Multi-modal target identification and tracking prediction method based on laser radar and binocular camera

Abstract

The invention relates to a multi-modal target identification and tracking prediction method based on a laser radar and a binocular camera, and belongs to the field of robot / unmanned system perception identification. Comprising the following steps: preprocessing a target two-dimensional image acquired by a binocular camera and target point cloud data acquired by a laser radar to obtain a preprocessed two-dimensional image and point cloud data; s2, performing two-dimensional target detection and tracking detection based on the preprocessed two-dimensional image to obtain a two-dimensional target detection and tracking result; s3, performing three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking result to obtain a three-dimensional detection result; and S4, based on the pixel coordinates in the two-dimensional target detection and tracking result and the pixel coordinates of the vertexes of the target point cloud boundary in the three-dimensional detection result, target fusion tracking prediction is carried out, and the target ID, the position, the category and the target point cloud boundary are obtained. And accurate identification and tracking of the target are realized.

Description

Technical Field

[0001] The present invention relates to the field of robot / unmanned system perception and recognition technology, and in particular to a multimodal target recognition and tracking prediction method based on laser radar and binocular camera. Background Art

[0002] Target recognition and tracking technology is mainly used in the front-end perception module of unmanned systems, which is responsible for accurately identifying and classifying designated target objects in the environment and continuously tracking them.

[0003] Relying only on the image information of the camera for target recognition and tracking, due to the lack of depth features, only the position information of the target in pixel coordinates can be obtained. Although the depth information in the environment can be extracted through inter-frame matching of monocular cameras, parallax of binocular cameras, or direct introduction of depth cameras, this method has low accuracy at the depth scale in larger scenes or scenes where the unmanned system moves rapidly, and there are large errors in the recognition and positioning of distant targets. At the same time, the depth extraction method based on matching needs to reduce the gap between matching frames and increase the matching frequency under the condition of rapid movement of the unmanned system, which leads to a slow running speed and is difficult to meet the real-time requirements of the actual operation of the unmanned system.

[0004] As the most commonly used 3D sensor in the unmanned field, LiDAR is often used in 3D target detection. Different from 2D image detection, 3D target detection based on LiDAR needs to process point cloud information with fewer features and more complex structures. Based on the performance of LiDAR, 3D target detection has higher accuracy in depth scale in larger scenes. Therefore, compared with 2D target detection methods, the positioning accuracy of the detected target is higher, and because point cloud features are simpler than image features, 3D target detection can have a higher working frequency. There are currently two mainstream 3D target detection methods. One is to project point cloud information into a bird's-eye view and detect it through a 2D detection framework. The defect of this method is that it is very dependent on the detection and tracking accuracy of the 2D detection framework; the other is to project the point cloud into regular 3D voxels for identification and tracking. The defect of this method is that the constructed voxels are not necessarily optimal, and information loss is likely to occur during the quantization process, and the final detection result is not necessarily accurate. Another mainstream method is to directly perform target detection on the original point cloud based on a deep learning network, refine the 3D detection method in the point cloud by combining semantic features and local spatial features, and finally generate the 3D bounding box of the target directly in the original point cloud. The drawback of this learning-based method is that the original point cloud information is sparse, and the feature extraction process in the network is often not efficient enough.

[0005] The above two-dimensional and three-dimensional target detection and tracking methods have the following limitations: the single two-dimensional target detection framework lacks depth scale and is difficult to adapt to the detection and positioning of targets in larger scenes; the mainstream projection-based three-dimensional detection method has poor accuracy, and the three-dimensional detection method based on raw point clouds is not efficient enough. Summary of the invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide a steel enterprise production process simulation system based on multi-agent, so as to solve the technical problems of the existing method of missing two-dimensional detection depth scale and inefficient three-dimensional detection point cloud processing.

[0007] The purpose of the present invention is mainly achieved through the following technical solutions:

[0008] The present invention provides a multimodal target recognition and tracking prediction method based on laser radar and binocular camera, comprising the following steps:

[0009] Step S1, preprocessing the target two-dimensional image acquired by the binocular camera and the target point cloud data acquired by the laser radar to obtain the preprocessed two-dimensional image and point cloud data;

[0010] Step S2: performing two-dimensional target detection and tracking detection based on the preprocessed two-dimensional image to obtain two-dimensional target detection and tracking results;

[0011] Step S3, performing three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain a three-dimensional detection result;

[0012] Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, target fusion tracking prediction is performed to obtain the target ID, position, category and target point cloud boundary.

[0013] Furthermore, the step S2 comprises:

[0014] The preprocessed multiple frames of two-dimensional images are sequentially input into a one-stage detection model to perform two-dimensional target detection, and obtain a two-dimensional target detection result; the two-dimensional target detection result includes the target included in each frame of the two-dimensional image, the category of the target, and the pixel coordinates and detection confidence of the corresponding target;

[0015] Based on the detection confidence, the two-dimensional target detection result corresponding to each frame of the image is divided into a high-confidence target, a low-confidence target, and a target between the high-confidence and the low-confidence;

[0016] The pixel coordinates of the high-confidence target are input into the Kalman filter. The Kalman filter predicts the new position of the target in the current frame according to the pixel coordinates of the historical positions of each high-confidence target, obtains the predicted detection frame position of the high-confidence target, and defines an ID for each high-confidence target;

[0017] Based on the preprocessed two-dimensional image of the current frame, obtain the real detection frame positions of all targets in the current frame;

[0018] Based on the real detection frames of the current frame and the predicted detection frames corresponding to the previous frame, and in combination with detecting new targets based on the low-confidence targets, the Hungarian algorithm is used to complete continuous tracking of the same target to obtain two-dimensional target detection and tracking results.

[0019] Furthermore, the first-stage target detection model uses YOLOv8 combined with NMS;

[0020] The YOLOv8 first performs an initial detection on the preprocessed two-dimensional image to obtain multiple bounding boxes and corresponding target categories, pixel coordinates, and confidence levels;

[0021] The NMS performs post-processing on the YOLOv8 output to suppress redundant bounding boxes, and obtains a two-dimensional object detection result by retaining the bounding box with the highest confidence and removing other bounding boxes with high overlap.

[0022] Furthermore, based on each real detection frame of the current frame and each predicted detection frame of the previous frame, the Hungarian algorithm is used to sequentially calculate the IOU values ​​of each real detection frame of the current frame and each predicted detection frame of the previous frame;

[0023] The target of the predicted detection frame of the previous frame with the largest IOU value with the real detection frame is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames;

[0024] If there is a target in the current frame that cannot find a matching target in the previous frame, it is considered that the target is lost, and it is necessary to call all the low-confidence targets in the most recent P frame that are currently saved and match them with the current frame target that failed to match using the Hungarian algorithm again; if the match is successful, the current frame target will inherit the ID of the low-confidence target to achieve re-identification of the lost target;

[0025] If a matching target still cannot be found, the current frame target is assigned a new ID as the newly appeared target.

[0026] Further, the pre-processed point cloud data of the latest received i-frame and the two-dimensional target detection and tracking results are subjected to three-dimensional target detection, including:

[0027] Predefine the prior geometric information of each target category;

[0028] Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking result, the length and width of the two-dimensional detection frame of the target in the pixel coordinates are calculated;

[0029] Based on the prior geometric information and the length and width of the target's two-dimensional detection box in pixel coordinates, the left side of the target pixel is converted to a 3D coordinate position P by similar coordinate transformation. c1 (x1,y1,z1); with P c1 (x1, y1, z1) is the center coordinate, and a three-dimensional bounding box Box1 is generated based on the length, width and height preset based on the number of targets in the detection area and the complexity of the scene;

[0030] Traverse all points in the point cloud of the current frame. If the point is in the bounding box Box1, it is classified as a point to be detected.

[0031] Performing Euclidean clustering on all the points to be detected in the boundary box Box1 to obtain multiple point cloud clusters, screening out a point cloud cluster with the highest credibility from the multiple point cloud clusters, and constructing a three-dimensional boundary box Box2 based on the boundary of the screened point cloud cluster and the length, width and height of the prior geometric information of the target category to obtain a three-dimensional detection result;

[0032] The three-dimensional detection result is a three-dimensional bounding box Box2.

[0033] Furthermore, selecting a point cloud cluster with the highest credibility from the multiple point cloud clusters includes:

[0034] Based on the scale of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The distance of (x1, y1, z1) is weighted to obtain the weighted value of each point cloud cluster;

[0035] Filter out the point cloud cluster with the highest weighted value.

[0036] Further, determine the 3D position P of the center of the 3D bounding box Box2 in Box1 c2 (x2,y2,z2);

[0037] The P c2 (x2, y2, z2) and Box2 constitute the final three-dimensional detection result.

[0038] Furthermore, the step S4 comprises:

[0039] Initialize the extended Kalman filter EKF and set the initial state and covariance matrix of the target;

[0040] Based on the initial state and covariance matrix of the target, weighted fusion is performed on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results to obtain fused target state information;

[0041] Use the state transition model and process noise to predict the next state of the target and update the covariance matrix;

[0042] Compare the predicted next state of the target with the observed value of the target in the actual three-dimensional detection, calculate the Kalman gain, and update the target state and covariance matrix, and output the state estimation and covariance matrix of the target at time k+1;

[0043] Process 2D and 3D detection results in parallel at each time step, and use EKF to fuse and update the detection results to continuously track and locate the target;

[0044] Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box Box2.

[0045] Furthermore, the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted fused as follows:

[0046]

[0047] in, is the target state information after weighted fusion; α and β are the weight coefficients of 3D and 2D detection respectively; The pixel coordinates of the 8 vertices of the bounding box of Box2 for 3D detection; Predicted target pixel coordinates for 2D tracking.

[0048] Furthermore, the geometric prior information includes the shape and size of the target category, and the size includes the prior length, width and height of the target.

[0049] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0050] 1. The present invention determines the exact position of the target point cloud through two detection boxes Box1 and Box2 of different sizes based on point cloud observation and two-dimensional detection results; by using the precise three-dimensional spatial information provided by the laser radar and the rich two-dimensional image details provided by the binocular camera, the present invention can realize accurate recognition and positioning of the target, especially in complex environments, significantly improving the detection accuracy compared with a single sensor system;

[0051] 2. Fusing two-dimensional image information and three-dimensional point cloud information for target tracking prediction, using the target state information obtained by weighted fusion to correct the prediction process of the extended Kalman filter and optimize the tracking prediction results; by fusing the two-dimensional and three-dimensional detection results with the extended Kalman filter, the present invention can maintain stable tracking when the target moves quickly or is partially blocked, and reduce target tracking loss caused by environmental interference;

[0052] 3. Based on two-dimensional detection, the present invention integrates three-dimensional point cloud information for target detection, which can make up for the lack of depth scale when using only a camera for detection and obtain more accurate point cloud target positioning information;

[0053] 4. The present invention directly uses the original point cloud data after voxel downsampling, without the need for projection processing of the point cloud, and can retain more target features. It uses a non-learning method for 3D target detection, clusters through two three-dimensional detection boxes, and quickly locates the target position, which has higher real-time performance;

[0054] 5. The method of the present invention is designed with the needs of real-time processing in mind. Through optimized data processing procedures and algorithms, it ensures that fast data processing and response can be achieved even in scenarios where unmanned systems move quickly or targets frequently appear and disappear. The length, width, and height dimensions of the three-dimensional bounding box Box1 are dynamically adjusted according to the number of targets in the detection area and the complexity of the scene. The present invention can more flexibly adapt to different detection environments, optimize detection performance, and reduce false detections and missed detections;

[0055] 6. Through accurate fusion of two-dimensional and three-dimensional data and effective target tracking prediction, the present invention reduces the demand for computing resources and improves the working efficiency of the system, making the method suitable for unmanned systems with limited resources;

[0056] 7. By optimizing the target matching process using the Hungarian algorithm, the present invention can accurately distinguish and track multiple targets in a multi-target tracking scenario, and can maintain accurate tracking even when the targets are close to or intersecting each other;

[0057] 8. The method of the present invention is not limited to a specific type of environment or target, but has wide applicability and can adapt to a variety of different application scenarios and be flexibly applied across scenarios, such as autonomous driving, robot navigation, intelligent monitoring, etc.

[0058] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0060] Figure 1 This is a flow chart of a multimodal target recognition and tracking prediction method based on laser radar and binocular camera in an embodiment of the present invention;

[0061] Figure 2 Schematic diagram of a two-dimensional target tracking prediction process in an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of two-stage target detection in three-dimensional target detection in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0064] The present invention aims to solve the technical problems of missing depth scale in two-dimensional detection and inefficient point cloud processing in three-dimensional detection by improving the fusion method of two-dimensional and three-dimensional target detection, so as to realize accurate recognition and tracking and positioning of any target in a large scene.

[0065] In the present invention, the binocular camera provides rich two-dimensional image information, and the laser radar provides high-precision three-dimensional spatial information. The integrated use of the two can significantly improve the environmental perception capability and target tracking accuracy of the unmanned system.

[0066] In the present invention, binocular cameras and laser radars are used as perception modules of unmanned systems or robots. For example, these sensors are installed in unmanned system platform application scenarios such as unmanned vehicles, drones, automated guided vehicles, and service robots.

[0067] A specific embodiment of the present invention discloses a multi-modal target recognition and tracking prediction method based on laser radar and binocular camera, such as Figure 1 As shown, the following steps are included:

[0068] Step S1, preprocessing the target two-dimensional image acquired by the binocular camera and the target point cloud data acquired by the laser radar to obtain the preprocessed two-dimensional image and point cloud data;

[0069] Step S2: performing two-dimensional target detection and tracking detection based on the preprocessed two-dimensional image to obtain two-dimensional target detection and tracking results;

[0070] Step S3, performing three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain a three-dimensional detection result;

[0071] Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, target fusion tracking prediction is performed to obtain the target ID, position, category and target point cloud boundary.

[0072] Step S1 is divided into steps S11-S12, specifically.

[0073] Step S11: The binocular camera continuously acquires a target two-dimensional image and performs preprocessing to obtain a preprocessed two-dimensional image.

[0074] The binocular camera is continuously called to capture the two-dimensional image of the target, and the two-dimensional image is preprocessed.

[0075] Scale the 2D image size to the input layer size of the one-stage object detection model as the input for the subsequent 2D object detection step.

[0076] Exemplarily, the two-dimensional image size is scaled to 640×480 pixels.

[0077] Step S12: simultaneously using the laser radar to acquire target point cloud data, and preprocessing the point cloud data to obtain preprocessed point cloud data.

[0078] The original point cloud data of the laser radar is received and preprocessed, and voxel grid downsampling (Voxel Grid Downsampling) is performed to obtain the preprocessed point cloud data. The preprocessed point cloud data is the original point cloud data after voxel grid downsampling, which is still the original point cloud, but the number of point clouds is reduced.

[0079] The acquired raw point cloud data is downsampled to a voxel grid to reduce the number of raw point clouds while retaining important spatial information.

[0080] The purpose of step S1 is to provide standardized, high-quality data input for subsequent multimodal target recognition and tracking prediction by preprocessing and downsampling the binocular camera image and lidar point cloud data.

[0081] The step S2 comprises:

[0082] The preprocessed multiple frames of two-dimensional images are sequentially input into a one-stage detection model to perform two-dimensional target detection, and obtain a two-dimensional target detection result; the two-dimensional target detection result includes the target included in each frame of the two-dimensional image, the category of the target, and the pixel coordinates and detection confidence of the corresponding target;

[0083] Based on the detection confidence, the two-dimensional target detection result corresponding to each frame of the image is divided into a high-confidence target, a low-confidence target, and a target between the high-confidence and the low-confidence;

[0084] The pixel coordinates of the high-confidence target are input into the Kalman filter. The Kalman filter predicts the new position of the target in the current frame according to the pixel coordinates of the historical positions of each high-confidence target, obtains the predicted detection frame position of the high-confidence target, and defines an ID for each high-confidence target;

[0085] Based on the preprocessed two-dimensional image of the current frame, obtain the real detection frame positions of all targets in the current frame;

[0086] Based on the real detection frames of the current frame and the predicted detection frames corresponding to the previous frame, and in combination with detecting new targets based on the low-confidence targets, the Hungarian algorithm is used to complete continuous tracking of the same target to obtain two-dimensional target detection and tracking results.

[0087] Step S2 is divided into steps S21-S22.

[0088] Step S21: Perform two-dimensional target detection based on the preprocessed two-dimensional image of the target.

[0089] The preprocessed multiple frames of two-dimensional images are sequentially input into the one-stage detection model for two-dimensional target detection to obtain the two-dimensional target detection results.

[0090] Exemplarily, the first-stage target detection model uses a YOLO (You Only Look Once) model, which is integrated on a computing device.

[0091] Initialize the two-dimensional detector. During the initialization process, maintain a detection result class to store the two-dimensional target detection results output by the one-stage target detection model.

[0092] This class contains the target category, 4 pixel coordinates (x, y), and detection confidence. The pixel coordinates are the coordinates of the four intersection points of the detected target bounding box (x min ,y min )、(x max ,y min )、(x min ,y max )、(x max ,y max ), the detection confidence of the target is mainly evaluated by whether there is an object in the target bounding box and the intersection and union ratio of the target predicted bounding box and the target real bounding box.

[0093] The two-dimensional detector is a soft module integrated on the computing device.

[0094] Exemplarily, the present invention adopts the YOLOv8 version, which is integrated on the computing device.

[0095] The one-stage object detection model adopts YOLOv8 combined with NMS;

[0096] The YOLOv8 first performs an initial detection on the preprocessed two-dimensional image to obtain multiple bounding boxes and the corresponding object categories, pixel coordinates, and confidence levels;

[0097] The NMS performs post-processing on the output of the YOLOv8 to suppress redundant bounding boxes. By retaining the bounding box with the highest confidence level and removing other bounding boxes with high overlap with it, a two-dimensional object detection result is obtained.

[0098] The YOLO model first takes the preprocessed target two-dimensional image as input, then performs model inference on the input image, and performs non-maximum suppression NMS (Non-Maximum Suppression) on the basis of the model output result to screen out the final two-dimensional object detection result. The output two-dimensional object detection result includes the category of the object, 4 pixel coordinates, and detection confidence level, which are saved and maintained in the detection result class.

[0099] According to the detection confidence level, it is divided into high-confidence objects (confidence > m) and low-confidence objects (confidence < n), and objects between high confidence and low confidence;

[0100] m and n are preset values, and m > n; Exemplarily, m = 0.6 and n = 0.5.

[0101] The non-maximum suppression NMS is integrated on the computing device.

[0102] The YOLO model is an object detection model mainly used in the field of object detection. It can identify the object categories and positions in the image through one scan. It is a real-time object detection algorithm that can provide a high processing speed while ensuring the detection quality. The characteristics of the YOLO model include its simplicity and efficiency. It does not need to first find the possible object regions and then classify them like traditional object detection methods. The YOLO model outputs the category of the object, the detection bounding box, the detection confidence level, and the category.

[0103] NMS is a post-processing technique used to filter and optimize the output results of the YOLO model. In object detection, NMS is used to solve the problem of multiple bounding boxes surrounding the same object. NMS is used to suppress redundant bounding boxes, reducing redundancy and false detections by retaining the bounding box with the highest confidence and removing other bounding boxes with high overlap. After NMS processing, redundant bounding boxes can be effectively removed, retaining the bounding box that is most likely to contain the object.

[0104] The relationship between YOLO and NMS:

[0105] (1) Complementarity: YOLO may generate multiple overlapping bounding boxes when detecting an object, especially when the object is partially occluded in the image or there are similar objects near the object. NMS, as a post-processing step of YOLO output, helps YOLO improve detection accuracy and reduce redundancy by suppressing unnecessary bounding boxes.

[0106] (2) Process order: In the object detection process of YOLO, the preprocessed two-dimensional image is first processed using the YOLO model to obtain a series of bounding boxes and corresponding confidence scores. Then, NMS is applied to these outputs to remove overlapping bounding boxes and retain the bounding boxes that are most likely to contain the object.

[0107] Function: NMS helps to improve the overall quality of YOLO detection results, reduce false detections and duplicate detections, and thus improve the precision and recall of the YOLO model.

[0108] The output of 2D object detection: object category, 4 pixel coordinates, and detection confidence.

[0109] The purpose of using a binocular camera for two-dimensional target detection is to provide the unmanned system with basic information such as the target category, pixel coordinate position and detection confidence, so that subsequent steps can perform accurate target tracking and precise positioning in three-dimensional space.

[0110] Step S22: Track and predict the two-dimensional target.

[0111] Two-dimensional target tracking prediction adopts a target matching and tracking method based on detection confidence.

[0112] like Figure 2 As shown, the two-dimensional target detection result of a single frame image is used as input to continuously track and match high-confidence targets;

[0113] Save the low-confidence target. When the subsequent tracking target is lost, call the currently saved low-confidence target again for re-identification and tracking (such as Figure 2 The dashed line part shows the

[0114] At most low-confidence targets in the most recent P-frame detection results are saved; illustratively, P=30.

[0115] Unclassified categories (i.e., targets with confidence between m and n) are not processed in the current frame. For example, unclassified categories, such as categories with confidence between 0.6 and 0.5, are not processed in the current frame.

[0116] (1) For high confidence thresholds (e.g. > 0.6):

[0117] Targets with detection confidence greater than 0.6 are considered reliable enough to be directly used for target tracking. This is because the detection results of these targets have a high probability of being correct and can therefore be directly used in the subsequent tracking process;

[0118] (2) For low confidence thresholds (e.g. <0.5):

[0119] Targets with confidence less than 0.5 are considered unreliable and are not directly useful for tracking. However, the detection results of these targets are not completely ignored, but saved. When a loss occurs during tracking (i.e. the tracked target cannot be detected in subsequent frames), these low-confidence targets can be used as potential candidates for reinitialization or resumption of tracking;

[0120] (3) For the medium confidence range (between 0.5 and 0.6):

[0121] Targets with confidence levels between 0.5 and 0.6 are usually in a "gray area" and are neither directly used for tracking nor completely ignored. There are many false positives in this range and they occur very frequently. Generally, it is more efficient to use a smaller confidence level for re-identification.

[0122] Not using targets with confidence levels between 0.5 and 0.6 for tracking detection is to strike a balance between tracking accuracy and computing resources while avoiding introducing too many potential false tracking. This design decision is usually based on the needs and limitations of a specific application scenario.

[0123] This step is based on the target list after confidence classification, and obtains a list of high-confidence targets and low-confidence targets. Prepare the target list for subsequent tracking prediction and determine which targets need to be tracked.

[0124] Perform preliminary tracking prediction of the target. In this part, the Kalman filter is introduced. The high-confidence target pixel coordinates of the historical frame are input into the Kalman filter. The Kalman filter predicts the new position of the target in the current frame based on the pixel coordinate data of the historical position of each target, and obtains the predicted detection box position of all targets in the current frame in the next frame and the ID of each target.

[0125] Based on the real detection frames of the current frame and the predicted detection frames of the previous frame, the Hungarian algorithm is used to calculate the IOU (Intersection over Union) values ​​of the real detection frames of the current frame and the predicted detection frames of the previous frame in sequence;

[0126] The target of the predicted detection frame of the previous frame with the largest IOU value with the real detection frame is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames;

[0127] If there is a target in the current frame that cannot find a matching target in the previous frame, it is considered that the target is lost, and it is necessary to call all the low-confidence targets in the most recent P frame that are currently saved and match them with the current frame target that failed to match using the Hungarian algorithm again; if the match is successful, the current frame target will inherit the ID of the low-confidence target to achieve re-identification of the lost target;

[0128] If a matching target still cannot be found, the current frame target is assigned a new ID as the newly appeared target.

[0129] Continue to obtain the next frame of image from the binocular camera as the current frame and repeat all the above work. After completing the two-dimensional target detection of this frame of image,

[0130] The true detection box positions of all objects in the current frame and the predicted detection box positions of all objects in the previous frame;

[0131] The IOU features of each detection box in the current frame and each predicted detection box in the previous frame are calculated in sequence to measure the matching degree of each target between frames.

[0132] Get the target matching result of the k+1th frame. Output all the identified targets of the k+1th frame, and the information of each target includes: target ID, 4 pixel coordinates, and detection confidence.

[0133] The role of step S2 is to achieve accurate recognition, positioning and continuous tracking of targets in two-dimensional images by combining the target detection of the YOLO model and the prediction of the Kalman filter, and to optimize the target matching process to improve the target tracking performance of the unmanned system in complex environments.

[0134] Step S3, specifically.

[0135] Performing three-dimensional target detection on the pre-processed point cloud data of the latest received i-frame and the two-dimensional target detection and tracking results, including:

[0136] Predefine the prior geometric information of each target category;

[0137] Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking result, the length and width of the two-dimensional detection frame of the target in the pixel coordinates are calculated;

[0138] Based on the prior geometric information and the length and width of the target's two-dimensional detection box in pixel coordinates, the left side of the target pixel is converted to a 3D coordinate position P by similar coordinate transformation. c1 (x1,y1,z1); with P c1 (x1, y1, z1) is the center coordinate, and a three-dimensional bounding box Box1 is generated based on the length, width and height preset based on the number of targets in the detection area and the complexity of the scene;

[0139] Traverse all points in the point cloud of the current frame. If the point is in the bounding box Box1, it is classified as a point to be detected.

[0140] Performing Euclidean clustering on all the points to be detected in the boundary box Box1 to obtain multiple point cloud clusters, screening out a point cloud cluster with the highest credibility from the multiple point cloud clusters, and constructing a three-dimensional boundary box Box2 based on the boundary of the screened point cloud cluster and the length, width and height of the prior geometric information of the target category to obtain a three-dimensional detection result;

[0141] The three-dimensional detection result is a three-dimensional bounding box Box2.

[0142] The planned prior information of each target category is pre-defined.

[0143] The geometric prior information includes the shape and size of the target category, and the size includes the prior length, width and height of the target.

[0144] The three-dimensional target detection method of the present invention is performed in an original point cloud environment with voxel grid downsampling, and is mainly divided into two detection stages: a preliminary detection stage and a secondary detection stage.

[0145] The most recently received i-frame point cloud is saved and voxel grid downsampled and maintained.

[0146] i is a preset value, and illustratively, i is 5. Every 5 frames of point cloud are processed once, that is, each frame of point cloud is firstly downsampled by voxel grid, and then the point cloud data after 5 frames of voxel downsampling are retained.

[0147] First, the preliminary detection stage:

[0148] The preliminary detection is based on the downsampled point cloud data and 2D object detection and tracking results.

[0149] This step uses the similar coordinate transformation method to make a preliminary prediction of the target's three-dimensional information in the processed point cloud scene.

[0150] Input the 2D target detection and tracking results (4 intersection pixel coordinates), including the length and width of the target's 2D detection box in pixel coordinates, and use the 4 intersection pixel coordinates of the 2D detection box to calculate the length and width of the 2D detection box in pixel coordinates.

[0151] Output the 3D coordinate position P of the target object after similar coordinate transformation c1 (x1,y1,z1), as shown in formula (1).

[0152]

[0153] Among them, z is the depth scale information of the target, that is, z1; w and h are the prior width and prior height of the target respectively; w bbox ,h bbox are the width and length of the two-dimensional detection box of the target respectively; f x , f y are the focal length in the x direction and the focal length in the y direction respectively; α is the width and height weight coefficient; C lidar is the coordinate of the target in the radar coordinate system; R cam2pix is the rotation matrix from the pixel coordinate system to the camera coordinate system; R lidar2cam is the rotation matrix from the camera coordinate system to the radar coordinate system; p pix is the pixel coordinate of the center point of the target detection frame, which is converted to the radar coordinate system to obtain (x1, y1); cam2lidar is the offset vector from the camera coordinate system to the radar coordinate system; Q odom is the quaternion rotation matrix from the lidar coordinate system to the world coordinate system; p odom is the position vector of the laser radar in the world coordinate system; C world It is the coordinate information of the target in the world coordinate system.

[0154] In addition, the three-dimensional detection information is defined by a larger three-dimensional bounding box Box1, wherein the boundary information of the bounding box, including length, width and height, is preset, such as Figure 3 Shown in the dashed box.

[0155] If there are many objects in the detection area or the scene is complex, the length, width and height of the bounding box should be preset to be smaller.

[0156] Otherwise, if there are fewer objects in the detection area and the scene complexity is low, the length, width and height of the bounding box are preset to be larger.

[0157] The length L, width W and height H of Box1 are adjusted according to the number of targets N and the scene complexity C, as shown in formula (2):

[0158]

[0159] Among them, L max , W max and H max are the maximum length, width and height of the target scene respectively; η, μ and θ are the adjustment coefficients of length L, width W and height H respectively; f(N,C) is a function, which is defined according to the actual needs of the number of targets and the complexity of the scene. It can be a simple linear function or a model based on deep learning, which is used to more accurately reflect the impact of the number of targets and the complexity of the scene on the size of Box1.

[0160] For Box1, P c1 (x1, y1, z1) are the center coordinates, and a three-dimensional bounding box Box1 is generated with the preset length, width and height.

[0161] The three-dimensional coordinate position Pc1 and the three-dimensional bounding box Box1 of the target object are preliminarily predicted.

[0162] Based on the pixel coordinates of the four intersection points of the two-dimensional detection box, Pc1(x1, y1, z1) is obtained. At this time, the detection result of each target includes: target ID, target category, four two-dimensional pixel coordinates, detection confidence, Pc1(x1, y1, z1), Box1.

[0163] Continue to the secondary detection stage:

[0164] The three-dimensional bounding box Box1 obtained in the preliminary detection stage and the 3D coordinate position P of the target object preliminarily predicted c1 (x1,y1,z1).

[0165] Traverse all points in the point cloud of the current frame. If the point is in the bounding box Box1, it is classified as a point to be detected.

[0166] Perform Euclidean clustering on all points to be detected in the bounding box Box1 to obtain multiple point cloud clusters such as Figure 3 The shaded areas are shown in .

[0167] The step of selecting a point cloud cluster with the highest credibility from the multiple point cloud clusters includes:

[0168] Based on the scale of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The distance of (x1, y1, z1) is weighted to obtain the weighted value of each point cloud cluster;

[0169] Filter out the point cloud cluster with the highest weighted value.

[0170] According to the size of the point cloud cluster after clustering (the number of points in the point cloud) and the distance from the center point of the bounding box Box1, a weighted calculation is performed to select the point cloud cluster with the highest credibility, such as Figure 3 The upper right corner is marked with P c2 As shown in formula (3), the point cloud cluster centered on is shown.

[0171]

[0172] Among them, P j is the jth point cloud cluster; S j is the point cloud cluster P j The number of points in C j Represents the point cloud cluster P j The center point of D j is the center point C of the point cloud cluster j The distance from the center point Box1 of the bounding box; They are the center points C of the point cloud clusters. j The coordinates of the center point of the bounding box Box1; W j Represents the point cloud cluster P j The credibility of ; λ is the weight parameter.

[0173] Based on the boundary of the filtered point cloud cluster, a smaller three-dimensional bounding box Box2 is maintained, such as Figure 3 As shown in the solid box, the 3D bounding box center P c2 (x2, y2, z2) is used as the final detection and positioning result of the target.

[0174] Determine the 3D position P of the center of the 3D bounding box Box2 within Box1 c2 (x2,y2,z2);

[0175] The P c2 (x2, y2, z2) and Box2 constitute the final three-dimensional detection result.

[0176] The final 3D detection result includes the 3D position of the target and the corresponding bounding box Box2, where the data of Box2 is the coordinates of the 8 vertices of the cube Box2.

[0177] The step S4 comprises:

[0178] Initialize the extended Kalman filter EKF and set the initial state and covariance matrix of the target;

[0179] Based on the initial state and covariance matrix of the target, weighted fusion is performed on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results to obtain fused target state information;

[0180] Use the state transition model and process noise to predict the next state of the target and update the covariance matrix;

[0181] Compare the predicted next state of the target with the observed value of the target in the actual three-dimensional detection, calculate the Kalman gain, and update the target state and covariance matrix, and output the state estimation and covariance matrix of the target at time k+1;

[0182] Process 2D and 3D detection results in parallel at each time step, and use EKF to fuse and update the detection results to continuously track and locate the target;

[0183] Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box Box2.

[0184] By fusing the 2D image information (the pixel coordinates of the 4 intersection points of the 2D bounding box) and the 3D point cloud information (the pixel coordinates of the 8 vertices of Box2), the target position is tracked and predicted again in the point cloud scene.

[0185] (1) First initialize the extended Kalman filter EKF

[0186] In the secondary detection stage of three-dimensional target detection, an extended Kalman filter (EKF) is introduced as a parallel process of the secondary detection stage of three-dimensional target detection.

[0187] The target fusion tracking prediction system is a soft module integrated on the computing device. Before the work starts, the Kalman filter is initialized and the initial state and covariance matrix of the target are set.

[0188]

[0189] in, is the predicted state of the target at time k, is the prediction covariance matrix at time k; x init is the preset initial value of the target state, P init is the preset initial value of the covariance matrix, and k is the timestamp.

[0190] (2) Weighted fusion of target states

[0191] The target state information predicted by 2D tracking (the pixel coordinates of the 4 intersection points of the 2D detection box) and the 3D detection state information (the pixel coordinates of the 8 intersection points of Box2).

[0192] After completing the secondary detection phase and identifying and locating the target object, the Kalman filter continuously maintains the target position (pixel coordinates). Before the EKF prediction phase, the target state information (4 pixel coordinates) predicted by two-dimensional tracking and the three-dimensional detection state information (8 corner point cloud coordinates of the Box2 bounding box) are weighted fused as the target state information of the prediction phase.

[0193] The pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted fused, as shown in formula (5):

[0194]

[0195] in, is the target state information after weighted fusion; α and β are the weight coefficients of 3D and 2D detection respectively; The pixel coordinates of the 8 vertices of the bounding box of Box2 for 3D detection; Predicted target pixel coordinates for 2D tracking.

[0196] The fused target state information is shown in formula (6).

[0197]

[0198] in, Indicates the target state information of 3D detection represents the target state information of two-dimensional tracking prediction, Represents the target state information after weighted fusion (the four pixel coordinates of the two-dimensional detection box and the pixel coordinates of the eight vertices of Box1 are combined into a vector); (x min ,x max ,y min ,y max ,z min ,z max ) represents the specific coordinates in the 3D detection state information (using the x, y, z range of the bounding box to maintain the coordinates of the 8 bounding box corner points); Represents the specific coordinates in the target state information predicted by two-dimensional tracking (using the x and y ranges of the detection box to maintain the coordinates of the intersection of the four detection boxes);

[0199] This part obtains the fused target state information;

[0200] (3) Predict the next state of the target.

[0201] Based on the fused target state information, the state transition model and process noise are used to predict the next state of the target and update the covariance matrix, as shown in formula (7).

[0202]

[0203] in, is the predicted state of the target at time k+1, F k is the state transfer matrix at time k, is the prediction covariance matrix at time k+1, Q k is the process noise covariance matrix at time k.

[0204] Output the predicted target next state.

[0205] (4) Update the target status.

[0206] Based on the predicted next target state and the observed values ​​of the target in the actual 3D detection.

[0207] After predicting the next target state, the predicted value is compared with the observed value of the target in the actual three-dimensional detection, and the Kalman gain is calculated;

[0208] Get the Kalman gain K k+1 After that, the target state and covariance matrix are updated as shown in formula (8).

[0209]

[0210] Among them, H k+1 is the observation matrix, H k+1 The transposed matrix, R k+1 is the observation noise covariance matrix, K k+1 is the Kalman gain.

[0211] The state estimation of the target at time k+1 is shown in formula (9).

[0212]

[0213] Among them, x k+1 is the updated target state estimate after time k+1, P k+1 is the updated covariance matrix after time k+1, z k+1 is the target observation data at time k+1 (i.e., the position of the target in the global coordinate system at time k+1), and I is the unit matrix.

[0214] Get the state estimate of the target at time k+1 and the covariance matrix at time k+1.

[0215] (5) Parallel processing and continuous tracking.

[0216] Output results based on 2D and 3D detection.

[0217] Each detector works in parallel with the filter, and the detector completes two-dimensional and three-dimensional detection of the target;

[0218] At the same time, the extended Kalman filter EKF completes the fusion and update of the output results of the above two detectors;

[0219] By repeatedly predicting and updating the target at each time step, the target can be continuously tracked and located.

[0220] The final output is the position of the target in the world coordinate system, the target category, the point cloud bounding box Box2, and the tracking target ID at the current moment.

[0221] In summary, the multimodal target recognition and tracking prediction method based on laser radar and binocular camera according to the embodiment of the present invention has the following beneficial effects:

[0222] 1. The present invention determines the exact position of the target point cloud through two detection boxes Box1 and Box2 of different sizes based on point cloud observation and two-dimensional detection results; by using the precise three-dimensional spatial information provided by the laser radar and the rich two-dimensional image details provided by the binocular camera, the present invention can realize accurate recognition and positioning of the target, especially in complex environments, significantly improving the detection accuracy compared with a single sensor system;

[0223] 2. Fusing two-dimensional image information and three-dimensional point cloud information for target tracking prediction, using the target state information obtained by weighted fusion to correct the prediction process of the extended Kalman filter and optimize the tracking prediction results; by fusing the two-dimensional and three-dimensional detection results with the extended Kalman filter, the present invention can maintain stable tracking when the target moves quickly or is partially blocked, and reduce target tracking loss caused by environmental interference;

[0224] 3. Based on two-dimensional detection, the present invention integrates three-dimensional point cloud information for target detection, which can make up for the lack of depth scale when using only a camera for detection and obtain more accurate point cloud target positioning information;

[0225] 4. The present invention directly uses the original point cloud data after voxel downsampling, without the need for projection processing of the point cloud, and can retain more target features. It uses a non-learning method for 3D target detection, clusters through two three-dimensional detection boxes, and quickly locates the target position, which has higher real-time performance;

[0226] 5. The method of the present invention is designed with the needs of real-time processing in mind. Through optimized data processing procedures and algorithms, it ensures that fast data processing and response can be achieved even in scenarios where unmanned systems move quickly or targets frequently appear and disappear. The length, width, and height dimensions of the three-dimensional bounding box Box1 are dynamically adjusted according to the number of targets in the detection area and the complexity of the scene. The present invention can more flexibly adapt to different detection environments, optimize detection performance, and reduce false detections and missed detections;

[0227] 6. Through accurate fusion of two-dimensional and three-dimensional data and effective target tracking prediction, the present invention reduces the demand for computing resources and improves the working efficiency of the system, making the method suitable for unmanned systems with limited resources;

[0228] 7. By optimizing the target matching process using the Hungarian algorithm, the present invention can accurately distinguish and track multiple targets in a multi-target tracking scenario, and can maintain accurate tracking even when the targets are close to or intersecting each other;

[0229] 8. The method of the present invention is not limited to a specific type of environment or target, but has wide applicability and can adapt to a variety of different application scenarios and be flexibly applied across scenarios, such as autonomous driving, robot navigation, intelligent monitoring, etc.

[0230] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multimodal target recognition and tracking prediction method based on laser radar and binocular camera, characterized in that: The steps include: Step S1, preprocessing the target two-dimensional image acquired by the binocular camera and the target point cloud data acquired by the laser radar to obtain the preprocessed two-dimensional image and point cloud data; Step S2: performing two-dimensional target detection and tracking detection based on the preprocessed two-dimensional image to obtain two-dimensional target detection and tracking results; Step S3, performing three-dimensional target detection based on the preprocessed point cloud data and the two-dimensional target detection and tracking results to obtain a three-dimensional detection result; Step S4: Based on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the target point cloud boundary in the three-dimensional detection results, target fusion tracking prediction is performed to obtain the target ID, position, category and target point cloud boundary.

2. The method according to claim 1, characterized in that: The step S2 comprises: The preprocessed multiple frames of two-dimensional images are sequentially input into a one-stage detection model to perform two-dimensional target detection, and obtain a two-dimensional target detection result; the two-dimensional target detection result includes the target included in each frame of the two-dimensional image, the category of the target, and the pixel coordinates and detection confidence of the corresponding target; Based on the detection confidence, the two-dimensional target detection result corresponding to each frame of the image is divided into a high-confidence target, a low-confidence target, and a target between the high-confidence and the low-confidence; The pixel coordinates of the high-confidence target are input into the Kalman filter. The Kalman filter predicts the new position of the target in the current frame according to the pixel coordinates of the historical positions of each high-confidence target, obtains the predicted detection frame position of the high-confidence target, and defines an ID for each high-confidence target; Based on the preprocessed two-dimensional image of the current frame, obtain the real detection frame positions of all targets in the current frame; Based on the real detection frames of the current frame and the predicted detection frames corresponding to the previous frame, and in combination with detecting new targets based on the low-confidence targets, the Hungarian algorithm is used to complete continuous tracking of the same target to obtain two-dimensional target detection and tracking results.

3. The method according to claim 2, characterized in that: The first-stage target detection model uses YOLOv8 combined with NMS; The YOLOv8 first performs an initial detection on the preprocessed two-dimensional image to obtain multiple bounding boxes and corresponding target categories, pixel coordinates, and confidence levels; The NMS performs post-processing on the YOLOv8 output to suppress redundant bounding boxes, and obtains a two-dimensional object detection result by retaining the bounding box with the highest confidence and removing other bounding boxes with high overlap.

4. The method according to claim 2, characterized in that: Based on the real detection frames of the current frame and the predicted detection frames of the previous frame, the Hungarian algorithm is used to calculate the IOU values ​​of the real detection frames of the current frame and the predicted detection frames of the previous frame in sequence; The target of the predicted detection frame of the previous frame with the largest IOU value with the real detection frame is taken as the best matching object, and the target ID of the best matching object is inherited to achieve continuous tracking and matching of the same target between frames; If there is a target in the current frame that cannot find a matching target in the previous frame, it is considered that the target is lost, and it is necessary to call all the low-confidence targets in the most recent P frame that are currently saved and match them with the current frame target that failed to match using the Hungarian algorithm again; if the match is successful, the current frame target will inherit the ID of the low-confidence target to achieve re-identification of the lost target; If a matching target still cannot be found, the current frame target is assigned a new ID as the newly appeared target.

5. The method according to claim 1, characterized in that: Performing three-dimensional target detection on the pre-processed point cloud data of the latest received i-frame and the two-dimensional target detection and tracking results, including: Predefine the prior geometric information of each target category; Based on the pixel coordinates of the four intersection points of each target in the two-dimensional target detection and tracking result, the length and width of the two-dimensional detection frame of the target in the pixel coordinates are calculated; Based on the prior geometric information and the length and width of the target's two-dimensional detection box in pixel coordinates, the left side of the target pixel is converted to a 3D coordinate position P by similar coordinate transformation. c1 (x1,y1,z1); with P c1 (x1, y1, z1) is the center coordinate, and a three-dimensional bounding box Box1 is generated based on the length, width and height preset based on the number of targets in the detection area and the complexity of the scene; Traverse all points in the point cloud of the current frame. If the point is in the bounding box Box1, it is classified as a point to be detected. Performing Euclidean clustering on all the points to be detected in the boundary box Box1 to obtain multiple point cloud clusters, screening out a point cloud cluster with the highest credibility from the multiple point cloud clusters, and constructing a three-dimensional boundary box Box2 based on the boundary of the screened point cloud cluster and the length, width and height of the prior geometric information of the target category to obtain a three-dimensional detection result; The three-dimensional detection result is a three-dimensional bounding box Box2.

6. The method according to claim 5, characterized in that: The step of selecting a point cloud cluster with the highest credibility from the multiple point cloud clusters includes: Based on the scale of each point cloud cluster in the bounding box Box1 and its relationship with the center point P of the bounding box Box1 c1 The distance of (x1, y1, z1) is weighted to obtain the weighted value of each point cloud cluster; Filter out the point cloud cluster with the highest weighted value.

7. The method according to claim 5, characterized in that: Determine the 3D position P of the center of the 3D bounding box Box2 within Box1 c2 (x2,y2,z2); The P c2 (x2, y2, z2) and Box2 constitute the final three-dimensional detection result.

8. The method according to claim 1, characterized in that: The step S4 comprises: Initialize the extended Kalman filter EKF and set the initial state and covariance matrix of the target; Based on the initial state and covariance matrix of the target, weighted fusion is performed on the pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results to obtain fused target state information; Use the state transition model and process noise to predict the next state of the target and update the covariance matrix; Compare the predicted next state of the target with the observed value of the target in the actual three-dimensional detection, calculate the Kalman gain, and update the target state and covariance matrix, and output the state estimation and covariance matrix of the target at time k+1; Process 2D and 3D detection results in parallel at each time step, and use EKF to fuse and update the detection results to continuously track and locate the target; Output the target ID, the target's position in the world coordinate system, the target's category, and the point cloud bounding box Box2.

9. The method according to claim 8, characterized in that: The pixel coordinates in the two-dimensional target detection and tracking results and the pixel coordinates of the vertices of the point cloud boundary Box2 in the three-dimensional detection results are weighted fused as follows: in, is the target state information after weighted fusion; α and β are the weight coefficients of 3D and 2D detection respectively; The pixel coordinates of the 8 vertices of the bounding box of Box2 for 3D detection; Predicted target pixel coordinates for 2D tracking.

10. The method according to claim 6, characterized in that: The geometric prior information includes the shape and size of the target category, and the size includes the prior length, width and height of the target.

Images