A road-side multi-source sensor fusion target perception method and system for open-pit mines

Abstract

The present invention discloses a road-side multi-source sensor fusion target perception method and system for open-pit mines, specifically including: data collection, reading data from sensor devices, and format conversion of data; performing instance segmentation and depth estimation on images; fusing point cloud data, clustering foreground point cloud data, and constructing a three-dimensional detection frame for point cloud clusters; performing heterogeneous information fusion, associating the fusion detection results with the trajectory, performing optimal estimation of the target observation value, and outputting the future motion trajectory. The present invention adapts to the vehicle-road collaborative perception environment, achieves full coverage of scene range perception through the fusion of multi-camera and multi-lidar perception results, and improves the robustness of the perception algorithm in bad weather. By combining high-precision map information, accurate prediction of the target's future trajectory is achieved, showing the characteristics of low power consumption, good effect, and high robustness, and is very suitable for large-scale commercial implementation and promotion.

Description

Technical Field

[0001] The present invention relates to a method and system for fusion target perception of multi-source sensors at the road side, and in particular to a method and system for fusion target perception of multi-source sensors at the road side of an open-pit mine. Background Art

[0002] With the advancement of deep learning technology and the increase in computing power of edge hardware platforms, autonomous driving is becoming a core technology for future mobility. Existing autonomous driving algorithms mostly focus on single-vehicle intelligence, deploying multiple sensors and AI algorithms onboard to achieve vehicle intelligence. Single-vehicle intelligence works well in simple road environments, but it struggles with complex intersections. Intersections involve complex interactions between various types of vehicles and pedestrians, and because the sensors are mounted low on the vehicle, their field of view is easily obscured by obstacles such as large vehicles, bridges, and buildings. These issues lead to systemic deficiencies in single-vehicle intelligent systems in complex intersections. To address these deficiencies, roadside collaborative perception systems have emerged.

[0003] A roadside intelligent unit (RSU) is a system deployed at intersections to monitor dynamic objects within the entire area. This system provides intelligent vehicles within the area with information such as the global position, velocity, and predicted trajectory of objects in the area, providing additional assurance for safe driving. Most existing 3D perception frameworks are tailored for vehicle-side applications, with few specifically designed for roadside environments. Migrating vehicle-side perception algorithms to the roadside is challenging for three main reasons: 1. Vehicle-side sensors are constantly moving, and the context of the data they collect is constantly changing, while the roadside is stationary, making algorithms developed based on vehicle-side data inappropriate for the roadside environment. 2. The vehicle-side perspective differs significantly from the roadside perspective, providing more comprehensive and richer scene information. This also means that roadside perception is more susceptible to environmental interference, such as dust, rain, and snow, creating new challenges for roadside perception algorithms. 3. Vehicle-side computing resources are more abundant than those available to the roadside, requiring more efficient and power-efficient algorithms to ensure real-time performance. Therefore, a dedicated environmental perception algorithm must be developed specifically for roadside scenarios.

[0004] Patent CN112990129A proposes a three-dimensional object detection method based on the combination of vision and lidar. The method first obtains the current point cloud frame and video frame, then performs visual detection on the video frame to obtain the visual detection result, and then performs depth judgment on the visual detection result to obtain the visual approximate depth; for the point cloud frame data, the method first converts it into a sparse depth map, and then extracts the candidate depth frame in the sparse depth map based on the visual detection result, and then constructs the candidate point cluster of the current point cloud frame based on the candidate depth frame and the visual approximate depth, and finally constructs the three-dimensional object detection result based on the point cluster. The method uses the image detection results to extract the target point cloud cluster, and uses the image depth estimation model to estimate the depth of the detection target to further segment the point cloud. The performance of this three-dimensional object detection method is limited by image detection, and it cannot handle the situation where the image target is occluded well. In addition, due to the limited field of view of the image, the algorithm cannot use point cloud information beyond the field of view of the image.

[0005] Patent CN113095172A proposes a deep learning-based method for 3D object detection in point clouds. This method first uses a hierarchical voxel encoding module to extract feature representations of non-empty voxels in both sparse and dense regions of a point cloud scene. Then, an attention module fuses these voxel features to obtain an effective voxel-wise representation. Furthermore, a height information supplementation module introduces a bird's-eye view of the point cloud to supplement the height information in the voxel feature map. A channel attention module extracts useful information from the masked feature map to enhance the network's geometric structure perception. This method constructs a feature learning network to mine high-level semantic information from the feature map and adds a voxel segmentation task at the output to determine whether non-empty voxels belong to the target object. The proposed model structure is very complex, which poses significant limitations for engineering deployment. Furthermore, like all deep learning-based detection methods, this method is data-driven, making model performance highly dependent on the dataset. The network also has poor generalization capabilities. Changing scene models requires re-collecting, annotating, and training data, a time-consuming, resource-intensive, and labor-intensive process that is highly unsuitable for implementation. Current deep learning-based methods also face interpretability challenges and struggle to handle long-tail scenarios. Finally, the network-based 3D detection model consumes a lot of computing resources, and it is difficult to achieve real-time detection on the edge computing platform.

[0006] In the field of trajectory prediction, patent CN113763434A proposes a target trajectory prediction method based on switching multiple motion models using a Kalman filter. This method first establishes multiple Kalman filter motion models, then collects motion information of the target over a period of time (including at least the target's initial coordinates, real-time velocity, and real-time acceleration). Based on this motion information, the algorithm determines the target's motion state (including at least decelerating straight ahead, constant speed straight ahead, decelerating lane change, constant speed lane change, and accelerating lane change). The algorithm then switches the Kalman filter motion model based on the change in motion state to calculate the target's predicted trajectory. This method hardcodes the target's motion model into different Kalman filters, making it difficult to cope with the complexity of driving conditions in real-world scenarios. It also fails to utilize prior information about the road in the scene. Relying solely on the Kalman filter, this method can only predict the target's trajectory over a very short period of time, but cannot predict the target's long-term motion trends. Summary of the Invention

[0007] The purpose of the present invention is to provide a road-side multi-source sensor fusion target perception method and system for open-pit mines, which uses multi-source sensor fusion technology to achieve accurate and reliable three-dimensional target detection and tracking in the mining area, and can combine target tracking results and high-precision map information to predict the target's future driving trajectory information, thereby solving the defects of the existing technology.

[0008] The present invention provides the following solutions:

[0009] A method for target perception using multi-source sensor fusion at the roadside of an open-pit mine comprises the following steps:

[0010] Step 1) Data collection: collect data, read data from sensor devices, and convert the data format;

[0011] Step 2) Image multi-task perception: Perform instance segmentation and depth estimation on the collected road target images to obtain instance segmentation results and depth maps;

[0012] Step 3) Point cloud target detection: Multi-radar fusion is performed on the point cloud data to obtain fused point cloud data with unified coordinates. Background filtering is then performed to obtain foreground point cloud data. This foreground point cloud data is clustered using a clustering algorithm. A 3D detection box is constructed for the resulting point cloud clusters to obtain a 3D target box.

[0013] Step 4) Heterogeneous information fusion: Perform heterogeneous information fusion on image instance segmentation, depth map, and 3D target bounding box, and output the target 3D detection result;

[0014] Step 5) Multi-target tracking: Establish a trajectory based on the first frame result after heterogeneous information fusion, associate the fused detection result with the trajectory, and make an optimal estimate of the target observation value;

[0015] Step 6) Trajectory prediction: Based on the high-precision map information of the roadside area and the target tracking results, the future movement trajectory is output.

[0016] Furthermore, in step 1), time synchronization is performed while data is being collected. A trigger signal is sent through a unified clock source to trigger different sensors to collect data, and all data are timestamped with the triggering moment, achieving nanosecond-level time synchronization of different sensor data.

[0017] Furthermore, in step 2), an industrial camera is used to capture road target images from a roadside perspective, and instance segmentation and depth estimation are performed on the captured road target images. A multi-task deep learning network is used to train and implement instance segmentation and depth estimation on a dataset of a specific application scenario.

[0018] Furthermore, in step 3):

[0019] The point cloud frames in the local coordinate systems output by different lidar sensors are transferred to a unified coordinate system to obtain a fused point cloud frame with a complete field of view.

[0020] Receive fused point cloud frame data and perform filtering operations, including downsampling, illegal point removal, outlier removal, and region of interest filtering, and output the processed point cloud data;

[0021] Filter the background point cloud in the point cloud frame, divide the detection range into different voxels, collect several point cloud frames online, count the point cloud density of each voxel, make a background table, and set the threshold to filter the background points according to the point cloud density value of the voxel corresponding to the current point in the background table;

[0022] The DBSCAN algorithm is used to cluster the foreground point clouds, and the point clouds of the same target are clustered into one category;

[0023] The OBB directional bounding box algorithm is used to construct a three-dimensional detection box for each point cloud cluster.

[0024] Furthermore, in step 4), the image instance segmentation, depth estimation, and multi-radar point cloud fusion data are fused to output the target 3D detection result, which includes semantic category size and 3D pose information;

[0025] In step 5), a trajectory is established based on the first frame result after heterogeneous information fusion. When the next frame fusion result arrives, the Hungarian bipartite graph matching algorithm is used to associate the fusion detection result with the trajectory, and the Kalman filter technology is used to optimally estimate the target observation value.

[0026] Furthermore, in step 6), the future motion trajectory is output based on the target tracking results and high-precision map information. The map information includes lane guide lines, flatness and slope information, and the motion trajectory includes position information, speed, and acceleration.

[0027] A road-side multi-source sensor fusion target perception system for open-pit mines, specifically comprising:

[0028] Data acquisition module, which collects data, reads data from sensor devices, and converts the data format;

[0029] The image multi-task perception module performs instance segmentation and depth estimation on the collected road target images to obtain instance segmentation results and depth maps;

[0030] The point cloud target detection module performs multi-radar fusion on the point cloud data to obtain fused point cloud data with unified coordinates. It obtains foreground point cloud data through background filtering, clusters the foreground point cloud data using a clustering algorithm, and constructs a 3D detection frame for the point cloud clusters obtained through clustering to obtain a 3D target frame.

[0031] Heterogeneous information fusion module: It performs heterogeneous information fusion on image instance segmentation, depth map, and 3D target box, and outputs the target 3D detection result;

[0032] The multi-target tracking module is used to establish a trajectory based on the first frame result after heterogeneous information fusion, associate the fused detection result with the trajectory, and make the optimal estimate of the target observation value;

[0033] The trajectory prediction module is used to output the future motion trajectory based on the target tracking results and high-precision map information. The high-precision map information includes lane guide lines, flatness, and slope information. The motion trajectory includes position information, speed, and acceleration.

[0034] Furthermore, the point cloud target detection module includes a preprocessing module, a background filtering module, a point cloud clustering module, and a 3D box reconstruction module, wherein;

[0035] The preprocessing module is used to receive the original point cloud frame data or the fused point cloud frame data output by the multi-point cloud fusion module, perform filtering operations on the fused point cloud frame data, including downsampling, illegal point removal, outlier removal, and region of interest filtering operations, and output the processed point cloud data;

[0036] The background filtering module is used to filter the background point cloud in the point cloud frame, formalize the detection range into a three-dimensional voxel representation, collect several point cloud frames online, count the point cloud density of each voxel, and create a background table. According to the point cloud density value of the voxel corresponding to the current point in the background table, background points are filtered according to the threshold;

[0037] The point cloud clustering module is used to cluster the foreground point clouds using the DBSCAN algorithm, clustering point clouds of the same target into one category;

[0038] The 3D box reconstruction module uses the OBB directional bounding box algorithm to construct a 3D detection box for each point cloud cluster. Each point cloud cluster obtained by the point cloud clustering module needs to have a corresponding 3D box constructed by the 3D box reconstruction module;

[0039] In the multi-target tracking module, the fusion results of the heterogeneous information fusion module are finally input into the multi-target tracking module. The multi-target tracking module first establishes a trajectory based on the fusion results of the first frame. When the fusion results of the next frame arrive, the Hungarian bipartite graph matching algorithm is used to associate the fusion detection results and the trajectory, and then the Kalman filtering technology is used to make the optimal estimate of the target observation value.

[0040] An electronic device includes: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of a method for fusion target perception of multi-source sensors at the roadside of an open-pit mine.

[0041] A computer-readable storage medium is characterized in that it stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of a road-end multi-source sensor fusion target perception method for an open-pit mine.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] The present invention provides a roadside perception algorithm suitable for open-pit mines. This algorithm can well match the characteristics of roadside perception in the specific application scenario of vehicle-road collaboration in open-pit mines, realize accurate perception of dynamic targets in the scene, and provide blind spot filling, beyond-visual-range and redundant verification for vehicle-side perception, thus providing guarantee for the safe driving of intelligent driving vehicles at complex intersections.

[0044] This invention addresses point cloud target detection technology for roadside platforms and is adaptable to the fusion detection of multiple numbers and types of LiDARs. Point cloud target detection utilizes background filtering to preemptively filter static targets from the scene, significantly improving the speed and accuracy of point cloud target detection. Fusion of heterogeneous information from image detection results and point cloud target detection fully leverages the strengths of each sensor, enabling beyond-line-of-sight sensing for roadside sensing units and accurate perception in harsh environments such as dust, rain, and snow.

[0045] The present invention also combines a target trajectory prediction algorithm with high-precision map information. In the specific application scenario of vehicle-road collaboration in open-pit mines, this algorithm constrains the target's movement direction through the lane guide line information in the high-precision map, and uses the flatness and slope information to constrain the target's speed and acceleration, thereby achieving accurate prediction of the target's future movement trajectory.

[0046] The multi-source sensor perception algorithm proposed in this paper is specifically adapted for vehicle-infrastructure collaborative perception environments. By integrating the perception results of multiple cameras and multiple lidars, it achieves full coverage of the scene range and improves the algorithm's robustness in inclement weather. Furthermore, this framework also proposes a trajectory prediction module that accurately predicts the target's future trajectory by integrating high-precision map information. In specific application scenarios, the perception algorithm demonstrates low power consumption, high performance, and high robustness, making it well-suited for large-scale commercial deployment and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 It is a flow chart of the target perception method of multi-source sensor fusion at the road end of open-pit mines.

[0049] Figure 2 This is the architecture diagram of the road-side multi-source sensor fusion target perception system in open-pit mines.

[0050] Figure 3 This is the block diagram of the open-pit mine roadside scene perception algorithm.

[0051] Figure 4 This is the system architecture diagram of the point cloud target detection module.

[0052] Figure 5 It is a flowchart of point cloud target detection.

[0053] Figure 6 This is the structural diagram of the heterogeneous information fusion module.

[0054] Figure 7 This is a schematic diagram of the Frenet coordinate system.

[0055] Figure 8 It is a system architecture diagram of an electronic device. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] Most existing three-dimensional environment perception algorithms are developed based on vehicle-side perspective data, and these methods cannot be well transferred to roadside perception scenarios. On the one hand, there are significant differences in the perspectives of sensors on the vehicle side and the roadside side, which further lead to data differences. This makes the perception algorithms adapted to the vehicle side unable to work well in roadside scenarios. On the other hand, the vehicle-side sensors are in motion, and the background information in the collected data is constantly changing, which poses a great challenge to the perception algorithm. Roadside poles are fixed, and the background information in the scene can be filtered out based on high-precision maps using methods such as background filtering. This not only reduces the amount of data to be processed, but also improves the accuracy of target detection. Therefore, there are also differences in the perception algorithm processes between the roadside side and the vehicle side. The present invention proposes a perception algorithm for roadside environments. This algorithm uses background filtering and multi-source sensor fusion strategies to improve the performance of roadside perception algorithms and can meet the real-time requirements of the autonomous driving field.

[0058] For target trajectory prediction, existing methods include those based on Kalman filtering and those based on deep learning technology. For the former, Kalman filtering technology is highly dependent on the target's motion model, and cannot provide accurate trajectory prediction for situations where the target's motion is more complex. In addition, this method also has difficulty predicting the target's motion trajectory over a long period of time. For the latter, deep learning methods have poor generalization capabilities, and their performance is highly dependent on data. The annotation cost is high and the computational effort is large, making it impossible to meet the real-time requirements of autonomous driving on embedded devices. The target trajectory prediction method proposed in the present invention combines high-precision map information of the scene, including lane guide lines, flatness, slope and other information, and can obtain accurate trajectory information of the target in the next few seconds. These prediction information can significantly enhance the planning performance of autonomous driving vehicles at complex intersections and improve the vehicle's driving efficiency and safety.

[0059] From the description of the prior art in the background technology, it can be seen that the defects of the prior art include: it is unable to handle the situation where the image target is occluded, the image field of view is limited, and it is unable to utilize point cloud information beyond the image field of view (CN112990129A, involving a three-dimensional object detection method combining vision and lidar); the data-driven method debugging relies on the data set, and retraining is required when the scene changes. It lacks interpretability, consumes a lot of computing resources, and is difficult to detect the effect in real time on the edge computing platform (CN113095172A, involving deep learning point to three-dimensional object detection); it is difficult to cope with the complexity of driving conditions in actual scenarios, does not utilize prior information in the scene road, and only relies on Kalman filtering to predict the trajectory of the target in a very short time, but cannot predict the long-term movement trend of the target (CN113763434A, involving trajectory prediction).

[0060] The roadside perception algorithm proposed in this invention is specifically designed for use in open-pit mining environments. It enables beyond-line-of-sight perception and highly robust perception in adverse weather conditions, and can run efficiently and in real time on edge computing platforms. Existing solutions are mostly vehicle-side, consume high power, and cannot guarantee real-time performance. Therefore, this invention has technical benefits that exceed the expectations of those skilled in the art.

[0061] Example 1: Figure 1 The road-side multi-source sensor fusion target perception method for open-pit mine vehicle-road collaboration shown in the figure specifically includes:

[0062] Step S1) Data Collection: Data collection is performed, data is read from sensor devices, and the data format is converted. In this embodiment, the execution body of the data collection module includes road-end / roadside equipment such as industrial cameras and multi-lidar radars.

[0063] Step S2) Image multi-task perception: Perform instance segmentation and depth estimation on the collected road target image to obtain instance segmentation results and depth maps;

[0064] Depth estimation involves estimating the distance of each pixel in an image relative to the source of the image, using a single RGB image or a single or multiple perspectives. Existing methods for depth estimation typically involve building a convolutional neural network model for depth estimation, inspired by single-view / multi-view geometry principles. This involves collecting a large number of images and obtaining a depth map (for example, by interpolating depth using LiDAR data or directly capturing data using a depth camera). The network model is then optimized and trained using RGB images labeled with the depth value of each pixel. This optimized and trained network model can then be used to estimate the depth of a target image, obtaining the depth value of each pixel in the target image.

[0065] Step S3) Point cloud target detection: Perform multi-radar fusion on the point cloud data to obtain fused point cloud data with unified coordinates. Then, perform background filtering to obtain foreground point cloud data. Cluster the foreground point cloud data using a clustering algorithm. Construct a 3D detection frame for the point cloud clusters obtained through clustering to obtain a 3D target frame.

[0066] Step S4) Heterogeneous information fusion: Perform heterogeneous information fusion on image instance segmentation, depth map, and 3D target bounding box, and output the target 3D detection result. In this embodiment, heterogeneous information fusion is to fuse heterogeneous data from different data sources and different systems, such as image instance segmentation, depth estimation, and multi-radar fusion point cloud data.

[0067] Step S5) Multi-target tracking: Establish a trajectory based on the first frame result after heterogeneous information fusion, associate the fused detection result with the trajectory, and perform optimal estimation of the target observation value to obtain smoother and more accurate detection results. At the same time, information such as the target's speed and acceleration is also calculated.

[0068] Step S6) Trajectory Prediction: Based on the high-precision map information of the roadside area and the target tracking results, the future motion trajectory (the target's motion trajectory can be output several seconds in the future) is output. The map information includes lane guide lines, flatness, and slope information, and the motion trajectory includes position information, speed, and acceleration.

[0069] Preferably, time synchronization is performed while data is being collected. A trigger signal is sent through a unified clock source to trigger different sensors to collect data, and all data are timestamped with the triggering moment, thereby achieving nanosecond-level time synchronization of different sensor data.

[0070] Preferably, an industrial camera is used to capture road target images from a roadside perspective, and instance segmentation and depth estimation are performed on the captured road target images. A multi-task deep learning network is used to train and implement instance segmentation and depth estimation on a data set of a specific application scenario.

[0071] Preferably, the point cloud frames in the local coordinate systems output by different lidar sensors are transferred to a unified coordinate system, that is, the point cloud data under different viewing angles are spliced ​​together to obtain a fused point cloud frame with a complete field of view;

[0072] Receive fused point cloud frame data and perform filtering operations, including downsampling (voxel filtering), illegal point removal, outlier removal, and region of interest filtering, and output the processed point cloud data;

[0073] Filter the background point cloud in the point cloud frame, formalize the detection range into different voxels for representation, collect several point cloud frames online, count the point cloud density of each voxel, make a background table, and filter the background points according to the point cloud density value of the voxel corresponding to the current point in the background table according to the threshold;

[0074] The DBSCAN (density-based clustering with noise) algorithm is used to cluster the foreground point clouds, and the point clouds of the same target are clustered into one category;

[0075] The OBB directional bounding box algorithm is used to construct a three-dimensional detection box for each point cloud cluster.

[0076] Preferably, in step S4), image instance segmentation, depth estimation, and multi-radar point cloud fusion data are fused to output a 3D detection result of the target, where the 3D detection result includes semantic category size and 3D pose information;

[0077] In step S5), a trajectory is established based on the first frame result after heterogeneous information fusion. When the next frame fusion result arrives, the Hungarian bipartite graph matching algorithm is used to associate the fusion detection result with the trajectory. The Kalman filter technology is used to optimally estimate the target observation value to obtain a smoother and more accurate detection result. At the same time, information such as the target's speed and acceleration is also calculated.

[0078] Preferably, in step S6), the future motion trajectory is output based on the target tracking result and high-precision map information, the map information includes lane guide lines, flatness and slope information, and the motion trajectory includes position information, speed, and acceleration.

[0079] Example 2: Figure 2 and Figure 3 As shown, this embodiment is a system corresponding to a multi-source sensor fusion target perception method for roadside open-pit mines. This system comprises six modules: a data acquisition module, an image multi-task perception module, a point cloud target detection module, a heterogeneous information fusion module, a multi-target tracking module, and a trajectory prediction module. The data acquisition module is responsible for collecting real-time images and point cloud data. The image data is input into the image multi-task perception module to obtain results such as instance segmentation and depth estimation. The point cloud data is input into the point cloud target detection module to obtain 3D detection results. The results of the image multi-task perception module and the point cloud target detection module are input into the heterogeneous information fusion module. The fusion algorithm integrates the precise position information of the point cloud with the rich semantic information of the image to obtain more accurate 3D detection results. The results of the fusion module are then input into the multi-target tracking module to obtain information such as the target's tracking ID and velocity and acceleration. Finally, the target tracking information is input into the trajectory prediction module, which combines high-precision map information to accurately predict the target's future motion trajectory.

[0080] This system can realize the target perception method of multi-source sensor fusion at the roadside of open-pit mines through the combination of software and hardware. The basic functional modules of this system include:

[0081] The data acquisition module collects data, reads data from sensor devices, converts the data format, and transmits it to the point cloud object detection module. In addition to data encoding, the data acquisition module also includes functions such as time synchronization and spatial synchronization between data. Time synchronization utilizes hardware to generate a pulse signal. All sensors are triggered by this pulse, and each trigger calibrates their own clocks, eliminating accumulated errors in the clock source. This is crucial for online processing of time series data and ensures the possibility of multi-source sensor data fusion.

[0082] This module uses a unified clock source to send trigger signals to different sensors to collect data, and assigns a timestamp to all data at the triggering moment. This achieves nanosecond-level time synchronization of data from different sensors. Furthermore, spatial data synchronization, or calibration of different sensors, is also performed in the data acquisition module. Since the entire perception system includes lidar and cameras, sensor calibration involves both lidar-to-lidar calibration and camera-to-lidar calibration.

[0083] The image multi-task perception module is used to perform instance segmentation and depth estimation on the collected road target images; the input of the image multi-task perception module is the road target image collected by a high-speed industrial camera from the roadside perspective, and the output is the image instance segmentation result and the depth estimation result. This module trains a lightweight multi-task network model by collecting a large amount of data. The network can simultaneously complete the image instance segmentation and depth estimation tasks. Without loss of generality, the open source multi-task deep network can be used to train on a specific application scenario data set to achieve instance segmentation and depth estimation. For example, in the article "Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations", a lightweight multi-task detection network is proposed. Those skilled in the art can access and download the article through the online link https: / / arxiv.org / pdf / 1809.04766v2.pdf. The article is part of the prior art. The relevant information of the article is:

[0084] Author: Vladimir Nekrasov1, Thanuja Dharmasiri2, Andrew Spek2, TomDrummond2, Chunhua Shen1 and Ian Reid1

[0085] 1School of Computer Science, University of Adelaide, Australia2Monash University, Australia

[0086] Source: 2019-ICRA

[0087] Citation: Nekrasov, V., Dharmasiri, T., Spek, A., Drummond, T., Shen, C., & Reid, I. (2019, May). Real-time joint semantic segmentation and depthestimation using asymmetric annotations. In 2019 International Conference on Robotics and Automation (ICRA) (pp. 7101-7107). IEEE.

[0088] Improved from the paper: Nekrasov's 2018 article Light-Weight RefineNet for Real-Time Semantic Segmentation

[0089] The network architecture used in this module is similar to the lightweight multi-task detection network described above. Both consist of an encoder network that extracts high-level semantic features from images, followed by a lightweight decoder network that performs different prediction tasks. During the network processing phase, images from all cameras at the same time are batched and fed into the network simultaneously for processing, significantly improving network processing speed and throughput. To further accelerate network processing, this module uses NVIDIA's TensorRT framework to accelerate the model, enabling the multi-task image perception module on the Xavier development kit to meet the real-time requirements of autonomous driving.

[0090] like Figure 4 The point cloud target detection module shown in the figure performs multi-radar fusion on the point cloud data to obtain fused point cloud data with unified coordinates, obtains foreground point cloud data through background filtering, clusters the foreground point cloud data through a clustering algorithm, constructs a three-dimensional detection frame for the point cloud clusters obtained by clustering, and obtains a three-dimensional target frame.

[0091] In the case of collaborative perception of multiple lidars, the point cloud data collected by multiple sensors will first be input into the multi-point cloud fusion module for data pre-fusion to obtain fused point cloud data in a unified coordinate system, and then input into the point cloud target detection module.

[0092] Preferably, Figure 5 As shown in the figure, the point cloud target detection module contains four submodules, namely the preprocessing module, the background filtering module, the point cloud clustering module and the 3D frame reconstruction module. The three-dimensional target detection information output by the point cloud target detection module will finally be passed to the heterogeneous information fusion module and fused with the image detection results to obtain a detection result with richer information. The multi-point cloud fusion module is used to transfer the point cloud frames in the local coordinate systems output by different lidar sensors to a unified coordinate system, that is, to splice the point cloud data from different perspectives together to obtain a frame of fused point cloud frame with a complete field of view;

[0093] The preprocessing module is used to receive the original point cloud frame data or the fused point cloud frame data output by the multi-point cloud fusion module, perform filtering operations on the fused point cloud frame data, including downsampling, illegal point removal, outlier removal, and region of interest filtering operations, and output the processed point cloud data;

[0094] The background filtering module is used to filter the background point cloud in the point cloud frame, formalize the detection range into a three-dimensional voxel representation, collect several point cloud frames online, count the point cloud density of each voxel, and create a background table. According to the point cloud density value of the voxel corresponding to the current point in the background table, background points are filtered according to the threshold;

[0095] The point cloud clustering module is used to cluster the foreground point clouds using the DBSCAN (density-based clustering with noise) algorithm, clustering point clouds of the same target into one category;

[0096] After the preprocessed point cloud passes through the background filtering module, only the foreground point cloud data remains. The point cloud clustering module uses the DBSCAN (density-based clustering method with noise) algorithm to complete the clustering of the foreground point clouds and cluster the point clouds of the same target into one category.

[0097] Specifically, the DBSCAN clustering algorithm generally assumes that clusters can be determined by the density of sample distribution, and clustering is achieved by grouping closely connected samples together. For point cloud data, connectivity can be constructed based on the Euclidean distance between points, and then clustering can be performed using the DBSCAN algorithm.

[0098] The 3D box reconstruction module uses the OBB directional bounding box algorithm to construct a 3D detection box for each point cloud cluster. Each point cloud cluster generated by the point cloud clustering module requires the 3D box reconstruction module to construct a corresponding 3D box. The 3D box reconstruction module uses the OBB directional bounding box algorithm to construct a 3D detection box for each point cloud cluster. The OBB algorithm aims to construct a minimum cuboid in three-dimensional space that can enclose a point cloud cluster. The algorithm primarily utilizes principal component analysis to iteratively calculate the three major axis (X, Y, and Z) directions of the point cloud. Once the three major axis directions are determined, the target's position and size information can be obtained by projecting the point cloud in these three directions, thereby completing the target detection function.

[0099] like Figure 6 As shown, the heterogeneous information fusion module is used to perform heterogeneous information fusion. Preferably, it combines image instance segmentation, depth estimation (depth map), and fused point cloud data (3D object bounding boxes) and outputs a 3D object detection result. Preferably, the 3D detection result includes semantic category size and 3D pose information. The heterogeneous information fusion module takes as input the results of the image multi-task perception module and the point cloud object detection module, and outputs the final 3D object detection result, including semantic category size and 3D pose information. The fusion module implements two different fusion strategies based on distance. At close range (within the effective sensing range of the lidar), the fusion module prioritizes point cloud object detection results. However, inclement weather conditions such as dust, rain, and snow can introduce significant noise into the point cloud, leading to detection errors. Image perception algorithms, by contrast, are less susceptible to noise and are better suited to handling perception tasks in these conditions. Therefore, the algorithm projects the point cloud of the 3D detected target into image space. It then uses the image instance segmentation results to filter out points outside the projected target mask. The remaining point cloud is then used to recalculate the 3D bounding box, thereby filtering out false detections and achieving more accurate 3D target detection results. At long distances, the target point cloud is sparse, and point cloud object detection algorithms often struggle to accurately detect the target. However, the visual range of an image is much greater than that of a LiDAR, allowing for accurate detection of targets at distances of 200 to 300 meters. Therefore, the fusion module first uses the image instance segmentation results and depth estimation information to obtain depth information for all pixels within the target mask area. Based on this depth information, a pseudo point cloud is generated, and the target's size and 3D pose are then calculated. Due to errors in depth estimation, the calculated target information is generally inaccurate. To restore the target's size and 3D pose as accurately as possible, the algorithm projects the original point cloud data onto the image instance segmentation mask and uses the depth information of the point cloud that falls within the target mask to correct the image depth estimation results. The fusion module fully considers the advantages and disadvantages of each sensor, realizes the beyond-visual-range perception of the roadside perception algorithm and adapts to dust, rain and snow, and improves the perception distance and robustness of the perception algorithm in harsh environments.

[0100] The multi-target tracking module establishes a trajectory based on the first frame of heterogeneous information fusion results, associates the fused detection results with the trajectory, and optimally estimates the target observations, resulting in smoother and more accurate detection results. It also calculates information such as the target's velocity and acceleration. The fusion results from the heterogeneous information fusion module are ultimately input into the multi-target tracking module. The module first establishes a trajectory based on the first frame of fusion results. When the next frame of fusion results arrives, it associates the fused detection results with the trajectory using the Hungarian bipartite graph matching algorithm. Then, using Kalman filtering technology, it optimally estimates the target observations, resulting in smoother and more accurate detection results. It also calculates information such as the target's velocity and acceleration.

[0101] Kalman filter is an optimal estimation algorithm. The algorithm uses the optimal estimate X at K-1 time. k-1 Based on the prediction of the state variable X at time K ^ k / k-1 , and at the same time observe the state and get the observed variable Z k , and then analyze between prediction and observation, or correct the prediction with the observation, so as to obtain the optimal state estimate X at time K k In the process of multi-target tracking, the state variables are the three-dimensional position and velocity of the target ([x, y, z, v x , v y , v z ]), which can be calculated using the target motion model. The observation variable is the target's 3D position ([x, y, z]), obtained using the target detection algorithm. In practical applications, the target's orientation and size information can also be incorporated into the state variables and observation variables to obtain optimal estimates of the target's orientation and size. Through the multi-target tracking module, each fused target is managed using a trajectory, establishing temporal correlation between targets across frames. This facilitates the use of perception results by the trajectory prediction module.

[0102] like Figure 7 The trajectory prediction module shown is used to output the future motion trajectory (which can output the target's motion trajectory for the next few seconds) based on the target tracking results and high-precision map information. The high-precision map information includes lane guide lines, flatness, and slope information, and the motion trajectory includes position information, speed, and acceleration.

[0103] The input of the trajectory prediction module is the tracking result of the target and the high-precision map information of the roadside area, including the lane guide line (usually the lane center line), flatness and slope information, and the output is the target's motion trajectory in the next few seconds, including position information, speed, and acceleration information. First, the prediction module uses the target posture, speed, acceleration and other information obtained by multi-target tracking to preliminarily build a motion model of the target, and then uses the lane guide line to constrain the target's motion direction. The constraint on the motion direction works in the Frenet coordinate system. The Frenet coordinate system is based on the lane guide line, with the direction along the guide line as the vertical axis (s), and the horizontal axis perpendicular to the guide line as the horizontal axis (l), such as Figure 7 Given a lane guideline, the module projects the vehicle's position onto the guideline. Based on the projection point, the module decomposes the vehicle's current motion state (x, y, θ, v, a) in the map coordinate system to obtain the position, velocity, and acceleration along the guideline, as well as the position, velocity, and acceleration relative to the guideline. The "velocity" and "acceleration" here are not the first-order / second-order derivatives of displacement with respect to time in the general sense, but rather the first-order / second-order derivatives of lateral displacement with respect to longitudinal displacement. They describe the changing trend of the geometric shape and are calculated as follows:

[0104]

[0105] After motion decomposition, the vehicle's motion model in the map coordinate system can be converted to the Frenet coordinate system. Constraining the vehicle's future motion direction can be achieved by limiting the vehicle's motion component along the horizontal axis l, ensuring that the vehicle's predicted trajectory follows the lane guide lines as closely as possible, improving trajectory prediction accuracy.

[0106] In addition, the trajectory prediction module uses the slope information to make a linear correction to the predicted speed, and then uses the flatness information to constrain the target's movement speed and acceleration. Unlike the constraint on the direction of movement, this is performed in the map coordinate system.

[0107] The trajectory prediction module uses the above scheme to iteratively calculate the target's future position, velocity, and acceleration information. This information can provide the vehicle with the future state of the scene, thereby improving the accuracy and efficiency of the vehicle's path planning and velocity planning.

[0108] It is worth noting that although only basic functional modules are disclosed in the text description and architecture diagram of this system, it does not mean that this system is limited to the basic functional modules. On the contrary, what this patent wants to express is: based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technologies to form an infinite number of embodiments or technical solutions. In other words, this system is open rather than closed. Just because this embodiment only discloses individual basic functional modules, it cannot be considered that the scope of protection of the patent claims is limited to the disclosed basic functional modules.

[0109] like Figure 8 As shown, the present invention discloses a method and system for target perception of multi-source sensor fusion at the roadside of an open-pit mine, and also discloses electronic equipment and storage media corresponding to the method and system:

[0110] An electronic device comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above methods.

[0111] A computer-readable storage medium stores a computer program executable by an electronic device, wherein when the computer program runs on the electronic device, the electronic device executes the steps of any one of the above methods.

[0112] The electronic device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory. The operating system can be any one or more computer operating systems that implement electronic device control through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. In addition, in an embodiment of the present invention, the electronic device can be a handheld device such as a smart phone or a tablet computer, or an electronic device such as a desktop computer or a portable computer, which is not particularly limited in the embodiment of the present invention. The execution subject of the electronic device control in the embodiment of the present invention can be an electronic device, or a functional module in the electronic device that can call and execute a program.

[0113] The electronic device can obtain the firmware corresponding to the storage medium. The firmware corresponding to the storage medium is provided by the supplier. The firmware corresponding to different storage media can be the same or different, and this is not limited here. After the electronic device obtains the firmware corresponding to the storage medium, it can write the firmware corresponding to the storage medium into the storage medium, specifically, burning the firmware corresponding to the storage medium into the storage medium. The process of burning the firmware into the storage medium can be implemented using existing technologies and will not be described in detail in the embodiments of the present invention.

[0114] The electronic device can also obtain a reset command corresponding to the storage medium. The reset command corresponding to the storage medium is provided by the supplier. The reset commands corresponding to different storage media can be the same or different, and are not limited here. In this case, the storage medium of the electronic device is a storage medium with the corresponding firmware written therein. The electronic device can respond to the reset command corresponding to the storage medium in the storage medium with the corresponding firmware written therein, so that the electronic device resets the storage medium with the corresponding firmware written therein according to the reset command corresponding to the storage medium. The process of resetting the storage medium according to the reset command can be implemented by existing technology and will not be described in detail in the embodiments of the present invention.

[0115] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0117] The present application can be used in a wide variety of general-purpose or specialized computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0118] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for target perception using multi-source sensor fusion at the roadside of an open-pit mine, characterized in that: The specific steps include: Step 1) Data collection: collect data, read data from sensor devices, and convert the data format; Step 2) Image multi-task perception: Perform instance segmentation and depth estimation on the collected road target image to obtain instance segmentation results and depth maps; Step 3) Point cloud target detection: Multi-radar fusion is performed on the point cloud data to obtain fused point cloud data with unified coordinates. The foreground point cloud data is then obtained through background filtering. The foreground point cloud data is clustered using a clustering algorithm. A 3D detection box is constructed for the point cloud clusters obtained through clustering to obtain a 3D target box. Step 4) Heterogeneous Information Fusion: Perform heterogeneous information fusion on the image instance segmentation, depth map, and 3D target bounding box, and output the target 3D detection result. Heterogeneous information fusion implements two different fusion methods based on the distance, including short-range method and long-range method: A short-range method is used within the effective sensing range of the LiDAR, prioritizing the trust of point cloud target detection results. The point cloud of the 3D detected target is projected into the image space. The image instance segmentation results are used to filter out points projected outside the target mask. The remaining point cloud is used to recalculate the 3D bounding box to filter out false detection results and obtain more accurate 3D target detection results. In the long-range method, the image instance segmentation results and depth estimation information are used to obtain the depth information of all pixels in the target mask area. Based on this depth information, a pseudo point cloud is generated to calculate the size and 3D pose of the target. The original point cloud data is projected into the image instance segmentation mask, and the depth information of the point cloud falling within the target mask is used to correct the image depth estimation result. Step 5) Multi-target tracking: Establish a trajectory based on the first frame result after heterogeneous information fusion, associate the fused detection result with the trajectory, and make an optimal estimate of the target observation value; Step 6) Trajectory prediction: Based on the high-precision map information of the roadside area and the target tracking results, the future movement trajectory is output.

2. The method for target perception by multi-source sensor fusion at the road end of an open-pit mine according to claim 1, characterized in that: In step 1), time synchronization is performed while data is collected. A trigger signal is sent through a unified clock source to trigger different sensors to collect data, and all data are timestamped with the triggering moment, achieving nanosecond-level time synchronization of different sensor data.

3. The method for target perception by multi-source sensor fusion at the road end of an open-pit mine according to claim 1, characterized in that: In step 2), an industrial camera is used to capture road target images from a roadside perspective, and instance segmentation and depth estimation are performed on the captured road target images. A multi-task deep learning network is used to train and implement instance segmentation and depth estimation on a dataset of a specific application scenario.

4. The method for target perception by multi-source sensor fusion at the road end of an open-pit mine according to claim 1, characterized in that: In step 3): The point cloud frames in the local coordinate systems output by different lidar sensors are transferred to a unified coordinate system to obtain a fused point cloud frame with a complete field of view. Receive fused point cloud frame data and perform filtering operations, including downsampling, illegal point removal, outlier removal, and region of interest filtering, and output the processed point cloud data; Filter the background point cloud in the point cloud frame, divide the detection range into different voxels, collect several point cloud frames online, count the point cloud density of each voxel, make a background table, and set the threshold to filter the background points according to the point cloud density value of the voxel corresponding to the current point in the background table; The DBSCAN algorithm is used to cluster the foreground point clouds, and the point clouds of the same target are clustered into one category; The OBB directional bounding box algorithm is used to construct a three-dimensional detection box for each point cloud cluster.

5. The method for target perception using multi-source sensor fusion at the roadside of an open-pit mine according to claim 1, characterized in that: In step 4), the image instance segmentation, depth estimation, and multi-radar point cloud fusion data are fused to output the target 3D detection result, which includes semantic category size and 3D pose information; In step 5), the trajectory is established based on the first frame result after heterogeneous information fusion. When the next frame fusion result arrives, The Hungarian bipartite graph matching algorithm is used to associate the fusion detection results with the trajectory, and the Kalman filter technique is used to optimally estimate the target observation value.

6. The method for target perception by multi-source sensor fusion at the road end of an open-pit mine according to claim 1, characterized in that: In step 6), the future motion trajectory is output based on the target tracking results and high-precision map information. The map information includes lane guide lines, flatness and slope information, and the motion trajectory includes position information, speed, and acceleration.

7. A road-side multi-source sensor fusion target perception system for open-pit mines, characterized in that: Specifically include: Data acquisition module, which collects data, reads data from sensor devices, and converts the data format; The image multi-task perception module performs instance segmentation and depth estimation on the collected road target images to obtain instance segmentation results and depth maps; The point cloud target detection module performs multi-radar fusion on the point cloud data to obtain fused point cloud data with unified coordinates. It obtains foreground point cloud data through background filtering, clusters the foreground point cloud data using a clustering algorithm, and constructs a 3D detection frame for the point cloud clusters obtained through clustering to obtain a 3D target frame. The heterogeneous information fusion module performs heterogeneous information fusion on image instance segmentation, depth map, and three-dimensional target box, and outputs the target 3D detection result. Among them, the heterogeneous information fusion module implements two different fusion methods according to the distance, including short-range method and long-range method: the short-range method is used within the effective perception range of the lidar, and the point cloud target detection result is trusted first. The point cloud of the three-dimensional detection target is projected into the image space, and the image instance segmentation result is used to filter out the points projected outside the target mask. The 3D bounding box is recalculated with the remaining point cloud to filter out false detection results and obtain more accurate target three-dimensional detection results; in the long-range method, the image instance segmentation result and depth estimation information are used to obtain the depth information of all pixels in the target mask area, and a pseudo point cloud is generated based on this depth information to calculate the size and three-dimensional pose of the target; the original point cloud data is projected into the image instance segmentation mask, and the depth information of the point cloud falling within the target mask is used to correct the image depth estimation result; The multi-target tracking module is used to establish a trajectory based on the first frame result after heterogeneous information fusion, associate the fused detection result with the trajectory, and make the optimal estimate of the target observation value; The trajectory prediction module is used to output the future motion trajectory based on the target tracking results and high-precision map information. The high-precision map information includes lane guide lines, flatness, and slope information. The motion trajectory includes position information, speed, and acceleration.

8. The road-side multi-source sensor fusion target perception system for open-pit mines according to claim 7 is characterized in that: The point cloud target detection module includes a preprocessing module, a background filtering module, a point cloud clustering module, and a 3D frame reconstruction module, among which; The preprocessing module is used to receive the original point cloud frame data or the fused point cloud frame data output by the multi-point cloud fusion module, perform filtering operations on the fused point cloud frame data, including downsampling, illegal point removal, outlier removal, and region of interest filtering operations, and output the processed point cloud data; The background filtering module is used to filter the background point cloud in the point cloud frame, formalize the detection range into a three-dimensional voxel representation, collect several point cloud frames online, count the point cloud density of each voxel, and create a background table. According to the point cloud density value of the voxel corresponding to the current point in the background table, background points are filtered according to the threshold; The point cloud clustering module is used to cluster the foreground point clouds using the DBSCAN algorithm, clustering point clouds of the same target into one category; The 3D box reconstruction module uses the OBB directional bounding box algorithm to construct a 3D detection box for each point cloud cluster. Each point cloud cluster obtained by the point cloud clustering module needs to have a corresponding 3D box constructed by the 3D box reconstruction module; In the multi-target tracking module, the fusion results of the heterogeneous information fusion module are finally input into the multi-target tracking module. The multi-target tracking module first establishes a trajectory based on the fusion results of the first frame. When the fusion results of the next frame arrive, the Hungarian bipartite graph matching algorithm is used to associate the fusion detection results and the trajectory, and then the Kalman filtering technology is used to make the optimal estimate of the target observation value.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory stores a computer program, When the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method according to any one of claims 1 to 6.

Images