Mine unmanned dynamic obstacle perception system and method

Abstract

The present application relates to the technical field of mine unmanned driving, and particularly relates to a mine unmanned driving dynamic obstacle sensing system and method, comprising a multi-sensor sensing module, a data preprocessing module, an adaptive fusion algorithm module, a dynamic obstacle identification and trajectory prediction module and an output module; the present application changes the fixed weight fusion of the multi-sensor into adaptive fusion switching with the working condition, and cooperates with the complementary characteristics of the four types of sensors, so as to realize the reliable identification and trajectory prediction of the dynamic obstacle in the complex environment of the mine.

Description

Technical Field

[0001] This invention relates to the field of unmanned driving technology in mining, specifically to a dynamic obstacle perception system and method for unmanned driving in mining. Background Technology

[0002] Unmanned mining trucks in open-pit mines typically operate between the loading / unloading area, transport roads, and spoil heaps. These areas simultaneously contain personnel, other mining vehicles, and dynamic obstacles such as rocks rolling down slopes. To enable timely obstacle avoidance, mining trucks are generally equipped with environmental perception devices such as millimeter-wave radar, lidar, visual sensors, or infrared thermal imaging sensors. The perception system processes the collected data and provides information such as the type and location of obstacles to the truck's decision-making and control system.

[0003] The mining environment is unique, with common problems such as severe dust, variable lighting (no lighting or strong light interference at night), frequent fog, and bumpy roads, which limit the environmental perception capabilities of single sensors: millimeter-wave radar is easily interfered with by metallic obstacles and has insufficient accuracy in identifying non-metallic obstacles (such as personnel and falling rocks); lidar suffers severe range attenuation in dusty and foggy environments, and data point clouds are prone to missing; visual sensors depend on lighting conditions, and their accuracy drops significantly at night or in low visibility environments; while infrared thermal imaging can detect heat source targets (such as personnel and vehicles) at night and in low visibility, it is less effective in identifying non-heat source obstacles (such as falling rocks).

[0004] To compensate for the limitations of a single sensor, existing mine sensing systems fuse data from multiple sensors. A common approach is to weight the data from each sensor with fixed weights or to directly overlay the detection results from each sensor. This approach works when the environment is relatively stable, but mine conditions are constantly changing. The same road can be dusty during the day, dark at night, and bumpy when turning a bend.

[0005] During periods of dust pollution, the lidar point cloud is largely missing, yet it is still assigned a high weight under clear conditions during fusion. The missing data is introduced as valid information, which reduces the reliability of the fusion result. At night, visual images are basically ineffective, but the fixed weight still retains a large proportion of them, which can easily lead to false detections. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a dynamic obstacle perception system and method for unmanned mining operations. By determining the environmental type of the mining truck in real time and dynamically adjusting the fusion weights of each sensor accordingly, the fusion result prioritizes the most reliable sensor under different operating conditions such as dust, nighttime, heavy fog, and bumpy conditions, thereby improving the accuracy and real-time performance of dynamic obstacle recognition.

[0007] This invention is achieved through the following technical solution:

[0008] A dynamic obstacle perception system for unmanned mining operations includes a multi-sensor perception module, a data preprocessing module, an adaptive fusion algorithm module, a dynamic obstacle recognition and trajectory prediction module, and an output module.

[0009] The multi-sensor perception module is installed on the unmanned mining truck and is used to collect multi-source sensor data of the mining environment.

[0010] The data preprocessing module is connected to the multi-sensor sensing module and is used to preprocess the multi-source sensor data to obtain preprocessed multi-source sensor data.

[0011] The adaptive fusion algorithm module is connected to the data preprocessing module and includes an environmental assessment unit and a dynamic weight allocation unit. The environmental assessment unit is used to determine the current environmental type of the unmanned mining truck. The dynamic weight allocation unit is used to determine the fusion weight of each sensor according to the environmental type, and to perform weighted fusion of the preprocessed multi-source sensor data based on the fusion weight to obtain fused perception data.

[0012] The dynamic obstacle recognition and trajectory prediction module is connected to the adaptive fusion algorithm module and is used to identify the type of dynamic obstacle based on the fused perception data and predict the motion trajectory of the dynamic obstacle.

[0013] The output module is connected to the dynamic obstacle recognition and trajectory prediction module and is used to output the type and trajectory of the dynamic obstacle to the decision control system of the unmanned mining truck.

[0014] Optionally, the multi-sensor perception module includes millimeter-wave radar, lidar, visual sensors, and infrared thermal imaging sensors, with each sensor respectively installed on the roof, front, and both sides of the unmanned mining truck.

[0015] The data preprocessing module includes a time synchronization unit, a noise reduction unit, and a spatial calibration unit connected in sequence.

[0016] Specifically, it also includes an inertial measurement unit and a data processing unit, both of which are mounted on the unmanned mining truck;

[0017] The inertial measurement unit is connected to the adaptive fusion algorithm module and is used to collect the attitude information of the unmanned mining truck.

[0018] The adaptive fusion algorithm module and the dynamic obstacle recognition and trajectory prediction module are integrated into the data processing unit.

[0019] A dynamic obstacle perception method for unmanned mining operations includes:

[0020] Multi-source sensor data of the mining environment are collected by millimeter-wave radar, lidar, visual sensors and infrared thermal imaging sensors installed on unmanned mining trucks.

[0021] The multi-source sensor data is time-synchronized, noise-reduced, and spatially calibrated to obtain pre-processed multi-source sensor data.

[0022] The environment type of the unmanned mining truck is determined based on the preprocessed multi-source sensor data. The fusion weight of each sensor is determined based on the environment type. The preprocessed multi-source sensor data is then weighted and fused based on the fusion weight to obtain fused perception data.

[0023] The type of dynamic obstacle is identified based on the fused perception data, and the trajectory of the dynamic obstacle is predicted.

[0024] The type and trajectory of the dynamic obstacle are output to the decision control system of the unmanned mining truck.

[0025] Optionally, a method for determining the current environment type of the unmanned mining truck includes:

[0026] Analyze the confidence level of the multi-source sensor data, and determine the environment type based on the confidence level. The environment type includes at least one of the following: normal environment, dusty environment, foggy environment, nighttime environment, and bumpy road environment.

[0027] The confidence level includes at least one of the following: lidar point cloud integrity, visual image sharpness, millimeter-wave radar signal strength, and infrared thermal imaging contrast.

[0028] Optionally, the method for determining the fusion weights of each sensor based on the environment type includes:

[0029] The fusion weights are determined based on the inherent confidence coefficients of each sensor and the real-time environment adaptability. The inherent confidence coefficients are pre-calibrated, and the real-time environment adaptability is determined based on the environment type.

[0030] The fusion weight is calculated using the following formula: In the formula, For the first The fusion weights of individual sensors, For the first The inherent confidence coefficient of each sensor, For the first Real-time environmental adaptability of individual sensors This represents the total number of sensors.

[0031] Optionally, when the environment type is a dusty environment or a foggy environment, the fusion weight of the millimeter-wave radar and the infrared thermal imaging sensor is increased;

[0032] When the environment type is nighttime, increase the fusion weight of the infrared thermal imaging sensor;

[0033] When the environment type is a bumpy road environment, the fusion weights of each sensor are adjusted according to the attitude information collected by the inertial measurement unit.

[0034] Optionally, a method for processing the multi-source sensor data includes:

[0035] Time alignment of the multi-source sensor data is performed based on timestamps;

[0036] Kalman filtering is used to reduce noise in the millimeter-wave radar data, and statistical filtering is used to remove outliers in the point cloud of the lidar.

[0037] Based on hand-eye calibration, the data from the lidar, vision sensor, and infrared thermal imaging sensor are converted to the coordinate system of the millimeter-wave radar.

[0038] Optionally, the method for predicting the trajectory of the dynamic obstacle includes:

[0039] Based on the thermal radiation information in the fused sensing data, heat source obstacles and non-heat source obstacles are distinguished;

[0040] The motion trajectory of the dynamic obstacle is predicted by using a long short-term memory network combined with a Kalman filter.

[0041] Specifically, it also includes: determining the risk level of the dynamic obstacle based on the relative distance and relative speed between the dynamic obstacle and the unmanned mining truck, and outputting the risk level to the decision control system of the unmanned mining truck.

[0042] This invention, through environmental assessment and dynamic weight allocation, transforms the fixed-weight fusion of multiple sensors into adaptive fusion that changes with operating conditions. Furthermore, by leveraging the complementary characteristics of four types of sensors, it achieves reliable identification and trajectory prediction of dynamic obstacles in complex mining environments. Compared with existing technologies, this invention has the following features and beneficial effects:

[0043] By employing multi-source fusion of millimeter-wave radar, lidar, visual sensors, and infrared thermal imaging sensors, and complementing the advantages of each sensor, the ability to identify dynamic obstacles in complex environments such as dust, night, fog, and bumpy roads is improved. The accuracy of identifying people, vehicles, and falling rocks is increased, and the number of missed detections and false detections is effectively reduced.

[0044] An adaptive fusion algorithm is proposed, which can dynamically adjust the sensor fusion weights according to the environment, avoid the performance degradation of fixed weight fusion in complex environments, and ensure the stability and reliability of the sensing system.

[0045] By combining a long short-term memory network with a Kalman filter to predict the trajectory of obstacles, and correcting the observation noise with a Kalman filter based on the trend prediction given by the long short-term memory network, real-time identification and accurate trajectory prediction of dynamic obstacles are achieved. Attached Figure Description

[0046] The accompanying drawings illustrate exemplary embodiments of the present invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, but do not constitute a limitation on the embodiments of the present invention.

[0047] Figure 1 This is a schematic diagram showing the layout of each sensor in the multi-sensor perception module on an unmanned mining truck, as provided in an embodiment of the present invention.

[0048] Figure 2 This is a structural block diagram of a dynamic obstacle perception system for unmanned mining vehicles based on multi-sensor fusion, provided in an embodiment of the present invention.

[0049] Figure 3 A schematic diagram illustrating the process of determining fusion weights in the adaptive fusion algorithm module provided in this embodiment of the invention;

[0050] Figure 4 This is a flowchart illustrating the dynamic obstacle perception method for unmanned mining operations based on multi-sensor fusion, as provided in an embodiment of the present invention.

[0051] Figure reference numerals: 100-Unmanned mining truck; 10-Multi-sensor perception module; 11-Millimeter-wave radar; 12-LiDAR; 13-Vision sensor; 14-Infrared thermal imaging sensor; 20-Data preprocessing module; 21-Time synchronization unit; 22-Noise reduction unit; 23-Spatial calibration unit; 30-Adaptive fusion algorithm module; 31-Environmental assessment unit; 32-Dynamic weight allocation unit; 40-Dynamic obstacle recognition and trajectory prediction module; 50-Output module; 60-Inertial measurement unit; 70-Data processing unit. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] It should also be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0054] Where there is no conflict, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] The dynamic obstacles mentioned in this article include, but are not limited to, workers, mining vehicles, and rocks rolling down slopes, which may change position within the mining operation area, as well as other movable targets that need to be avoided by unmanned mining trucks.

[0056] The environmental types mentioned in this article refer to the mining conditions that affect the detection performance of each sensor, including but not limited to normal environment, dusty environment, foggy environment, nighttime environment, and bumpy road environment, as well as other conditions that change the detection performance of one or more sensors.

[0057] Example 1

[0058] This embodiment provides a dynamic obstacle perception system for unmanned mining operations based on multi-sensor fusion. For example... Figure 2 As shown, the system includes a multi-sensor perception module, a data preprocessing module, an adaptive fusion algorithm module, a dynamic obstacle recognition and trajectory prediction module, and an output module. These modules are connected sequentially along the data flow direction.

[0059] A multi-sensor perception module is installed on the unmanned mining truck to collect multi-source sensor data of the mining environment. For example... Figure 1 As shown, the multi-sensor perception module includes millimeter-wave radar, lidar, a vision sensor, and an infrared thermal imaging sensor. These sensors are respectively installed on the roof, front, and both sides of the unmanned mining truck, ensuring seamless connectivity of the perception range in front of and to the sides of the truck without blind spots.

[0060] Each sensor has a different function:

[0061] Millimeter-wave radar is used to acquire information about the range, velocity, and azimuth of a target;

[0062] LiDAR is used to acquire three-dimensional point cloud data of the environment, providing the outline and distance of the target;

[0063] Visual sensors are used to acquire environmental images, providing texture information about the target;

[0064] Infrared thermal imaging sensors are used to collect thermal radiation information of targets, enabling the detection of heat sources at night and in low visibility conditions.

[0065] In a specific example, the sampling frequency of the millimeter-wave radar is 10Hz to 20Hz, such as 10Hz, 15Hz or 20Hz; the detection range of the lidar is 0.5m to 200m; the resolution of the visual sensor is not less than 1920×1080 and the frame rate is not less than 30fps; the detection range of the infrared thermal imaging sensor is 0.5m to 100m and the thermal sensitivity is ≤50mK.

[0066] The data preprocessing module is connected to the multi-sensor sensing module and is used to preprocess the acquired multi-source sensor data. The data preprocessing module includes a time synchronization unit, a noise reduction unit, and a spatial calibration unit connected in sequence, which respectively align the data of each sensor on the time axis, suppress noise, and unify the spatial coordinates.

[0067] The adaptive fusion algorithm module is connected to the data preprocessing module and includes an environment assessment unit and a dynamic weight allocation unit. The environment assessment unit is used to determine the current environment type of the unmanned mining truck; the dynamic weight allocation unit is used to determine the fusion weight of each sensor according to the environment type, and to perform weighted fusion of the preprocessed multi-source sensor data according to the fusion weight to obtain fused perception data.

[0068] The dynamic obstacle recognition and trajectory prediction module is connected to the adaptive fusion algorithm module. It identifies the type of dynamic obstacle based on the fused perception data and predicts the trajectory of that obstacle. The output module is connected to the dynamic obstacle recognition and trajectory prediction module and outputs the type and trajectory of the dynamic obstacle to the decision control system of the unmanned mining truck for obstacle avoidance decisions.

[0069] In some embodiments, the system further includes an inertial measurement unit and a data processing unit, both of which are mounted on the unmanned mining truck.

[0070] The inertial measurement unit (IMU) is connected to the adaptive fusion algorithm module to collect the attitude information of the unmanned mining truck, which is used to adjust the fusion weights under bumpy road conditions. The data processing unit is the hardware that carries the computation, and the adaptive fusion algorithm module and the dynamic obstacle recognition and trajectory prediction module are integrated into the data processing unit.

[0071] In a specific example, the data processing unit may be an in-vehicle embedded computing platform with computing power sufficient to support the parallel processing of multi-channel sensor data, such as 200 TOPS.

[0072] Example 2

[0073] This embodiment provides a dynamic obstacle perception method for unmanned mining operations based on multi-sensor fusion. This method can be executed by the perception system described in Embodiment 1, but is not limited to being executed by that system. For example... Figure 4As shown, the method includes the following steps.

[0074] Step S1: Collect multi-source sensor data of the mining environment by using millimeter-wave radar, lidar, vision sensor and infrared thermal imaging sensor set on the unmanned mining truck.

[0075] Step S2 involves performing time synchronization, noise reduction, and spatial calibration on the multi-source sensor data to obtain preprocessed multi-source sensor data.

[0076] Step S3: Determine the current environment type of the unmanned mining truck based on the preprocessed multi-source sensor data, determine the fusion weight of each sensor based on the environment type, and perform weighted fusion of the preprocessed multi-source sensor data based on the fusion weight to obtain fused perception data.

[0077] Step S4: Identify the type of dynamic obstacle based on the fused perception data and predict the movement trajectory of the dynamic obstacle.

[0078] Step S5: Output the type and trajectory of the dynamic obstacle to the decision control system of the unmanned mining truck.

[0079] The process of determining the environmental type in step S3 is completed by the environmental assessment unit.

[0080] The environmental assessment unit analyzes the confidence levels of multi-source sensor data and determines the environment type based on the confidence levels. The environment type includes at least one of the following: normal environment, dusty environment, foggy environment, nighttime environment, and bumpy road environment.

[0081] Confidence is used to measure the availability of current output data from each sensor, including at least one of lidar point cloud integrity, visual image sharpness, millimeter-wave radar signal strength, and infrared thermal imaging contrast.

[0082] For example, when the integrity of the lidar point cloud decreases significantly while the signal strength of the millimeter-wave radar is basically normal, it can be determined that the current environment is dusty or foggy; when the clarity of the visual image decreases significantly while the contrast of the infrared thermal imaging is high, it can be determined that the environment is nighttime.

[0083] In some embodiments, the determination of the environment type can also be achieved through a pre-trained classifier, that is, the confidence level of each sensor data is used as input and the environment type is used as output, and the classifier gives the determination result; in other embodiments, information such as the clock and headlight status of the mining truck can also be combined to assist in the determination of the nighttime environment.

[0084] Example 3

[0085] This embodiment further explains the process of determining the fusion weights and performing weighted fusion in step S3, which is completed by the dynamic weight allocation unit.

[0086] like Figure 3 As shown, the dynamic weight allocation unit determines the fusion weights based on the inherent confidence coefficients and real-time environmental adaptability of each sensor. The inherent confidence coefficients characterize the reliability of the sensor under standard conditions, are obtained in advance through experimental calibration, and remain unchanged during operation after calibration. The real-time environmental adaptability is determined based on the environmental type given by the environmental assessment unit, reflecting the availability of the sensor under the current operating conditions.

[0087] The fusion weights are calculated using the following formula: In the formula, For the first The fusion weights of individual sensors, Corresponding millimeter-wave radar, Corresponding LiDAR, Corresponding visual sensors, Corresponding infrared thermal imaging sensor;

[0088] For the first The inherent confidence coefficient of each sensor is obtained in advance through experimental calibration; For the first The real-time environmental adaptability of each sensor is output by the environmental assessment unit, with a value ranging from 0 to 1. The closer the value is to 1, the better the performance of the sensor in the current environment. The confidence scores of the four sensors are weighted and summed such that the sum of the fusion weights satisfies the following condition. .

[0089] In a specific example, the inherent confidence coefficients of millimeter-wave radar, lidar, visual sensor, and infrared thermal imaging sensor are calibrated to 0.8, 0.9, 0.75, and 0.85, respectively.

[0090] In a specific example, the inherent confidence coefficients of millimeter-wave radar, lidar, visual sensor, and infrared thermal imaging sensor can be calibrated to 0.80, 0.95, 0.90, and 0.70, respectively. The real-time environment adaptability ranges from 0 to 1, with a value closer to 1 indicating better sensor performance in the current environment.

[0091] The dynamic weighting unit assigns weights to each sensor based on the environment type, which results in the more reliable sensors having a larger share in the fused data.

[0092] Under normal conditions, LiDAR and vision sensors are the main components. For example, the fusion weights of each sensor can be set as follows: LiDAR approximately 0.35, vision sensor approximately 0.30, millimeter-wave radar approximately 0.175, and infrared thermal imaging sensor approximately 0.175.

[0093] In dusty or foggy environments, increase the fusion weight of millimeter-wave radar and infrared thermal imaging sensor, and decrease the fusion weight of lidar and vision sensor. For example, set them as follows: millimeter-wave radar approximately 0.40, infrared thermal imaging sensor approximately 0.30, lidar approximately 0.15, and vision sensor approximately 0.15.

[0094] In nighttime environments, the fusion weights of infrared thermal imaging sensors can be increased, for example, by setting them as follows: approximately 0.45 for infrared thermal imaging sensors, approximately 0.30 for millimeter-wave radar, approximately 0.15 for visual sensors, and approximately 0.10 for lidar.

[0095] In bumpy road conditions, the inertial measurement unit collects the attitude information of the mining truck. The dynamic weight allocation unit adjusts the real-time environmental adaptability of each sensor according to attitude changes, and deweights sensors that are sensitive to attitude disturbances or whose data stability decreases accordingly, in order to maintain the stability of the fused data. The above weight values ​​are examples, and the actual values ​​can fluctuate under the corresponding operating conditions according to the sensor configuration and calibration results.

[0096] After obtaining the fusion weights, the dynamic weight allocation unit performs weighted fusion on the preprocessed multi-source sensor data according to the weights to obtain unified fused sensing data.

[0097] Example 4

[0098] This embodiment further explains the preprocessing process in step S2 and the identification and trajectory prediction process in step S4.

[0099] In step S2, the time synchronization unit performs time alignment on the multi-source sensor data based on the timestamp, so that data from different sensors and different sampling frequencies are grouped into the same time reference.

[0100] The noise reduction unit uses Kalman filtering to reduce noise in the millimeter-wave radar data and statistical filtering to remove outliers in the point cloud of the lidar.

[0101] The spatial calibration unit, based on hand-eye calibration, converts data from lidar, visual sensors, and infrared thermal imaging sensors to the coordinate system of the millimeter-wave radar, so that the data from each sensor corresponds to the same target in space.

[0102] In other embodiments, noise reduction can also be achieved using median filtering, wavelet denoising, or other methods, and spatial calibration can also be achieved using joint calibration based on a calibration board.

[0103] In step S4, the dynamic obstacle recognition and trajectory prediction module first identifies the obstacle type and then predicts its trajectory. During recognition, it distinguishes between heat source obstacles and non-heat source obstacles based on the thermal radiation information in the fused sensing data: people, moving vehicles, etc., which have obvious thermal radiation, are classified as heat source obstacles; falling rocks, etc., which have small temperature differences from the background and have insignificant thermal radiation, are classified as non-heat source obstacles, and are identified by combining point cloud contours and image textures. In some embodiments, obstacle type recognition is achieved through a deep learning-based target detection network, such as the YOLO series of networks, and in a specific example, an improved YOLOv8 network is used; the three-dimensional contour features of the obstacle can be extracted through a point cloud target detection network, such as the PointPillars network. In other embodiments, the target detection network can also be Faster R-CNN, SSD, etc., and the point cloud processing network can also be VoxelNet, etc., all of which can achieve obstacle type recognition.

[0104] During trajectory prediction, a motion model of the dynamic obstacle is established using a Long Short-Term Memory (LSTM) network combined with Kalman filtering, based on the obstacle's historical position, velocity, and acceleration information. In a specific example, the LSM network's prediction output is: In the formula, Obstacles predicted for long short-term memory networks in the future Location at any given moment This corresponds to the next 1 to 5 seconds.

[0105] The current position of the obstacle; Set the length of the historical data window, for example, to 10; The prediction function is a Long Short-Term Memory network, which is trained using a dynamic obstacle dataset of a mining scene.

[0106] The prediction results of the Long Short-Term Memory (LSTM) network are then corrected by Kalman filtering. The predicted positions of the LSM network and the Kalman filter are then weighted and fused according to their respective prediction error covariance. In the formula, For the final predicted location; The position predicted by the Kalman filter; It is the identity matrix; For filter gain, in The prediction error covariance of the Long Short-Term Memory network. The prediction error covariance of the Kalman filter is represented by the value of the side with the smaller prediction error covariance, which carries a greater weight in the fusion result. In other embodiments, the Long Short-Term Memory network can also be replaced by a time-series model such as a gated recurrent unit or a temporal convolutional network.

[0107] In some embodiments, the output module, while outputting the obstacle type and trajectory, also determines the risk level of the obstacle based on the relative distance, relative speed, and relative acceleration between the dynamic obstacle and the unmanned mining truck, and outputs the risk level to the decision control system. The risk level is used to classify the degree of danger of the obstacle; the closer the distance and the greater the relative speed, the higher the risk level.

[0108] In a specific example, risk level Calculate using the following formula: In the formula, The risk level of the obstacle ranges from 0 to 10. Low risk Medium risk High risk; The relative velocity between the obstacle and the mining truck is expressed in m / s. The value is positive when the obstacle is facing the mining truck and negative when it is moving away from it. This represents the real-time distance between the obstacle and the mining truck, in meters (m). The relative acceleration between the obstacle and the mining truck is expressed in m / s², and is positive when accelerating towards it. The safe distance in mining scenarios is preset to 50m and can be dynamically adjusted according to the speed of the mining truck. , These are weighting coefficients, calibrated experimentally, for example, taking... For 6, The value is set to 4 to ensure a relatively balanced contribution of the two indicators to the risk level. When the calculated result exceeds 0 to 10, the corresponding boundary value is set to 0 or 10. The decision control system can adopt different obstacle avoidance strategies based on the risk level it belongs to.

[0109] It should be noted that the functional modules and units in this invention can be integrated into the data processing unit in software form or implemented by dedicated hardware; the model, quantity and installation position of each sensor can be adjusted according to the type of mining truck and the operating scenario. For example, a model with a longer detection range can be selected for millimeter-wave radar, and a model with higher thermal sensitivity can be selected for infrared thermal imaging sensor, without affecting the implementation of this invention.

[0110] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0112] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above invention, and these changes or modifications still fall within the scope of the present invention.

Claims

1. A dynamic obstacle perception system for unmanned mining operations, characterized in that, It includes a multi-sensor perception module, a data preprocessing module, an adaptive fusion algorithm module, a dynamic obstacle recognition and trajectory prediction module, and an output module; The multi-sensor perception module is installed on the unmanned mining truck and is used to collect multi-source sensor data of the mining environment. The data preprocessing module is connected to the multi-sensor sensing module and is used to preprocess the multi-source sensor data to obtain preprocessed multi-source sensor data. The adaptive fusion algorithm module is connected to the data preprocessing module and includes an environmental assessment unit and a dynamic weight allocation unit. The environmental assessment unit is used to determine the current environmental type of the unmanned mining truck. The dynamic weight allocation unit is used to determine the fusion weight of each sensor according to the environmental type, and to perform weighted fusion of the preprocessed multi-source sensor data based on the fusion weight to obtain fused perception data. The dynamic obstacle recognition and trajectory prediction module is connected to the adaptive fusion algorithm module and is used to identify the type of dynamic obstacle based on the fused perception data and predict the motion trajectory of the dynamic obstacle. The output module is connected to the dynamic obstacle recognition and trajectory prediction module and is used to output the type and trajectory of the dynamic obstacle to the decision control system of the unmanned mining truck.

2. The dynamic obstacle perception system for unmanned mining operations according to claim 1, characterized in that, The multi-sensor perception module includes millimeter-wave radar, lidar, visual sensor and infrared thermal imaging sensor, with each sensor respectively installed on the roof, front and sides of the unmanned mining truck. The data preprocessing module includes a time synchronization unit, a noise reduction unit, and a spatial calibration unit connected in sequence.

3. The dynamic obstacle perception system for unmanned mining operations according to claim 1, characterized in that, It also includes an inertial measurement unit and a data processing unit, both of which are mounted on the unmanned mining truck; The inertial measurement unit is connected to the adaptive fusion algorithm module and is used to collect the attitude information of the unmanned mining truck. The adaptive fusion algorithm module and the dynamic obstacle recognition and trajectory prediction module are integrated into the data processing unit.

4. A dynamic obstacle perception method for unmanned mining operations, characterized in that, include: Multi-source sensor data of the mining environment are collected by millimeter-wave radar, lidar, visual sensors and infrared thermal imaging sensors installed on unmanned mining trucks. The multi-source sensor data is time-synchronized, noise-reduced, and spatially calibrated to obtain pre-processed multi-source sensor data. The environment type of the unmanned mining truck is determined based on the preprocessed multi-source sensor data. The fusion weight of each sensor is determined based on the environment type. The preprocessed multi-source sensor data is then weighted and fused based on the fusion weight to obtain fused perception data. The type of dynamic obstacle is identified based on the fused perception data, and the trajectory of the dynamic obstacle is predicted. The type and trajectory of the dynamic obstacle are output to the decision control system of the unmanned mining truck.

5. The dynamic obstacle perception method for unmanned mining operations according to claim 4, characterized in that, A method for determining the current environment type of the unmanned mining truck includes: Analyze the confidence level of the multi-source sensor data, and determine the environment type based on the confidence level. The environment type includes at least one of the following: normal environment, dusty environment, foggy environment, nighttime environment, and bumpy road environment. The confidence level includes at least one of the following: lidar point cloud integrity, visual image sharpness, millimeter-wave radar signal strength, and infrared thermal imaging contrast.

6. The dynamic obstacle perception method for unmanned mining operations according to claim 4, characterized in that, The method for determining the fusion weights of each sensor based on the environment type includes: The fusion weights are determined based on the inherent confidence coefficients of each sensor and the real-time environment adaptability. The inherent confidence coefficients are pre-calibrated, and the real-time environment adaptability is determined based on the environment type. The fusion weight is calculated using the following formula: In the formula, For the first The fusion weights of individual sensors, For the first The inherent confidence coefficient of each sensor, For the first Real-time environmental adaptability of individual sensors This represents the total number of sensors.

7. The dynamic obstacle perception method for unmanned mining operations according to claim 6, characterized in that, When the environment type is a dusty environment or a foggy environment, the fusion weight of the millimeter-wave radar and the infrared thermal imaging sensor is increased; When the environment type is nighttime, increase the fusion weight of the infrared thermal imaging sensor; When the environment type is a bumpy road environment, the fusion weights of each sensor are adjusted according to the attitude information collected by the inertial measurement unit.

8. The dynamic obstacle perception method for unmanned mining operations according to claim 4, characterized in that, A method for processing the multi-source sensor data includes: Time alignment of the multi-source sensor data is performed based on timestamps; Kalman filtering is used to reduce noise in the millimeter-wave radar data, and statistical filtering is used to remove outliers in the point cloud of the lidar. Based on hand-eye calibration, the data from the lidar, vision sensor, and infrared thermal imaging sensor are converted to the coordinate system of the millimeter-wave radar.

9. The dynamic obstacle perception method for unmanned mining operations according to claim 4, characterized in that, A method for predicting the trajectory of the dynamic obstacle includes: Based on the thermal radiation information in the fused sensing data, heat source obstacles and non-heat source obstacles are distinguished; The motion trajectory of the dynamic obstacle is predicted by using a long short-term memory network combined with a Kalman filter.

10. The dynamic obstacle perception method for unmanned mining operations according to claim 4, characterized in that, Also includes: The risk level of the dynamic obstacle is determined based on the relative distance and relative speed between the dynamic obstacle and the unmanned mining truck, and the risk level is output to the decision control system of the unmanned mining truck.

Images