Multi-mode competitive collaborative fusion sensing method for unmanned vehicle in mining area

Abstract

The invention discloses a mining area unmanned vehicle multi-mode competitive collaborative fusion sensing method, and relates to the technical field of fusion sensing, and the method comprises the following implementation steps: employing infrared thermal imaging and visible light image fusion to forward sense an obstacle, employing an ultrasonic radar and a laser infrared distance measurement sensor to sense a near-area obstacle, and employing a laser infrared distance measurement sensor to sense the near-area obstacle; whether an obstacle approaches or not is judged according to distance measurement information of sensors at four corners and four sides; a laser radar, a depth visible light imaging head and an inertial measurement unit are fused to perceive obstacles, and six modes are fused to construct a complete environment perception domain. According to the multi-mode competitive collaborative fusion perception method for the unmanned carrying vehicle in the mining area, posture changes are detected and tracked in real time, the angle of a vehicle steering engine is adjusted in advance, it is guaranteed that an image can accurately reflect an actual scene, the precision of state prediction of the carrying vehicle is further improved, and the safety and reliability of operation of the unmanned carrying vehicle in the mining area are improved.

Description

Technical Field

[0001] The present invention relates to the field of fusion perception technology, and in particular to a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas. Background Art

[0002] As mining progresses, the mining environment becomes increasingly complex and diverse, which places higher demands on the safety and efficiency of mining operations. Traditional mining perception methods mainly rely on a single type of sensor, such as lidar, cameras, and radars. However, due to the variability and complexity of the mining environment, a single sensor is often severely limited in its perception capabilities when faced with severe weather, changing lines, and especially in situations such as insufficient lighting, dust, and humidity underground. In addition, mining areas usually have complex terrain with a large number of obstacles, slopes, potholes, etc., which poses a huge challenge to the equipment's precise navigation and obstacle detection.

[0003] Publication No. CN115236673A discloses a multi-radar fusion perception system and method for large vehicles, which includes: storing the data of each sensor in a domain controller and establishing a cache area; regularly reading the data in the cache area to obtain synchronized data; and inputting it into the perception software system module; extracting boundaries from the laser point cloud data, performing point cloud clustering on obstacle points and suspected obstacle points; and constructing obstacle information; fusing the traditional laser radar module with the laser deep learning module to obtain the perception results and the millimeter wave radar perception results, and fusing the two to obtain the final detection results. The present invention ensures the accuracy of the perception system, reduces the impact of the system on the harsh environment, improves the accuracy and recall rate of obstacle detection, and improves the detection reliability.

[0004] After searching the above patents, it was found that there are still some deficiencies in the operation of unmanned transport vehicles in mining areas: 1. When the robot or vehicle is driving, it will encounter uneven roads, which will cause obvious jitter in the video image, which is not conducive to subsequent image processing and affects the stability of vehicle operation; 2. In the process of capturing, sampling, quantifying and transmitting visible light and infrared images, these images are easily interfered by various noises in the external environment, resulting in the image being unable to accurately reflect the actual scene, which in turn causes poor image quality and affects the operation of the vehicle; 3. When driving, unmanned transport vehicles in mining areas will encounter uneven roads. The vehicle may not be able to avoid obstacles in advance and accurately identify obstacles, affecting the safety and reliability of the operation of unmanned vehicles in mining areas.

[0005] Therefore, a new multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas is proposed to solve the above problems. Summary of the invention

[0006] The main purpose of the present invention is to provide a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas to solve the problems raised in the above background.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is: a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas, the method comprising the following implementation steps:

[0008] S1: Use infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction, detect and track posture changes in real time, generate steering control signals based on errors, automatically adjust the vehicle steering angle, and perform image denoising, image registration and image fusion;

[0009] S2: Use ultrasonic radar and laser infrared ranging sensors to sense nearby obstacles, and use the ranging information of sensors at the four corners and four sides to determine whether obstacles are approaching;

[0010] S3: It uses laser radar, deep visible light imaging head and inertial measurement unit to sense obstacles, combined with millimeter wave radar to sense obstacles and perform target screening;

[0011] S4: The six modalities are integrated to construct a complete environmental perception domain. The six modalities are fused according to different algorithms through the extended Kalman filter fusion algorithm to obtain the multi-modal competitive collaborative fusion perception results of unmanned vehicles in mining areas.

[0012] The method of using infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction in S1 includes image stabilization, image denoising, image registration, fusion of infrared thermal imaging image and visible light image, and fusion image obstacle detection, wherein the method of image stabilization is as follows:

[0013] S111. Install a dual-degree-of-freedom servo and attitude sensor for the camera on the vehicle;

[0014] S112, using Kalman filtering to fuse sensor data to obtain the current posture;

[0015] S113, calculating the error between the current posture of the vehicle and the target posture, the calculation formula is as follows:

[0016]

[0017] Among them, the basic center point of the vehicle posture sensor is set as the origin, and the coordinate system is established to obtain the error between the current posture angle of the vehicle and the target posture angle. α is the angle between the current posture angle of the vehicle and the target posture angle in the positive direction of the X-axis, θ is the angle between the current posture angle and the target posture angle in the positive direction of the Y-axis, A is the vehicle height, Δx is the vector of the vehicle posture in the X-axis direction, Δy is the vector of the vehicle posture in the Y-axis direction, if the vector is equal to 0, it means the posture is normal, if the vector is not equal to 0, it means the posture is abnormal;

[0018] S114, using a PID control algorithm, generating a steering gear control signal according to the error, sending the control signal to the steering gear, and adjusting the angle of the transport vehicle according to the error data.

[0019] The image denoising is used to calculate the denoising value of each pixel block, and the specific steps are as follows:

[0020] The denoising value of each pixel is calculated using similar pixel blocks in the image. The calculation formula is as follows: Given an image I, the denoising value of the pixel value I(i) at position i can be expressed as:

[0021]

[0022] in, is the pixel value after denoising; Ω is the search window; ω(i,j) is the weight of positions i and j, which is determined by the similarity of the pixel blocks corresponding to the two;

[0023] The weight ω(i,j) is defined as:

[0024]

[0025] Among them, P i and P j are the pixel blocks around i and j respectively; Z(i) is the normalization constant; h is the smoothing parameter that controls the degree of denoising.

[0026] The steps of image registration are as follows:

[0027] S131, applying SURF algorithm to detect key points in the image;

[0028] S132, using FLANN to perform feature point matching;

[0029] S133, eliminating wrong matches and estimating model parameters through RANSAC algorithm;

[0030] S134, calculating affine transformation parameters based on the accurately matched feature points, and generating an infrared image and a visible light image with the same field of view.

[0031] The infrared thermal imaging image and the visible light image are fused by using a TIF algorithm to fuse the registered images;

[0032] The fusion image obstacle detection steps are as follows:

[0033] S151. In the ground and underground environment of the mining area, the environment is photographed by infrared thermal imaging cameras and visible light imaging cameras, and diversified image data under different lighting conditions and weather conditions are obtained in real time by data trackers and data collectors;

[0034] S152, receiving diversified image data in real time through a data receiver, and fusing the infrared thermal imaging image with the visible light image according to TIF;

[0035] S153, annotating the fused image in detail, where the detailed annotation content includes the operating personnel, vehicles, mining equipment and obstacles in the mining area;

[0036] S154. After the labeling is completed, the fused image data set is trained using the YOLOv8 detection algorithm, and the training process combines the data enhancement technology to perform at least one of image rotation, scaling or flipping;

[0037] S155. After the training is completed, the trained model is used to perform real-time obstacle detection on the mining area image, the visible light imaging head and the infrared camera are used to obtain and fuse the image in real time, and the target object is identified through the YOLOV8 target detection algorithm.

[0038] S2 includes ranging perception, elimination of noise and outliers, confidence weighted fusion, and comprehensive judgment of whether an obstacle is approaching;

[0039] The distance sensing is used to measure the distance between the vehicle and the obstacle in real time through the TOF algorithm. The ultrasonic radar calculation formula is:

[0040]

[0041] where d us is the distance between the sensor and the target object; v is the propagation speed of ultrasound in the air; t is the total time from the emission of ultrasound to the receipt of the reflected signal;

[0042] Laser infrared ranging sensor:

[0043]

[0044] Among them, d ir is the distance between the sensor and the target object; c is the speed of light; t is the total time from the emission of laser or infrared light to the receipt of the reflected signal;

[0045] Noise and outlier elimination Kalman filter algorithm is used to eliminate environmental noise and outliers;

[0046] The confidence weighted fusion dynamically adjusts the confidence of the sensor according to the environmental conditions, including light intensity and weather conditions. At long distances, the confidence of the laser sensor is higher; at close distances, the confidence of the ultrasonic radar is higher. The confidence of the two sensors is combined to weight the detection results, and the final position and distance of the surrounding obstacles are calculated using the formula:

[0047]

[0048] Among them, w s and w ir are the confidence weights of ultrasonic radar and laser infrared ranging sensor, d us and d ir They are the filtered measured distances of ultrasonic radar and laser infrared sensor respectively;

[0049] The comprehensive judgment of whether an obstacle is approaching is used to judge whether an obstacle is approaching through the distance measurement results of the sensors at the four corners and four sides, and the chassis sensor is used to judge whether there is a step or a road depression in front. If d final If it is less than 1m, it means the obstacle is approaching. final If it is greater than or equal to 1m, it means that the obstacle is not approaching.

[0050] The S3 uses a laser radar, a deep visible light imaging head and an inertial measurement unit to sense obstacles and generate a point cloud map through the R3LIVE algorithm.

[0051] The steps of the millimeter wave radar sensing obstacles are as follows:

[0052] S31. Processing the radar signal by a constant false alarm rate algorithm, and calculating in real time the false alarm rate of the millimeter wave radar perception in a noisy environment, the calculation formula is E=N(PFA-1 / N-1), where N-1 represents the set detection threshold, and E represents the probability density function of the detection statistic N under the noise assumption condition;

[0053] S32. Use the OS CFAR algorithm to effectively screen the target and obtain effective data of millimeter wave radar perception. The screening method is as follows:

[0054] If there are at least min P points (including p itself) in the ε-neighborhood of a point p, then p is a valid data, formally defined as:

[0055]

[0056] Among them, |N ΕP| represents the number of valid data in the ε-neighborhood of p, Core(p) represents the data set of the ε-neighborhood, and min P represents the valid data of the ε-neighborhood;

[0057] S33. Cluster multiple valid data echoes from the same object into one target using the DBSCAN algorithm.

[0058] The S4 includes time synchronization, space synchronization and data fusion to build a perception map;

[0059] The time synchronization is used to achieve data time synchronization by transferring the data of each sensor into the buffer area and aligning the data frame time;

[0060] The spatial synchronization is used to arrange the position of the sensor and perform coordinate calibration conversion, as follows:

[0061] The coordinates (x′, y′, z′) of the point (x, y, z) after rotating counterclockwise around the origin by angle θ are converted using the following formula:

[0062]

[0063] in, Represents the fused data after 3D coordinate calibration and conversion.

[0064] The data fusion to construct the perception map is used to fuse the six modalities according to different algorithms through the extended Kalman filter fusion algorithm to obtain the environmental perception result, which is as follows:

[0065] S41, predict the environmental perception error covariance matrix at the next moment, the formula is:

[0066]

[0067] Among them, P K|K-1 is the prior error covariance of the environment perception at time k, P K-1|K-1 yes K-1 The covariance of the posterior error of environmental perception at the moment, F K is the Jacobian matrix of the environmental state transfer function, Q K is the noise covariance matrix in the environmental perception process;

[0068] S42. By iteratively running the EKF algorithm, the above formula is repeatedly calculated to obtain the average value of the data set, which is the environmental perception result.

[0069] The present invention has the following beneficial effects:

[0070] 1. In the present invention, by using infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction, and detecting and tracking posture changes in real time, when there is an angle error between the current posture of the vehicle and the target posture, the vehicle steering gear angle can be adjusted in advance when the posture is abnormal, so that the vehicle steering gear angle always tends to a stable range, ensuring that even if the vehicle encounters an uneven road surface during driving, there will be no obvious video screen jitter phenomenon, which increases the convenience for subsequent image processing and improves the stability of vehicle operation;

[0071] 2. In the present invention, ultrasonic radar and laser infrared ranging sensors are used to sense near-field obstacles, and similar pixel blocks in the image are used to calculate the denoising value of each pixel, so that in the process of capturing, sampling, quantizing and transmitting visible light and infrared images, these images can reduce the interference of various noises in the external environment, further ensuring that the images can accurately reflect the actual scene, thereby improving the image quality and ensuring the stability of vehicle data fusion processing;

[0072] 3. In the present invention, the six modes are fused according to different algorithms through the extended Kalman filter fusion algorithm to obtain the multi-modal competitive collaborative fusion perception result of the unmanned transport vehicle in the mining area, and the six modes are fused to construct a complete environmental perception domain, so that the six modes can realize data fusion according to different algorithms, and the final data fusion value is iteratively predicted to further increase the accuracy of the vehicle state prediction, so that the unmanned transport vehicle in the mining area can avoid obstacles in advance and accurately identify obstacles even if it encounters uneven roads when driving, thereby increasing the safety and reliability of the operation of the unmanned transport vehicle in the mining area. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 This is an overall flow chart of a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas according to the present invention;

[0074] Figure 2 This is a schematic diagram of the architecture of step 1 of a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas according to the present invention;

[0075] Figure 3 It is a schematic diagram of the architecture of step 2 of a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas according to the present invention;

[0076] Figure 4 This is a schematic diagram of the architecture of step four of a multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas according to the present invention. DETAILED DESCRIPTION

[0077] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0078] Embodiment 1

[0079] Please refer to Figure 1 to Figure 2 As shown: A multi-modal competitive collaborative fusion perception method for unmanned vehicles in a mining area, the method includes the following implementation steps:

[0080] S1: Use infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction, detect and track posture changes in real time, generate steering control signals based on errors, automatically adjust the vehicle steering angle, and perform image denoising, image registration and image fusion;

[0081] S2: Use ultrasonic radar and laser infrared ranging sensors to sense nearby obstacles, and use the ranging information of sensors at the four corners and four sides to determine whether obstacles are approaching;

[0082] S3: It uses laser radar, deep visible light imaging head and inertial measurement unit to sense obstacles, combined with millimeter wave radar to sense obstacles and perform target screening;

[0083] S4: The six modalities are integrated to construct a complete environmental perception domain. The six modalities are fused according to different algorithms through the extended Kalman filter fusion algorithm to obtain the multi-modal competitive collaborative fusion perception results of unmanned vehicles in mining areas.

[0084] In S1, the forward obstacle perception using infrared thermal imaging and visible light image fusion includes image stabilization, image denoising, image registration, infrared thermal imaging image and visible light image fusion, and fusion image obstacle detection. The image stabilization method is as follows:

[0085] S111. Install a dual-degree-of-freedom servo and attitude sensor for the camera on the vehicle;

[0086] S112, using Kalman filtering to fuse sensor data to obtain the current posture;

[0087] S113, calculating the error between the current posture of the vehicle and the target posture, the calculation formula is as follows:

[0088]

[0089] Among them, the basic center point of the vehicle posture sensor is set as the origin, and the coordinate system is established to obtain the error between the current posture angle of the vehicle and the target posture angle. α is the angle between the current posture angle of the vehicle and the target posture angle in the positive direction of the X-axis, θ is the angle between the current posture angle and the target posture angle in the positive direction of the Y-axis, A is the vehicle height, Δx is the vector of the vehicle posture in the X-axis direction, Δy is the vector of the vehicle posture in the Y-axis direction, if the vector is equal to 0, it means the posture is normal, if the vector is not equal to 0, it means the posture is abnormal;

[0090] S114. Use PID control algorithm to generate a servo control signal according to the error, send the control signal to the servo, adjust the angle of the vehicle according to the error data, and track the posture changes in real time. When there is an angle error between the current posture of the vehicle and the target posture, the vehicle servo angle can be adjusted in advance when the posture is abnormal, so that the vehicle servo angle always tends to be in a stable range.

[0091] Embodiment 2

[0092] Please refer to Figure 3 As shown: Based on the first embodiment, image denoising is used to calculate the denoising value of each pixel block, and the specific steps are as follows:

[0093] The denoising value of each pixel is calculated using similar pixel blocks in the image. The calculation formula is as follows: Given an image I, the denoising value of the pixel value I(i) at position i can be expressed as:

[0094]

[0095] in, is the pixel value after denoising; Ω is the search window; ω(i,j) is the weight of positions i and j, which is determined by the similarity of the pixel blocks corresponding to the two;

[0096] The weight ω(i,j) is defined as:

[0097]

[0098] Among them, P i and P j are the pixel blocks around i and j respectively; Z(i) is the normalization constant; h is the smoothing parameter that controls the degree of denoising.

[0099] The steps of image registration are as follows:

[0100] S131. Use the SURF algorithm to detect key points in the image. SURF is an efficient feature detection method that can identify image features that are robust to rotation, scale changes, and lighting conditions.

[0101] S132, using FLANN for feature point matching. FLANN is an efficient algorithm that can quickly find the nearest neighbor, thereby achieving accurate matching between image features;

[0102] S133, eliminating false matches and estimating model parameters by using the RANSAC algorithm, which is a statistical method used to estimate the parameters of a mathematical model from a set of data containing outliers to ensure the accuracy and reliability of the final model;

[0103] S134, calculating affine transformation parameters based on the accurately matched feature points, and generating an infrared image and a visible light image with the same field of view.

[0104] The infrared thermal imaging image and the visible light image are fused using the TIF algorithm to fuse the registered images;

[0105] The steps of fusion image obstacle detection are as follows:

[0106] S151. In the ground and underground environment of the mining area, the environment is photographed by infrared thermal imaging cameras and visible light imaging cameras, and diversified image data under different lighting conditions and weather conditions are obtained in real time by data trackers and data collectors;

[0107] S152, receiving diversified image data in real time through a data receiver, and fusing the infrared thermal imaging image with the visible light image according to TIF;

[0108] S153, annotating the fused image in detail, where the detailed annotation content includes the operating personnel, vehicles, mining equipment and obstacles in the mining area;

[0109] S154. After the labeling is completed, the fused image data set is trained using the YOLOv8 detection algorithm, and the training process combines the data enhancement technology to perform at least one of image rotation, scaling or flipping;

[0110] S155. After the training is completed, the trained model is used to perform real-time obstacle detection on the mining area images. The visible light imaging head and the infrared camera are used to obtain and fuse the images in real time. The YOLOV8 target detection algorithm is used to identify the target objects. The similar pixel blocks in the image are used to calculate the denoising value of each pixel. In this way, when the vehicle is capturing, sampling, quantizing and transmitting visible light and infrared images, these images can reduce the interference of various noises in the external environment, further ensuring that the images can accurately reflect the actual scene.

[0111] Embodiment 3

[0112] Please refer to Figure 4 As shown: Based on the first embodiment, S2 includes ranging perception, noise and outlier elimination, confidence weighted fusion, and comprehensive judgment of whether an obstacle is approaching;

[0113] Distance sensing is used to measure the distance between the vehicle and the obstacle in real time through the TOF algorithm. The ultrasonic radar calculation formula is:

[0114]

[0115] where d usis the distance between the sensor and the target object; v is the propagation speed of ultrasound in the air; t is the total time from the emission of ultrasound to the receipt of the reflected signal;

[0116] Laser infrared ranging sensor:

[0117]

[0118] Among them, d ir is the distance between the sensor and the target object; c is the speed of light; t is the total time from the emission of laser or infrared light to the receipt of the reflected signal;

[0119] Eliminate noise and outliers Kalman filter algorithm is used to eliminate environmental noise and outliers. The data of each sensor is cached with a timestamp and processed by Kalman filter algorithm to eliminate environmental noise and outliers.

[0120] Confidence weighted fusion dynamically adjusts the confidence of the sensor according to environmental conditions, including light intensity and weather conditions. At long distances, the confidence of the laser sensor is higher; at close distances, the confidence of the ultrasonic radar is higher. The confidence of the two sensors is combined to weight the detection results and calculate the final position and distance of the surrounding obstacles using the formula:

[0121]

[0122] Among them, w s and wi r are the confidence weights of ultrasonic radar and laser infrared ranging sensor, d us and d ir They are the filtered measured distances of ultrasonic radar and laser infrared sensor respectively;

[0123] Comprehensive judgment of whether an obstacle is approaching is used to determine whether an obstacle is approaching through the distance measurement results of the sensors at the four corners and four sides, and the chassis sensor is used to determine whether there are steps or road depressions ahead. If d final If it is less than 1m, it means the obstacle is approaching. final If it is greater than or equal to 1m, it means that the obstacle is not approaching.

[0124] The S3 uses a fusion of laser radar, deep visible light imaging head and inertial measurement unit to sense obstacles and generate a point cloud map through the R3LIVE algorithm. In the R3LIVE algorithm, the laser radar parameters are first set, such as scanning range, resolution, etc. The R3LIVE algorithm will simulate the scanning process of the laser radar, including emitting laser pulses and receiving reflected signals, and then calculate the distance of each point based on the received signal and convert it into three-dimensional coordinates. Finally, all the calculated three-dimensional coordinate points are combined to form a complete point cloud map.

[0125] The steps for millimeter wave radar to sense obstacles are as follows:

[0126] S31. Processing the radar signal through a constant false alarm rate algorithm, and calculating in real time the virtual mirror rate of the millimeter wave radar perception in a noisy environment. The calculation formula is E=N(PFA-1 / N-1), where N-1 represents the set detection threshold, and E represents the probability density function of the detection statistic N under the noise assumption condition. The virtual mirror rate dynamically adjusts the detection threshold by analyzing the noise level around the target, so that a low false alarm rate can be maintained in a high noise environment;

[0127] S32. Use the OS CFAR algorithm to effectively screen the target and obtain effective data of millimeter wave radar perception. The screening method is as follows:

[0128] If there are at least min P points (including p itself) in the ε-neighborhood of a point p, then p is a valid data, formally defined as:

[0129]

[0130] Among them, |N Ε P| represents the number of valid data in the ε-neighborhood of p, Core(p) represents the data set of the ε-neighborhood, and min P represents the valid data of the ε-neighborhood;

[0131] S33. Multiple valid data echoes from the same object are clustered into one target through the DBSCAN algorithm. DBSCAN is a density-based clustering algorithm, which is particularly suitable for processing noisy data sets. In the application of millimeter-wave radar, it can effectively cluster multiple echoes from the same object into one target, and screen the targets detected by the millimeter-wave radar before fusion to improve the running speed and accuracy of the fusion algorithm.

[0132] S4 includes time synchronization, space synchronization and data fusion to build perception maps;

[0133] Time synchronization is used to achieve data time synchronization by transferring the data of each sensor into the buffer area and aligning the data frame time;

[0134] Spatial synchronization is used to arrange the position through the sensor and perform coordinate calibration conversion, as follows:

[0135] The coordinates (x′, y′, z′) of the point (x, y, z) after rotating counterclockwise around the origin by angle θ are converted using the following formula:

[0136]

[0137] in, Represents the fused data after 3D coordinate calibration and conversion.

[0138] Data fusion is used to build a perception map. It is used to fuse the six modalities according to different algorithms through the extended Kalman filter fusion algorithm to obtain the environmental perception results, as follows:

[0139] S41, predict the environmental perception error covariance matrix at the next moment, the formula is:

[0140]

[0141] Among them, P KK-1 is the prior error covariance of the environment perception at time k, P K-1K-1 yes K-1 The covariance of the posterior error of environmental perception at the moment, F K is the Jacobian matrix of the environmental state transfer function, Q K is the noise covariance matrix in the environmental perception process;

[0142] S42. Through the iterative operation of the EKF algorithm, the above formula is repeatedly calculated to obtain the average value of the data set, which is the environmental perception result. The six modes realize data fusion according to different algorithms, and the final data fusion value is iteratively predicted to further increase the accuracy of the state prediction of the transport vehicle. When the unmanned transport vehicle in the mining area is driving, even if it encounters uneven roads, the vehicle can avoid obstacles in advance and accurately identify obstacles, thereby increasing the safety and reliability of the operation of unmanned transport vehicles in the mining area.

[0143] In the present invention, a multi-modal competitive collaborative fusion perception method for unmanned transport vehicles in mining areas is firstly provided with a dual-degree-of-freedom steering gear and a posture sensor for the camera on the vehicle, and obstacles are perceived forward by using infrared thermal imaging and visible light image fusion, and posture changes are detected and tracked in real time. A steering gear control signal is generated according to the error, and the vehicle steering gear angle is automatically adjusted, and image denoising, image registration and image fusion are performed. When there is an angular error between the current posture of the vehicle and the target posture, the vehicle steering gear angle can be adjusted in advance when the posture is abnormal, so that the vehicle steering gear angle always tends to a stable range, ensuring that even if the vehicle encounters an uneven road surface during driving, there will be no obvious video screen jitter phenomenon, which increases the convenience for subsequent image processing and improves the stability of vehicle operation; ultrasonic radar and laser infrared ranging sensors are used to perceive near-area obstacles, and the ranging information of the sensors at the four corners and four sides is used to determine whether the obstacle is approaching; laser radar, deep The visible light imaging head and inertial measurement unit are integrated to perceive obstacles, and the millimeter wave radar is combined to perceive obstacles and perform target screening, so that in the process of capturing, sampling, quantizing and transmitting visible light and infrared images, these images can reduce the interference of various noises in the external environment, further ensuring that the images can accurately reflect the actual scene, thereby improving the image quality and ensuring the stability of vehicle data fusion processing; the six modes are integrated to construct a complete environmental perception domain, and the six modes are fused according to different algorithms through the extended Kalman filter fusion algorithm to obtain the multi-modal competition and collaborative fusion perception results of unmanned vehicles in mining areas, so that the six modes can realize data fusion according to different algorithms, and the final data fusion value is iteratively predicted to further increase the accuracy of vehicle state prediction, so that when the unmanned vehicles in mining areas are driving, even if they encounter uneven roads, the vehicles can avoid obstacles in advance and accurately identify obstacles. The whole method is simple and efficient, and is suitable for the multi-modal competition and collaborative fusion perception method of unmanned vehicles in mining areas.

[0144] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-modal competitive collaborative fusion perception method for unmanned vehicles in mining areas, characterized in that: The method comprises the following implementation steps: S1: Use infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction, detect and track posture changes in real time, generate steering control signals based on errors, automatically adjust the vehicle steering angle, and perform image denoising, image registration and image fusion; S2: Use ultrasonic radar and laser infrared ranging sensors to sense nearby obstacles, and use the ranging information of sensors at the four corners and four sides to determine whether obstacles are approaching; S3: It uses laser radar, deep visible light imaging head and inertial measurement unit to sense obstacles, combined with millimeter wave radar to sense obstacles and perform target screening; S4: The six modalities are integrated to construct a complete environmental perception domain. The six modalities are fused according to different algorithms through the extended Kalman filter fusion algorithm to obtain the multi-modal competitive collaborative fusion perception results of unmanned vehicles in mining areas.

2. The method according to claim 1, characterized in that: The method of using infrared thermal imaging and visible light image fusion to perceive obstacles in the forward direction in S1 includes image stabilization, image denoising, image registration, fusion of infrared thermal imaging image and visible light image, and fusion image obstacle detection, wherein the method of image stabilization is as follows: S111. Install a dual-degree-of-freedom servo and attitude sensor for the camera on the vehicle; S112, using Kalman filtering to fuse sensor data to obtain the current posture; S113, calculating the error between the current posture of the vehicle and the target posture, the calculation formula is as follows: Among them, the basic center point of the vehicle posture sensor is set as the origin, and the coordinate system is established to obtain the error between the current posture angle of the vehicle and the target posture angle. α is the angle between the current posture angle of the vehicle and the target posture angle in the positive direction of the X-axis, θ is the angle between the current posture angle and the target posture angle in the positive direction of the Y-axis, A is the vehicle height, Δx is the vector of the vehicle posture in the X-axis direction, Δy is the vector of the vehicle posture in the Y-axis direction, if the vector is equal to 0, it means the posture is normal, if the vector is not equal to 0, it means the posture is abnormal; S114, using a PID control algorithm, generating a steering gear control signal according to the error, sending the control signal to the steering gear, and adjusting the angle of the transport vehicle according to the error data.

3. The method according to claim 2, characterized in that: The image denoising is used to calculate the denoising value of each pixel block, and the specific steps are as follows: The denoising value of each pixel is calculated using similar pixel blocks in the image. The calculation formula is as follows: Given an image I, the denoised value of its pixel value I(i) at position i can be expressed as: in, is the pixel value after denoising; Ω is the search window; ω(i,j) is the weight of positions i and j, which is determined by the similarity of the pixel blocks corresponding to the two; The weight ω(i,j) is defined as: Among them, P i and P j are the pixel blocks around i and j respectively; Z(i) is the normalization constant; h is the smoothing parameter that controls the degree of denoising.

4. The method according to claim 3, characterized in that: The steps of image registration are as follows: S131, applying SURF algorithm to detect key points in the image; S132, using FLANN to perform feature point matching; S133, eliminating wrong matches and estimating model parameters through RANSAC algorithm; S134, calculating affine transformation parameters based on the accurately matched feature points, and generating an infrared image and a visible light image with the same field of view.

5. The method according to claim 2, characterized in that: The infrared thermal imaging image and the visible light image are fused by using a TIF algorithm to fuse the registered images; The fusion image obstacle detection steps are as follows: S151. In the ground and underground environment of the mining area, the environment is photographed by infrared thermal imaging cameras and visible light imaging cameras, and diversified image data under different lighting conditions and weather conditions are obtained in real time by data trackers and data collectors; S152, receiving diversified image data in real time through a data receiver, and fusing the infrared thermal imaging image with the visible light image according to TIF; S153, annotating the fused image in detail, where the detailed annotation content includes the operating personnel, vehicles, mining equipment and obstacles in the mining area; S154. After the labeling is completed, the fused image data set is trained using the YOLOv8 detection algorithm, and the training process combines the data enhancement technology to perform at least one of image rotation, scaling or flipping; S155. After the training is completed, the trained model is used to perform real-time obstacle detection on the mining area image, the visible light imaging head and the infrared camera are used to obtain and fuse the image in real time, and the target object is identified through the YOLOV8 target detection algorithm.

6. The method according to claim 1, characterized in that: S2 includes ranging perception, elimination of noise and outliers, confidence weighted fusion, and comprehensive judgment of whether an obstacle is approaching; The distance sensing is used to measure the distance between the vehicle and the obstacle in real time through the TOF algorithm. The ultrasonic radar calculation formula is: where d us is the distance between the sensor and the target object; v is the propagation speed of ultrasound in the air; t is the total time from the emission of ultrasound to the receipt of the reflected signal; Laser infrared ranging sensor: Among them, d ir is the distance between the sensor and the target object; c is the speed of light; t is the total time from the emission of laser or infrared light to the receipt of the reflected signal; Noise and outlier elimination Kalman filter algorithm is used to eliminate environmental noise and outliers; The confidence weighted fusion dynamically adjusts the confidence of the sensor according to the environmental conditions, including light intensity and weather conditions. At long distances, the confidence of the laser sensor is higher; at close distances, the confidence of the ultrasonic radar is higher. The confidence of the two sensors is combined to weight the detection results, and the final position and distance of the surrounding obstacles are calculated using the formula: Among them, w s and w ir are the confidence weights of ultrasonic radar and laser infrared ranging sensor, d us and d ir They are the filtered measured distances of ultrasonic radar and laser infrared sensor respectively; The comprehensive judgment of whether an obstacle is approaching is used to judge whether an obstacle is approaching through the distance measurement results of the sensors at the four corners and four sides, and the chassis sensor is used to judge whether there is a step or a road depression in front. If d final If it is less than 1m, it means the obstacle is approaching. final If it is greater than or equal to 1m, it means that the obstacle is not approaching.

7. The method according to claim 1, characterized in that: The S3 uses a laser radar, a deep visible light imaging head and an inertial measurement unit to sense obstacles and generate a point cloud map through the R3LIVE algorithm.

8. The method according to claim 7, characterized in that: The steps of the millimeter wave radar sensing obstacles are as follows: S31. Processing the radar signal by a constant false alarm rate algorithm, and calculating in real time the false alarm rate of the millimeter wave radar perception in a noisy environment, the calculation formula is E=N(PFA-1 / N-1), where N-1 represents the set detection threshold, and E represents the probability density function of the detection statistic N under the noise assumption condition; S32. Use the OS CFAR algorithm to effectively screen the target and obtain effective data of millimeter wave radar perception. The screening method is as follows: If there are at least min P points (including p itself) in the ε-neighborhood of a point p, then p is a valid data, formally defined as: Among them, |N Ε P| represents the number of valid data in the ε-neighborhood of p, Core(p) represents the data set of the ε-neighborhood, and min P represents the valid data of the ε-neighborhood; S33. Cluster multiple valid data echoes from the same object into one target using the DBSCAN algorithm.

9. The method according to claim 1, characterized in that: The S4 includes time synchronization, space synchronization and data fusion to build a perception map; The time synchronization is used to achieve data time synchronization by transferring the data of each sensor into the buffer area and aligning the data frame time; The spatial synchronization is used to arrange the position of the sensor and perform coordinate calibration conversion, as follows: The coordinates (x′, y′, z′) of the point (x, y, z) after rotating counterclockwise around the origin by angle θ are converted using the following formula: in, Represents the fused data after 3D coordinate calibration and conversion.

10. The method according to claim 9, characterized in that: The data fusion to construct the perception map is used to fuse the six modalities according to different algorithms through the extended Kalman filter fusion algorithm to obtain the environmental perception result, which is as follows: S41, predict the environmental perception error covariance matrix at the next moment, the formula is: Among them, P KK-1 is the prior error covariance of the environment perception at time k, P K-1K-1 yes K-1 The covariance of the posterior error of environmental perception at the moment, F K is the Jacobian matrix of the environmental state transfer function, Q K is the noise covariance matrix in the environmental perception process; S42. By iteratively running the EKF algorithm, the above formula is repeatedly calculated to obtain the average value of the data set, which is the environmental perception result.

Images