Mine broken unmanned aerial vehicle inspection method based on fusion of multispectral image and laser radar data

Abstract

The mine broken unmanned plane inspection method based on multispectral image and laser radar data fusion contains a mine broken unmanned plane inspection auxiliary device, a data fusion processing module and a monitoring result output module; the mine broken unmanned plane inspection auxiliary device includes a mine inspection unmanned plane and a ore pile light supplement instrument; a multispectral image acquisition module and a laser radar data acquisition module are installed on the unmanned plane; the multispectral image acquisition module can adjust the shooting angle according to different inspection requirements during the flight of the unmanned plane, and collect multispectral image data of multiple angles for the same feature; the laser radar data acquisition module obtains three-dimensional point cloud data of the mine broken area, and extracts the spatial distribution and granularity size information of the mine broken area; the data fusion processing module fuses the multispectral image data, the three-dimensional point cloud data, the unmanned plane flight attitude data and the environmental meteorological data. The application improves the monitoring efficiency and safety, and improves the accuracy of image analysis.

Description

Technical Field

[0001] This invention relates to a method for unmanned aerial vehicle (UAV) inspection of mine crushing based on the fusion of multispectral images and lidar data. Background Technology

[0002] During the mining process, it is essential to grasp the geological structure of the mine in a timely and accurate manner. Currently, the methods for studying the geological structure of mines are plan and cross-sectional line drawings, which are quite difficult and have low visualization capabilities, failing to present the structural situation in a visual way. With the development of drone technology, drones are now widely used in agricultural and mining production, especially in terrain surveying and information monitoring, where they are superior to older methods. Therefore, the application of drones in the mining field is becoming increasingly widespread and mature. Blasting in mines is very complex. Before and after blasting, the terrain must be carefully surveyed, and the mine situation must be monitored in real time during blasting to prevent unauthorized personnel from entering the blasting area. Therefore, drones are needed for observation.

[0003] During blasting, a large number of sharp objects such as gravel will fly up from the blasting area. After the blast, the stone is crushed. In the process of crushing the stone, relevant equipment such as cone crushers and sand making machines are required.

[0004] Due to the complex geographical environment of the mine (complex outdoor weather conditions and a lot of dust generated during blasting), the image information obtained by drone inspections is of poor clarity. Therefore, it is impossible to obtain accurate inspection results on the blasting situation and particle size of the mine.

[0005] Traditional methods for monitoring crushing performance in mines primarily rely on manual inspections and simple image analysis. These methods have significant shortcomings, such as the difficulty of image analysis and inaccurate equipment status monitoring, resulting in the inability to adjust crushing parameters in real time and impacting product quality and resource utilization. Therefore, achieving efficient and accurate monitoring of crushing performance in mines has become an important research direction in the field of mineral processing technology.

[0006] While existing technologies have improved the monitoring of crushing effects in mines to some extent, several problems and shortcomings remain. First, current manual inspection methods are inefficient, unsafe, unable to achieve real-time monitoring, and highly susceptible to human error. Second, existing image analysis methods suffer from significant limitations in areas such as the difficulty of image analysis and inaccurate equipment status monitoring, making accurate monitoring of crushing effects difficult. Furthermore, existing technical solutions have limitations in practical applications and cannot meet the monitoring needs of crushing effects in different mining environments. Therefore, developing a novel method for monitoring crushing effects in mines has significant practical importance and application value. Summary of the Invention

[0007] The present invention aims to solve three technical problems: 1) the low efficiency and poor safety of traditional manual inspection, which makes it impossible to achieve real-time monitoring; 2) the technical problems of simple image analysis methods in terms of high image analysis difficulty and inaccurate equipment status monitoring; and 3) the technical problems of existing technical solutions failing to meet the monitoring needs of crushing effect in different mining environments. Therefore, the present invention provides a mine crushing drone inspection method based on the fusion of multispectral images and lidar data.

[0008] To solve the above problems, the present invention is achieved through the following technical solution: A method for mine crushing drone inspection based on multispectral image and lidar data fusion includes a mine crushing drone inspection auxiliary device, a data fusion processing module, and a monitoring result output module. The aforementioned mine crushing drone inspection auxiliary device includes a mine inspection drone and a ore pile supplementary lighting device. Both the mine inspection drone and the ore pile supplementary lighting device are connected to the main controller. The data fusion processing module and the monitoring result output module run in the main controller, which is located in the mine control room. Both the multispectral image acquisition module and the lidar data acquisition module are installed on the UAV. The multispectral image acquisition module can adjust the shooting angle according to different inspection needs during the flight of the UAV to acquire multispectral image data of the same feature from multiple angles. The lidar data acquisition module acquires three-dimensional point cloud data of the crushed area of ​​the mine and extracts the spatial distribution and particle size information of the crushed area of ​​the mine. The data fusion processing module establishes a multi-dimensional data fusion model, which integrates multispectral image data, 3D point cloud data, UAV flight attitude data and environmental meteorological data to establish multi-dimensional data correlation relationships. The monitoring result output module displays the real-time monitoring results of the mine crushing effect. When the monitoring results are abnormal, the alarm device issues an alarm signal to prompt relevant personnel to handle the situation.

[0009] The method includes the following steps: Step 1: The drone takes off, and the multispectral camera and lidar sensor begin collecting data on the crushed area of ​​the mine; Step 2: Obtain multi-source data; Step 2.1: The multispectral camera acquires multispectral image data of the crushed area in the mine and transmits it to the main controller through the multispectral image acquisition module; the multispectral camera acquires multispectral image data of the crushed area in the mine, including four bands: red, green, blue, and near-infrared. Step 2.2: The lidar sensor collects three-dimensional point cloud data of the crushed area in the mine and transmits it to the main controller through the lidar data acquisition module; Step 2.3: Various sensors, including speed sensors, acceleration sensors and GPS sensors, are installed on the drone to obtain the drone's flight attitude data and transmit the drone's flight attitude data to the main controller; Step 2.4: Install temperature and humidity sensors and wind speed sensors on the mine to obtain environmental meteorological data and transmit the environmental meteorological data to the main controller; Step 3: Input the collected multispectral image data, 3D point cloud data, UAV flight attitude data and environmental meteorological data into the multidimensional data fusion model, establish the correlation between the data, identify the characteristics and boundaries of the crushed material, and predict the crushing particle size based on the spectral characteristics; Step 4: Based on the data fusion results, determine the particle size distribution of the crushed material and adjust the crushing parameters of the crushing equipment in real time; Step 5: Display the monitoring results on the display device in real time. When the monitoring results are abnormal, the alarm device will issue an alarm signal to prompt relevant personnel to handle the situation.

[0010] Step 2.1 specifically includes: Step 2.1.1: Adjust the initial position of the ore pile supplementary light device located near the ore pile, cone crusher, or sand making machine discharge conveyor belt. The main controller sends a control signal to the UAV microcontroller, and the UAV microcontroller controls the spectral camera to take the first multispectral image of the same object. Step 2.1.2: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take a second and third multispectral image of the same object; Step 2.1.3: After taking three multispectral images, the UAV microcontroller 4 sends a trigger signal to the main controller. After receiving the trigger signal, the main controller sends a control signal to the supplementary lighting microcontroller 12. The supplementary lighting microcontroller 12 outputs a control signal to the supplementary lighting rotation motor. The supplementary lighting rotation motor rotates the supplementary lighting turntable 10, causing the supplementary lighting lamp 11 to turn to another angle. After the supplementary lighting lamp turns, the supplementary lighting microcontroller 12 sends a turning completion signal to the main controller. After receiving the turning completion signal, the main controller sends a control signal to the UAV microcontroller 4. Step 2.1.4: After receiving the control signal from step 2.1.3, the UAV microcontroller 4 takes a fourth multispectral image; Step 2.1.5: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take the fifth and sixth multispectral images of the same object; Step 2.1.6: The UAV microcontroller 4 transmits the data of six multispectral images to the data fusion processing module of the main controller through the data transmission module.

[0011] Step 3 specifically includes: Step 3.1: Building a multi-dimensional data fusion model: Step 3.2, Data Quality Assessment and Weight Allocation: Step 3.2.1, Multispectral Image Data Quality Assessment: Evaluate the image's sharpness, contrast, and other indicators, and preprocess images that do not meet the quality requirements; Step 3.2.2, 3D point cloud data quality assessment: assess the density, accuracy and other indicators of the point cloud, and preprocess point clouds that do not meet the quality requirements.

[0012] Step 3.2.3, Weight Allocation: Based on the data quality assessment results, assign different weights to different data to improve the reliability of the fusion results; Step 3.3, Post-processing and Performance Evaluation: Step 3.3.1, Graph Neural Network Post-processing: Introduce a graph neural network to post-process the matching results, further improving the accuracy of feature matching; Step 3.3.2, Performance Evaluation: Establish an evaluation index system based on information entropy and overlap rate to evaluate the performance of the multi-dimensional data fusion model, and use cross-validation to optimize the model.

[0013] Step 3.1 specifically includes: Step 3.1.1: Feature Extraction of Multispectral Image Data: A convolutional neural network from deep learning is used to extract features from the multispectral image data. The specific steps are as follows: Step 3.1.1.1 Selecting a CNN model: Select ResNet-50 as the base model. This model contains 50 convolutional and fully connected layers and can automatically learn features in images. Step 3.1.1.2, Data preprocessing: Normalize the multispectral image data and scale the pixel values ​​to the range of [0,1] to improve the stability and convergence speed of model training; Step 3.1.1.3, Model Training: The ResNet-50 model is pre-trained using the ImageNet dataset, and then fine-tuned using a dataset for a specific environmental monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.1.4, Feature Extraction: Apply the trained ResNet-50 model to the multispectral image data to extract the texture and color features of the image and obtain the feature vector; Step 3.1.2: Feature Extraction of 3D Point Cloud Data: The PointNet++ network is used to extract features from the 3D point cloud data. The specific steps are as follows: Step 3.1.2.1 Select the PointNet++ model; Step 3.1.2.2, Data Preprocessing: Denoise the 3D point cloud data by using statistical filtering to remove outliers and improve the quality of the point cloud data; Step 3.1.2.3, Model Training: Pre-train the PointNet++ model using the ModelNet dataset, and then fine-tune the model using a dataset for a specific environment monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.2.4, Feature Extraction: Apply the trained PointNet++ model to the 3D point cloud data to extract the shape and surface normal vector features of the point cloud to obtain the feature vector; Step 3.1.3: Feature Matching and Data Fusion: Step 3.1.3.1, Feature matching: Use the SuperPoint algorithm to extract feature points and descriptors from multispectral images and 3D point cloud data, and then use the SuperGlue algorithm to perform feature matching to find matching feature point pairs; Step 3.1.3.2, Data Fusion: Using the matched feature points as a bridge, the color information of the multispectral image is mapped onto the 3D point cloud data, thereby realizing the fusion of multispectral image data and 3D point cloud data.

[0014] Following step 3.1.3.2, a linear regression model is used to correlate the UAV flight attitude data and environmental meteorological data with the fused multispectral image and 3D point cloud data to predict the key parameters of the fused data.

[0015] Compared with existing technologies, the present invention has the following beneficial effects: 1. Improved monitoring efficiency and safety: By installing multispectral cameras and lidar sensors on drones, real-time monitoring of the crushing area in the mine is achieved, improving inspection efficiency and reducing the safety risks of manual inspection. 2. Enhanced accuracy of image analysis: Deep learning algorithms are used to analyze multispectral images, combined with three-dimensional point cloud data from lidar, reducing the difficulty of image analysis and improving the accuracy of identifying the characteristics and boundaries of crushed materials. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention; Figure 2 This is a block diagram illustrating the principles of the multispectral image acquisition module and the lidar data acquisition module. Figure 3 A structural diagram of a mine inspection drone; Figure 4 The structure of a spectral camera mount; Figure 5 A structural diagram of a ore pile supplemental lighting device; Figure 6 This is a structural diagram of a ore pile supplementary lighting device according to another embodiment; Figure 7 This describes the feature extraction process for multispectral image data. Figure 8 This describes the feature extraction process for 3D point clouds. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0019] like Figure 1 As shown, the method for mine crushing UAV inspection based on multispectral image and lidar data fusion includes a mine crushing UAV inspection auxiliary device, a data fusion processing module, and a monitoring result output module.

[0020] like Figure 2 As shown, the mine crushing drone inspection auxiliary device includes a mine inspection drone and a ore pile supplementary lighting device. Both the mine inspection drone and the ore pile supplementary lighting device are connected to the main controller. The data fusion processing module and the monitoring result output module operate in the main controller, which is located in the mine control room.

[0021] Both the multispectral image acquisition module and the lidar data acquisition module are installed on the UAV. The multispectral image acquisition module can adjust the shooting angle during UAV flight to acquire multispectral image data of the same feature from multiple angles according to different inspection needs. The lidar data acquisition module acquires three-dimensional point cloud data of the mine's crushed area, extracting spatial distribution and particle size information of the crushed area.

[0022] The mine inspection drone includes a drone body 1, on which a spectral camera 2, a lidar 3, and a drone microcontroller 4 are mounted. The drone microcontroller 4 is communicatively connected to a main controller. The drone microcontroller 4 includes a first microcontroller and a data transmission module connected to it. The spectral camera 2 is communicatively connected to the data transmission module. The lidar 3 is fixedly mounted on the drone body 1, and the spectral camera 2 is mounted on the drone body 1 via angle adjustment and altitude adjustment mechanisms.

[0023] It should be noted that both the angle adjustment mechanism and the altitude adjustment mechanism adopt existing structures, and only one implementation method is given below. The angle adjustment mechanism is a turntable 5, which is installed at the bottom of the UAV body 1. The turntable 5 is connected to a rotating motor, and a spectral camera bracket is installed on the turntable 5. For example... Figure 4 As shown, the spectral camera support includes a fixed frame 6 fixed on the turntable 5. The fixed frame 6 is inverted U-shaped and has a through hole on its side wall. A rotating frame 7 is mounted inside the fixed frame 6 via a rotating shaft, which is installed at the through hole on the side wall of the fixed frame 6 via a bearing. A spectral camera 2 is fixedly mounted at one end of the rotating frame 7, and an arc-shaped tooth is provided at the other end of the rotating frame 7. The arc-shaped tooth meshes with a lead screw, which is driven by a height adjustment motor (not shown in the figure). The height adjustment motor can be mounted on the fixed frame 6. Both the rotating motor in the angle adjustment mechanism and the height adjustment motor in the height adjustment mechanism are connected to the UAV microcontroller 4.

[0024] The ore pile supplementary lighting device includes a support frame 8, on which a supplementary lighting turntable 10 is mounted, and the turntable 10 is connected to a supplementary lighting rotation motor. Supplementary lighting lamps 11 are mounted on the turntable 10, and the lamps 11 are connected to a supplementary lighting microcontroller 12, which is communicatively connected to a main controller. The microcontroller 12 includes a second microcontroller, which is connected to a brightness adjuster and the supplementary lighting rotation motor.

[0025] Furthermore, the bracket 8 is a support rod, and a folding tripod 9 is installed at the lower part of the support rod. This structure is a simple structure, such as... Figure 5 As shown, it is suitable for installation near the conveyor belt of a cone crusher or sand making machine. In this embodiment, the supplementary light 11 is located at the top of the support rod.

[0026] Furthermore, such as Figure 6As shown, the support frame 8 consists of two pillars, with a solar panel 13 positioned between them. The two pillars and the solar panel 13 together form an open frame structure. The supplementary lighting microcontroller 12 is located within this frame structure below the solar panel 13. This design is relatively stable and suitable for installation near ore piles. In this embodiment, the supplementary lighting lamp 11 faces outwards, and the supplementary lighting turntable 10 rotates the lamp 11 upwards, allowing for supplementary lighting of the ore pile from different heights.

[0027] Furthermore, the supplementary light 11 is an LED panel.

[0028] Furthermore, the brightness adjuster is connected to a rotary button, which is located on the housing of the supplementary lighting microcontroller 12.

[0029] Due to the complex geographical environment of the mine (unpredictable weather conditions and significant dust generated during blasting and crushing), the image information acquired by drones during inspections is often of poor clarity. Therefore, ore pile supplementary lighting devices are installed near the cone crusher, sand making machine, and ore pile. This allows the drone to capture images from multiple angles under the supplementary lighting of these devices when it begins its inspection, providing a hardware foundation for subsequent software analysis of the blasting and crushing effects.

[0030] The data fusion processing module establishes a multi-dimensional data fusion model, which integrates multispectral image data, 3D point cloud data, UAV flight attitude data, and environmental meteorological data to establish multi-dimensional data correlations, gain a more comprehensive understanding of the situation in the crushing area of ​​the mine, and further improve the accuracy of particle size judgment of crushed materials.

[0031] The monitoring results output module displays the real-time monitoring results of the mine crushing effect, including information such as the particle size distribution of the crushed material and equipment status. When the monitoring results are abnormal, the alarm device issues an alarm signal to prompt relevant personnel to handle the situation.

[0032] A method for UAV inspection of mine crushing based on multispectral image and lidar data fusion includes the following steps: Step 1: The drone takes off, and the multispectral camera and lidar sensor begin collecting data on the fractured area of ​​the mine. The drone takes off at an altitude of 100m-200m and a speed of 10m / s-20m / s. The multispectral camera and lidar sensor begin collecting data on the fractured area of ​​the mine, with the acquisition frequency set to 1Hz-5Hz.

[0033] Step 2: Obtain multi-source data; Step 2.1: The multispectral camera acquires multispectral image data of the crushed area of ​​the mine and transmits it to the main controller through the multispectral image acquisition module; the multispectral camera acquires multispectral image data of the crushed area of ​​the mine with an image resolution of 4000×3000, including four bands: red, green, blue, and near-infrared. Step 2.1.1: Adjust the initial position of the ore pile supplementary light device located near the ore pile, cone crusher or sand making machine discharge conveyor belt. The main controller sends a control signal to the UAV microcontroller 4. The UAV microcontroller 4 controls the spectral camera 2 to take the first multispectral image of the same object. Step 2.1.2: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take a second and third multispectral image of the same object; Step 2.1.3: After taking three multispectral images, the UAV microcontroller 4 sends a trigger signal to the main controller. After receiving the trigger signal, the main controller sends a control signal to the supplementary lighting microcontroller 12. The supplementary lighting microcontroller 12 outputs a control signal to the supplementary lighting rotation motor. The supplementary lighting rotation motor rotates the supplementary lighting turntable 10, causing the supplementary lighting lamp 11 to turn to another angle. After the supplementary lighting lamp turns, the supplementary lighting microcontroller 12 sends a turning completion signal to the main controller. After receiving the turning completion signal, the main controller sends a control signal to the UAV microcontroller 4. Step 2.1.4: After receiving the control signal from step 2.1.3, the UAV microcontroller 4 takes a fourth multispectral image; Step 2.1.5: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take the fifth and sixth multispectral images of the same object; Step 2.1.6: The UAV microcontroller 4 transmits the data of six multispectral images to the data fusion processing module of the main controller through the data transmission module.

[0034] In applications such as remote sensing, changes in the observation angle directly affect the radiance received by the sensor. This effect is particularly pronounced in the shortwave band where atmospheric scattering is strong, ultimately leading to errors in reflectance inversion. Therefore, to obtain accurate data, photometric correction is necessary, acquiring data under different observation angles and illumination levels.

[0035] Step 2.2: The lidar sensor collects three-dimensional point cloud data of the crushed area of ​​the mine and transmits it to the main controller through the lidar data acquisition module; the spectral range is set to 400nm~1000nm, and the point cloud density is set to 1 point / m³~5 points / m³. Step 2.3: Various sensors, including speed sensors, acceleration sensors, GPS sensors, etc., are installed on the drone to obtain the drone's flight attitude data and transmit the drone's flight attitude data to the main controller; Step 2.4: Install temperature and humidity sensors and wind speed sensors on the mine to obtain environmental meteorological data, and then transmit the environmental meteorological data to the main controller.

[0036] Step 3: Input the collected multispectral image data, 3D point cloud data, UAV flight attitude data, and environmental meteorological data into the multidimensional data fusion model to establish the correlation between the data, identify the characteristics and boundaries of the crushed material, and predict the crushing particle size based on the spectral characteristics. The model's recognition accuracy is set at 90%~95%, and the particle size prediction error is set at 5%~10%, thereby providing a more comprehensive understanding of the crushing area in the mine.

[0037] Step 3.1: Building a multi-dimensional data fusion model: Step 3.1.1: Feature Extraction of Multispectral Image Data: A Convolutional Neural Network (CNN) from deep learning is used to extract features from the multispectral image data. The specific steps are as follows: Step 3.1.1.1 Selecting a CNN model: Select ResNet-50 as the base model. This model contains 50 convolutional and fully connected layers and can automatically learn features in images. Step 3.1.1.2, Data preprocessing: Normalize the multispectral image data and scale the pixel values ​​to the range of [0,1] to improve the stability and convergence speed of model training; Step 3.1.1.3, Model Training: The ResNet-50 model is pre-trained using the ImageNet dataset, and then fine-tuned using a dataset for a specific environmental monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.1.4, Feature Extraction: Apply the trained ResNet-50 model to multispectral image data to extract features such as texture and color of the image, and obtain feature vectors.

[0038] Step 3.1.2: Feature Extraction of 3D Point Cloud Data: The PointNet++ network is used to extract features from the 3D point cloud data. The specific steps are as follows: Step 3.1.2.1 Select the PointNet++ model: PointNet++ is a point cloud feature extraction model based on graph convolutional networks, which can extract features such as shape and surface normal vectors of point cloud data; Step 3.1.2.2, Data Preprocessing: Denoise the 3D point cloud data by using statistical filtering to remove outliers and improve the quality of the point cloud data; Step 3.1.2.3, Model Training: Pre-train the PointNet++ model using the ModelNet dataset, and then fine-tune the model using a dataset for a specific environment monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.2.4, Feature Extraction: Apply the trained PointNet++ model to the 3D point cloud data to extract features such as the shape and surface normal vectors of the point cloud, and obtain feature vectors.

[0039] Step 3.1.3: Feature Matching and Data Fusion: Step 3.1.3.1, Feature matching: Use the SuperPoint algorithm to extract feature points and descriptors from multispectral images and 3D point cloud data, and then use the SuperGlue algorithm to perform feature matching to find matching feature point pairs.

[0040] Step 3.1.3.2, Data Fusion: Using the matched feature points as a bridge, the color information of the multispectral image is mapped onto the 3D point cloud data, thereby realizing the fusion of multispectral image data and 3D point cloud data.

[0041] Better yet, a linear regression model can be used to correlate UAV flight attitude data and environmental meteorological data with the fused multispectral images and 3D point cloud data to predict key parameters of the fused data, such as the position and size of the target object.

[0042] Step 3.2, Data Quality Assessment and Weight Allocation: Step 3.2.1, Multispectral Image Data Quality Assessment: Evaluate the image's sharpness, contrast, and other indicators. For images that do not meet the quality requirements, perform preprocessing, such as image enhancement. Step 3.2.2, 3D point cloud data quality assessment: assess the density, accuracy and other indicators of the point cloud, and preprocess point clouds that do not meet the quality requirements, such as point cloud filtering.

[0043] Step 3.2.3, Weight Allocation: Based on the data quality assessment results, assign different weights to different data to improve the reliability of the fusion results.

[0044] Step 3.3, Post-processing and Performance Evaluation: Step 3.3.1, Graph Neural Network Post-processing: Introduce a graph neural network (GNN) to post-process the matching results, further improving the accuracy of feature matching.

[0045] Step 3.3.2, Performance Evaluation: Establish an evaluation index system based on information entropy and overlap rate to evaluate the performance of the multi-dimensional data fusion model, and use cross-validation to optimize the model.

[0046] This allows for the effective integration of multispectral images, 3D point clouds, and UAV flight attitude and environmental meteorological data to construct a multi-dimensional data fusion model, thereby improving the accuracy and efficiency of environmental monitoring.

[0047] Step 4: Based on the data fusion results, determine the particle size distribution of the crushed material, adjust the crushing parameters of the crushing equipment in real time, and improve product quality and resource utilization.

[0048] Step 5: Display the monitoring results on the display device in real time. When the monitoring results are abnormal, the alarm device will issue an alarm signal to prompt relevant personnel to handle the situation.

[0049] Compared with existing technologies, the beneficial effects of this technical solution are as follows: 1. Improved monitoring efficiency and safety: By installing multispectral cameras and lidar sensors on drones, real-time monitoring of the crushing area in the mine is achieved, improving inspection efficiency and reducing the safety risks of manual inspection. 2. Enhanced accuracy of image analysis: Using deep learning algorithms to analyze multispectral images, combined with 3D point cloud data from lidar, reduces the difficulty of image analysis and improves the accuracy of identifying crushed material characteristics and boundaries. 3. Enhanced targeting and efficiency of data acquisition: By studying the reflection characteristics of crushed materials in different spectral bands, key spectral bands are selected, improving the targeting and efficiency of data acquisition. 4. Improved accuracy of crushed material particle size judgment: By fusing multi-dimensional data, including multispectral images, 3D point clouds, flight attitude, and environmental meteorological data, a multi-dimensional data fusion model is established, further improving the accuracy of crushed material particle size judgment. 5. Adaptability to different mining environments and working conditions: The data fusion model and data acquisition frequency are dynamically adjusted according to the drone's flight stage and different crushing areas, enabling the solution to better adapt to different mining environments and working conditions, improving the flexibility and accuracy of crushing effect monitoring. 6. Real-time adjustment of crushing parameters to improve product quality and resource utilization: By accurately judging the particle size of the crushed material, crushing parameters can be adjusted in real time to optimize the crushing process and improve product quality and resource utilization.

[0050] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several changes and improvements without departing from the overall concept of the present invention, and these should also be considered within the scope of protection of the present invention.

Claims

1. A method for unmanned aerial vehicle (UAV) inspection of mine crushing based on multispectral image and lidar data fusion, characterized in that: It includes a mine crushing drone inspection auxiliary device, a data fusion processing module, and a monitoring result output module; The aforementioned mine crushing drone inspection auxiliary device includes a mine inspection drone and a ore pile supplementary lighting device. Both the mine inspection drone and the ore pile supplementary lighting device are connected to the main controller. The data fusion processing module and the monitoring result output module run in the main controller, which is located in the mine control room. Both the multispectral image acquisition module and the lidar data acquisition module are installed on the UAV. The multispectral image acquisition module can adjust the shooting angle according to different inspection needs during the flight of the UAV to acquire multispectral image data of the same feature from multiple angles. The lidar data acquisition module acquires three-dimensional point cloud data of the crushed area of ​​the mine and extracts the spatial distribution and particle size information of the crushed area of ​​the mine. The data fusion processing module establishes a multi-dimensional data fusion model, which integrates multispectral image data, 3D point cloud data, UAV flight attitude data and environmental meteorological data to establish multi-dimensional data correlation relationships. The monitoring result output module displays the real-time monitoring results of the mine crushing effect. When the monitoring results are abnormal, the alarm device issues an alarm signal to prompt relevant personnel to handle the situation.

2. The method for mine crushing UAV inspection based on multispectral image and lidar data fusion as described in claim 1, characterized in that, Includes the following steps: Step 1: The drone takes off, and the multispectral camera and lidar sensor begin collecting data on the crushed area of ​​the mine; Step 2: Obtain multi-source data; Step 2.1: The multispectral camera acquires multispectral image data of the crushed area in the mine and transmits it to the main controller through the multispectral image acquisition module; the multispectral camera acquires multispectral image data of the crushed area in the mine, including four bands: red, green, blue, and near-infrared. Step 2.2: The lidar sensor collects three-dimensional point cloud data of the crushed area in the mine and transmits it to the main controller through the lidar data acquisition module; Step 2.3: Various sensors, including speed sensors, acceleration sensors and GPS sensors, are installed on the drone to obtain the drone's flight attitude data and transmit the drone's flight attitude data to the main controller; Step 2.4: Install temperature and humidity sensors and wind speed sensors on the mine to obtain environmental meteorological data and transmit the environmental meteorological data to the main controller; Step 3: Input the collected multispectral image data, 3D point cloud data, UAV flight attitude data and environmental meteorological data into the multidimensional data fusion model, establish the correlation between the data, identify the characteristics and boundaries of the crushed material, and predict the crushing particle size based on the spectral characteristics; Step 4: Based on the data fusion results, determine the particle size distribution of the crushed material and adjust the crushing parameters of the crushing equipment in real time; Step 5: Display the monitoring results on the display device in real time. When the monitoring results are abnormal, the alarm device will issue an alarm signal to prompt relevant personnel to handle the situation.

3. The method for mine crushing UAV inspection based on multispectral image and lidar data fusion as described in claim 2, characterized in that: Step 2.1 specifically includes: Step 2.1.1: Adjust the initial position of the ore pile supplementary light device located near the ore pile, cone crusher, or sand making machine discharge conveyor belt. The main controller sends a control signal to the UAV microcontroller, and the UAV microcontroller controls the spectral camera to take the first multispectral image of the same object. Step 2.1.2: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take a second and third multispectral image of the same object; Step 2.1.3: After taking three multispectral images, the UAV microcontroller 4 sends a trigger signal to the main controller. After receiving the trigger signal, the main controller sends a control signal to the supplementary lighting microcontroller 12. The supplementary lighting microcontroller 12 outputs a control signal to the supplementary lighting rotation motor. The supplementary lighting rotation motor rotates the supplementary lighting turntable 10, causing the supplementary lighting lamp 11 to turn to another angle. After the supplementary lighting lamp turns, the supplementary lighting microcontroller 12 sends a turning completion signal to the main controller. After receiving the turning completion signal, the main controller sends a control signal to the UAV microcontroller 4. Step 2.1.4: After receiving the control signal from step 2.1.3, the UAV microcontroller 4 takes a fourth multispectral image; Step 2.1.5: The UAV microcontroller 4 adjusts the angle adjustment mechanism and / or altitude adjustment mechanism, and the UAV microcontroller 4 controls the spectroscopic camera 2 to take the fifth and sixth multispectral images of the same object; Step 2.1.6: The UAV microcontroller 4 transmits the data of six multispectral images to the data fusion processing module of the main controller through the data transmission module.

4. The method for mine crushing UAV inspection based on multispectral image and lidar data fusion as described in claim 2, characterized in that: Step 3 specifically includes: Step 3.1: Building a multi-dimensional data fusion model: Step 3.2, Data Quality Assessment and Weight Allocation: Step 3.2.1, Multispectral Image Data Quality Assessment: Evaluate the image's sharpness, contrast, and other indicators, and preprocess images that do not meet the quality requirements; Step 3.2.2, 3D point cloud data quality assessment: assess the density, accuracy and other indicators of the point cloud, and preprocess point clouds that do not meet the quality requirements; Step 3.2.3, Weight Allocation: Based on the data quality assessment results, assign different weights to different data to improve the reliability of the fusion results; Step 3.3, Post-processing and Performance Evaluation: Step 3.3.1, Graph Neural Network Post-processing: Introduce a graph neural network to post-process the matching results, further improving the accuracy of feature matching; Step 3.3.2, Performance Evaluation: Establish an evaluation index system based on information entropy and overlap rate to evaluate the performance of the multi-dimensional data fusion model, and use cross-validation to optimize the model.

5. The method for mine crushing UAV inspection based on multispectral image and lidar data fusion according to claim 4, characterized in that: Step 3.1 specifically includes: Step 3.1.1: Feature Extraction of Multispectral Image Data: A convolutional neural network from deep learning is used to extract features from the multispectral image data. The specific steps are as follows: Step 3.1.1.1 Selecting a CNN model: Select ResNet-50 as the base model. This model contains 50 convolutional and fully connected layers and can automatically learn features in images. Step 3.1.1.2, Data preprocessing: Normalize the multispectral image data and scale the pixel values ​​to the range of [0,1] to improve the stability and convergence speed of model training; Step 3.1.1.3, Model Training: The ResNet-50 model is pre-trained using the ImageNet dataset, and then fine-tuned using a dataset for a specific environmental monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.1.4, Feature Extraction: Apply the trained ResNet-50 model to the multispectral image data to extract the texture and color features of the image and obtain the feature vector; Step 3.1.2: Feature Extraction of 3D Point Cloud Data: The PointNet++ network is used to extract features from the 3D point cloud data. The specific steps are as follows: Step 3.1.2.1 Select the PointNet++ model; Step 3.1.2.2, Data Preprocessing: Denoise the 3D point cloud data by using statistical filtering to remove outliers and improve the quality of the point cloud data; Step 3.1.2.3, Model Training: Pre-train the PointNet++ model using the ModelNet dataset, and then fine-tune the model using a dataset for a specific environment monitoring task to adapt to the feature extraction requirements of the specific task. Step 3.1.2.4, Feature Extraction: Apply the trained PointNet++ model to the 3D point cloud data to extract the shape and surface normal vector features of the point cloud to obtain the feature vector; Step 3.1.3: Feature Matching and Data Fusion: Step 3.1.3.1, Feature matching: Use the SuperPoint algorithm to extract feature points and descriptors from multispectral images and 3D point cloud data, and then use the SuperGlue algorithm to perform feature matching to find matching feature point pairs; Step 3.1.3.2, Data Fusion: Using the matched feature points as a bridge, the color information of the multispectral image is mapped onto the 3D point cloud data, thereby realizing the fusion of multispectral image data and 3D point cloud data.

6. The method for mine crushing UAV inspection based on multispectral image and lidar data fusion according to claim 5, characterized in that: Following step 3.1.3.2, a linear regression model is used to correlate the UAV flight attitude data and environmental meteorological data with the fused multispectral image and 3D point cloud data to predict the key parameters of the fused data.

Images