Three-dimensional panoramic intelligent monitoring method and device for coal conveying gallery

Abstract

The invention discloses a three-dimensional panoramic intelligent monitoring method and device for a coal conveying gallery, and relates to the technical field of industrial automation and intelligent monitoring, and the method comprises the steps: obtaining initial point cloud data and an initial multispectral image of the coal conveying gallery through a rotary laser radar, a high-dynamic-range camera and an infrared thermal imager; inertial measurement units are arranged in the devices to collect vibration data, and the vibration data are used for carrying out vibration compensation on point cloud and images to generate target point cloud data and target multispectral images; thirdly, performing three-dimensional modeling and multispectral texture mapping on the coal conveying gallery based on the data to obtain a target three-dimensional model; and finally, mapping equipment, personnel and environment data into the three-dimensional model in real time, displaying each piece of state information, and realizing comprehensive monitoring. According to the application, high-precision three-dimensional panoramic monitoring can be realized in a complex vibration environment of the coal conveying gallery, and the safety management level and the operation and maintenance efficiency of the coal conveying gallery are effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of industrial automation and intelligent monitoring, and particularly to a three-dimensional panoramic intelligent monitoring method and device for a coal conveying corridor. Background Art

[0002] In modern mining and power industries, the coal conveying corridor is the core link of coal transportation, and its safe and efficient operation is crucial for production. With the advancement of intelligent production, traditional monitoring methods can no longer meet the production requirements of high efficiency and low risk. Therefore, an intelligent solution that can monitor the operation status of the coal conveying corridor in real time and comprehensively is needed to improve the operation efficiency of equipment, reduce the failure rate, and ensure the safety of personnel and equipment.

[0003] Currently, the monitoring of the coal conveying corridor is mainly achieved through the following methods: (1) Traditional manual inspection: relying on manual regular inspection of the equipment operation status, but this method has low efficiency, high labor intensity, and it is difficult to discover potential problems in a timely manner. (2) Video monitoring system: real-time monitoring of the coal conveying corridor through cameras, but traditional video monitoring can only provide limited visual information.

[0004] Although the existing practices have improved the monitoring level of the coal conveying corridor to a certain extent, there are still the following problems: insufficient monitoring accuracy, difficult data fusion, insufficient real-time performance and response speed, and poor environmental adaptability (there is a lot of dust and vibration in the coal conveying corridor, which puts higher requirements on the stability and reliability of monitoring equipment). Therefore, how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor has become an urgent problem to be solved.

[0005] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The purpose of this application is to provide a three-dimensional panoramic intelligent monitoring method and device for a coal conveying corridor, aiming to solve the technical problem of how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor.

[0007] To achieve the above purpose, this application proposes a three-dimensional panoramic intelligent monitoring method for a coal conveying corridor, and the method includes:

[0008] Obtain the initial point cloud data collected by a rotary lidar and the initial multi-spectral images collected by a high-dynamic range camera and an infrared thermal imager. The rotary lidar is longitudinally deployed on the coal conveying corridor at equal intervals. An inertial measurement unit is embedded in the rotary lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data;

[0009] Performing vibration compensation on the initial point cloud data and the initial hyperspectral image according to the vibration data to obtain target point cloud data and a target hyperspectral image;

[0010] Performing three-dimensional modeling and hyperspectral texture mapping on the coal conveying corridor according to the target point cloud data and the target hyperspectral image to obtain a target three-dimensional model;

[0011] Mapping the obtained equipment data, personnel data, and environmental data into the target three-dimensional model to display the equipment status, personnel status, and environmental status.

[0012] In one embodiment, the step of performing three-dimensional modeling and hyperspectral texture mapping on the coal conveying corridor according to the target point cloud data and the target hyperspectral image to obtain a target three-dimensional model includes: estimating the normal direction of the target point cloud data to obtain point cloud data with normal vectors; constructing an initial triangular mesh model according to the point cloud data with normal vectors; performing mesh smoothing and hole filling on the initial triangular mesh model to obtain a reference three-dimensional model; performing hyperspectral texture mapping on the reference three-dimensional model according to the target hyperspectral image to obtain a target three-dimensional model.

[0013] In one embodiment, the step of constructing an initial triangular mesh model according to the point cloud data with normal vectors includes: performing multi-resolution spatial partitioning on the point cloud data with normal vectors to obtain a hierarchical node distribution; performing normal vector projection on the hierarchical node distribution to obtain discretized gradient field data; performing integral operation on the gradient field data to obtain an implicit surface indicator function; performing isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model.

[0014] In one embodiment, the step of performing isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model includes: defining the surface boundary and performing uniform voxel partitioning on the implicit surface indicator function to generate structured three-dimensional grid data; performing voxel vertex state matching, edge intersection interpolation calculation, and topological connection on the structured three-dimensional grid data to generate an initial triangular patch set; merging the duplicate vertices in the initial triangular patch set to obtain a target triangular patch set; encapsulating the target triangular patch set into a standard grid format to obtain an initial triangular mesh model.

[0015] In one embodiment, the step of performing multi-spectral texture mapping on the reference 3D model according to the target multi-spectral image to obtain the target 3D model includes: parameterizing and unfolding the reference 3D model to obtain a topological mapping relationship with 2D texture coordinates; obtaining a high-dimensional spectral feature texture atlas according to the target multi-spectral image and the topological mapping relationship; and performing non-rigid registration on the high-dimensional spectral feature texture atlas and the reference 3D model to obtain the target 3D model.

[0016] In one embodiment, the step of obtaining a high-dimensional spectral feature texture atlas according to the target multi-spectral image and the topological mapping relationship includes: calibrating corresponding points of the UV coordinates of the target multi-spectral image and the topological mapping relationship through feature point matching to obtain a pixel-vertex mapping matrix; performing illumination compensation on the multi-spectral pixels covered by the mapping matrix through a radiation correction model to obtain multi-spectral texture information with consistent brightness; and performing band stacking and weight optimization on the multi-spectral texture information to obtain a high-dimensional spectral feature texture atlas.

[0017] In one embodiment, the step of performing vibration compensation on the initial point cloud data and the initial multi-spectral image according to the vibration data to obtain the target point cloud data and the target multi-spectral image includes: obtaining a vibration displacement time-domain signal according to the vibration data; inputting the vibration displacement time-domain signal into a Kalman filter state space model to obtain a vibration displacement prediction value; performing spatial compensation on the initial point cloud data according to the vibration displacement prediction value to obtain the target point cloud data; and performing affine transformation on the initial multi-spectral image according to the vibration displacement prediction value to obtain the target multi-spectral image.

[0018] In one embodiment, the step of obtaining a vibration displacement time-domain signal according to the vibration data includes: performing low-pass filtering on the three-axis acceleration signals in the vibration data to obtain an acceleration time-domain sequence; performing fast Fourier transform on the acceleration time-domain sequence to obtain vibration frequency spectrum characteristics; extracting the main frequency component from the vibration frequency spectrum characteristics; and inputting the main frequency component into a vibration displacement prediction model to obtain a vibration displacement time-domain signal.

[0019] In one embodiment, the step of mapping the obtained device data, personnel data, and environmental data into the target 3D model to display the device state, personnel state, and environmental state includes: registering the reference points of the target 3D model and the coal conveying corridor through a spatial calibration matrix to obtain a virtual-real space correspondence relationship; and mapping the obtained device data, personnel data, and environmental data into the target 3D model according to the virtual-real space correspondence relationship to display the device state, personnel state, and environmental state.

[0020] In addition, to achieve the above object, the present application also proposes a three-dimensional panoramic intelligent monitoring device for a coal conveying corridor, and the device includes:

[0021] A data acquisition module, configured to acquire initial point cloud data collected by a rotary lidar and initial multispectral images collected by a high-dynamic range camera and an infrared thermal imager. The rotary lidar is longitudinally and equidistantly deployed on the coal conveying corridor. An inertial measurement unit is embedded in each of the rotary lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data;

[0022] A vibration compensation module, configured to perform vibration compensation on the initial point cloud data and the initial multispectral images according to the vibration data to obtain target point cloud data and target multispectral images;

[0023] A model establishment module, configured to perform three-dimensional modeling and multispectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multispectral images to obtain a target three-dimensional model;

[0024] A data mapping module, configured to map the acquired device data, personnel data, and environmental data into the target three-dimensional model to display the device state, personnel state, and environmental state.

[0025] In addition, to achieve the above object, the present application also proposes a three-dimensional panoramic intelligent monitoring device for a coal conveying corridor, and the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor as described above.

[0026] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor as described above.

[0027] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor as described above.

[0028] One or more technical solutions proposed by the present application have at least the following technical effects:

[0029] First, the intelligent monitoring system collects initial point cloud data through rotation lidars deployed longitudinally at equal intervals on the coal conveying corridor, and uses high-dynamic range cameras and infrared thermal imagers to collect initial multi-spectral images. At the same time, vibration data is collected with the help of the inertial measurement unit embedded in the device, providing a basis for subsequent data processing. Secondly, the system performs vibration compensation on the initial point cloud data and the initial multi-spectral images according to the collected vibration data, thereby eliminating the errors caused by vibration and obtaining more accurate target point cloud data and target multi-spectral images. Then, the system uses the target point cloud data to perform three-dimensional modeling on the coal conveying corridor, constructs an accurate geometric structure model, and maps the target multi-spectral images onto the surface of the three-dimensional model to complete multi-spectral texture mapping, generating a target three-dimensional model with rich details and realism. This process not only restores the physical structure of the coal conveying corridor, but also endows the model with a more intuitive visual effect through texture mapping, enhancing the visualization ability of monitoring. Finally, the system maps the collected device data, personnel data, and environmental data into the target three-dimensional model, accurately locates various types of data to the corresponding positions in the model, and realizes the visual display of the device state, personnel state, and environmental state. Through this series of steps, the intelligent monitoring system can achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor, effectively improving the safety management level and operation and maintenance efficiency of the coal conveying corridor. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a schematic flowchart provided for the first embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application;

[0033] Figure 2 It is a schematic diagram of the coal conveying corridor scene provided for the first embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application;

[0034] Figure 3 It is a schematic diagram of the three-dimensional model of the coal conveying corridor provided for the first embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application;

[0035] Figure 4 It is a schematic flowchart provided for the second embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application;

[0036] Figure 5 It is a schematic diagram of the module structure of the three-dimensional panoramic intelligent monitoring device for the coal conveying corridor in the embodiment of the present application;

[0037] Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor in the embodiment of the present application.

[0038] The realization of the purpose, functional characteristics and advantages of the present application will be further described in conjunction with the embodiments with reference to the accompanying drawings. Detailed implementation manners

[0039] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0040] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0041] In the coal transportation in the mining and power industries, the safe and efficient operation of the coal conveying corridor is crucial. However, traditional monitoring methods such as manual inspections and video monitoring can no longer meet the needs of intelligent production. Manual inspections are inefficient and it is difficult to detect problems in a timely manner, while traditional video monitoring can only provide limited visual information. Although these methods have improved the monitoring level to a certain extent, they still face problems such as insufficient monitoring accuracy, difficulty in data fusion, slow real-time response, and poor adaptability to harsh environments. Especially in the coal conveying corridor with a lot of dust and large vibrations, a more stable and reliable intelligent monitoring solution is needed to improve the equipment operation efficiency, reduce the failure rate, and ensure safety.

[0042] The main solution of the embodiment of the present application is: the intelligent monitoring system respectively collects the point cloud data and multi-spectral images of the coal conveying corridor through rotation laser radars, high-dynamic range cameras and infrared thermal imagers deployed at equal intervals longitudinally, and obtains vibration data through the built-in inertial measurement unit. Based on the vibration data, the initial data is compensated to generate accurate target point cloud data and multi-spectral images. Then, three-dimensional modeling is carried out using the target point cloud data and texture mapping is completed in combination with the multi-spectral images to create a detailed three-dimensional model of the coal conveying corridor. Finally, the device, personnel and environmental data are mapped into this three-dimensional model to realize the visual display of each state and enhance the monitoring effect.

[0043] It should be noted that the execution subject of the embodiment of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an intelligent monitoring system, etc. that can realize the above functions. Hereinafter, the intelligent monitoring system will be taken as an example to illustrate this embodiment and the following embodiments.

[0044] Based on this, the embodiments of the present application provide a three-dimensional panoramic intelligent monitoring method for a coal conveying corridor. Referring to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application.

[0045] In this embodiment, the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor includes steps S10 to S40:

[0046] Step S10, obtain the initial point cloud data collected by a rotating lidar and the initial multi-spectral images collected by a high-dynamic range camera and an infrared thermal imager. The rotating lidar is longitudinally and equidistantly deployed on the coal conveying corridor. An inertial measurement unit is embedded in each of the rotating lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data.

[0047] It should be noted that the initial point cloud data refers to the original three-dimensional coordinate data collected by the rotating lidar. These data include the three-dimensional coordinates and reflection intensities of each point obtained by laser ranging during the scanning process of the lidar. The initial multi-spectral images refer to the image data collected by the high-dynamic range camera and the infrared thermal imager. These images contain information in different spectral ranges. By combining visible light images and infrared images, multi-spectral images are formed. The vibration data refers to the data collected by an inertial measurement unit (IMU). An IMU usually includes a three-axis accelerometer and a three-axis gyroscope, and is used to measure the acceleration and angular velocity of the device in space.

[0048] It can be understood that first, the intelligent monitoring system collects the initial point cloud data through the rotating lidar longitudinally and equidistantly deployed on the coal conveying corridor. Then, the system uses the high-dynamic range camera and the infrared thermal imager to synchronously collect the initial multi-spectral images, including visible light images and thermal imaging images, to comprehensively monitor the device operation status and the environmental temperature distribution, enhancing the comprehensiveness and accuracy of the monitoring. Finally, the system collects the vibration data through the IMU embedded in the lidar, camera, and thermal imager, which is used to correct the sensor data in real time, compensate for the errors caused by device vibration, and ensure the high precision and reliability of the monitoring data.

[0049] Step S20, perform vibration compensation on the initial point cloud data and the initial multi-spectral images according to the vibration data to obtain target point cloud data and target multi-spectral images.

[0050] It should be noted that vibration compensation refers to the real-time monitoring of the vibration state of the device and the use of relevant algorithms to correct the collected data to eliminate or reduce the impact of vibration on the measurement results. The target point cloud data refers to the point cloud data after vibration compensation processing. These data are corrected in terms of spatial position and reflection intensity and can more accurately reflect the actual three-dimensional structure of the coal conveying corridor. The target multi-spectral image refers to the multi-spectral image after vibration compensation processing. After vibration compensation, the positions and boundaries of the objects in the image are clearer, and it can more accurately reflect the device state and environmental information. Point cloud vibration compensation formula:

[0051] P corrected = R(θ)·(P raw - t)

[0052] Where, P raw refers to the original point cloud coordinates (3D vector); R(θ) refers to the rotation matrix (3×3 matrix) calculated from the IMU vibration angle (θ) and is used to eliminate attitude offsets; t refers to the translation vector (3D vector) obtained by integrating the IMU linear acceleration to compensate for position vibration. Image vibration compensation formula:

[0053] I corrected (x, y) = I raw (x + Δx, y + Δy) Where, I raw refers to the original image pixel value; Δx, Δy refer to the image displacement amounts calculated from the IMU vibration data (by optical flow method or direct integration).

[0054] It can be understood that first, the intelligent monitoring system uses the vibration data collected by the IMU to perform vibration compensation on the initial point cloud data and the initial multi-spectral image through a filtering algorithm. Secondly, the system adjusts the three-dimensional coordinates of the point cloud data and the pixel positions of the multi-spectral image according to the frequency and amplitude characteristics of the vibration data to eliminate the errors caused by vibration.

[0055] As an example, the step of performing vibration compensation on the initial point cloud data and the initial multi-spectral image according to the vibration data to obtain the target point cloud data and the target multi-spectral image includes: obtaining the vibration displacement time-domain signal according to the vibration data; inputting the vibration displacement time-domain signal into the Kalman filter state space model to obtain the vibration displacement prediction value; performing spatial compensation on the initial point cloud data according to the vibration displacement prediction value to obtain the target point cloud data; performing affine transformation on the initial multi-spectral image according to the vibration displacement prediction value to obtain the target multi-spectral image.

[0056] The vibration displacement time-domain signal refers to the signal of the vibration displacement varying with time obtained after processing the vibration data collected by the IMU. The Kalman filter state space model is a filtering algorithm based on a linear dynamic system, which is used to estimate the state of the system from a series of measurement data with noise. It describes the dynamic behavior of the system through the state equation and the observation equation, and uses the Kalman gain to fuse the predicted value and the observed value to obtain the optimal state estimate.

[0057] The predicted value of the vibration displacement refers to the estimated value of the future vibration displacement obtained after processing the vibration displacement time-domain signal through the Kalman filter state space model. This value reflects the dynamic change trend of the vibration displacement and can be used for subsequent data compensation. Spatial compensation refers to correcting the three-dimensional coordinates of each point in the initial point cloud data according to the predicted value of the vibration displacement. Affine transformation refers to performing a geometric transformation on the initial multispectral image according to the predicted value of the vibration displacement to correct the image position offset and deformation caused by vibration. Affine transformation matrix:

[0058]

[0059] Among them, the translation components d x (t), d y (t), d z (t): The predicted value of the vibration displacement, compensating for the offset of the image caused by vibration. Image transformation operation: For each pixel coordinate u = [x, y, 1] of the image T Apply u′ = Mu.

[0060] First, the intelligent monitoring system processes the vibration data collected by the IMU to calculate the vibration displacement time-domain signal, that is, the signal of the vibration displacement varying with time. Then, the vibration displacement time-domain signal is input into the Kalman filter state space model, and the predicted value of the vibration displacement is obtained using its prediction function. Next, spatial compensation is performed on the initial point cloud data according to the predicted value of the vibration displacement. By adjusting the three-dimensional coordinates of the point cloud data, the position deviation caused by vibration is eliminated, so as to obtain more accurate target point cloud data. Finally, an affine transformation is performed on the initial multispectral image using the predicted value of the vibration displacement. Through operations such as translation, rotation, and scaling, the deformation and position offset of the image are corrected to generate a clear and accurate target multispectral image.

[0061] As an example, the step of obtaining the vibration displacement time-domain signal according to the vibration data includes: performing low-pass filtering on the three-axis acceleration signal in the vibration data to obtain the acceleration time-domain sequence; performing fast Fourier transform on the acceleration time-domain sequence to obtain the vibration frequency spectrum characteristics; extracting the main frequency component from the vibration frequency spectrum characteristics; inputting the main frequency component into the vibration displacement prediction model to obtain the vibration displacement time-domain signal.

[0062] The three-axis acceleration signal refers to the acceleration data collected by the IMU along three orthogonal directions, and these signals reflect the acceleration changes of the device in three directions in space. The acceleration time-domain sequence refers to the three-axis acceleration signal after low-pass filtering. It is a time series representing the change of acceleration over time. The purpose of low-pass filtering is to remove high-frequency noise and retain the main components in the vibration signal, making subsequent processing more accurate. The vibration spectrum feature refers to the frequency-domain information obtained by performing a fast Fourier transform (FFT) on the acceleration time-domain sequence. It reflects the energy distribution of the vibration signal at different frequencies and can reveal the main frequency components and periodic characteristics of the vibration. The main frequency component refers to the most prominent frequency component extracted from the vibration spectrum feature. The main frequency component usually corresponds to the most significant periodic change in the vibration signal, reflects the main characteristic frequency of the vibration, and is the key information for analyzing vibration characteristics and performing subsequent processing. The vibration displacement prediction model is a mathematical model established based on the characteristics of the vibration signal and is used to predict the vibration displacement from the main frequency component. By analyzing the relationship between the main frequency component and the displacement, this model outputs the time-domain signal of the vibration displacement, providing a basis for subsequent vibration compensation. Vibration displacement prediction model:

[0063]

[0064] where {f i , A i} is the main frequency component, φ i is the instantaneous phase calculated by the Hilbert transform, and ∈(t) is the white noise term (variance = 0.01A).

[0065] First, the intelligent monitoring system performs low-pass filtering on the three-axis acceleration signal collected by the IMU to remove high-frequency noise and retain the low-frequency vibration components in the signal, thereby obtaining the acceleration time-domain sequence. This is done to eliminate high-frequency interference and make the signal smoother. Second, the system performs a fast Fourier transform on the filtered acceleration time-domain sequence to convert it from the time domain to the frequency domain, obtaining the vibration spectrum feature. This step can clearly show the energy distribution of different frequency components in the signal and help identify the main vibration frequencies. Then, the main frequency component with the strongest energy is extracted from the vibration spectrum feature. The main frequency component reflects the most significant periodic characteristics in the vibration signal. By extracting the main frequency component, the signal processing process can be simplified and the calculation efficiency can be improved. Finally, the main frequency component is input into the pre-established vibration displacement prediction model to calculate the vibration displacement time-domain signal. By analyzing the relationship between the main frequency component and the displacement, this model outputs the time-domain signal of the vibration displacement, providing accurate displacement information for subsequent vibration compensation.

[0066] Step S30, perform three-dimensional modeling and multi-spectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multi-spectral image to obtain a target three-dimensional model.

[0067] It should be noted that in the monitoring of the coal conveying corridor, 3D modeling is based on target point cloud data. Through algorithms, the discrete points in the point cloud are fitted into continuous 3D geometric shapes, thereby constructing an accurate spatial model of the coal conveying corridor.

[0068] Multi-spectral texture mapping refers to the process of mapping the texture information in multi-spectral images onto the surface of a 3D model. Specifically, it aligns the multi-spectral image after vibration compensation processing with the 3D model, and attaches details such as colors and textures in the image to the model surface, so that the model not only has a geometric shape but also rich visual information.

[0069] The target 3D model refers to the final model obtained after 3D modeling and multi-spectral texture mapping. It is a complete digital model that contains the geometric structure and texture details of the coal conveying corridor, and can intuitively display information such as the internal structure, equipment layout, and temperature distribution of the coal conveying corridor, providing a visual basis for subsequent monitoring and analysis.

[0070] It can be understood that, first, the intelligent monitoring system uses target point cloud data for 3D modeling of the coal conveying corridor, constructs an accurate spatial model of the coal conveying corridor through the geometric information of the point cloud data, and truly restores its internal and external structures and equipment layout. Secondly, the system performs multi-spectral texture mapping on the target multi-spectral image and the 3D model, accurately maps the texture information in the image onto the model surface, enables the model to have rich visual details and a sense of reality, and generates a target 3D model. This model not only contains the accurate geometric shape of the coal conveying corridor but also integrates the texture information of the multi-spectral image, and can intuitively display the operating state and equipment details of the coal conveying corridor.

[0071] Please refer to Figure 2 , Figure 2 FIG. 18 is a schematic diagram of a coal conveying corridor scene provided for the first embodiment of the 3D panoramic intelligent monitoring method for a coal conveying corridor in the present application, which includes an orange conveyor belt support, and a black conveyor belt is laid above the support for transporting coal. There is obvious coal accumulation on the conveyor belt, showing the main function of the coal conveying corridor. The walls of the corridor are composed of gray rocks, and pipes are installed on the top for ventilation or other industrial purposes.

[0072] Please refer to Figure 3 , Figure 3This is a schematic diagram of the three-dimensional model of the coal conveying corridor provided in the first embodiment of the three-dimensional panoramic intelligent monitoring method of this application. It includes two conveyor belts (marked as Belt One and Belt Two). Each conveyor belt is supported by an orange bracket and connected by a blue structure. There is coal being transported on the conveyor belt, showing the main function of the corridor. Various sensors and devices are marked on the three-dimensional model, such as sensors for monitoring parameters such as belt speed, temperature, voltage, current, and tension. These data are displayed digitally beside the model, providing information for real-time monitoring. In addition, the model integrates vibration compensation and multi-spectral texture mapping technologies to ensure the accuracy and realism of the model. The entire three-dimensional model provides an intuitive and detailed visualization tool for the monitoring, maintenance, and management of the coal conveying corridor, helping to improve operation efficiency and safety.

[0073] Step S40: Map the obtained device data, personnel data, and environmental data to the target three-dimensional model to display the device status, personnel status, and environmental status.

[0074] It should be noted that device data refers to the operating parameters and status information of various devices in the coal conveying corridor, such as the operating time, power consumption, temperature, vibration frequency, fault alarm signals, etc. of the devices. These data are collected by sensors and transmitted to the monitoring system. Personnel data refers to the relevant information of the staff in the coal conveying corridor, including the location, activity trajectory, working status (such as whether in a dangerous area), identity information, etc. of the personnel. These data are usually obtained through personnel positioning systems and monitoring cameras. Environmental data refers to the environmental parameters in the coal conveying corridor, such as temperature, humidity, dust concentration, harmful gas concentration, etc. These data are collected by environmental sensors.

[0075] Device status refers to the operating health status of the device, including whether it is operating normally, whether there are potential faults, device efficiency, etc. Personnel status refers to the safety status and working status of the staff. For example, whether the personnel are in a safe area, whether an accident has occurred, whether emergency rescue is needed, etc. By mapping the personnel data to the three-dimensional model, the location and activity trajectory of the personnel can be tracked in real time. Environmental status refers to the safety and suitability of the environment in the coal conveying corridor. For example, whether there is excessive harmful gas, too high dust concentration, abnormal temperature, etc.

[0076] It can be understood that, first, the intelligent monitoring system associates the collected device data with the device positions in the target 3D model, and intuitively displays the operating status of the devices on the model through color coding or icon prompts, etc. For example, green indicates normal operation, and red indicates faults or anomalies. Secondly, the system maps the personnel data into the 3D model, and displays the distribution and dynamics of personnel in the coal conveying corridor in real time to ensure that personnel are in a safe area and potential dangers can be detected in a timely manner. Finally, the environmental data is corresponded to the spatial positions in the model, and the environmental status is displayed in the form of a heat map or numerical labels to help monitoring personnel quickly identify abnormal environmental areas, so as to realize the comprehensive visual monitoring of the equipment, personnel and environmental status in the coal conveying corridor.

[0077] As an example, the step of mapping the obtained device data, personnel data, and environmental data into the target 3D model to display the device status, personnel status, and environmental status includes: registering the reference points of the target 3D model and the coal conveying corridor through a spatial calibration matrix to obtain the corresponding relationship between the virtual and real spaces; mapping the obtained device data, personnel data, and environmental data into the target 3D model according to the corresponding relationship between the virtual and real spaces to display the device status, personnel status, and environmental status.

[0078] The spatial calibration matrix refers to a transformation matrix used to describe the geometric relationship between the target 3D model and the actual coal conveying corridor, which usually includes rotation and translation parameters and is used to align the virtual model with the real scene.

[0079] Reference point registration refers to selecting several corresponding reference points in the target 3D model and the actual coal conveying corridor, and calculating the rigid transformation relationship between the model and the reality by using the spatial calibration matrix, so as to achieve the precise alignment of the virtual model and the real scene.

[0080] The corresponding relationship between the virtual and real spaces refers to the spatial mapping relationship between the virtual 3D model and the actual coal conveying corridor obtained through reference point registration, and this relationship enables the points in the virtual model to correspond one by one with the corresponding points in the real scene.

[0081] First, select several corresponding reference points in the model and the real scene, and calculate the rigid transformation relationship between the two by using the calibration matrix, so as to achieve the precise alignment of the virtual model and the real scene. Secondly, according to this corresponding relationship between the virtual and real spaces, associate the data with the geometric positions in the model. For example, label the device status information on the position of the corresponding device in the model through coordinate transformation, display the personnel position information in the corresponding area in the model, and display the environmental parameters in the form of a heat map or numerical labels in the corresponding spatial position in the model, so that the device status, personnel status, and environmental status can be intuitively displayed on the 3D model, realizing the comprehensive visual monitoring of the operation of the coal conveying corridor.

[0082] This embodiment provides a three-dimensional panoramic intelligent monitoring method for a coal conveying corridor. First, the intelligent monitoring system collects initial point cloud data through rotation lidars deployed longitudinally at equal intervals on the coal conveying corridor, and uses a high-dynamic range camera and an infrared thermal imager to collect initial multi-spectral images. At the same time, vibration data is collected by means of an inertial measurement unit embedded in the device, providing a basis for subsequent data processing. Secondly, the system performs vibration compensation on the initial point cloud data and the initial multi-spectral images according to the collected vibration data, thereby eliminating the errors caused by vibration and obtaining more accurate target point cloud data and target multi-spectral images. Then, the system uses the target point cloud data to perform three-dimensional modeling on the coal conveying corridor, constructs an accurate geometric structure model, and maps the target multi-spectral images onto the surface of the three-dimensional model to complete multi-spectral texture mapping, generating a target three-dimensional model with rich details and a sense of reality. This process not only restores the physical structure of the coal conveying corridor, but also endows the model with a more intuitive visual effect through texture mapping, enhancing the visualization ability of monitoring. Finally, the system maps the collected device data, personnel data, and environmental data into the target three-dimensional model, accurately locates various types of data to the corresponding positions in the model, and realizes the visual display of the device state, personnel state, and environmental state. Through this series of steps, the intelligent monitoring system can achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor, effectively improving the safety management level and operation and maintenance efficiency of the coal conveying corridor.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be elaborated hereinafter. On this basis, please refer to Figure 4 , Figure 4 which is a schematic flowchart of the second embodiment of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor of the present application. The step S30 of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor includes steps S31 to S34:

[0084] Step S31, estimate the normal direction of the target point cloud data to obtain point cloud data with normal vectors.

[0085] It should be noted that the normal direction estimation refers to calculating the normal vector of each point in the point cloud data to describe the direction of the point cloud surface at each point. The normal direction estimation formula:

[0086]

[0087] where p i is the neighborhood point coordinate (3D vector) of a certain point in the point cloud; μ is the mean coordinate of the neighborhood points; C is the covariance matrix, and the eigenvector corresponding to the smallest eigenvalue of it is the normal direction.

[0088] It can be understood that, first, a central point is selected from the target point cloud data, and all points within its neighborhood are found within a predefined search radius. Then, a local plane is constructed using these neighborhood points, and the normal vector of this plane is calculated through principal component analysis (PCA) (by calculating the covariance matrix of the neighborhood points and performing eigenvalue decomposition on the matrix, and the eigenvector corresponding to the smallest eigenvalue is the normal vector of the point). Finally, the system associates the calculated normal vector with each point to generate point cloud data with normal vectors. This process can provide important geometric feature information for subsequent 3D modeling and surface reconstruction, enhancing the detail representation ability of the model.

[0089] Step S32: Construct an initial triangular mesh model based on the point cloud data with normal vectors.

[0090] It should be noted that the initial triangular mesh model refers to a 3D mesh model composed of triangles generated by processing the point cloud data with normal vectors. This model is the basic structure for constructing a 3D surface using the point cloud data.

[0091] It can be understood that, first, the system analyzes the point cloud data with normal vectors, and selects a suitable triangulation algorithm (such as Delaunay triangulation or Ball - Pivoting algorithm) according to the geometric distribution of the point cloud and the normal vector information. Then, the algorithm calculates the spatial relationships between the points in the point cloud, connects the points in the point cloud into triangles one by one, and gradually constructs the entire mesh structure. Finally, the generated initial triangular mesh model can approximately represent the surface geometry of the coal conveying corridor in a discretized form, providing a basis for subsequent 3D modeling and surface optimization, while ensuring the surface continuity and geometric accuracy of the model.

[0092] As an example, the step of constructing an initial triangular mesh model based on the point cloud data with normal vectors includes: performing multi - resolution spatial partitioning on the point cloud data with normal vectors to obtain a hierarchical node distribution; performing normal vector projection on the hierarchical node distribution to obtain discretized gradient field data; performing integral operations on the gradient field data to obtain an implicit surface indicator function; and performing isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model.

[0093] Multi - resolution spatial partitioning refers to using an octree structure to perform multi - level partitioning of the point cloud data with normal vectors in space. Each level represents a different resolution, gradually refining from coarse to fine to obtain node distributions at different levels. Hierarchical node distribution means that after multi - resolution spatial partitioning, the point cloud data is organized into nodes with a hierarchical structure. Each node contains point cloud data within a certain range, and the level of the node reflects the coarseness or fineness of the spatial partitioning.

[0094] Normal vector projection refers to projecting the normal vector information in the point cloud data onto each node of the hierarchical node distribution through trilinear interpolation, so that each node has a normal vector, providing a basis for the subsequent generation of gradient field data. Discretized gradient field data refers to the normal vector information of each node obtained through normal vector projection. These normal vectors form a discrete gradient field in space, describing the gradient direction of the point cloud surface at each node. Trilinear interpolation normal vector projection formula:

[0095]

[0096] Among them, n i,j,k refers to the normal vectors of the adjacent 8 nodes; w i (x) refers to the linear weight along the x-axis (such as ).

[0097] The implicit surface indicator function refers to the function obtained by integrating the discretized gradient field data through a Poisson equation solver. This function defines an implicit surface in space. The value of the surface is zero on the isosurface, and is interpolated according to the normal vector and distance of the surface at other positions. Isosurface extraction refers to using the marching cubes algorithm to process the implicit surface indicator function and extract the isosurface where the function value is zero, that is, the implicit surface, so as to obtain the initial triangular mesh model. The marching cubes algorithm determines the position of the isosurface within the cube by analyzing the sign changes of the function values at the cube vertices and generates the corresponding triangular mesh. Poisson equation solving implicit surface formula:

[0098]

[0099] Among them, Φ refers to the implicit function (scalar field), and its isosurface (such as Φ = 0.5) defines the surface boundary; V refers to the gradient field (obtained by normal vector interpolation); refers to the Laplace operator, is the divergence operator.

[0100] First, the octree structure is used to perform multi-resolution spatial partitioning on the point cloud data with normal vectors. The point cloud data is hierarchically organized into nodes with different resolutions, forming a hierarchical node distribution. The specific approach is to divide the entire point cloud data space into multiple cubes of different sizes. The point cloud data within each cube constitutes a node, and the size of the node is dynamically adjusted according to the resolution requirements. This is done to efficiently process large-scale point cloud data while retaining detailed information at different levels. Secondly, the trilinear interpolation method is used to project the normal vectors of each node in the hierarchical node distribution, transferring the normal vector information of the point cloud to the nodes and generating discretized gradient field data. The specific approach is to calculate the average normal vector of each node through trilinear interpolation based on the normal vector information of the point cloud within the node, thereby obtaining the discretized gradient field of the entire space. This is done to convert the local geometric information (normal vector) of the point cloud into the attributes of the nodes, providing direction information for subsequent surface reconstruction. Then, the discretized gradient field is used as the input, and a continuous implicit surface function is calculated using a Poisson equation solver. This function defines the geometric shape of the target object in space. Finally, the distribution of the implicit surface indicator function in space is analyzed, and a triangular mesh is extracted from the isosurface through the marching cubes algorithm, obtaining an initial triangular mesh model for subsequent visualization and further processing, while ensuring the geometric accuracy and detail performance of the model.

[0101] As an example, the step of performing isosurface extraction on the implicit surface indicator function to obtain the initial triangular mesh model includes: defining the surface boundary and uniformly dividing the voxels of the implicit surface indicator function to generate structured three-dimensional grid data; performing voxel vertex state matching, edge intersection interpolation calculation, and topological connection on the structured three-dimensional grid data to generate an initial set of triangular patches; merging the duplicate vertices in the initial set of triangular patches to obtain a target set of triangular patches; and encapsulating the target set of triangular patches into a standard grid format to obtain the initial triangular mesh model.

[0102] Surface boundary definition refers to clearly defining the boundary range of the isosurface in the implicit surface indicator function, determining the position and range of the isosurface in space, and providing a basis for subsequent mesh generation. Uniform voxel division refers to uniformly dividing the space defined by the implicit surface indicator function into small cubes (voxels), forming a structured three-dimensional grid. Uniform voxel division formula:

[0103]

[0104] where Δ refers to the voxel side length, which controls the mesh resolution.

[0105] Structured three-dimensional grid data refers to a three-dimensional grid obtained by uniform voxel division, where the vertex positions and function values of each voxel are known and used for subsequent isosurface extraction. Voxel vertex state matching means judging whether each voxel vertex is inside or outside the isosurface according to the value of the implicit surface indicator function. The vertex state is usually represented in binary and is used for subsequent isosurface extraction. Edge intersection interpolation calculation means calculating the intersection position of the isosurface and the edge on the edge of the voxel by linear interpolation. The specific method is to calculate the exact position of the intersection point through the interpolation formula according to the function values of the two endpoints of the edge. Topological connection means determining the topological structure of the isosurface within the voxel and generating triangular patches based on the states of the voxel vertices and the positions of the edge intersections. This process is usually implemented through a lookup table to ensure that the generated triangular patches can be correctly connected.

[0106] The initial triangular patch set refers to the set of all triangular patches generated by the marching cubes algorithm. These triangular patches are an approximate representation of the isosurface but may contain duplicate vertices. The target triangular patch set refers to the final triangular patch set obtained after merging the duplicate vertices in the initial triangular patch set. Merging duplicate vertices can reduce mesh redundancy and improve the efficiency and quality of the mesh. The standard mesh format means encapsulating the target triangular patch set into a common three-dimensional mesh format (such as PLY or OBJ) for subsequent visualization and further processing.

[0107] First, define the surface boundary of the implicit surface indicator function to clarify the range of the isosurface in space. Then, evenly divide the space into small cubes (voxels) according to the boundary range to generate structured three-dimensional grid data. The purpose of doing this is to discretize the continuous implicit surface for subsequent processing and ensure the accuracy and efficiency of isosurface extraction. Second, process each voxel in the structured three-dimensional grid data: judge whether the function value of the voxel vertex is greater than or less than the isosurface threshold to determine the vertex state; then calculate the intersection position of the isosurface and the edge on the edge of the voxel by linear interpolation; finally, determine the topological structure within the voxel and generate the initial triangular patch set according to the vertex state and the intersection position through the lookup table of the marching cubes algorithm. This can approximate the implicit surface as a series of triangular patches and provide a basis for subsequent optimization. Then, merge the duplicate vertices in the initial triangular patch set to eliminate redundancy, optimize the mesh structure, reduce the complexity of the mesh, and improve the efficiency and quality of the model to obtain the target triangular patch set. Finally, encapsulate the target triangular patch set into the standard mesh format so that it can be read and processed by common 3D software, thereby obtaining the initial triangular mesh model that can be used for subsequent visualization and analysis.

[0108] Step S33, perform mesh smoothing and hole filling on the initial triangular mesh model to obtain a reference three-dimensional model.

[0109] It should be noted that the reference 3D model refers to the 3D model after optimization processing (such as mesh smoothing and hole filling), which serves as the benchmark model for subsequent modeling, analysis, or modification.

[0110] It can be understood that, first, the initial triangular mesh model is subjected to mesh smoothing processing. The new positions of each vertex are calculated through the Laplace smoothing algorithm, and it is moved to the average position of its neighboring vertices, thereby reducing the noise and sharp features on the mesh surface and making the model surface smoother. Secondly, the holes in the mesh are filled. By detecting the hole boundaries, triangular patches are generated using the minimum angle method or Delaunay triangulation method or radial basis function interpolation method, and the hole regions are gradually filled, and the boundary information is updated until the holes are completely filled. Then, the filled mesh is optimized, and the vertex positions are adjusted by the least squares method to further improve the mesh quality. Finally, the optimized mesh is encapsulated into a standard mesh format to obtain the reference 3D model. This model has undergone smoothing and hole filling processing and has a higher-quality geometric representation, which can be used for subsequent analysis and applications. Laplace smoothing formula:

[0111]

[0112] where λ is the smoothing coefficient (0 - 1), which controls the iteration amplitude; N(i) is the set of adjacent vertices of vertex i.

[0113] Radial basis function interpolation formula:

[0114]

[0115] where φ(r) is the radial basis function (such as φ(r) = r 3 );c i is the known point on the hole boundary.

[0116] Step S34, perform multi-spectral texture mapping on the reference 3D model according to the target multi-spectral image to obtain the target 3D model.

[0117] It can be understood that, first, texture coordinates are calculated for each vertex of the reference 3D model. These coordinates define the mapping relationship between the model surface and the target multi-spectral image. Specifically, by assigning two-dimensional texture coordinates (u, v) to each vertex, the model surface is associated with the image pixels. Secondly, through the texture sampling algorithm (such as bilinear interpolation), pixel values are sampled from the multi-spectral image according to the texture coordinates and assigned to the corresponding positions on the model surface. This process ensures that the texture can fit smoothly onto the model surface. Finally, the sampled texture information is completely mapped onto the reference 3D model to generate the target 3D model, enabling the model surface to have rich details and a sense of reality, and enhancing its performance in visualization and analysis.

[0118] As an example, the step of performing multi - spectral texture mapping on the reference 3D model according to the target multi - spectral image to obtain the target 3D model includes: parametrically unfolding the reference 3D model to obtain a topological mapping relationship with two - dimensional texture coordinates; obtaining a high - dimensional spectral feature texture atlas according to the target multi - spectral image and the topological mapping relationship; and performing non - rigid registration on the high - dimensional spectral feature texture atlas and the reference 3D model to obtain the target 3D model.

[0119] Two - dimensional texture coordinates refer to two - dimensional coordinates defined on each vertex of the 3D model surface, used to specify the specific position of the vertex in the texture image. These coordinates map the pixels of the texture image to the points on the model surface, thus realizing the correct fitting of the texture. The topological mapping relationship refers to the mapping relationship obtained by parametrically unfolding the 3D model surface onto a two - dimensional plane. Specifically, it is to unfold the surface structure of the 3D model into a two - dimensional topological structure by calculating the two - dimensional texture coordinates of each vertex for subsequent texture mapping. The high - dimensional spectral feature texture atlas refers to extracting the spectral feature information from the target multi - spectral image and organizing it into a high - dimensional texture atlas. This atlas contains the feature information of the multi - spectral image in different bands and is used to enhance the details and expressiveness of texture mapping.

[0120] Non - rigid registration refers to a non - rigid alignment process of the high - dimensional spectral feature texture atlas and the reference 3D model during the texture mapping process. Different from rigid registration, non - rigid registration allows the texture atlas to deform during the mapping process to better adapt to the surface structure of the 3D model. The target 3D model refers to the final 3D model obtained after multi - spectral texture mapping and non - rigid registration processing. This model not only has an accurate geometric shape but also integrates the texture information of the multi - spectral image, being able to more realistically reflect the appearance and details of the target object. Non - rigid registration:

[0121]

[0122] Among them, A is an affine transformation matrix (3×3), representing global linear deformation (such as scaling, rotation); b is a translation vector (3×1), representing overall displacement; p i is the position of the control point (extracted from feature matching, such as SIFT matching point pairs); c i is the weight coefficient (3×1) associated with the control point, solved by minimizing the registration error:

[0123]

[0124] Among them, x j is the model vertex, y jis a texture feature point, λ is a regularization parameter to prevent overfitting; φ(r) is a kernel function (commonly r 2 logr), which is used to define the intensity of the deformation influence of the control point pair on the surrounding area. This function is smooth at r = 0 and gradually decays with the increase of distance, ensuring local elasticity and global continuity of the deformation.

[0125] First, the surface of the 3D model is cut into continuous and non-overlapping 2D patches through the UV unwrapping algorithm: Using the least squares conformal mapping (LSCM) or the edge-weight-based unwrapping method, seam lines are selected on the 3D mesh to divide the surface, and the 3D coordinates of each triangular vertex are mapped to the UV plane, ensuring that the boundaries of adjacent patches are aligned and the stretching distortion is minimized, generating a 2D coordinate grid that strictly corresponds to the model topology, providing a geometric correspondence for texture mapping. Second, the target multispectral image data is mapped to the texture atlas according to the UV coordinates: Pixel-level registration is performed on each band of the multispectral image (such as RGB, near-infrared), and the corresponding multispectral pixel values of each 3D mesh patch are filled into the corresponding positions of the UV atlas using bilinear interpolation, generating a multi-layer (such as n-channel) texture map, so that each UV pixel contains the complete spectral features of the corresponding 3D surface point. Finally, non-rigid registration based on energy optimization is adopted: A joint energy function containing geometric constraints (such as Laplacian coordinate preservation) and spectral consistency (such as texture gradient matching the model curvature) is constructed, and the 3D vertex positions are adjusted through gradient descent or Newton iteration, so that after the geometric deformation of the model surface, the multispectral texture gradients (such as edges, patches) in its UV mapping area are aligned with the deformed 3D curvature feature space, thus realizing high-precision spectral-geometric fusion 3D reconstruction while preserving the original connectivity, and eliminating texture-geometric misalignment caused by perspective or deformation. UV unwrapping formula:

[0126]

[0127] where J f is the Jacobian matrix of the triangular patch (the linear part of the parametric mapping); s f is the scaling factor, and R f is the rotation matrix.

[0128] As an example, the step of obtaining the high-dimensional spectral feature texture atlas according to the target multispectral image and the topological mapping relationship includes: Calibrating the corresponding points of the UV coordinates of the target multispectral image and the topological mapping relationship through feature point matching to obtain a pixel-vertex mapping matrix; Performing illumination compensation on the multispectral pixels covered by the mapping matrix through a radiation correction model to obtain multispectral texture information with consistent brightness; (Through a texture fusion algorithm) Performing band stacking and weight optimization on the multispectral texture information to obtain a high-dimensional spectral feature texture atlas.

[0129] Feature point matching refers to detecting local feature points such as SIFT and ORB (e.g., edges, corners) in the target multispectral image and on the surface of the UV-unwrapped model, and comparing their similarities with the 3D model geometric features (such as curvature extreme points) at the corresponding positions in the UV coordinates to establish the spatial correspondence between the two. Its function is to solve the perspective difference between the image and the model and ensure the geometric alignment of texture mapping. Corresponding point calibration refers to manually or automatically screening out high-confidence matching point pairs based on the feature point matching results and recording the correspondence between the pixel coordinates in the multispectral image and the vertex indices in the UV-unwrapped map to form a discrete mapping relationship dataset. The pixel-vertex mapping matrix refers to expanding the discrete corresponding point calibration results into a continuous mapping relationship through an interpolation algorithm (such as barycentric coordinate interpolation) to generate a sparse matrix. The matrix elements represent the weight associations between each pixel in the multispectral image and the vertices of the 3D model, which are used for the allocation of pixel values to vertices during subsequent texture mapping. SIFT descriptor:

[0130]

[0131] Among them, D a 、D b are the 128-dimensional SIFT feature vectors of the image and the UV map.

[0132] The radiation correction model refers to using the empirical linear method or physical models (such as the atmospheric radiation transfer model) to numerically correct the pixel brightness deviations in the multispectral image caused by differences in lighting conditions and sensor responses, eliminating interferences such as shadows or overexposure, and making the pixel radiation values at different positions / times comparable. Radiation correction model:

[0133]

[0134]

[0135] Among them, Lsensor refers to the radiation value received by the sensor; Lpath refers to the atmospheric path radiation (scattering light interference); τ refers to the atmospheric transmittance; θ refers to the solar zenith angle.

[0136] Band stacking refers to stacking the texture information of multiple spectral bands along the channel dimension to form a multi-channel texture atlas (e.g., RGB + near-infrared constitutes a 4-channel atlas). By leveraging the complementarity between bands, the information content of the texture is enhanced. For example, the near-infrared band can highlight vegetation features. Weight optimization refers to dynamically adjusting the contribution weights of each band in the texture atlas based on feature importance (such as band distinctiveness) or task requirements (such as classification, detection) using PCA or deep learning networks. This suppresses redundant bands and strengthens key features, improving the accuracy of subsequent 3D analysis.

[0137] Band stacking and weight optimization:

[0138]

[0139] where p k is the probability that the pixel gray value k appears in the i-th band, calculated through histogram statistics. For example, if a certain band has 1000 pixels and 50 pixels have a gray value of 50, then p 50 = 0.05; H i refers to the information entropy of the i-th band, measuring the information richness of this band; w i is the entropy-based weight, with higher entropy bands having greater weights and lower entropy bands (such as overexposed regions) having reduced weights.

[0140] First, local feature points such as edges and corners are extracted from the target multi-spectral image and the UV-unwrapped model surface using the SIFT or ORB algorithm. The similarity is calculated using feature descriptors to establish the spatial correspondence between image pixels and UV coordinate vertices. Manually select or eliminate mis-matched points based on RANSAC to generate discrete pixel-vertex mapping pairs, and then expand the discrete points into a sparse mapping matrix covering the entire grid through barycentric coordinate interpolation. Each element in the matrix represents the radiative contribution weight of the pixel to the vertex, thus solving the geometric deformation differences between the image perspective and the model after unwrapping. Second, for the multi-spectral pixels covered by the mapping matrix, the atmospheric radiative transfer model or empirical linear method is used to perform radiative calibration and shadow compensation on the pixel values of each band based on reference whiteboard data or light distribution estimation, eliminating the brightness fluctuations caused by light angles and uneven sensor responses, making the texture radiation values of the same material in different regions tend to be consistent and enhancing physical authenticity. Finally, the corrected multi-spectral textures are stacked along the band dimension into a multi-channel atlas. The information entropy and correlation of each band are calculated through PCA, and weights are dynamically assigned (such as increasing the weights of high-distinctiveness bands), and a weighted fusion algorithm (such as Laplacian pyramid fusion) is used to eliminate noise and redundancy between bands, generating a texture atlas with both high spectral resolution and spatial continuity, enabling each vertex on the 3D model surface to be associated with multi-dimensional spectral features to support fine-grained material classification or environmental parameter inversion.

[0141] In this embodiment, the normal direction of the target point cloud data is estimated first. By calculating the normal vector of each point, the direction of the point cloud surface at each point is described. This step can provide important geometric feature information for subsequent surface reconstruction and texture mapping, enhancing the detail representation ability of the model. Next, an initial triangular mesh model is constructed based on the point cloud data with normal vectors. The points in the point cloud are connected into a triangular mesh through an algorithm to form a discretized geometric representation, providing a basis for subsequent 3D modeling and surface optimization. Then, the initial triangular mesh model is smoothed and holes are filled. Through algorithms such as Laplacian smoothing, the noise and sharp features on the mesh surface are reduced, making the mesh surface smoother. At the same time, the holes in the mesh are filled to optimize the mesh structure, obtaining a reference 3D model. Finally, multi-spectral texture mapping is performed on the reference 3D model according to the target multi-spectral image. The texture information in the multi-spectral image is mapped onto the model surface, enabling the model to not only have an accurate geometric shape but also incorporate the texture information of the multi-spectral image, and being able to more realistically reflect the appearance and details of the target object, obtaining the target 3D model. This series of steps can achieve the reconstruction from point cloud data to a high-precision 3D model, providing intuitive and accurate visualization support for the monitoring and analysis of the coal conveying corridor.

[0142] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the 3D panoramic intelligent monitoring method for the coal conveying corridor of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0143] This application also provides a 3D panoramic intelligent monitoring device for a coal conveying corridor. Please refer to Figure 5 , the 3D panoramic intelligent monitoring device for the coal conveying corridor includes:

[0144] A data acquisition module 10, configured to acquire the initial point cloud data collected by a rotary lidar and the initial multi-spectral images collected by a high-dynamic range camera and an infrared thermal imager. The rotary lidar is longitudinally and equally spaced on the coal conveying corridor. An inertial measurement unit is embedded in the rotary lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data;

[0145] A vibration compensation module 20, configured to perform vibration compensation on the initial point cloud data and the initial multi-spectral images according to the vibration data to obtain target point cloud data and target multi-spectral images;

[0146] A model establishment module 30, configured to perform 3D modeling and multi-spectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multi-spectral images to obtain a target 3D model;

[0147] A data mapping module 40, configured to map the acquired device data, personnel data, and environmental data into the target three-dimensional model to display the device status, personnel status, and environmental status.

[0148] In one embodiment, the model building module 30 is further configured to estimate the normal direction of the target point cloud data to obtain point cloud data with normal vectors; construct an initial triangular mesh model according to the point cloud data with normal vectors; perform mesh smoothing and hole filling on the initial triangular mesh model to obtain a reference three-dimensional model; perform multispectral texture mapping on the reference three-dimensional model according to the target multispectral image to obtain a target three-dimensional model.

[0149] In one embodiment, the model building module 30 is further configured to perform multi-resolution spatial partitioning on the point cloud data with normal vectors to obtain a hierarchical node distribution; perform normal vector projection on the hierarchical node distribution to obtain discretized gradient field data; perform integral operation on the gradient field data to obtain an implicit surface indicator function; perform isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model.

[0150] In one embodiment, the model building module 30 is further configured to define the surface boundary and perform uniform voxel partitioning on the implicit surface indicator function to generate structured three-dimensional grid data; perform voxel vertex state matching, edge intersection interpolation calculation, and topological connection on the structured three-dimensional grid data to generate an initial set of triangular patches; merge the duplicate vertices in the initial set of triangular patches to obtain a target set of triangular patches; encapsulate the target set of triangular patches into a standard grid format to obtain an initial triangular mesh model.

[0151] In one embodiment, the model building module 30 is further configured to perform parametric unfolding on the reference three-dimensional model to obtain a topological mapping relationship with two-dimensional texture coordinates; obtain a high-dimensional spectral feature texture atlas according to the target multispectral image and the topological mapping relationship; perform non-rigid registration on the high-dimensional spectral feature texture atlas and the reference three-dimensional model to obtain a target three-dimensional model.

[0152] In one embodiment, the model building module 30 is further configured to perform corresponding point calibration on the UV coordinates of the target multispectral image and the topological mapping relationship through feature point matching to obtain a pixel-vertex mapping matrix; perform illumination compensation on the multispectral pixels covered by the mapping matrix through a radiation correction model to obtain multispectral texture information with consistent brightness; perform band stacking and weight optimization on the multispectral texture information to obtain a high-dimensional spectral feature texture atlas.

[0153] In one embodiment, the vibration compensation module 20 is further configured to obtain a vibration displacement time-domain signal based on the vibration data; input the vibration displacement time-domain signal into a Kalman filter state space model to obtain a vibration displacement prediction value; perform spatial compensation on the initial point cloud data according to the vibration displacement prediction value to obtain target point cloud data; and perform an affine transformation on the initial multi-spectral image according to the vibration displacement prediction value to obtain a target multi-spectral image.

[0154] In one embodiment, the vibration compensation module 20 is further configured to perform low-pass filtering on the triaxial acceleration signal in the vibration data to obtain an acceleration time-domain sequence; perform a fast Fourier transform on the acceleration time-domain sequence to obtain vibration frequency spectrum characteristics; extract a main frequency component from the vibration frequency spectrum characteristics; and input the main frequency component into a vibration displacement prediction model to obtain a vibration displacement time-domain signal.

[0155] In one embodiment, the data mapping module 40 is further configured to perform reference point registration on the target three-dimensional model and the coal conveying corridor through a spatial calibration matrix to obtain a virtual-real space correspondence relationship; and map the acquired equipment data, personnel data, and environmental data into the target three-dimensional model according to the virtual-real space correspondence relationship to display the equipment state, personnel state, and environmental state.

[0156] The coal conveying corridor three-dimensional panoramic intelligent monitoring device provided by the present application adopts the coal conveying corridor three-dimensional panoramic intelligent monitoring method in the above embodiment, and can solve the technical problem of how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor. Compared with the prior art, the beneficial effects of the coal conveying corridor three-dimensional panoramic intelligent monitoring device provided by the present application are the same as those of the coal conveying corridor three-dimensional panoramic intelligent monitoring method provided by the above embodiment, and other technical features in the coal conveying corridor three-dimensional panoramic intelligent monitoring device are the same as the features disclosed in the method of the above embodiment, and will not be elaborated herein.

[0157] The present application provides a coal conveying corridor three-dimensional panoramic intelligent monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the coal conveying corridor three-dimensional panoramic intelligent monitoring method in the first embodiment above.

[0158] Next, refer to Figure 6, which shows a schematic structural diagram of a three-dimensional panoramic intelligent monitoring device for a coal conveying corridor suitable for implementing the embodiments of the present application. The three-dimensional panoramic intelligent monitoring device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions, tablet computers), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The shown three-dimensional panoramic intelligent monitoring device for the coal conveying corridor is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0159] As Figure 6 shown, the three-dimensional panoramic intelligent monitoring device for the coal conveying corridor may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 into a RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the three-dimensional panoramic intelligent monitoring device for the coal conveying corridor are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the three-dimensional panoramic intelligent monitoring device for the coal conveying corridor to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a three-dimensional panoramic intelligent monitoring device with various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0160] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0161] The three-dimensional panoramic intelligent monitoring device for a coal conveying corridor provided by the present application adopts the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor in the above embodiment, and can solve the technical problem of how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of a coal conveying corridor. Compared with the prior art, the beneficial effects of the three-dimensional panoramic intelligent monitoring device for a coal conveying corridor provided by the present application are the same as those of the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor provided by the above embodiment, and other technical features in the three-dimensional panoramic intelligent monitoring device for a coal conveying corridor are the same as the features disclosed in the method of the previous embodiment, which will not be elaborated here.

[0162] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0163] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0164] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the three-dimensional panoramic intelligent monitoring method for a coal conveying corridor in the above embodiment.

[0165] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or flash memory), optical fibers, CD-ROM (Compact Disc - Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0166] The above computer-readable storage medium can be included in the three-dimensional panoramic intelligent monitoring device for coal conveying corridors; or it can exist separately and not be assembled into the three-dimensional panoramic intelligent monitoring device for coal conveying corridors.

[0167] The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the three-dimensional panoramic intelligent monitoring device for coal conveying corridors, the three-dimensional panoramic intelligent monitoring device for coal conveying corridors is enabled to: obtain the initial point cloud data collected by the rotating lidar and the initial multi-spectral images collected by the high-dynamic range camera and the infrared thermal imager. The rotating lidar is longitudinally deployed on the coal conveying corridor at equal intervals, and inertial measurement units are embedded in the rotating lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data; perform vibration compensation on the initial point cloud data and the initial multi-spectral images according to the vibration data to obtain target point cloud data and target multi-spectral images; perform three-dimensional modeling and multi-spectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multi-spectral images to obtain a target three-dimensional model; map the obtained device data, personnel data, and environmental data into the target three-dimensional model to display the device status, personnel status, and environmental status.

[0168] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or it can be connected to an external computer (for example, by connecting through an Internet service provider via the Internet).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0170] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0171] The readable storage medium provided by this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for performing the above-mentioned three-dimensional panoramic intelligent monitoring method for the coal conveying corridor, and can solve the technical problem of how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor provided in the above embodiments, and will not be elaborated here.

[0172] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor as described above.

[0173] The computer program product provided by the present application can solve the technical problem of how to achieve high-precision three-dimensional panoramic monitoring in the complex vibration environment of the coal conveying corridor. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the three-dimensional panoramic intelligent monitoring method for the coal conveying corridor provided by the above embodiments, and will not be elaborated here.

[0174] The above are only some embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A three-dimensional panoramic intelligent monitoring method for a coal conveying corridor, characterized in that, The method includes: Obtaining initial point cloud data collected by a rotary lidar, and initial multispectral images collected by a high-dynamic-range camera and an infrared thermal imager. The rotary lidar is longitudinally and equidistantly deployed on a coal conveying corridor. Inertial measurement units are embedded in the rotary lidar, the high-dynamic-range camera, and the infrared thermal imager to collect vibration data; Performing vibration compensation on the initial point cloud data and the initial multispectral images according to the vibration data to obtain target point cloud data and target multispectral images; Performing three-dimensional modeling and multispectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multispectral images to obtain a target three-dimensional model; Mapping the obtained equipment data, personnel data, and environmental data into the target three-dimensional model to display the equipment state, personnel state, and environmental state.

2. The method according to claim 1, wherein The step of performing three-dimensional modeling and multispectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multispectral images to obtain a target three-dimensional model includes: Estimating the normal direction of the target point cloud data to obtain point cloud data with normal vectors; Constructing an initial triangular mesh model according to the point cloud data with normal vectors; Performing mesh smoothing and hole filling on the initial triangular mesh model to obtain a reference three-dimensional model; Performing multispectral texture mapping on the reference three-dimensional model according to the target multispectral images to obtain a target three-dimensional model.

3. The method according to claim 2, wherein The step of constructing an initial triangular mesh model according to the point cloud data with normal vectors includes: Performing multi-resolution spatial partitioning on the point cloud data with normal vectors to obtain a hierarchical node distribution; Performing normal vector projection on the hierarchical node distribution to obtain discretized gradient field data; Performing integral operation on the gradient field data to obtain an implicit surface indicator function; Performing isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model.

4. The method according to claim 3, wherein The step of performing isosurface extraction on the implicit surface indicator function to obtain an initial triangular mesh model includes: Defining the surface boundary and performing uniform voxel partitioning on the implicit surface indicator function to generate structured three-dimensional grid data; Performing voxel vertex state matching, edge intersection interpolation calculation, and topological connection on the structured three-dimensional grid data to generate an initial triangular patch set; Merging duplicate vertices in the initial triangular patch set to obtain a target triangular patch set; Encapsulating the target triangular patch set into a standard grid format to obtain an initial triangular mesh model.

5. The method according to claim 2, wherein The step of performing multispectral texture mapping on the reference three-dimensional model according to the target multispectral images to obtain a target three-dimensional model includes: Performing parametric unfolding on the reference three-dimensional model to obtain a topological mapping relationship with two-dimensional texture coordinates; Obtaining a high-dimensional spectral feature texture atlas according to the target multispectral images and the topological mapping relationship; Performing non-rigid registration on the high-dimensional spectral feature texture atlas and the reference three-dimensional model to obtain a target three-dimensional model.

6. The method according to claim 5, wherein The step of obtaining a high-dimensional spectral feature texture atlas according to the target multispectral images and the topological mapping relationship includes: Perform corresponding point calibration on the UV coordinates of the target multispectral image and the topological mapping relationship through feature point matching to obtain a pixel-vertex mapping matrix; Perform illumination compensation on the multispectral pixels covered by the mapping matrix through a radiation correction model to obtain multispectral texture information with consistent brightness; Perform band stacking and weight optimization on the multispectral texture information to obtain a high-dimensional spectral feature texture atlas.

7. The method according to claim 1, characterized in that, The step of performing vibration compensation on the initial point cloud data and the initial multispectral image according to the vibration data to obtain target point cloud data and a target multispectral image includes: Obtain a vibration displacement time-domain signal according to the vibration data; Input the vibration displacement time-domain signal into a Kalman filter state space model to obtain a vibration displacement prediction value; Perform spatial compensation on the initial point cloud data according to the vibration displacement prediction value to obtain target point cloud data; Perform an affine transformation on the initial multispectral image according to the vibration displacement prediction value to obtain a target multispectral image.

8. The method according to claim 7, characterized in that The step of obtaining a vibration displacement time-domain signal according to the vibration data includes: Perform low-pass filtering on the three-axis acceleration signals in the vibration data to obtain an acceleration time-domain sequence; Perform a fast Fourier transform on the acceleration time-domain sequence to obtain vibration spectrum characteristics; Extract the main frequency component from the vibration spectrum characteristics; Input the main frequency component into a vibration displacement prediction model to obtain a vibration displacement time-domain signal.

9. The method according to any one of claims 1 to 8, characterized in that, The step of mapping the obtained device data, personnel data, and environmental data into the target three-dimensional model to display the device state, personnel state, and environmental state includes: Perform reference point registration on the target three-dimensional model and the coal conveying corridor through a spatial calibration matrix to obtain a virtual-real space correspondence; Map the obtained device data, personnel data, and environmental data into the target three-dimensional model according to the virtual-real space correspondence to display the device state, personnel state, and environmental state.

10. A three-dimensional panoramic intelligent monitoring device for a coal conveying corridor, characterized in that, The device includes: A data acquisition module for acquiring initial point cloud data collected by a rotating lidar and initial multispectral images collected by a high-dynamic range camera and an infrared thermal imager. The rotating lidar is longitudinally and equidistantly deployed on the coal conveying corridor, and an inertial measurement unit is embedded in each of the rotating lidar, the high-dynamic range camera, and the infrared thermal imager to collect vibration data; A vibration compensation module for performing vibration compensation on the initial point cloud data and the initial multispectral image according to the vibration data to obtain target point cloud data and a target multispectral image; A model establishment module for performing three-dimensional modeling and multispectral texture mapping on the coal conveying corridor according to the target point cloud data and the target multispectral image to obtain a target three-dimensional model; A data mapping module for mapping the obtained device data, personnel data, and environmental data into the target three-dimensional model to display the device state, personnel state, and environmental state.

Images