A coal mine inspection machine AI fault analysis method for a low-light dust environment

Abstract

This invention relates to the field of intelligent robot inspection technology, and discloses an AI fault analysis method for coal mine inspection robots in dusty, low-light environments. The method involves acquiring a three-dimensional spatial dust transmittance grid sequence and extracting the transmittance time series. It predicts the time interval where the transmittance is at a local maximum, triggers a visible light camera and an infrared thermal imager to perform synchronous global exposure, and records the exposure midpoint timestamp. A virtual instantaneous transmittance matrix is ​​reconstructed by combining the timestamp and the grid sequence, and the infrared temperature measurement matrix is ​​inversely compensated to obtain a thermal feature matrix. If the instantaneous transmittance is lower than the optical signal-to-noise ratio failure threshold, the absolute center coordinates of the target equipment are extracted to reconstruct the phase delay compensation amount, and the acoustic envelope feature vector is extracted. A dynamic confidence weighting factor is calculated based on the transmittance, and the visual, infrared, and acoustic features are fused to output the mechanical fault probability result. This invention overcomes the interference of high-concentration dust obstruction, ensuring the accuracy and continuity of diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot inspection technology, specifically to an AI fault analysis method for coal mine inspection machines in dusty and low-light environments. Background Technology

[0002] Coal mine underground working faces generally present harsh environmental conditions such as high concentrations of suspended dust and low light levels. With the advancement of intelligent mining, the use of inspection robots equipped with multi-source sensors for condition monitoring and fault analysis of electromechanical equipment has become an industry trend. However, existing inspection robot perception and diagnostic systems still have significant limitations in dealing with such complex working conditions.

[0003] Because dust particles exhibit violent, disordered turbulent motion under the ventilation airflow in mines, they cause severe dynamic obstruction to visual sensors. Traditional inspection systems typically use fixed frequencies or application-layer software commands to trigger image acquisition, which lacks the ability to perceive the transient evolution trend of dust concentration. Application-layer task scheduling introduces random system response delays, causing optical sensors to be unable to complete capture within the short window when dust is relatively sparse. The resulting visible light images generally have a low signal-to-noise ratio, making it difficult to meet the requirements for identifying surface defects in target equipment.

[0004] In terms of equipment thermal condition monitoring, existing infrared temperature measurement methods mostly rely directly on the apparent output of thermal imagers or simply introduce a single static ambient temperature for basic correction. In actual operating environments, high-temperature dust in the detection optical path not only attenuates the effective thermal radiation of the target surface, but its aerosols also release background thermal radiation that superimposes onto the sensor receiver. Due to the lack of a three-dimensional inverse compensation mechanism that incorporates real-time spatial dust distribution, the system struggles to reconstruct the true absolute temperature field of the equipment. Furthermore, lidar used for environmental spatial perception employs a low-frequency continuous mechanical scanning system, while thermal imagers provide instantaneous global exposure. This difference in underlying hardware sampling systems results in inherent asynchrony between the two data over time. Directly using spatial point clouds from different moments to calibrate and compensate infrared images will lead to spatial mapping misalignment in dynamic scenarios with intense dust diffusion.

[0005] When extremely high concentrations of dust emerge from a localized area, the photoelectric sensing channel becomes severely congested. Existing inspection systems, when faced with such visual failures, lack cross-modal spatial guidance mechanisms, making it difficult to schedule sensors unaffected by dust, such as acoustic arrays, to take over and accurately focus. Furthermore, conventional multimodal fusion algorithms often employ fixed feature stitching or fixed-weight decision logic, failing to adaptively assess the current data quality of each modality based on fluctuations in environmental transmittance. This static fusion mechanism allows contaminated, low-signal-to-noise-ratio features to directly participate in the final classification, thereby reducing the overall reliability of mechanical fault diagnosis results for electromechanical equipment. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an AI-based fault analysis method for coal mine inspection robots in dusty and low-light environments. This method solves the problems of poor sensor data quality caused by asynchronous sensor triggering and dust obstruction in high-dynamic dust environments, temperature field distortion caused by the lack of three-dimensional spatiotemporal alignment and reverse compensation in infrared thermometry, and low reliability of equipment fault diagnosis due to the lack of cross-modal spatial guidance and adaptive dynamic weight fusion when optical sensing fails under extreme conditions.

[0007] To achieve the above objectives, this invention provides an AI-based fault analysis method for coal mine inspection machines in dusty, low-light environments, comprising the following steps: Obtain a three-dimensional dust transmittance mesh sequence; The time interval is determined based on the three-dimensional spatial dust transmittance grid sequence, and visible light image features and infrared temperature measurement matrix are obtained within the time interval; A virtual instantaneous transmittance matrix is ​​reconstructed based on the three-dimensional dust transmittance grid sequence, and the infrared temperature measurement matrix is ​​compensated using the virtual instantaneous transmittance matrix to obtain a thermodynamic feature matrix. If the average instantaneous transmittance of the virtual instantaneous transmittance matrix is ​​lower than the optical signal-to-noise ratio failure threshold calibrated offline based on the dark area shot noise limit of the visible light camera hardware with global shutter speed, then the acoustic envelope feature vector of the target device is obtained. The dynamic confidence weighting factor is determined based on the average instantaneous transmittance. The visible light image features, the thermal feature matrix, and the acoustic envelope feature vector are fused and calculated based on the dynamic confidence weighting factor to output the mechanical failure probability result.

[0008] Preferably, obtaining a three-dimensional dust transmittance grid sequence specifically includes: The system receives raw point cloud data and classifies the scanning points in the raw point cloud data whose echo intensity is greater than or equal to the rigid body reflection intensity threshold determined by benchmark measurement in a dust-free environment as solid phase point cloud, and classifies the scanning points whose echo intensity is less than the rigid body reflection intensity threshold and whose echo order is marked as the first echo as gas phase suspended point cloud. A mapping model is established based on the echo intensity data of the gas-phase suspended point cloud processed by the aerosol lidar equation, and the spatial extinction coefficient is extracted by inversion. The atmospheric transmittance along the detection optical path at the coordinate point is calculated using the spatial extinction coefficient, thereby generating the three-dimensional spatial dust transmittance grid sequence.

[0009] Preferably, the time interval is determined based on the three-dimensional spatial dust transmittance grid sequence, specifically including: Extract the transmittance time series from the three-dimensional spatial dust transmittance grid sequence; The transmittance time series is filtered and smoothed using a Kalman filter to obtain the mean transmittance series and the first and second time derivatives corresponding to the mean transmittance series. When the first time derivative is greater than zero and the second time derivative is less than zero, the predicted time point when the first time derivative decays to zero is calculated by extrapolation based on the current derivative value. The time interval is defined by extending forward and backward from the predicted time point. When the absolute value of the predicted transmittance corresponding to the time interval is greater than the minimum optical signal-to-noise ratio threshold calibrated based on the dark current noise parameter and quantum efficiency characteristics of the camera sensor, the corresponding time interval is confirmed as the time interval for performing subsequent acquisition operations.

[0010] Preferably, the process of acquiring visible light image features and infrared thermometry matrix within the time interval further includes: When the internal system clock reaches the start time of the time interval, it drives the general-purpose input / output pins to generate a level flip, sending hardwired trigger pulses in parallel to the external trigger pins of the visible light camera and the infrared thermal imager; The hardwired trigger pulse is used to force all pixel units of the visible light camera and the infrared thermal imager to start and end exposure operations simultaneously, thereby obtaining the visible light image features and the infrared temperature measurement matrix. At the moment the exposure cycle ends, a feedback signal is captured via a hardware interrupt. The result is combined with the calibrated exposure duration to calculate and record the global exposure midpoint timestamp.

[0011] Preferably, a virtual instantaneous transmittance matrix is ​​reconstructed based on the three-dimensional spatial dust transmittance grid sequence, and the infrared thermometry matrix is ​​compensated using the virtual instantaneous transmittance matrix to obtain a thermal characteristic matrix, specifically including: Based on the global exposure midpoint timestamp, obtain the three-dimensional spatial dust transmittance grid sequence of two frames that are immediately before and after the global exposure midpoint timestamp in time; Perform temporal linear interpolation on the two frames of the three-dimensional spatial dust transmittance grid sequence to generate the virtual instantaneous transmittance matrix aligned with the exposure instant; By combining the pre-calibrated rigid transformation matrix and optical intrinsic parameter matrix, the cumulative attenuation value corresponding to the ray penetrating the grid voxel in the virtual instantaneous transmittance matrix is ​​extracted using the line-of-sight ray tracing algorithm, and the two-dimensional view projection transmittance matrix is ​​obtained. By combining the obtained ambient background radiation temperature, and based on the modified Stefan-Boltzmann law, the infrared temperature measurement matrix is ​​subjected to inverse radiation attenuation compensation pixel by pixel using the two-dimensional perspective projection transmittance matrix to obtain the thermodynamic feature matrix that restores the actual surface temperature of the device.

[0012] Preferably, obtaining the acoustic envelope feature vector of the target device specifically includes: Obtain a solid point cloud of the target environment, perform a density-based spatial clustering algorithm on the solid point cloud, and extract the geometric absolute center coordinates of the point cloud clusters; The rigid body spatial pose transformation is performed by calling the pre-calibrated rigid translation and rotation transformation matrix, and the geometric absolute center coordinates are mapped to the local acoustic coordinate system to obtain the spatial absolute center coordinates. The acoustic envelope feature vector is extracted based on the absolute center coordinates of the space.

[0013] Preferably, the acoustic envelope feature vector is extracted based on the absolute center coordinates of the space, specifically including: Extract the known hardware coordinates of each microphone channel and calculate the actual time-varying speed of sound based on the ambient background radiation temperature positive compensation. Using the absolute center coordinates of the space, the hardware coordinates, and the time-varying sound speed of the actual environment, the delay time is calculated based on the sound wave transmission path length to obtain the phase delay compensation amount for each channel. The original acoustic array signal is shifted and aligned along the time axis according to the phase delay compensation amount, and a weighted summation coherence operation is performed to synthesize a one-dimensional time master signal. A fast Fourier transform is performed on the one-dimensional time master signal to extract the frequency domain amplitude envelope information, thereby generating the acoustic envelope feature vector.

[0014] Preferably, determining the dynamic confidence weighting factor based on the average instantaneous transmittance specifically includes: Based on the value of the average instantaneous transmittance, the visual basic confidence score corresponding to the visible light image feature and the infrared basic confidence score corresponding to the infrared thermometry matrix are calculated respectively. Assign an acoustic fundamental confidence score to the data extraction modality that is constant for the acoustic envelope feature vector; The visual baseline confidence score, the infrared baseline confidence score, and the acoustic baseline confidence score are redistributed using a normalized exponential function that includes a temperature parameter to obtain the normalized dynamic confidence weight factor.

[0015] Preferably, before performing the fusion calculation of the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector based on the dynamic confidence weight factor, the following alignment step is further included: Multiple feature extraction neural network branches are invoked in parallel to map the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector into visual feature vectors, infrared feature vectors, and acoustic feature vectors of a unified dimension, respectively. When the acoustic envelope feature vector is not extracted, the acoustic feature vector is assigned a zero vector that matches the dimension of the feature latent space to maintain tensor alignment of the fused input structure.

[0016] Preferably, the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector are fused and calculated based on the dynamic confidence weighting factor to output the mechanical failure probability result, specifically including: Using the dynamic confidence weighting factor, an element-wise algebraic weighted summation operation is performed on the visual feature vector, the infrared feature vector, and the acoustic feature vector to obtain a multimodal fusion feature vector; The multimodal fusion feature vector is input into a multilayer perceptron classifier for dimensionality reduction mapping, and the output is a probability distribution vector corresponding to the mechanical fault category in the system. The probability distribution vector of multiple consecutive diagnostic cycles is smoothed by multi-frame probability mean processing, and the term with the largest weighted probability is output as the mechanical fault probability result.

[0017] This invention provides an AI-based fault analysis method for coal mine inspection machines in dusty, low-light environments. It offers the following advantages: 1. This invention establishes a transmittance time series prediction model, which can predict the window when the dust concentration is at a local minimum value, and uses the underlying hardwire to send synchronous trigger pulses to heterogeneous optical sensors, eliminating the time delay error caused by traditional software instruction scheduling, and forcing the vision and infrared cameras to perform synchronous exposure at the best time when the dust obscuration is weakest, thereby obtaining basic images and temperature measurement data with usable signal-to-noise ratio under the high dynamic dust conditions of coal mines.

[0018] 2. This invention uses temporal linear interpolation to reconstruct a virtual instantaneous transmittance matrix aligned with the camera's exposure moment, and combines this with a line-of-sight ray tracing algorithm to extract the cumulative attenuation value of the two-dimensional viewpoint, quantifying the attenuation degree in the line-of-sight direction of the heterogeneous sensor. By performing inverse compensation pixel by pixel on the infrared thermometry matrix through the Stefan-Boltzmann law, the background radiation interference of the high-temperature aerosol itself and the absorption loss of target thermal radiation by the dust layer are effectively removed, restoring the true absolute temperature field distribution on the device surface and improving the accuracy of infrared thermal diagnosis.

[0019] 3. This invention establishes a cross-modal adaptive takeover and multi-modal fusion mechanism based on the optical signal-to-noise ratio failure threshold. When the optical channel is blocked due to excessively low environmental transmittance, the system uses the spatial absolute center coordinates extracted from the solid-phase point cloud to guide the microphone array to perform beam phase delay compensation, and dynamically updates the confidence weight factors of each mode according to the instantaneous transmittance. This allows the system to smoothly transition to an acoustically-dominated diagnostic mode under extreme conditions where vision and infrared signals are blocked, avoiding monitoring blind spots caused by the failure of a single sensor and ensuring the continuity of the electromechanical equipment fault diagnosis process. Attached Figure Description

[0020] Figure 1 This is an architecture diagram of an AI fault analysis system for coal mine inspection machines in dusty and low-light environments, provided in an embodiment of the present invention. Figure 2 Flowchart of an AI-based fault analysis method for coal mine inspection machines in dusty, low-light environments, provided in an embodiment of the present invention. Figure 3 A schematic diagram illustrating the prediction of environmental dust transmittance evolution trend and the triggering timing of underlying hardware provided in an embodiment of the present invention; Figure 4 A comparison diagram of the reverse compensation effect of infrared thermometry under different dust transmittances provided in the embodiments of the present invention; Figure 5 This is a comparison chart of the system fault diagnosis accuracy under different test conditions provided in the embodiments of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 , Figure 1 This is an architecture diagram of an AI fault analysis system for coal mine inspection machines in dusty, low-light environments, according to an embodiment of the present invention. The present invention provides an AI fault analysis system for coal mine inspection machines in dusty, low-light environments, which may include: The edge computing inspection robot is equipped with a multi-echo 3D LiDAR, a global shutter visible light camera, an infrared thermal imager, and a microphone array. These sensors are connected to the edge computing main control unit via a high-speed data bus.

[0023] The general-purpose input / output pins of the edge computing main control unit are directly connected via hardwired to the external trigger pins of the global shutter visible light camera and the infrared thermal imager to transmit hardwired trigger pulse level signals. The multi-echo 3D LiDAR, global shutter visible light camera, infrared thermal imager, and microphone pickup array complete external parameter calibration in the spatial coordinate system to obtain the rigid transformation matrix between the coordinate systems of each sensor.

[0024] The transmittance calculation module is configured to receive point cloud and echo intensity data from a multi-echo 3D lidar, separate the solid phase point cloud from the gas phase suspended point cloud, and calculate and generate a three-dimensional spatial dust transmittance grid sequence.

[0025] The timing control module is configured to receive grid sequence data output by the transmittance calculation module, predict the evolution trend of dust transmittance, and obtain the global exposure midpoint timestamp of the image acquisition device by sending hardwired trigger pulses through the edge computing main control unit.

[0026] The compensation module is configured to reconstruct a virtual instantaneous transmittance matrix based on the global exposure midpoint timestamp, and use this matrix to perform inverse compensation calculation on the infrared temperature measurement matrix output by the infrared thermal imager, outputting a thermal feature matrix.

[0027] The beam reconstruction module is configured to extract the spatial absolute center coordinates of the target device when the instantaneous transmittance is lower than a set threshold, and reconstruct the phase delay compensation of each channel in the microphone pickup array accordingly, and extract the acoustic envelope feature vector.

[0028] The fault decision module is configured to receive thermal feature matrix, acoustic envelope feature vector and visible light image features output by a global shutter visible light camera, allocate dynamic confidence weights through nonlinear mapping, and calculate and output the probability result of mechanical fault of the equipment.

[0029] See attached document Figure 2 , Figure 2 This is a flowchart of an AI-based fault analysis method for coal mine inspection machines in dusty, low-light environments, according to an embodiment of the present invention. The present invention provides an AI-based fault analysis method for coal mine inspection machines in dusty, low-light environments, comprising the following steps: S10, the edge computing inspection robot performs inspection tasks on the target working surface, and the multi-echo 3D LiDAR continuously scans the target environment; the transmittance calculation module analyzes the multi-echo data, and separates the solid phase point cloud representing the equipment surface and the gas phase suspended point cloud representing air suspended particles according to the echo intensity; calculates the extinction coefficient of the spatial grid corresponding to the gas phase suspended point cloud, and generates a three-dimensional spatial dust transmittance grid sequence representing the attenuation state of the spatial environment. S20, the timing control module extracts the transmittance time series of the region of interest of the target device, uses a filter to smooth the evolution trend and calculates the time derivative to predict the time interval where the transmittance is at a local maximum; at the beginning of the interval, the timing control module sends a hard-wired trigger pulse through the general-purpose input / output pin to force the global shutter visible light camera and infrared thermal imager to perform synchronous global exposure operation, and the edge computing main control unit synchronously records the returned global exposure midpoint timestamp; S30, the compensation module performs temporal linear interpolation on the two frames of three-dimensional spatial dust transmittance grid sequence before and after the exposure time based on the global exposure midpoint timestamp, generating a virtual instantaneous transmittance matrix aligned with the exposure moment of the infrared thermal imager; combined with the acquired ambient background radiation temperature, the compensation module uses the virtual instantaneous transmittance matrix to perform inverse radiation attenuation compensation on the infrared temperature measurement matrix pixel by pixel, obtaining a thermal feature matrix that restores the actual surface temperature of the device. S40, the fault decision module evaluates the virtual instantaneous transmittance matrix in real time; if the average instantaneous transmittance of the virtual instantaneous transmittance matrix is ​​lower than the preset optical signal-to-noise ratio failure threshold, the beam reconstruction module extracts the spatial absolute center coordinates of the target device from the pre-isolated solid point cloud, and after the coordinates are mapped by a rigid translation and rotation transformation matrix, they are input into the beamforming algorithm to dynamically calculate the phase delay compensation of each channel of the microphone pickup array; the microphone pickup array completes spatial focusing according to the phase delay compensation and outputs the acoustic envelope feature vector of the target device; S50, the fault decision module calculates the dynamic confidence weight factor of each modal feature based on the numerical value of the average instantaneous transmittance; it performs feature-level fusion calculation of visible light image features, thermal feature matrix and acoustic envelope feature vector with the corresponding dynamic confidence weight factor, and outputs the mechanical fault probability result used to characterize the state of the inspection target.

[0030] When the edge computing inspection robot performs inspection tasks on the target working surface, a multi-echo 3D LiDAR continuously scans the target environment. The transmittance calculation module performs the following steps to calculate and generate a transmittance mesh sequence: S11, Separate the solid-phase point cloud from the gas-phase suspended point cloud. The transmittance calculation module receives the raw point cloud data output by the multi-echo 3D lidar. As the basis for the system's environmental perception, the raw point cloud data includes the three-dimensional spatial coordinates of each scanning point, echo intensity, and echo order information. Considering that when the laser penetrates dusty air, tiny suspended particles will produce weak Rayleigh or Mie scattering, while impacting the solid surface of the device will form high-intensity diffuse reflection, the system establishes separation logic based on this causal relationship. In this embodiment, the system pre-programs a calibrated rigid body reflection intensity threshold.

[0031] This threshold is typically determined based on benchmark measurements in a dust-free environment, serving as a boundary for distinguishing media morphology. The transmittance calculation module classifies scan points with echo intensity greater than or equal to the rigid body reflection intensity threshold as solid-phase point clouds, thus characterizing the rigid surface contours of equipment in the target environment. Simultaneously, for scan points with echo intensity less than the rigid body reflection intensity threshold and marked as the first echo, the transmittance calculation module classifies them as gas-phase suspended point clouds. The reason for selecting the first echo as the criterion is that the first point of contact and scattering after the light beam is emitted is inevitably the airborne particles filling the space. Through this classification method based on multi-dimensional parameters combined with feature determination, the system can effectively extract gas-phase parameters reflecting dust distribution characteristics from invalid environmental data typically filtered out by conventional navigation and obstacle avoidance algorithms.

[0032] S12 calculates the extinction coefficient of the spatial grid corresponding to the gas-phase suspended point cloud. For the three-dimensional discrete voxel grid occupied by the gas-phase suspended point cloud in space, the transmittance calculation module needs to extract the core physical quantity characterizing the attenuation of the optical signal with distance. Based on this technical objective, the system processes the echo intensity data of the gas-phase suspended point cloud based on the aerosol lidar equation. Generally speaking, the echo signal power received by the lidar system is proportional to the backscattering coefficient of the measured medium and also proportional to the two-way transmittance of the medium. The transmittance calculation module establishes a mapping model between echo intensity and extinction coefficient based on this general rule. The specific derivation relies on the following expression of the lidar equation: ; In the formula, Represents distance The power of the gas-phase suspended point cloud echo signal received at the location directly corresponds to the echo intensity value in the scan point data. Represents the laser pulse emission power; The system constants representing the multi-echo 3D LiDAR are obtained through the equipment's factory calibration. Represents the absolute spatial distance from the radar optical center to the scattering target; Represents distance The spatial volume backscattering coefficient of the dust medium; Represents distance Spatial extinction coefficient of dusty media; Represents the integral distance variable along the direction of the radar beam; This represents a constant characterizing the round-trip transmission path of a light beam; The definite integral represents the distance from the optical center to the target scattering point. The differentiable element representing the distance variable rr; This represents an exponential function with the natural constant as its base.

[0033] When extracting the spatial extinction coefficient using this equation, it is necessary to eliminate the coupling between the backscattering coefficient and the extinction coefficient. As a preferred approach, the system adopts a fixed extinction backscattering ratio assumption to establish a numerical solution model. The selection of this fixed ratio depends on the specific optical scattering characteristics of coal dust particles and is usually determined during the calibration stage by comparing with measured data from a transmission meter. Since division operations are inevitably involved in the numerical solution process, to prevent the denominator from approaching zero and causing algorithm divergence due to a sharp increase in local dust concentration or the appearance of a pure air gap, the transmittance calculation module introduces a boundary constraint mechanism within the iterative calculation framework. Specifically, it sets maximum and minimum threshold values ​​for the extinction coefficient, and automatically replaces the calculated result with the neighborhood smoothed mean when it exceeds the limits. Combined with the above comprehensive protection mechanism, the system robustly completes the inversion of the extinction coefficient at each coordinate point.

[0034] S13 generates a three-dimensional spatial dust transmittance grid sequence characterizing the attenuation state of the space environment. The transmittance calculation module establishes a three-dimensional grid coordinate system with the optical center position of the multi-echo 3D LiDAR as the origin. For the discrete time corresponding to the current scanning frame time, the transmittance calculation module calculates the atmospheric transmittance along the probe optical path at any coordinate point within the field of view based on the extinction coefficient of each grid point obtained from the previous steps. This calculation process aims to quantify the proportion of energy retained after the light signal penetrates a specific spatial region. The calculation formula is as follows: ; In the formula, Representing discrete time Lower three-dimensional coordinate points The cumulative atmospheric transmittance at that location; Represents the distance from the origin to the target coordinate point. The straight-line distance; Representing discrete time Distance point on the lower light path Local extinction coefficient at the location; Represents the distance variable along the integral optical path; The definite integral representing the distance from the optical center to the straight line is calculated. Distance variable Differential elements.

[0035] After completing the integration calculation for a single optical path, the transmittance calculation module matrix-arranges the atmospheric transmittance values ​​of all three-dimensional discrete grid coordinate points within the field of view, generating a three-dimensional spatial dust transmittance grid at a specific discrete time. During this environmental reconstruction process, the grid size is determined by both the sensor's angular resolution and the parallel processing power of the edge computing control unit, ensuring a balance between spatial sampling accuracy and real-time performance. Based on the characteristic of multi-echo 3D LiDAR operating continuously at a fixed scanning frequency, the transmittance calculation module outputs three-dimensional spatial dust transmittance grids at multiple discrete time points, which are then combined to form a three-dimensional spatial dust transmittance grid sequence. The output three-dimensional spatial dust transmittance grid sequence provides a basic environmental attenuation benchmark containing spatiotemporal two-dimensional evolution information for subsequent modules.

[0036] After acquiring the three-dimensional dust transmittance grid sequence, the system needs to overcome random occlusion caused by dynamic dust turbulence to guide the passive optical sensor to acquire data at the optimal time. The timing control module performs the following steps to achieve precise exposure triggering: S21: Extract the regional transmittance sequence and perform filtering, smoothing, and derivative calculation. The timing control module receives the 3D spatial dust transmittance grid sequence output by the transmittance calculation module. Since the core technology of industrial site inspection aims to monitor the operating status of specific equipment, directly processing the panoramic grid would introduce a large amount of irrelevant environmental background interference. Based on this, the system, according to the pre-set 3D bounding box coordinates of the equipment, clips out the local grid corresponding to the region of interest of the target equipment from the grid sequence. The timing control module calculates the spatial mean of the transmittance of all discrete voxels within this local grid, generating a time series of target equipment transmittance varying with discrete time.

[0037] Coal dust particles exhibit violent turbulent and disordered motion due to the influence of mine ventilation airflow, resulting in a large amount of high-frequency random noise in the original transmittance time series. To extract the macroscopic envelope trend characterizing the evolution of dust concentration while filtering out noise, the time series control module establishes a Kalman filter based on a kinematic model. In this embodiment, the system constructs a three-dimensional state vector, including the mean transmittance, the first-order time derivative of transmittance, and the second-order time derivative of transmittance. The state transition equation is expressed as follows: ; In the formula, Represents the current discrete moment The prior state estimation vector; The state transition matrix is ​​represented by a matrix internally constructed based on the fundamental principle of uniformly accelerated time-difference kinematics. Represents the previous discrete time. The system state posterior update vector; Represents the system process noise vector; subscript and These represent adjacent discrete sampling time steps. As a preferred approach, The covariance matrix is ​​calibrated offline based on the historical environmental wind speed fluctuation variance collected by the anemometer at the target working face to reflect the true disturbance intensity.

[0038] In the subsequent iterative process of calculating the Kalman gain and updating observations, the system inevitably involves inverting the innovation covariance matrix. Considering that if the environmental dust remains absolutely still for an extremely short time, the covariance matrix may exhibit ill-conditioned or even singular behavior, the timing control module forcibly superimposes a small diagonal regularization matrix of the thermal noise level before inversion to ensure absolute stability of the numerical calculations. Through the aforementioned filter prediction and observation update iterative calculations, the system outputs a smoothed mean transmittance sequence and its corresponding first and second time derivatives. The mean transmittance represents the absolute degree of current dust obstruction, the first time derivative represents the rate of change of dust concentration, and the second time derivative represents the acceleration of the dust change trend.

[0039] S22, predicts the time interval corresponding to the local maximum of transmittance. Based on the acquired smoothed derivative data, the timing control module executes extremum optimization logic to lock in a window where dust is relatively sparse. The system continuously monitors the numerical sign of the first-order time derivative. When the first-order time derivative is greater than zero and the second-order time derivative is less than zero, it physically means that the transmittance of the current environment is increasing but the rate of increase is beginning to slow down, and the dust concentration is about to reach a local minimum.

[0040] Under this evolutionary trend, the timing control module uses the current derivative value to extrapolate based on a first-order Taylor expansion to calculate the predicted time point when the first-order time derivative decays to zero. This extrapolation process essentially involves dividing the current first-order time derivative by the second-order time derivative to obtain the time increment. To prevent system crashes caused by the denominator being zero during division when the second-order time derivative approaches zero (i.e., when transmittance changes at a uniform rate), the timing control module sets a small denominator tolerance threshold. If the absolute value of the second-order time derivative is detected to be less than this tolerance threshold, the system directly determines that the current trend is not an extreme value evolution trend and skips the prediction. After passing the division safety check, the system obtains the predicted time point when the first-order derivative returns to zero; this time point is the moment when transmittance reaches a local maximum.

[0041] Since the photoelectric sensors of global shutter visible light cameras and infrared thermal imagers require a certain photon integration time to complete a full exposure, the system cannot rely solely on a single time point as the trigger. The timing control module uses this predicted time point as the center and, combined with the preset equipment exposure duration requirements, defines a time interval extending forward and backward. In this embodiment, the width of this time interval is set at the millisecond level. This millisecond-level width is based on the fact that the duration of the trough in the macroscopic dust envelope is typically on the order of tens of milliseconds, sufficient to cover the camera's normal exposure requirements. The system defines this defined millisecond-level time interval as the dust trough window.

[0042] To prevent invalid triggering in extremely murky environments, the system incorporates multi-dimensional judgment logic. The timing control module extracts the absolute value of the predicted transmittance at the predicted time point while defining the time interval. The system only confirms a valid trigger window if the predicted value is greater than a preset minimum optical signal-to-noise ratio (OSN) threshold. This minimum OSN threshold is calibrated at the factory based on the camera sensor's dark current noise parameters and quantum efficiency characteristics, representing the minimum amount of light allowed for the sensor to distinguish the outline of a valid target. If the predicted extreme value is still below this threshold, it indicates that the current local dust level is insufficient to meet imaging requirements, and the system abandons the current triggering and waits for the next evolution cycle.

[0043] S23 achieves microsecond-level exposure triggering via underlying hard-wired pulses. When the system's internal high-precision clock reaches the start of the aforementioned effective time interval, the edge computing main control unit executes the image acquisition command. Because conventional operating systems use time-slice round-robin task scheduling, there is uncontrollable thread switching overhead. Using application-layer software commands to trigger the camera generates random latency jitter ranging from several milliseconds to tens of milliseconds, causing the actual exposure action to deviate from the predicted dust trough window.

[0044] To eliminate this spatiotemporal misalignment, this embodiment employs a low-level control scheme that bypasses the system software layer. The low-level registers of the edge computing main control unit directly drive the general-purpose input / output pins to generate level toggles, sending hard-wired trigger pulses in parallel to the external trigger pins of the global shutter visible light camera and the infrared thermal imager. Since the low-level hardware response delay of the electrical signals is on the order of microseconds or less, this mechanism forces the two heterogeneous optical sensors to synchronously start the global shutter for photon capture within a predicted millisecond time interval with extremely low time error.

[0045] Upon receiving a hard-wired trigger pulse, the field-programmable gate array (FPGA) at the bottom layer of the global shutter visible light camera and infrared thermal imager controls all pixel units of the photosensitive chip to simultaneously begin and end exposure. At the instant the exposure cycle ends, the edge computing main control unit captures the feedback signal via a hardware interrupt and, combined with the known exposure duration, calculates the global exposure midpoint timestamp for this image acquisition. This global exposure midpoint timestamp accurately records the core time node of the optical sensor recording the real-world scene, providing an absolute time reference for subsequent temporal alignment and compensation of cross-sensor data.

[0046] After obtaining the global exposure midpoint timestamp, the compensation module performs the following steps to eliminate the spatiotemporal misalignment caused by the different sampling systems of heterogeneous sensors and restore the true absolute temperature of the device surface: S31 reconstructs a virtual instantaneous transmittance matrix based on a time reference. Considering the realities of underlying perception, multi-echo 3D LiDAR relies on a mechanical rotating structure for continuous periodic scanning, generating a three-dimensional spatial dust transmittance grid sequence with discrete and low-frequency temporal attributes. In contrast, infrared thermal imagers perform instantaneous global exposure under hard-wired pulse triggering. This fundamental difference in underlying hardware sampling mechanisms means that the spatial perception data of the LiDAR and the image capture data of the camera cannot be naturally aligned on the time axis. If non-synchronous transmittance grids are directly used to compensate for the infrared image, severe spatial misalignment errors in temperature measurement will be introduced when the target moves or dust diffuses violently.

[0047] To address the aforementioned spatiotemporal asynchrony issue, the compensation module receives the global exposure midpoint timestamp recorded by the edge computing master control unit. Using this absolute time node as a reference, the system retrieves two consecutive valid data frames temporally adjacent to and following this exposure timestamp from the 3D dust transmittance grid sequence. The compensation module then uses the time nodes of these two data frames to perform temporal linear interpolation on the discrete 3D grid. The specific interpolation model is as follows: ; In the formula, Represents the timestamp in the global exposure. The three-dimensional coordinate points obtained by time reconstruction The corresponding virtual instantaneous transmittance; and These represent the first and second moments before and after the exposure time. Frame and the Timestamps of the frame grid sequence; and These represent the actual transmittance values ​​extracted from the discrete grid at the corresponding time points; This represents the absolute timestamp of the global exposure midpoint recorded by the edge computing control unit. Since the multi-echo 3D LiDAR rotates at a fixed mechanical frequency, the time difference between two adjacent frames, represented by the denominator of the formula, is a constant non-zero positive number, eliminating the risk of algorithm crashes due to a zero denominator from the underlying mathematical logic. Through this interpolation calculation, the system outputs a virtual instantaneous transmittance matrix that is absolutely aligned with the instant of exposure of the infrared thermal imager.

[0048] S32, establish the spatial mapping relationship between infrared pixels and the 3D mesh. Based on the absolute alignment results at the time level, the compensation module needs to reduce the dimensionality of the 3D spatial mesh to the 2D imaging plane of the infrared thermal imager. The system calls the rigid transformation matrix obtained from the previous external parameter calibration, and performs coordinate system transformation in combination with the optical intrinsic parameter matrix of the infrared thermal imager. For any valid pixel in the infrared temperature measurement matrix, the compensation module emits a virtual ray along the optical line of sight of that pixel, and extracts the cumulative attenuation value corresponding to the voxel of the mesh penetrated by the ray in the virtual instantaneous transmittance matrix. In this embodiment, for the case where the virtual ray passes through the interface of multiple discrete voxel meshes, the system adopts a line-of-sight ray tracing algorithm with a preset step size to perform multiplication of the transmittance along the way. As a preferred method, the sampling step size of the ray tracing algorithm is synchronously set according to the minimum voxel side length of the 3D mesh to ensure that the algorithm does not excessively consume processor computing power, nor does it miss any local dust density change characteristics. By traversing all pixels, the system obtains a 2D view projection transmittance matrix corresponding one-to-one with the imaging array of the infrared thermal imager.

[0049] S33 utilizes a modified Stefan-Boltzmann law to perform pixel-by-pixel inverse compensation. The total radiant energy received by the infrared thermal imager includes not only the effective thermal radiation emitted from the target device surface and penetrating the dust layer, but also the background thermal radiation emitted by the dust aerosols in the high-temperature environment and captured by the lens. To eliminate this environmental thermal radiation interference and restore the true target surface temperature, the compensation module constructs an infrared temperature field inverse compensation model based on energy conservation and the Stefan-Boltzmann law. Essentially, this model subtracts the contribution of environmental background radiation from the total radiance received by the sensor, and then divides by the spatial transmittance to compensate for path loss. The model calculation formula is as follows: ; In the formula, Represents infrared image coordinates The actual absolute temperature of the equipment surface after reduction; This represents the measured temperature value directly output by the infrared thermal imager at this coordinate. This represents the transmittance value in the two-dimensional view projection transmittance matrix obtained by the preceding steps in the viewing direction of this pixel. This represents the ambient background radiation temperature. In this embodiment, the ambient background radiation temperature... Real-time data is collected by an externally mounted contact-type ambient temperature sensor on the edge computing inspection robot.

[0050] When performing the compensation calculation pixel by pixel, high concentrations of coal dust may completely obstruct local views, causing the projected transmittance to approach zero. Directly substituting into the formula would cause division overflow, forcing the system to push the calculation result to infinity, thus causing thermal feature extraction to fail. Therefore, the compensation module presets a hard transmittance truncation threshold. This threshold is typically set between 0.05 and 0.15, with its specific value determined experimentally before shipment based on the infrared thermal imager's noise equivalent temperature difference parameter. It represents the boundary between effective thermal radiation and underlying background thermal noise that the system can distinguish. When the projected transmittance value is detected to be less than this threshold, the system determines that the optical channel corresponding to the current pixel has irreversibly failed, automatically skips the square root operation for that pixel, and marks its temperature value as an invalid background value. Furthermore, considering that the underlying thermal noise of the measuring equipment may cause a small negative number in the numerator, the system applies an absolute value non-negativity constraint to protect it before performing the fourth root calculation.

[0051] After the above calculations and anomaly removal, the compensation module outputs the compensated global thermal characteristic matrix. This matrix eliminates the interference from radiation absorption and stray emission of dust aerosols, reflecting the true temperature field distribution on the equipment surface, and providing an objective basis for subsequent system evaluation of mechanical faults such as bearing overheating or motor stator overload.

[0052] In coal mine inspection operations, high concentrations of dust emissions can cause severe or even complete attenuation of high-frequency electromagnetic waves. Based on these harsh sensing conditions, the beam reconstruction module performs the following steps: utilizing acoustic sensors with strong resistance to dust particle scattering interference to take over environmental condition perception, and relying on previously acquired three-dimensional spatial coordinates for acoustic directional guidance: S41, an acoustic takeover control strategy is triggered based on multi-dimensional optical failure determination. The beam reconstruction module continuously monitors the overall spatiotemporal evolution of the three-dimensional dust transmittance grid sequence. For the region of interest of the target device, the system extracts the transmittance of all local grid points at the current discrete moment and calculates its spatial mean to obtain the average instantaneous transmittance. To objectively evaluate the current availability of the optical sensing channel, the system compares the average instantaneous transmittance with a preset optical signal-to-noise ratio failure threshold in real time. In this embodiment, the optical signal-to-noise ratio failure threshold is set to a range of 0.01 to 0.03. This value range is calibrated offline based on the dark area shot noise limit of the global shutter visible light camera hardware itself, representing the critical point where the photon penetration probability can no longer meet the minimum image recognition requirements.

[0053] Considering the potential for brief, localized obstructions caused by splashing coal or transient airflow in industrial environments, relying solely on extreme or average drops at a single moment for mode switching can easily lead to frequent oscillations and false triggers in the system's perception logic. As a preferred approach, the system introduces a multi-dimensional weighted judgment logic based on a time-sliding window. The system not only monitors the average transmittance of a single frame but also records its duration. Only when the average transmittance of multiple consecutive sampling frames (typically set to 3-5 frames) falls below the optical signal-to-noise ratio failure threshold does the system definitively confirm irreversible blockage of the conventional visual perception channel. Based on this comprehensive judgment, the system automatically issues a bus hardware takeover command, triggering the microphone array mounted on the inspection robot's chassis to initiate broadband acoustic detection, effectively avoiding overall monitoring blind spots caused by the failure of a single sensor.

[0054] S42, extract the absolute center coordinates of the target device in space and perform cross-coordinate system mapping. To achieve accurate acoustic spatial focusing subsequently, the beam reconstruction module must first obtain the precise three-dimensional position of the target sound source in the physical world. Based on this technical objective, the system retrieves the solid-phase point cloud data successfully separated by the transmittance calculation module during the initial environmental scanning phase. This point cloud data represents a high-reflectivity rigid surface, is not deeply contaminated by gaseous dust attenuation, and completely preserves the geometric contour information of the target device's rigid shell. The beam reconstruction module performs a density-based spatial clustering algorithm on this local solid-phase point cloud. For the specific iterative optimization of the core distance and neighborhood radius parameters in this algorithm, those skilled in the art can consult conventional publicly available literature in the field of three-dimensional point cloud processing; the specific cluster generation process is a well-known technique in this field and will not be elaborated upon here.

[0055] After completing point cloud clustering and removing discrete noise points, the system extracts the geometric absolute center coordinates of the effective point cloud clusters of the target device. Due to the inherent geometric and mechanical offset between the optical center of the multi-echo 3D LiDAR and the center of the microphone array on the inspection robot chassis, directly using these center coordinates would result in acoustic focusing deviation. To eliminate spatial misalignment errors between heterogeneous sensors, the beam reconstruction module calls a rigid translation and rotation transformation matrix pre-calibrated at the factory. The system performs a rigid body spatial pose transformation on the point cloud geometric center coordinates in the LiDAR coordinate system using standard four-dimensional homogeneous coordinate multiplication, absolutely mapping them to a local acoustic coordinate system with the microphone array center as the origin. After dimensionality reduction and realignment of this coordinate system, the system outputs the spatial absolute center coordinates of the target device in the acoustic perception dimension.

[0056] S43, dynamically recalculates the phase delay compensation for each channel and extracts the acoustic envelope feature vector. After obtaining the high-precision sound source location, the beam reconstruction module uses delay-summation beamforming technology to guide the system in signal enhancement. The underlying principle of this general technology is to apply a specific time delay to the array receiving channels at different spatial locations, so that the target signal originating from the target direction is precisely phase-aligned and coherently superimposed in the time domain, while stray noise from other non-target directions is incoherently canceled due to phase interleaving.

[0057] Because the transmission path lengths from the spatial sound source of the target device to each microphone channel are inconsistent, the acquired analog electrical signals have an inherent spatial phase deviation on the time axis. The beam reconstruction module uses the remapped absolute center coordinates of the target device space to calculate the time delay compensation for each channel. The specific delay calculation formula is as follows: ; In the formula, Representative regarding the first The time delay compensation amount is calculated for each microphone channel; This represents the absolute center coordinates of the target device space extracted and mapped after the aforementioned steps. Representing the The known hardware coordinates of a microphone in the local acoustic coordinate system; Represents the linear spatial distance from the reference microphone channel to the absolute center coordinates of the target device. The reference air speed constant at standard atmospheric pressure at zero degrees Celsius (usually taken as 331.4 m / s). Represents the temperature compensation coefficient for the speed of sound; This represents the ambient temperature collected in real time by the aforementioned external contact temperature sensor (since the sensor is the same, this variable is equivalent to the aforementioned ambient background radiation temperature, and is used here to characterize the local air absolute temperature).

[0058] Considering the continuous operation of electromechanical equipment in the underground working face, which causes drastic fluctuations in local ambient temperature, directly using a fixed ambient temperature sound velocity for division spatial mapping would lead to severe misalignment of the phase alignment of high-frequency sound waves. As a preferred approach, the system incorporates the aforementioned real-time acquired ambient background radiation temperature to positively compensate for the actual sound velocity. In this embodiment, based on the ideal gas thermodynamic sound velocity approximation theory, this sound velocity temperature compensation coefficient... The value is set to 0.606 m / (s·℃). The denominator of this formula fully represents the actual time-varying sound speed in the compensated environment. Based on the objective principles of basic acoustics, the actual sound speed under normal conditions is always positive and much greater than zero, thus avoiding the risk of microprocessor crashes caused by the denominator of division operations approaching zero from the underlying mathematical logic.

[0059] Based on precisely calculated time delay compensation amounts, the system performs smooth time-axis shifting and alignment of the original acoustic array signals in the hardware sampling buffers of each channel, and then performs weighted summation coherent operations. Through this spatiotemporally coordinated beamforming operation, the mechanical noise energy at the location of the target device is coherently amplified several times. After completing the temporal domain enhancement and reconstruction, the beam reconstruction module performs a Fast Fourier Transform on the synthesized one-dimensional time master signal, extracts the frequency domain amplitude envelope information within its specific diagnostic frequency band, and then generates an acoustic envelope feature vector characterizing the mechanical vibration characteristics of the device. This feature vector provides high signal-to-noise ratio, attenuated auditory dimension feature support for subsequent fault diagnosis under extreme obstruction conditions where both visual and infrared sensing channels fail.

[0060] After acquiring multi-dimensional sensing data with prior spatiotemporal alignment, the core technical challenge faced by the system lies in the drastic fluctuations in the signal-to-noise ratio of single-modal data caused by dynamic dust occlusion. Based on this complex operating condition, the fault decision module performs the following steps to achieve fault diagnosis that adaptively adjusts feature weights based on the current environmental state: S51 dynamically calculates the basic confidence scores for each mode based on spatial transmittance and performs normalized redistribution. In a real mine environment, the sensitivity of visual, infrared, and acoustic sensors to suspended dust particles exhibits different mechanisms. Visible light is easily blocked by gaseous dust, resulting in Mie scattering; long-wave infrared light has a certain penetrating ability but is still affected by aerosol absorption; while sound waves, as mechanical waves, are negligibly affected by dust concentration on a macroscopic scale. Based on these objective laws, the fault decision module uses the average instantaneous transmittance extracted at the current moment to evaluate the current reliability of the sensing data for each mode in real time.

[0061] In practice, the system is a global shutter visible light camera (defined as a visual modality). Infrared thermal imager (defined as infrared mode) ) and microphone pickup arrays (defined as acoustic modalities) The system constructs basic confidence score functions for each mode. For the visual and infrared modes, the system sets their respective basic confidence scores as linear mapping values ​​related to the mean transmittance of the region of interest of the current target device, with the mapping value limited to the range of 0 to 1. For the acoustic mode, based on its resistance to dust obstruction, the system assigns a constant acoustic basic confidence score to the data extraction mode corresponding to the acoustic envelope feature vector, based on the reverse calibration of the measured signal-to-noise ratio under unobstructed normal conditions. To convert the above basic scores of different dimensions into probability distribution weights that can be used for matrix operations, the system introduces a normalized exponential function based on temperature hyperparameter adjustment for confidence redistribution. The redistribution formula is as follows: ; In the formula, This represents the allocation to a specific mode after reallocation. Normalized dynamic weights; variables and The set of values ​​is These correspond to the three modes of vision, infrared, and acoustic, respectively. and These represent the basic confidence scores obtained from the calculations of the corresponding modalities mentioned above; Temperature hyperparameter representing the smoothness of the control weight distribution; Represents an exponential function with the natural constant as its base; This represents a summation operation on all modal elements within the set. In this embodiment, the temperature hyperparameter... The value range is set between 1.5 and 2.5. Its specific value is determined based on the joint noise immunity test results of the multimodal sensor in the standard calibration chamber. The introduction of this parameter effectively avoids a step-like change in the weight of a single mode when the transmittance crosses the critical point, ensuring a smooth transition of the system's weighted logic. Simultaneously, since the exponential function's range is always greater than zero, the denominator of this normalization formula is always positive, naturally avoiding the risk of microprocessor crashes due to the denominator approaching zero during division operations from a fundamental mathematical perspective.

[0062] S52 extracts high-dimensional abstract features from heterogeneous modalities and performs latent space dimension alignment. Before performing weighted fusion, the system must eliminate incompatibility issues at the data structure level between multiple sources. The system input data includes wide... high The three-channel two-dimensional visible light image, the single-channel two-dimensional infrared thermal feature matrix, and the one-dimensional acoustic envelope feature vector reconstructed across modes differ fundamentally in their tensor dimensions. Directly performing algebraic fusion operations on these heterogeneous data will trigger a system error due to matrix dimension mismatch.

[0063] To establish a consistent mathematical foundation, the fault decision-making module invokes three sets of feature extraction neural network branches with specific structures in parallel. As a preferred approach, before feeding data into the network, the system performs uniform preprocessing operations: global pixel maximum and minimum value normalization is performed on the image and infrared matrix to map their values ​​to the [0,1] interval; zero-mean unit variance standardization is performed on the acoustic envelope vector to eliminate the interference of absolute amplitudes of different modes on gradient calculation.

[0064] After preprocessing, for the visual modality, the system employs a ResNet-18 residual convolutional network with the fully connected layers at the end removed for classification. It utilizes the global average pooling layer of its deep feature maps to map the visible light image into a fixed-length 512-dimensional visual feature vector. For the infrared modality, the system constructs a lightweight feature extractor consisting of three consecutive cascaded 2D convolutional layers (Conv2D) and max-pooling layers (MaxPooling). After flattening the ends, it outputs a 512-dimensional infrared feature vector through a fully connected layer. For the acoustic modality, the system calls a temporal feature network containing multiple 1D convolutional layers (Conv1D) and a 1D global average pooling layer (GlobalAveragePooling1D) to map the acoustic envelope features into a 512-dimensional acoustic feature vector. During this process, if the current operating condition does not reach the optical signal-to-noise ratio failure threshold and thus does not trigger acoustic modal takeover, the system automatically assigns the acoustic envelope feature vector to a zero vector that matches the dimension of the latent space, in order to maintain the absolute alignment of the tensors in the subsequent multimodal fusion network input structure. Through the above-mentioned independent network mapping processes, the three heterogeneous data representing different attributes are forcibly aligned and projected into a feature latent space of a unified dimension.

[0065] S53, perform confidence-weighted feature fusion and output multi-dimensional comprehensive equipment fault classification results. After obtaining the aligned latent space features mentioned above, the fault decision module uses the dynamic weights calculated in S51 to perform element-wise algebraic weighted summation on the three sets of feature vectors. The technical purpose of this step is to dynamically suppress the features of the currently heavily dust-contaminated sensing mode and amplify the feature responses of the uncontaminated mode through weight factors, so that the final synthesized feature vector always contains the highest quality equipment operating status information under the current operating conditions. The specific fusion calculation formula is as follows: ; In the formula, This represents the final synthesized 512-dimensional multimodal fusion feature vector; , and These represent the visual, infrared, and acoustic feature vectors extracted from each branch of the neural network, respectively. , and The normalized dynamic weights obtained from the aforementioned steps are calculated and summed to 1.

[0066] After obtaining the fused feature vector, the system inputs it into a multilayer perceptron (MLP) classifier consisting of two fully connected layers and a dropout layer cascaded together. This classifier uses a non-linear activation function to reduce the dimensionality of the high-dimensional features, and finally outputs a probability distribution vector corresponding to a preset mechanical fault category via a softmax layer. These specific categories cover clearly defined states such as severe bearing wear, motor stator insulation aging, rotor imbalance, and normal operation. To avoid frequent state jumps caused by relying on extreme results from a single frame, the system introduces a multi-frame probability mean smoothing strategy at the output, comprehensively considering the classification probabilities of five consecutive diagnostic cycles, and using the term with the highest weighted probability as the final fault diagnosis result for the current device.

[0067] To ensure the reliable predictive capabilities of the aforementioned neural network branches, this solution discloses its offline training mechanism. Training samples are derived from multimodal synchronous datasets collected by inspection robots equipped with the same type of sensors in simulated dust laboratories and real mine working faces. The classification labels of each working condition sample are confirmed by on-site equipment maintenance engineers through downtime disassembly and comparison, and mathematical annotation is performed using one-hot encoding. During the supervised learning phase, the system uses the cross-entropy loss function to measure the distribution error between the forward propagation predicted probability and the true label. For the calculation of the backpropagation error gradient based on this loss function, and the updating of the internal network weight matrix parameters using the Adam optimizer, those skilled in the art can consult publicly available literature in the field of deep learning; the specific network iterative convergence process is a well-known technique in this field and will not be elaborated upon here. After offline training and verification using a large-scale sample library, the solidified model parameters are deployed to the edge computing main control unit to execute real-time industrial inference.

[0068] Specific application examples: Inspection of drive motor faults in coal mine main conveyor belt To provide a more intuitive understanding of the technical solution of this invention, the following detailed description will be based on the inspection process of the drive motor of the main conveyor belt in the 1203 coal mining face.

[0069] Initial environmental state perception (corresponding to S10) The edge computing inspection robot moved to a distance of 3 meters from the drive motor. At this time, the coal mining machine had just passed by, and a large amount of high-concentration coal dust was released into the environment. The tunnel was in extremely harsh working conditions with low light and high dust.

[0070] The multi-echo 3D LiDAR scans the motor area at a frequency of 10Hz. The transmittance calculation module identifies point clouds with echo intensity > 85 (rigid body threshold) as the motor casing (solid phase), and point clouds with echo intensity < 85 and being the first echo as coal dust aerosol (gas phase).

[0071] The system calculated that the average dust transmittance between the target motor and the robot was only 0.18 (compared to 0.95 or higher in a typical cleanroom environment), indicating that the line of sight was severely obstructed.

[0072] Timing prediction and hardware-level exposure triggering (corresponding to S20) Mine ventilation airflow causes dust concentration to fluctuate wildly in a wave-like pattern. The timing control module monitors the first and second derivatives of transmittance using a Kalman filter and predicts that after 1.250 seconds, the transmittance will reach an extreme low point (i.e., the dust concentration will become extremely low), and the transmittance will rebound to 0.45.

[0073] When the predicted time point (1.250s) is reached, the bottom register of the edge computing master control unit directly pulls the GPIO pin high, generating a microsecond-level (approximately 2μs) hardwired trigger pulse.

[0074] The visible light camera and infrared thermal imager simultaneously expose the image, and the master control records the global exposure midpoint timestamp with precision as follows: .

[0075] Infrared temperature field inverse compensation (corresponding to S30) Because dust aerosols absorb infrared radiation and are superimposed with ambient background thermal radiation (measuring ambient temperature) At this time, the apparent temperature of the motor bearing surface directly output by the infrared thermal imager is only 45℃ (within the normal threshold).

[0076] Compensation module extraction The instantaneous virtual instantaneous transmittance matrix (obtained by linear interpolation in the time domain) is used for pixel-by-pixel inverse compensation using the modified Stefan-Boltzmann law.

[0077] After compensation, the system restored the bearing's true surface temperature to 78°C, which exceeded the warning threshold (75°C), exposing the thermal anomaly that was covered by dust.

[0078] Acoustic take-off and beam reconfiguration (corresponding to S40) During the inspection, a strong gust of wind suddenly passed by, carrying a large amount of coal dust. The instantaneous transmittance dropped to 0.02 (below the set optical failure threshold of 0.03), and four consecutive visible light images were almost completely black, indicating an extremely high risk of overflow in the infrared compensation denominator.

[0079] The beam reconfiguration module immediately triggers acoustic takeover. The absolute coordinates of the motor center (2.8m, 1.2m, -0.5m) are extracted from the previously preserved solid point cloud.

[0080] By combining the real-time measured temperature (28℃ with a 0.606-fold compensation for the speed of sound), the system calculates the precise phase delay of each channel of the microphone array. After beamforming, the abnormal frictional high-frequency envelope characteristics at around 500Hz inside the motor are successfully extracted, completely eliminating low-frequency noise interference from the coal mining machine in the background environment.

[0081] Dynamic weight allocation and fault decision-making (corresponding to S50) Due to the extremely low current transmittance (0.02), the system dynamically allocates confidence weights: visual weights. Infrared weight Acoustic weight .

[0082] The multi-source feature vectors are fused and input into the MLP classifier. Combining the 78℃ overheating feature previously captured by infrared with the current high-weight acoustic high-frequency friction feature, the system finally outputs a probability of 94.5% for severe wear of the motor drive bearing, successfully achieving high-precision early warning in a blind state.

[0083] Experimental verification and effect comparison To verify the effectiveness and advancement of the system of this invention, the research and development team conducted a comparative experiment for a total of 1,000 hours in a full-size simulated coal mine dust laboratory (50m long × 4m wide × 3m high) and a real working face of a mine with a capacity of 10 million tons.

[0084] Experimental setup: Fault Sample Library: Contains 8 common mechanical faults such as motor stator aging, bearing wear, rotor imbalance, belt tearing, etc., with a total of 1200 independent test samples.

[0085] Comparison method: Baseline Method A (Traditional Photoelectric Monitoring): Relies solely on uncompensated visible light + infrared images, triggered by application-layer software.

[0086] Baseline Method B (Static Multimodal Fusion): Includes visual, infrared, and acoustic methods, but uses fixed weights (average 0.33) and does not incorporate dynamic dust compensation or rigid body coordinate acoustic guidance.

[0087] The present invention (dynamic adaptive AI system) applies transmittance prediction, hardware triggering, thermal compensation and dynamic weight allocation throughout the entire process.

[0088] Quantitative Effect Comparison Table:

[0089] Conclusion Analysis: Combining the above table with Figure 3As can be seen, in the high-frequency turbulent dust environment of coal mines, the original measured transmittance value exhibits severe random fluctuations and a large amount of high-frequency noise due to the disordered movement of dust. The system successfully extracted a smooth envelope reflecting the evolution trend of macroscopic dust concentration changes through Kalman filtering technology. More importantly, the system accurately predicted the local maximum of transmittance (i.e., the instant when the dust concentration is relatively low) based on this smooth envelope, and precisely issued the underlying hard-wired trigger point within this time window. This underlying hardware-level microsecond-level triggering mechanism eliminates the random latency caused by thread scheduling in traditional operating systems, reducing the camera exposure trigger latency from 12-18ms to 0.002ms as shown in the table, ensuring that the optical sensor can capture images under the optimal viewing window.

[0090] Combining the above table with Figure 4 As can be seen, as the dust transmittance on the horizontal axis gradually decreases from 1.0 (clear) to 0.05 (extremely turbid), the temperature readings output by baseline method A (surface temperature) are severely interfered with by the superposition of coal dust aerosol absorption and ambient background thermal radiation, resulting in an extremely nonlinear drop that completely deviates from the actual equipment temperature. In contrast, the proposed solution (reverse compensation) utilizes a reconstructed virtual instantaneous transmittance matrix to perform precise radiation attenuation reverse compensation calculations pixel by pixel, ensuring that the output temperature readings always closely match the physical temperature limits. The introduction of this mechanism perfectly validates the experimental result in the table above, where the average absolute temperature measurement error decreased from 14.6℃ to 1.5℃, effectively avoiding underreporting of equipment overheating caused by high dust levels.

[0091] Based on the experimental data in the table above, it can be seen that during the inspection process, when the environment encounters a sudden change in dense dust, the confidence weights of the multimodal fusion undergo adaptive redistribution. The visual modal weights and infrared modal weights, representing optical perception, are dynamically suppressed by the system due to the sharp decrease in transmittance. At the same time, the acoustic modal weights, which have strong anti-dust obstruction characteristics, rapidly expand and take over the main perception decision-making power. This dynamic weight allocation strategy based on environmental transmittance avoids the risk of system blinding caused by the failure of a single optical sensor, explaining the underlying logic that the system false alarm rate (false alarm rate) can be reduced to 2.1% in the table above. Combining the above table with Figure 5It can be seen that in the normal test environment with low dust, the fault diagnosis accuracy of the three methods is similar. However, as the environment deteriorates and enters the extremely high dust obstruction area, the accuracy of baseline method A (traditional photoelectric) drops to 21.4% due to the obstruction of the optical channel, while the accuracy of baseline method B (static mode) barely maintains 68.5% due to the failure to shield the negative interference of the contaminated mode. In stark contrast, the intelligent fusion solution of this invention, with its multi-dimensional collaborative mechanism of underlying hardware triggering, infrared reverse compensation, and dynamic acoustic takeover, still maintains a high recognition rate of 93.2% in the extremely high dust obstruction area. Finally, in the comprehensive average scenario, the intelligent fusion solution of this invention achieves a global fault diagnosis accuracy of 96.8%, fully verifying the robustness and advancement of this invention in the harsh mining environment.

Claims

1. A method for AI-based fault analysis of coal mine inspection machines in dusty, low-light environments, characterized in that, Includes the following steps: Obtain a three-dimensional dust transmittance mesh sequence; The time interval is determined based on the three-dimensional spatial dust transmittance grid sequence, and visible light image features and infrared temperature measurement matrix are obtained within the time interval; A virtual instantaneous transmittance matrix is ​​reconstructed based on the three-dimensional dust transmittance grid sequence, and the infrared temperature measurement matrix is ​​compensated using the virtual instantaneous transmittance matrix to obtain a thermodynamic feature matrix. If the average instantaneous transmittance of the virtual instantaneous transmittance matrix is ​​lower than the optical signal-to-noise ratio failure threshold calibrated offline based on the dark area shot noise limit of the visible light camera hardware with global shutter speed, then the acoustic envelope feature vector of the target device is obtained. The dynamic confidence weighting factor is determined based on the average instantaneous transmittance. The visible light image features, the thermal feature matrix, and the acoustic envelope feature vector are fused and calculated based on the dynamic confidence weighting factor to output the mechanical failure probability result.

2. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 1, characterized in that, Obtaining a three-dimensional dust transmittance mesh sequence specifically includes: The system receives raw point cloud data and classifies the scanning points in the raw point cloud data whose echo intensity is greater than or equal to the rigid body reflection intensity threshold determined by benchmark measurement in a dust-free environment as solid phase point cloud, and classifies the scanning points whose echo intensity is less than the rigid body reflection intensity threshold and whose echo order is marked as the first echo as gas phase suspended point cloud. A mapping model is established based on the echo intensity data of the gas-phase suspended point cloud processed by the aerosol lidar equation, and the spatial extinction coefficient is extracted by inversion. The atmospheric transmittance along the detection optical path at the coordinate point is calculated using the spatial extinction coefficient, thereby generating the three-dimensional spatial dust transmittance grid sequence.

3. The AI ​​fault analysis method for coal mine inspection machines in dusty, low-light environments, as described in claim 1, is characterized in that... The time interval is determined based on the three-dimensional dust transmittance grid sequence, specifically including: Extract the transmittance time series from the three-dimensional spatial dust transmittance grid sequence; The transmittance time series is filtered and smoothed using a Kalman filter to obtain the mean transmittance series and the first and second time derivatives corresponding to the mean transmittance series. When the first time derivative is greater than zero and the second time derivative is less than zero, the predicted time point when the first time derivative decays to zero is calculated by extrapolation based on the current derivative value. The time interval is defined by extending forward and backward from the predicted time point. When the absolute value of the predicted transmittance corresponding to the time interval is greater than the minimum optical signal-to-noise ratio threshold calibrated based on the dark current noise parameter and quantum efficiency characteristics of the camera sensor, the corresponding time interval is confirmed as the time interval for performing subsequent acquisition operations.

4. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 1, characterized in that, The process of acquiring visible light image features and infrared thermometry matrix within the stated time interval also includes: When the internal system clock reaches the start time of the time interval, it drives the general-purpose input / output pins to generate a level flip, sending hardwired trigger pulses in parallel to the external trigger pins of the visible light camera and the infrared thermal imager; The hardwired trigger pulse is used to force all pixel units of the visible light camera and the infrared thermal imager to start and end exposure operations simultaneously, thereby obtaining the visible light image features and the infrared temperature measurement matrix. At the moment the exposure cycle ends, a feedback signal is captured via a hardware interrupt. The result is combined with the calibrated exposure duration to calculate and record the global exposure midpoint timestamp.

5. The AI ​​fault analysis method for coal mine inspection machines in dusty, low-light environments according to claim 4, characterized in that, A virtual instantaneous transmittance matrix is ​​reconstructed based on the three-dimensional dust transmittance grid sequence. This virtual instantaneous transmittance matrix is ​​then used to compensate the infrared thermometry matrix to obtain a thermal characteristic matrix, specifically including: Based on the global exposure midpoint timestamp, obtain the three-dimensional spatial dust transmittance grid sequence of two frames that are immediately before and after the global exposure midpoint timestamp in time; Perform temporal linear interpolation on the two frames of the three-dimensional spatial dust transmittance grid sequence to generate the virtual instantaneous transmittance matrix aligned with the exposure instant; By combining the pre-calibrated rigid transformation matrix and optical intrinsic parameter matrix, the cumulative attenuation value corresponding to the ray penetrating the grid voxel in the virtual instantaneous transmittance matrix is ​​extracted using the line-of-sight ray tracing algorithm, and the two-dimensional view projection transmittance matrix is ​​obtained. By combining the obtained ambient background radiation temperature, and based on the modified Stefan-Boltzmann law, the infrared temperature measurement matrix is ​​subjected to inverse radiation attenuation compensation pixel by pixel using the two-dimensional perspective projection transmittance matrix to obtain the thermodynamic feature matrix that restores the actual surface temperature of the device.

6. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 1, characterized in that, Obtaining the acoustic envelope feature vector of the target device specifically includes: Obtain a solid point cloud of the target environment, perform a density-based spatial clustering algorithm on the solid point cloud, and extract the geometric absolute center coordinates of the point cloud clusters; The rigid body spatial pose transformation is performed by calling the pre-calibrated rigid translation and rotation transformation matrix, and the geometric absolute center coordinates are mapped to the local acoustic coordinate system to obtain the spatial absolute center coordinates. The acoustic envelope feature vector is extracted based on the absolute center coordinates of the space.

7. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 6, characterized in that, The acoustic envelope feature vector is extracted based on the absolute center coordinates of the space, specifically including: Extract the known hardware coordinates of each microphone channel and calculate the actual time-varying speed of sound based on the ambient background radiation temperature positive compensation. Using the absolute center coordinates of the space, the hardware coordinates, and the time-varying sound speed of the actual environment, the delay time is calculated based on the sound wave transmission path length to obtain the phase delay compensation amount for each channel. The original acoustic array signal is shifted and aligned along the time axis according to the phase delay compensation amount, and a weighted summation coherence operation is performed to synthesize a one-dimensional time master signal. A fast Fourier transform is performed on the one-dimensional time master signal to extract the frequency domain amplitude envelope information, thereby generating the acoustic envelope feature vector.

8. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 1, characterized in that, The dynamic confidence weighting factor is determined based on the average instantaneous transmittance, specifically including: Based on the value of the average instantaneous transmittance, the visual basic confidence score corresponding to the visible light image feature and the infrared basic confidence score corresponding to the infrared thermometry matrix are calculated respectively. Assign an acoustic fundamental confidence score to the data extraction modality that is constant for the acoustic envelope feature vector; The visual baseline confidence score, the infrared baseline confidence score, and the acoustic baseline confidence score are redistributed using a normalized exponential function that includes a temperature parameter to obtain the normalized dynamic confidence weight factor.

9. The AI ​​fault analysis method for coal mine inspection machines in dusty and low-light environments according to claim 1, characterized in that, Before fusing the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector based on the dynamic confidence weighting factor, the following alignment step is also included: Multiple feature extraction neural network branches are invoked in parallel to map the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector into visual feature vectors, infrared feature vectors, and acoustic feature vectors of a unified dimension, respectively. When the acoustic envelope feature vector is not extracted, the acoustic feature vector is assigned a zero vector that matches the dimension of the feature latent space to maintain tensor alignment of the fused input structure.

10. The AI ​​fault analysis method for coal mine inspection machines in dusty, low-light environments according to claim 9, characterized in that, Based on the dynamic confidence weighting factor, the visible light image features, the thermal feature matrix, and the acoustic envelope feature vector are fused and calculated to output the mechanical failure probability result, specifically including: Using the dynamic confidence weighting factor, an element-wise algebraic weighted summation operation is performed on the visual feature vector, the infrared feature vector, and the acoustic feature vector to obtain a multimodal fusion feature vector; The multimodal fusion feature vector is input into a multilayer perceptron classifier for dimensionality reduction mapping, and the output is a probability distribution vector corresponding to the mechanical fault category in the system. The probability distribution vector of multiple consecutive diagnostic cycles is smoothed by multi-frame probability mean processing, and the term with the largest weighted probability is output as the mechanical fault probability result.

Images