Unmanned mine car monitoring and early warning method, system, equipment and medium

Abstract

The invention provides an unmanned mine car monitoring and early warning method, system and device and a medium, and belongs to the technical field of unmanned mine car monitoring. According to the method, multi-source heterogeneous data of an unmanned mine car, an excavator and a mining area environment are collected in real time, time synchronization, space registration and fusion processing are carried out, unified sensing data are generated, and key information such as the position posture, the load and the working state of equipment is extracted; on the basis, updating a mining area two-dimensional map in real time, constructing operation scene data, and generating a congestion, risk and efficiency dynamic thermodynamic diagram; and finally, superposing and rendering the map, the thermodynamic diagram and the real-time video stream to form a visual visual interface, and realizing predictive early warning of behavior abnormality, equipment failure, environmental risk and scheduling efficiency based on an intelligent analysis model. According to the method, the problems of data fusion deficiency, visualization capability limitation, dynamic perception insufficiency, early warning mechanism lag and the like in the prior art are solved, and accurate perception and intelligent early warning of the mining area operation global situation are realized.

Description

Technical Field

[0001] This invention belongs to the field of unmanned mining truck monitoring technology, and more specifically relates to a method, system, equipment and medium for monitoring and early warning of unmanned mining trucks. Background Technology

[0002] Traditional mining transportation operations have long relied on manual driving, resulting in inherent drawbacks such as low production efficiency, high safety risks, and harsh working environments. While the introduction of autonomous driving technology has alleviated these problems to some extent, the monitoring and intervention system, which is the core of management, remains underdeveloped. Its shortcomings have become a key bottleneck restricting the overall intelligent upgrading of mining areas. Existing systems often only provide preliminary positioning and status display for unmanned mining vehicles, failing to fundamentally change the passive, lagging, and one-sided monitoring mode.

[0003] Currently, most unmanned mining truck monitoring systems face severe challenges at the data layer. The massive amounts of data generated by various sensors (such as GNSS, LiDAR, and cameras) are often independent, forming "data silos" and lacking an effective real-time multi-source heterogeneous fusion mechanism. This results in the system's inability to generate comprehensive and accurate situational awareness of the vehicle's own status and its surrounding environment, leading to a weak decision-making foundation. Simultaneously, in terms of information presentation, existing systems suffer from severely inadequate visualization capabilities, mostly limited to two-dimensional planar views or simple trajectory playback, making it difficult to intuitively and three-dimensionally recreate the real-time dynamics of complex mining areas, hindering supervisors from quickly grasping the overall operational situation.

[0004] Furthermore, existing systems have significant shortcomings in dynamic situation perception and early warning. The systems typically cannot calculate and present key dynamic information such as vehicle density, traffic pressure, congestion hotspots, and potential risk areas within the mining area in real time. This makes scheduling decisions still highly dependent on the personal experience of management personnel, resulting in low efficiency and a lack of scientific rigor. In addition, mainstream early warning mechanisms mostly adopt rule-triggered modes based on single thresholds, lacking the ability to perform multi-dimensional data correlation analysis. This leads to their inability to effectively identify complex behavioral anomalies, predict potential equipment failures, or perceive environmental risks, resulting in a prominent problem of delayed early warning. Summary of the Invention

[0005] To address the above problems, the present invention aims to provide a method, system, device, and medium for monitoring and early warning of unmanned mining trucks. Through the comprehensive application of deep fusion of multi-source heterogeneous data, overlay rendering of real-time two-dimensional maps and dynamic heat maps, and intelligent early warning models, it achieves comprehensive and accurate perception of the mining area's operational status, enhances predictive early warning capabilities, optimizes overall operational efficiency, and significantly improves the level of safety risk management.

[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a method for monitoring and early warning of unmanned mining trucks, including: Real-time data collection of operational data from mining equipment and mining environment data generates multi-source heterogeneous data; the operational equipment includes unmanned mining trucks and unmanned excavators. Multi-source heterogeneous data is synchronized in time, registered in space, and fused to generate unified sensing data. Key information is then extracted based on the unified sensing data. The key information includes the real-time position and attitude information, load information, and operation status information of the operating equipment. Based on the real-time location and attitude information of the operating equipment, the two-dimensional map of the mining area is updated in real time, and the load information and operation status information are mapped to the corresponding operating equipment location to form real-time operation scene data of the mining area. Based on key information and real-time operation scenario data in the mining area, at least one dynamic heat map is generated to form dynamic heat map data; Based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video stream, a two-dimensional visualization interface is generated through overlay rendering. Based on unified sensing data, key information, and dynamic heat map data, intelligent analysis is performed to identify abnormal patterns and trigger predictive warnings.

[0007] In one optional implementation, the multi-source heterogeneous data includes: GNSS positioning data, inertial measurement unit data, lidar point cloud data, visual image data, CAN bus device status data, equipment operation sensor data, loading and unloading operation status data, and edge environment sensor data; wherein, the CAN bus device status data includes speed, load, fuel consumption, and component temperature; the equipment operation sensor data includes boom angle sensor data and bucket capacity sensor data of unmanned excavators, as well as on-board weighing sensor data and hydraulic suspension system pressure sensor data of unmanned mining trucks; and the edge environment sensor data includes meteorological data and road condition data.

[0008] In an optional implementation, the process of time synchronization, spatial registration, and fusion of multi-source heterogeneous data to generate unified sensing data, and the extraction of key information based on the unified sensing data, includes the real-time position and attitude information, load information, and operational status information of the operating equipment, including: By employing extended Kalman filtering or particle filtering algorithms, and fusing GNSS positioning data, inertial measurement unit data, and local positioning results generated by synchronous positioning and mapping algorithms based on lidar point cloud data or visual image data, the global precise position and attitude angle of the working equipment are calculated, and real-time position and attitude information of the working equipment is generated. Based on data from onboard weighing sensors or hydraulic suspension system pressure sensors, the real-time load capacity of the unmanned mining truck is estimated through data filtering and pre-calibration models, generating load information. By analyzing CAN bus device status data and equipment operation sensor data, and using state machine-based logical judgment or machine learning classifiers, the operating mode of the operating equipment in real time, whether it is driving, loading, unloading or idling, is identified, and operation status information is generated.

[0009] In an optional implementation, the step of updating the two-dimensional map of the mining area in real time based on the real-time position and attitude information of the operating equipment, and mapping the load information and operation status information to the corresponding operating equipment positions to form real-time operation scene data of the mining area, includes: Based on the real-time location and attitude information of the operating equipment, the global coordinates (E, N) of the operating equipment are generated; where E is the eastward coordinate value and N is the northward coordinate value; through the geographic information system coordinate transformation algorithm, the global coordinates (E, N) are mapped to the plane coordinate system (x, y) of the high-precision two-dimensional map; Integrate load information and operation status information, and bind them as attribute data to the corresponding operation equipment location on the map; Using incremental real-time rendering technology, the display of the two-dimensional map is dynamically refreshed based on the updated location and attribute data of the operating equipment, so as to form and output real-time operating scene data of the mining area.

[0010] In an optional implementation, the step of generating at least one dynamic heatmap based on key information and real-time mining operation scenario data to form dynamic heatmap data includes: Generate at least one dynamic heatmap from key information and real-time operation scenario data in the mining area, including congestion heatmap, risk heatmap, and efficiency heatmap. The process of generating the congestion heatmap includes: Based on the position and velocity data in the real-time position and attitude information of the working equipment, a grid is divided on the two-dimensional map, and the size of the grid is Δx×Δy; The real-time device density ρ within each grid is calculated using the formula, where ρ = N. device / (Δx×Δy), N device This represents the total number of operating devices within the grid, based on location data statistics. For each grid, the speed data of each operating device is recorded as being lower than the preset congestion speed threshold v. threshold Cumulative stagnation time T stop ; Using real-time device density ρ and cumulative dwell time T stop As a core congestion indicator, a spatial distribution map is generated using a kernel density estimation algorithm, serving as a congestion heat map. The process of generating the risk heatmap includes: Based on the real-time position and attitude information and load information of the working equipment, the real-time relative distance and relative velocity between the working equipment are calculated through a kinematic model, and then the potential collision risk index R is calculated. collision ; Real-time acquisition of equipment fault warning level L from predictive early warning information fault The road slipperiness W is calculated based on environmental data and vehicle motion data. slip ; Through formula R total =wc×R collision +wf×L fault +ws×W slip Calculate the comprehensive risk index R total Where wc, wf, and ws are the preset weights for collision risk, equipment failure, and slippery road conditions, respectively. R total The risk heatmap is generated by converting the color map into a color gradient from low risk to high risk and then performing spatial rendering. The process of generating the efficiency heatmap includes: Based on the job status information, obtain the waiting time T of the work equipment at each loading and unloading point. wait ; Based on the work cycle duration from load information and work status information, the loading or transportation volume per unit time at each work point is calculated as an efficiency indicator. ; The waiting time T wait or efficiency indicators As a core efficiency indicator, it is spatially visualized by color mapping to the corresponding geographical location, generating an efficiency heatmap.

[0011] In an optional implementation, the step of overlaying and rendering based on real-time mining operation scene data and dynamic heat map data, combined with real-time video streams, to generate a two-dimensional visualization interface includes: By using image overlay and transparency blending technology, real-time operation scene data, dynamic heat map data, and real-time video streams of operating equipment in the mining area are overlaid and rendered in real time to generate a two-dimensional visualization interface that supports multi-view switching, zooming, and roaming interactive operations.

[0012] In an optional implementation, the step of performing intelligent analysis based on unified sensing data, key information, and dynamic heatmap data to identify abnormal patterns and trigger predictive warnings includes: Real-time sensing data from the unified sensing data is input into a time-series anomaly detection model built on an LSTM autoencoder. The time-series anomaly detection model uses historical normal operation data to construct a baseline of normal behavior and outputs an anomaly score S. When the anomaly score S exceeds a preset threshold... When this occurs, an abnormal behavior warning is triggered; Historical time-series data of key component sensors in the unified sensing data are input into a deep learning time-series prediction model built based on a long short-term memory network or a Transformer model. This deep learning time-series prediction model is then used to predict future trends in sensor data and apply formulas. Calculate the failure probability P fault When P fault Exceeding the preset threshold When this occurs, a device fault warning is triggered; in, This represents the hidden state of a deep learning time series prediction model at time t. Let be the weight matrix from the hidden state to the output layer. For the bias term of the output layer, The activation function is used; the historical time-series data of the key component sensor includes temperature T(t), vibration V(t), and pressure P(t); Using environmental feature data from unified sensing data as input, a classification prediction model is used to calculate the probability P of a specific environmental risk occurring. slide When P slide Exceeding the preset threshold When this occurs, an environmental risk warning is triggered; the probability P... slide Using the logistic regression formula Calculate, where, The i-th environmental feature input variable includes rainfall R, slope G, and soil moisture H. This represents the regression coefficient corresponding to the i-th environmental feature input variable. The intercept term represents the logistic regression model; Obtain the waiting time T of the equipment at each loading and unloading point. wait When T wait Exceeding the preset scheduling efficiency threshold At that time, a scheduling efficiency warning is triggered.

[0013] Secondly, this application also provides an unmanned mining truck monitoring and early warning system, including: a multi-source data acquisition module, a multi-modal data fusion processing module, a map update module, a dynamic heat map generation module, a two-dimensional visualization rendering module, and an intelligent analysis and early warning module; A multi-source data acquisition module is used to collect real-time operating data of mining equipment and mining environment data, generating multi-source heterogeneous data; the operating equipment includes unmanned mining trucks and unmanned excavators. The multimodal data fusion processing module is used to perform time synchronization, spatial registration and fusion processing on multi-source heterogeneous data to generate unified perception data, and extract key information based on the unified perception data; the key information includes the real-time position and attitude information, load information and operation status information of the operating equipment. The map update module is used to update the two-dimensional map of the mining area in real time based on the real-time location and attitude information of the operating equipment, and to map the load information and operation status information to the corresponding operating equipment location to form real-time operation scene data of the mining area. The dynamic heat map generation module is used to generate at least one dynamic heat map based on key information and real-time operation scenario data in the mining area, thus forming dynamic heat map data. The 2D visualization rendering module is used to overlay and render data based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video streams, to generate a 2D visualization interface. The intelligent analysis and early warning module is used to perform intelligent analysis based on unified sensing data, key information, and dynamic heat map data, identify abnormal patterns, and trigger predictive early warnings.

[0014] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the unmanned mining truck monitoring and early warning method described in any of the above descriptions.

[0015] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the unmanned mining truck monitoring and early warning method described in any of the above embodiments.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: The unmanned mining truck monitoring and early warning method provided in this application ensures the comprehensiveness and accuracy of the perceived data through multi-source heterogeneous data acquisition and fusion processing. The map update and scene construction realize the intuitive presentation of the mining area operation status. Combined with multi-type dynamic heat maps, it reflects key scene characteristics such as congestion, risk, and efficiency in multiple dimensions. The overlaid and rendered two-dimensional visualization interface improves the intuitiveness and interactivity of monitoring. Furthermore, through intelligent analysis methods such as time-series anomaly detection and deep learning prediction, it realizes predictive early warning of anomalies such as behavioral anomalies, equipment failures, environmental risks, and scheduling efficiency. Overall, it significantly improves the accuracy of unmanned mining operation monitoring and the timeliness of early warning, effectively ensuring operational safety and helping to optimize operational efficiency.

[0017] This application collects multi-source heterogeneous data from equipment such as unmanned mining trucks and excavators, and constructs a complete processing system that integrates time synchronization, spatial registration, and filtering fusion. It efficiently integrates diverse information such as positioning, attitude, load, equipment status, and environment to generate comprehensive and high-precision unified perception data, providing solid and reliable data support for subsequent monitoring and early warning.

[0018] This application dynamically updates the two-dimensional map of the mining area based on the real-time position and attitude of the equipment, binds attributes such as load and operation status to the equipment position and refreshes them in real time. The constructed real-time operation scene data can intuitively and three-dimensionally restore the dynamics of the complex mining area, allowing monitoring personnel to fully grasp the overall operation layout and equipment operating conditions, forming an active, real-time and comprehensive monitoring mode.

[0019] This application generates dynamic heat maps of various types, including congestion, risk, and efficiency, transforming abstract data such as equipment density, collision risk, and operational efficiency into visualized spatial distribution features. This helps monitoring personnel quickly locate congested road sections, high-risk areas, and inefficient work sites, providing data support for scheduling decisions and improving the scientific nature and efficiency of scheduling.

[0020] This application integrates real-time mining operation scene data, dynamic heat map data, and real-time video streams through overlay rendering technology to generate a two-dimensional visualization interface that supports multi-view switching, zooming, and roaming. This allows monitoring personnel to flexibly interact and view details of the target area, greatly improving the convenience of monitoring operations and the efficiency of overall situational awareness.

[0021] This application utilizes intelligent models such as time-series anomaly detection and deep learning prediction, combined with unified perception data, key information, and dynamic heat map data, to achieve multi-dimensional correlation analysis and predictive early warning of behavioral anomalies, equipment failures, environmental risks, and scheduling efficiency. This enables early avoidance of safety risks such as collisions and equipment failures, while optimizing work processes through scheduling efficiency early warning, thus facilitating the intelligent upgrading of mining areas. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating the unmanned mining truck monitoring and early warning method provided in this application.

[0024] Figure 2 This is a schematic diagram of the unmanned mining truck monitoring and early warning system provided in this application.

[0025] Figure 3A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] The various embodiments of this disclosure will be described more fully in the following detailed description of the specific steps of the unmanned mining truck monitoring and early warning method. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0027] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 The diagram shown is a flowchart of a method for monitoring and early warning of unmanned mining trucks in a specific embodiment. The method includes: S1: Real-time collection of operating data of mining equipment and mining environment data to generate multi-source heterogeneous data; the operating equipment includes unmanned mining trucks and unmanned excavators.

[0030] In this specific implementation, the core task of this step is to construct a perception network covering the entire mining area, providing a comprehensive and real-time data foundation for subsequent processing. First, a hardware perception layer is deployed in the mining area, including GNSS base stations, inertial measurement units, lidar, vision cameras, and roadside environmental sensors. Unmanned mining trucks and unmanned excavators continuously output CAN bus device status data, including speed, load, fuel consumption, and component temperature, through the onboard network system.

[0031] In practice, GNSS positioning equipment provides real-time positioning data with centimeter-level accuracy; inertial measurement units collect motion data at frequencies above 100Hz to compensate for short-term drift of GNSS signals; lidar constructs a precise point cloud model of the surrounding environment by emitting and receiving laser beams; and visual imaging equipment captures rich texture information for environmental perception. Specifically, equipment operation sensor data includes boom angle sensor data and bucket capacity sensor data for unmanned excavators, as well as onboard weighing sensor data and hydraulic suspension system pressure sensor data for unmanned mining trucks. Simultaneously, edge environment sensors collect real-time meteorological and road condition data, forming a complete environmental perception capability.

[0032] All sensor data is transmitted to the central processing system via a high-speed industrial network, forming a unified multi-source heterogeneous dataset. During data transmission, a timestamp synchronization mechanism is used to ensure that all types of data have a unified time reference during subsequent processing.

[0033] S2: Perform time synchronization, spatial registration and fusion processing on multi-source heterogeneous data to generate unified perception data, and extract key information based on the unified perception data; the key information includes the real-time position and attitude information of the operating equipment, load information and operating status information.

[0034] In a specific implementation, the multi-source heterogeneous data is first synchronized in time and registered in space to eliminate the spatiotemporal differences between sensors and form unified sensing data.

[0035] For real-time position and attitude extraction, an extended Kalman filter algorithm is employed, fusing GNSS positioning data, inertial measurement unit (IMU) data, and local positioning results generated by simultaneous positioning and mapping algorithms based on lidar point cloud data or visual image data. This algorithm uses the absolute position provided by GNSS as the observed value and the relative displacement calculated by the IMU as the predicted value, effectively correcting positioning errors in signal-blocked areas, and ultimately outputting the precise global position and attitude angles of the operating equipment.

[0036] In terms of load information extraction, based on data from vehicle-mounted weighing sensors or hydraulic suspension system pressure sensors, median filtering is first used to remove signal noise. Then, a pre-calibrated load-sensor signal model is used to convert the physical signal into an accurate load value. Specifically, a mapping relationship is established between pressure sensor readings and load tonnage: M load =k×P+b, where k is the calibration coefficient, P is the sensor reading, and b is the offset.

[0037] For operation status identification, the system analyzes CAN bus device status data and equipment operation sensor data, employing a logic judgment method based on a finite state machine. The system monitors parameters such as vehicle speed, excavator boom angle, and load changes in real time. Through preset state transition conditions, it accurately identifies the operating mode of the equipment: driving, loading, unloading, or idling. For example, when the vehicle speed is detected to be 0, the excavator boom angle is within the working range, and the load continues to increase, the current state is determined to be "loading" operation.

[0038] S3: Based on the real-time location and attitude information of the operating equipment, update the two-dimensional map of the mining area in real time, and map the load information and operation status information to the corresponding operating equipment location to form real-time operation scene data of the mining area.

[0039] In a specific implementation, the goal of this step is to create a digital operational scenario that reflects the real-time dynamics of the mining area. During implementation, a high-precision two-dimensional map based on a geographic information system is maintained, and real-time location and attitude information of the operating equipment from step S2 is received, including the eastward coordinate E and the northward coordinate N.

[0040] A geographic information system (GIS) coordinate transformation algorithm is used to map the global coordinates (E, N) onto the planar coordinate system (x, y) of a high-precision 2D map. In practice, the UTM coordinate transformation method is employed to ensure accurate mapping of location information. Simultaneously, the extracted load information and operational status information are used as attribute data and dynamically bound to the corresponding operational equipment locations on the map.

[0041] In the visualization update phase, incremental real-time rendering technology is employed. Only equipment elements whose position or status changes are redrawn; for example, when the location of a mining truck is updated or its operational status changes, the corresponding map display is dynamically refreshed. This approach significantly improves rendering efficiency, ensures the system's real-time performance in large-scale mining scenarios, and ultimately outputs real-time mining operation scene data containing complete equipment information and status data.

[0042] S4: Based on key information and real-time operation scenario data in the mining area, generate at least one dynamic heat map to form dynamic heat map data.

[0043] In a specific implementation, this step transforms abstract operational data into an intuitive, visualized situation map. During implementation, based on key information and real-time operational scenario data of the mining area, at least one dynamic heat map is generated, including a congestion heat map, a risk heat map, and an efficiency heat map.

[0044] During the generation of the congestion heatmap, based on the position and velocity data from the real-time location and attitude information of the operating equipment, a grid of size Δx×Δy is divided on the two-dimensional map. This is achieved using the formula ρ=N. device / (Δx×Δy) calculates the real-time device density for each grid, where Ndevice This represents the total number of operating devices within a grid, based on location data. Additionally, for each grid, the speed data of each operating device is recorded as being below a preset congestion speed threshold v. threshold Cumulative stagnation time T stop Finally, the real-time device density ρ and cumulative dwell time T are used as the basis for the calculation. stop As a core congestion indicator, a spatial distribution map is generated using a kernel density estimation algorithm based on a Gaussian kernel function.

[0045] During the generation of the risk heatmap, based on the real-time position and attitude information and load information of the operating equipment, the real-time relative distance and relative velocity between the equipment are calculated through a kinematic model, and then the potential collision risk index R is calculated. collision Simultaneously, the equipment fault warning level L in the predictive early warning information is acquired in real time. fault The degree of road slipperiness W is calculated based on environmental data and vehicle motion data. slip Through formula R total =wc×R collision +wf×L fault +ws×W slip A comprehensive risk index is calculated, where wc, wf, and ws are preset weights for collision risk, equipment failure, and slippery road conditions, respectively. These weighting coefficients are adjusted and optimized based on the actual working environment. Finally, R... total Spatial rendering is performed by converting color mapping into a color gradient from low risk to high risk.

[0046] During the generation of the efficiency heatmap, the waiting time T for each work point is obtained based on the work status information. wait Simultaneously, based on the load information and the operation cycle duration in the operation status information, the loading or transportation volume per unit time at each operation point is calculated as an efficiency indicator. Waiting time T wait or efficiency indicators As a core efficiency indicator, color mapping is used to visualize the spatial distribution of mining area operation efficiency in relation to the corresponding geographical location, providing an intuitive view of the distribution of operational efficiency.

[0047] S5: Based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video stream, it performs overlay rendering to generate a two-dimensional visualization interface.

[0048] In a specific implementation, this step is a crucial step in achieving comprehensive information display. During implementation, image overlay and transparency blending techniques are used to integrate and render real-time mining operation scene data, dynamic heat map data, and real-time video streams from the operating equipment.

[0049] In practice, a high-performance graphics rendering engine based on WebGL is used. Real-time mining operation scene data is used as the base map, dynamic heat map data is used as a semi-transparent overlay, and real-time video stream windows are superimposed at key locations. Through precise alpha blending control, it is ensured that each information layer is both clearly distinguishable and coordinated.

[0050] This method also provides complete interactive operation functions, supporting multi-view switching, map zooming, and scene roaming. Users can flexibly adjust the displayed content through the graphical interface, such as switching between different types of dynamic heat maps or focusing on viewing the operation details of a specific area. This implementation provides users with a high-fidelity, immersive monitoring experience, significantly improving situational awareness capabilities.

[0051] For example, a high-performance rendering engine is built using the WebGL graphics library. Through image overlay and alpha blending techniques, real-time mining scene data is used as the base layer, dynamic heatmap data as a semi-transparent overlay, and real-time video stream as a floating window for multi-layered fusion rendering. Specifically, the heatmap layer's transparency is set to 40% to ensure the underlying map information is visible while highlighting situational information. The real-time video stream is displayed in picture-in-picture format next to the corresponding device location, supporting dynamic dragging and zooming. The system provides rich interactive functions, including mouse wheel zooming, drag-and-drop navigation, and view switching. Users can quickly locate and view detailed information by double-clicking device elements, creating an immersive monitoring experience.

[0052] S6: Based on unified sensing data, key information, and dynamic heat map data, perform intelligent analysis to identify abnormal patterns and trigger predictive warnings.

[0053] In specific implementations, this step is the core of the intelligent decision-making process, achieving a leap from passive monitoring to proactive early warning through multi-model fusion analysis. A modular analysis architecture is adopted to deeply mine unified sensing data, key information, and dynamic heatmap data, establishing a four-layer early warning mechanism.

[0054] The behavioral anomaly warning system employs a deep learning-based temporal anomaly detection framework. In practice, a training dataset is first constructed: three months of normal mining truck operation data are collected, including speed sequences v(t), acceleration a(t), steering angle θ(t), and position coordinates (E, N), forming multi-dimensional temporal samples. An LSTM autoencoder is used for unsupervised training; the encoder compresses the input sequence into latent features, and the decoder attempts to reconstruct the original sequence. After training, online detection is performed, with real-time perceived data input into the model via a sliding window, and the reconstruction error is calculated as the anomaly score S. When abnormal driving behavior occurs, such as deviation from the preset path, frequent speeding, or sudden braking, the reconstruction error increases significantly. Based on a set dynamic threshold... When S> In addition to triggering warnings, the system also uses real-time location information to pinpoint abnormal devices and categorizes anomalies based on error patterns. Simultaneously, it records abnormal segments for continuous model optimization and driver behavior analysis.

[0055] Equipment failure early warning is based on predictive maintenance principles and employs a multivariate time-series prediction model based on LSTM. Taking key components such as engines, hydraulic systems, and braking systems as examples, it continuously monitors sensor data such as temperature T(t), vibration V(t), and pressure P(t). Data preprocessing is performed first, including outlier removal, sequence alignment, and feature normalization. The model uses an encoder-decoder architecture, where the encoder learns the hidden states of historical sequences. The decoder is based on Predict sensor readings for multiple future time steps. Calculate the degree of deviation by comparing the predicted values ​​with the actual observed values, and then use the formula... Mapped to failure probability, where This is the Sigmoid activation function. When... Exceeding the threshold In addition to triggering an early warning, the system will also combine information such as equipment runtime and maintenance records to assess the remaining useful life (RUL) and generate a tiered warning: Attention Level (0.3≤ <0.6), Warning level (0.6≤ <0.8), Hazard level ( (≥0.8). Simultaneously, it automatically retrieves the equipment's historical maintenance records and similar equipment failure modes to support maintenance decisions.

[0056] Environmental risk early warning employs a statistical learning model that integrates multi-source information. Taking landslide early warning as an example, it integrates multi-dimensional features such as real-time rainfall R, slope gradient G, soil moisture H, and geological structure data. First, feature engineering is performed, including numerical standardization, feature cross-validation, and temporal feature extraction. A logistic regression model is then used to establish a risk probability calculation framework. .in, The i-th environmental feature input variable includes rainfall R, slope G, and soil moisture H. This represents the regression coefficient corresponding to the i-th environmental feature input variable. This represents the intercept term of the logistic regression model; the model weight coefficients are obtained through training on historical risk data and are periodically updated with the latest data. In implementation, the system establishes a multi-level early warning mechanism: when... A blue alert is issued when the value is in the range [0.3, 0.6), and it is recommended to strengthen patrols. A yellow alert is issued when the value is in the range [0.6, 0.8), and it is recommended to restrict operations in the surrounding area. A red alert is issued when the temperature is ≥ 0.8, and personnel and equipment are immediately evacuated. Simultaneously, by combining meteorological forecast data, predictive warnings can be issued several hours in advance, allowing sufficient time for emergency decision-making.

[0057] Dispatch efficiency early warning is based on real-time performance indicator monitoring and trend analysis. It continuously tracks the waiting time T at each loading / unloading point through operation status information. wait Key performance indicators such as equipment utilization rate and task completion rate. Establish a dynamic threshold mechanism, including scheduling efficiency thresholds. It adaptively adjusts based on historical efficiency data, equipment quantity, production plan, and other factors. When T wait Exceed When the time comes, a scheduling efficiency warning is triggered, and root cause analysis is automatically performed: by analyzing the equipment distribution heat map, congestion points are identified, the impact range of faulty equipment is checked, and the balance of task allocation is assessed.

[0058] In this embodiment, by constructing a multi-source heterogeneous data acquisition network and a multi-level fusion processing architecture, a comprehensive and accurate perception of the mine operation status is achieved; by using dynamic heat maps and multi-dimensional data overlay rendering technology, the visualization and monitoring capabilities in complex environments are significantly improved; and based on deep learning and predictive analysis models, an intelligent early warning system covering behavior, equipment, environment, and scheduling efficiency is constructed, thereby fundamentally improving the safety, efficiency, and intelligent management level of mine transportation operations.

[0059] like Figure 2 As shown, the following are embodiments of the unmanned mining truck monitoring and early warning system provided in this disclosure. This system and the unmanned mining truck monitoring and early warning methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the unmanned mining truck monitoring and early warning system, please refer to the embodiments of the unmanned mining truck monitoring and early warning methods described above.

[0060] An unmanned mining truck monitoring and early warning system includes: a multi-source data acquisition module, a multi-modal data fusion and processing module, a map update module, a dynamic heat map generation module, a two-dimensional visualization rendering module, and an intelligent analysis and early warning module.

[0061] The multi-source data acquisition module is used to collect real-time operating data of mining equipment and mining environment data, and generate multi-source heterogeneous data; the operating equipment includes unmanned mining trucks and unmanned excavators.

[0062] The multimodal data fusion processing module is used to perform time synchronization, spatial registration and fusion processing on multi-source heterogeneous data to generate unified perception data, and extract key information based on the unified perception data; the key information includes the real-time position and attitude information of the operating equipment, load information and operating status information.

[0063] The map update module is used to update the two-dimensional map of the mining area in real time based on the real-time location and attitude information of the operating equipment, and to map the load information and operation status information to the corresponding operating equipment location to form real-time operation scene data of the mining area.

[0064] The dynamic heat map generation module is used to generate at least one dynamic heat map based on key information and real-time operation scenario data in the mining area, thus forming dynamic heat map data.

[0065] The 2D visualization rendering module is used to overlay and render data based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video streams, to generate a 2D visualization interface.

[0066] The intelligent analysis and early warning module is used to perform intelligent analysis based on unified sensing data, key information, and dynamic heat map data, identify abnormal patterns, and trigger predictive early warnings.

[0067] The unmanned mining truck monitoring and early warning system provided in this embodiment generates high-precision unified perception data through time synchronization, spatial registration, and filtering fusion of multi-source heterogeneous data. It dynamically updates two-dimensional maps based on equipment position and attitude and binds attributes to construct real-time operation scenarios, generates multiple types of dynamic heat maps to achieve multi-dimensional feature visualization, and constructs a flexible and interactive visualization interface by combining overlay rendering technology. Furthermore, it conducts multi-dimensional correlation analysis through intelligent models such as time-series anomaly detection and deep learning prediction to achieve predictive early warning. It comprehensively achieves the goals of accurate data support, global situation control, scientific scheduling decisions, and efficient monitoring interaction, effectively improving operation safety and efficiency, and providing core technical support for the intelligent upgrading of mining areas.

[0068] Figure 3 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0069] The unmanned mining truck monitoring and early warning method provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0070] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0071] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0072] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0073] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0074] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0075] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0076] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0077] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0078] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0079] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0080] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0081] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0082] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0083] The aforementioned electronic equipment enables the unmanned mining truck monitoring and early warning method of this application to achieve real-time scene construction and dynamic heat map generation through time synchronization, spatial registration and filtering fusion of multi-source heterogeneous data, as well as intelligent model analysis and early warning such as overlay rendering visualization, time-series anomaly detection, and deep learning prediction. This achieves the beneficial effects of more accurate data support, controllable global situation, scientific scheduling decisions, and efficient monitoring interaction, while simultaneously improving both operational safety and efficiency, and providing core support for the intelligent upgrading of mining areas.

[0084] The storage medium provided in this application stores a program product capable of implementing a monitoring and early warning method for unmanned mining trucks.

[0085] Monitoring and early warning methods for unmanned mining trucks include: Real-time data collection of operational data from mining equipment and mining environment data generates multi-source heterogeneous data; the operational equipment includes unmanned mining trucks and unmanned excavators. Multi-source heterogeneous data is synchronized in time, registered in space, and fused to generate unified sensing data. Key information is then extracted based on the unified sensing data. The key information includes the real-time position and attitude information, load information, and operation status information of the operating equipment. Based on the real-time location and attitude information of the operating equipment, the two-dimensional map of the mining area is updated in real time, and the load information and operation status information are mapped to the corresponding operating equipment location to form real-time operation scene data of the mining area. Based on key information and real-time operation scenario data in the mining area, at least one dynamic heat map is generated to form dynamic heat map data; Based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video stream, a two-dimensional visualization interface is generated through overlay rendering. Based on unified sensing data, key information, and dynamic heat map data, intelligent analysis is performed to identify abnormal patterns and trigger predictive warnings.

[0086] In some possible implementations, the unmanned mining truck monitoring and early warning method of this disclosure can be implemented as a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0087] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0088] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring and early warning of unmanned mining trucks, characterized in that, include: Real-time collection of operational data from mining equipment and mining environment data to generate multi-source heterogeneous data; The operating equipment includes unmanned mining trucks and unmanned excavators; Multi-source heterogeneous data is synchronized in time, registered in space, and fused to generate unified sensing data. Key information is then extracted based on the unified sensing data. The key information includes the real-time position and attitude information, load information, and operation status information of the operating equipment. Based on the real-time location and attitude information of the operating equipment, the two-dimensional map of the mining area is updated in real time, and the load information and operation status information are mapped to the corresponding operating equipment location to form real-time operation scene data of the mining area. Based on key information and real-time operation scenario data in the mining area, at least one dynamic heat map is generated to form dynamic heat map data; Based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video stream, a two-dimensional visualization interface is generated through overlay rendering. Based on unified sensing data, key information, and dynamic heat map data, intelligent analysis is performed to identify abnormal patterns and trigger predictive warnings.

2. The unmanned mining truck monitoring and early warning method according to claim 1, characterized in that, The multi-source heterogeneous data includes: GNSS positioning data, inertial measurement unit data, lidar point cloud data, visual image data, CAN bus device status data, equipment operation sensor data, loading and unloading operation status data, and edge environment sensor data; wherein, the CAN bus device status data includes speed, load, fuel consumption, and component temperature; the equipment operation sensor data includes boom angle sensor data and bucket capacity sensor data of unmanned excavators, as well as vehicle-mounted weighing sensor data and hydraulic suspension system pressure sensor data of unmanned mining trucks; and the edge environment sensor data includes meteorological data and road condition data.

3. The unmanned mining truck monitoring and early warning method according to claim 2, characterized in that, The process involves time synchronization, spatial registration, and fusion of multi-source heterogeneous data to generate unified perception data, and key information is extracted based on the unified perception data. The key information includes the real-time location and attitude information of the operating equipment, load information, and operating status information, including: By employing extended Kalman filtering or particle filtering algorithms, and fusing GNSS positioning data, inertial measurement unit data, and local positioning results generated by synchronous positioning and mapping algorithms based on lidar point cloud data or visual image data, the global precise position and attitude angle of the working equipment are calculated, and real-time position and attitude information of the working equipment is generated. Based on data from onboard weighing sensors or hydraulic suspension system pressure sensors, the real-time load capacity of the unmanned mining truck is estimated through data filtering and pre-calibration models, generating load information. By analyzing CAN bus device status data and equipment operation sensor data, and using state machine-based logical judgment or machine learning classifiers, the operating mode of the operating equipment in real time, whether it is driving, loading, unloading or idling, is identified, and operation status information is generated.

4. The unmanned mining truck monitoring and early warning method according to claim 3, characterized in that, The process involves updating the two-dimensional map of the mining area in real time based on the real-time location and attitude information of the operating equipment, and mapping the load information and operating status information to the corresponding operating equipment locations to form real-time operating scene data of the mining area, including: Based on the real-time location and attitude information of the operating equipment, the global coordinates (E, N) of the operating equipment are generated; where E is the eastward coordinate value and N is the northward coordinate value; through the geographic information system coordinate transformation algorithm, the global coordinates (E, N) are mapped to the plane coordinate system (x, y) of the high-precision two-dimensional map; Integrate load information and operation status information, and bind them as attribute data to the corresponding operation equipment location on the map; Using incremental real-time rendering technology, the display of the two-dimensional map is dynamically refreshed based on the updated location and attribute data of the operating equipment, so as to form and output real-time operating scene data of the mining area.

5. The unmanned mining truck monitoring and early warning method according to claim 4, characterized in that, The process of generating at least one dynamic heatmap based on key information and real-time operational data in the mining area, forming dynamic heatmap data, includes: Generate at least one dynamic heatmap from key information and real-time operation scenario data in the mining area, including congestion heatmap, risk heatmap, and efficiency heatmap. The process of generating the congestion heatmap includes: Based on the position and velocity data in the real-time position and attitude information of the working equipment, a grid is divided on the two-dimensional map, and the size of the grid is Δx×Δy; The real-time device density ρ within each grid is calculated using the formula, where ρ = N. device / (Δx×Δy), N device This represents the total number of operating devices within the grid, based on location data statistics. For each grid, the speed data of each operating device is recorded as being lower than the preset congestion speed threshold v. threshold Cumulative stagnation time T stop ; Using real-time device density ρ and cumulative dwell time T stop As a core congestion indicator, a spatial distribution map is generated using a kernel density estimation algorithm, serving as a congestion heat map. The process of generating the risk heatmap includes: Based on the real-time position and attitude information and load information of the working equipment, the real-time relative distance and relative velocity between the working equipment are calculated through a kinematic model, and then the potential collision risk index R is calculated. collision ; Real-time acquisition of equipment fault warning level L from predictive early warning information fault The road slipperiness W is calculated based on environmental data and vehicle motion data. slip ; Through formula R total =wc×R collision +wf×L fault +ws×W slip Calculate the comprehensive risk index R total Where wc, wf, and ws are the preset weights for collision risk, equipment failure, and slippery road conditions, respectively. R total The risk heatmap is generated by converting the color map into a color gradient from low risk to high risk and then performing spatial rendering. The process of generating the efficiency heatmap includes: Based on the job status information, obtain the waiting time T of the work equipment at each loading and unloading point. wait ; Based on the work cycle duration from load information and work status information, the loading or transportation volume per unit time at each work point is calculated as an efficiency indicator. ; The waiting time T wait or efficiency indicators As a core efficiency indicator, it is spatially visualized by color mapping to the corresponding geographical location, generating an efficiency heatmap.

6. The unmanned mining truck monitoring and early warning method according to claim 5, characterized in that, The method of generating a two-dimensional visualization interface by overlaying and rendering real-time mining operation scene data and dynamic heat map data with real-time video stream includes: By using image overlay and transparency blending technology, real-time operation scene data, dynamic heat map data, and real-time video streams of operating equipment in the mining area are overlaid and rendered in real time to generate a two-dimensional visualization interface that supports multi-view switching, zooming, and roaming interactive operations.

7. The unmanned mining truck monitoring and early warning method according to claim 6, characterized in that, The process of intelligently analyzing unified sensing data, key information, and dynamic heatmap data to identify abnormal patterns and trigger predictive warnings includes: Real-time sensing data from the unified sensing data is input into a time-series anomaly detection model built on an LSTM autoencoder. The time-series anomaly detection model uses historical normal operation data to construct a baseline of normal behavior and outputs an anomaly score S. When the anomaly score S exceeds a preset threshold... When this occurs, an abnormal behavior warning is triggered; Historical time-series data of key component sensors in the unified sensing data are input into a deep learning time-series prediction model built based on a long short-term memory network or a Transformer model. This deep learning time-series prediction model is then used to predict future trends in sensor data and apply formulas. Calculate the failure probability P fault When P fault Exceeding the preset threshold When this occurs, a device fault warning is triggered; in, This represents the hidden state of a deep learning time series prediction model at time t. Let be the weight matrix from the hidden state to the output layer. For the bias term of the output layer, The activation function is used; the historical time-series data of the key component sensor includes temperature T(t), vibration V(t), and pressure P(t); Using environmental feature data from unified sensing data as input, a classification prediction model is used to calculate the probability P of a specific environmental risk occurring. slide When P slide Exceeding the preset threshold When this occurs, an environmental risk warning is triggered; the probability P... slide Using the logistic regression formula Calculate, where, The i-th environmental feature input variable includes rainfall R, slope G, and soil moisture H. This represents the regression coefficient corresponding to the i-th environmental feature input variable. The intercept term represents the logistic regression model; Obtain the waiting time T of the equipment at each loading and unloading point. wait When T wait Exceeding the preset scheduling efficiency threshold At that time, a scheduling efficiency warning is triggered.

8. A monitoring and early warning system for unmanned mining trucks, characterized in that, The system employs the unmanned mining truck monitoring and early warning method as described in any one of claims 1 to 7; The system includes: a multi-source data acquisition module, a multi-modal data fusion and processing module, a map update module, a dynamic heat map generation module, a two-dimensional visualization rendering module, and an intelligent analysis and early warning module; A multi-source data acquisition module is used to collect real-time operating data of mining equipment and mining environment data, generating multi-source heterogeneous data; the operating equipment includes unmanned mining trucks and unmanned excavators. The multimodal data fusion processing module is used to perform time synchronization, spatial registration and fusion processing on multi-source heterogeneous data to generate unified perception data, and extract key information based on the unified perception data; the key information includes the real-time position and attitude information, load information and operation status information of the operating equipment. The map update module is used to update the two-dimensional map of the mining area in real time based on the real-time location and attitude information of the operating equipment, and to map the load information and operation status information to the corresponding operating equipment location to form real-time operation scene data of the mining area. The dynamic heat map generation module is used to generate at least one dynamic heat map based on key information and real-time operation scenario data in the mining area, thus forming dynamic heat map data. The 2D visualization rendering module is used to overlay and render data based on real-time operation scene data and dynamic heat map data in the mining area, combined with real-time video streams, to generate a 2D visualization interface. The intelligent analysis and early warning module is used to perform intelligent analysis based on unified sensing data, key information, and dynamic heat map data, identify abnormal patterns, and trigger predictive early warnings.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the unmanned mining truck monitoring and early warning method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the unmanned mining truck monitoring and early warning method as described in any one of claims 1 to 7.

Images