Mine equipment inspection method and equipment based on multi-modal data, and medium

Abstract

The embodiment of the invention discloses a mining equipment inspection method and device based on multi-modal data and a medium, and relates to the technical field of the Internet of Things, and the method comprises the steps: obtaining a predetermined current inspection task, collecting real-time multi-modal data corresponding to each target mining equipment according to a preset inspection point in the current inspection task, and sending the real-time multi-modal data to the target mining equipment; and performing feature fusion on each piece of real-time multi-modal data, determining a real-time fusion feature corresponding to each piece of target mining equipment, performing anomaly detection and fault prediction on the target mining equipment based on the real-time fusion features, and determining a remote inspection result. The remote inspection result comprises equipment state data and fault probability data corresponding to each piece of target mining equipment; and according to the remote inspection result corresponding to each target mining device, optimizing the current inspection task, determining an optimized inspection task, and updating the current inspection task based on the optimized inspection task, the optimized inspection task including an inspection path and an inspection frequency.

Description

Technical Field

[0001] This specification relates to the field of Internet of Things technology, and in particular to a mining equipment inspection method, equipment, and medium based on multimodal data. Background Art

[0002] In mining production systems, equipment inspections are a core component of ensuring safe production. Traditional inspections rely heavily on manual on-site verification, requiring operators to delve into high-risk areas underground to collect equipment status information through visual, auditory, and handheld instrumentation.

[0003] Mine tunnel environments are subject to extreme conditions such as strong noise, low illumination, and high dust levels, which significantly reduce the reliability of data collected by a single sensor. For example, visible light cameras cannot capture surface cracks on equipment in smoky areas; audio sensors have difficulty separating abnormal equipment noise from environmental noise in the crusher operating area; and vibration probes on mobile mining trucks are easily affected by bumps and produce false signals. In other words, environmental factors are deeply coupled with equipment failure characteristics, making it difficult for existing single-modal perception systems to achieve reliable status diagnosis. During the inspection process, most existing methods use inspection strategies with fixed cycles and preset paths, which cannot adapt to dynamic changes in equipment status. Repeated inspections of equipment with low failure rates result in a waste of resources. Due to insufficient inspection frequency, high-load equipment cannot capture potential faults in a timely manner, resulting in mismatched operation and maintenance resources and delayed fault response.

[0004] Therefore, the operating environment of mining equipment has extreme conditions, the data collected by a single sensor is not reliable enough, and the fixed inspection strategy ignores the dynamic changes of mining equipment, which poses the risk of mismatch of operation and maintenance resources and delayed fault response. Summary of the Invention

[0005] One or more embodiments of this specification provide a mining equipment inspection method, equipment, and medium based on multimodal data, which are used to solve the following technical problems: the operating environment of mining equipment has extreme conditions, the data collected by a single sensor is not reliable enough, and the fixed inspection strategy ignores the dynamic changes of mining equipment, resulting in the risk of mismatch of operation and maintenance resources and delayed fault response.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide a mining equipment inspection method based on multimodal data, the method comprising: obtaining a predetermined current inspection task to collect real-time multimodal data corresponding to each target mining equipment according to preset inspection points in the current inspection task, wherein the multimodal data includes equipment operation data, equipment image data, and equipment audio data; performing feature fusion on each of the real-time multimodal data to determine a real-time fusion feature corresponding to each target mining equipment, performing anomaly detection and fault prediction on the target mining equipment based on the real-time fusion feature, and determining a remote inspection result, wherein the remote inspection result includes equipment status data and fault probability data corresponding to each target mining equipment; optimizing the current inspection task based on the remote inspection result corresponding to each target mining equipment to determine an optimized inspection task, and updating the current inspection task based on the optimized inspection task, wherein the optimized inspection task includes an inspection path and an inspection frequency.

[0008] One or more embodiments of this specification provide a mining equipment inspection device based on multimodal data, including:

[0009] at least one processor; and,

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0012] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above method.

[0013] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the above technical solution, starting from the current inspection task, targeted collection is carried out according to the inspection mining equipment in the current inspection task, avoiding the amount of data generated by full collection, ensuring that the collected data is the monitoring data required for the inspection task, further ensuring the timeliness of inspection of high-risk equipment, and helping to timely discover safety hazards of risky equipment; and optimizing the collection parameters according to the real-time environment, avoiding the problem of visual failure under low visibility conditions, ensuring the reliability of the original data, and significantly reducing the correction cost of subsequent feature fusion; through the collection and processing of multimodal data, it is possible to comprehensively and comprehensively use various types of data to inspect the inspection targets. The system can analyze the inspection targets, effectively overcome the limitations of single-modal data inspection, accurately judge the working status of the inspection targets, thereby improving the accuracy, reliability and comprehensiveness of the inspection, helping to timely discover equipment failures and abnormal operation conditions, and ensure the normal operation of the equipment; carry out targeted remote inspections based on the inspection tasks optimized by the previous inspection, and then optimize the inspection tasks based on the analysis results of the actual collected data, so as to facilitate the use of the optimized inspection paths for on-site secondary inspections, and further improve the comprehensiveness and accuracy of the inspections of mining equipment; in addition, the inspection tasks are optimized based on the analysis results of the actual collected data, realizing strong monitoring of high-risk targets and weak coverage of low-risk equipment, and precise resource allocation, solving the mismatch problem of mine inspection resource allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0015] Figure 1 A flowchart of a mining equipment inspection method based on multimodal data provided in an embodiment of this specification;

[0016] Figure 2 A schematic diagram of the structure of a mining equipment inspection system based on multimodal data provided in an embodiment of this specification;

[0017] Figure 3 A schematic diagram of the structure of another mining equipment inspection system based on multimodal data provided in an embodiment of this specification;

[0018] Figure 4 A schematic diagram of the structure of a mining equipment inspection device based on multimodal data provided in an embodiment of this specification. DETAILED DESCRIPTION

[0019] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0020] The embodiments of this specification provide a mining equipment inspection method based on multimodal data. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A flowchart of a mining equipment inspection method based on multimodal data is provided in an embodiment of this specification, such as Figure 1 As shown, it mainly includes the following steps:

[0021] Step S101: obtaining a predetermined current inspection task, so as to collect real-time multimodal data corresponding to each target mining equipment according to preset inspection points in the current inspection task.

[0022] The multimodal data includes device operation data, device image data, and device audio data;

[0023] Obtaining a predetermined current inspection task specifically includes: if the current inspection process is the first inspection, obtaining the mining equipment distribution information, mining equipment attribute information and historical mining equipment operation data in the target inspection area; determining the usage frequency data corresponding to each mining equipment based on the mining equipment attributes and the historical mining equipment operation data, and quantifying it as a frequency coefficient; based on the mining equipment distribution information and the frequency coefficient, performing an equipment importance assessment on each mining equipment, and determining the equipment importance index corresponding to each mining equipment; determining the initial inspection task through the equipment importance index, wherein the initial inspection task includes an initial inspection path sequence corresponding to multiple mining equipment.

[0024] Mine tunnel environments are complex and ever-changing, with equipment distribution exhibiting spatial heterogeneity. For example, equipment is densely packed in mining areas, while equipment is dispersed along transportation lines. Furthermore, the production value and safety risks of different equipment vary significantly. Simply applying a fixed inspection template will lead to insufficient coverage of high-risk equipment and waste of resources in low-value areas.

[0025] In one embodiment of this specification, if the current inspection process is the first inspection, it is described as an initial inspection task. If the current inspection process is not the first inspection, the optimized inspection task generated in the previous inspection process is used as the current inspection task corresponding to this inspection.

[0026] In the case of the first inspection, the inspection initialization settings need to be performed. First, the mining equipment management system (EAM) provides an equipment basic attribute library, including the equipment attribute information of each mining equipment. The equipment data information here includes equipment type, purchase cost, safety level, and standby machine configuration status. Equipment types include crushers, hoists, etc. The safety level can be divided according to the existing mine safety management rules. The production control system synchronously outputs the location information of the equipment in the process flow to identify the key nodes of the main process, such as the necessary equipment for the ore crushing line. For new equipment with no historical operating data, the default rule library of the equipment type is enabled, the frequency of continuously running equipment is mapped according to the manufacturer's recommended average daily duration, and the benchmark frequency of emergency equipment is set according to the annual activation upper limit.

[0027] Based on the mining equipment attributes and historical mining equipment operation data, the usage frequency data corresponding to each mining equipment is determined. The mining equipment attributes are used to obtain the operating time and number of startups of the mining equipment in the historical mining equipment operation data. The frequency coefficient is calculated for different equipment types. For continuously operating equipment, the ratio of the average daily operating time to the 24-hour period is calculated as the frequency coefficient; for intermittently operating equipment, the ratio of the average monthly startup times to the 30-day period is calculated as the frequency coefficient; for emergency standby equipment, the annual activation number is obtained as the frequency coefficient.

[0028] Based on the mining equipment distribution information and the frequency coefficient, each piece of mining equipment is evaluated for its criticality and its corresponding equipment criticality index is determined. A scorecard is constructed to determine this index, combining four dimensions: production criticality, safety risk, cost weight, and redundancy configuration. The production criticality dimension assigns a value based on the equipment's position in the process flow, with core process equipment receiving the highest weight. For example, the production criticality weight range for the production criticality dimension ranges from 0 to 1, with core process equipment receiving the highest weight of 1. The safety risk dimension references the equipment safety profile, assigning a risk factor of 1 to specialized equipment like hoists and 0 to standard equipment. Cost weight is assigned a score based on equipment price tiers, with higher price tiers receiving a greater cost weight. Equipment is divided into multiple price ranges, with each price range assigned a weight between 0 and 1. For example, within a five-price range, if the equipment price falls within the first, highest price range, the price weight is set to 1; if the equipment price falls within the fifth, lowest price range, the price weight is set to 0.2. The redundancy dimension detects the presence of backup devices. Devices without backups trigger risk compensation. The risk compensation weight can be set to 1 or 0. If a backup device is present, the risk compensation weight is 0; if not, the risk compensation weight is 1. The four dimensions are summed and multiplied by the frequency coefficient to output the device importance index.

[0029] Set the inspection frequency for each mining equipment based on the equipment importance index. The larger the importance index, the more important the equipment, and the higher the corresponding inspection frequency. The specific frequency setting value can be set according to the equipment's safety instructions and inspection requirements. For example, if the safety instructions recommend monthly inspections, this mining equipment has a high equipment importance index, so the inspection frequency can be set to less than once every 30 days. Based on the inspection frequency, mining equipment with the same inspection frequency can be grouped as an inspection task. Set the time trigger condition based on the inspection frequency to trigger the inspection task when the time trigger condition is met.

[0030] Based on the distribution information of mining equipment, the tunnel network structure is extracted, and traffic constraints such as slope exceeding limit areas, narrow passages with a width of less than 3m, and gas gathering areas are marked. The target area is divided into sub-domains such as mining area, transportation area, and crushing area according to production function. According to the inspection frequency, multiple equipment monitoring priority sequences are formed. The inspection frequency of each equipment monitoring priority sequence is the same, and each equipment monitoring priority sequence corresponds to an inspection task. In each sub-domain, each equipment node is arranged in descending order of the equipment importance index in the equipment monitoring priority sequence. The equipment nodes in the sub-domain are connected by the Dijkstra algorithm, and the path weight introduces "distance × 0.7 + safety risk × 0.3 ” A composite cost function is used; when crossing subdomains, the main transport lanes are connected to form a closed-loop path. Finally, the initial inspection path sequence and frequency configuration table with priority markings are output.

[0031] By integrating static equipment attributes with dynamic operational characteristics such as usage frequency, a key equipment metric system is constructed to accurately locate core monitoring targets. Furthermore, based on lane topology constraints, a spatially optimal path is generated, providing a reliable initial solution for subsequent dynamic optimization and preventing the algorithm from falling into local optimality.

[0032] According to the preset inspection points in the current inspection task, real-time multimodal data corresponding to each target mining equipment is collected, specifically including: according to the preset inspection points in the current inspection task, at least one Internet of Things collection device corresponding to the preset inspection points and the corresponding target mining equipment are matched in sequence; through the environmental parameters corresponding to the preset inspection points and the equipment attribute information corresponding to the target mining equipment, the inspection control parameters corresponding to each Internet of Things collection device are generated; the inspection control parameters are sent to the corresponding Internet of Things collection device to collect real-time multimodal data corresponding to each target mining equipment through the Internet of Things collection device.

[0033] Traditional fixed-parameter data collection methods face limitations in complex underground environments. Audio sensors are susceptible to interference from equipment operating in high-noise environments. Visible-light cameras struggle to capture surface defects in equipment in rainy and foggy conditions. And deviations in the mounting angle of vibration sensors on mobile mining trucks can distort data. A one-size-fits-all data collection strategy will not only fail to capture valid information, but may also lead to misjudgments due to distorted data. Furthermore, mining equipment is diverse and numerous, and some equipment may not require inspection during current inspections. Utilizing a full-data collection approach would require significant processing time, impacting the efficiency of inspections for high-risk equipment.

[0034] In one embodiment of the present specification, first, according to the preset inspection points in the current inspection task, the target mining equipment corresponding to the preset inspection points are matched in sequence. The target mining equipment in the current inspection task is the equipment that needs to be inspected, and in order to monitor the target mining equipment, multiple IoT collection devices are pre-deployed. Specifically, the edge computing platform parses the preset point codes in the inspection task, such as P101 representing the crusher point in the mining area. Based on the point radius of 50 meters, available IoT devices are searched, such as thermal imaging camera CAM_07 and vibration sensor VD_12, and collection terminals that meet the equipment accuracy requirements are selected, such as a camera with a resolution of >1920×1080 required to detect bearing cracks.

[0035] Real-time acquisition of point environmental parameters. For example, through a mining intrinsically safe environmental sensor, when the light intensity is lower than 50 lux, the visible light collection channel is closed, the thermal imaging gain is increased to +3 levels, and the fill light is triggered at the same time. If the ambient noise exceeds 85dB for 3 seconds, the active noise reduction filtering algorithm is enabled, focusing on the 200Hz-2kHz device characteristic frequency band. For equipment in sloping tunnels with a slope greater than 15°, the sampling frequency is increased from 1kHz to 2kHz to suppress harmonic interference from transportation bumps.

[0036] Control instruction sets are generated based on device attributes and environmental conditions. For visual equipment, exposure compensation is set to +2 stops in low-light environments of 50-500 lux, increasing the gain to 60%. In rainy and foggy conditions, the thermal imager sampling rate is increased from 1Hz to 2Hz, and the temperature sensitivity is adjusted to ±0.5°C. For acoustic equipment, narrowband filters are enabled in high-noise areas exceeding 85dB, focusing on the device's characteristic frequency bands, such as bearing noise (2-8kHz). For status sensors, such as mobile devices like mining trucks, motion compensation algorithms are used to suppress vibration noise. The temperature sampling cycle in high-temperature areas is shortened to 10 seconds. Parameter packages are pushed to IoT terminals via the industrial-grade MQTT protocol, and then sent to designated IoT devices. These designated IoT collection devices then collect real-time multimodal data for each target mining device. After data collection, the devices upload data quality labels, such as clarity score and signal-to-noise ratio. Automatic re-collection is triggered when the data falls below a threshold.

[0037] Through the above technical solution, starting from the current inspection task, targeted data collection is carried out according to the inspection mining equipment in the current inspection task, avoiding the amount of data generated by full collection, ensuring that the collected data is the monitoring data required for the inspection task, further ensuring the timeliness of inspections of high-risk equipment, and helping to promptly discover safety hazards of risky equipment; and optimizing the collection parameters according to the real-time environment, avoiding visual failure problems under low visibility conditions, ensuring the reliability of the original data, and significantly reducing the correction costs of subsequent feature fusion.

[0038] Step S102 , feature fusion is performed on each real-time multimodal data to determine the real-time fusion feature corresponding to each target mining equipment, so as to perform anomaly detection and fault prediction on the target mining equipment based on the real-time fusion feature and determine the remote inspection result.

[0039] The remote inspection results include equipment status data and fault probability data corresponding to each target mining equipment;

[0040] Performing feature fusion on each of the real-time multimodal data to determine the real-time fusion features corresponding to each of the target mining equipment, specifically including: obtaining real-time environmental state parameters, wherein the real-time environmental state parameters include ambient light intensity, ambient noise level, and weather type code; dynamically determining the fusion weight corresponding to each modal data based on the environmental state parameters; extracting the modal feature vector corresponding to each modal data in the real-time multimodal data, and performing weighted fusion on the modal feature vector according to the fusion weight to generate the real-time fusion feature.

[0041] Mine tunnel environments are subject to sudden changes in lighting, such as dimly lit areas and brightly lit areas underground. Furthermore, there is the aliasing of noise from equipment, such as high-frequency rumble, and meteorological interference, such as rain, fog, and dust, which can severely distort single-modal data analysis. If a fixed-weight fusion strategy is employed, visible light images, even when ineffective in fog, will still be given a high weight, masking surface cracks in equipment. Equipment noise can be incorrectly filtered in high-noise scenarios, missing bearing wear warnings. Vibration data can generate false alarms when mobile equipment is jolted, triggering spurious faults. Through a closed loop of environmental parameters, fusion weights, and feature reconstruction, the physical environment is deeply coupled with data credibility, ensuring data authenticity during subsequent diagnostics.

[0042] In one embodiment of this specification, a network of meteorological sensors deployed underground continuously collects ambient light intensity, noise level, and weather type codes to construct a dynamic environmental parameter matrix. Based on this dynamic environmental parameter matrix, a weighted decision engine analyzes the impact of environmental factors on the credibility of multimodal data. When light attenuation or rain and fog codes are detected, the weight of visible light features is automatically reduced and the fusion ratio of thermal imaging features is increased. In the face of high-intensity background noise, the contribution of audio features is suppressed while the weight of vibration signal analysis is increased. This environmental adaptive mechanism ensures a sound balance of modal features under different operating conditions. At the feature processing layer, multimodal feature extraction is performed in parallel. The visual channel utilizes a deep convolutional network to capture the device's surface texture and thermal distribution. The audio channel applies spectral analysis techniques to separate the device's soundprint from environmental interference. The sensor channel incorporates a motion compensation algorithm to eliminate vibration noise from the mobile device. The extracted feature vectors are input into a weighted fusion module, where they are linearly superimposed according to the aforementioned dynamic weights to generate a fused feature vector representing the device's comprehensive status. The fused feature vector incorporates complementary information across modalities, forming a highly robust device health profile. After receiving the fused feature vector, the fault diagnosis engine performs a multi-level comparison against a library of equipment benchmark models. The anomaly detection module identifies immediate faults through feature shifts. For example, bearing overheating manifests as a coordinated alarm signal from a high-temperature area in the thermal image and an abnormal vibration spectrum. The prediction module predicts potential faults based on feature trend analysis. For example, decaying soundprint energy indicates progressive gear wear. The diagnostic results are ultimately output as a device status label and a fault probability estimate. This forms a closed loop of environmental perception, weighted decision-making, feature fusion, and collaborative diagnosis, effectively overcoming the risks of perception distortion and misjudgment in complex mining environments.

[0043] Through the above technical solution, through dynamic weight distribution driven by environmental parameters, at the data input layer, the dominant mode is switched in real time according to the ambient light intensity, and the weight of thermal imaging is greatly enhanced in rainy and foggy weather, solving the problem of visual failure under low visibility conditions; at the feature processing layer, the filtering parameters are dynamically configured based on the noise spectrum characteristics to effectively separate the abnormal sound of equipment and the environmental background sound; at the diagnostic decision layer, the weighted collaboration of multimodal features significantly improves the accuracy of complex fault identification, such as the concurrent detection of bearing overheating and gear wear. The environmental adaptive fusion mechanism ensures the reliability of the diagnostic results and greatly reduces the frequency of manual review.

[0044] Step S103 : Optimizing the current inspection task according to the remote inspection result corresponding to each target mining equipment, determining the optimized inspection task, and updating the current inspection task based on the optimized inspection task.

[0045] The optimized inspection task includes inspection paths and inspection frequencies.

[0046] According to the remote inspection results corresponding to each target mining equipment, the current inspection task is optimized to determine the optimized inspection task, specifically including: obtaining the equipment status data and fault probability data corresponding to each target mining equipment in the remote inspection results, and determining the high-risk mining equipment whose fault probability data is higher than a preset threshold among multiple target mining equipment through the fault probability data corresponding to each target mining equipment; dynamically adjusting the inspection frequency of the target mining equipment according to the equipment operating status data and the fault probability data to determine the optimized inspection frequency; and updating the current inspection path in the current inspection task based on the position coordinates of the high-risk mining equipment and the optimized inspection frequency to determine the optimized inspection path.

[0047] The operating status of mining equipment is highly time-varying and spatially heterogeneous. The failure rate of equipment in the mining area rises sharply due to high-load operation, while auxiliary equipment may be in a low-risk state for a long time. Under the traditional fixed inspection mode, high-frequency coverage of low-risk equipment causes waste of resources, and low-frequency missed inspection of high-risk equipment causes safety accidents, resulting in problems of resource mismatch and delayed response.

[0048] In one embodiment of the present specification, based on the device status data and failure probability data corresponding to each target mining device in the remote inspection results, the failure probability data corresponding to each target mining device is utilized to identify high-risk mining devices among multiple target mining devices whose failure probability data exceeds a preset threshold. The preset threshold here can generally be set to a probability greater than 50%, and the specific value can be determined based on inspection requirements. If the failure probability data exceeds the preset threshold, it indicates that the failure probability of this device is high, indicating that the device is high-risk mining equipment.

[0049] Based on the equipment operating status data and the fault probability data, the inspection frequency of the target mining equipment is dynamically adjusted to determine the optimized inspection frequency, specifically including: classifying multiple target mining equipment according to the equipment operating status data to determine a normal equipment set and an abnormal equipment set; based on the fault probability data, according to a preset differentiated optimization strategy, differentially adjusting the inspection frequencies corresponding to the normal equipment set and the abnormal equipment set to determine the optimized inspection frequency. Based on the fault probability data and in accordance with a preset differentiated optimization strategy, the inspection frequencies corresponding to the normal equipment set and the abnormal equipment set are differentially adjusted to determine the optimized inspection frequency, specifically including: using the fault probability data, screening a first equipment subset below the preset threshold in the normal equipment set to dynamically increase the inspection interval in the first equipment subset and determine the inspection frequency corresponding to the first equipment subset; performing an inspection interval maintenance operation on the other mining equipment in the normal equipment set except the first equipment subset and determining the inspection frequency corresponding to the other mining equipment; obtaining the abnormal type information corresponding to the abnormal mining equipment in the abnormal equipment set and dynamically shortening the inspection interval of the high-risk mining equipment based on the abnormal type information to determine the high-risk inspection frequency of the high-risk mining equipment; when the inspection frequency corresponding to the first equipment subset, the inspection frequency corresponding to the other mining equipment and the high-risk inspection frequency of the dangerous mining equipment meet the preset safety interval boundary constraints, determining the optimized inspection frequency of the target mining equipment.

[0050] In one embodiment of this specification, target mining equipment is first classified in detail. Based on equipment operating status data, the equipment is divided into two core categories: a normal equipment set and an abnormal equipment set. Within the normal equipment set, a low-risk equipment subset is further selected. This subset consists of equipment with a failure probability below a preset threshold; the remaining equipment is classified as stable operating equipment. The abnormal equipment set focuses on high-risk equipment with a failure probability exceeding the threshold.

[0051] After the classification is completed, differentiated frequency adjustments are made. For a subset of low-risk equipment, a progressive inspection interval relaxation strategy is implemented, gradually extending the detection cycle while ensuring the safety margin; the original inspection frequency of steady-state operating equipment remains unchanged; for high-risk equipment groups, a graded response mechanism is implemented in combination with the characteristics of the abnormality type. For example, mechanical wear-related abnormalities trigger medium-intensity frequency compression, electrical fault-related abnormalities implement deep frequency enhancement, and complex abnormalities enable a superimposed compression strategy to form a multi-level risk defense system. During the execution adjustment process, a multi-dimensional security verification mechanism is simultaneously started. At the device level, the minimum safety detection interval is enforced to avoid monitoring blind spots caused by excessive compression. The final generated optimized inspection frequency configuration is sent to the edge execution layer in the form of structured data packets to achieve dynamic matching of inspection resources and equipment risk status.

[0052] Through the risk classification response mechanism, the failure probability, abnormality type and equipment value are associated to achieve strong monitoring of high-risk targets and weak coverage of low-risk equipment, and precise resource allocation to solve the mismatch problem of mine inspection resource allocation.

[0053] Based on the location coordinates of the high-risk mining equipment and the optimized inspection frequency, the current inspection path in the current inspection task is updated to determine the optimized inspection path, specifically including: setting a sampling weight probability for each device node corresponding to the high-risk mining equipment according to the optimized inspection frequency of each high-risk mining equipment, wherein the sampling weight probability is positively correlated with the optimized inspection frequency; optimizing the current inspection path according to the sampling weight probability and the location coordinates to determine the optimized inspection path.

[0054] In one embodiment of the present specification, based on the list of high-risk mining equipment and the optimized inspection frequency data of each high-risk mining equipment, the equipment monitoring frequency is mapped to the spatial sampling weight through weight conversion. High-frequency monitoring equipment is assigned a significantly higher weight value, forming a risk thermal distribution map covering the three-dimensional space of the tunnel, which intuitively presents the differential distribution of inspection value in each area. The improved RRT* algorithm is used for path optimization. After the algorithm is initialized, a random tree structure is constructed at the starting point, and a directional expansion mechanism is used to guide node growth. The high-risk equipment focus strategy guides the expansion direction towards the weight peak area with a dominant probability, and automatically shortens the expansion step in the equipment-dense area to improve the path accuracy. The historical path is reused synchronously, and dynamic disturbances are injected on the basis of retaining the verified safe path. The disturbance amplitude is inversely proportional to the equipment risk weight. The entire expansion process is subject to real-time tunnel constraint detection, and the resampling mechanism is immediately triggered when the node enters the prohibited area or approaches the dangerous distance of the tunnel wall.

[0055] A composite cost function model is constructed to simultaneously optimize two key metrics: path length and high-risk equipment coverage. This model incorporates a dynamic balancing factor to automatically prioritize equipment coverage in high-risk areas. The generated initial path is processed by a smoothing engine, using curve fitting to eliminate sharp corners. Auxiliary checkpoints are inserted in sharp turns to ensure the stability of inspection equipment. This path now maintains spatial continuity and ensures safe passage. The system continuously monitors changes in the roadway environment. When a sudden obstacle, such as a landslide or water accumulation, is detected by the LiDAR, local path replanning is initiated, starting from the current location. The resulting optimized path sequence is processed using an incremental compression algorithm, encapsulating only the modified nodes into lightweight instruction packets. These packets are then sent to the edge execution layer via a dedicated communication protocol, ensuring that inspection resources are always focused on the highest-value target areas. A deep coupling mechanism between risk weighting and spatial search is established. High-risk equipment receives priority algorithm access through weight mapping. Roadway physical constraints ensure path feasibility through real-time collision detection. Historical path reuse significantly reduces computational complexity. The resulting optimized path meets both the business requirement of prioritizing high-risk equipment coverage and the engineering requirements of path efficiency and traffic safety.

[0056] It should be noted that the inspection process in the embodiments of this specification can be implemented using intelligent inspection equipment or remotely through pre-deployed sensor networks and data acquisition modules. In one embodiment, after obtaining the optimized inspection task in step S103, an intelligent inspection device, such as an inspection robot, is used to conduct an on-site inspection. Following the remote inspection in step S102, the inspection path is optimized to facilitate a secondary on-site inspection, further improving the comprehensiveness and accuracy of the inspection of mining equipment.

[0057] Through the above technical solution, starting from the current inspection task, targeted collection is carried out according to the inspection mining equipment in the current inspection task, avoiding the amount of data generated by full collection, ensuring that the collected data is the monitoring data required for the inspection task, further ensuring the timeliness of inspection of high-risk equipment, and helping to timely discover safety hazards of risky equipment; and optimizing the collection parameters according to the real-time environment, avoiding the problem of visual failure under low visibility conditions, ensuring the reliability of the original data, and significantly reducing the correction cost of subsequent feature fusion; through the collection and processing of multi-modal data, it is possible to comprehensively and comprehensively use multiple types of data to analyze the inspection target, effectively overcoming the limitations of single-modal data. According to the limitations of inspections, the working status of inspection targets can be accurately judged, thereby improving the accuracy, reliability and comprehensiveness of inspections, helping to timely discover equipment failures and abnormal operation conditions, and ensuring the normal operation of equipment; targeted remote inspections are carried out based on the inspection tasks optimized by the previous inspection, and then the inspection tasks are optimized based on the actual data analysis results, so as to facilitate the use of the optimized inspection paths for on-site secondary inspections, further improving the comprehensiveness and accuracy of mining equipment inspections; in addition, the inspection tasks are optimized based on the actual data analysis results, achieving strong monitoring of high-risk targets and weak coverage of low-risk equipment, and precise resource allocation, solving the mismatch problem of mine inspection resource allocation.

[0058] Figure 2 This is a schematic diagram of the structure of a mining equipment inspection system based on multimodal data provided in an embodiment of this specification, such as Figure 2 As shown in the figure, the mining equipment inspection system includes a data acquisition and real-time monitoring module, an inspection task management and scheduling module, a fault warning and prediction module, and a data analysis and report generation module.

[0059] The data acquisition and real-time monitoring module deploys a variety of sensors to collect various types of data from devices in real time, including operational status data, audio data, and video data. These parameters reflect the device's operating status and workload. Managers can monitor the device's real-time status at any time to identify potential faults. It should be noted that video data includes visible light, thermal, and multispectral images; operational status data includes electrical parameters, temperature, humidity, and vibration data; and audio data includes the operating sound frequency of the device.

[0060] The inspection task management and scheduling module automatically generates inspection tasks based on the equipment's actual operating status, maintenance cycles, and pre-set inspection plans. Upon receiving an inspection task, the edge computer reads data from front-end sensors, such as cameras and audio sensors, in the order of the preset points listed in the inspection task. After preprocessing the collected multimodal data, it uses a deep learning-based intelligent recognition algorithm for fusion analysis to generate inspection results. The edge computer can also optimize the inspection path using a path optimization algorithm.

[0061] The fault warning and prediction module continuously monitors equipment operating status through real-time analysis of sensor data. Leveraging computer vision and machine learning algorithms, it automatically analyzes collected real-time operating, video, and audio data to identify abnormal equipment conditions or potential faults. By integrating multi-source data such as images, videos, text, and sensors, and through cross-modal alignment and joint reasoning, it enables a global analysis of equipment status, environmental risks, and human behavior. The data analysis and report generation module utilizes deep learning algorithms to perform joint reasoning on multimodal fusion features to identify potential risks or equipment failures. Inspection reports and decisions are generated based on the identified potential risks or equipment failures.

[0062] Figure 3 A flowchart of another method for inspecting mining equipment based on multimodal data is provided in the embodiment of this specification, as shown in FIG. Figure 3 As shown, first, an inspection task is generated and issued. According to the pre-configured inspection task, this inspection task is triggered according to the set time rules and sent to the edge platform via the MQTT protocol. The data acquisition module hardware includes all sensors and instruments deployed on instruments in the industrial environment, as well as the handheld mobile terminals used by inspection personnel. Inspection personnel are equipped with multifunctional handheld mobile terminals. These terminals integrate multiple functions, including RFID tag scanning, operating parameter reporting, real-time positioning, and a high-precision clock. The high-precision clock module accurately records timestamps to ensure accurate time recording. The data acquisition module collects real-time operating data from various devices in the industrial environment and data returned by handheld mobile terminals. This includes, but is not limited to, physical parameters such as device temperature, pressure, speed, and vibration frequency; device operating status information (such as power on, power off, and faults); video image data; and audio data. The collected audio and video data can also be pre-processed.

[0063] Receive data collected and uploaded during device operation in real time, such as temperature, pressure, flow, vibration, and other device operating status data. Analyze the collected data based on preset judgment rules to determine whether the device has any abnormalities and the type of abnormality. In addition to real-time monitoring, to prevent abnormalities in some devices from going undetected, abnormal events discovered during manual inspections are also uploaded to the IoT platform via handheld mobile terminals and stored in the data storage module. Upon receiving the inspection task, the edge platform reads data from front-end sensors such as cameras and audio sensors in the order of the preset points listed in the inspection task. The collected multimodal data is preprocessed and analyzed using an intelligent recognition algorithm based on deep learning to form the inspection results.

[0064] First, real-time data such as current, voltage, temperature, and vibration from the device is collected to generate multimodal signals, which are then fed into a neural network for state prediction. Based on the matching multimodal fusion method, deep learning algorithms are used to extract features from both video and audio, and the extracted features are then fused multimodally. A video gateway collects surveillance video and images from the device in real time. Preliminary image features are extracted using a ResNet50 model. Different image features are then mapped through different layers to train different weak classifiers in the subsequent multimodal feature fusion phase. Audio information from the device's operation is collected via a sound sensor. Mel-frequency cepstral coefficient (MFCC) features are used as the device's voiceprint signal signature. An MFCC-Transformer recognition model is constructed and trained to identify the device's operating status. The multimodal feature fusion phase uses the Adaboost ensemble algorithm as the overall framework, training each weak classifier sequentially until all are integrated into a single strong classifier capable of detecting all categories. Finally, a deep learning algorithm is used to jointly infer the multimodal fusion features to identify potential risks or equipment failures in industrial inspection areas. Inspection reports and decisions are generated based on the identified potential risks or equipment failures.

[0065] The inspection path is optimized based on the graph theory model and the shortest path algorithm, and the inspection frequency and path are dynamically adjusted according to the health status of the equipment, which significantly improves the inspection efficiency and the accuracy of fault identification. Considering that the rapid expansion random tree algorithm (RRT) may be somewhat blind when solving the most planned path, in order to optimize the subsequent planning effect, the equipment inspection target is first probabilistically sampled. For the inspection target node, a sampling threshold of p between 0 and 1 will be generated in advance based on the monitoring status of the inspection equipment in the inspection task. bias, the higher the probability of equipment failure monitoring, the greater the threshold, and the nodes in the inspection task are closer to the front. On the basis of inspection target probability sampling, by combining the Informed RRT* algorithm, the path cost spent on each random tree node is calculated to achieve full coverage inspection path planning. The principle of the RRT algorithm is to construct random trees at the starting point and end point respectively in an incremental manner, and then search for an optimal planning path in the spatial area through target probability sampling. Informed RRT* combines the rapid exploration capability of the RRT algorithm and the guiding capability of the heuristic function to effectively search for the optimal path in complex environments. The Informed RRT* algorithm improves the efficiency of the algorithm and the quality of the path by using the heuristic function to guide the sampling process.

[0066] In the inspection path optimization algorithm, the Informed RRT* algorithm is improved. (x, y, z) is used to store the information of any node in the random tree, where x and y represent the two-dimensional coordinates of the node respectively; z represents the parent node of the node. Assume that the node x near , to generate a new tree node x new , there is a 10% probability of sampling from the historical path. If there is a historical path, randomly select a historical path, and then randomly select a node from the path, and then generate a new sampling point near this node by adding a certain range of random perturbations. Then, the extended node x new Save it, otherwise, you need to delete the newly expanded tree node to ensure the execution effect of the inspection task.

[0067] The specific calculation formula of θ is as follows:

[0068]

[0069] Among them, x near Represents the nearest planning node, x parent Represents the parent node of the nearest planning node. The nearest planning node is mainly related to the path cost, and its specific calculation formula is:

[0070] x near =min(Cost(x tree , x start )+dis(x tree , x start )

[0071] In the formula, Cost(x tree , x start ) represents the path cost between the node on the search tree and the starting point of the planned path; dis(x tree , x start) represents the distance between a node in the search tree and the end point of the planned path, which can be calculated using the Manhattan distance formula. Based on the above process, the path cost of the search tree node is calculated, and the optimal planned path is found through a bidirectional search and iteration method.

[0072] After receiving the returned inspection results, multimodal result data, including images and audio, is stored in the object repository, with storage path links generated. Indicator attribute information is stored in the relational database, and the result data supports multi-dimensional queries such as time and task number. Unstructured data, such as voice work orders and maintenance records, is parsed in real time to build a company-specific inspection knowledge base, improving the accuracy of anomaly warnings.

[0073] The embodiment of this specification also provides a mining equipment inspection device based on multimodal data, such as Figure 4 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0074] The embodiments of this specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0075] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0076] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0078] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0084] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, the phrase "comprising a..." … ” The specified elements do not exclude the existence of other identical elements in the process, method, product or equipment that includes the elements.

[0086] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A mining equipment inspection method based on multimodal data, characterized in that: The method comprises: Obtaining a predetermined current inspection task to collect real-time multimodal data corresponding to each target mining equipment according to preset inspection points in the current inspection task, wherein the multimodal data includes equipment operation data, equipment image data, and equipment audio data; Performing feature fusion on each of the real-time multimodal data to determine a real-time fusion feature corresponding to each of the target mining equipment, performing anomaly detection and fault prediction on the target mining equipment based on the real-time fusion feature, and determining a remote inspection result, wherein the remote inspection result includes equipment status data and fault probability data corresponding to each of the target mining equipment; According to the remote inspection results corresponding to each target mining equipment, the current inspection task is optimized to determine an optimized inspection task, so as to update the current inspection task based on the optimized inspection task, wherein the optimized inspection task includes an inspection path and an inspection frequency.

2. A mining equipment inspection method based on multimodal data according to claim 1, characterized in that: Get the current pre-determined inspection tasks, including: If the current inspection process is the first inspection, obtain the mining equipment distribution information, mining equipment attribute information and historical mining equipment operation data in the target inspection area; Determine usage frequency data corresponding to each mining device based on the mining device attributes and the historical mining device operation data, and quantify the data into a frequency coefficient; Based on the mining equipment distribution information and the frequency coefficient, performing an equipment importance assessment on each mining equipment, and determining an equipment importance index corresponding to each mining equipment; An initial inspection task is determined according to the equipment importance index, wherein the initial inspection task includes an initial inspection path sequence corresponding to a plurality of mining equipment.

3. The mining equipment inspection method based on multimodal data according to claim 1, characterized in that: According to the preset inspection points in the current inspection task, real-time multimodal data corresponding to each target mining equipment is collected, including: According to the preset inspection points in the current inspection task, at least one IoT collection device corresponding to the preset inspection points and the corresponding target mining equipment are matched in sequence; Generate inspection control parameters corresponding to each of the IoT collection devices based on the environmental parameters corresponding to the preset inspection points and the device attribute information corresponding to the target mining equipment; The inspection control parameters are sent to the corresponding Internet of Things collection device, so that the real-time multimodal data corresponding to each target mining equipment is collected through the Internet of Things collection device.

4. The mining equipment inspection method based on multimodal data according to claim 1, characterized in that: Performing feature fusion on each of the real-time multimodal data to determine the real-time fusion features corresponding to each of the target mining equipment specifically includes: Acquiring real-time environmental status parameters, wherein the real-time environmental status parameters include ambient light intensity, ambient noise level, and weather type code; Based on the environmental state parameters, dynamically determining the fusion weight corresponding to each modal data; A modal feature vector corresponding to each modal data in the real-time multimodal data is extracted, and the modal feature vector is weightedly fused according to the fusion weight to generate the real-time fusion feature.

5. The mining equipment inspection method based on multimodal data according to claim 1, characterized in that: Optimize the current inspection task according to the remote inspection results corresponding to each target mining equipment to determine the optimized inspection task, specifically including: Obtaining device status data and fault probability data corresponding to each target mining device in the remote inspection result, and determining, based on the fault probability data corresponding to each target mining device, high-risk mining devices whose fault probability data is higher than a preset threshold among the multiple target mining devices; Dynamically adjusting the inspection frequency of the target mining equipment according to the equipment operating status data and the fault probability data to determine an optimized inspection frequency; Based on the position coordinates of the high-risk mining equipment and the optimized inspection frequency, the current inspection path in the current inspection task is updated to determine the optimized inspection path.

6. A mining equipment inspection method based on multimodal data according to claim 5, characterized in that: Dynamically adjusting the inspection frequency of the target mining equipment according to the equipment operating status data and the fault probability data to determine an optimized inspection frequency, specifically including: Classifying the plurality of target mining equipment according to the equipment operation status data to determine a normal equipment set and an abnormal equipment set; According to the fault probability data and in accordance with a preset differentiated optimization strategy, the inspection frequencies corresponding to the normal device set and the abnormal device set are differentially adjusted to determine an optimized inspection frequency.

7. The mining equipment inspection method based on multimodal data according to claim 5, characterized in that: Based on the location coordinates of the high-risk mining equipment and the optimized inspection frequency, the current inspection path in the current inspection task is updated to determine the optimized inspection path, specifically including: According to the optimized inspection frequency of each high-risk mining equipment, a sampling weight probability is set for the equipment node corresponding to each high-risk mining equipment, wherein the sampling weight probability is positively correlated with the optimized inspection frequency; The current inspection path is optimized according to the sampling weight probability and the position coordinates to determine an optimized inspection path.

8. The mining equipment inspection method based on multimodal data according to claim 6, characterized in that: Based on the fault probability data and in accordance with a preset differentiated optimization strategy, differential adjustments are made to the inspection frequencies corresponding to the normal device set and the abnormal device set to determine an optimized inspection frequency, specifically including: Using the failure probability data, a first subset of devices whose faults are below the preset threshold is selected from the normal device set, so as to dynamically increase the inspection interval in the first subset of devices and determine the inspection frequency corresponding to the first subset of devices; For the other mining equipment in the normal equipment set except the first equipment subset, performing an inspection interval maintenance operation to determine the inspection frequency corresponding to the other mining equipment; Acquire abnormality type information corresponding to abnormal mining equipment in the abnormal equipment set, and dynamically shorten the inspection interval of the high-risk mining equipment based on the abnormality type information to determine the high-risk inspection frequency of the high-risk mining equipment; When the inspection frequency corresponding to the first equipment subset, the inspection frequency corresponding to the other mining equipment, and the high-risk inspection frequency of the dangerous mining equipment meet the preset safety interval boundary constraints, the optimized inspection frequency of the target mining equipment is determined.

9. A mining equipment inspection device based on multimodal data, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images