A coal flow multi-modal autonomous perception monitoring method, system, device and medium

Abstract

The application discloses a coal flow multi-modal autonomous perception monitoring method, system, device and medium, relates to the technical field of industrial intelligent monitoring and digital data processing, and comprises the following steps: deploying equipment, collecting in real time, and generating a multi-modal data set; pre-processing the multi-modal data and extracting features to obtain a joint representation vector; generating an abnormal probability based on cloud-edge collaboration; triggering an early warning signal in combination with the abnormal probability and a hierarchical threshold; and adjusting a dynamic distribution to optimize a monitoring strategy in combination with the early warning signal, thereby improving the coal flow perception monitoring precision. The application can timely find various abnormalities and potential safety hazards in operation, such as deviation, foreign matter and coal stacking, avoid production accidents caused by equipment failure, and ensure production safety. Meanwhile, abnormal information such as coal stacking, foreign matter, deviation and carrier roller failure in the operation process is statistically analyzed, and the safety and production efficiency in the coal flow transportation process are further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial intelligent monitoring and digital data processing, in particular to a coal flow multi-modal autonomous perception monitoring method, system, device and medium. BACKGROUND

[0002] With the continuous development of the coal industry, the coal flow transportation system plays a crucial role in mine production. As the core equipment of the coal flow transportation system, the belt conveyor directly affects the smooth progress of the entire production process. However, due to the harsh coal mine environment, equipment aging and other reasons, the belt conveyor is prone to various abnormal situations during operation, such as deviation, tearing, slipping, etc. These problems not only affect production efficiency, but also may cause serious safety accidents.

[0003] In order to ensure the normal operation of the coal flow transportation system and improve production efficiency and safety production level, it is necessary to develop an intelligent monitoring algorithm for the coal flow transportation scene. By introducing artificial intelligence technology, real-time monitoring and fault warning of the belt conveyor are realized, thereby reducing the risk of accidents and improving the reliability and service life of the equipment.

[0004] Single belt is long and the environment is complex, and is under the influence of factors such as high buried depth geothermal, high mineralization degree water, etc. There are many safety hazards in the actual operation of the mine coal flow transportation system. The failure of the carrier roller occurs frequently, and a large number of damaged carrier rollers cannot be replaced in time, which seriously threatens the safe operation of the belt conveyor. The belt is prone to wear and tear during operation, and problems such as belt tearing, foreign matter, deviation, coal stacking, etc. may occur during coal transportation. If the belt is severely damaged, the entire belt needs to be replaced, which is costly. In order to ensure the safe operation of the main coal flow system, multiple manual inspections are required per shift, which is labor-intensive and time-consuming.

[0005] The present application develops an intelligent monitoring algorithm for the coal flow transportation scene based on cloud-edge collaboration and edge-learning, realizes the key technology of developing a belt abnormal AI monitoring model, forms a coal flow system health protection system, integrates inspection control and data analysis, realizes belt damage tracking, timely warning of dangerous situations and business closed loop, effectively assists field production and belt maintenance work, and is widely applied, which has achieved remarkable economic and social benefits. SUMMARY

[0006] In view of the above problems, the present application is proposed.

[0007] Therefore, the problem to be solved by this invention is: how to solve the problems of frequent belt and roller failures, low efficiency of manual inspection, untimely anomaly detection, high equipment maintenance costs and poor environmental adaptability in coal mine coal transportation systems through AI visual inspection, multimodal data fusion, cloud-edge collaborative architecture and other technical means, so as to achieve intelligent, efficient and safe operation of coal transportation systems.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a multimodal autonomous sensing and monitoring method for coal flow, comprising: deploying equipment to collect data in real time and generate a multimodal dataset; preprocessing the multimodal data and extracting features to obtain a joint representation vector; the feature extraction includes using a convolutional neural network to extract surface damage texture features and temperature gradient features, and using short-time Fourier transform and Mel-frequency cepstral coefficients to extract sound spectrum features; generating anomaly probabilities based on cloud-edge collaboration; the anomaly probability includes utilizing cloud-edge collaboration, that is, fusing the detection results obtained by the real-time target detection algorithm at the edge with the results obtained by the residual network analysis in the cloud to calculate the anomaly probability; combining the anomaly probability with a graded threshold to trigger an early warning signal; and combining the early warning signal with the dynamic distribution optimization monitoring strategy to improve the accuracy of coal flow sensing and monitoring.

[0009] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the multimodal dataset includes images, temperature distribution, operating noise, belt conveyor speed, coal quantity distribution, and idler vibration signals.

[0010] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the result of the preprocessing is expressed as follows:

[0011] ,

[0012] in, This indicates a composite processing operation. This represents a transformation of the original data. Indicates the first Types of data Transformation parameters, Indicates the first Weighting coefficients for the transformation of this type of data. Indicates the number of operations, take =6, Indicates an index variable. This indicates the result of preprocessing. Indicates the first Various data types, including Image data, Temperature distribution Operating noise Belt conveyor speed Coal quantity distribution and Idler roller vibration signal.

[0013] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the joint characterization vector includes a surface damage texture feature vector, a temperature gradient feature vector, and a sound spectrum feature vector; the surface damage texture feature and temperature gradient feature extracted using a convolutional neural network are represented as follows:

[0014] ,

[0015] ,

[0016] in, Represents the parameters of a convolutional neural network. This represents the preprocessed image data. This represents the extracted texture features with surface damage. This represents the temperature distribution data after preprocessing. This represents the extracted temperature gradient features. This represents a convolutional neural network; the sound spectral features extracted using short-time Fourier transform and Mel-frequency cepstral coefficients are represented as follows:

[0017] ,

[0018] ,

[0019] in, This indicates the preprocessed noise data. The parameters represent the short-time Fourier transform. The parameter representing the Mel frequency cepstral coefficients. This represents the obtained sound spectrum characteristics. This represents the result of the short-time Fourier transform. Represents the short-time Fourier transform. This represents the Mel frequency cepstral coefficients.

[0020] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the calculated anomaly probability is expressed as:

[0021] ,

[0022] in, This represents the probability of an anomaly detected by the real-time object detection algorithm. This represents the anomaly probability obtained from the residual network. , Indicates the weighting coefficient. These represent the parameters of a real-time object detection algorithm. The parameters represent the residual network. The results represent the probability of anomalies. The detection results obtained by the real-time target detection algorithm include deploying a lightweight real-time target detection algorithm model at the edge to detect abnormal targets in the area image in real time, including foreign objects, large pieces of coal, and deviations, generating target detection results, including target category, location, and confidence level, and uploading them to the cloud. The results obtained by the residual network analysis include receiving the detection results uploaded from the edge at the cloud and performing joint analysis with thermal imaging and sound data. The joint analysis includes using the residual network to extract temperature gradient features to determine the risk of local overheating; using the residual network to extract sound spectrum features to determine the risk of abnormal wear of the idler rollers; and generating the probability distribution of anomalies in the thermal imaging and sound data.

[0023] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the early warning signal includes three levels of early warning, which are generated by combining anomaly probability and graded thresholds; the three levels of early warning include high-risk early warning, medium-risk early warning, and low-risk early warning, as shown below:

[0024] ,

[0025] in, This indicates the threshold for high-risk classification. Indicates the low-risk classification threshold; when When a high risk is detected, the equipment linkage control module is triggered, automatically adjusting the belt speed and pushing real-time images and the coordinates of the potential hazard location to the monitoring terminal via a dynamic alarm video wall; when When the risk level is medium, an early warning message is pushed to the monitoring terminal, prompting the operator to conduct inspections and interventions; when... When the risk is low, the warning information is recorded for subsequent analysis.

[0026] As a preferred embodiment of the multimodal autonomous sensing and monitoring method for coal flow described in this invention, the dynamic distribution optimization monitoring strategy is expressed as follows:

[0027] ,

[0028] in, This indicates a dynamic distribution optimization monitoring strategy. This indicates that a high-priority dynamic distribution strategy is triggered under high-risk conditions. This indicates that a medium-risk scenario warrants a medium-priority response. This indicates that a low-priority response should be adopted in low-risk situations. This indicates parameter adjustments made in response to high-risk warnings. This indicates the operational parameters for issuing medium-risk warnings. representing the operating parameters for low-risk warning, representing the dynamic distribution parameters; when in high-risk, increasing the sampling frequency, processing important data preferentially, and responding in real time; when in medium-risk, optimizing the sensor sampling rate and adjusting the resource allocation; when in low-risk, reducing the system load and optimizing the data transmission efficiency.

[0029] Another object of the present application is to provide a coal flow multi-modal autonomous perception monitoring system, which realizes autonomous perception, real-time anomaly detection and accurate early warning of multi-modal data in the coal flow monitoring process by integrating data acquisition, feature extraction, anomaly detection, early warning linkage and model optimization function modules, and improves the intelligent and automatic level of coal flow monitoring, ensures the safe operation of equipment and reduces the dependence on manual intervention.

[0030] To solve the above technical problems, the present application provides the following technical scheme: a coal flow multi-modal autonomous perception monitoring system, comprising a data acquisition module, a feature extraction module, an anomaly detection module, an early warning linkage module and a monitoring optimization module; the data acquisition module is used for deploying equipment, performing real-time acquisition and generating a multi-modal data set; the feature extraction module is used for pre-processing and feature extraction of the multi-modal data to obtain a joint representation vector; the anomaly detection module is used for generating an anomaly probability based on cloud-edge collaboration; the early warning linkage module is used for combining the anomaly probability and a hierarchical threshold to trigger an early warning signal; and the monitoring optimization module is used for combining the early warning signal, adjusting the dynamic distribution and optimizing the monitoring strategy to improve the coal flow perception monitoring accuracy.

[0031] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the coal flow multi-modal autonomous perception monitoring method as described above when executing the computer program.

[0032] A computer readable storage medium stores a computer program, and the computer program implements the steps of the coal flow multi-modal autonomous perception monitoring method as described above when executed by a processor.

[0033] The present application has the following advantages: the coal flow multi-modal autonomous perception monitoring method provided by the present application can timely discover various abnormalities and potential safety hazards in the running process, such as deviation, foreign matter and coal stacking, avoid production accidents caused by equipment failure, and ensure production safety. The developed coal flow multi-modal autonomous perception monitoring system integrates computer vision, equipment control and big data analysis technology, realizes intelligent detection and alarm of the belt roller and the belt surface, simultaneously performs statistical analysis on abnormal information such as coal stacking, foreign matter, deviation and roller failure in the running process, and realizes AI video speed regulation, video management analysis, video stream management analysis and platform alarm prompt. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative labor.

[0035] Figure 1 A flow chart of a coal flow multi-modal autonomous perception monitoring method provided for the first embodiment of the present application.

[0036] Figure 2 A structure diagram of a coal flow multi-modal autonomous perception monitoring system provided for the second embodiment of the present application.

[0037] Figure 3 An experimental comparison diagram of a coal flow multi-modal autonomous perception monitoring method provided for the third embodiment of the present application.

[0038] In the figure, 100, a data acquisition module; 200, a feature extraction module; 300, an anomaly detection module; 400, a pre-warning linkage module; 500, a monitoring optimization module. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0040] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0041] Embodiment 1

[0042] Reference Figure 1 For the first embodiment of the present application, the embodiment provides a coal flow multi-modal autonomous perception monitoring method, which comprises: deploying equipment, performing real-time acquisition, and generating a multi-modal data set; pre-processing and feature extraction of the multi-modal data to obtain a joint representation vector; generating an abnormal probability based on cloud-edge collaboration; combining the abnormal probability with a hierarchical threshold to trigger a pre-warning signal; combining the pre-warning signal to optimize the monitoring strategy by adjusting the dynamic distribution, and improving the coal flow perception monitoring accuracy.

[0043] S1, deploying equipment, performing real-time acquisition, and generating a multi-modal data set.

[0044] It should be noted that the multi-modal data set includes images, temperature distribution, running noise, belt running speed, coal quantity distribution and roller vibration signal.

[0045] Furthermore, the generation of multimodal datasets relies on traditional sensor data acquisition, combined with computer vision and intelligent algorithms, to ensure the accurate capture of subtle anomalies; for example, image data can help detect surface damage or foreign objects in coal seams, while temperature distribution data can provide early signals of equipment overheating or localized failures.

[0046] Furthermore, the combination of multi-dimensional data such as temperature distribution, coal quantity distribution, and idler vibration signals not only improves the accuracy of the data, but also enables the identification of potential risk points through intelligent analysis algorithms. For example, changes in idler vibration signals may indicate idler failure, and combining them with other data can help predict the occurrence of failures in advance, thereby reducing unexpected downtime and equipment damage.

[0047] S2. Preprocess the multimodal data and extract features to obtain a joint representation vector.

[0048] It should be noted that data preprocessing involves extracting and fusing features from different modalities through composite processing operations, and then constructing a joint representation vector; the result of the preprocessing is expressed as follows:

[0049] ,

[0050] in, This indicates a composite processing operation. This represents a transformation of the original data. Indicates the first Types of data Transformation parameters, Indicates the first Weighting coefficients for the transformation of this type of data. Indicates the number of operations, take =6, Indicates an index variable. This indicates the result of preprocessing. Indicates the first Various data types, including Image data, Temperature distribution Operating noise Belt conveyor speed Coal quantity distribution and Idler roller vibration signal.

[0051] Further, the joint representation vector includes a belt surface damage texture feature vector, a temperature gradient feature vector, and a sound spectrum feature vector; the feature extraction step includes using a convolutional neural network (CNN) to extract the belt surface damage texture feature and the temperature gradient feature, and using a short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCC) to extract the sound spectrum feature.

[0052] wherein the using the convolutional neural network to extract the belt surface damage texture feature and the temperature gradient feature is represented as:

[0053] ,

[0054] ,

[0055] wherein, represents a convolutional neural network parameter, represents preprocessed image data, represents an extracted belt surface damage texture feature, represents preprocessed temperature distribution data, represents an extracted temperature gradient feature, represents a convolutional neural network.

[0056] The using the short-time Fourier transform and the Mel-frequency cepstral coefficients to extract the sound spectrum feature is represented as:

[0057] ,

[0058] ,

[0059] wherein, represents preprocessed operating noise data, represents a short-time Fourier transform parameter, represents a Mel-frequency cepstral coefficient parameter, represents a sound spectrum feature obtained, represents a short-time Fourier transform result, represents a short-time Fourier transform, represents a Mel-frequency cepstral coefficient.

[0060] Further, in the process of preprocessing and feature extraction of multi-modal data, after the data is subjected to composite processing operation, the features of different modalities are effectively extracted and fused, thereby constructing a high-dimensional joint representation vector. Specifically, the application of CNN in the extraction of surface damage texture features and temperature gradient features involves a fine feature abstraction process. Through hierarchical learning of multiple convolution layers in the network, the micro features of damage texture and temperature change are revealed. When using CNN for feature extraction, the preprocessed image data and temperature distribution data are input into the network for deep convolution to generate high-level feature representation. The texture features of surface damage are extracted by the multi-layer convolution kernel of CNN, which reflects the local damage of coal flow. The temperature gradient features reflect the possible overheating risk of the system through the spatial temperature distribution change captured by the network. In the process of sound spectrum feature extraction, STFT can convert the running noise signal to the frequency domain to reveal the potential frequency features.

[0061] S3, generating an abnormal probability based on cloud-edge collaboration.

[0062] It should be noted that the abnormal probability utilizes cloud-edge collaboration, that is, the detection results obtained by the edge real-time target detection algorithm (You Only Look Once, YOLO) are fused with the results obtained by the residual network analysis in the cloud to calculate the abnormal probability, which is represented as:

[0063]

[0064] Among them, represents the abnormal probability detected by YOLO, represents the abnormal probability obtained by the residual network, , represents a weighting coefficient, represents the parameters of YOLO, represents the parameters of the residual network, represents the abnormal probability.

[0065] Further, the detection results obtained by YOLO include deploying a lightweight YOLO model on the edge to detect abnormal targets in the surface image in real time, including foreign matter, large coal, and deviation; generating target detection results, including target category, position, and confidence, and uploading to the cloud.

[0066] The results obtained by the residual network analysis include receiving the detection results uploaded by the edge in the cloud, and combining thermal imaging and sound data for joint analysis; including using the residual network to extract temperature gradient features to judge the local overheating risk; using the residual network to extract sound spectrum features to judge the abnormal wear risk of the roller; generating the abnormal probability distribution of thermal imaging and sound data.

[0067] ​Further, the synergy between the cloud and the edge significantly enhances the system's real-time response capability. Through precise multi-modal data joint analysis (such as thermal imaging data and sound signals), the system can more efficiently detect abnormalities and improve accuracy. For example, the lightweight YOLO model deployed on the edge can quickly identify large foreign objects, coal blocks, and potential risks such as deviation in the image, while the residual network in the cloud conducts in-depth analysis of temperature gradient and sound spectrum data to accurately determine whether the equipment is at risk of overheating or abnormal wear. By fusing these multi-dimensional data, the generated abnormal probability can more accurately reflect the running state of the equipment, thereby supporting more timely and effective early warning and decision-making. Furthermore, the system provides great flexibility in multi-modal data fusion by dynamically adjusting weight coefficients, which can optimize the priority of specific abnormalities according to the specific running state of the equipment and environmental context. For example, the system can prioritize the weight of relevant data sources based on specific issues such as high temperature or sound abnormalities to respond more quickly to these urgent situations. This ensures that the system can maintain efficient monitoring capabilities and respond promptly in complex and changing environments.

[0068] S4, combine the abnormal probability with the grading threshold to trigger an early warning signal.

[0069] It should be noted that the early warning signal includes three levels of early warning, and the alarm information is generated by combining the abnormal probability with the grading threshold.

[0070] Further, the three levels of early warning include high-risk early warning, medium-risk early warning, and low-risk early warning, represented as:

[0071] ,

[0072] wherein, represents the high-risk grading threshold, represents the low-risk grading threshold.

[0073] Further, when the high-risk early warning is triggered, the system will immediately start the device linkage control module and automatically regulate the device based on the abnormal probability and grading threshold. In the case of high risk, the belt speed will be automatically adjusted to alleviate potential equipment overload or abnormal state. At the same time, the system will display real-time monitoring images through a dynamic alarm video wall, timely push the hidden danger location coordinates to the monitoring terminal, and provide comprehensive visual information for operators to understand the specific location and impact range of the problem.

[0074] When the system identifies a medium risk, it will push a series of warning information to the monitoring terminal according to the preset weight coefficient and threshold, prompting the operator to check and intervene in time; this medium risk warning does not immediately trigger device regulation, but the system will provide detailed warning descriptions, including possible risk types, affected device parts, historical data comparison analysis, etc.; the operator will obtain this information for further confirmation; the risk level is dynamically adjusted according to the operator's feedback, and the subsequent decision is guided according to the inspection results.

[0075] Among them, dynamically adjusting the risk level includes threshold adjustment, data processing frequency, and monitoring parameters.

[0076] Among them, threshold adjustment includes: if the abnormal probability value of the device (such as temperature, vibration, noise, etc.) is usually high under certain environmental conditions, the high-risk threshold value should be appropriately increased; for example, if the temperature fluctuation is large, but it does not always cause device failure, the high-risk threshold value can be increased to avoid excessive triggering of high-risk alarms; if in some cases, if the temperature fluctuation is small, it may not affect long-term operation, at this time the low-risk threshold value can be appropriately reduced so that potential minor problems can be discovered earlier and adjusted in time.

[0077] Data processing frequency includes: when the abnormal probability is high, the system identifies a high-risk signal, the data acquisition frequency and processing frequency should be increased; by increasing the sensor sampling rate and accelerating data processing, any changes can be captured in real time, reducing reaction delay; in the low-risk stage, reduce the data acquisition and processing frequency to avoid wasting computing resources due to frequent processing of unnecessary data.

[0078] Monitoring parameters include: increasing monitoring accuracy or coverage in high-risk situations, adding more monitoring points, parameters, or data sources to ensure that problems can be captured at the earliest stage; for example, adding more temperature sensors and using more sensitive vibration sensors for accurate monitoring; in low-risk situations, reduce the number of monitoring points or monitoring range, reduce unnecessary monitoring pressure, and reduce the accuracy requirements of monitoring parameters.

[0079] In the low-risk situation, the system will record the warning information to ensure that the data is properly archived to provide a basis for subsequent analysis and decision-making; low-risk warning does not require immediate intervention measures, but the system will generate detailed historical data reports through the intelligent analysis platform for subsequent system evaluation, performance optimization or strategy adjustment; in this stage, the operator views historical trends, monitoring data and prediction results to provide data support for future monitoring and risk control.

[0080] S5, improve detection accuracy through dynamic distribution.

[0081] It should be noted that dynamic distribution optimizes monitoring strategy, which means:

[0082] ,

[0083] wherein, represents the dynamic distribution optimization monitoring strategy, represents triggering a high-priority dynamic distribution strategy in a high-risk situation, represents taking a medium-priority response in a medium-risk situation, represents taking a low-priority response in a low-risk situation, represents parameter adjustment for high-risk warning, represents operating parameters for medium-risk warning, represents operating parameters for low-risk warning, represents dynamic distribution parameters.

[0084] Further, when the system detects a high-risk state, the system will automatically start a high-priority dynamic distribution strategy, which ensures real-time collection of each key parameter by increasing the data sampling frequency, especially for important indicators such as temperature, pressure, and vibration for more frequent monitoring; In this process, high-risk data is prioritized and dynamically responded to based on real-time data; When an anomaly occurs, automatically adjust resource allocation, mobilize more computing power for real-time analysis to quickly locate the problem and execute preventive measures; For example, if the device temperature rises sharply, the system will automatically increase the sampling frequency of the temperature sensor and analyze the collected data in real time to ensure timely response to potential device failures.

[0085] In a medium-risk state, the system will enter a medium-priority response mode; At this time, the system will optimize the sampling rate of the sensor, reduce the system load while ensuring data quality; Dynamically adjust resource allocation according to real-time feedback; For example, preferentially schedule data collection devices with high load to ensure that important data is collected and processed first; In this phase, the operator will receive optimized data push, and can decide whether to intervene manually or continue to observe the system state; The system automatically pushes risk analysis reports and warning information to remind the operator of potential risks in some devices or links, helping the operator to make timely decisions.

[0086] In a low-risk state, the system reduces unnecessary resource consumption through a low-priority response mode to ensure efficient operation of the system without excessive resource waste; Automatically reduce the load, optimize the data transmission efficiency, reduce the network burden and improve the speed and accuracy of data transmission; The data collection frequency will also be dynamically reduced according to the demand to improve the overall operating efficiency of the system.

[0087] The above is a schematic scheme of the coal flow multi-modal autonomous perception monitoring method of the embodiment. It should be noted that the technical scheme of the coal flow multi-modal autonomous perception monitoring system is the same as the technical scheme of the coal flow multi-modal autonomous perception monitoring method described above, and the details of the technical scheme of the coal flow multi-modal autonomous perception monitoring system in the embodiment are not described in detail, which can be referred to the description of the technical scheme of the coal flow multi-modal autonomous perception monitoring method.

[0088] Embodiment 2

[0089] Referring to Figure 2 For the second embodiment of the present application, which is different from the previous embodiment, a coal flow multi-modal autonomous perception monitoring system is provided, comprising: a data acquisition module 100, a feature extraction module 200, an anomaly detection module 300, a pre-warning linkage module 400 and a monitoring optimization module 500.

[0090] The data acquisition module 100 is used to deploy equipment for real-time acquisition to generate a multi-modal data set.

[0091] The feature extraction module 200 is used to pre-process and extract features from the multi-modal data to obtain a joint representation vector.

[0092] The anomaly detection module 300 is used to generate an anomaly probability based on cloud-edge collaboration.

[0093] The pre-warning linkage module 400 is used to combine the anomaly probability with a hierarchical threshold to trigger a pre-warning signal.

[0094] The monitoring optimization module 500 is used to combine the pre-warning signal to optimize the monitoring strategy by adjusting the dynamic distribution, thereby improving the coal flow perception monitoring accuracy.

[0095] The embodiment also provides a computing device suitable for a coal flow multi-modal autonomous perception monitoring method, comprising:

[0096] A memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the coal flow multi-modal autonomous perception monitoring method as proposed in the above embodiment.

[0097] The embodiment also provides a storage medium having a computer program stored thereon, which is executed by a processor to realize the coal flow multi-modal autonomous perception monitoring method as proposed in the above embodiment.

[0098] The storage medium proposed in the embodiment and the coal flow multi-modal autonomous perception monitoring method proposed in the above embodiment belong to the same inventive concept, and the technical details not described in detail in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0099] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0100] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the purpose of this specification, "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0101] More specific examples (non-exhaustive list) of the computer readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CD ROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0102] It should be understood that various parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits (ASICs) having appropriate combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0103] Example 3

[0104] Reference Figure 3 In a third embodiment of the present application, a coal flow multi-modal autonomous perception monitoring method is provided. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiments are used for scientific demonstration.

[0105] Experimental steps:

[0106] Set up experimental group: through dynamic distribution optimization monitoring and optimization of federal learning framework, i.e. the method of the present application, so that the coal flow transportation system can realize high-precision anomaly detection in complex and changeable environment.

[0107] Set up control group: compare the differences in anomaly detection accuracy, response time, etc. between traditional monitoring method and system based on federal learning framework.

[0108] Experimental data collection, as shown in Table 1.

[0109] Table 1 Experimental data

[0110] ,

[0111] Experimental evaluation:

[0112] Data collection and preprocessing:

[0113] The same data collection method is used in the experimental group and the control group to ensure the accuracy of environmental change data (such as temperature, humidity, etc.).

[0114] Feature extraction and training:

[0115] Deep learning algorithm is used to extract multi-modal data features in the experimental group.

[0116] In the control group, traditional threshold method is used for feature extraction and anomaly detection.

[0117] Model training and real-time updating:

[0118] The experimental group uses a federated learning framework for real-time model updating and parameter adjustment.

[0119] The control group uses a fixed threshold method for monitoring.

[0120] Evaluation and comparison:

[0121] By comparing the detection accuracy, response speed, false positive rate, etc. of the two groups, the performance difference between the experimental group and the control group is evaluated.

[0122] The experimental results are analyzed as shown in Figure 3 : Detection accuracy comparison: The experimental group can effectively adapt to environmental changes by dynamically adjusting the threshold and real-time model updating, achieving a detection accuracy of 95% in complex environments such as high temperature, high humidity, and overload. The control group has a lower detection accuracy due to the use of fixed models and threshold methods, only achieving 80% in these environmental changes.

[0123] Abnormal probability:

[0124] The experimental group can dynamically adjust the model parameters according to the changing environmental factors (such as temperature, humidity, etc.), thereby reducing the false positive rate and improving the accuracy of detection.

[0125] The control group has a lower abnormal detection accuracy due to the inability to adjust the model according to environmental changes, resulting in a higher false positive rate.

[0126] Overall, detection accuracy: The experimental group should be significantly better than the control group, especially in complex environments (such as high temperature, high humidity, and overload), because the experimental group can adapt to environmental changes by dynamically adjusting the threshold and real-time model updating. Response time: The experimental group achieves fast response through edge computing and federated learning framework, reducing the delay, while the control group may have higher response delay. False positive rate: The experimental group has a lower false positive rate due to dynamic optimization of model parameters, while the control group has a relatively high false positive rate.

[0127] The dynamic optimization method of the experimental group significantly improves the abnormal detection accuracy of the coal flow transportation system, especially in complex environments. The control group uses a fixed monitoring method that can effectively adapt to environmental changes, thereby improving detection accuracy.

[0128] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be included in the scope of the claims of the present application.

Claims

1. A coal flow multi-modal autonomous perception monitoring method, characterized in that: comprising, deploying devices to collect in real time, generating a multi-modal dataset; preprocessing and feature extraction of multi-modal data to obtain joint representation vectors; the feature extraction includes using convolutional neural networks to extract belt surface damage texture features and temperature gradient features, using short-time Fourier transform and mel-frequency cepstral coefficients to extract sound spectrum features; based on cloud-edge collaboration, generating an anomaly probability; the anomaly probability includes utilizing cloud-edge collaboration, that is, fusing the detection results obtained by the real-time target detection algorithm at the edge with the results obtained by the residual network analysis at the cloud, to calculate the anomaly probability; combining the anomaly probability with the grading threshold to trigger an early warning signal; combined with the early warning signal, the monitoring strategy is optimized by adjusting the dynamic distribution, and the coal flow perception monitoring accuracy is improved; the multi-modal dataset includes images, temperature distribution, operating noise, belt speed, coal distribution, and roller vibration signals; the results of the preprocessing are represented as, wherein, represents a composite processing operation, represents a transformation of the original data, θ i represents a transformation parameter of the i-th data type D i , w i represents a weight coefficient of the transformation of the i-th data, n represents the number of operations, n = 6, i represents an index variable, D processed represents a result of pre-processing, D i represents the i-th data type, including D1 image data, D2 temperature distribution, D3 running noise, D4 belt running speed, D5 coal quantity distribution, and D6 roller vibration signal; the joint representation vectors include belt surface damage texture feature vectors, temperature gradient feature vectors, and sound spectrum feature vectors; using convolutional neural networks to extract belt surface damage texture features and temperature gradient features is represented as, wherein θ cnn denotes a convolutional neural network parameter, denotes pre-processed image data, V texture denotes extracted band surface damage texture features, denotes pre-processed temperature distribution data, V temperature denotes extracted temperature gradient features, CNN denotes a convolutional neural network; using short-time Fourier transform and mel-frequency cepstral coefficients to extract sound spectrum features is represented as, V specturun = MFCC(S stft , θ mfcc ) wherein, represents pre-processed running noise data, θ stft represents a parameter of a short-time Fourier transform, θ mfcc represents a parameter of a mel-frequency cepstral coefficient, V spectrun represents a resulting sound spectrum feature, S stft represents a short-time Fourier transform result, STFT represents a short-time Fourier transform, and MFCC represents a mel-frequency cepstral coefficient. the calculation of the anomaly probability is represented as, wherein P YOLO represents the abnormal probability detected by the real-time target detection algorithm, P resnet represents the abnormal probability obtained by the residual network, ω1, ω2 represent the weighting coefficients, θ yolo represents the parameters of the real-time target detection algorithm, θ resnet represents the parameters of the residual network, P abnormal represents the abnormal probability; the detection results obtained by the real-time target detection algorithm include deploying a lightweight real-time target detection algorithm model at the edge, detecting abnormal targets in the belt surface image in real time, including foreign matter, large coal, and deviation, generating target detection results including target category, location, and confidence, and uploading to the cloud; the results obtained by the residual network analysis include receiving the detection results uploaded by the edge at the cloud, and combining thermal imaging and sound data for joint analysis, including using a residual network to extract temperature gradient features to judge local overheating risk; using a residual network to extract sound spectrum features to judge roller abnormal wear risk; generating abnormal probability distribution of thermal imaging and sound data; by fusing multi-dimensional data, the generated anomaly probability reflects the running state of the equipment, supports early warning and processing decisions, and then by dynamically adjusting the weight coefficient, the priority of a specific anomaly is optimized according to the specific running state of the equipment and the environmental situation, and the weight of the relevant data source is preferentially improved according to the specific problems of temperature overheating or sound anomaly; the early warning signal includes three-level early warning, and alarm information is generated by combining the anomaly probability with the grading threshold; the three-level early warning includes high-risk early warning, medium-risk early warning, and low-risk early warning, represented as, where T high represents a high risk classification threshold, T low represents a low risk classification threshold; When P abnormal > T high , it indicates high risk, triggers the device linkage control module, automatically adjusts the belt speed, and pushes the real-time picture and hidden danger location coordinates to the monitoring terminal through the dynamic alarm video wall. When P abnormal ≤ T high , it indicates a medium risk, and a warning information is pushed to the monitoring terminal to prompt the operator to check and intervene. When P abnormal <T low Low risk, record warning information for subsequent analysis. the dynamic distribution optimization monitoring strategy is represented as, wherein, represents a dynamic distribution optimization monitoring strategy, represents triggering a high priority dynamic distribution strategy in case of high risk, represents taking a medium priority response in case of medium risk, represents taking a low priority response in case of low risk, high represents parameter adjustment for high risk alert, medium represents operational parameters for medium risk alert, low represents operational parameters for low risk alert, represents a dynamic distribution parameter; when in high-risk, increase the sampling frequency, prioritize processing important data, and respond in real time, when an anomaly occurs, automatically adjust resource allocation, mobilize more computing power for real-time analysis, to quickly locate the problem and execute preventive measures; when in medium-risk, optimize the sensor sampling rate, adjust resource allocation, and prioritize scheduling data collection devices with high processing load to ensure that important data is collected and processed first; when in low-risk, reduce system load and optimize data transmission efficiency.

2. The coal flow multi-modal autonomous perception monitoring method of claim 1, characterized in that: According to the feedback, the risk level is dynamically adjusted, and the dynamic adjustment of the risk level includes threshold adjustment, data processing frequency, and monitoring parameters. The data processing frequency includes increasing the data acquisition frequency and processing frequency when a high-risk signal is identified; in the low-risk stage, the data acquisition and processing frequency are reduced to avoid wasting computing resources due to frequent processing of unnecessary data; The monitoring parameters include increasing the monitoring accuracy or coverage in high-risk situations, adding more monitoring points, parameters, or data sources; in low-risk situations, the monitoring points or monitoring range can be reduced to reduce unnecessary monitoring pressure and reduce the accuracy requirements of monitoring parameters, and the warning information is recorded to ensure that the data is properly archived.

3. A coal flow multi-modal autonomous perception monitoring system, applying a coal flow multi-modal autonomous perception monitoring method according to any one of claims 1-2, characterized in that: It includes a data acquisition module (100), a feature extraction module (200), an anomaly detection module (300), a warning linkage module (400), and a monitoring optimization module (500); The data acquisition module (100) is used to deploy equipment for real-time acquisition to generate a multi-modal data set; The feature extraction module (200) is used to preprocess and extract features from multi-modal data to obtain a joint representation vector; The anomaly detection module (300) is used to generate an anomaly probability based on cloud-edge collaboration; The warning linkage module (400) is used to combine the anomaly probability and the grading threshold to trigger a warning signal; The monitoring optimization module (500) is used to combine the warning signal to adjust the dynamic distribution and optimize the monitoring strategy to improve the coal flow perception monitoring accuracy.

4. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to realize the steps of the coal flow multi-modal autonomous perception monitoring method in any one of claims 1-2.

5. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the coal flow multi-modal autonomous perception monitoring method in any one of claims 1-2.

Images