Early warning and response method, system and device for coal preparation plant based on multi-modal analysis

Abstract

The application provides a coal preparation plant early warning and response method, system and device based on multi-modal analysis, comprising: acquiring multi-modal data corresponding to the coal preparation plant, and performing time and space alignment processing on the multi-modal data to obtain a multi-modal data sequence; performing cross-modal feature clustering on the multi-modal data sequence through a pre-constructed abnormal early warning network, identifying an abnormal event occurring in the coal preparation plant based on an abnormal event feature cluster obtained through the cross-modal feature clustering, and obtaining an abnormal early warning result; determining an event decision result corresponding to the abnormal event based on the abnormal early warning result and a coal preparation industry knowledge graph through a pre-constructed abnormal decision network, and performing emergency response on the abnormal event according to the event decision result. The application can significantly improve the early warning and response processing efficiency, reduce the false alarm rate, and also meet the needs of more special and complex business scenarios.

Description

Technical Field

[0001] The present invention relates to the field of safety monitoring technology, and in particular to a coal preparation plant early warning and response method, system and equipment based on multimodal analysis. Background Art

[0002] Traditional video surveillance systems primarily rely on manual review of camera footage for problem monitoring. In recent years, the empowerment of various technologies has enabled intelligent identification of abnormal scenarios through video footage. This technology has been widely adopted across various industries, with the most common application scenarios including facial recognition at security barriers and identification of non-compliant drivers on roads. In coal preparation plant applications, the most common use cases are regular safety monitoring scenarios such as boundary intrusions, helmet violations, and personnel collapses. However, these current technologies primarily identify and issue alerts based on simple scenarios and a single video modality, resulting in a high false alarm rate. Intelligent monitoring capabilities are currently unavailable for specialized and complex business scenarios. After an alarm occurs, analysis and subsequent processing rely heavily on the experience of management personnel, resulting in low problem resolution efficiency. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a coal preparation plant early warning and response method, system and equipment based on multimodal analysis, which can significantly improve the efficiency of early warning and response processing, reduce the false alarm rate, and meet the needs of more special and complex business scenarios.

[0004] In a first aspect, the present invention provides a coal preparation plant early warning and response method based on multimodal analysis, comprising:

[0005] Acquire multimodal data corresponding to the coal preparation plant and perform temporal and spatial alignment processing on the multimodal data to obtain a multimodal data sequence; wherein the multimodal data includes video modal data, audio modal data, and text modal data, and the multimodal data sequence includes multimodal data of different spaces within the coal preparation plant at multiple time points;

[0006] Through the pre-built abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence. The abnormal event feature clusters obtained based on the cross-modal feature clustering are used to identify abnormal events occurring in the coal preparation plant and obtain abnormal warning results.

[0007] Through the pre-built abnormal decision network, based on the abnormal warning results and the coal preparation industry knowledge graph, the event decision results corresponding to the abnormal events are determined, so that emergency responses can be made to the abnormal events according to the event decision results.

[0008] In one embodiment, the abnormality warning network includes a multimodal conditional random field model, a cross-modal clustering model, and an abnormality warning model. The pre-built abnormality warning network performs cross-modal feature clustering on the multimodal data sequence, and uses abnormal event feature clusters obtained based on the cross-modal feature clustering to identify abnormal events occurring in the coal preparation plant, thereby obtaining abnormality warning results, including:

[0009] Through a multimodal conditional random field model, multimodal feature information of video modal data, audio modal data, and text modal data at the same time point and in the same space is extracted respectively; wherein the multimodal feature information includes video modal feature information, audio modal feature information, and text modal feature information;

[0010] Through the cross-modal clustering model, multimodal feature information is cross-modal clustered to obtain multiple abnormal event feature clusters; wherein the abnormal event feature cluster is a data cluster composed of multimodal feature information belonging to the same abnormal event;

[0011] Through the abnormal warning model, based on the multimodal feature information contained in the abnormal event feature cluster, the probability of abnormal events corresponding to the abnormal event feature cluster occurring in the coal preparation plant is determined, and the abnormal warning result is determined based on the probability corresponding to each abnormal event feature cluster.

[0012] In one embodiment, cross-modal feature clustering is performed on multimodal feature information to obtain multiple abnormal event feature clusters, including:

[0013] The video modal feature information, audio modal feature information and text modal feature information are mapped to a common subspace, and the similarity between the video modal feature information, audio modal feature information and text modal feature information is determined in the common subspace, so as to divide the video modal feature information, audio modal feature information and text modal feature information belonging to the same abnormal event into the same data cluster based on the similarity, thereby obtaining multiple abnormal event feature clusters.

[0014] In one embodiment, a pre-built abnormal decision network is used to determine the event decision result corresponding to the abnormal event based on the abnormal warning result and the coal preparation industry knowledge graph, including:

[0015] Based on the coal preparation industry knowledge graph, determine the set of alternative abnormal causes and alternative strategies corresponding to the abnormal events included in the abnormal warning results; the coal preparation industry knowledge graph is used to describe the relationship between abnormal events, their causes, and their response strategies;

[0016] Through the pre-built abnormal decision network, based on the abnormal event feature cluster, alternative abnormal cause set and alternative strategy set corresponding to the abnormal event, the target abnormal cause and target response strategy corresponding to the abnormal event are determined as the event decision result corresponding to the abnormal event.

[0017] In one embodiment, determining a target abnormal cause and a target response strategy corresponding to the abnormal event based on the abnormal event feature cluster, the set of candidate abnormal causes, and the set of candidate strategies corresponding to the abnormal event includes:

[0018] Based on the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, the target historical abnormal events are roughly screened out;

[0019] Perform feature encoding and vector concatenation on the abnormal event feature cluster corresponding to the abnormal event, the alternative abnormal causes included in the alternative abnormal cause set, and the alternative response strategies included in the alternative strategy set to construct multiple current multi-dimensional vectors corresponding to the abnormal event; and perform feature encoding and vector concatenation on the abnormal event feature cluster, target abnormal cause, and target response strategy corresponding to the target historical abnormal event to construct a historical multi-dimensional vector;

[0020] Based on multiple current multi-dimensional vectors and historical multi-dimensional vectors, the target abnormal causes and target response strategies corresponding to abnormal events are carefully screened.

[0021] In one embodiment, performing time alignment processing on multimodal data includes:

[0022] A fixed-length window is slid on the time axis to associate all multimodal data within the window to the same time point.

[0023] In one embodiment, the video modality data includes: monitoring data collected by multispectral cameras deployed at various equipment in the coal preparation plant and monitoring data collected by inspection equipment;

[0024] Audio modal data include: sound data collected by microphones deployed in the coal preparation plant;

[0025] The text modal data includes: the equipment status and equipment parameters corresponding to each equipment in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and pre-configured safety operation specifications.

[0026] In a second aspect, the present invention further provides a coal preparation plant early warning and response system based on multimodal analysis, comprising:

[0027] The data acquisition and processing module is configured to acquire multi-modal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multi-modal data to obtain a multi-modal data sequence; wherein the multi-modal data comprises video modal data, audio modal data and text modal data, and the multi-modal data sequence comprises multi-modal data of different spaces in the coal preparation plant at multiple time points;

[0028] The anomaly early warning module is configured to perform cross-modal feature clustering on the multi-modal data sequence by using a pre-constructed anomaly early warning network, to identify an abnormal event occurring in the coal preparation plant based on an abnormal event feature cluster obtained through the cross-modal feature clustering, and to obtain an anomaly early warning result.

[0029] The anomaly emergency response module is configured to determine an event decision result corresponding to the abnormal event based on the anomaly early warning result and a coal industry knowledge graph by using a pre-constructed anomaly decision network, and to perform emergency response to the abnormal event according to the event decision result.

[0030] In a third aspect, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.

[0031] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer executable instructions, and the computer executable instructions, when called and executed by a processor, cause the processor to implement the method of any one of the first aspect.

[0032] The present invention provides a coal preparation plant early warning and response method, system and equipment based on multimodal analysis. First, multimodal data corresponding to the coal preparation plant is obtained, and time and space alignment processing is performed on the multimodal data to obtain a multimodal data sequence, where the multimodal data includes video modal data, audio modal data and text modal data, and the multimodal data sequence includes multimodal data of different spaces in the coal preparation plant at multiple time points; then, cross-modal feature clustering is performed on the multimodal data sequence through a pre-constructed abnormal early warning network, and abnormal events occurring in the coal preparation plant are identified based on abnormal event feature clusters obtained based on cross-modal feature clustering to obtain abnormal early warning results; finally, based on the abnormal early warning results and the coal preparation industry knowledge graph, an event decision result corresponding to the abnormal event is determined through a pre-constructed abnormal decision network, so that an emergency response is performed for the abnormal event according to the event decision result. The above method obtains multimodal information corresponding to the coal preparation plant and utilizes the cross-validation and complementary characteristics of multimodal data to improve environmental adaptability and anti-interference ability. On this basis, the anomaly warning network is used to perform cross-modal feature clustering on the multimodal time series to identify anomaly warning results. The anomaly decision network is further used, combined with the coal preparation industry knowledge graph to derive specific event resolution strategies. This can not only significantly improve the efficiency of early warning and response processing and reduce the false alarm rate, but also meet the needs of more special and complex business scenarios.

[0033] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 A schematic flow chart of a coal preparation plant early warning and response method based on multimodal analysis provided by an embodiment of the present invention;

[0037] Figure 2 A technical framework diagram of a coal preparation plant early warning and response method based on multimodal analysis provided by an embodiment of the present invention;

[0038] Figure 3 A schematic structural diagram of a coal preparation plant early warning and response system based on multimodal analysis provided by an embodiment of the present invention;

[0039] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0041] At present, traditional video surveillance systems have the following problems: (1) Serious data silos: Video, sensor, production environment, operation behavior and other data are not integrated into the scene, and cross-modal abnormality judgment cannot be achieved. For example, when a maintenance worker stands on a sieve, it is not possible to determine whether the equipment has been powered off. In the case of power off, it is a safe behavior, but in the case of power on, it is an unsafe behavior; (2) Rigid rule engine: The alarm strategy based on fixed thresholds cannot adapt to dynamic scenarios. For example, there are differences in room temperature between winter and summer, day and night, and the optimal alarm threshold cannot be adjusted based on environmental adaptation; (3) Single intelligent monitoring scenario: When performing hot work on site, if the operator has multiple unsafe behaviors, they can only be detected one by one based on the recognition capability of the camera deployment. The recognition capability is limited. In an actual scenario, only the recognition function of not wearing a helmet is usually deployed. If the operator is not equipped with fire extinguishing equipment, the distance between oxygen and acetylene cylinders is less than the safe distance, or the operator is not wearing fire-resistant gloves, detection and alarm cannot be achieved; (4) Delayed emergency response: Abnormal events rely on manual decision-making and lack automated handling links.

[0042] Based on this, the implementation of the present invention provides a coal preparation plant early warning and response method, system and equipment based on multimodal analysis, which can significantly improve the efficiency of early warning and response processing, reduce the false alarm rate, and meet the needs of more special and complex business scenarios.

[0043] To facilitate understanding of this embodiment, firstly, a coal preparation plant early warning and response method based on multimodal analysis disclosed in an embodiment of the present invention is introduced in detail, see Figure 1 The flowchart of a coal preparation plant early warning and response method based on multimodal analysis is shown, and the method mainly includes the following steps S102 to S106:

[0044] Step S102: Acquire multimodal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multimodal data to obtain a multimodal data sequence.

[0045] Multimodal data includes video, audio, and text. Video data includes monitoring data collected by multispectral cameras and inspection equipment deployed at various equipment within the coal preparation plant. Audio data includes sound data collected by microphones deployed within the plant. Text data includes the status and parameters of each piece of equipment within the plant, the current environmental status of the plant, and pre-configured safety operating procedures. Multimodal data sequences include multimodal data from different spaces within the plant at multiple time points.

[0046] In one example, the multimodal data corresponding to the coal preparation plant can be obtained through various methods such as equipment collection and manual uploading. Since the uploaded multimodal data has different time and space, it is necessary to perform time and space alignment processing on it to obtain multimodal data in different spaces at multiple time points.

[0047] In step S104, cross-modal feature clustering is performed on the multimodal data sequence through the pre-built abnormal warning network, and abnormal events occurring in the coal preparation plant are identified based on the abnormal event feature clusters obtained by cross-modal feature clustering to obtain abnormal warning results.

[0048] The anomaly warning network includes a multimodal conditional random field model, a cross-modal clustering model, and an anomaly warning model. Anomaly event feature clusters are data clusters composed of multimodal feature information belonging to the same abnormal event. Anomaly warning results include abnormal events occurring within the coal preparation plant and their probability of occurrence. In one example, after a multimodal data sequence is input into the anomaly warning network, the conditional random field model is used to extract the multimodal feature information from the multimodal data sequence. The cross-modal clustering model then performs cross-modal feature clustering on the multimodal feature information to generate abnormal event feature clusters. The anomaly warning model then generates an anomaly warning result based on the abnormal event feature clusters.

[0049] Step S106: Determine the event decision result corresponding to the abnormal event through the pre-built abnormal decision network based on the abnormal warning result and the coal preparation industry knowledge graph, so as to perform an emergency response to the abnormal event according to the event decision result.

[0050] The coal preparation industry knowledge graph is used to describe the relationships between abnormal events, their causes, and their response strategies. In one example, the coal preparation industry knowledge graph can be used to identify a set of candidate causes and strategies corresponding to the abnormal events included in the abnormal warning results. The abnormal decision network, combined with the aforementioned abnormal event feature clusters, can then be used to assess the target abnormal cause and target response decision, which can be used as the event decision-making result to guide emergency response.

[0051] The coal preparation plant early warning and response method based on multimodal analysis provided by the embodiment of the present invention obtains multimodal information corresponding to the coal preparation plant and utilizes the cross-validation and complementary characteristics between multimodal data to improve environmental adaptability and anti-interference ability. On this basis, the abnormal early warning network is used to perform cross-modal feature clustering on the multimodal time series to identify abnormal early warning results. The abnormal decision network is further used, combined with the coal preparation industry knowledge graph to derive specific event resolution strategies. This can not only significantly improve the efficiency of early warning and response processing and reduce the false alarm rate of alarms, but also meet the needs of more special and complex business scenarios.

[0052] The present invention provides a coal preparation plant early warning and response method based on multimodal analysis, a multimodal safety supervision system integrating video surveillance, Internet of Things sensing, edge computing and large model reasoning, which is suitable for safety supervision scenarios such as coal preparation plant personnel behavior compliance monitoring and equipment operation status early warning. Figure 2 The following is a technical framework diagram of a coal preparation plant early warning and response method based on multimodal analysis. This method is applied to a multimodal safety supervision system that integrates the four-layer architecture of "perception-analysis-decision-execution".

[0053] The perception layer deploys explosion-proof multispectral cameras (visible light + thermal imaging), on-site monitoring sensors for various production links, microphones, etc. to collect multimodal data, such as Figure 2 As shown in , it includes video modal data and other modal data. Video modal data includes monitoring data collected by multispectral cameras deployed at gun cameras, dome cameras, and control dome cameras, as well as monitoring data collected by inspection equipment such as inspection robots and inspection recorders. Other modal data also include audio modal data and text modal data, such as Figure 2 As shown in the figure, the analog signals (including equipment status and equipment parameters) and text collected by the production process detection sensors are all text modal data.

[0054] The analysis layer Figure 2 The multimodal fusion analysis system shown in the figure is used to integrate multimodal feature fusion algorithms based on spatiotemporal alignment, such as video, audio, and text, and conduct comprehensive analysis based on the surrounding working environment and equipment operating status to generate abnormal warning results.

[0055] The decision-making level Figure 2 The large model intelligent early warning system shown in the background art is used for analysis and decision-making in combination with the coal preparation industry knowledge graph and the abnormal decision network to generate event decision results. The abnormal decision network can adopt an LLM (Large Language Model) network.

[0056] The execution layer is also the intelligent interaction terminal / system shown in the background art, which is used for linkage with a PLC (Programmable Logic Controller) control system, an audible and visual alarm, and various intelligent interaction terminals to realize millisecond-level emergency response. Figure 2

[0057] The system can realize online real-time inference detection of various scenes such as personnel safety and equipment abnormality in the coal preparation production site through the perception layer, the data analysis layer, the inference decision layer, and the execution layer of information collection. The detection information is combined with the site device working condition and the surrounding environment for comprehensive analysis and judgment, and then the actual existing safety problems on the site are inferred to realize real-time safety supervision of the coal preparation production process.

[0058] On the basis of the foregoing embodiments, the present embodiment provides a specific implementation of a coal preparation plant early warning and response method based on multi-modal analysis.

[0059] (I) Obtain multi-modal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multi-modal data to obtain a multi-modal data sequence.

[0060] The video modal data is collected by a multispectral camera. The multispectral camera mainly serves as a medium for video information collection. A conventional video collection terminal is mainly directed to a fixedly installed gun camera or a ball camera. The collected video pictures are returned to the system. In addition to the conventional camera, the multispectral camera covered in the present embodiment also includes a mobile surveillance ball camera, a patrol robot, a recorder, and other mobile video collection terminals. The multispectral camera can cover various fixed scenes and mobile monitoring scenes to realize all-around, dead-angle-free, dynamic and static combined video collection capability.

[0061] The biggest difference between multi-modal analysis and a conventional video monitoring system is that multi-modal analysis is based on comprehensive analysis of multiple modal data, including but not limited to text, image, video, audio, sensor data, space-time information, and the like. Therefore, the present embodiment also needs to collect other information related to the scene, such as the running state of the equipment, whether it is powered on, the system coal carrying capacity, and various production data information. In addition, the present embodiment also needs to collect various environmental and spatial data information such as the environmental temperature and the geographical position. The various multi-modal data information required above can be obtained in real time through production link detection sensors.

[0062] ​Multimodal visual analysis offers significant advantages over single-modality analysis by fusing multiple data sources, including images, text, and voice. Multimodal data can be cross-validated and complementary, enhancing environmental adaptability and anti-interference capabilities, thereby improving the accuracy of judgments. Examples include infrared and visible light fusion for enhanced nighttime recognition, and comprehensive analysis and judgment of abnormal behavior and equipment operating status. Furthermore, cross-modal feature association mining can overcome the limitations of single-modal information and achieve more fine-grained semantic understanding. For example, combining motion and voice recognition in video analysis improves the accuracy of behavioral interpretation. Multimodal learning enhances model generalization through knowledge transfer, performing particularly well in data-scarce scenarios. However, single-modality analysis is susceptible to constraints such as data quality and environmental noise, making it difficult to support the intelligent analysis needs of complex scenarios.

[0063] The multimodal data collected above is aligned in both time and space. In the time dimension: a multi-source data fusion framework based on GNSS+PTP clock synchronization can be used to solve the spatial calibration problem of mobile inspection robot data and fixed camera data (alignment error <3ms). This can also address the time lag between real-time production data and the corresponding time of the video capture screen. A fixed-length window can also be slid on the time axis to associate all multimodal data within the window to the same time point for rough time alignment. In the spatial dimension: spatial matching and spatial interpolation processing can be performed based on the spatial alignment carried by the multimodal data to achieve alignment in the spatial dimension.

[0064] (2) Through the pre-built abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence, and abnormal events occurring in the coal preparation plant are identified by abnormal event feature clusters obtained based on cross-modal feature clustering, thereby obtaining abnormal warning results. Specifically, it includes:

[0065] (2.1) Using a multimodal conditional random field model, multimodal feature information is extracted from video, audio, and text modal data at the same time and in the same space. The term "conditional random field model" refers to a model based on a conditional random field algorithm and is used for image segmentation, video action recognition, speech recognition (integrating large language model information), and other applications. This extracts multimodal feature information, which includes video, audio, and text modal feature information.

[0066] (2.2) Using a cross-modal clustering model, cross-modal feature clustering is performed on the multimodal feature information to obtain multiple abnormal event feature clusters. In a specific embodiment, the cross-modal feature clustering process is as follows: video modality feature information, audio modality feature information, and text modality feature information are mapped to a common subspace, and similarities between the video modality feature information, audio modality feature information, and text modality feature information are determined within the common subspace. Based on the similarities, the video modality feature information, audio modality feature information, and text modality feature information belonging to the same abnormal event are grouped into the same data cluster to obtain multiple abnormal event feature clusters.

[0067] Specifically, the canonical correlation analysis (CCA) algorithm can be used to map the multimodal feature information output by the multimodal conditional random field model to a 256-dimensional common subspace. The cosine similarity is used in the common subspace to measure the similarity between the multimodal feature information. The multimodal feature information is then divided based on the similarity and a preset similarity threshold to obtain abnormal event feature clusters corresponding to different abnormal events.

[0068] (2.3) Using the anomaly warning model, based on the multimodal feature information contained in the abnormal event feature clusters, the probability of an abnormal event corresponding to each abnormal event feature cluster occurring in the coal preparation plant is determined. The anomaly warning result is then determined based on the probability corresponding to each abnormal event feature cluster. The anomaly warning model can employ an LSTM (Long Short-Term Memory) model, which takes abnormal event feature clusters as input and outputs the abnormal warning result.

[0069] In the embodiments of the present invention, safety is an important regulatory direction in coal preparation production management. Safety management includes three aspects: equipment safety, personnel safety, and environmental safety. The application scenarios of the embodiments of the present invention are mainly aimed at the above three aspects. The specific scenarios and monitoring rules are described as follows:

[0070] (I) Equipment Safety Monitoring: Equipment safety monitoring primarily targets various types of sorting equipment, dewatering equipment, and transport and transfer equipment in coal preparation plants. Monitoring items include, but are not limited to, uneven screen distribution, screen breakage, belt conveyor head blockage, belt spillage, and other production scenarios. Directed gun cameras, fixed dome cameras, and inspection robots are used to capture real-time operating images of each type of equipment. Multimodal technology is also used to combine video information with equipment status, operating parameters, and ambient temperature. The LSTM model dynamically learns equipment operating conditions and automatically corrects abnormal parameter thresholds (e.g., differentiated motor temperature thresholds for heavy and no-load modes). Multi-dimensional, multimodal data is used to comprehensively determine whether the current image information is abnormal, enabling intelligent monitoring and early warning at the production site.

[0071] (II) Personnel safety monitoring: Personnel safety monitoring mainly monitors the daily behavior of on-site production workers, and integrates multimodal information, including real-time equipment parameter information and safe operation specification texts. By interpreting the environment and personnel actions in the monitoring area, combined with the requirements for safe operation behavior specifications in different operation scenarios, it analyzes and identifies unsafe personnel behaviors. The achievable warning content includes but is not limited to: personnel not wearing safety helmets, hot work not complying with safe operation specifications, high-altitude work not complying with safe behavior specifications, live work, and other complex information scenarios, realizing intelligent supervision of personnel safety.

[0072] (III) Environmental safety monitoring: Environmental safety monitoring mainly focuses on the safety supervision of the production operation environment, including supervision of environmental stability, coal dust, hazardous gases, etc. At the same time, it can integrate text modal information. By monitoring real-time data on site and combining it with current production status information and industry standards and specifications, it can comprehensively analyze the safety risks in the current operating environment and realize intelligent supervision.

[0073] (3) Through the pre-built abnormal decision network, based on the abnormal warning results and the coal preparation industry knowledge map, determine the event decision results corresponding to the abnormal event. Specifically:

[0074] (3.1) Based on the coal preparation industry knowledge graph, determine the set of candidate abnormal cause and strategy corresponding to the abnormal events included in the abnormal warning results. In one example, since the coal preparation industry knowledge graph describes the relationship between abnormal events, their causes, and their response strategies, and the abnormal events included in the abnormal warning results are known, the set of candidate abnormal cause and strategy corresponding to each abnormal event can be directly extracted from the coal preparation industry knowledge graph.

[0075] (3.2) Using a pre-built abnormal decision network, based on the abnormal event feature cluster, the set of candidate abnormal causes, and the set of candidate strategies corresponding to the abnormal event, a target abnormal cause and a target response strategy corresponding to the abnormal event are determined as the event decision result corresponding to the abnormal event. In one embodiment:

[0076] (3.21) Based on the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, the target historical abnormal events are roughly screened. In one example, the cosine similarity can be used to calculate the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal event feature clusters.

[0077] (3.22) Feature encoding and vector concatenation are performed on the abnormal event feature cluster corresponding to the abnormal event, the alternative abnormal causes contained in the alternative abnormal cause set, and the alternative response strategies contained in the alternative strategy set to construct multiple current multi-dimensional vectors corresponding to the abnormal event; and feature encoding and vector concatenation are performed on the abnormal event feature cluster, target abnormal cause, and target response strategy corresponding to the target historical abnormal event to construct a historical multi-dimensional vector.

[0078] The process of feature encoding and vector concatenation of relevant information of abnormal events is as follows:

[0079] Perform one-hot encoding on each alternative cause, and use the same one-hot encoding method as the alternative cause set to concatenate the abnormal event feature cluster vector, the alternative cause encoding vector, and the alternative strategy encoding vector in sequence to obtain the current multi-dimensional vector.

[0080] The process of feature encoding and vector splicing for relevant information of historical abnormal events is as follows: one-hot encode the target abnormal cause, one-hot encode the target response strategy, and splice the historical abnormal event feature cluster vector, target cause encoding vector, and target strategy encoding vector to obtain a historical multi-dimensional vector.

[0081] (3.23) Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, target anomaly causes and target response strategies corresponding to the abnormal event are precisely screened. In one example, the Euclidean distance can be used to calculate the distance between the current multi-dimensional vector and each historical multi-dimensional vector, thereby screening the target anomaly causes and target response strategies corresponding to the abnormal event.

[0082] In an optional implementation, a coal preparation-specific knowledge graph and an abnormal decision network are trained and embedded with clauses from the "Safety Regulations for Coal Preparation Plants" to achieve end-to-end output from "violation identification" to "compliance recommendations." For example, when an excessively high coal pile is detected, the "Stacker-Reclaimer Operating Height Limitation" is automatically pushed and a three-dimensional stacking simulation diagram is generated, thereby achieving a closed-loop process of discovering, analyzing, and solving problems.

[0083] In this embodiment of the present invention, real-time online monitoring is enabled by collecting multi-dimensional, multimodal data, such as video information and equipment operating parameters, combined with various application scenarios in coal preparation plant production. Leveraging the system's knowledge graphs and large-scale model analysis and reasoning capabilities, the system accurately identifies safety hazards in the monitored area. Furthermore, by integrating public and private expert knowledge bases, including safety regulations and production management measures, the system analyzes the causes of identified anomalies and recommends solutions. This information can be pushed to end users via various intelligent interactive terminals. Violations can trigger AR glasses to overlay corrective operating instructions (such as demonstrating the correct use of a safety rope, such as hanging it high and using it low), further facilitating problem resolution. This large-scale model intelligent early warning system can identify potential risks, abnormal events, or key trends in real time or near real time, and proactively trigger early warnings or decision support to mitigate risks or improve response efficiency.

[0084] In summary, the core technical points of the embodiments of the present invention are: (1) combining video, surveillance cameras, inspection robots and recorders to collect video information, and then combining real-time production parameters and industry specifications, based on multimodal analysis and reasoning capabilities, to achieve safety supervision and emergency response of coal preparation plants; (2) monitoring various types of abnormal information and combining knowledge graphs and large models for reasoning, analyzing the causes of problems and providing problem-solving measures and regulatory requirements to achieve closed-loop management of problems; (3) the focus is on combining large model technology to realize a multimodal intelligent safety supervision and early warning system in the field of coal preparation production. On the basis of the above core technical points, the embodiment of the present invention combines multimodal fusion analysis with a large model enhanced with industry knowledge for the first time, overcoming the technical pain points of "difficult to perceive, slow to judge, and late to deal with" safety incidents in the complex environment of coal preparation plants. Compared with traditional methods, it has at least the following characteristics: (1) The missed reporting rate of personnel violation identification is greatly reduced: by integrating multiple video acquisition terminal devices, the production site monitoring is basically achieved without blind spots and without loopholes; (2) The accuracy of equipment fault warning is significantly improved: through the multimodal comprehensive analysis algorithm, some false alarms and misreporting problems are effectively shielded, which can greatly improve the accuracy of alarms; (3) The average time consumption of emergency response is greatly reduced: after the system detects an abnormal situation, it can realize online reasoning in seconds and quickly transmit information to the end user through the intelligent interactive system, which greatly shortens the information communication and flow link and greatly improves the timeliness of problem response; (4) The threshold of personnel skill level requirement is lowered: through the establishment of a proprietary knowledge base, knowledge spectrum and reasoning large model, the cause of the problem and the solution can be analyzed online according to the problem, which improves the efficiency of judgment and solves the problem of analysis relying on personnel experience.

[0085] Based on the above embodiments, the present invention provides a coal preparation plant early warning and response system based on multimodal analysis, see Figure 3The following is a schematic diagram of the structure of a coal preparation plant early warning and response system based on multimodal analysis. The system mainly includes the following parts:

[0086] The data acquisition and processing module 302 is used to acquire multimodal data corresponding to the coal preparation plant and perform temporal and spatial alignment processing on the multimodal data to obtain a multimodal data sequence; wherein the multimodal data includes video modal data, audio modal data, and text modal data, and the multimodal data sequence includes multimodal data of different spaces within the coal preparation plant at multiple time points;

[0087] The abnormality warning module 304 is configured to perform cross-modal feature clustering on the multimodal data sequence using a pre-built abnormality warning network, identify abnormal events occurring in the coal preparation plant based on abnormal event feature clusters obtained through cross-modal feature clustering, and obtain abnormality warning results;

[0088] The abnormal emergency response module 306 is used to determine the event decision result corresponding to the abnormal event based on the abnormal warning result and the coal preparation industry knowledge graph through a pre-built abnormal decision network, so as to perform an emergency response to the abnormal event according to the event decision result.

[0089] The coal preparation plant early warning and response system based on multimodal analysis provided by the embodiment of the present invention obtains multimodal information corresponding to the coal preparation plant and utilizes the cross-validation and complementary characteristics between multimodal data to improve environmental adaptability and anti-interference ability. On this basis, the abnormal early warning network is used to perform cross-modal feature clustering on the multimodal time series to identify abnormal early warning results. The abnormal decision network is further used, combined with the coal preparation industry knowledge graph to derive specific event resolution strategies. This can not only significantly improve the efficiency of early warning and response processing and reduce the false alarm rate of alarms, but also meet the needs of more special and complex business scenarios.

[0090] In one embodiment, the anomaly warning network includes a multimodal conditional random field model, a cross-modal clustering model, and an anomaly warning model; the anomaly warning module 304 is specifically used to:

[0091] Through a multimodal conditional random field model, multimodal feature information of video modal data, audio modal data, and text modal data at the same time point and in the same space is extracted respectively; wherein the multimodal feature information includes video modal feature information, audio modal feature information, and text modal feature information;

[0092] Through the cross-modal clustering model, multimodal feature information is cross-modal clustered to obtain multiple abnormal event feature clusters; wherein the abnormal event feature cluster is a data cluster composed of multimodal feature information belonging to the same abnormal event;

[0093] Through the abnormal warning model, based on the multimodal feature information contained in the abnormal event feature cluster, the probability of abnormal events corresponding to the abnormal event feature cluster occurring in the coal preparation plant is determined, and the abnormal warning result is determined based on the probability corresponding to each abnormal event feature cluster.

[0094] In one embodiment, the abnormality warning module 304 is specifically configured to:

[0095] The video modal feature information, audio modal feature information and text modal feature information are mapped to a common subspace, and the similarity between the video modal feature information, audio modal feature information and text modal feature information is determined in the common subspace, so as to divide the video modal feature information, audio modal feature information and text modal feature information belonging to the same abnormal event into the same data cluster based on the similarity, thereby obtaining multiple abnormal event feature clusters.

[0096] In one embodiment, the abnormal emergency response module 306 is specifically configured to:

[0097] Based on the coal preparation industry knowledge graph, determine the set of alternative abnormal causes and alternative strategies corresponding to the abnormal events included in the abnormal warning results; the coal preparation industry knowledge graph is used to describe the relationship between abnormal events, their causes, and their response strategies;

[0098] Through the pre-built abnormal decision network, based on the abnormal event feature cluster, alternative abnormal cause set and alternative strategy set corresponding to the abnormal event, the target abnormal cause and target response strategy corresponding to the abnormal event are determined as the event decision result corresponding to the abnormal event.

[0099] In one embodiment, the abnormal emergency response module 306 is specifically configured to:

[0100] Based on the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, the target historical abnormal events are roughly screened out;

[0101] Perform feature encoding and vector concatenation on the abnormal event feature cluster corresponding to the abnormal event, the alternative abnormal causes included in the alternative abnormal cause set, and the alternative response strategies included in the alternative strategy set to construct multiple current multi-dimensional vectors corresponding to the abnormal event; and perform feature encoding and vector concatenation on the abnormal event feature cluster, target abnormal cause, and target response strategy corresponding to the target historical abnormal event to construct a historical multi-dimensional vector;

[0102] Based on multiple current multi-dimensional vectors and historical multi-dimensional vectors, the target abnormal causes and target response strategies corresponding to abnormal events are carefully screened.

[0103] In one embodiment, the data acquisition and processing module 302 is specifically configured to:

[0104] A fixed-length window is slid on the time axis to associate all multimodal data within the window to the same time point.

[0105] In one embodiment, the video modality data includes: monitoring data collected by multispectral cameras deployed at various equipment in the coal preparation plant and monitoring data collected by inspection equipment;

[0106] Audio modal data include: sound data collected by microphones deployed in the coal preparation plant;

[0107] The text modal data includes: the equipment status and equipment parameters corresponding to each equipment in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and pre-configured safety operation specifications.

[0108] The system provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0109] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage system. The storage system stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0110] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present invention. The electronic device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43. The processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0111] Memory 41 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between the system network element and at least one other network element is achieved through at least one communication interface 43 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0112] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0113] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the system for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0114] Processor 40 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in processor 40. The above processor 40 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processing unit (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 41 , and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0115] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0116] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0117] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A coal preparation plant early warning and response method based on multimodal analysis, characterized in that: include: Acquire multimodal data corresponding to the coal preparation plant, and perform temporal and spatial alignment processing on the multimodal data to obtain a multimodal data sequence; wherein the multimodal data includes video modal data, audio modal data, and text modal data, and the multimodal data sequence includes the multimodal data at different spaces within the coal preparation plant at multiple time points; Through a pre-built abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence, and abnormal events occurring in the coal preparation plant are identified by abnormal event feature clusters obtained based on the cross-modal feature clustering, thereby obtaining abnormal warning results; Through a pre-built abnormal decision network, based on the abnormal warning results and the coal preparation industry knowledge graph, an event decision result corresponding to the abnormal event is determined, so as to perform an emergency response to the abnormal event according to the event decision result; The abnormal warning network includes a multimodal conditional random field model, a cross-modal clustering model and an abnormal warning model; through the pre-constructed abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature cluster obtained by cross-modal feature clustering, and obtain an abnormal warning result, including: through the multimodal conditional random field model, respectively extracting multimodal feature information of the video modal data, the audio modal data and the text modal data at the same time point and the same space; wherein the multimodal feature information includes video frequency modal feature information, audio modal feature information and text modal feature information; performing cross-modal feature clustering on the multimodal feature information through the cross-modal clustering model to obtain a plurality of abnormal event feature clusters; wherein the abnormal event feature cluster is a data cluster composed of the multimodal feature information belonging to the same abnormal event; determining the probability of the abnormal event corresponding to the abnormal event feature cluster occurring in the coal preparation plant through the abnormal early warning model based on the multimodal feature information contained in the abnormal event feature cluster, and determining the abnormal early warning result based on the probability corresponding to each abnormal event feature cluster; Cross-modal feature clustering is performed on the multimodal feature information to obtain multiple abnormal event feature clusters, including: mapping the video modal feature information, the audio modal feature information, and the text modal feature information to a common subspace, determining the similarity between the video modal feature information, the audio modal feature information, and the text modal feature information in the common subspace, and dividing the video modal feature information, the audio modal feature information, and the text modal feature information belonging to the same abnormal event into the same data cluster based on the similarity, to obtain multiple abnormal event feature clusters.

2. The coal preparation plant early warning and response method based on multimodal analysis according to claim 1 is characterized in that: Through the pre-built abnormal decision network, based on the abnormal warning results and the coal preparation industry knowledge graph, the event decision result corresponding to the abnormal event is determined, including: Based on the coal preparation industry knowledge graph, determining a set of alternative abnormal causes and a set of alternative strategies corresponding to the abnormal events included in the abnormal warning results; wherein the coal preparation industry knowledge graph is used to describe the association between abnormal events, causes of abnormal events, and response strategies for abnormal events; Through a pre-constructed abnormal decision network, based on the abnormal event feature cluster, the set of alternative abnormal causes and the set of alternative strategies corresponding to the abnormal event, the target abnormal cause and target response strategy corresponding to the abnormal event are determined as the event decision result corresponding to the abnormal event.

3. The coal preparation plant early warning and response method based on multimodal analysis according to claim 2 is characterized in that: Determining a target abnormal cause and a target response strategy corresponding to the abnormal event based on the abnormal event feature cluster, the set of candidate abnormal causes, and the set of candidate strategies corresponding to the abnormal event includes: Based on the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, roughly screening out the target historical abnormal events; Performing feature coding and vector concatenation on the abnormal event feature cluster corresponding to the abnormal event, the alternative abnormal causes included in the alternative abnormal cause set, and the alternative response strategies included in the alternative strategy set to construct a plurality of current multi-dimensional vectors corresponding to the abnormal event; and performing feature coding and vector concatenation on the abnormal event feature cluster, target abnormal cause, and target response strategy corresponding to the target historical abnormal event to construct a historical multi-dimensional vector; Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, target abnormal causes and target response strategies corresponding to the abnormal events are precisely screened out.

4. The coal preparation plant early warning and response method based on multimodal analysis according to claim 1, characterized in that: Performing time alignment processing on the multimodal data includes: A fixed-length window is slid on the time axis to associate all multimodal data within the window to the same time point.

5. The coal preparation plant early warning and response method based on multimodal analysis according to claim 1, characterized in that: The video modality data includes: monitoring data collected by multispectral cameras deployed at various equipment in the coal preparation plant and monitoring data collected by inspection equipment; The audio modal data includes: sound data collected by microphones deployed in the coal preparation plant; The text modal data includes: equipment status and equipment parameters corresponding to each equipment in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and pre-configured safety operation specifications.

6. A coal preparation plant early warning and response system based on multimodal analysis, characterized in that: include: a data acquisition and processing module, configured to acquire multimodal data corresponding to the coal preparation plant and perform temporal and spatial alignment processing on the multimodal data to obtain a multimodal data sequence; wherein the multimodal data includes video modal data, audio modal data, and text modal data, and the multimodal data sequence includes the multimodal data at different spaces within the coal preparation plant at multiple time points; an abnormality warning module, configured to perform cross-modal feature clustering on a multimodal data sequence using a pre-built abnormality warning network, identify abnormal events occurring in the coal preparation plant based on abnormal event feature clusters obtained through the cross-modal feature clustering, and obtain abnormality warning results; An abnormal emergency response module is used to determine the event decision result corresponding to the abnormal event based on the abnormal warning result and the coal preparation industry knowledge graph through a pre-built abnormal decision network, so as to perform an emergency response to the abnormal event according to the event decision result; The abnormal warning network includes a multimodal conditional random field model, a cross-modal clustering model and an abnormal warning model; the abnormal warning module is specifically used to: extract multimodal feature information of the video modal data, the audio modal data and the text modal data at the same time point and in the same space through the multimodal conditional random field model; wherein the multimodal feature information includes video modal feature information, audio modal feature information and text modal feature information; through the cross-modal clustering model, perform cross-modal feature clustering on the multimodal feature information to obtain multiple abnormal event feature clusters; wherein the abnormal event feature cluster is a data cluster composed of the multimodal feature information belonging to the same abnormal event; through the abnormal warning model, based on the multimodal feature information contained in the abnormal event feature cluster, determine the probability of the abnormal event corresponding to the abnormal event feature cluster occurring in the coal preparation plant, and determine the abnormal warning result based on the probability corresponding to each abnormal event feature cluster; The abnormal warning module is specifically used to: map the video modal feature information, the audio modal feature information and the text modal feature information to a common subspace, determine the similarity between the video modal feature information, the audio modal feature information and the text modal feature information in the common subspace, and divide the video modal feature information, the audio modal feature information and the text modal feature information belonging to the same abnormal event into the same data cluster based on the similarity, thereby obtaining multiple abnormal event feature clusters.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images