Coal preparation plant early warning and response method, system and equipment based on multi-modal analysis

Abstract

The invention provides a coal preparation plant early warning and response method, system and device based on multi-modal analysis, and the method comprises the steps: obtaining multi-modal data corresponding to a coal preparation plant, and carrying out the time and space alignment processing of the multi-modal data, and obtaining a multi-modal data sequence; performing cross-modal feature clustering on the multi-modal data sequence through a pre-constructed abnormality early warning network, and identifying an abnormal event occurring in the coal preparation plant based on an abnormal event feature cluster obtained by the cross-modal feature clustering to obtain an abnormality early warning result; and determining an event decision result corresponding to the abnormal event based on the abnormal early warning result and a coal preparation industry knowledge graph through a pre-constructed abnormal decision network, so as to carry out emergency response on the abnormal event according to the event decision result. According to the method, the early warning and response processing efficiency can be remarkably improved, the false alarm rate is reduced, and the requirements of more special and complex service scenes can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of safety monitoring, and in particular, to a warning and response method, system and device for coal preparation plants based on multimodal analysis. Background Art

[0002] Traditional video surveillance systems mainly rely on manual viewing of camera footage for problem monitoring. In recent years, with the empowerment of some technical means, the intelligent recognition function of abnormal scenes through video footage has been realized. This technology has been widely applied in various industries. The most common application scenarios include face recognition at security gates and identification of non-compliant driving on roads. In the application scenario of coal preparation plants, the most commonly used ones are conventional safety monitoring scenarios such as out-of-bounds alarms (such as perimeter intrusion), not wearing safety helmets, and personnel falling to the ground. However, at present, these technical means are basically based on simple scenes and single video modalities for problem recognition and alarm, resulting in a relatively high false alarm rate. For special and complex business scenarios, the current intelligent monitoring capabilities cannot be achieved. After an alarm occurs, the analysis and subsequent processing of the alarm basically rely on the experience of management personnel, and the problem handling efficiency is relatively low. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a warning and response method, system and device for coal preparation plants based on multimodal analysis, which can significantly improve the warning and response processing efficiency, reduce the false alarm rate of alarms, and also meet the needs of more special and complex business scenarios.

[0004] In a first aspect, the present invention provides a warning and response method for coal preparation plants based on multimodal analysis, including: Obtain multimodal data corresponding to a coal preparation plant, and perform time and space alignment processing on the multimodal data to obtain a multimodal data sequence; wherein, the multimodal data includes video modality data, audio modality data and text modality data, and the multimodal data sequence includes multimodal data at different spaces in a coal preparation plant at multiple time points; Through a pre-constructed abnormal warning network, perform cross-modal feature clustering on the multimodal data sequence to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by cross-modal feature clustering, and obtain an abnormal warning result; Through a pre-constructed abnormal decision network, based on the abnormal warning result and the knowledge graph of the coal preparation industry, determine the event decision result corresponding to the abnormal event, so as to perform an emergency response to the abnormal event according to the event decision result.

[0005] In one implementation, the anomaly warning network includes a multi-modal conditional random field model, a cross-modal clustering model, and an anomaly warning model; through the pre-constructed anomaly warning network, cross-modal feature clustering is performed on the multi-modal data sequence to identify the anomaly events occurring in the coal preparation plant based on the anomaly event feature clusters obtained from the cross-modal feature clustering, and the anomaly warning result is obtained, including: Through the multi-modal conditional random field model, multi-modal feature information of video modal data, audio modal data, and text modal data at the same time point and the same space is extracted respectively; among them, the multi-modal feature information includes video modal feature information, audio modal feature information, and text modal feature information; Through the cross-modal clustering model, cross-modal feature clustering is performed on the multi-modal feature information to obtain multiple anomaly event feature clusters; among them, the anomaly event feature cluster is a data cluster composed of multi-modal feature information belonging to the same anomaly event; Through the anomaly warning model, based on the multi-modal feature information included in the anomaly event feature cluster, the probability of the anomaly event corresponding to the anomaly event feature cluster occurring in the coal preparation plant is determined, and the anomaly warning result is determined based on the probability corresponding to each anomaly event feature cluster.

[0006] In one implementation, performing cross-modal feature clustering on the multi-modal feature information to obtain multiple anomaly event feature clusters includes: Mapping the video modal feature information, audio modal feature information, and text modal feature information to a common subspace, determining the similarity between the video modal feature information, audio modal feature information, and text modal feature information in the common subspace, and based on the similarity, dividing the video modal feature information, audio modal feature information, and text modal feature information belonging to the same anomaly event into the same data cluster to obtain multiple anomaly event feature clusters.

[0007] In one implementation, through the pre-constructed anomaly decision network, based on the anomaly warning result and the coal preparation industry knowledge graph, the event decision result corresponding to the anomaly event is determined, including: Based on the coal preparation industry knowledge graph, determining the set of alternative anomaly causes and the set of alternative strategies corresponding to the anomaly events included in the anomaly warning result; among them, the coal preparation industry knowledge graph is used to describe the association relationship between anomaly events, the causes of anomaly events, and the response strategies of anomaly events; Through the pre-constructed anomaly decision network, based on the anomaly event feature cluster, the set of alternative anomaly causes, and the set of alternative strategies corresponding to the anomaly event, the target anomaly cause and the target response strategy corresponding to the anomaly event are determined as the event decision result corresponding to the anomaly event.

[0008] In one implementation manner, determining a target abnormal cause and a target response strategy corresponding to an abnormal event based on an abnormal event feature cluster, an alternative abnormal cause set, and an alternative strategy set corresponding to the abnormal event includes: Based on the similarity between the abnormal event feature cluster corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, roughly screen out the target historical abnormal events; Perform feature encoding and vector splicing on the abnormal event feature cluster corresponding to the abnormal event, the alternative abnormal causes included in the alternative abnormal cause set, and the alternative response strategies included in the alternative strategy set to construct multiple current multi-dimensional vectors corresponding to the abnormal event; and perform feature encoding and vector splicing on the abnormal event feature cluster, the target abnormal cause, and the target response strategy corresponding to the target historical abnormal event to construct historical multi-dimensional vectors; Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, finely screen out the target abnormal cause and the target response strategy corresponding to the abnormal event.

[0009] In one implementation manner, performing time alignment processing on multimodal data includes: Slide a window of a fixed length on the time axis, and associate all the multimodal data reached within the window to the same time point.

[0010] In one implementation manner, the video modality data includes: monitoring data collected by multispectral cameras deployed at various devices in the coal preparation plant and monitoring data collected by inspection devices; The audio modality data includes: sound data collected by pickups deployed inside the coal preparation plant; The text modality data includes: the device status and device parameters corresponding to each device in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and the pre-configured safety operation specifications.

[0011] In a second aspect, the present invention further provides a coal preparation plant warning and response system based on multimodal analysis, including: A data acquisition and processing module, configured to acquire multimodal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multimodal data to obtain a multimodal data sequence; wherein, the multimodal data includes video modality data, audio modality data, and text modality data, and the multimodal data sequence includes multimodal data at different spaces in the coal preparation plant at multiple time points; An abnormal warning module, configured to perform cross-modal feature clustering on the multimodal data sequence through a pre-constructed abnormal warning network, so as to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by the cross-modal feature clustering, and obtain an abnormal warning result; An abnormal emergency response module is used to determine an event decision result corresponding to an abnormal event based on an abnormal warning result and a coal preparation industry knowledge graph through a pre-constructed abnormal decision network, so as to perform an emergency response to the abnormal event according to the event decision result.

[0012] In a third aspect, the present invention also provides an electronic device, including a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of the first aspect.

[0013] In a fourth aspect, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions cause the processor to implement the method according to any one of the first aspect.

[0014] A coal preparation plant warning and response method, system and device based on multimodal analysis provided by the present invention first obtains multimodal data corresponding to the coal preparation plant, performs time and space alignment processing on the multimodal data to obtain a multimodal data sequence. The multimodal data includes video modal data, audio modal data and text modal data, and the multimodal data sequence includes multimodal data at different spaces in the coal preparation plant at multiple time points; then through a pre-constructed abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by cross-modal feature clustering, and an abnormal warning result is obtained; finally, through a pre-constructed abnormal decision network, based on the abnormal warning result and the coal preparation industry knowledge graph, an event decision result corresponding to the abnormal event is determined, so as to perform an emergency response to the abnormal event according to the event decision result. The above method improves the environmental adaptability and anti-interference ability by obtaining the multimodal information corresponding to the coal preparation plant and using the cross-validation and complementarity characteristics between multimodal data. On this basis, an abnormal warning network is used to perform cross-modal feature clustering on the multimodal time series to identify the abnormal warning result, and further an abnormal decision network is used to combine the coal preparation industry knowledge graph to obtain specific event solution strategies, which can not only significantly improve the warning and response processing efficiency and reduce the false alarm rate of alarms, but also meet the needs of more special and complex business scenarios.

[0015] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the specification, claims and drawings.

[0016] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. Brief Description of the Drawings

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic flow chart of a coal preparation plant warning and response method based on multimodal analysis provided by an embodiment of the present invention; Figure 2 It is a technical framework diagram of a coal preparation plant warning and response method based on multimodal analysis provided by an embodiment of the present invention; Figure 3 It is a schematic structural diagram of a coal preparation plant warning and response system based on multimodal analysis provided by an embodiment of the present invention; Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Description of the Embodiments

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0020] Currently, the traditional video surveillance system has the following problems: (1) Serious data islands: Data such as videos, sensors, production environments, and operation behaviors are not scene-integrated, and cross-modal anomaly judgment cannot be achieved. For example, when maintenance personnel stand on the sieve, it is not linked whether the equipment has been powered off. In the case of power off, it is a safe behavior, and in the case of no power off, it belongs to an unsafe behavior; (2) Rigid rule engine: The alarm strategy based on fixed thresholds cannot adapt to dynamic scenarios. For example, the room temperature in winter and summer, day and night is different, and the optimal alarm threshold cannot be adaptively adjusted based on the environment; (3) Single intelligent monitoring scenario: When hot work is carried out on-site and the operator has multiple unsafe behavior operations, it can only be detected one by one according to the recognition ability of the camera deployment, and the recognition ability is limited. In an actual scenario, usually only the function of not wearing a safety helmet is deployed. If the operator does not configure fire extinguishing equipment, the distance between the oxygen and acetylene cylinders is less than the safe distance, and the operator does not wear fire-resistant gloves, etc., detection and alarm cannot be achieved; (4) Lag in emergency response: Abnormal events rely on manual decision-making, lacking an automated disposal link.

[0021] Based on this, the embodiments of the present invention provide a coal preparation plant warning and response method, system and device based on multimodal analysis, which can significantly improve the warning and response processing efficiency, reduce the false alarm rate of alarms, and can also meet the requirements of more special and complex business scenarios.

[0022] For the convenience of understanding this embodiment, first, a coal preparation plant warning and response method based on multimodal analysis disclosed in the embodiments of the present invention will be introduced in detail. Refer to Figure 1 the schematic flowchart of a coal preparation plant warning and response method based on multimodal analysis shown in the figure. This method mainly includes the following steps S102 to S106: Step S102, obtain the multimodal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multimodal data to obtain a multimodal data sequence.

[0023] Among them, the multimodal data includes video modal data, audio modal data and text modal data. The video modal data includes: monitoring data collected by multispectral cameras deployed at each device in the coal preparation plant and monitoring data collected by inspection devices; the audio modal data includes: sound data collected by pickups deployed in the coal preparation plant; the text modal data includes: the device status and device parameters corresponding to each device in the coal preparation plant, as well as the current environmental status corresponding to the coal preparation plant, and the pre-configured safety operation specifications. The multimodal data sequence includes multimodal data at different spaces in the coal preparation plant at multiple time points.

[0024] In one example, the multimodal data corresponding to the coal preparation plant can be obtained through various methods such as device collection and manual upload. Since the uploaded multimodal data is different in time and space, it is necessary to perform time and space alignment processing on it to obtain multimodal data at different spaces at multiple time points.

[0025] Step S104, through a pre-constructed anomaly warning network, perform cross-modal feature clustering on the multimodal data sequence, and identify the abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by cross-modal feature clustering, so as to obtain an anomaly warning result.

[0026] Among them, the anomaly warning network includes a multimodal conditional random field model, a cross-modal clustering model and an anomaly warning model; the abnormal event feature cluster is a data cluster composed of multimodal feature information belonging to the same abnormal event; the anomaly warning result includes the abnormal events occurring in the coal preparation plant and their occurrence probabilities. In one example, after the multimodal data sequence is input into the anomaly warning network, the multimodal feature information of the multimodal data sequence will be extracted through the conditional random field model, the cross-modal feature clustering of the multimodal feature information will be performed through the cross-modal clustering model to obtain the abnormal event feature cluster, and the anomaly warning result will be generated based on the abnormal event feature cluster through the anomaly warning model.

[0027] Step S106: Based on the pre-constructed anomaly decision network, and using the anomaly warning result and the knowledge graph of the coal preparation industry, determine the event decision result corresponding to the abnormal event, so as to conduct an emergency response to the abnormal event according to the event decision result.

[0028] Among them, the knowledge graph of the coal preparation industry is used to describe the association relationships among abnormal events, the causes of abnormal events, and the response strategies for abnormal events. In one example, the knowledge graph of the coal preparation industry can be used to determine the set of alternative causes and the set of alternative strategies corresponding to the abnormal events included in the abnormal warning result. Then, the anomaly decision network is used to evaluate the target abnormal cause and the target response decision in combination with the aforementioned abnormal event feature clusters, and use them as the event decision result to guide the emergency response.

[0029] The coal preparation plant warning and response method based on multimodal analysis provided by the embodiments of the present invention obtains the multimodal information corresponding to the coal preparation plant, and uses the characteristics of cross-verification and complementarity between multimodal data to improve the environmental adaptability and anti-interference ability. On this basis, the anomaly warning network is used to perform cross-modal feature clustering on the multimodal time series to identify the anomaly warning result. Further, the anomaly decision network is used to combine the knowledge graph of the coal preparation industry to obtain specific event solution strategies, which can not only significantly improve the warning and response processing efficiency and reduce the false alarm rate of alarms, but also meet the needs of more special and complex business scenarios.

[0030] A coal preparation plant warning and response method based on multimodal analysis provided by the embodiments of the present invention, which integrates a multimodal safety supervision system of video monitoring, Internet of Things sensing, edge computing and large model reasoning, is applicable to safety supervision scenarios such as personnel behavior compliance monitoring and equipment operation status warning in coal preparation plants. See Figure 2 the technical framework diagram of a coal preparation plant warning and response method based on multimodal analysis shown in The perception layer deploys explosion-proof multispectral cameras (visible light + thermal imaging), monitoring sensors for each production link on site, pick-up microphones, etc., for collecting multimodal data, such as Figure 2 shown, including video modal data and other modal data. The video modal data includes the monitoring data collected by the multispectral cameras deployed at bullet cameras, dome cameras, and PTZ cameras, and also includes the monitoring data collected by patrol devices such as patrol robots and patrol recorders; the other modal data is also audio modal data and text modal data. As Figure 2 shown, the analog signals (including equipment status and equipment parameters), text, etc. collected by the production link detection sensors are all text modal data.

[0031] The analysis layer is also Figure 2The multimodal fusion analysis system shown in the figure is used to integrate multimodal feature fusion algorithms based on spatiotemporal alignment, such as video, audio, and text, and conduct comprehensive analysis based on the surrounding working environment and equipment operating status to generate abnormal warning results.

[0032] The decision-making level Figure 2 The large-scale intelligent early warning system shown in the figure is used to combine the coal preparation industry knowledge graph with the abnormal decision network for analysis and decision-making, generating event decision results. The abnormal decision network can use the LLM (Large Language Model) network.

[0033] The execution layer is Figure 2 The intelligent interactive terminal / system shown in the figure is used to interact with a PLC (Programmable Logic Controller) control system, an audible and visual alarm, and various intelligent interactive terminals to achieve millisecond-level emergency response.

[0034] Through the perception layer, data analysis layer, reasoning decision layer and execution layer of information collection, the system can realize online real-time reasoning detection of various scenarios such as personnel safety and equipment abnormalities at the coal preparation production site. The detection information is combined with the working conditions of the on-site equipment and the surrounding environment to conduct comprehensive analysis and judgment, and then infer the actual safety problems on site, realizing real-time safety supervision of the coal preparation production process.

[0035] Based on the above embodiments, the embodiments of the present invention provide a specific implementation of a coal preparation plant early warning and response method based on multimodal analysis.

[0036] (1) Obtain multimodal data corresponding to the coal preparation plant and perform temporal and spatial alignment processing on the multimodal data to obtain a multimodal data sequence.

[0037] Video modal data is collected by multispectral cameras, which mainly serve as a medium for video information collection. Conventional video acquisition terminals are mainly used for fixed-mounted box cameras or dome cameras, which transmit the captured video images back to the system. In addition to conventional cameras, the multispectral cameras covered by the embodiments of the present invention also include: mobile control dome cameras, inspection robots, recorders and other mobile video acquisition terminals. They can cover various fixed scenes and mobile monitoring scenes, realizing all-round, no-blind-angle, and dynamic and static combined video acquisition capabilities.

[0038] The biggest difference between multimodal analysis and traditional video surveillance systems lies in that multimodal analysis conducts comprehensive analysis based on various types of modal data, including but not limited to text, images, videos, audio, sensor data, spatio-temporal information, etc. Therefore, the embodiments of the present invention also need to collect other information related to the scenario, such as the operating status of equipment, whether it is powered on, the coal-carrying capacity of the system, and other various production data information, as well as various environmental and spatial data information such as ambient temperature and geographical location. All the above-mentioned various types of multimodal data information can be obtained in real time through detection and sensing in the production process.

[0039] Multimodal visual analysis has significant advantages compared to single-modal analysis by fusing multi-source data such as images, text, and speech. Cross-validation and complementarity can be achieved among multimodal data, improving environmental adaptability and anti-interference ability, and thus enhancing the accuracy of judgment. For example, the fusion of infrared and visible light enhances night recognition, and the comprehensive analysis and judgment of abnormal behaviors and equipment operating states, etc. At the same time, cross-modal feature correlation mining can break through the limitations of single-modal information and achieve more fine-grained semantic understanding. For example, in video analysis, combining action and speech recognition improves the accuracy of behavior interpretation. Multimodal learning enhances the generalization ability of the model through knowledge transfer, especially performing excellently in scenarios with scarce data. In contrast, single-modal analysis is easily restricted by factors such as data quality and environmental noise and is difficult to support the intelligent analysis requirements of complex scenarios.

[0040] For the multimodal data collected above, alignment processing is performed in the time and space dimensions. In the time dimension: Based on a multi-source data fusion framework of GNSS + PTP clock synchronization, the problem of spatial calibration of data collected by mobile inspection robots and fixed cameras can be solved (alignment error < 3ms), and at the same time, the time lag problem between real-time production data information and the corresponding time of video acquisition frames can be solved. It is also possible to slide a window of a fixed length on the time axis and associate all the multimodal data reaching within the window to the same time point for rough time alignment. In the space dimension: Based on the spatial alignment carried by multimodal data, spatial matching and spatial interpolation processing, etc. can be performed to achieve alignment processing in the space dimension.

[0041] (2) Through a pre-constructed abnormal warning network, cross-modal feature clustering is performed on the multimodal data sequence to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained from cross-modal feature clustering, and an abnormal warning result is obtained. Specifically, it includes: (2.1)Extract multi-modal feature information for video modal data, audio modal data, and text modal data at the same time point and in the same space through a multi-modal conditional random field model. Among them, the conditional random field model is short for a model based on the conditional random field algorithm, which is used for image segmentation, video action recognition, speech recognition (integrating large language model information), etc., so as to extract multi-modal feature information, and the multi-modal feature information includes video modal feature information, audio modal feature information, and text modal feature information.

[0042] (2.2)Cluster the multi-modal feature information through a cross-modal clustering model to obtain multiple abnormal event feature clusters. In a specific implementation, the process of cross-modal feature clustering is as follows: Map the video modal feature information, audio modal feature information, and text modal feature information to a common subspace, and determine the similarity between the video modal feature information, audio modal feature information, and text modal feature information in the common subspace, so as to divide the video modal feature information, audio modal feature information, and text modal feature information belonging to the same abnormal event into the same data cluster based on the similarity, and obtain multiple abnormal event feature clusters.

[0043] Specifically, the canonical correlation analysis (CCA) algorithm can be used to map the multi-modal feature information output by the multi-modal conditional random field model to a 256-dimensional common subspace, and the cosine similarity is used to measure the similarity between the multi-modal feature information in the common subspace. Furthermore, based on the similarity and a preset similarity threshold, the multi-modal feature information is divided to obtain abnormal event feature clusters corresponding to different abnormal events.

[0044] (2.3)Based on the multi-modal feature information included in the abnormal event feature cluster, determine the probability of the abnormal event corresponding to the abnormal event feature cluster occurring in the coal preparation plant through an abnormal warning model, and determine the abnormal warning result based on the probability corresponding to each abnormal event feature cluster. Among them, the abnormal warning model can adopt the LSTM (Long Short-Term Memory) model, and the input of this model is the abnormal event feature cluster, and the output is the abnormal warning result.

[0045] In the embodiments of the present invention, safety is an important supervision direction in coal preparation production management, and safety management includes three aspects: equipment safety, personnel safety, and environmental safety. The application scenarios of the embodiments of the present invention mainly target the above three aspects, and the specific scenarios and monitoring rules are described as follows: (I) Equipment safety monitoring: Equipment safety monitoring mainly monitors various types of separation equipment, dewatering equipment, and transportation and transfer equipment in the coal preparation plant. The monitoring items include, but are not limited to, problems in various production scenarios such as uneven screening cloth, material breakage on the screen surface, blockage at the head of the belt, and belt material spillage. Real-time collection of the operation images of each type of equipment is achieved through directional cameras, fixed ball cameras, and inspection robots. At the same time, combined with multi-modal technology, video image information is combined with equipment status information, operating parameters, environmental temperature and other information. Based on the LSTM model, the operating conditions of the equipment are dynamically learned, and the abnormal parameter thresholds are automatically corrected (such as the differential setting of the motor temperature threshold in the heavy-load / light-load mode). Through multi-dimensional and multi-modal data information, it is comprehensively judged whether there are abnormalities in the current image information, and intelligent monitoring and early warning of the production site are realized.

[0046] (II) Personnel safety monitoring: Personnel safety monitoring mainly monitors the daily behaviors of on-site production operators, and at the same time integrates multi-modal information, including equipment real-time parameter information and safety operation specification texts. By interpreting the environment and personnel actions in the monitored area, combined with the requirements of safety operation behavior specifications in different operation scenarios, unsafe behaviors of personnel are analyzed and identified. The warning content that can be realized includes, but is not limited to, various complex information scenarios such as personnel not wearing safety helmets, hot work not meeting safety operation specifications, climbing work not meeting safety behavior specifications, and live working, realizing intelligent supervision of personnel safety.

[0047] (III) Environmental safety monitoring: Environmental safety monitoring mainly focuses on the safety supervision of the production operation environment, including the supervision of environmental stability, coal dust, harmful gases, etc. At the same time, text modal information can be incorporated. By monitoring the on-site real-time data and combining the current production status information and industry standard requirements, the safety risks existing in the current operation environment are comprehensively analyzed to realize intelligent supervision.

[0048] (III) Through a pre-constructed abnormal decision network, based on the abnormal early warning results and the knowledge graph of the coal preparation industry, determine the event decision results corresponding to the abnormal events. Specifically: (3.1) Based on the knowledge graph of the coal preparation industry, determine the set of alternative abnormal causes and the set of alternative strategies corresponding to the abnormal events included in the abnormal early warning results. In one example, since the knowledge graph of the coal preparation industry describes the association relationships between abnormal events, the causes of abnormal events, and the response strategies of abnormal events, and the abnormal events included in the known abnormal early warning results are known, the set of alternative abnormal causes and the set of alternative strategies corresponding to each abnormal event can be directly extracted from the knowledge graph of the coal preparation industry.

[0049] (3.2)Based on the pre-constructed anomaly decision network, the target anomaly cause and the target response strategy corresponding to the anomaly event are determined based on the anomaly event feature cluster, the set of alternative anomaly causes, and the set of alternative strategies corresponding to the anomaly event, as the event decision result corresponding to the anomaly event. In one implementation: (3.21)Based on the similarity between the anomaly event feature cluster corresponding to the anomaly event and the historical anomaly events and their corresponding historical anomaly event feature clusters stored in the historical database, the target historical anomaly events are roughly screened out. In one example, the cosine similarity can be used to calculate the similarity between the anomaly event feature cluster corresponding to the anomaly event and the historical anomaly event feature clusters.

[0050] (3.22)Feature encoding and vector splicing are performed on the anomaly event feature cluster corresponding to the anomaly event, the alternative anomaly causes included in the set of alternative anomaly causes, and the alternative response strategies included in the set of alternative strategies to construct multiple current multi-dimensional vectors corresponding to the anomaly event; and feature encoding and vector splicing are performed on the anomaly event feature cluster, the target anomaly cause, and the target response strategy corresponding to the target historical anomaly event to construct historical multi-dimensional vectors.

[0051] The process of feature encoding and vector splicing for the relevant information of the anomaly event is as follows: One-hot encoding is performed on each alternative cause, and in the same one-hot encoding manner as the set of alternative causes, the anomaly event feature cluster vector, the alternative cause encoding vector, and the alternative strategy encoding vector are sequentially spliced to obtain the current multi-dimensional vector.

[0052] The process of feature encoding and vector splicing for the relevant information of the historical anomaly event is as follows: One-hot encoding is performed on the target anomaly cause, one-hot encoding is performed on the target response strategy, and the historical anomaly event feature cluster vector, the target cause encoding vector, and the target strategy encoding vector are spliced to obtain the historical multi-dimensional vector.

[0053] (3.23)Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, the target anomaly cause and the target response strategy corresponding to the anomaly event are precisely screened out. In one example, the Euclidean distance can be used to calculate the distance between the current multi-dimensional vectors and each historical multi-dimensional vector, and then the target anomaly cause and the target response strategy corresponding to the anomaly event are screened out.

[0054] In an alternative implementation, a special knowledge graph for coal preparation and an anomaly decision network are trained, and the provisions of the "Safety Regulations for Coal Preparation Plants" are embedded to achieve end-to-end output from "violation identification" to "compliance suggestions". For example, after detecting that the coal pile is too high, the "Height Limit for Stacker-Reclaimer Operation" is automatically pushed and a three-dimensional stacking simulation diagram is generated, thereby realizing a full-process closed-loop of problem discovery, problem analysis, and problem solving.

[0055] In the embodiments of the present invention, through multi-dimensional and multi-modal data information such as collected video information and equipment operation parameter information, combined with various application scenarios in coal preparation plant production, real-time online monitoring can be carried out. At the same time, with the help of various knowledge graphs established by the system and the analysis and reasoning capabilities of large models, it is possible to accurately judge and identify whether there are potential safety hazards in the current monitoring area. At the same time, combined with various public and private expert knowledge base data such as various safety regulation behavior requirements and production management measures, analyze the causes of various identified abnormal information and recommend solutions, and can push the information to terminal users through various intelligent interaction terminals. The ability to trigger the real-time overlay of correct operation guidelines on AR glasses for violation behaviors (such as showing the correct use method of hanging a safety rope high and using it low) and other capabilities, thereby promoting the handling of problems. This large model intelligent early warning system can identify potential risks, abnormal events or key trends in real-time or near real-time, and actively trigger early warnings or decision support to reduce risks or improve response efficiency.

[0056] In summary, the core technical points of the embodiments of the present invention are as follows: (1) Combine video, ball cameras, inspection robots and recorders, etc. to collect video information, and then combine real-time production parameters and industry specifications to achieve safety supervision and emergency response in coal preparation plants based on multi-modal analysis and reasoning capabilities; (2) Monitor various abnormal information, conduct reasoning by combining knowledge graphs and large models, analyze the causes of problems and give problem-solving measures and specification requirements to achieve closed-loop management of problems; (3) Focus on combining large model technology to achieve a multi-modal intelligent safety supervision and early warning system in the field of coal preparation production. Based on the above core technical points, the embodiments of the present invention first combine multi-modal fusion analysis with a large model enhanced by industry knowledge to overcome the technical pain points of "difficult to perceive, slow to judge, and late to handle" safety incidents in the complex environment of coal preparation plants. Compared with traditional methods, it has at least the following characteristics: (1) The false negative rate of personnel violation identification is significantly reduced: By integrating multiple video acquisition terminal devices, it is basically possible to achieve no dead ends in production site monitoring and no loopholes in safety; (2) The accuracy of equipment failure early warning is significantly improved: Through multi-modal comprehensive analysis algorithms, some false alarms and misreported problems are effectively screened, which can greatly improve the accuracy of alarms; (3) The average time-consuming of emergency response is significantly reduced: After the system detects an abnormal situation, it can achieve online reasoning in seconds and quickly transmit the information to terminal users through an intelligent interaction system, greatly shortening the information communication and circulation link and significantly improving the timeliness of problem response; (4) The threshold requirement for personnel skill level is reduced: Through the established proprietary knowledge base, knowledge spectrum and reasoning large model, it is possible to analyze the causes and solutions of problems online, improve the judgment efficiency, and also solve the problem of relying on personnel experience for analysis.

[0057] On the basis of the foregoing embodiments, the embodiments of the present invention provide a coal preparation plant warning and response system based on multi-modal analysis. See Figure 3Schematic diagram of the structure of a coal preparation plant warning and response system based on multimodal analysis. The system mainly includes the following parts: The data acquisition and processing module 302 is used to acquire multimodal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multimodal data to obtain a multimodal data sequence; wherein, the multimodal data includes video modal data, audio modal data, and text modal data, and the multimodal data sequence includes multimodal data at different spaces in the coal preparation plant at multiple time points; The abnormal warning module 304 is used to perform cross-modal feature clustering on the multimodal data sequence through a pre-constructed abnormal warning network, so as to identify abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by cross-modal feature clustering, and obtain an abnormal warning result; The abnormal emergency response module 306 is used to determine an event decision result corresponding to the abnormal event based on the abnormal warning result and the coal preparation industry knowledge graph through a pre-constructed abnormal decision network, so as to perform an emergency response to the abnormal event according to the event decision result.

[0058] The coal preparation plant warning and response system based on multimodal analysis provided by the embodiment of the present invention obtains multimodal information corresponding to the coal preparation plant, utilizes the characteristics of cross-verification and complementarity between multimodal data to improve environmental adaptability and anti-interference ability, and on this basis, uses an abnormal warning network to perform cross-modal feature clustering on the multimodal time series to identify an abnormal warning result. Further, using an abnormal decision network and combining the coal preparation industry knowledge graph to obtain a specific event solution strategy can not only significantly improve the warning and response processing efficiency and reduce the false alarm rate of alarms, but also meet the needs of more special and complex business scenarios.

[0059] In one implementation, the abnormal warning network includes a multimodal conditional random field model, a cross-modal clustering model, and an abnormal warning model; the abnormal warning module 304 is specifically used for: Through the multimodal conditional random field model, extract multimodal feature information of video modal data, audio modal data, and text modal data at the same time point and the same space respectively; wherein, the multimodal feature information includes video modal feature information, audio modal feature information, and text modal feature information; Through the cross-modal clustering model, perform cross-modal feature clustering on the multimodal feature information to obtain multiple abnormal event feature clusters; wherein, the abnormal event feature cluster is a data cluster composed of multimodal feature information belonging to the same abnormal event; Through the abnormal warning model, based on the multimodal feature information included in the abnormal event feature cluster, determine the probability of the abnormal event corresponding to the abnormal event feature cluster occurring in the coal preparation plant, and determine the abnormal warning result based on the probability corresponding to each abnormal event feature cluster.

[0060] In one implementation, the anomaly warning module 304 is specifically configured to: Map the video modality feature information, audio modality feature information, and text modality feature information to a common subspace, determine the similarity between the video modality feature information, audio modality feature information, and text modality feature information within the common subspace, and divide the video modality feature information, audio modality feature information, and text modality feature information belonging to the same anomaly event into the same data cluster based on the similarity to obtain multiple anomaly event feature clusters.

[0061] In one implementation, the anomaly emergency response module 306 is specifically configured to: Based on the coal preparation industry knowledge graph, determine the set of alternative anomaly causes and the set of alternative strategies corresponding to the anomaly events included in the anomaly warning result; wherein, the coal preparation industry knowledge graph is used to describe the association relationships between anomaly events, the causes of anomaly events, and the response strategies of anomaly events; Through a pre-constructed anomaly decision network, based on the anomaly event feature cluster, the set of alternative anomaly causes, and the set of alternative strategies corresponding to the anomaly event, determine the target anomaly cause and the target response strategy corresponding to the anomaly event as the event decision result corresponding to the anomaly event.

[0062] In one implementation, the anomaly emergency response module 306 is specifically configured to: Based on the similarity between the anomaly event feature cluster corresponding to the anomaly event and the historical anomaly events and their corresponding historical anomaly event feature clusters stored in the historical database, roughly screen out the target historical anomaly events; Perform feature encoding and vector splicing on the anomaly event feature cluster corresponding to the anomaly event, the alternative anomaly causes included in the set of alternative anomaly causes, and the alternative response strategies included in the set of alternative strategies to construct multiple current multi-dimensional vectors corresponding to the anomaly event; and perform feature encoding and vector splicing on the anomaly event feature cluster, the target anomaly cause, and the target response strategy corresponding to the target historical anomaly event to construct historical multi-dimensional vectors; Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, finely screen out the target anomaly cause and the target response strategy corresponding to the anomaly event.

[0063] In one implementation, the data acquisition and processing module 302 is specifically configured to: Slide a window of a fixed length on the time axis and associate all the multi-modal data reached within the window to the same time point.

[0064] In one implementation, the video modality data includes: the monitoring data collected by multi-spectral cameras deployed at each device in the coal preparation plant and the monitoring data collected by the inspection equipment; The audio modality data includes: the sound data collected by the pickups deployed inside the coal preparation plant; The text modal data includes: the device status and device parameters corresponding to each device in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and the pre-configured safety operation specifications.

[0065] The system provided by the embodiments of the present invention has the same implementation principle and technical effects as those of the foregoing method embodiments. For the sake of brief description, for the parts not mentioned in the system embodiments, reference may be made to the corresponding content in the foregoing method embodiments.

[0066] The embodiments of the present invention provide an electronic device. Specifically, the electronic device includes a processor and a storage system; a computer program is stored on the storage system, and when the computer program is run by the processor, it executes the method according to any one of the foregoing embodiments.

[0067] Figure 4 FIG. is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes: a processor 40, a memory 41, a bus 42, and a communication interface 43. The processor 40, the communication interface 43, and the memory 41 are connected through the bus 42; the processor 40 is configured to execute an executable module stored in the memory 41, such as a computer program.

[0068] Among them, the memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 43 (which may be wired or wireless), a communication connection between the system network element and at least one other network element is realized, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0069] The bus 42 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 4 only a bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0070] Among them, the memory 41 is used to store a program. After receiving an execution instruction, the processor 40 executes the program. The method executed by the system defined by the flow process disclosed in any one of the foregoing embodiments of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0071] The processor 40 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 40 or the instructions in the form of software. The above-mentioned processor 40 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute each method, step and logic block diagram disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and combines its hardware to complete the steps of the above method.

[0072] The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be elaborated here.

[0073] If the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs.

[0074] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A warning and response method for coal preparation plants based on multimodal analysis, characterized in that, Including: Obtain multi-modal data corresponding to a coal preparation plant, and perform time and space alignment processing on the multi-modal data to obtain a multi-modal data sequence; wherein, the multi-modal data includes video modal data, audio modal data, and text modal data, and the multi-modal data sequence includes the multi-modal data at different spaces in the coal preparation plant at multiple time points; Through a pre-constructed anomaly warning network, perform cross-modal feature clustering on the multi-modal data sequence to identify the anomalies occurring in the coal preparation plant based on the anomaly event feature clusters obtained from the cross-modal feature clustering, and obtain an anomaly warning result; Through a pre-constructed anomaly decision network, based on the anomaly warning result and the knowledge graph of the coal preparation industry, determine the event decision result corresponding to the anomaly event, so as to perform an emergency response to the anomaly event according to the event decision result.

2. The coal preparation plant warning and response method based on multimodal analysis according to claim 1, wherein The anomaly warning network includes a multi-modal conditional random field model, a cross-modal clustering model, and an anomaly warning model; through the pre-constructed anomaly warning network, performing cross-modal feature clustering on the multi-modal data sequence to identify the anomalies occurring in the coal preparation plant based on the anomaly event feature clusters obtained from the cross-modal feature clustering, and obtaining an anomaly warning result, including: Through the multi-modal conditional random field model, respectively extract multi-modal feature information of the video modal data, the audio modal data, and the text modal data at the same time point and the same space; wherein, the multi-modal feature information includes video modal feature information, audio modal feature information, and text modal feature information; Through the cross-modal clustering model, perform cross-modal feature clustering on the multi-modal feature information to obtain multiple anomaly event feature clusters; wherein, the anomaly event feature cluster is a data cluster composed of the multi-modal feature information belonging to the same anomaly event; Through the anomaly warning model, based on the multi-modal feature information included in the anomaly event feature cluster, determine the probability of the anomaly event corresponding to the anomaly event feature cluster occurring in the coal preparation plant, and determine the anomaly warning result based on the probability corresponding to each anomaly event feature cluster.

3. The coal preparation plant warning and response method based on multimodal analysis according to claim 2, wherein Performing cross-modal feature clustering on the multi-modal feature information to obtain multiple anomaly event feature clusters, including: Map the video modal feature information, the audio modal feature information, and the text modal feature information to a common subspace, determine the similarity between the video modal feature information, the audio modal feature information, and the text modal feature information in the common subspace, so as to divide the video modal feature information, the audio modal feature information, and the text modal feature information belonging to the same anomaly event into the same data cluster based on the similarity, and obtain multiple anomaly event feature clusters.

4. The coal preparation plant warning and response method based on multimodal analysis according to claim 1, characterized in that Through a pre-constructed anomaly decision network, based on the anomaly warning result and the knowledge graph of the coal preparation industry, determine the event decision result corresponding to the anomaly event, including: Based on the knowledge graph of the coal preparation industry, determine the set of alternative abnormal causes and the set of alternative strategies corresponding to the abnormal events included in the abnormal warning result; wherein, the knowledge graph of the coal preparation industry is used to describe the association relationships among abnormal events, the causes of abnormal events, and the response strategies of abnormal events. Based on the abnormal event feature clusters, the set of alternative abnormal causes, and the set of alternative strategies corresponding to the abnormal event, determine the target abnormal cause and the target response strategy corresponding to the abnormal event through a pre-constructed abnormal decision network, as the event decision result corresponding to the abnormal event.

5. The coal preparation plant warning and response method based on multimodal analysis according to claim 4, characterized in that Determining the target abnormal cause and the target response strategy corresponding to the abnormal event based on the abnormal event feature clusters, the set of alternative abnormal causes, and the set of alternative strategies corresponding to the abnormal event includes: Based on the similarity between the abnormal event feature clusters corresponding to the abnormal event and the historical abnormal events and their corresponding historical abnormal event feature clusters stored in the historical database, roughly screen out the target historical abnormal events. Perform feature encoding and vector splicing on the abnormal event feature clusters corresponding to the abnormal event, the alternative abnormal causes included in the set of alternative abnormal causes, and the alternative response strategies included in the set of alternative strategies to construct multiple current multi-dimensional vectors corresponding to the abnormal event; and perform feature encoding and vector splicing on the abnormal event feature clusters, the target abnormal causes, and the target response strategies corresponding to the target historical abnormal event to construct historical multi-dimensional vectors. Based on the multiple current multi-dimensional vectors and the historical multi-dimensional vectors, finely screen out the target abnormal cause and the target response strategy corresponding to the abnormal event.

6. The coal preparation plant warning and response method based on multimodal analysis according to claim 1, characterized in that Performing time alignment processing on the multi-modal data includes: Slide a window of a fixed length on the time axis and associate all the multi-modal data reached within the window to the same time point.

7. The coal preparation plant early warning and response method based on multimodal analysis according to claim 1, wherein The video modal data includes: monitoring data collected by multi-spectral cameras deployed at each device in the coal preparation plant and monitoring data collected by inspection devices. The audio modal data includes: sound data collected by pickups deployed inside the coal preparation plant. The text modal data includes: the device status and device parameters corresponding to each device in the coal preparation plant, the current environmental status corresponding to the coal preparation plant, and the pre-configured safety operation specifications.

8. A warning and response system for coal preparation plants based on multimodal analysis, characterized in that, Includes: A data acquisition and processing module, configured to acquire the multi-modal data corresponding to the coal preparation plant, and perform time and space alignment processing on the multi-modal data to obtain a multi-modal data sequence; wherein, the multi-modal data includes video modal data, audio modal data, and text modal data, and the multi-modal data sequence includes the multi-modal data at different spaces in the coal preparation plant at multiple time points. An abnormal warning module, configured to perform cross-modal feature clustering on the multi-modal data sequence through a pre-constructed abnormal warning network, so as to identify the abnormal events occurring in the coal preparation plant based on the abnormal event feature clusters obtained by the cross-modal feature clustering, and obtain an abnormal warning result. An abnormal emergency response module is used to determine an event decision result corresponding to the abnormal event through a pre-constructed abnormal decision network based on the abnormal warning result and the knowledge graph of the coal preparation industry, so as to perform an emergency response to the abnormal event according to the event decision result.

9. An electronic device, characterized in that, It includes a processor and a memory. The memory stores computer-executable instructions that can be executed by the processor. The processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions cause the processor to implement the method according to any one of claims 1 to 7.

Images