Knowledge graph-based multi-modal safety management and control method and device

Abstract

The application provides a knowledge graph-based multi-modal safety management and control method and device, which comprises the following steps: acquiring visual data of a target area of a coal preparation plant; identifying hidden danger points in the visual data based on a pre-constructed knowledge base; and performing cause analysis on the hidden danger points based on a pre-constructed knowledge graph and a first large language model, and generating an analysis report of the target area. The application improves the efficiency and reliability of safety supervision of the coal preparation plant.

Description

Technical Field

[0001] This invention relates to the field of security management technology, and in particular to a multimodal security management method and device based on knowledge graphs. Background Technology

[0002] Coal preparation plants, as a crucial link in coal production, are characterized by numerous hazardous work sites and complex operating environments. Currently, safety inspections at many coal preparation plants still rely on manual labor, employing limited methods and exhibiting inconsistent inspection standards, making them prone to omissions and resulting in low efficiency in safety supervision. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a multimodal safety management and control method and device based on knowledge graph, so as to improve the efficiency and reliability of safety supervision in coal preparation plants.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a multimodal safety management and control method based on knowledge graphs, comprising: acquiring visual data of a target area of ​​a coal preparation plant; identifying potential hazards in the visual data based on a pre-built knowledge base; and performing cause analysis on the potential hazards based on a pre-built knowledge graph and a first language model to generate an analysis report of the target area.

[0005] Optionally, based on a pre-built knowledge graph and the first major language model, a cause analysis of the hidden danger points is performed to generate an analysis report for the target area, including: performing cause reasoning on the hidden danger points based on the pre-built knowledge graph to obtain a preliminary analysis report; and inputting the knowledge base and the preliminary analysis report into the first major language model to generate an analysis report for the target area.

[0006] Optionally, before acquiring the visual data of the target area of ​​the coal preparation plant, the process may include: acquiring safety text data, historical failure cases, and corresponding historical solutions for the coal preparation plant; extracting entities from the safety text data based on a pre-trained second language model to obtain structured text; wherein the structured text includes at least: potential hazards in the target area, descriptions of potential hazards, entities, and entity relationships; and constructing a knowledge base and knowledge graph based on the structured text, historical failure cases, and historical solutions.

[0007] Optionally, based on a pre-built knowledge base, identify potential hazards in the visual data, including: constructing prompt words based on potential hazards and their descriptions in the knowledge base; and inputting the prompt words and visual data into a pre-built visual language model to obtain the potential hazards in the visual data.

[0008] Optionally, a preliminary analysis report is obtained by performing causal reasoning on the hidden danger points based on a pre-built knowledge graph, including: performing causal reasoning on the hidden danger points based on entity relationships and abnormal causal links in the knowledge graph to obtain preliminary causal information corresponding to the hidden danger points; and generating a preliminary analysis report based on the description information of the hidden danger points, the preliminary causal information, and the causal reasoning process.

[0009] Optionally, the first major language model includes an analysis module and a summary module. The knowledge base and preliminary analysis report are input into the first major language model to generate an analysis report for the target area, including: inputting the preliminary analysis report and historical solution strategies from the knowledge base into the analysis module to generate solution strategies for the hidden danger points; inputting the hidden danger points, preliminary analysis report, and solution strategies into the summary module to generate an analysis report for the target area.

[0010] Secondly, the present invention provides a multimodal safety management and control device based on a knowledge graph, comprising: a data acquisition module for acquiring visual data of a target area of ​​a coal preparation plant; a hazard identification module for identifying hazard points in the visual data based on a pre-built knowledge base; and a cause analysis module for performing cause analysis on the hazard points based on a pre-built knowledge graph and a first language model, and generating an analysis report of the target area.

[0011] Optionally, the cause analysis module is specifically used to: perform cause reasoning on potential hazards based on a pre-built knowledge graph to obtain a preliminary analysis report; and input the knowledge base and the preliminary analysis report into the first language model to generate an analysis report for the target area.

[0012] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method provided in any of the first aspects above.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the method provided in any of the first aspects above.

[0014] This invention brings the following beneficial effects: The multimodal safety management method and apparatus based on knowledge graphs provided by this invention first acquires visual data of the target area of ​​a coal preparation plant; then, based on a pre-built knowledge base, it identifies potential hazards in the visual data; finally, based on the pre-built knowledge graph and a large language model, it performs causal analysis on the hazards and generates an analysis report for the target area. In this method, the use of a pre-built knowledge base to identify potential hazards in the visual data, combined with causal analysis using a knowledge graph and a large language model, enhances the comprehensiveness of safety supervision of the coal preparation plant by connecting information from different sources, thereby improving the efficiency and reliability of safety supervision.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a data construction method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a knowledge graph-based multimodal security management method provided in this embodiment of the invention; Figure 3 A flowchart illustrating the inference stage of multimodal security management provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a knowledge graph-based multimodal security management and control device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, safety supervision methods in coal preparation plants face numerous bottlenecks: (1) Traditional safety supervision is inefficient: Currently, many coal preparation plants still rely on manual labor for safety inspections. The methods are limited and the inspection levels vary, making it easy to miss inspections.

[0021] (2) Serious data silos: The production, equipment, safety and other system data in the coal preparation plant are independent of each other, forming data silos, making it difficult to conduct comprehensive analysis and coordinated decision-making.

[0022] (3) Limited data sources and poor model scalability: Video analysis systems can only analyze and process video data and train recognition based on predefined patterns, resulting in poor scalability. For example, a helmet recognition model can only recognize helmets. To recognize people crossing belts, data must be collected again and the model must be trained, which is a complex and inefficient process.

[0023] (4) Difficulty in identifying complex environments: Coal preparation plants have complex working conditions such as low illumination and high dust, which makes it difficult to extract features and blur the boundary positioning of traditional image recognition methods, resulting in low recognition accuracy.

[0024] Based on this, the present invention provides a multimodal security management method and device based on knowledge graph, which can improve the efficiency and reliability of safety supervision in coal preparation plants.

[0025] To facilitate understanding of this embodiment, a data construction method disclosed in this embodiment of the invention will be described in detail first, mainly including the following process: First, acquire safety text data, historical fault cases, and corresponding historical solution strategies of the coal preparation plant; then, extract entities from the safety text data based on a pre-trained second language model to obtain structured text; wherein, the structured text includes at least: hidden danger points in the target area, description information of hidden danger points, entities, and entity relationships; finally, construct a knowledge base and knowledge graph based on the structured text, historical fault cases, and historical solution strategies.

[0026] In one implementation, see Figure 1As shown, data is constructed using an LLM module and a processing module. Specifically, the LLM module utilizes the language understanding and reasoning capabilities of a large language model to extract structured text from safety text data (e.g., safety procedure texts, equipment operation and maintenance manuals, standard operating procedures, etc.). The structured text includes: knowledge point descriptions (including but not limited to: hazard points and hazard point description information), entities, and entity relationships. Entities include at least: personnel, equipment (belt conveyors, vibrating screens, crushers, etc.), areas, PPE (safety helmets, dust masks, etc.), hazard types, and standard clauses. Entity relationships are the relationships between entities, such as: personnel - wearing - PPE, belt conveyor - located at - transfer point, not wearing a safety helmet - violating - safety procedure clause X, etc. Furthermore, in this embodiment of the invention, historical solutions can also be structured to form an abnormal causal chain of hazard - cause - solution.

[0027] The processing module processes the structured text output by the LLM module, including but not limited to operations such as text deduplication, validation, vectorization, and database insertion. Ultimately, it combines the structured text, historical fault cases, and historical solution strategies to form a knowledge base and knowledge graph. The knowledge graph includes entity relationships and anomaly causal chains.

[0028] Next, a multimodal security management method based on knowledge graphs, disclosed in the embodiments of the present invention, will be described in detail. This method can be executed by electronic devices, such as smartphones, computers, and tablets. See also Figure 2 The flowchart shown illustrates a multimodal security management method based on knowledge graphs, indicating that the method mainly includes the following steps S201 to S203: Step S201: Obtain visual data of the target area of ​​the coal preparation plant.

[0029] In one implementation, the target area can be a working area within a coal preparation plant (e.g., main plant, transfer point, crushing workshop, power distribution room, coal bunker, etc.) or specific production equipment. In this embodiment of the invention, visual data of the target area can be collected by cameras installed in the coal preparation plant, such as industrial scene monitoring and equipment operation footage.

[0030] Step S202: Identify potential hazards in the visual data based on a pre-built knowledge base.

[0031] In one implementation, potential hazards in a coal preparation plant include, but are not limited to: personnel behavior (not wearing a safety helmet, illegally crossing dangerous areas, working at heights without protection, not operating equipment according to procedures, etc.), equipment status (belt misalignment, screen blockage, motor overheating, oil leakage, missing protective covers, etc.), environmental risks (water accumulation, excessive dust concentration, insufficient lighting, blocked passages, etc.), and management deficiencies (no warning signs, missing fire-fighting equipment, closed emergency passages, etc.).

[0032] In this embodiment of the invention, the pre-built knowledge base includes potential hazards and corresponding hazard descriptions. Based on this, a visual large language model can be used to identify whether corresponding potential hazards exist in visual data according to the hazard descriptions in the knowledge base.

[0033] Step S203: Based on the pre-built knowledge graph and the first language model, perform cause analysis on the hidden danger points and generate an analysis report for the target area.

[0034] In one implementation, after obtaining the potential hazards in the visual data, a knowledge graph can be used for querying and causal reasoning to determine the root cause of the hazard; then, the first language model is used to combine the potential hazards and the root cause to generate corresponding solutions and suggestions, and an analysis report for the target area is generated.

[0035] The above-mentioned knowledge graph-based multimodal safety management method provided in this invention uses a pre-built knowledge base to identify potential hazards in visual data and combines knowledge graphs and large language models for cause analysis. By connecting information from different sources, it can improve the comprehensiveness of safety supervision of coal preparation plants, thereby improving the efficiency and reliability of safety supervision of coal preparation plants.

[0036] In one implementation, for the aforementioned step S202, that is, when identifying potential hazards in visual data based on a pre-built knowledge base, the following methods may be used, including but not limited to: First, constructing prompt words based on potential hazards and their descriptions in the knowledge base; then, inputting the prompt words and visual data into a pre-built visual language model to obtain the potential hazards in the visual data.

[0037] In practice, the visual data is first processed by frame extraction (e.g., 1 frame / second), and object detection models (YOLOv8, RT-DETR) are used to identify basic objects in the video: people, safety helmets, equipment, flames, smoke, water stains, etc. Then, hazard warning words are constructed using hazard points and their descriptions from a knowledge base. These warning words and video frames are then input into a Vision-Language Model (VLM) for semantic understanding to obtain the hazard points in the visual data. For example, if the VLM outputs: "A worker was not wearing a safety helmet while working next to a conveyor belt," the hazard point is initially determined to be "violation of PPE wearing regulations."

[0038] In one implementation, for the aforementioned step S203, i.e., when performing cause analysis on potential hazards based on a pre-built knowledge graph and the first major language model to generate an analysis report for the target area, the following methods may be used, including but not limited to: First, the causes of potential hazards are inferred based on a pre-built knowledge graph, resulting in a preliminary analysis report.

[0039] In practice, firstly, based on the entity relationships and abnormal causal links in the knowledge graph, the cause of the hidden danger is inferred to obtain the preliminary cause information corresponding to the hidden danger; then, based on the description information of the hidden danger, the preliminary cause information and the cause inference process, a preliminary analysis report is generated.

[0040] Specifically, the identified potential hazards are used as input, and the causes are inferred by combining entity relationships and abnormal causal links in the knowledge graph to obtain preliminary cause information corresponding to the potential hazards. Then, a preliminary analysis report is output, which includes: description information of the potential hazard, preliminary cause information, and cause reasoning process.

[0041] Then, the knowledge base and preliminary analysis report are input into the first language model to generate an analysis report for the target region.

[0042] In practical implementation, the first major language model includes an analysis module and a summary module. First, the preliminary analysis report and historical solution strategies from the knowledge base are input into the analysis module to generate solution strategies for the potential hazards. Then, the potential hazards, the preliminary analysis report, and the solution strategies are input into the summary module to generate an analysis report for the target area.

[0043] Specifically, firstly, the preliminary analysis report and historical solution strategies from the knowledge base (unstructured knowledge stored in the strategy library, which includes historical solution strategies and expert-compiled strategy experience, described using natural language) are input into the analysis module of the first major language model for analysis, resulting in actionable solution strategies (e.g., maintenance plans, process optimization suggestions, etc.). Then, the potential hazards, preliminary analysis report, and solution strategies are input into the summary module to integrate information from the entire process and generate an analysis report for the target area.

[0044] Considering that the visual language large model needs to be adapted to monitoring data from multiple scenarios in industrial settings such as coal preparation plants, in order to improve the scenario adaptability of the visual language large model, embodiments of the present invention may adopt methods including but not limited to the following: (1) Adopt scene-aware dynamic prompting engineering, dynamically generate VLM prompt words for different areas corresponding to different security rules.

[0045] (2) By using the visual-text-knowledge triad fusion, the original output of VLM is prevented from deviating from the actual procedure. Specifically, the VLM model generates a preliminary description of the video stream frame, then performs knowledge graph retrieval to match the hazard type and return the relevant procedure and causal chain; then, the output of the VLM model is reconstructed using the returned relevant procedure and causal chain, and relevant supporting information is added to obtain the final hazard point.

[0046] (3) Learn a scene context vector for each monitoring point as part of the VLM input. Specifically, a graph neural network (GNN) can be used to model the topology of the factory area, and the camera location, workshop, and equipment type can be encoded as embedding vectors and concatenated into image features or prompts.

[0047] For ease of understanding, this embodiment of the invention also provides a flowchart of the multimodal security control inference stage, see [link / reference]. Figure 3 As shown, the reasoning process is implemented based on a multimodal security control system, which includes an input layer, a perception layer, an analysis layer, and a decision layer.

[0048] (1) Input layer: dual-source input of vision and knowledge.

[0049] Visual stream: Input video stream, providing real-time or historical visual data (such as industrial scene monitoring, equipment operation screens, etc.).

[0050] Knowledge Flow: Input knowledge base, which includes domain expertise, historical failure cases, rule base, etc. (such as equipment operation and maintenance manuals and failure case library).

[0051] (2) Perception layer: VLM module (visual-language model).

[0052] By integrating video streams with visual and textual knowledge from a knowledge base, potential hazards in the scene (such as abnormal equipment status or operational violations) are identified and output, thus clearly identifying the problems or risks present in the visual scene.

[0053] (3) Analysis layer: root cause analysis and knowledge graph empowerment.

[0054] Specifically, the analysis layer includes: Root Cause Analysis Module: It receives potential hazards as input, combines them with a knowledge graph (including structured knowledge such as entity relationships and causal links), traces the root causes of potential hazards (such as design defects in equipment failures, process loopholes, etc.), and outputs an analysis report, which includes hazard description information, root cause derivation process, and preliminary conclusions.

[0055] (4) Decision-making level: LLM model and multi-source integration.

[0056] Specifically, the decision-making layer includes an LLM analysis module and an LLM summary module. The LLM analysis module receives analysis reports and a knowledge base (supplemented with unstructured knowledge and a strategy base), and generates actionable recommendations and measures (such as maintenance plans and process optimization suggestions).

[0057] The LLM summary module simultaneously receives hazard points, analysis reports, recommendations, and measures, integrates information from the entire process, and generates the final analysis report (including a complete closed loop of hazard, root cause, and solution).

[0058] The multimodal security management method based on knowledge graphs provided in this invention breaks through the limitations of traditional single data analysis. By connecting information from different sources, it forms a comprehensive understanding of security. For example, it correlates and analyzes real-time video-recognized personnel behavior with relevant safety regulations, clauses, and security knowledge graphs, cross-checking and verifying them to make the behavioral analysis data foundation more comprehensive and reliable. If security supervision requirements change, the system can import corresponding relevant documents and automatically change the visual detection mode through prompt word injection, without retraining the model, offering high flexibility and strong scalability. The above method can not only trigger alarms but also use the knowledge graph to find the attributes and real-time data of related upstream and downstream devices, analyze the causes of alarms, and provide repair suggestions based on knowledge points and historical fault data, ultimately achieving a closed loop of system monitoring, alarm triggering, cause analysis, and handling suggestions.

[0059] In addition to the knowledge graph-based multimodal security management method provided in the foregoing embodiments, this invention also provides a knowledge graph-based multimodal security management device. (See [link to relevant documentation]). Figure 4 The diagram shown illustrates the structure of a knowledge graph-based multimodal security management device, indicating that the device mainly comprises the following parts: The data acquisition module 401 is used to acquire visual data of the target area of ​​the coal preparation plant.

[0060] The hazard identification module 402 is used to identify hazard points in visual data based on a pre-built knowledge base.

[0061] The cause analysis module 403 is used to perform cause analysis on potential hazards based on a pre-built knowledge graph and the first language model, and generate an analysis report for the target area.

[0062] The multimodal safety management and control device based on knowledge graphs provided in this invention uses a pre-built knowledge base to identify potential hazards in visual data and combines knowledge graphs and large language models for cause analysis. By connecting information from different sources, it can improve the comprehensiveness of safety supervision of coal preparation plants, thereby improving the efficiency and reliability of safety supervision of coal preparation plants.

[0063] In one embodiment, the above-mentioned device further includes a data construction module, used for: acquiring safety text data of the coal preparation plant, historical failure cases and corresponding historical solution strategies; extracting entities from the safety text data based on a pre-trained second language model to obtain structured text; wherein the structured text includes at least: hidden danger points in the target area, description information of hidden danger points, entities and entity relationships; and constructing a knowledge base and knowledge graph based on the structured text, historical failure cases and historical solution strategies.

[0064] In one implementation, the aforementioned hazard identification module 402 is specifically used to: construct prompt words based on hazard points and hazard point description information in the knowledge base; input the prompt words and visual data into a pre-constructed visual language large model to obtain hazard points in the visual data.

[0065] In one implementation, the cause analysis module 403 is specifically used to: perform cause reasoning on the hidden danger points based on a pre-built knowledge graph to obtain a preliminary analysis report; input the knowledge base and the preliminary analysis report into the first large language model to generate an analysis report for the target area.

[0066] In one implementation, the cause analysis module 403 is specifically used to: perform cause reasoning on the hidden danger points based on entity relationships and abnormal causal links in the knowledge graph, and obtain preliminary cause information corresponding to the hidden danger points; and generate a preliminary analysis report based on the description information of the hidden danger points, the preliminary cause information and the cause reasoning process.

[0067] In one implementation, the first major language model includes an analysis module and a summary module; the aforementioned cause analysis module 403 is specifically used to: input the preliminary analysis report and historical solution strategies from the knowledge base into the analysis module to generate a solution strategy for the hidden danger points; and input the hidden danger points, the preliminary analysis report, and the solution strategies into the summary module to generate an analysis report for the target area.

[0068] It should be noted that the device provided in the embodiments of the present invention has the same implementation principle and technical effect as the aforementioned method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the aforementioned method embodiments.

[0069] This invention also provides an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0070] Figure 5 The present invention provides a schematic diagram of the structure of an electronic device 100, which includes a processor 50, a memory 51, a bus 52 and a communication interface 53. The processor 50, the communication interface 53 and the memory 51 are connected through the bus 52. The processor 50 is used to execute executable modules, such as computer programs, stored in the memory 51.

[0071] The memory 51 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 53 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0072] Bus 52 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0073] The memory 51 is used to store programs. After receiving an execution instruction, the processor 50 executes the programs. The method executed by the device for defining the flow process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0074] Processor 50 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 50 or by instructions in software form. Processor 50 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 51. The processor 50 reads the information in memory 51 and, in conjunction with its hardware, completes the steps of the above method.

[0075] The computer program product of the readable storage medium provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For specific implementation, please refer to the foregoing method embodiments, which will not be repeated here.

[0076] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal security management method based on knowledge graphs, characterized in that, include: Acquire visual data of the target area of ​​the coal preparation plant; Based on a pre-built knowledge base, potential hazards are identified in the visual data; Based on a pre-built knowledge graph and the first major language model, the causes of the potential hazards are analyzed, and an analysis report of the target area is generated.

2. The method according to claim 1, characterized in that, Based on a pre-built knowledge graph and the first major language model, the causes of the potential hazards are analyzed, and an analysis report of the target area is generated, including: Based on a pre-constructed knowledge graph, the causes of the potential hazards are inferred, and a preliminary analysis report is obtained. The knowledge base and the preliminary analysis report are input into the first large language model to generate an analysis report for the target region.

3. The method according to claim 2, characterized in that, Before acquiring visual data of the target area of ​​the coal preparation plant, the following steps are also included: Obtain the safety text data, historical fault cases, and corresponding historical solution strategies of the coal preparation plant; The security text data is subjected to entity extraction based on a pre-trained second language model to obtain structured text; wherein, the structured text includes at least: potential hazards in the target area, description information of the potential hazards, entities, and entity relationships; A knowledge base and knowledge graph are constructed based on the structured text, the historical failure cases, and the historical solution strategies.

4. The method according to claim 3, characterized in that, Based on a pre-built knowledge base, potential hazards in the visual data are identified, including: Based on the hazard points and hazard point descriptions in the knowledge base, prompt words are constructed; The prompt words and the visual data are input into a pre-built visual language model to obtain the potential hazards in the visual data.

5. The method according to claim 3, characterized in that, Based on a pre-constructed knowledge graph, the causes of the potential hazards are inferred, resulting in a preliminary analysis report, including: Based on the entity relationships and abnormal causal links in the knowledge graph, the cause reasoning of the potential risks is performed to obtain the preliminary cause information corresponding to the potential risks. Based on the description of the potential hazard, the preliminary cause information, and the cause reasoning process, a preliminary analysis report is generated.

6. The method according to claim 3, characterized in that, The first large language model includes an analysis module and a summary module; the knowledge base and the preliminary analysis report are input into the first large language model to generate an analysis report for the target region, including: The preliminary analysis report and the historical solutions in the knowledge base are input into the analysis module to generate a solution strategy for the hidden danger point; The potential hazards, the preliminary analysis report, and the solution strategy are input into the summary module to generate an analysis report for the target area.

7. A multimodal security management and control device based on knowledge graphs, characterized in that, include: The data acquisition module is used to acquire visual data of the target area of ​​the coal preparation plant; The hazard identification module is used to identify hazard points in the visual data based on a pre-built knowledge base; The cause analysis module is used to perform cause analysis on the potential hazards based on a pre-built knowledge graph and the first language model, and generate an analysis report for the target area.

8. The apparatus according to claim 7, characterized in that, The cause analysis module is specifically used for: Based on a pre-constructed knowledge graph, the causes of the potential hazards are inferred, and a preliminary analysis report is obtained. The knowledge base and the preliminary analysis report are input into the first large language model to generate an analysis report for the target region.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method described in any one of claims 1 to 6.

Images