Intelligent safety coordination management system based on multi-source perception and language large model

Abstract

The present application relates to the field of multi-source data management, and more particularly to an intelligent safety coordination management system based on multi-source perception and language large model. It comprises: a multi-source data acquisition module, which accesses various intelligent monitoring devices and converts them into a unified standard tuple format through a mapping function output; an information fusion module, which filters the candidate alarm set according to the space-time tolerance, generates composite alarm information by confidence sorting and semantic embedding weighted average; an RAG knowledge base module, which generates a composite event description through a large language model and vectorizes it as a retrieval index; an intelligent retrieval module, which uses a hybrid retrieval strategy to obtain relevant historical events and construct structured prompt words; an LLM decision module, which outputs root cause analysis and classified disposal suggestions based on composite alarms and prior knowledge; a dual-path response module, which distributes decision suggestions to management personnel and agent systems to realize collaborative execution; a feedback optimization module, which collects disposal data to adjust the weight of the knowledge base and the decision template to realize continuous optimization.

Description

Technical Field

[0001] This invention relates to the field of multi-source data management, and in particular to an intelligent security collaborative management system based on multi-source perception and a large language model. Background Technology

[0002] Traditional safety management systems have long been hampered by information silos and high false alarm rates. Alarm information from various sensors is often independent, lacking effective cross-modal correlation analysis capabilities. This fragmented data processing method forces maintenance personnel to manually correlate multiple isolated alarms (such as "video anomaly" + "sudden temperature rise"), which is inefficient and prone to missing key correlation points. With the popularization of IoT technology, the multi-source heterogeneous data generated in modern industrial environments is growing explosively. Traditional rule-based and threshold-based alarm systems can no longer cope with the safety monitoring needs of complex environments, resulting in high false alarm rates. The risk of missed alarms is particularly prominent in complex event scenarios, such as a single sensor pressure drop alarm without a video-confirmed liquid splash alarm, which may overlook leakage events.

[0003] In recent years, the development of multi-source data fusion technology has provided a new path to overcome this bottleneck. This technology transforms data from different sensors into a unified event description through spatiotemporal correlation analysis and contextual semantic modeling. For example, when video surveillance identifies abnormal behavior, the system can simultaneously correlate "illegal door opening" events from access control sensors in the same area with abnormal fluctuations from temperature sensors to generate a high-confidence "intrusion event" determination. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides an intelligent security collaborative management system based on multi-source perception and a large language model, aiming to improve the technical defects of traditional security management systems, such as severe information silos, high false alarm and false alarm rates, and low automation.

[0005] In a first aspect, the present invention provides the following technical solution: an intelligent security collaborative management system based on multi-source perception and a large language model, comprising:

[0006] The multi-source data acquisition module is used to connect to various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through a mapping function;

[0007] The information fusion module is used to filter candidate alarm sets based on time window tolerance and spatial error tolerance, and generate composite alarm information by using confidence ranking and splicing and semantic embedding weighted averaging.

[0008] The RAG knowledge base module is used to generate composite event descriptions from composite alarm information through a large language model, and then vectorize them as a retrieval index to build a structured knowledge base.

[0009] The intelligent retrieval module uses a hybrid retrieval strategy to retrieve relevant historical events from the knowledge base and dynamically constructs structured prompts.

[0010] The LLM decision module is used to output root cause analysis and differentiate between automated execution and manual review of classification and handling suggestions based on composite alarms and prior knowledge;

[0011] The dual-path response module is used to distribute decision recommendations to managers and automated agent systems in parallel, enabling collaborative execution of manual review and automated equipment control.

[0012] The feedback optimization module is used to collect disposal data and adjust the knowledge base weights and decision templates based on the disposal results, so as to achieve continuous system optimization.

[0013] Furthermore, the processing flow of the multi-source data acquisition module includes:

[0014] It can connect to various intelligent monitoring devices to obtain raw monitoring data from each sensor;

[0015] The heterogeneous sensor outputs are converted into a unified standard tuple format containing timestamps, location coordinates, data source type, alarm content, and confidence level through a mapping function.

[0016] Furthermore, the processing flow of the information fusion module includes:

[0017] Candidate alarm sets are filtered based on the time window tolerance and spatial error tolerance of the reference center point;

[0018] Composite alarm information that can be read by both humans and machines is generated by using a confidence-based sorting and splicing method;

[0019] The text is converted into vectors and fused using semantic embedding and confidence-weighted averaging to generate composite alarm text suitable for large language model input.

[0020] Furthermore, the construction process of the RAG knowledge base module includes:

[0021] The composite alarm information is used to generate a structured composite event description through a large language model;

[0022] Vectorize the composite events and use them as the retrieval index field;

[0023] Construct a complete knowledge base data structure that includes composite alarms, composite events, locations, root causes, handling suggestions, and handling results.

[0024] Furthermore, the retrieval process of the intelligent retrieval module includes:

[0025] Filter irrelevant data by limiting alarm location parameters through metadata filtering;

[0026] A dense vector construction method is used to perform vector similarity search in the filtered subset to find the most semantically similar historical events;

[0027] Sparse vectors are used for keyword retrieval, and the priority of the retrieval results is further adjusted by reordering.

[0028] Dynamic splicing includes structured prompts such as role definition, task description, current alarms, and prior knowledge.

[0029] Furthermore, the analysis process of the LLM decision module includes:

[0030] Receive dynamic prompts containing the current composite alarm, retrieved prior knowledge, and task instructions;

[0031] Intelligent analysis is performed based on multi-source fusion event descriptions and semantic matching results from the knowledge base.

[0032] The output includes a structured root cause analysis containing event type, confidence level, and evidence of related events;

[0033] Generate classification and handling suggestions that distinguish between automatic execution and manual review, and force additional manual review requirements when the confidence level is lower than a preset threshold.

[0034] Furthermore, the execution flow of the dual-path response module includes:

[0035] Intelligent decision-making suggestions are distributed in parallel to two terminals: management and maintenance personnel and automated agent system. Management and maintenance personnel receive instructions that require manual review and are responsible for manual judgment, decision support and key point review.

[0036] The automated agent system parses executable instructions and converts natural language commands into device control instructions through a preset instruction mapping table, which are then executed by standardized machinery.

[0037] Furthermore, the optimization process of the feedback optimization module includes:

[0038] Collect data on the original decisions, automated execution records, manual review records, and final handling results for each incident.

[0039] Strengthen the weight of relevant knowledge base items based on successful cases, mark decision templates that need revision based on failed cases, and update the root cause and handling suggestion database based on manual corrections;

[0040] The system continuously learns by implementing optimizations to prompt word templates, adjustments to search weights, and improvements to instruction generation strategies.

[0041] Secondly, this invention provides the following technical solution: an intelligent security collaborative management method based on multi-source perception and a large language model, comprising:

[0042] It can connect to various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through mapping functions;

[0043] Candidate alarm sets are selected based on time window tolerance and spatial error tolerance, and composite alarm information is generated by confidence ranking and splicing and semantic embedding weighted averaging.

[0044] Composite alarm information is used to generate composite event descriptions through a large language model, and these descriptions are vectorized and used as retrieval indexes to build a structured knowledge base.

[0045] The hybrid retrieval strategy retrieves relevant historical events from the knowledge base and dynamically constructs structured prompts;

[0046] Based on composite alarms and prior knowledge, the system outputs root cause analysis and classification and handling suggestions that differentiate between automated execution and manual review;

[0047] Decision recommendations are distributed in parallel to managers and automated agent systems, enabling collaborative execution of manual review and automated equipment control;

[0048] Collect and process data, and adjust the knowledge base weights and decision templates based on the processing results to achieve continuous system optimization.

[0049] The present invention has the following beneficial effects:

[0050] 1. In this invention, through multi-source data fusion, the system can correlate data from multiple sensors to generate composite event alarms. For example, in the event of abnormal temperature, video surveillance data is combined to verify abnormal behavior, reducing false alarms. Only after cross-verification of multiple data sources will the system trigger the final alarm, greatly improving the accuracy of alarms and reducing unnecessary interference.

[0051] 2. In this invention, intelligent analysis is performed based on preset business semantic templates. The system can automatically parse alarm information, analyze the causes, and generate emergency steps according to business rules, making the decision-making process more intelligent and automated. Through natural language processing technology, the system can quickly generate actionable suggestions and solutions, significantly improving response efficiency and decision-making quality.

[0052] 3. In this invention, by parsing the standardized instructions generated by the LLM, mechanical operations (such as valve control, equipment start-up and shutdown) can be executed. This automated response reduces the latency of traditional manual operations from minutes to milliseconds, greatly improving the system's response speed and execution efficiency, and ensuring that critical operations are quickly implemented in emergency situations. Attached Figure Description

[0053] Figure 1This is a structural diagram of the intelligent security collaborative management system based on multi-source perception and a large language model proposed in this invention.

[0054] Figure 2 This is an architecture diagram of the intelligent security collaborative management system proposed in this invention;

[0055] Figure 3 This is a flowchart of the RAG knowledge base construction process proposed in this invention;

[0056] Figure 4 This is a flowchart of the intelligent security collaborative management system based on multi-source perception and a large language model proposed in this invention. Detailed Implementation

[0057] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] In the first embodiment of the present invention, the present invention provides an intelligent security collaborative management system based on multi-source perception and a large language model, the specific architecture of which is as follows: Figure 1 As shown in the figure. Based on this architecture, a modular design is performed, resulting in the following structural diagram. Figure 2 As shown, it includes:

[0060] The multi-source data acquisition module is used to connect to various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through a mapping function;

[0061] Furthermore, the processing flow of the multi-source data acquisition module includes:

[0062] It can connect to various intelligent monitoring devices to obtain raw monitoring data from each sensor;

[0063] The heterogeneous sensor outputs are converted into a unified standard tuple format containing timestamps, location coordinates, data source type, alarm content, and confidence level through a mapping function.

[0064] Specifically, the intelligent monitoring device connected to the multi-source data acquisition module includes, but is not limited to: intelligent video monitoring (camera), intelligent temperature monitoring (thermal sensor), intelligent pressure monitoring (pressure gauge), intelligent optical monitoring (light signal sensor), and intelligent smoke monitoring (smoke sensor). In one feasible implementation, the set of various sensors is represented as follows: The data set contains data outputs from video, temperature, pressure, optical, and smoke sensors, respectively. Each sensor generates corresponding raw monitoring data and alarm information.

[0065] The raw sensor output is converted into a standardized vector description by a mapping function f, realizing the mapping of heterogeneous information to a standard tuple space and supporting cross-modal fusion. The specific mapping process is as follows:

[0066] ;

[0067] in, Indicates the timestamp, recording the exact time the alarm occurred. This indicates the coordinates of the point and records the spatial location information of the sensor. Indicates the data source type. Indicates the alarm content, Indicates the confidence level of the alarm. The reference time for fusion alignment is indicated, which is the "time anchor point" selected when performing alarm fusion. It is used to determine which alarms can be considered as candidates within the same time window. The subscript indicates the reference time. This indicates different monitoring types, including video, temperature, pressure, optical, and smoke.

[0068] Through the above standardization process, the information from heterogeneous sensors is mapped to a unified tuple space, providing standardized data input for subsequent information fusion modules and supporting correlation analysis and fusion processing between different modal data.

[0069] The information fusion module is used to filter candidate alarm sets based on time window tolerance and spatial error tolerance, and generate composite alarm information by using confidence ranking and splicing and semantic embedding weighted averaging.

[0070] Furthermore, the processing flow of the information fusion module includes:

[0071] Candidate alarm sets are filtered based on the time window tolerance and spatial error tolerance of the reference center point;

[0072] Composite alarm information that can be read by both humans and machines is generated by using a confidence-based sorting and splicing method;

[0073] The text is converted into vectors and fused using semantic embedding and confidence-weighted averaging to generate composite alarm text suitable for large language model input.

[0074] Specifically, the deployed information fusion module performs spatiotemporal fusion and analysis on the collected multi-source monitoring information. For example, when abnormal behavior is detected in a video of a certain area, accompanied by a sudden rise in temperature and pressure fluctuations, it is automatically stitched together into a composite event alarm. This process can greatly improve the confidence of the alarm and avoid false alarms. The system determines the location based on... and timestamp It automatically associates multi-source alarm information to generate composite alarm information, improving alarm accuracy and significantly reducing false alarm rate. The specific design is as follows:

[0075] Defined at the reference center point The set of all nearby candidate alarms is:

[0076] ;

[0077] in, Indicates the tolerance of the time window. This represents the spatial error tolerance, and dist() represents the spatial distance function.

[0078] The alarm content corresponding to the selected alarm information is concatenated. In one feasible implementation, the alarms are concatenated in order of confidence level, for example: Temperature rises sharply , Pressure fluctuations , Video error , = "Sudden temperature rise; pressure fluctuation; video anomaly".

[0079] For large language model input scenarios, an embedding + confidence-weighted semantic fusion approach is adopted. First, a semantic embedding unit is used to convert the text content of each alarm into a vector representation, realizing the mapping from text to numerical vectors. This step can be described as follows: Subsequently, the embedded vectors are subjected to a confidence-weighted average, and the fused semantic vector representation is obtained through weighted calculation. This step can be expressed as: Finally, a large language model is used to generate a composite alarm text description based on the weighted fused semantic vectors. This ensures that the generated text contains key information from multiple alarm sources while also possessing good semantic coherence and comprehensibility. This step can be represented as: .

[0080] Through the above fusion process, when abnormal behavior is detected in a video feed of a certain area, accompanied by a sudden rise in temperature and pressure fluctuations, the system will automatically combine these independent alarms into a single composite event alarm. This process greatly improves the confidence level of the alarm and effectively avoids false alarms.

[0081] The RAG knowledge base module is used to generate composite event descriptions from composite alarm information through a large language model, and then vectorize them as a retrieval index to build a structured knowledge base.

[0082] Furthermore, the construction process of the RAG knowledge base module is as follows: Figure 3As shown, it includes:

[0083] The composite alarm information is used to generate a structured composite event description through a large language model;

[0084] Vectorize the composite events and use them as the retrieval index field;

[0085] Construct a complete knowledge base data structure that includes composite alarms, composite events, locations, root causes, handling suggestions, and handling results.

[0086] Specifically, the RAG knowledge base module constructs a semantic index space for alarm content, further intelligently processing the composite alarm information from the information fusion module. First, the composite alarm message generated by the information fusion module is used as input, and then processed through preset enhancement template embedding. Next, leveraging the language understanding and generation capabilities of a large language model, the original composite alarm information is transformed into a structured, semantically rich composite event description. The resulting composite event description possesses better semantic expressiveness and contextual understanding, providing a high-quality semantic foundation for subsequent retrieval and matching.

[0087] The composite event description is vectorized and mapped to the embedding space. In this step, a text embedding model is used to convert the natural language description of the composite event into a high-dimensional vector representation. The vectorized composite event is then used as a vector index field in the RAG knowledge base, supporting efficient semantic similarity retrieval. The vector indexing mechanism ensures that the system can quickly locate relevant historical events and knowledge based on semantic similarity.

[0088] Finally, a complete knowledge base data structure is constructed, including the following core fields:

[0089] {

[0090] "Combined Alarm": [

[0091] {"Type": "Temperature Exceeded", "Value": "90 degrees"},

[0092] {"Type": "Smoke Alarm", "Value": "0.3"}

[0093] ],

[0094] "Combined event": "A temperature of up to 90 degrees Celsius was detected, along with a smoke concentration of 0.3."

[0095] Location: Equipment Room, Area A

[0096] Root cause: Potential fire risk due to equipment overheating.

[0097] "Recommendation": "Immediately activate the fire suppression system and evacuate all relevant personnel."

[0098] "Resolution Outcome": "Successfully resolved; equipment restored to normal operation."

[0099] }

[0100] By supporting semantic retrieval through vectorized indexing, when a new composite alarm is generated, the system can quickly retrieve relevant historical events based on semantic similarity. Simultaneously, a knowledge enhancement mechanism ensures that new events can draw upon historical experience, improving the accuracy and efficiency of decision-making. This RAG knowledge base architecture provides rich knowledge support and efficient retrieval capabilities for subsequent intelligent retrieval and decision generation.

[0101] The intelligent retrieval module uses a hybrid retrieval strategy to retrieve relevant historical events from the knowledge base and dynamically constructs structured prompts.

[0102] Furthermore, the retrieval process of the intelligent retrieval module includes:

[0103] Filter irrelevant data by limiting alarm location parameters through metadata filtering;

[0104] A dense vector construction method is used to perform vector similarity search in the filtered subset to find the most semantically similar historical events;

[0105] Sparse vectors are used for keyword retrieval, and the priority of the retrieval results is further adjusted by reordering.

[0106] Dynamic splicing includes structured prompts such as role definition, task description, current alarms, and prior knowledge.

[0107] Specifically, the intelligent retrieval module first performs a preliminary screening of the RAG knowledge base through a metadata filtering mechanism. The system establishes a spatial index based on the location information of the current alarm and filters out historical data that is not geographically relevant by restricting alarm location parameters.

[0108] In the data subset filtered by metadata, the system uses the BGE-M3 Embedding model to construct dense vectors for deep semantic retrieval. The system identifies the K most semantically similar historical event knowledge blocks through vector similarity search; this process can be represented as: ,in This represents the vectorized representation of historical events in the vector database, ensuring that deep semantic relationships can be captured.

[0109] To supplement the precise word matching that dense vector retrieval may miss, the system first uses the BM25 algorithm to construct sparse vectors for keyword retrieval, and then uses the BGE-Rerank re-ranking model to perform secondary optimization and ranking of the preliminary results of the hybrid retrieval.

[0110] Based on the optimized search results, the system dynamically constructs structured prompts comprising four core components. The role definition section defines the identity and responsibilities of professional security management personnel; the task definition section clarifies the analysis objectives and output requirements, with current actual alarms, current events, and prior knowledge serving as dynamic parameters; the current alarm input section concatenates real-time composite alarm information from the information fusion module; and the prior knowledge section integrates professional knowledge such as the root causes and handling suggestions of relevant historical events retrieved. The final structured prompts provide complete contextual information and knowledge support for subsequent LLM decision analysis.

[0111] Through a multi-level and multi-dimensional retrieval mechanism, the system can accurately locate the most relevant security management knowledge from massive amounts of historical data, providing a high-quality knowledge foundation for intelligent decision-making.

[0112] The LLM decision module is used to output root cause analysis and differentiate between automated execution and manual review of classification and handling suggestions based on composite alarms and prior knowledge;

[0113] Furthermore, the analysis process of the LLM decision module includes:

[0114] Receive dynamic prompts containing the current composite alarm, retrieved prior knowledge, and task instructions;

[0115] Intelligent analysis is performed based on multi-source fusion event descriptions and semantic matching results from the knowledge base.

[0116] The output includes a structured root cause analysis containing event type, confidence level, and evidence of related events;

[0117] Generate classification and handling suggestions that distinguish between automatic execution and manual review, and force additional manual review requirements when the confidence level is lower than a preset threshold.

[0118] Specifically, the LLM decision module receives complete dynamic prompts constructed by the intelligent retrieval module. These prompts include current composite alarm information, retrieved prior knowledge, and preset task instructions. The system performs structured parsing on the input prompts, extracting key information components, including real-time alarm data, experience in handling similar historical events, professional knowledge base content, and specific requirements of the analysis task.

[0119] Based on the received multi-source fusion event descriptions and semantic matching results from the RAG knowledge base, the LLM decision module performs deep intelligent analysis. Leveraging the powerful language understanding and reasoning capabilities of the large language model, the system correlates and analyzes composite alarm information from different sensors with historical event data to identify potential patterns and causal relationships within the events.

[0120] The system outputs structured root cause analysis results that include event type, confidence level, and evidence of related events. The specific output format includes root cause analysis fields, where the event type identifies the specific category of the safety event, the confidence level reflects the reliability of the analysis results, and the evidence of related events lists the specific sensor data and historical case matching information that support the root cause determination.

[0121] The LLM decision module generates categorized handling suggestions that differentiate between automated execution and manual review. For operations that can be standardized, the system outputs automated instructions marked "auto," including specific control instructions and execution confidence levels. For complex decisions requiring manual judgment, the system outputs manual review instructions marked "manual," including specific matters requiring manual confirmation and operational instructions. When the root cause probability or action confidence level falls below a preset threshold, the system forcibly adds a manual review requirement. The confidence level verification mechanism ensures the reliability of critical decisions by setting a safety threshold. When the system's analysis results are insufficiently certain, it automatically triggers a manual intervention process. In this case, the system adds an "Alarm Prompt" field to the output, containing "[HUMAN] Request expert intervention for assessment," ensuring that uncertain security events receive timely attention and handling from professionals. In one feasible implementation, the handling suggestion format is as follows:

[0122] {

[0123] Root cause analysis: {

[0124] Type: "Pipe rupture"

[0125] Confidence level: 0.85

[0126] Related Events: ["Sudden Temperature Rise + Smoke Alarm", "Historical Similar Event Match Rate 82%"]

[0127] },

[0128] "Recommendations": [

[0129] {

[0130] Steps: Close valve X

[0131] Type: "Automatic Execution"

[0132] "Control command": "[ACTION] VALVE_X CLOSE",

[0133] Confidence level: 0.91

[0134] },

[0135] {

[0136] Steps: "Confirm the toxicity of the leaked substance",

[0137] Type: Manual Review

[0138] Note: [HUMAN] Requires chemical sensor verification.

[0139] }

[0140] ],

[0141] Alert Notice: "[HUMAN] Requesting expert intervention for analysis"

[0142] }

[0143] The root cause analysis section provides event classification and confidence level assessment; the action recommendation section distinguishes between automated execution steps and manual review steps, with each step including specific operation content, execution type, and confidence level information; when manual intervention is required, the alarm prompt section clearly indicates the decision-making stage requiring expert participation. This standardized output format provides clear and actionable decision guidance for the subsequent dual-path response module.

[0144] The dual-path response module is used to distribute decision recommendations to managers and automated agent systems in parallel, enabling collaborative execution of manual review and automated equipment control.

[0145] Furthermore, the execution flow of the dual-path response module includes:

[0146] Intelligent decision-making suggestions are distributed in parallel to two terminals: management and maintenance personnel and automated agent system. Management and maintenance personnel receive instructions that require manual review and are responsible for manual judgment, decision support and key point review.

[0147] The automated agent system parses executable instructions and converts natural language commands into device control instructions through a preset instruction mapping table, which are then executed by standardized machinery.

[0148] Specifically, the system employs a message routing mechanism to automatically distribute different types of instructions to the corresponding processing terminals based on the type identifier (type: "auto" or type: "manual") in the handling suggestions. Management and maintenance personnel receive manual review instructions marked as "manual" and are responsible for handling complex situations requiring professional judgment and experience-based decision-making. The automated agent system specifically handles executable instructions marked as "auto." Through a pre-set instruction mapping table, it converts the natural language commands output by the large language model into specific device control instructions. The instruction mapping table establishes a correspondence between natural language descriptions and device control protocols. For example, it converts the natural language instruction "Close valve X" into the standardized control command "[ACTION]VALVE_X CLOSE," and then further converts it into a lower-level control protocol that the device can recognize.

[0149] The dual-path response module organically combines manual and automated execution. The automated path handles standardized, procedural safety operations, such as equipment start-up and shutdown, and valve control, ensuring rapid response in emergencies. The manual path focuses on decision-making processes requiring expertise, experience, and risk assessment, such as identifying hazardous substance types, assessing evacuation areas, and developing follow-up plans. This collaborative mechanism fully leverages the efficiency of the automated system and the flexibility of human decision-making, achieving optimal configuration for safety management.

[0150] The feedback optimization module collects processing data and adjusts the knowledge base weights and decision templates based on the processing results to achieve continuous system optimization. Further, the optimization process of the feedback optimization module includes:

[0151] Collect data on the original decisions, automated execution records, manual review records, and final handling results for each incident.

[0152] Strengthen the weight of relevant knowledge base items based on successful cases, mark decision templates that need revision based on failed cases, and update the root cause and handling suggestion database based on manual corrections;

[0153] The system continuously learns by implementing optimizations to prompt word templates, adjustments to search weights, and improvements to instruction generation strategies.

[0154] Specifically, after each security incident is handled, the feedback optimization module automatically collects a complete dataset covering the entire handling process. The raw decision data includes root cause analysis, handling recommendations, and confidence assessments output by the LLM decision module. The automatic execution log details the specific instruction sets executed by the automated agent system (such as "VALVE_XCLOSE", "FAN_ON", etc.) and their corresponding execution result statuses (such as "success", "timeout", "failure", etc.). In one feasible implementation, after each incident is handled, the system collects the following data and writes it to the emergency knowledge base:

[0155] {

[0156] "Original Decision": {Output from Step 5},

[0157] "Automatic Execution Log": {

[0158] "Instruction set": ["VALVE_X CLOSE", "FAN_ON"],

[0159] Execution result: ["Success", "Timeout"]

[0160] },

[0161] "Manual review record": {

[0162] "Confirmation Item": ["Confirmation of Leaked Substance"],

[0163] "Correction": ["The actual reason is..."]

[0164] },

[0165] Final outcome: "Resolved successfully"

[0166] }

[0167] The system dynamically adjusts the weights of the RAG knowledge base based on the evaluation results of the handling outcomes. For successfully handled cases, the system strengthens the retrieval weight of relevant knowledge base entries, giving these validated historical experiences a higher priority in subsequent similar event searches. Specifically, this is achieved by increasing the weight coefficient of successful cases, thereby improving their scores in semantic similarity calculation. For unsuccessful or ineffective cases, the system reduces the weight of the corresponding knowledge base entries and marks the decision templates and handling plans that need revision, ensuring that erroneous handling experiences do not negatively impact subsequent decisions.

[0168] Implement a continuous learning strategy encompassing three core dimensions. **Prompt word template optimization:** Adjust role definitions and task templates based on feedback data to improve the accuracy and adaptability of LLM decision-making, such as adjusting professional role descriptions and decision requirements based on treatment outcomes. **Search weight adjustment:** Improve the accuracy and relevance of RAG searches by analyzing the characteristic patterns of successful cases and optimizing the weight allocation of similarity calculations. **Command generation strategy improvement:** Improve the mapping rules from natural language to device control commands by recording and analyzing successfully executed automated command patterns, thereby increasing the success rate and reliability of automated execution.

[0169] The feedback optimization module establishes a continuous knowledge accumulation mechanism to ensure that valuable experience from each response is effectively absorbed and utilized by the system. The response suggestions output by LLM are compared and analyzed with the actual response results, automatically identifying the accuracy and effectiveness of the decisions and forming a quantitative assessment of decision quality. The system establishes a dynamic update mechanism for the emergency suggestion library, incorporating new and validated response plans into the standard knowledge base while eliminating suggestions that have been proven ineffective or outdated. The quality assurance mechanism ensures the authority and reliability of the knowledge base through expert review, effectiveness verification, and continuous monitoring, preventing the accumulation and spread of misinformation.

[0170] Example 2:

[0171] In the safety management of modern chemical industrial parks, there is a complex challenge in coordinating multi-source, heterogeneous monitoring equipment. Traditional safety monitoring systems typically operate independently, with video surveillance, temperature sensors, pressure monitoring, and gas detection devices functioning autonomously. When complex safety incidents such as chemical leaks occur, these sensors simultaneously generate numerous alarms, but an effective correlation analysis mechanism is lacking. Maintenance personnel often have to manually correlate and analyze alarm data from different systems within a short period, such as simultaneous alarms like "abnormal movement in the tank area video," "rapid rise in ambient temperature," "abnormal drop in pipeline pressure," and "excessive concentration of hazardous gases." This makes it difficult to quickly determine the root cause of the incident and formulate a reasonable emergency response plan, easily leading to delays or incorrect decisions.

[0172] To address the aforementioned issues, the intelligent security collaborative management method based on multi-source perception and a large language model, as provided in this invention, is adopted. The process is as follows: Figure 2 As shown. The specific implementation process of this method is as follows:

[0173] The multi-source data acquisition module deployed within the chemical industrial park simultaneously connects to video surveillance equipment, temperature sensor arrays, pipeline pressure monitoring systems, and gas detection devices in the tank area. When a minor leak occurs in a tank, each sensor generates raw monitoring data: video surveillance detects abnormal liquid flow near the tank, temperature sensors detect a local temperature rise from 25°C to 45°C, the pressure monitoring system shows a pipeline pressure drop from 2.5 MPa to 1.8 MPa, and the gas detection device detects excessive concentrations of volatile organic compounds. The system uses a mapping function to convert this heterogeneous data into a unified standard tuple format, including precise timestamps, GPS coordinates, sensor type identifiers, alarm descriptions, and confidence scores calculated based on sensor accuracy.

[0174] The system sets a time window tolerance of 30 seconds and a spatial error tolerance of 50 meters. Based on the tank area (GPS coordinates: 116.3974, 39.9093) and the event occurrence time of 14:32:15, relevant alarms are filtered. The system identifies alarms from four sensors that fall within the set spatiotemporal range. A composite alarm message is generated by combining the alarms with confidence scores: "Pipeline pressure drop (0.92); abnormal temperature rise (0.88); gas concentration exceeding standard (0.85); abnormal video flow (0.78)". Simultaneously, semantic embedding technology is used to convert each alarm text into a vector representation. A fused alarm description suitable for large language models is generated by weighted averaging based on confidence scores.

[0175] First, the composite alarm information is used to generate a structured event description of "a suspected chemical leak was detected in the tank area, accompanied by temperature rise and gas diffusion" using a large language model. The system then vectorizes this description and searches the knowledge base, finding three similar historical event cases: a propylene leak in a similar tank in 2023, a pipeline rupture case in 2022, and a valve seal failure event in 2021. The intelligent search module employs a hybrid search strategy, limiting the search to historical events in the tank area through metadata filtering, using the BGE-M3 model for semantic retrieval, combined with BM25 keyword matching, and finally determining the most relevant handling experience through a re-ranking model.

[0176] The system receives dynamic prompts containing current composite alarms, historical case knowledge, and professional role definitions. Based on the role settings of chemical safety experts, the system analyzes the data and outputs a structured root cause analysis: the event type is "chemical leak caused by tank valve seal failure," with a confidence level of 0.87. Corresponding evidence includes "the combination pattern of pressure drop + temperature rise + abnormal gas detection highly matches historical cases." Simultaneously, it generates categorized response recommendations, including automatically executed actions such as "closing the main control valve of the tank," "starting the area ventilation system," and "activating the leak alarm device," as well as manually reviewed actions such as "confirming the specific composition of the leaked substance" and "assessing the personnel evacuation area."

[0177] The decision recommendations were distributed in parallel to the maintenance personnel in the central control room of the chemical industrial park and the automated control system. Upon receiving the manual verification instruction, the maintenance personnel immediately dispatched professionals with a portable gas chromatograph to the site to confirm the composition of the leaked substance and determine the evacuation area based on wind direction and diffusion models. Simultaneously, the automated agent system converted "close the main control valve of the storage tank" into the specific DCS control instruction "TANK_A01_MAIN_VALVE_CLOSE" through a preset instruction mapping table, completing the valve closure operation within 3 seconds, immediately activating the area forced ventilation system, and activating the audible and visual alarm device to notify on-site personnel to evacuate.

[0178] Complete data on the handling of this incident was collected: the initial decision accurately identified the root cause of the valve seal failure; all three automatically executed operations were successfully completed within the expected timeframe; manual verification confirmed the leaked substance to be a low-toxicity industrial solvent; and the final outcome was "successful control of the leak, no casualties, and minimal environmental impact." Based on this successful case, the system strengthened the knowledge base weight of the "pressure drop + temperature rise + gas anomaly" pattern corresponding to valve sealing problems, and added this successful handling solution as a new standard case to the RAG knowledge base, providing more accurate decision support for similar future events.

[0179] Through the complete implementation process described above, this invention successfully shortens the time required for complex safety events, which traditionally require 15-30 minutes of manual analysis and decision-making, to within 3 minutes for intelligent analysis and automated response, significantly improving the safety management efficiency and response speed of chemical industrial parks.

[0180] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent security collaborative management system based on multi-source perception and a large language model, characterized in that: include: The multi-source data acquisition module is used to connect to various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through a mapping function; The information fusion module is used to filter candidate alarm sets based on time window tolerance and spatial error tolerance, and generate composite alarm information by using confidence ranking and splicing and semantic embedding weighted averaging. The RAG knowledge base module is used to generate composite event descriptions from composite alarm information through a large language model, and then vectorize them as a retrieval index to build a structured knowledge base. The intelligent retrieval module uses a hybrid retrieval strategy to retrieve relevant historical events from the knowledge base and dynamically constructs structured prompts. The LLM decision module is used to output root cause analysis and differentiate between automated execution and manual review of classification and handling suggestions based on composite alarms and prior knowledge; The dual-path response module is used to distribute decision recommendations to managers and automated agent systems in parallel, enabling collaborative execution of manual review and automated equipment control. The feedback optimization module is used to collect disposal data and adjust the knowledge base weights and decision templates based on the disposal results, so as to achieve continuous system optimization. The processing flow of the information fusion module includes: Candidate alarm sets are filtered based on the time window tolerance and spatial error tolerance of the reference center point; Composite alarm information that can be read by both humans and machines is generated by using a confidence-based sorting and splicing method; The text is converted into vectors and fused using semantic embedding and confidence-weighted averaging to generate composite alarm text suitable for large language model input. The construction process of the RAG knowledge base module includes: The composite alarm information is used to generate a structured composite event description through a large language model; Vectorize the composite events and use them as the retrieval index field; Construct a complete knowledge base data structure that includes composite alarms, composite events, locations, root causes, handling suggestions, and handling results; The retrieval process of the intelligent retrieval module includes: Filter irrelevant data by limiting alarm location parameters through metadata filtering; A dense vector construction method is used to perform vector similarity search in the filtered subset to find the most semantically similar historical events; Sparse vectors are used for keyword retrieval, and the priority of the retrieval results is further adjusted by reordering. Dynamic splicing includes structured prompts such as role definition, task description, current alarms, and prior knowledge.

2. The intelligent security collaborative management system according to claim 1, characterized in that, The processing flow of the multi-source data acquisition module includes: It can connect to various intelligent monitoring devices to obtain raw monitoring data from each sensor; The heterogeneous sensor outputs are converted into a unified standard tuple format containing timestamps, location coordinates, data source type, alarm content, and confidence level through a mapping function.

3. The intelligent security collaborative management system according to claim 1, characterized in that, The analysis process of the LLM decision module includes: Receive dynamic prompts containing the current composite alarm, retrieved prior knowledge, and task instructions; Intelligent analysis is performed based on multi-source fusion event descriptions and semantic matching results from the knowledge base. The output includes a structured root cause analysis containing event type, confidence level, and evidence of related events; Generate classification and handling suggestions that distinguish between automatic execution and manual review, and force additional manual review requirements when the confidence level is lower than a preset threshold.

4. The intelligent security collaborative management system according to claim 1, characterized in that, The execution flow of the dual-path response module includes: Intelligent decision-making suggestions are distributed in parallel to two terminals: management and maintenance personnel and automated agent system. Management and maintenance personnel receive instructions that require manual review and are responsible for manual judgment, decision support and key point review. The automated agent system parses executable instructions and converts natural language commands into device control instructions through a preset instruction mapping table, which are then executed by standardized machinery.

5. The intelligent security collaborative management system according to claim 1, characterized in that, The optimization process of the feedback optimization module includes: Collect data on the original decisions, automated execution records, manual review records, and final handling results for each incident. Strengthen the weight of relevant knowledge base items based on successful cases, mark decision templates that need revision based on failed cases, and update the root cause and handling suggestion database based on manual corrections; The system continuously learns by implementing optimizations to prompt word templates, adjustments to search weights, and improvements to instruction generation strategies.

6. A method for intelligent security collaborative management based on multi-source perception and a large language model, used in the intelligent security collaborative management system described in any one of claims 1-5, characterized in that, include: It can connect to various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through mapping functions; Candidate alarm sets are selected based on time window tolerance and spatial error tolerance, and composite alarm information is generated by confidence ranking and splicing and semantic embedding weighted averaging. Composite alarm information is used to generate composite event descriptions through a large language model, and these descriptions are vectorized and used as retrieval indexes to build a structured knowledge base. The hybrid retrieval strategy retrieves relevant historical events from the knowledge base and dynamically constructs structured prompts; Based on composite alarms and prior knowledge, the system outputs root cause analysis and classification and handling suggestions that differentiate between automated execution and manual review; Decision recommendations are distributed in parallel to managers and automated agent systems, enabling collaborative execution of manual review and automated equipment control; Collect and process data, and adjust the knowledge base weights and decision templates based on the processing results to achieve continuous system optimization; The processing flow of the information fusion module includes: Candidate alarm sets are filtered based on the time window tolerance and spatial error tolerance of the reference center point; Composite alarm information that can be read by both humans and machines is generated by using a confidence-based sorting and splicing method; The text is converted into vectors and fused using semantic embedding and confidence-weighted averaging to generate composite alarm text suitable for large language model input. The construction process of the RAG knowledge base module includes: The composite alarm information is used to generate a structured composite event description through a large language model; Vectorize the composite events and use them as the retrieval index field; Construct a complete knowledge base data structure that includes composite alarms, composite events, locations, root causes, handling suggestions, and handling results; The retrieval process of the intelligent retrieval module includes: Filter irrelevant data by limiting alarm location parameters through metadata filtering; A dense vector construction method is used to perform vector similarity search in the filtered subset to find the most semantically similar historical events; Sparse vectors are used for keyword retrieval, and the priority of the retrieval results is further adjusted by reordering. Dynamic splicing includes structured prompts such as role definition, task description, current alarms, and prior knowledge.

Images