Intelligent security collaborative management system based on multi-source perception and language large model

Abstract

The invention relates to the field of multi-source data management, in particular to an intelligent security collaborative management system based on multi-source perception and a large language model. Comprising a multi-source data acquisition module which is accessed to various intelligent monitoring devices and is output and converted into a unified standard tuple format through a mapping function; the information fusion module is used for screening a candidate alarm set according to the space-time tolerance, and generating composite alarm information by adopting confidence ranking and semantic embedding weighted averaging; the RAG knowledge base module is used for generating a composite event description through a large language model and vectorizing the composite event description as a retrieval index; the intelligent retrieval module is used for acquiring related historical events by adopting a mixed retrieval strategy and constructing structured cue words; the LLM decision module is used for outputting root cause analysis and classification disposal suggestions based on the composite alarm and the priori knowledge; the double-path response module is used for distributing decision suggestions to management personnel and an agent system to realize collaborative execution; and the feedback optimization module is used for collecting disposal data and adjusting the weight of the knowledge base and the decision template to realize continuous optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-source data management, and in particular to an intelligent safety collaborative management system based on multi-source perception and language large models. BACKGROUND

[0002] Traditional safety management systems have long been limited by the information silo effect and high false alarm rate problems. The alarm information generated by various sensors is often independent of each other, lacking effective cross-modal correlation analysis capability. This fragmented data processing method requires operators to manually correlate multiple isolated alarms (such as "video anomaly" + "temperature sudden rise"), which is inefficient and prone to missing critical correlation points. With the popularization of Internet of Things technology, the multi-source heterogeneous data generated in modern industrial environments has shown explosive growth, and traditional rule and threshold-based alarm systems have been unable to meet the safety monitoring needs in complex environments, with high false alarm rates and high risk of missed alarms. For example, a single sensor pressure drop alarm without video confirmation of a liquid spray alarm may overlook a leakage event.

[0003] In recent years, the development of multi-source data fusion technology has provided a new path to break this bottleneck. This technology converts data from different sensors into a unified event description through spatio-temporal correlation analysis and context semantic modeling. For example, when video monitoring identifies abnormal behavior, the system can synchronize and correlate "illegal door opening" events from access control sensors and abnormal fluctuations from temperature sensors in the same area to generate a high-confidence "intrusion event" determination. SUMMARY

[0004] To address the above shortcomings, the present application provides an intelligent safety collaborative management system based on multi-source perception and language large models, aiming to improve the technical defects of severe information silos, high false alarm and missed alarm rates, and low automation in traditional safety management systems.

[0005] In a first aspect, the present application provides the following technical solution: an intelligent safety collaborative management system based on multi-source perception and language large models, comprising:

[0006] A multi-source data acquisition module for accessing multiple intelligent monitoring devices and converting the raw output of each sensor into a unified standard tuple format through a mapping function;

[0007] An information fusion module for filtering a candidate alarm set according to time window tolerance and spatial error tolerance, and generating composite alarm information using confidence ranking splicing and semantic embedding weighted average;

[0008] An RAG knowledge base module for generating a composite event description from the composite alarm information through a large language model, and vectorizing it as a retrieval index to build a structured knowledge base;

[0009] An intelligent retrieval module acquires relevant historical events from the knowledge base using a hybrid retrieval strategy and dynamically constructs structured prompt words;

[0010] An LLM decision module outputs root cause analysis and differentiated automatic execution and manual review classification disposal suggestions based on composite alarms and prior knowledge;

[0011] A dual-path response module is configured to distribute decision suggestions to management personnel and automated agent systems in parallel to realize the collaborative execution of manual judgment review and automated device control;

[0012] A feedback optimization module is configured to collect disposal data and adjust the knowledge base weight and decision template based on the disposal effect to realize continuous optimization of the system.

[0013] Further, the processing flow of the multi-source data acquisition module includes:

[0014] Access multiple intelligent monitoring devices to obtain raw monitoring data from various sensors;

[0015] The heterogeneous sensor output is converted into a unified standard tuple format containing timestamps, location coordinates, data source types, alarm content, and confidence levels through a mapping function.

[0016] Further, the processing flow of the information fusion module includes:

[0017] Filter the candidate alarm set according to the time window tolerance and spatial error tolerance of the reference center point;

[0018] Generate composite alarm information for human-machine co-reading using a confidence ranking splicing method;

[0019] Use semantic embedding and confidence weighted average to convert text to vectors and perform fusion to generate composite alarm text suitable for large language model input.

[0020] Further, the construction process of the RAG knowledge base module includes:

[0021] Generate structured composite event descriptions from composite alarm information using a large language model;

[0022] Vectorize the composite event and use it as a retrieval index field;

[0023] Construct a complete knowledge base data structure containing composite alarms, composite events, locations, root causes, disposal suggestions, and disposal results.

[0024] Further, the retrieval process of the intelligent retrieval module includes:

[0025] Filter irrelevant data by limiting alarm location parameters through metadata filtering;

[0026] The dense vector construction method is used to search for the most similar historical events in the filtered subset according to the vector similarity;

[0027] The sparse vector is used for keyword retrieval, and the priority of the retrieval result is further adjusted through reordering;

[0028] The structured prompt word dynamically spliced includes role definition, task description, current alarm and priori knowledge.

[0029] Further, the analysis process of the LLM decision module includes:

[0030] Receive dynamic prompt words containing current composite alarms, retrieved prior knowledge and task instructions;

[0031] Intelligent analysis based on multi-source fusion event description and knowledge base semantic matching results;

[0032] Output structured root cause analysis including event type, confidence and associated event evidence;

[0033] Generate a classification disposal suggestion distinguishing between automatic execution and manual review, and when the confidence is lower than the preset threshold, forcibly add a manual review requirement.

[0034] Further, the execution process of the double-path response module includes:

[0035] Intelligent decision suggestions are distributed in parallel to management and operation personnel and automatic agent systems two terminals; management and operation personnel receive instructions requiring manual review, responsible for manual judgment, decision support and key point review;

[0036] The automatic agent system analyzes executable instructions, converts natural language commands to device control instructions through a preset instruction mapping table, and is executed by a standardized machine.

[0037] Further, the optimization process of the feedback optimization module includes:

[0038] Collect raw decisions, automatic execution records, manual review records and final disposal result data of each event processing;

[0039] Based on successful cases, the weight of related knowledge base entries is strengthened, based on failed cases, the decision template to be revised is marked, and according to the manual correction, the root cause and disposal suggestion library are updated;

[0040] Implement prompt word template optimization, retrieval weight adjustment and instruction generation strategy improvement to realize system continuous learning.

[0041] In a second aspect, the present application provides the following technical solutions: an intelligent safety collaborative management method based on multi-source perception and language large model, comprising:

[0042] Access a variety of intelligent monitoring devices, and convert the raw output of each sensor into a unified standard tuple format through a mapping function;

[0043] Filter the candidate alarm set according to the time window tolerance and spatial error tolerance, and generate composite alarm information by using confidence ranking splicing and semantic embedding weighted average;

[0044] Generate a composite event description through a large language model based on the composite alarm information, and vectorize it as a retrieval index to build a structured knowledge base;

[0045] A mixed retrieval strategy is used to obtain related historical events from the knowledge base and dynamically build structured prompt words;

[0046] Based on the composite alarm and prior knowledge, root cause analysis and classification of automatic execution and manual review are output to provide treatment suggestions;

[0047] The decision suggestions are distributed to management personnel and automated agent systems in parallel to realize the collaborative execution of manual judgment review and automated device control;

[0048] Collect treatment data and adjust the knowledge base weight and decision template based on the treatment effect to realize continuous optimization of the system.

[0049] The present application has the following beneficial effects:

[0050] 1. In the present application, through multi-source data fusion, the system can associate multiple sensor data to generate composite event alarms. For example, in the case of temperature abnormalities, video monitoring data is combined to verify abnormal behavior and reduce false positives. Only after cross-verification of multiple data, the system will trigger the final alarm, greatly improving the accuracy of the alarm and reducing unnecessary interference.

[0051] 2. In the present application, intelligent analysis is performed based on a pre-set business semantic template. The system can automatically analyze alarm information, analyze the cause and generate emergency steps according to business rules, making the decision-making process more intelligent and automated. Through natural language processing technology, the system can quickly generate actionable suggestions and solutions, significantly improving response efficiency and decision quality.

[0052] 3. In the present application, by analyzing the standardized instructions generated by LLM, mechanical operations such as valve control, device start-stop, etc. can be performed. This automated response reduces the traditional manual operation delay from minutes to milliseconds, greatly improving the response speed and execution efficiency of the system, ensuring that critical operations are quickly implemented in emergency situations. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1A structural diagram of the intelligent safety collaborative management system based on multi-source perception and language large model according to the present application is shown in

[0054] Figure 2 A structural diagram of the intelligent safety collaborative management system according to the present application is shown in

[0055] Figure 3 A flowchart of the process of constructing the RAG knowledge base according to the present application is shown in

[0056] Figure 4 A flowchart of the intelligent safety collaborative management system based on multi-source perception and language large model according to the present application is shown in DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0058] Embodiment One

[0059] In the first embodiment of the present application, the present application provides an intelligent safety collaborative management system based on multi-source perception and language large model, and the specific architecture is shown in Figure 1 Based on the architecture, modular design is performed, and the structural diagram obtained is shown in Figure 2

[0060] The multi-source data acquisition module is used to access multiple intelligent monitoring devices, and the original output of each sensor is converted into a unified standard tuple format through a mapping function;

[0061] Further, the processing flow of the multi-source data acquisition module includes:

[0062] Accessing multiple intelligent monitoring devices to obtain the original monitoring data of each sensor;

[0063] Converting the output of heterogeneous sensors into a unified standard tuple format containing time stamp, position coordinates, data source type, alarm content and confidence through a mapping function.

[0064] Specifically, the intelligent monitoring devices accessed by the multi-source data acquisition module include but are not limited to video intelligent monitoring (camera), temperature intelligent monitoring (thermal sensor), pressure intelligent monitoring (pressure gauge), optical intelligent monitoring (optical signal sensor) and smoke intelligent monitoring (smoke sensor), etc. In a feasible implementation manner, let the set of various sensors be represented as ​, each item in the set corresponds to the data output of the video, temperature, pressure, optical, and smoke sensor respectively. Each sensor generates corresponding raw monitoring data and alarm information.

[0065] The mapping of heterogeneous information to the standard tuple space is achieved by converting the raw sensor output into a standardized vector description through the mapping function f, supporting cross-modal fusion. The specific mapping process is as follows:

[0066] ;

[0067] wherein, represents the timestamp, recording the exact time of the alarm, represents the point coordinate, recording the spatial position information of the sensor, represents the data source type, represents the alarm content, represents the alarm confidence, represents the reference time of fusion alignment, i.e. the "time anchor point" selected when performing alarm fusion, used to determine which can be considered as candidates within the same time window, subscript represents different monitoring types, including video, temperature, pressure, optical, and smoke.

[0068] Through the above standardization process, the mapping of heterogeneous sensor information to a unified tuple space is achieved, providing standardized data input for the subsequent information fusion module, supporting correlation analysis and fusion processing between different modal data.

[0069] The information fusion module is used to filter the candidate alarm set according to the time window tolerance and spatial error tolerance, and generate composite alarm information using confidence ranking splicing and semantic embedding weighted average methods;

[0070] Further, the processing flow of the information fusion module includes:

[0071] Filtering the candidate alarm set according to the time window tolerance and spatial error tolerance of the reference center point;

[0072] Generating human-machine co-reading composite alarm information using confidence ranking splicing method;

[0073] Using semantic embedding and confidence weighted average method to convert text to vector and perform fusion, generating composite alarm text suitable for large language model input.

[0074] Specifically, through the deployed information fusion module, the collected multi-source monitoring information is fused and analyzed in time and space dimensions. For example: when abnormal behavior is detected in a certain area of video, accompanied by a sudden rise in temperature and pressure fluctuations, it is automatically spliced into a composite event alarm. This process can greatly improve the confidence of the alarm and avoid false positives. The system can automatically generate a composite alarm according to the location and timestamp , automatically associate multi-source alarm information to generate composite alarm information, improve alarm accuracy and significantly reduce false alarm rate. The specific design is as follows:

[0075] define all candidate alarm sets near the reference center point as:

[0076] ;

[0077] wherein, represents the time window tolerance, represents the spatial error tolerance, and dist() represents the spatial distance function.

[0078] Splice the alarm content corresponding to the selected alarm information. In a feasible implementation manner, the confidence level is spliced, for example: : temperature surge , : pressure fluctuation , : video anomaly , = “temperature surge; pressure fluctuation; video anomaly”.

[0079] For large language model input scenarios, an Embedding+confidence weighted semantic fusion method is adopted. First, a semantic embedder is used to convert the text content of each alarm into a vector representation, realizing the mapping from text to numerical vector. This step can be expressed as: . Then, the embedding vector is processed by confidence weighted average, and the fused semantic vector representation is obtained by weighted calculation. This step can be expressed as: . Finally, a large language model is used to generate a composite alarm text description based on the weighted fused semantic vector, ensuring that the generated text contains key information of multi-source alarms and has good semantic coherence and understandability. This step can be expressed as: .

[0080] Through the above fusion processing, when an abnormal behavior is detected in a certain area video, and at the same time accompanied by a temperature surge and pressure fluctuation, the system will automatically splice these independent alarms into a composite event alarm. This process can greatly improve the confidence of the alarm and effectively avoid false alarms.

[0081] The RAG knowledge base module is used to generate a composite event description through a large language model, and the vectorization is used as a retrieval index to build a structured knowledge base.

[0082] Further, the construction process of the RAG knowledge base module is as follows Figure 3As shown, including:

[0083] Generate structured composite event description through large language model

[0084] Vectorize the composite event as a retrieval index field

[0085] Build a complete knowledge base data structure containing composite alarms, composite events, locations, root causes, treatment suggestions, and treatment results.

[0086] Specifically, the RAG knowledge base module constructs a semantic index space for alarm content and further processes the composite alarm information from the information fusion module. First, the composite alarm message generated by the information fusion module is taken as input, and then embedded processing is performed through a pre-set promotion template. Then, the language understanding and generation capabilities of the large language model are used to convert the original composite alarm information into a structured and semantically rich composite event description. Finally, the generated composite event description has better semantic expression ability and context understanding, providing a high-quality semantic basis for subsequent retrieval and matching.

[0087] Vectorize the composite event description and map it to the embedding space In this step, a text embedding model is used to convert the natural language description of the composite event into a high-dimensional vector representation. The vectorized composite event is then used as a vector index field in the RAG knowledge base, supporting efficient semantic similarity retrieval. The vector index mechanism ensures that the system can quickly locate related historical events and knowledge based on semantic similarity.

[0088] Finally, a complete knowledge base data structure is constructed, containing the following core fields:

[0089] {

[0090] "Composite Alarms": [

[0091] {"Type": "Temperature Exceeded", "Value": "90 degrees"},

[0092] {"Type": "Smoke Alarm", "Value": "0.3"}

[0093] ],

[0094] "Composite Event": "Detected temperature up to 90 degrees, smoke concentration 0.3",

[0095] "Location": "A area equipment room",

[0096] "Root Cause": "Potential fire risk caused by equipment overheating",

[0097] "Disposal suggestion": "Start fire extinguishing system immediately, evacuate relevant personnel"

[0098] "Disposal result": "Successful disposal, equipment returned to normal"

[0099] }.

[0100] The semantic retrieval function is supported by vectorization index. When a new composite alarm occurs, the system can quickly retrieve related historical events based on semantic similarity. At the same time, the knowledge enhancement mechanism ensures that new events can learn from historical experience, improving the accuracy and efficiency of decision-making. This RAG knowledge base architecture provides rich knowledge support and efficient retrieval capability for subsequent intelligent retrieval and decision-making generation.

[0101] The intelligent retrieval module uses a hybrid retrieval strategy to obtain relevant historical events from the knowledge base and dynamically constructs structured prompt words;

[0102] Further, the retrieval process of the intelligent retrieval module includes:

[0103] Filtering by metadata limits the alarm location parameter to filter out irrelevant data;

[0104] A dense vector construction method is used to search for the most similar historical events in the filtered subset based on vector similarity;

[0105] Sparse vectors are used for keyword retrieval, and the priority of the retrieval results is further adjusted through reordering;

[0106] Dynamically splice structured prompt words containing role definition, task description, current alarm, and prior knowledge.

[0107] Specifically, the intelligent retrieval module first filters the RAG knowledge base through the metadata filtering mechanism. The system establishes a spatial index based on the location information of the current alarm and filters out historical data that is not related to the geographical location by limiting the alarm location parameter.

[0108] In the subset of data after metadata filtering, the system uses the BGE-M3 Embedding model to construct a dense vector for deep semantic retrieval. The system finds the K most similar historical event knowledge blocks in terms of semantics through vector similarity search. This process can be represented as: Where represents the vectorization representation of the corresponding historical event in the vector database, ensuring that the deep semantic relationship can be captured.

[0109] To complement the precise lexical matching that dense vector retrieval might miss, the system first constructs a sparse vector using the BM25 algorithm for keyword retrieval, and then uses the BGE-Rerank re-ranking model to perform secondary optimization sorting on the preliminary results of hybrid retrieval.

[0110] Based on the optimized retrieval results, the system dynamically constructs a structured prompt word containing four core components. The role definition part sets the professional safety management personnel identity and responsibility requirements; the task definition part clearly defines the analysis target and output requirements, with the current actual alarm, current event and prior knowledge as dynamic parameters; the current alarm input part splices real-time composite alarm information from the information fusion module; the prior knowledge part integrates the root cause, disposal suggestions and other professional knowledge of the relevant historical events retrieved. The structured prompt word finally constructed provides complete contextual information and knowledge support for subsequent LLM decision analysis.

[0111] Through a multi-level, multi-dimensional retrieval mechanism, the system can accurately locate the most relevant safety management knowledge from massive historical data, providing a high-quality knowledge base for intelligent decision-making.

[0112] The LLM decision module is used to output root cause analysis and classified disposal suggestions for automatic execution and manual review based on composite alarms and prior knowledge;

[0113] Further, the analysis process of the LLM decision module includes:

[0114] Receiving dynamic prompt words containing current composite alarms, retrieved prior knowledge and task instructions;

[0115] Intelligent analysis based on multi-source fusion event description and knowledge base semantic matching results;

[0116] Outputting structured root cause analysis containing event type, confidence and associated event evidence;

[0117] Generating classified disposal suggestions for automatic execution and manual review, and when the confidence is lower than the preset threshold, forcibly adding the requirement for manual review.

[0118] Specifically, the LLM decision module receives the complete dynamic prompt word constructed by the intelligent retrieval module, which contains current composite alarm information, retrieved prior knowledge and preset task instructions. The system performs structured analysis on the input prompt word, extracting key information components including real-time alarm data, disposal experience of similar historical events, professional knowledge base content and specific requirements of analysis tasks.

[0119] Based on the received multi-source fusion event description and the semantic matching results of the RAG knowledge base, the LLM decision module conducts deep intelligent analysis. The system utilizes the powerful language understanding and reasoning capabilities of large language models to correlate and analyze complex alarm information from different sensors with historical event data, identifying potential patterns and causal relationships of events.

[0120] The system outputs structured root cause analysis results containing event type, confidence, and associated event evidence. The specific output format includes a root cause analysis field, where the event type identifies the specific classification of the security event, the confidence value reflects the reliability of the analysis results, and the associated event evidence lists specific sensor data and historical case matching information supporting the root cause judgment.

[0121] The LLM decision module generates two types of classification disposal suggestions: automatic execution and manual review. For operations that can be standardized, the system outputs an automatic instruction marked as "auto" type, containing specific control instruction content and execution confidence. For complex decisions that require human judgment, the system outputs a manual review instruction marked as "manual" type, containing specific matters that need human confirmation and operation instructions. When the root cause probability or action confidence is below the preset threshold, the system will forcibly add a manual review requirement. The confidence checking mechanism ensures the reliability of critical decisions by setting a safety threshold. When the system's analysis results are not certain enough, it will automatically trigger a manual intervention process. In this case, the system will add an "alert prompt" field in the output results, with the content "[HUMAN] Request expert intervention and research", ensuring that uncertain security events can be promptly attended to and handled by professionals. In a feasible implementation, the format of the disposal suggestion is:

[0122] {

[0123] "Root Cause Analysis": {

[0124] "Type": "Pipeline rupture",

[0125] "Confidence": 0.85,

[0126] "Associated Events": ["Temperature sudden rise + smoke alarm", "Historical similar event matching rate 82%"]

[0127] },

[0128] "Disposal Suggestions": [

[0129] {

[0130] "Step": "Close valve X",

[0131] "Type": "Automatic execution",

[0132] "Control instruction": "[ACTION] VALVE_X CLOSE",

[0133] "Confidence": 0.91

[0134] },

[0135] {

[0136] "Step": "Confirm the toxicity of the leaked substance",

[0137] "Type": "Manual review",

[0138] "Explanation": "[HUMAN] Requires chemical sensor review"

[0139] }

[0140] ],

[0141] "Alert prompt": "[HUMAN] Request expert intervention for judgment"

[0142] }。

[0143] The root cause analysis section provides event classification and confidence evaluation; the treatment suggestion section distinguishes between automatic execution steps and manual review steps, each containing specific operation content, execution type and confidence information; when manual intervention is required, the alarm prompt section clearly marks the decision-making link that requires expert participation. This standardized output format provides clear and operable decision guidance for the subsequent double-path response module.

[0144] The double-path response module is used to distribute decision suggestions to management personnel and automated agent systems in parallel, realizing the coordinated execution of manual judgment review and automated device control.

[0145] Further, the execution process of the double-path response module includes:

[0146] Intelligent decision suggestions are distributed to management and operation personnel and automated agent systems in parallel; management and operation personnel receive instructions that require manual review and are responsible for manual judgment, decision support and key point review.

[0147] The automated agent system parses executable instructions, converts natural language commands to device control instructions through a pre-set instruction mapping table, and performs standard mechanical execution.

[0148] Specifically, the system adopts a message routing mechanism to automatically distribute different types of instructions to corresponding processing terminals according to the type identification (type: "auto" or type: "manual") in the treatment suggestion. The management and operation personnel terminal receives manual review instructions marked as "manual" type, and is responsible for handling complex situations that require professional judgment and experience decision-making. The automated agent system is specially designed to handle executable instructions marked as "auto" type, and converts natural language commands output by the large language model into specific device control instructions through a pre-configured instruction mapping table. The instruction mapping table establishes a correspondence between natural language descriptions and device control protocols, for example, converting the natural language instruction "close valve X" into the standardized control command "[ACTION] VALVE_X CLOSE", and further converting it into a bottom-layer control protocol that can be recognized by the device.

[0149] The dual-path response module realizes the organic combination of manual and automated execution. The automated path is responsible for executing standardized and programmed safety operations, such as device start-stop, valve control, and other mechanical actions, to ensure rapid response in emergency situations. The manual path focuses on decision-making aspects that require professional knowledge, experience judgment, and risk assessment, such as confirming the type of hazardous substances, assessing the evacuation range, and developing subsequent disposal plans. This division of labor and cooperation mechanism fully utilizes the efficiency of the automated system and the flexibility of manual decision-making, achieving optimal configuration of safety management.

[0150] The feedback optimization module is used to collect treatment data and adjust the knowledge base weights and decision templates based on the treatment effect, realizing continuous optimization of the system. Further, the optimization process of the feedback optimization module includes:

[0151] Collecting raw decision-making, automatic execution records, manual review records, and final treatment result data for each event handling;

[0152] Based on successful cases, the weights of relevant knowledge base entries are strengthened, and based on failed cases, the decision templates that need to be revised are marked, and the root cause and treatment suggestion library are updated according to manual correction;

[0153] Implementing prompt word template optimization, retrieval weight adjustment, and instruction generation strategy improvement to realize continuous learning of the system.

[0154] Specifically, the feedback optimization module automatically collects a complete dataset covering the entire treatment process after each safety event is handled. The original decision data includes the root cause analysis, treatment recommendations, and confidence assessments output by the LLM decision module, among other decision-making content. The automatic execution record details the specific instruction set executed by the automated agent system (such as "VALVE_XCLOSE," "FAN_ON," etc.) and its corresponding execution result status (such as "success," "timeout," "failure," etc.). In a feasible implementation, after each event handling is completed, the system collects the following data and writes it into the emergency knowledge base:

[0155] {

[0156] "Original Decision": {Step 5 output content},

[0157] "Automatic Execution Record": {

[0158] "Instruction Set": ["VALVE_X CLOSE", "FAN_ON"],

[0159] "Execution Result": ["success", "timeout"]

[0160] },

[0161] "Manual Review Record": {

[0162] "Confirmation Items": ["leakage substance confirmation"],

[0163] "Correction Items": ["actual cause is..."]

[0164] },

[0165] "Final Treatment Result": "treatment success"

[0166] }。

[0167] Based on the evaluation results of the treatment effect, the RAG knowledge base is dynamically adjusted in weight. For cases where the treatment is successful, the system will strengthen the retrieval weight of the relevant knowledge base entries, so that these verified effective historical experiences will have a higher priority in the retrieval of similar events in the future. The specific implementation is to increase the weight coefficient of successful cases to improve their scores in semantic similarity calculation. For cases where the treatment fails or the effect is not good, the system will reduce the weight of the corresponding knowledge base entries and mark the decision templates and treatment plans that need to be revised to ensure that the wrong treatment experience does not have a negative impact on subsequent decision-making.

[0168] The system continuous learning strategy is implemented, which includes three core dimensions. The prompt word template optimization adjusts the role definition and task template based on feedback data, improving the accuracy and adaptability of LLM decision-making, such as adjusting the description and decision requirements of professional roles according to the treatment effect. Retrieval weight adjustment optimizes the weight distribution of similarity calculation by analyzing the feature patterns of successful cases, improving the accuracy and relevance of RAG retrieval. Instruction generation strategy improvement records and analyzes the successful execution of automated command patterns, optimizes the mapping rules from natural language to device control instructions, and improves the success rate and reliability of automated execution.

[0169] The feedback optimization module establishes a knowledge continuous sedimentation mechanism to ensure that the valuable experience of each treatment can be effectively absorbed and utilized by the system. The treatment suggestions of LLM output are compared and analyzed with the actual treatment results to automatically identify the accuracy and effectiveness of the decision-making, forming a quantitative evaluation of decision-making quality. The system establishes a dynamic updating mechanism for the emergency suggestion library, incorporating verified effective new treatment plans into the standard knowledge base, while eliminating treatment suggestions that have been proven ineffective or outdated. The quality assurance mechanism ensures the authority and reliability of the knowledge base through expert review, effect verification and continuous monitoring, preventing the accumulation and spread of false information.

[0170] Embodiment two:

[0171] In the safety management of modern chemical parks, there is a complex problem of collaborative operation of multiple source heterogeneous monitoring devices. In traditional safety monitoring systems, video monitoring, temperature sensors, pressure monitoring, gas detection and other devices operate independently. When a complex safety event such as chemical leakage occurs, a large number of alarm information will be generated simultaneously by each sensor, but there is a lack of effective correlation analysis mechanism. The operation and maintenance personnel often need to manually correlate and analyze the alarm data from different systems within a short period of time, such as "abnormal movement of video in storage tank area", "rapid rise of ambient temperature", "abnormal drop of pipeline pressure" and "excessive concentration of harmful gas", etc. It is difficult to quickly determine the root cause of the event and develop a reasonable emergency response plan, which may lead to delayed response or incorrect decision-making.

[0172] To solve the above problems, the intelligent safety collaborative management method based on multi-source perception and language large model provided by the present application is adopted, and the process is as shown in Figure 2 The specific implementation process of the method is as follows:

[0173] The multi-source data collection module deployed in the chemical industrial park simultaneously accesses the video monitoring equipment, temperature sensor array, pipeline pressure monitoring system, and gas detection device in the storage tank area. When a slight leak occurs in the storage tank, each sensor generates raw monitoring data: the video monitoring detects abnormal liquid flow near the storage tank, the temperature sensor monitors a local temperature rise from 25°C to 45°C, the pressure monitoring system shows a pipeline pressure drop from 2.5 MPa to 1.8 MPa, and the gas detection device detects a volatile organic compound concentration exceeding the standard. The system converts these heterogeneous data into a unified standard tuple format through a mapping function, including an accurate timestamp, GPS coordinates, sensor type identification, alarm content description, and confidence value based on sensor accuracy calculation.

[0174] The time window tolerance is set to 30 seconds, and the spatial error tolerance is set to 50 meters. Based on the storage tank area (GPS coordinates: 116.3974, 39.9093) and the event occurrence time 14:32:15, relevant alarms are filtered. The system identifies that the alarms of four sensors are within the set time and space range, and generates a composite alarm information "pipeline pressure drop (0.92); temperature anomaly rise (0.88); gas concentration exceeds standard (0.85); video abnormal flow (0.78)" by confidence sorting splicing. At the same time, the semantic embedding technology is used to convert each alarm text into a vector representation, and a weighted average of the confidence is used to generate a fusion alarm description suitable for large language model processing.

[0175] First, the composite alarm information is generated by the large language model to generate a structured event description "storage tank area detects suspected chemical leakage, accompanied by temperature rise and gas diffusion". The system vectorizes this description and retrieves it in the knowledge base, matching 3 similar historical event cases, including a 2023 propylene leakage event similar to the storage tank, a 2022 pipeline rupture case, and a 2021 valve seal failure event. The intelligent retrieval module uses a hybrid retrieval strategy, filters historical events in the storage tank area through metadata, uses the BGE-M3 model for semantic retrieval, combines BM25 keyword matching, and finally determines the most relevant disposal experience through a reordering model.

[0176] Receive dynamic prompt words containing current composite alarms, historical case knowledge, and professional role definitions. The system analyzes based on the role setting of chemical safety experts and outputs a structured root cause analysis: the event type is "chemical leakage caused by storage tank valve seal failure", with a confidence of 0.87, and the associated evidence includes "the combination of pressure drop + temperature rise + gas detection anomaly matches the historical case highly". At the same time, classification disposal suggestions are generated, including automatic execution of "closing the storage tank main control valve", "starting the area ventilation system", and "activating the leakage alarm device", as well as manual review of "confirming the specific composition of the leaked substance" and "evaluating the personnel evacuation range".

[0177] The decision suggestion is distributed to the central control room operation and maintenance personnel and the automatic control system of the chemical industry park in parallel. After receiving the manual review instruction, the operation and maintenance personnel immediately dispatch professional personnel to carry portable gas chromatograph to the site to confirm the leakage material composition, and determine the evacuation area according to the wind direction and diffusion model. At the same time, the automatic agent system converts "close the main valve of the storage tank" into a specific DCS control instruction "TANK_A01_MAIN_VALVE_CLOSE" through the preset instruction mapping table, completes the valve closing operation within 3 seconds, and then starts the regional forced ventilation system, and activates the sound and light alarm device to notify the on-site personnel to evacuate.

[0178] The complete disposal data of this event is collected: the original decision accurately identifies the root cause of the valve sealing failure, the three automatic operations are successfully completed within the expected time, the manual review confirms that the leakage material is a low-toxicity industrial solvent, and the final disposal result is "leakage is successfully controlled, no personnel casualties, and environmental impact is minimized". Based on this successful case, the system strengthens the knowledge base weight of the "pressure drop + temperature rise + gas anomaly" mode corresponding to the valve sealing problem, and adds the successful disposal scheme of this time as a new standard case to the RAG knowledge base, providing more accurate decision support for future similar events.

[0179] Through the above complete implementation process, the present application successfully shortens the traditional 15-30 minutes of manual analysis and decision-making for complex safety events to 3 minutes of intelligent analysis and automatic response, significantly improving the safety management efficiency and response speed of the chemical industry park.

[0180] Finally, it should be noted that: the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features, as long as they are within the spirit and principles of the present application. Any modification, equivalent replacement, improvement, etc. made shall be included in the protection scope of the present application.

Claims

1. An intelligent security collaborative management system based on multi-source perception and language large model, characterized in that, The application comprises the following steps: A multi-source data acquisition module is used to access various intelligent monitoring devices and convert the raw output of each sensor into a unified standard tuple format through a mapping function. An information fusion module is used to filter candidate alarm sets according to the time window tolerance and spatial error tolerance, generate composite alarm information by confidence sorting splicing and semantic embedding weighted average, and output root cause analysis and classification disposal suggestions for automatic execution and manual review. An RAG knowledge base module is used to generate composite event descriptions through a large language model, vectorize the composite alarm information, and use the vectorized information as a retrieval index to build a structured knowledge base. An intelligent retrieval module is used to adopt a hybrid retrieval strategy to obtain relevant historical events from the knowledge base and dynamically build structured prompt words. An LLM decision module is used to output root cause analysis and classification disposal suggestions for automatic execution and manual review based on composite alarms and prior knowledge. A dual-path response module is used to distribute decision suggestions to management personnel and automated agent systems in parallel to realize the collaborative execution of manual judgment review and automated device control. A feedback optimization module is used to collect disposal data and adjust the knowledge base weight and decision template based on the disposal effect to realize continuous optimization of the system.

2. The intelligent safety synergic management system according to claim 1, wherein, The processing flow of the multi-source data acquisition module comprises the following steps: Various intelligent monitoring devices are accessed to obtain raw monitoring data of each sensor. Heterogeneous sensor outputs are converted into a unified standard tuple format containing time stamp, location coordinates, data source type, alarm content and confidence level through a mapping function.

3. The intelligent safety synergic management system of claim 1, wherein, The processing flow of the information fusion module comprises the following steps: Candidate alarm sets are filtered according to the time window tolerance and spatial error tolerance of the reference center point. A confidence sorting splicing method is used to generate composite alarm information suitable for human-machine reading. A semantic embedding and confidence weighted average method is used to convert text into vectors and perform fusion to generate composite alarm text suitable for input of a large language model.

4. The intelligent safety synergic management system of claim 1, wherein, The construction flow of the RAG knowledge base module comprises the following steps: Structured composite event descriptions are generated through a large language model. The composite event is vectorized and used as a retrieval index field. A complete knowledge base data structure containing composite alarms, composite events, locations, root causes, disposal suggestions and disposal results is constructed.

5. The intelligent safety synergic management system of claim 1, wherein, The retrieval flow of the intelligent retrieval module comprises the following steps: Metadata filtering is used to filter irrelevant data by limiting alarm location parameters. A dense vector construction method is used to search for the most similar historical events in the filtered subset based on vector similarity. A sparse vector is used for keyword retrieval, and the priority of the retrieval results is further adjusted through reordering. Dynamic splicing of structured prompt words containing role definition, task description, current alarm and prior knowledge is performed.

6. The intelligent safety synergic management system of claim 1, wherein, The analysis flow of the LLM decision module comprises the following steps: Dynamic prompt words containing current composite alarms, retrieved prior knowledge and task instructions are received. Intelligent analysis is performed based on multi-source fusion event descriptions and knowledge base semantic matching results. Structured root cause analysis containing event type, confidence and associated event evidence is outputted. Classification disposal suggestions for automatic execution and manual review are generated, and manual review requirements are forcibly added when the confidence is lower than a preset threshold.

7. The intelligent safety synergic management system of claim 1, wherein, The execution flow of the dual-path response module comprises the following steps: Intelligent decision suggestions are distributed to management and operation personnel and automation agent system in parallel; management and operation personnel receive instructions that need to be manually reviewed, responsible for manual judgment, decision support and key point review; The automation agent system analyzes executable instructions, converts natural language commands into device control instructions through a preset instruction mapping table, and is executed by a standardized machine.

8. The intelligent safety synergic management system of claim 1, wherein, The optimization process of the feedback optimization module includes: Collecting raw decisions, automatic execution records, manual review records, and final treatment result data for each event handling; Based on successful cases, strengthen the weight of related knowledge base entries, based on failed cases, mark the decision templates that need to be revised, and update the root cause and treatment suggestion library according to manual correction; Implementing prompt word template optimization, retrieval weight adjustment and instruction generation strategy improvement to achieve system continuous learning.

9. The intelligent safety collaborative management method based on multi-source perception and language large model, applied to the intelligent safety collaborative management system of any one of claims 1-8, characterized in that, Including: Accessing a variety of intelligent monitoring devices and converting the raw output of each sensor into a unified standard tuple format through a mapping function; According to the time window tolerance and spatial error tolerance, filter the candidate alarm set, use confidence sorting splicing and semantic embedding weighted average to generate composite alarm information; The composite alarm information is vectorized after being generated by a large language model to generate a composite event description, which is used as a retrieval index to build a structured knowledge base; A hybrid retrieval strategy is used to obtain related historical events from the knowledge base and dynamically build structured prompts; Based on the composite alarm and prior knowledge, output root cause analysis and classification of automatic execution and manual review of classification treatment suggestions; Distribute decision suggestions to management personnel and automation agent system in parallel to realize the collaborative execution of manual judgment review and automated device control; Collect treatment data and adjust the knowledge base weight and decision template based on the treatment effect to achieve system continuous optimization.

Images