A hierarchical alarm handling system and method based on visual language models

Abstract

The application discloses a hierarchical alarm handling system and method based on a visual language model, which comprises a target detection module of a video stream medium, which generates a safety event, extracts key frames and assembles them into an event package, generates output data and outputs; a VLM causal analysis module, which receives data output from the target detection module of the video stream medium, immediately performs data analysis, finally generates output data and outputs; a historical tracing module, which receives data output from the target detection module of the video stream medium and the VLM causal analysis module, and then outputs a risk cumulative evaluation description; an alarm level analysis module, which receives data output from the target detection module of the video stream medium, the VLM causal analysis module and the historical tracing module, and finally outputs data by an LLM model. It not only performs deep semantic understanding and historical context tracing on the causes of the event, but also accurately distinguishes the alarm into multiple levels, so as to realize a full-link intelligent alarm handling closed loop.

Description

Technical Field

[0001] This invention relates to the field of system monitoring technology, and in particular to a hierarchical alarm handling system and method based on a visual language model. Background Technology

[0002] With the rapid development of deep learning and computer vision technologies, intelligent video surveillance systems have been widely used in security scenarios such as campuses, transportation, and industry. Existing intelligent video alarm handling systems typically adopt a basic architecture of "target detection + alarm triggering": front-end cameras capture video streams, and target detection models (such as the YOLO series algorithms) deployed at the edge or on the server side perform real-time analysis of the video footage to identify security events such as fights, intrusions, and abnormal gatherings, and trigger corresponding alarms.

[0003] With the rise of multimodal large-scale model technology, some improved solutions in recent years have begun to introduce large models for secondary verification based on traditional object detection. For example, some solutions adopt a two-level architecture of "small model initial detection + multimodal large-scale model re-judgment," that is, after the object detection model completes the initial abnormal behavior identification, the detection results and images are input into multimodal large-scale models such as Qwen-VL for secondary verification. In addition, in terms of alarm classification, some solutions have begun to perform simple alarm classification.

[0004] However, these existing solutions still have the following drawbacks: (1) lack of in-depth semantic understanding of the causes of security incidents; (2) lack of retrospective analysis of the historical background and evolution of security incidents; (3) single basis for alarm classification, lacking multi-dimensional comprehensive judgment capabilities.

[0005] Therefore, it is necessary to improve existing technologies. Summary of the Invention

[0006] This invention aims to at least partially solve one of the problems existing in the existing related technologies. To this end, this invention proposes a hierarchical alarm handling system and method based on a visual language model, which can perform deep semantic understanding and historical context tracing of the cause of an event, thereby realizing a closed loop of intelligent alarm handling across the entire chain.

[0007] The above objective is achieved through the following technical solution: A hierarchical alarm handling system based on a visual language model includes: The target detection module of the video streaming media performs video decoding and target detection inference. When a target is detected, a security event is generated. The module extracts the N consecutive video segments before the trigger time from the video cache, extracts M keyframe images from the video segments and assembles them into an event packet. Finally, the module generates and outputs the output data of the target detection module of the video streaming media. The VLM causal analysis module receives data from the target detection module of the video streaming media, then combines the "role definition" rule and the "analysis instruction containing event information" rule to perform data analysis, and finally generates and outputs the output data of the VLM causal analysis module. The historical tracing module receives data from the target detection module and the VLM causal analysis module of the video streaming media, and then performs logical analysis. The data after logical analysis is output as a risk cumulative assessment description through the LLM model. The alarm level analysis module receives data from the target detection module, VLM causal analysis module, and historical tracing module of the video streaming media. It then organizes the received information and the weights of the relevant items and inputs it back into the "analysis instruction containing event information" rule. Finally, the LLM model outputs the data.

[0008] In some embodiments, the output data of the target detection module in the video streaming media includes: Security incident types include whether there are security incidents such as fighting, intrusion, unusual gatherings, or falls at the target location; Trigger timestamp, used to determine when a security event was triggered; Camera description, including the camera's location information and camera ID; The preceding video segment includes a continuous video segment N seconds before the triggering time; The keyframe image set includes M keyframe images extracted from the preceding video; Detailed detection results, including a list of targets detected in each keyframe; Event confidence level: The level of confidence that this security event will occur.

[0009] In some embodiments, the output data of the VLM causality analysis module includes: Event semantic description, including natural language description of the entire process of a security event; The direct cause, the immediate trigger that led to the safety incident; Environmental factors, which are external environmental conditions that may contribute to security incidents; Contextual factors, the context in which a security incident occurs; Triggering event chains are key steps in the evolution of security incidents. The number of targets involved in the detection, and the number of targets directly involved in security incidents; Severity, the initial severity assessed by VLM; Analyze the confidence level, which is the degree of confidence that VLM has in its own analytical results.

[0010] In some embodiments, the data received by the history tracing module includes: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The semantic description of the event comes from the VLM causal analysis module; The time range for tracing can be dynamically adjusted based on system feedback through system configuration.

[0011] In some embodiments, the data processing logic of the historical tracing module includes: After constructing multi-dimensional search criteria by combining security event type, location, and semantic description, the system queries the event database to obtain a complete list of events. Then, through time-series trend analysis and feature vector library retrieval, it integrates and generates a historical trace report.

[0012] In some embodiments, the output data of the history tracing module includes: Total historical occurrences: The total number of times this type of security incident has occurred at this location. 7-day frequency: the number of times sent within 7 days; 30-day frequency: the number of times it occurs within 30 days; Time of first occurrence; time of first occurrence of this type of security incident at this location. The most recent occurrence time, the location, and the most recent occurrence time of the type of security incident; Frequency change trends are determined based on historical data. Identify periodic patterns and peak occurrence times; List of similar events, Top-K similar historical events; Accumulated risk assessment and comprehensive judgment in a targeted description.

[0013] In some embodiments, the data received by the alarm level analysis module includes: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The confidence score for target detection events is derived from the target detection module in the video streaming media. The semantic description of the event comes from the VLM causal analysis module; Severity tendency, derived from the VLM causal analysis module; The confidence level of VLM analysis comes from the VLM causal analysis module; The historical 30-day frequency is derived from the historical tracing module; The frequency change trend is derived from the historical tracing module; The average severity of similar events is derived from the historical tracing module; The cumulative risk assessment is derived from the historical tracing module.

[0014] In some embodiments, the weights of the data received by the alarm level analysis module include: security event type 0.20, location of occurrence 0.15, target detection event confidence 0.12, event semantic description 0.12, severity tendency 0.08, VLM analysis confidence 0.08, historical 30-day frequency 0.05, frequency change trend 0.05; average severity of similar events 0.05, and cumulative risk assessment 0.10.

[0015] In some embodiments, the LLM model output data includes: Alarm level, which forms the final alarm level determination; The comprehensive risk score forms the basis for quantitative rating. The basis for the grading, and a brief explanation of the reasons for the grading determination; Recommendations for handling the situation, including a list of differentiated handling procedures for each level; The deadline for handling the situation is the response time limit set according to the level.

[0016] A hierarchical alarm handling method based on a visual language model, applicable to the aforementioned hierarchical alarm handling system based on a visual language model, includes: Step 1: The target detection module of the video streaming media performs video decoding and target detection inference. If a target is detected, a security event is generated. At the same time, the event packet is sent to the VLM causal analysis module and the security event is output to the history tracing module. Step 2: The VLM cause-effect analysis module receives data from the target detection module of the video streaming media, performs data analysis, and then outputs the analyzed data to the historical tracing module and the alarm level analysis module respectively. Step 3: The historical tracing module receives data from the target detection module and the VLM cause-effect analysis module of the video streaming media, performs data analysis, and then outputs the analysis data to the alarm level analysis module. Step 4: The alarm level analysis module receives data from the target detection module, VLM cause-effect analysis module, and historical tracing module of the video streaming media, performs data analysis, and finally generates and outputs alarm levels and handling suggestions by the LLM model.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: This invention proposes a hierarchical alarm handling system and method based on a visual language model, which not only performs deep semantic understanding and historical context tracing of the cause of the event, but also accurately classifies alarms into multiple levels, thereby realizing a closed-loop intelligent alarm handling system across the entire chain. Attached Figure Description

[0018] Figure 1 This is a block diagram of the hierarchical alarm handling system in the embodiment; Figure 2 This is a flowchart of the historical tracing module processing logic in the embodiment. Detailed Implementation

[0019] The following embodiments illustrate the present invention, but the present invention is not limited to these embodiments. Modifications to the specific embodiments of the present invention or equivalent substitutions for some technical features, without departing from the spirit of the present invention, should all be covered within the scope of the technical solutions claimed in the present invention.

[0020] This embodiment provides a hierarchical alarm handling system based on a visual language model, including: The target detection module of the video streaming media performs video decoding and target detection inference. When a target is detected, a security event is generated. The module extracts the N consecutive video segments before the trigger time from the video cache, extracts M keyframe images from the video segments and assembles them into an event packet. Finally, the module generates and outputs the output data of the target detection module of the video streaming media. The VLM (Visual Language Model) causal analysis module receives data from the target detection module of the video streaming media, then combines the "role definition" rule and the "analysis instruction containing event information" rule to perform data analysis, and finally generates and outputs the output data of the VLM causal analysis module. The historical tracing module receives data from the target detection module and the VLM causal analysis module of the video streaming media, and then performs logical analysis. The data after logical analysis is output as a risk cumulative assessment description through the LLM model. The alarm level analysis module receives data from the target detection module, VLM causal analysis module, and historical tracing module of the video streaming media. It then organizes the received information and the weights of the relevant items and inputs it back into the "analysis instruction containing event information" rule. Finally, the LLM model outputs the data.

[0021] The existing alarm linkage system based on a multimodal large model has the following technical problems: (1) Lack of deep semantic understanding of the causes of the event.

[0022] Existing alarm verification schemes based on multimodal large models are limited to a secondary confirmation of whether the current alarm is genuine—that is, verifying whether the event identified by the target detection model is a false alarm. Whether it's a two-tier architecture of "small model initial detection + large model secondary judgment," or a method of inputting detection results into a large model for feature comparison, the role of the large model is limited to verifying the authenticity of the footage at a single alarm moment, without performing any semantic-level causal reasoning analysis on the deeper causes of the event. Existing video anomaly detection methods are fundamentally still "reactive"—identifying anomalies only after they have occurred.

[0023] (2) Lack of historical background and evolutionary pattern analysis of the event.

[0024] Existing solutions typically rely solely on a single frame of image or a very short video clip within a short time window when assessing security incidents. Even when historical alarm data is incorporated as a weighting factor in alarm classification, it merely treats the number of historical alarms as a static value in the calculation, failing to truly trace the event's occurrence, spatiotemporal distribution patterns, and evolutionary trends within historical records. The system cannot answer questions such as "Have similar events frequently occurred at this location in the past?" or "Does this event show a gradually escalating trend?"

[0025] (3) The alarm rating is based on a single basis and lacks the ability to make comprehensive judgments from multiple dimensions.

[0026] Existing alarm classification schemes either employ composite weighted algorithms to quantitatively score based on structured data such as sensor device attributes, historical alarm counts, and environmental factors; or they rely on rule bases for simple classification and grading. Neither of these schemes organically combines "visual semantic understanding of the current event" with "analysis of the spatiotemporal distribution patterns of historical events," failing to achieve a multi-dimensional comprehensive assessment from "what the event is," "why it happened," to "whether there are historical precedents," and "what level of response should be given." This results in alarm level determination lacking sufficient semantic basis and historical reference.

[0027] The present invention proposes a hierarchical alarm handling system and method based on a visual language model. After a security event is triggered by the target detection module of the video streaming media, it does not only analyze the current alarm frame, but also calls the VLM causal analysis module to perform causal reasoning analysis on a continuous video sequence before the event to understand the underlying cause of the event. At the same time, it traces the occurrence and evolution of the event in the historical record through the historical tracing module, combining the "semantic understanding of the current event" with the "spatial and temporal distribution pattern of historical events". Through multi-dimensional comprehensive judgment, the alarm level analysis module accurately classifies the alarm into multiple levels such as fatal, severe, and general. This realizes a closed loop of intelligent alarm handling from "event discovery - cause analysis - historical tracing - intelligent classification - hierarchical handling", effectively overcoming the technical defects of existing technologies that can only perform authenticity verification or simple scoring and cannot deeply understand the cause and historical background of the event.

[0028] Furthermore, the output data of the object detection module in video streaming includes: Security incident types include whether there are security incidents such as fighting, intrusion, unusual gatherings, or falls at the target location; Trigger timestamp, used to determine when a security event was triggered; Camera description, including the camera's location information and camera ID; The preceding video segment includes a continuous video segment N seconds before the triggering time; The keyframe image set includes M keyframe images extracted from the preceding video; Detailed detection results, including a list of targets detected in each keyframe; Event confidence level: The level of confidence that this security event will occur.

[0029] The output data settings of the aforementioned target detection module in video streaming media provide an overall description and video transmission of security events, facilitating further analysis.

[0030] Furthermore, the output data of the VLM causal analysis module includes: Event semantic description, including natural language description of the entire process of a security event; The direct cause, the immediate trigger that led to the safety incident; Environmental factors, which are external environmental conditions that may contribute to security incidents; Contextual factors, the context in which a security incident occurs; Triggering event chains are key steps in the evolution of security incidents. The number of targets involved in the detection, and the number of targets directly involved in security incidents; Severity, the initial severity assessed by VLM; Analyze the confidence level, which is the degree of confidence that VLM has in its own analytical results.

[0031] The VLM causal analysis module receives data from the target detection module of the video streaming media, along with role definitions (systemprompt) and analysis instructions (userprompt) containing event information. It then outputs the VLM causal analysis module's output data.

[0032] Furthermore, the data received by the historical tracing module includes: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The semantic description of the event comes from the VLM causal analysis module; The time range for tracing can be dynamically adjusted based on system feedback through system configuration.

[0033] Furthermore, the data processing logic of the historical tracing module includes: The system constructs multi-dimensional search criteria (including spatial, event type, and semantic dimensions) by combining security event type, location, and semantic description. This is then queried through an event database (including but not limited to total occurrences, 7-day frequency, 30-day frequency, and first and last occurrence times). Historical records are then compiled and retrieved to obtain a complete list of events (including but not limited to time, description, handling results, and severity). Finally, time-series trend analysis (including but not limited to grouping by day / week, calculating linear regression slope to determine rising / falling / stabilizing events, and identifying high-incidence periods) and feature vector database retrieval (including but not limited to retrieving feature vectors from the database based on the current event semantics and returning Top-K similar historical events) are used to integrate and generate a historical trace report. Notably, this information is used to output a cumulative risk assessment description through an LLM model.

[0034] Furthermore, the output data of the historical tracing module includes: Total historical occurrences: The total number of times this type of security incident has occurred at this location. 7-day frequency: the number of times sent within 7 days; 30-day frequency: the number of times it occurs within 30 days; Time of first occurrence; time of first occurrence of this type of security incident at this location. The most recent occurrence time, the location, and the most recent occurrence time of the type of security incident; Frequency change trends are determined based on historical data. Identify periodic patterns and peak occurrence times; A list of similar events (including but not limited to time, description, handling, result, and alarm level), and a Top-K list of similar historical events; Accumulated risk assessment and comprehensive judgment in a targeted description.

[0035] The output data of the historical tracing module includes, but is not limited to, the above-mentioned content. This facilitates a deep semantic understanding of the causes of security incidents and the tracing of their historical context.

[0036] Furthermore, the alarm level analysis module receives the following data: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The confidence score for target detection events is derived from the target detection module in the video streaming media. The semantic description of the event comes from the VLM causal analysis module; Severity tendency, derived from the VLM causal analysis module; The confidence level of VLM analysis comes from the VLM causal analysis module; The historical 30-day frequency is derived from the historical tracing module; The frequency change trend is derived from the historical tracing module; The average severity of similar events is derived from the historical tracing module; The cumulative risk assessment is derived from the historical tracing module.

[0037] Then, the above information and the weights of the relevant items are sorted and input back into the "Analysis Instruction Containing Event Information" rule, and finally the data is output by the LLM model.

[0038] Furthermore, the weights of the data received by the alarm level analysis module include: security event type 0.20, location of occurrence 0.15, target detection event confidence 0.12, event semantic description 0.12, severity tendency 0.08, VLM analysis confidence 0.08, historical 30-day frequency 0.05, frequency change trend 0.05; average severity of similar events 0.05, and cumulative risk assessment 0.10.

[0039] In this embodiment, the weights of the data received by the alarm level analysis module are as described above. However, in practical applications, the weight ratios include, but are not limited to, the settings described above.

[0040] Furthermore, the LLM model output data includes: Alarm level, which forms the final alarm level determination; The comprehensive risk score forms the basis for quantitative rating. The basis for the grading, and a brief explanation of the reasons for the grading determination; Recommendations for handling the situation, including a list of differentiated handling procedures for each level; The deadline for handling the situation is the response time limit set according to the level.

[0041] This enables a closed-loop intelligent alarm handling process that covers the entire chain, from "event discovery - cause analysis - historical tracing - intelligent classification - graded handling".

[0042] A hierarchical alarm handling method based on a visual language model, applicable to the aforementioned hierarchical alarm handling system based on a visual language model, includes: Step 1: The target detection module of the video streaming media performs video decoding and target detection inference. If a target is detected, a security event is generated. At the same time, the event packet is sent to the VLM causal analysis module and the security event is output to the history tracing module. Step 2: The VLM cause-effect analysis module receives data from the target detection module of the video streaming media, performs data analysis, and then outputs the analyzed data to the historical tracing module and the alarm level analysis module respectively. Step 3: The historical tracing module receives data from the target detection module and the VLM cause-effect analysis module of the video streaming media, performs data analysis, and then outputs the analysis data to the alarm level analysis module. Step 4: The alarm level analysis module receives data from the target detection module, VLM cause-effect analysis module, and historical tracing module of the video streaming media, performs data analysis, and finally generates and outputs alarm levels and handling suggestions by the LLM model.

[0043] This embodiment presents a hierarchical alarm handling method based on a visual language model. The steps are simple, the content of each module is clearly analyzed, and through multi-dimensional comprehensive judgment, it effectively reduces the bias of single-dimensional judgment and improves the accuracy of alarm level identification and the scientific nature of decision-making.

[0044] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A hierarchical alarm handling system based on a visual language model, characterized in that, include: The target detection module of the video streaming media performs video decoding and target detection inference. When a target is detected, a security event is generated. The module extracts the N consecutive video segments before the trigger time from the video cache, extracts M keyframe images from the video segments and assembles them into an event packet. Finally, the module generates and outputs the output data of the target detection module of the video streaming media. The VLM causal analysis module receives data from the target detection module of the video streaming media, then combines the "role definition" rule and the "analysis instruction containing event information" rule to perform data analysis, and finally generates and outputs the output data of the VLM causal analysis module. The historical tracing module receives data from the target detection module and the VLM causal analysis module of the video streaming media, and then performs logical analysis. The data after logical analysis is output as a risk cumulative assessment description through the LLM model. The alarm level analysis module receives data from the target detection module, VLM causal analysis module, and historical tracing module of the video streaming media. It then organizes the received information and the weights of the relevant items and inputs it back into the "analysis instruction containing event information" rule. Finally, the LLM model outputs the data.

2. The hierarchical alarm handling system based on a visual language model according to claim 1, characterized in that, The output data of the object detection module in video streaming includes: Security incident types include whether there are security incidents such as fighting, intrusion, unusual gatherings, or falls at the target location; Trigger timestamp, used to determine when a security event was triggered; Camera description, including the camera's location information and camera ID; The preceding video segment includes a continuous video segment N seconds before the triggering time; The keyframe image set includes M keyframe images extracted from the preceding video; Detailed detection results, including a list of targets detected in each keyframe; Event confidence level: The level of confidence that this security event will occur.

3. The hierarchical alarm handling system based on a visual language model according to claim 1, characterized in that, The output data of the VLM causality analysis module includes: Event semantic description, including natural language description of the entire process of a security event; The direct cause, the immediate trigger that led to the safety incident; Environmental factors, which are external environmental conditions that may contribute to security incidents; Contextual factors, the context in which a security incident occurs; Triggering event chains are key steps in the evolution of security incidents. The number of targets involved in the detection, and the number of targets directly involved in security incidents; Severity, the initial severity assessed by VLM; Analyze the confidence level, which is the degree of confidence that VLM has in its own analytical results.

4. The hierarchical alarm handling system based on a visual language model according to claim 1, characterized in that, The data received by the historical tracing module includes: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The semantic description of the event comes from the VLM causal analysis module; The time range for tracing can be dynamically adjusted based on system feedback through system configuration.

5. A hierarchical alarm handling system based on a visual language model according to claim 4, characterized in that, The data processing logic of the historical tracing module includes: After constructing multi-dimensional search criteria by combining security event type, location, and semantic description, the system queries the event database to obtain a complete list of events. Then, through time-series trend analysis and feature vector library retrieval, it integrates and generates a historical trace report.

6. A hierarchical alarm handling system based on a visual language model according to claim 5, characterized in that, The output data of the historical tracing module includes: Total historical occurrences: The total number of times this type of security incident has occurred at this location. 7-day frequency: the number of times sent within 7 days; 30-day frequency: the number of times it occurs within 30 days; Time of first occurrence; time of first occurrence of this type of security incident at this location. The most recent occurrence time, the location, and the most recent occurrence time of the type of security incident; Frequency change trends are determined based on historical data. Identify periodic patterns and peak occurrence times; List of similar events, Top-K similar historical events; Accumulated risk assessment and comprehensive judgment in a targeted description.

7. A hierarchical alarm handling system based on a visual language model according to claim 1, characterized in that, The data received by the alarm level analysis module includes: Security incident types originate from the target detection module of the video streaming media; The location of the event is derived from the target detection module of the video streaming media; The confidence score for target detection events is derived from the target detection module in the video streaming media. The semantic description of the event comes from the VLM causal analysis module; Severity tendency, derived from the VLM causal analysis module; The confidence level of VLM analysis comes from the VLM causal analysis module; The historical 30-day frequency is derived from the historical tracing module; The frequency change trend is derived from the historical tracing module; The average severity of similar events is derived from the historical tracing module; The cumulative risk assessment is derived from the historical tracing module.

8. A hierarchical alarm handling system based on a visual language model according to claim 7, characterized in that, The weights of the data received by the alarm level analysis module include: security event type 0.20, location of occurrence 0.15, target detection event confidence 0.12, event semantic description 0.12, severity tendency 0.08, VLM analysis confidence 0.08, historical 30-day frequency 0.05, frequency change trend 0.05; average severity of similar events 0.05, and cumulative risk assessment 0.

10.

9. A hierarchical alarm handling system based on a visual language model according to claim 8, characterized in that, The LLM model output data includes: Alarm level, which forms the final alarm level determination; The comprehensive risk score forms the basis for quantitative rating. The basis for the grading, and a brief explanation of the reasons for the grading determination; Recommendations for handling the situation, including a list of differentiated handling procedures for each level; The deadline for handling the situation is the response time limit set according to the level.

10. A hierarchical alarm handling method based on a visual language model, applicable to the hierarchical alarm handling system based on a visual language model as described in any one of claims 1-9, characterized in that, include: Step 1: The target detection module of the video streaming media performs video decoding and target detection inference. If a target is detected, a security event is generated. At the same time, the event packet is sent to the VLM causal analysis module and the security event is output to the history tracing module. Step 2: The VLM cause-effect analysis module receives data from the target detection module of the video streaming media, performs data analysis, and then outputs the analyzed data to the historical tracing module and the alarm level analysis module respectively. Step 3: The historical tracing module receives data output from the target detection module and the VLM cause-effect analysis module of the video streaming media, performs data analysis, and then outputs the analysis data to the alarm level analysis module. Step 4: The alarm level analysis module receives data from the target detection module, VLM cause-effect analysis module, and historical tracing module of the video streaming media, performs data analysis, and finally generates and outputs alarm levels and handling suggestions by the LLM model.

Images