Intelligent identification and pre-emptive disposal system for abnormal behavior in key places

Abstract

The application discloses a kind of key place-oriented abnormal behavior intelligent identification and pre-emptive disposal system, preliminary abnormal event sequence is generated by collecting multi-modal environment data, it is fused with the semantics of preset scene knowledge graph to form composite abnormal event description information, then based on large language model inference generates dynamic disposal preplan, is decomposed into collaborative control instruction set by central dispatch intelligent agent, drives video analysis intelligent agent, broadcast dissuasion intelligent agent, anxi linkage intelligent agent executes collaborative disposal operation, and generates situation evolution information in shared event canvas through environment feedback data, realizes the dynamic optimization adjustment of disposal strategy.The system realizes the full-process intelligentization of abnormal event from identification to disposal, improves the semantic understanding ability of system to complex scene and the efficiency of multi-agent collaborative response, enhances the pertinence and adaptive adjustment ability of disposal preplan.

Description

Technical Field

[0001] This invention relates to the field of smart security technology, specifically to an intelligent identification and proactive handling system for abnormal behavior in key locations. Background Technology

[0002] Security management of key locations is a crucial component of smart city construction, involving real-time monitoring and anomaly response to multi-dimensional information such as personnel behavior and environmental conditions. Currently, intelligent recognition systems based on technologies like video analytics and IoT sensing are gradually being introduced into this field, capable of detecting specific behavioral patterns and triggering alarms. Some systems are also attempting to integrate security and fire protection resources to build a unified management platform, enabling the visualization and coordinated control of multi-source sensing data.

[0003] However, existing technical solutions still face certain challenges in the recognition and handling of abnormal events: on the one hand, the system's semantic understanding of multi-source information in complex scenarios is limited, making it difficult to grasp the overall development of abnormal events; on the other hand, the lack of dynamic adjustment and collaborative optimization mechanisms in the generation and execution of contingency plans leads to insufficient response efficiency and adaptability when facing unexpected or complex abnormal events. How to improve the system's overall recognition of abnormal events and achieve dynamic generation and multi-unit collaborative execution of handling strategies has become a direction that needs further exploration in the field of intelligent security for key locations. Summary of the Invention

[0004] In view of the above problems, the present invention provides an intelligent identification and preemptive handling system for abnormal behavior in key locations. By semantically fusing preliminary abnormal events with scene knowledge graphs and generating dynamic handling plans based on a large language model, the system achieves multi-agent collaborative control and dynamic adjustment, solving the problems of insufficient cognition of complex abnormal events and lack of adaptability of handling strategies in existing systems.

[0005] To achieve the above objectives, this application provides an intelligent identification and proactive handling system for abnormal behavior in key locations, comprising: Acquire multimodal environmental data from sensing devices deployed in key locations; Perform abnormal behavior identification and analysis on multimodal environmental data to generate a preliminary abnormal event sequence; The initial sequence of abnormal events is semantically fused with a pre-defined scene knowledge graph to generate composite description information of abnormal events. The description information of the complex abnormal event is input into the dynamic contingency plan generation engine, and reasoning and calculation are performed based on the large language model to output a dynamic contingency plan containing multiple handling actions. According to the dynamic response plan, the central dispatching intelligent agent decomposes the tasks and generates a set of collaborative control instructions for different functional intelligent agents, including video analysis intelligent agents, broadcast guidance intelligent agents, and security and fire linkage intelligent agents. It executes a set of collaborative control instructions to drive video analysis agents, broadcast guidance agents, and security and fire linkage agents to perform collaborative handling operations, and collects environmental feedback data in real time. Environmental feedback data is input into a shared event canvas for multi-source information fusion to generate situational evolution information; Based on situational evolution information, the collaborative control instruction set is dynamically adjusted through decision control algorithms until the abnormal event is handled.

[0006] In some embodiments, abnormal behavior identification and analysis are performed on multimodal environmental data to generate a preliminary abnormal event sequence, including: Video analysis algorithms are used to parse video streams in multimodal environmental data and extract dynamic features including at least human posture, movement trajectory and object state. The audio stream in the multimodal environmental data is parsed using audio processing algorithms to extract audio features, including at least voiceprint features, sound source location, and anomalous phonemes. Dynamic features and audio features are input into a multimodal fusion detection model for joint analysis to identify initial anomalous events that include at least the behavior type, location of occurrence, and severity level. Cluster analysis is performed on continuously identified initial anomalous events based on time windows to generate a preliminary sequence of anomalous events with spatiotemporal correlation.

[0007] In some embodiments, a preliminary sequence of abnormal events is semantically fused with a preset scenario knowledge graph to generate composite abnormal event description information, including: Based on the spatiotemporal attributes of each anomalous event in the preliminary anomalous event sequence, a graph neural network algorithm is used to mine the associated paths in the scene knowledge graph to identify combinations of anomalous events with causal relationships. The identified abnormal event combinations are semantically enhanced, and the event elements are associated with the location structure entities, equipment entities and personnel role entities in the knowledge graph through the entity linking algorithm to obtain the association mapping results; Based on the correlation mapping results, an event evolution reasoning model is used to predict the development trend of abnormal event combinations, generating composite abnormal event description information that includes event correlation, evolution path, and scope of influence.

[0008] In some embodiments, based on the spatiotemporal attributes of each anomalous event in the preliminary anomalous event sequence, a graph neural network algorithm is used to mine association paths in the scene knowledge graph to identify combinations of anomalous events with causal relationships, including: Extract the spatial coordinates and timestamps of each anomalous event from the preliminary anomalous event sequence to construct a spatiotemporal feature vector; The spatiotemporal feature vectors are input into the graph neural network algorithm, and multi-hop neighbor node traversal is performed in the preset scene knowledge graph to calculate the correlation strength between nodes of abnormal events. Based on the correlation strength, anomalous event combinations that satisfy spatiotemporal proximity and logical correlation are identified through causal reasoning algorithms; Semantic enhancement processing is performed on the identified combinations of abnormal events. Event elements are then linked to location structure entities, equipment entities, and personnel role entities in the knowledge graph through entity links, resulting in the following association mapping results: Analyze the event elements in the abnormal event combination and extract the type and function attributes of the event elements; The entity linking algorithm is used to perform similarity matching between the type and functional attributes of event elements and the location structure entities, equipment entities, and personnel role entities in the preset scene knowledge graph. Establish semantic association mapping relationships between event elements and knowledge graph entities to form an enhanced semantic network, which is the association mapping result; Based on the correlation mapping results, an event evolution inference model is used to predict the development trend of abnormal event combinations, generating composite abnormal event description information that includes event correlations, evolution paths, and impact scope, including: An enhanced semantic network input event evolution reasoning model is used to simulate the diffusion path of abnormal events in the site structure based on a spatiotemporal propagation algorithm; By using an impact range calculation model, combined with the functional zoning of the venue and the population density, the impact range of a combination of abnormal events can be predicted. By combining the diffusion path and the scope of impact, a composite description of the abnormal event is generated, which includes the event's correlation, evolution path, and scope of impact.

[0009] In some embodiments, composite abnormal event description information is input into a dynamic contingency plan generation engine, inference calculations are performed based on a large language model, and a dynamic contingency plan containing multiple handling actions is output, including: The descriptive information of complex abnormal events is structured and parsed to extract core semantic elements, which include at least the event type, key location, involved objects, and current situation level. The core semantic elements are matched and retrieved with the contingency plan knowledge base to obtain the matching results. The contingency plan knowledge base stores historical handling cases, standard operating procedures and expert experience rules, and an initial contingency plan framework is generated based on the matching results. The initial contingency plan framework and real-time environmental status data are input into the large language model. The real-time environmental status data includes at least the dynamic distribution of personnel, the availability of equipment, and the passage capacity of channels. By using a large language model to perform multi-round inference calculations based on reinforcement learning strategies, the expected effects of different handling strategies are simulated, and the initial contingency plan framework is optimized and reconstructed to generate a dynamic contingency plan that includes resource scheduling schemes, action sequence logic, and expected control objectives. The dynamic response plan is logically consistent, including checking for timing conflicts of response actions, resource allocation conflicts, and compliance with site management rules, and outputting a dynamic response plan that has passed the verification.

[0010] In some embodiments, according to a dynamic response plan, a central scheduling agent decomposes tasks and generates a set of collaborative control instructions for different functional agents, including: Semantic parsing is performed on the dynamic response plan to extract multiple atomic response actions and their corresponding execution constraints contained in the dynamic response plan; Based on the task planning algorithm, atomic processing actions are sorted and grouped according to spatiotemporal logical relationships to form a task chain with sequential dependencies; Based on the capability profiles of each functional intelligent agent, the actions in the task chain are assigned to the corresponding video analysis intelligent agent, broadcast guidance intelligent agent, and security and fire linkage intelligent agent. Add execution parameters and triggering conditions to each assigned task to generate a machine-readable set of collaborative control instructions. The execution parameters include at least the action target, resource quota and timeout setting, and the triggering conditions include the event status threshold and the completion signal of the previous task.

[0011] In some embodiments, a collaborative control instruction set is executed to drive the video analysis agent, the broadcast guidance agent, and the security and fire linkage agent to perform collaborative handling operations, and to collect environmental feedback data in real time, including: Based on the video control instructions received from the collaborative control instruction set, the video analysis agent adjusts the monitoring parameters and viewing angle of the video acquisition equipment, continuously tracks and analyzes the target area, and generates video analysis data that includes the target's movement trajectory, changes in personnel density, and the continuous state of abnormal behavior. The broadcast guidance intelligent agent dynamically synthesizes scene-adaptive voice alarm content through a text generation algorithm based on the guidance strategy parameters in the collaborative control instruction set, and controls the broadcasting equipment to broadcast in the designated area. At the same time, it collects the on-site sound pressure characteristics and personnel flow response data after the voice broadcast. Based on the equipment operation logic in the collaborative control instruction set, the safety and fire protection linkage intelligent agent automatically triggers the pre-activation state of fire protection facilities, adjusts the access permissions of emergency passages, and generates resource scheduling plans, while collecting equipment execution status signals and environmental safety parameter change data. Video analytics data, personnel flow response data, and environmental safety parameter change data are used as environmental feedback data. After timestamp alignment and data fusion, the data is output to the shared event canvas.

[0012] In some embodiments, environmental feedback data is input to a shared event canvas for multi-source information fusion to generate situational evolution information, including: Spatiotemporal alignment processing is performed on the environmental feedback data input to the shared event canvas to establish spatiotemporal correlation between video analysis data, personnel flow response data, and environmental safety parameter change data; By using a multi-source information fusion algorithm to perform feature fusion on the spatiotemporally aligned data, fused features are obtained that include at least the trend of personnel aggregation, the spread range of abnormal behavior, and the equipment response efficiency. Based on the fusion characteristics, a situational simulation model is used to predict the evolution trend of abnormal events, generating situational evolution information that includes situational level, key impact areas and potential derivative risks. Furthermore, the situational evolution information is correlated and mapped with the description information of complex abnormal events to update the global situational awareness data in the shared event canvas.

[0013] In some embodiments, based on fusion features, a situational extrapolation model is used to predict the evolution trend of anomalous events, generating situational evolution information including situational level, key impact areas, and potential derivative risks, including: The fused features are input into a pre-trained spatiotemporal graph neural network model, and the propagation probability of abnormal events in the spatial topology of the venue is calculated through a node propagation algorithm. Based on propagation probability, the Monte Carlo tree search algorithm is used to simulate the evolution path of anomalous events at multiple time steps in the future, generating multiple potential evolution scenarios with different occurrence probabilities. For each potential evolution scenario, an impact assessment is conducted, and the corresponding situation level, key impact area range, and potential derivative risk type are calculated using a risk quantification model; By integrating the simulation results of all potential evolution scenarios, an evidence theory algorithm is used for fusion decision-making to generate situation evolution information including situation level prediction with confidence intervals, probability distribution of key impact areas, and derivative risk spectrum.

[0014] In some embodiments, based on situational evolution information, the cooperative control instruction set is dynamically adjusted through a decision control algorithm until the handling of abnormal events is completed, including: The situation evolution information is quantitatively analyzed to extract the trend of situation level change, the diffusion speed of key affected areas, and the probability of potential derivative risks. The analyzed situation quantification indicators are compared with the preset handling target thresholds, and a handling strategy adjustment vector is generated through a fuzzy decision algorithm; Based on the adjustment vector of the handling strategy, the parameters of the currently executing collaborative control instruction set are optimized. The parameter optimization includes at least adjusting the monitoring focus area of ​​the video analysis agent, updating the alarm content priority of the broadcast guidance agent, and replanning the resource scheduling path of the security and fire linkage agent. Drive each functional agent to perform adjusted disposal operations based on the optimized collaborative control instruction set, and continue to collect a new round of environmental feedback data; By continuously monitoring the situation level indicators in the situation evolution information through the convergence judgment algorithm, when all situation level indicators are continuously lower than the preset threshold and reach a stable duration, the abnormal event handling is determined to be completed.

[0015] Unlike existing technologies, the above-mentioned technical solution provides an intelligent identification and proactive handling system for abnormal behavior in key locations. It generates a preliminary abnormal event sequence by collecting multimodal environmental data, then semantically fuses this sequence with a pre-defined scenario knowledge graph to form a composite abnormal event description. Subsequently, it generates a dynamic handling plan based on a large language model, which is then decomposed into a collaborative control instruction set by a central dispatching agent. This command drives video analysis agents, broadcast guidance agents, and security and fire linkage agents to execute collaborative handling operations. Furthermore, environmental feedback data generates situational evolution information in a shared event canvas, enabling dynamic optimization and adjustment of the handling strategy. This system achieves full-process intelligent management of abnormal events from identification to handling, improving the system's semantic understanding of complex scenarios and the efficiency of multi-agent collaborative response, and enhancing the pertinence and adaptive adjustment capabilities of the handling plan.

[0016] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0017] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0018] In the accompanying drawings of the instruction manual: Figure 1 This is a schematic diagram illustrating steps S101 to S108 of the intelligent identification and pre-emptive handling system for abnormal behavior described in a specific implementation. Figure 2The diagram illustrates steps S201 to S204 of the intelligent identification and pre-emptive handling system for abnormal behavior as described in the specific implementation. Detailed Implementation

[0019] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0020] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0021] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0022] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0023] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0024] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0025] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0026] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0027] The processor described in the embodiments of this application can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0028] The computer program involved in the embodiments can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical, or chemical structures that can achieve the same or equivalent functions as the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium, or distributed and stored in multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device, or can be connected to the device involved in the embodiments as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0029] Please see Figure 1 This embodiment provides an intelligent identification and proactive handling system for abnormal behavior in key locations, including: S101. Acquire multimodal environmental data from sensing devices deployed in key locations; S102. Perform abnormal behavior identification and analysis on multimodal environmental data to generate a preliminary abnormal event sequence; S103. Semantically fuse the preliminary abnormal event sequence with the preset scenario knowledge graph to generate composite abnormal event description information; S104. Input the description information of the complex abnormal event into the dynamic contingency plan generation engine, perform reasoning calculation based on the large language model, and output a dynamic contingency plan containing multiple handling actions. S105. According to the dynamic response plan, the central dispatching intelligent agent decomposes the tasks and generates a set of collaborative control instructions for different functional intelligent agents, including video analysis intelligent agents, broadcast guidance intelligent agents, and security and fire linkage intelligent agents. S106. Execute the collaborative control instruction set to drive the video analysis intelligent agent, the broadcast guidance intelligent agent, and the security and fire linkage intelligent agent to perform collaborative handling operations, and collect environmental feedback data in real time. S107. Input environmental feedback data into the shared event canvas to perform multi-source information fusion and generate situational evolution information; S108. Based on situational evolution information, the collaborative control instruction set is dynamically adjusted through decision control algorithms until the abnormal event is handled.

[0030] In step S101, multimodal environmental data refers to a heterogeneous data set collected by various sensing devices deployed in key locations, including visual data acquired by video surveillance equipment, acoustic data acquired by audio acquisition equipment, and physical parameter data acquired by environmental sensors. These sensing devices are distributed according to the functional zoning of the locations, forming a three-dimensional sensing network covering key areas. Through multi-source data acquisition and standardized processing, a comprehensive environmental perception foundation is provided for subsequent abnormal behavior identification. This step achieves a comprehensive digital mapping of the environmental status of key locations by constructing a unified access mechanism for multi-source heterogeneous data.

[0031] In step S102, the abnormal behavior identification and analysis employs multimodal fusion detection technology. By extracting features and performing correlation analysis on heterogeneous data such as video and audio streams, it identifies behavioral patterns with abnormal characteristics. The preliminary abnormal event sequence is a structured event description formed by spatiotemporal clustering of continuously detected abnormal behaviors. Each event includes key attributes such as behavior type, location, timestamp, and confidence level. This step enhances the reliability of single-modal detection and reduces the false alarm rate through multimodal data complementarity. This step realizes the transformation from raw perceptual data to structured event descriptions, providing standardized input for subsequent semantic understanding.

[0032] In step S103, the preset scene knowledge graph is a pre-constructed semantic network containing elements such as location spatial structure, equipment layout, personnel roles, and behavioral rules. The semantic fusion process uses a graph neural network algorithm to map event elements in the initial abnormal event sequence to entity nodes in the knowledge graph, identifying semantic associations and causal relationships between events. The composite abnormal event description information is an enhanced event description generated on this basis, containing the association relationships, evolution trends, and potential impact range of the event combination. This step enhances the system's semantic understanding ability of complex abnormal scenarios by introducing domain knowledge.

[0033] In step S104, the dynamic contingency plan generation engine is an intelligent decision-making module built on a large language model. It performs in-depth analysis of the descriptive information of complex abnormal events using natural language understanding and reasoning techniques. Supported by a contingency plan knowledge base, the large language model combines real-time environmental state data to perform multiple rounds of reasoning, generating handling plans for specific abnormal scenarios. The dynamic contingency plan includes multiple handling actions ordered by priority, with each action specifying its execution content, required resources, and expected effects. This step utilizes the semantic understanding and reasoning capabilities of the large language model to achieve intelligent generation and optimization of the contingency plan.

[0034] In step S105, the central dispatching agent acts as the control hub responsible for coordinating the collaborative work of multiple agents. It decomposes the dynamic response plan into tasks, transforming complex response schemes into specific execution instructions. The collaborative control instruction set is a set of machine-readable instructions generated according to functional divisions, with control parameters and execution logic defined for agents with different functions such as video analysis, broadcast guidance, and security and fire prevention linkage. This step ensures the efficient execution of the response plan through task allocation and coordination mechanisms among agents.

[0035] In step S106, each functional agent executes corresponding handling operations based on the received set of collaborative control instructions. The video analysis agent adjusts the monitoring strategy for target tracking, the broadcast guidance agent generates and broadcasts voice alarms, and the security and fire linkage agent controls security and fire protection equipment for coordinated response. Environmental feedback data consists of various sensor data collected in real time during the handling process, used to evaluate the handling effect and changes in environmental conditions. This step forms a complete handling closed loop through the collaborative execution of multiple agents.

[0036] In step S107, the shared event canvas is a core component of multi-source information fusion and visualization. By performing spatiotemporal alignment and feature extraction on environmental feedback data, it generates evolutionary information reflecting the development trend of abnormal events. This evolutionary information includes key indicators such as the spread trend, impact range, and risk level of the abnormal events, providing a basis for subsequent decision-making adjustments. This step improves the system's accuracy in perceiving the development trend of abnormal events through multi-source data fusion.

[0037] In step S108, the decision control algorithm dynamically optimizes and adjusts the set of cooperative control instructions being executed based on situational evolution information. By continuously monitoring the handling effect and environmental changes, the system can adaptively adjust the control parameters and execution strategies of each agent until the abnormal event is effectively handled. This step ensures the adaptability and effectiveness of the handling process through a closed-loop feedback control mechanism.

[0038] This embodiment constructs a complete technology chain from environmental perception to intelligent response, enabling rapid identification and proactive handling of abnormal behaviors in key locations. The system employs multimodal data fusion technology to improve identification accuracy, enhances semantic understanding through knowledge graphs, utilizes large language models to generate intelligent contingency plans, leverages multi-agent collaborative execution of response plans, and dynamically optimizes and adjusts based on real-time feedback. This embodiment improves the timeliness, accuracy, and adaptability of abnormal event handling through the comprehensive application of intelligent technologies, providing new technical support for the safety management of key locations.

[0039] Please see Figure 2 In some embodiments, abnormal behavior identification and analysis are performed on multimodal environmental data to generate a preliminary abnormal event sequence, including: S201. Analyze the video stream in the multimodal environment data using video analysis algorithms to extract dynamic features including at least human posture, movement trajectory and object state; S202. The audio stream in the multimodal environment data is parsed using an audio processing algorithm to extract audio features including at least voiceprint features, sound source location, and abnormal phonemes. S203. Input the dynamic features and audio features into the multimodal fusion detection model for joint analysis to identify initial abnormal events that include at least the behavior type, location of occurrence, and severity level; S204. Based on the time window, perform cluster analysis on the continuously identified initial abnormal events to generate a preliminary abnormal event sequence with spatiotemporal correlation.

[0040] In step S201, the video analysis algorithm employs deep learning-based target detection and pose estimation techniques, extracting dynamic features from the video stream through a convolutional neural network. Human pose features are obtained through a joint detection model, representing the relative positions and motion states of various parts of the human body; movement trajectory features are calculated using a multi-target tracking algorithm, recording the positional changes of targets in consecutive frames; and object state features are analyzed using a target recognition model to examine the position and shape changes of key objects in the scene. These dynamic features collectively constitute a quantitative description of behavioral patterns in the video scene, providing a visual analytical foundation for abnormal behavior recognition.

[0041] In step S202, the audio processing algorithm employs acoustic signal analysis and pattern recognition techniques to extract features from the audio stream. Voiceprint features are obtained through Mel-frequency cepstral coefficient analysis, used to distinguish the identity characteristics of different sound sources; the sound source location is calculated using microphone array delay estimation and beamforming techniques to determine the spatial orientation of the sound source; abnormal phonemes are identified through a pre-trained audio classification model, detecting abnormal sound patterns such as screams and impacts. These audio features complement visual features, enhancing the system's perception capabilities in complex environments.

[0042] In step S203, the multimodal fusion detection model employs an attention-based neural network architecture to fuse dynamic and audio features across modalities. The model learns the correlations between different modal features through a combination of feature-level and decision-level fusion, comprehensively judging the occurrence of abnormal behavior. The initial abnormal event identification process includes behavior classification, location localization, and severity assessment. Behavior type is classified based on a predefined behavior dictionary, the location is determined by the spatial consistency of multimodal features, and the severity level is comprehensively assessed based on the degree of abnormality and duration of the behavior features.

[0043] In step S204, the time window clustering analysis employs a density-based spatiotemporal clustering algorithm to group the continuously identified initial anomalous events. The algorithm considers the temporal proximity and spatial correlation of events, aggregating anomalous events that are spatially close within a certain time range into a semantically complete sequence of anomalous events. The clustering process also considers the continuity and evolution of event types to ensure that the generated preliminary sequence of anomalous events accurately reflects the development process of anomalous behavior.

[0044] This embodiment achieves accurate identification and structured description of abnormal behavior through multimodal feature extraction and fusion analysis. The complementarity of video and audio features enhances the system's robustness in complex environments, while spatiotemporal clustering analysis ensures the integrity and continuity of abnormal event sequences. This embodiment improves the accuracy and reliability of abnormal behavior identification through multi-source information fusion, providing high-quality input data for subsequent semantic understanding and intelligent processing.

[0045] In some embodiments, a preliminary sequence of abnormal events is semantically fused with a preset scenario knowledge graph to generate composite abnormal event description information, including: Based on the spatiotemporal attributes of each anomalous event in the preliminary anomalous event sequence, a graph neural network algorithm is used to mine the associated paths in the scene knowledge graph to identify combinations of anomalous events with causal relationships. The identified abnormal event combinations are semantically enhanced, and the event elements are associated with the location structure entities, equipment entities and personnel role entities in the knowledge graph through the entity linking algorithm to obtain the association mapping results; Based on the correlation mapping results, an event evolution reasoning model is used to predict the development trend of abnormal event combinations, generating composite abnormal event description information that includes event correlation, evolution path, and scope of influence.

[0046] In this embodiment, the graph neural network algorithm performs association analysis between anomalous events and knowledge graph nodes through a spatiotemporal attention mechanism. Specifically, the algorithm first encodes the spatiotemporal coordinates of the anomalous event into a feature vector, and then calculates the association weights between the feature vector and location / device nodes in the knowledge graph through a graph attention layer. This attention-weighted association mining effectively identifies combinations of anomalous events that are spatiotemporally adjacent and semantically related, overcoming the limitations of traditional fixed-threshold matching.

[0047] The entity linking algorithm employs a multi-feature fusion similarity calculation method, comprehensively considering the textual description similarity, spatiotemporal co-occurrence probability, and functional relevance of event elements. The algorithm maps event elements and knowledge graph entities to the same vector space through a pre-trained semantic embedding model, and then performs joint similarity calculation by combining the location information of the event with the spatial distribution of the entities. This multi-dimensional similarity evaluation ensures the accuracy of entity linking, particularly in resolving ambiguities when dealing with entities with the same name or similar entities in complex environments.

[0048] The event evolution reasoning model is built upon spatiotemporal propagation dynamics. By analyzing the functional dependencies and spatial connectivity between entities in a knowledge graph, it simulates the propagation path of anomalous events along physical spaces and logical connections. The model employs a random walk algorithm to explore multiple paths on an enhanced semantic network, combining this with the distribution of personnel density and equipment operating status to predict the potential impact range and evolution trend of anomalous events. This graph-based evolutionary reasoning effectively captures the chain reactions of anomalous events in complex environments.

[0049] This embodiment significantly enhances the depth and breadth of semantic understanding of abnormal events by introducing a spatiotemporal attention mechanism, multi-feature fusion entity linking, and graph-based evolutionary reasoning. Compared with traditional rule matching methods, this embodiment can automatically discover potential causal relationships and accurately predict event development trends, providing a richer semantic context and a more accurate situation assessment basis for subsequent dynamic contingency plan generation.

[0050] In some embodiments, based on the spatiotemporal attributes of each anomalous event in the preliminary anomalous event sequence, a graph neural network algorithm is used to mine association paths in the scene knowledge graph to identify combinations of anomalous events with causal relationships, including: Extract the spatial coordinates and timestamps of each anomalous event from the preliminary anomalous event sequence to construct a spatiotemporal feature vector; The spatiotemporal feature vectors are input into the graph neural network algorithm, and multi-hop neighbor node traversal is performed in the preset scene knowledge graph to calculate the correlation strength between nodes of abnormal events. Based on the correlation strength, anomalous event combinations that satisfy spatiotemporal proximity and logical correlation are identified through causal reasoning algorithms; Semantic enhancement processing is performed on the identified combinations of abnormal events. Event elements are then linked to location structure entities, equipment entities, and personnel role entities in the knowledge graph through entity links, resulting in the following association mapping results: Analyze the event elements in the abnormal event combination and extract the type and function attributes of the event elements; The entity linking algorithm is used to perform similarity matching between the type and functional attributes of event elements and the location structure entities, equipment entities, and personnel role entities in the preset scene knowledge graph. Establish semantic association mapping relationships between event elements and knowledge graph entities to form an enhanced semantic network, which is the association mapping result; Based on the correlation mapping results, an event evolution inference model is used to predict the development trend of abnormal event combinations, generating composite abnormal event description information that includes event correlations, evolution paths, and impact scope, including: An enhanced semantic network input event evolution reasoning model is used to simulate the diffusion path of abnormal events in the site structure based on a spatiotemporal propagation algorithm; By using an impact range calculation model, combined with the functional zoning of the venue and the population density, the impact range of a combination of abnormal events can be predicted. By combining the diffusion path and the scope of impact, a composite description of the abnormal event is generated, which includes the event's correlation, evolution path, and scope of impact.

[0051] In this embodiment, the spatiotemporal feature vector is constructed using a multi-scale encoding method, which maps spatial coordinates to spatial distribution features through a Gaussian kernel function, while timestamps are converted into periodic temporal features. This encoding method can simultaneously capture the absolute location information and relative spatiotemporal relationship of abnormal events, providing rich feature representations for subsequent correlation analysis.

[0052] The multi-hop neighbor node traversal in graph neural network algorithms employs a weighted message passing mechanism, determining the association strength by calculating semantic similarity and spatiotemporal distance between nodes. Specifically, the algorithm performs a finite-step random walk on the knowledge graph, collecting node features along the path, and then aggregates these features through an attention mechanism to calculate the association strength score. This multi-hop traversal mechanism can discover potential associations between non-directly connected nodes, effectively identifying causal chains hidden in complex network structures.

[0053] The causal reasoning algorithm combines temporal logic and statistical correlation analysis, using Granger causality tests and transfer entropy calculations to verify the causal direction between event combinations. The algorithm also considers spatiotemporal proximity constraints, ensuring that identified event combinations not only have a temporal sequence but also a reasonable spatial propagation path. This statistically validated causal reasoning improves the reliability of event association identification.

[0054] The similarity matching in the entity linking algorithm employs a multi-level filtering strategy. First, a coarse screening is performed based on type attributes, followed by fine-tuning based on the semantic similarity of functional attributes. Type attribute matching uses an ontology-based hierarchical classification method, while functional attribute matching calculates semantic distance using a word vector model. This hierarchical matching mechanism improves computational efficiency while ensuring link accuracy.

[0055] The spatiotemporal propagation algorithm in the event evolution inference model employs a diffusion model based on partial differential equations to simulate the propagation process of abnormal events in a spatial environment. The model abstracts the site structure as a weighted graph network and calculates the propagation speed and extent of the event's impact by solving the diffusion equations. The impact range calculation model combines the site's functional zoning capacity and real-time population distribution data, predicting population density changes in the affected area through a risk propagation model.

[0056] This embodiment achieves precise quantification of anomalous event correlation mining and evolution prediction through refined algorithm design and computational models. Multi-scale spatiotemporal feature encoding provides a solid foundation for correlation analysis, causal inference based on statistical verification ensures the reliability of event combination identification, hierarchical entity linking strategies balance accuracy and efficiency, and physically inspired propagation models provide realistic event evolution predictions. This embodiment constitutes a complete computational chain from raw event sequences to composite event descriptions, providing deep situational understanding support for intelligent response decisions.

[0057] In some embodiments, composite abnormal event description information is input into a dynamic contingency plan generation engine, inference calculations are performed based on a large language model, and a dynamic contingency plan containing multiple handling actions is output, including: The descriptive information of complex abnormal events is structured and parsed to extract core semantic elements, which include at least the event type, key location, involved objects, and current situation level. The core semantic elements are matched and retrieved with the contingency plan knowledge base to obtain the matching results. The contingency plan knowledge base stores historical handling cases, standard operating procedures and expert experience rules, and an initial contingency plan framework is generated based on the matching results. The initial contingency plan framework and real-time environmental status data are input into the large language model. The real-time environmental status data includes at least the dynamic distribution of personnel, the availability of equipment, and the passage capacity of channels. By using a large language model to perform multi-round inference calculations based on reinforcement learning strategies, the expected effects of different handling strategies are simulated, and the initial contingency plan framework is optimized and reconstructed to generate a dynamic contingency plan that includes resource scheduling schemes, action sequence logic, and expected control objectives. The dynamic response plan is logically consistent, including checking for timing conflicts of response actions, resource allocation conflicts, and compliance with site management rules, and outputting a dynamic response plan that has passed the verification.

[0058] In this embodiment, the structured analysis process employs natural language processing technology based on semantic role labeling to identify core elements such as participants, time, and location from the description information of complex abnormal events. Event types are standardized and categorized using a predefined event classification system; key locations are precisely located using geographic information system coordinate transformation; involved objects are identified using named entity recognition technology to extract specific person and object identifiers; and the current situation level is comprehensively assessed based on the severity and scope of the event. This structured analysis ensures the accuracy and relevance of subsequent contingency plans.

[0059] The contingency plan knowledge base uses a graph database architecture to organize and store content. Historical handling cases include complete records of the handling process and effect evaluation data. Standard operating procedures are stored in the form of process documents, and expert experience rules are expressed through a rule engine. The matching and retrieval process adopts a multi-level indexing mechanism. First, it performs coarse screening based on event type and situation level, and then performs fine matching through semantic similarity calculation to ensure that the retrieved contingency plan framework is highly relevant to the current scenario.

[0060] After receiving the initial contingency plan framework and real-time environmental state data, the large language model analyzes the feasibility of the response actions using thought chain reasoning technology. Based on a reinforcement learning strategy, the model evaluates the risk-reward ratio of each option by simulating the execution effects of different response plans in a virtual environment. The optimization and reconstruction process includes dynamic adjustment of resource scheduling schemes, logical optimization of action sequence, and priority reordering of control objectives, ensuring that the generated dynamic contingency plan is both theoretically optimal and practically operable.

[0061] Logical consistency verification employs formal verification methods, examining dependencies between actions through a temporal logic model to avoid resource conflicts and timing inconsistencies. Compliance verification of site management rules is executed through a rule engine, ensuring that the contingency plan complies with site safety management regulations and operational procedures. Verified dynamic contingency plans are output as structured documents, containing complete execution steps and resource configuration schemes.

[0062] This embodiment achieves intelligent generation of anomaly event descriptions and executable contingency plans through a complete process of structured parsing, knowledge base matching, large language model reasoning, and logical verification. This embodiment fully leverages the reasoning capabilities of the large language model and the domain knowledge of the knowledge base, combined with real-time environmental data for dynamic optimization, ensuring that the generated contingency plans are both theoretically sound and meet actual execution conditions, effectively improving the intelligence level and response efficiency of anomaly event handling.

[0063] In some embodiments, according to a dynamic response plan, a central scheduling agent decomposes tasks and generates a set of collaborative control instructions for different functional agents, including: Semantic parsing is performed on the dynamic response plan to extract multiple atomic response actions and their corresponding execution constraints contained in the dynamic response plan; Based on the task planning algorithm, atomic processing actions are sorted and grouped according to spatiotemporal logical relationships to form a task chain with sequential dependencies; Based on the capability profiles of each functional intelligent agent, the actions in the task chain are assigned to the corresponding video analysis intelligent agent, broadcast guidance intelligent agent, and security and fire linkage intelligent agent. Add execution parameters and triggering conditions to each assigned task to generate a machine-readable set of collaborative control instructions. The execution parameters include at least the action target, resource quota and timeout setting, and the triggering conditions include the event status threshold and the completion signal of the previous task.

[0064] In this embodiment, the semantic parsing process employs dependency parsing-based techniques to identify atomic actions and their associated information from the natural language description of the dynamic response plan. Atomic actions refer to indivisible basic operational units, such as "adjusting the camera angle" or "playing a warning message." Execution constraints include elements such as time windows, spatial ranges, and resource limitations, which are extracted from the plan text and transformed into structured data using a conditional extraction model. This fine-grained parsing ensures the accuracy of subsequent task decomposition.

[0065] The task planning algorithm employs a graph planning method based on temporal logic. By analyzing the causal relationships and resource dependencies between atomic actions, it constructs a task network with partial order relationships. The algorithm considers the spatiotemporal constraints of action execution, ensuring that the order of actions is satisfied while optimizing overall execution efficiency. An exception handling mechanism is also considered during the formation of the task chain, setting up backup execution paths for critical tasks to improve the system's robustness.

[0066] Capability profiles are constructed by analyzing historical execution data and equipment performance parameters of each functional agent, including dimensions such as processing power, response speed, and resource consumption. The video analytics agent excels in visual analysis and target tracking, the broadcasting and traffic control agent specializes in speech synthesis and area broadcasting, and the security and fire protection linkage agent is responsible for the coordinated control of security and fire protection equipment. Task allocation based on capability profiles ensures that each agent can fully leverage its professional strengths.

[0067] The action target in the execution parameters is determined through coordinate transformation and semantic mapping; resource quotas are dynamically allocated based on equipment capabilities and task requirements; and timeout settings are set based on historical execution time statistics. The event status thresholds in the trigger conditions are obtained through machine learning model analysis of historical abnormal event data, and the completion signal of the previous task is acquired in real time through a task status monitoring mechanism. This fine-grained configuration of parameters ensures the executability of the instruction set and the timeliness of the system response.

[0068] This embodiment achieves precise transformation from macro-level contingency plans to micro-level control instructions through fine-grained task decomposition and intelligent resource allocation. Semantic parsing ensures accurate communication of intent, task planning optimizes the execution process, capability profiling leverages the professional strengths of each agent, and parameterized configuration guarantees the operability of the instructions. This hierarchical and progressive task decomposition mechanism effectively improves the collaborative efficiency and overall handling effectiveness of the multi-agent system.

[0069] In some embodiments, a collaborative control instruction set is executed to drive the video analysis agent, the broadcast guidance agent, and the security and fire linkage agent to perform collaborative handling operations, and to collect environmental feedback data in real time, including: Based on the video control instructions received from the collaborative control instruction set, the video analysis agent adjusts the monitoring parameters and viewing angle of the video acquisition equipment, continuously tracks and analyzes the target area, and generates video analysis data that includes the target's movement trajectory, changes in personnel density, and the continuous state of abnormal behavior. The broadcast guidance intelligent agent dynamically synthesizes scene-adaptive voice alarm content through a text generation algorithm based on the guidance strategy parameters in the collaborative control instruction set, and controls the broadcasting equipment to broadcast in a designated area. At the same time, it collects the on-site sound pressure characteristics and personnel flow response data after the voice broadcast. Based on the equipment operation logic in the collaborative control instruction set, the safety and fire protection linkage intelligent agent automatically triggers the pre-activation state of fire protection facilities, adjusts the access permissions of emergency passages, and generates resource scheduling plans, while collecting equipment execution status signals and environmental safety parameter change data. Video analytics data, personnel flow response data, and environmental safety parameter change data are used as environmental feedback data. After timestamp alignment and data fusion, the data is output to the shared event canvas.

[0070] In this embodiment, the video analytics agent dynamically adjusts the pitch, azimuth, and focal length of the PTZ camera by parsing the target area coordinates and tracking parameters in the video control commands. Preferably, the monitoring parameter adjustment employs a visual servo-based control algorithm to ensure the target is always within the optimal analysis area of ​​the image; the target's trajectory is calculated using a multi-target tracking algorithm, changes in personnel density are analyzed using a crowd density estimation model, and the persistence of abnormal behavior is evaluated using the confidence output of a behavior recognition model. This dynamic adjustment mechanism ensures the accuracy and real-time performance of the video analytics.

[0071] The text generation algorithm in the broadcast guidance agent employs template-based semantic filling technology, dynamically generating targeted alarm content based on the current abnormal event type, severity level, and impact range. Guidance strategy parameters include broadcast frequency, voice intensity, and coverage area, optimized by analyzing population density and the spread trend of abnormal events. On-site sound pressure characteristics are collected and analyzed using a microphone array, while population flow response data is obtained through the fusion of video analysis results and infrared sensor data, used to evaluate the actual effectiveness of guidance measures.

[0072] The intelligent security and fire protection system executes tiered pre-activation operations on fire protection facilities based on the linkage rules in the equipment operation logic. Emergency exit access permissions are adjusted dynamically through the access control system based on real-time personnel distribution and the location of abnormal events. The resource scheduling plan comprehensively considers equipment availability, response time, and handling priorities to generate an optimal resource allocation plan. Equipment execution status signals include equipment readiness status and fault information, while environmental safety parameter change data includes key safety indicators such as smoke concentration and temperature changes.

[0073] Timestamp alignment employs a synchronization mechanism based on a network time protocol to ensure a unified time base for multi-source data. The data fusion process uses a Kalman filter algorithm to perform spatiotemporal registration and noise filtering on heterogeneous sensor data, forming a consistent environmental state description. This refined data acquisition and processing workflow provides high-quality input data for subsequent situation assessment.

[0074] This embodiment achieves efficient implementation of the response plan through the specialized division of labor and collaborative execution of various functional intelligent agents. Dynamic adjustments via video analytics ensure monitoring effectiveness, intelligent broadcasting improves evacuation efficiency, security and fire coordination ensures the rational allocation of emergency resources, and multi-source data fusion provides the system with comprehensive environmental awareness. This embodiment fully leverages the professional advantages of each intelligent agent through a collaborative execution model, forming a complete closed-loop response system.

[0075] In some embodiments, environmental feedback data is input to a shared event canvas for multi-source information fusion to generate situational evolution information, including: Spatiotemporal alignment processing is performed on the environmental feedback data input to the shared event canvas to establish spatiotemporal correlation between video analysis data, personnel flow response data, and environmental safety parameter change data; By using a multi-source information fusion algorithm to perform feature fusion on the spatiotemporally aligned data, fused features are obtained that include at least the trend of personnel aggregation, the spread range of abnormal behavior, and the equipment response efficiency. Based on the fusion characteristics, a situational simulation model is used to predict the evolution trend of abnormal events, generating situational evolution information that includes situational level, key impact areas and potential derivative risks. Furthermore, the situational evolution information is correlated and mapped with the description information of complex abnormal events to update the global situational awareness data in the shared event canvas.

[0076] In this embodiment, the spatiotemporal alignment process employs a data registration method based on a geographic coordinate system and a unified time reference. By converting the pixel coordinates in the video analysis data into actual geographic coordinates and spatially associating them with personnel flow sensor data and environmental monitoring equipment data, a unified spatiotemporal reference system for cross-modal data is established. This alignment process ensures the consistency of data from different sources in the spatiotemporal dimension, providing a reliable foundation for subsequent fusion analysis.

[0077] The multi-source information fusion algorithm employs a deep learning-based feature-level fusion architecture, extracting key features from each data source through a multi-head attention mechanism. The trend of population density changes is calculated by analyzing time-series data of population density; the spread range of abnormal behavior is estimated by combining video analysis results and population flow patterns; and equipment response efficiency is evaluated based on the difference between equipment status data and expected response. This multi-dimensional feature fusion can comprehensively reflect the development trend of abnormal events.

[0078] The situation simulation model employs a sequence prediction method based on recurrent neural networks. By learning the evolutionary patterns of historical anomalous events, it predicts the future development trend of current events. The situation level is comprehensively assessed based on the severity and spread rate of the event. Key impact areas are calculated using a spatial propagation model, and potential derivative risks are analyzed based on the correlation between event types. This model can identify potential chain reactions and secondary risks in advance.

[0079] The association mapping process uses a semantic matching algorithm to associate real-time situational evolution information with initial descriptions of complex anomalies, updating the event status, impact range, and risk level in the global situational awareness data. This dynamic update mechanism ensures the timeliness and accuracy of the system's recognition of anomalies.

[0080] This embodiment achieves precise control over the development of abnormal events through refined multi-source information fusion and situational analysis. Spatiotemporal alignment ensures data consistency, feature fusion provides a comprehensive situational description, situational analysis enables prediction of future trends, and correlation mapping maintains cognitive continuity. The progressive analysis method in this embodiment provides reliable situational awareness support for the system's dynamic decision-making adjustments.

[0081] In some embodiments, based on fusion features, a situational extrapolation model is used to predict the evolution trend of anomalous events, generating situational evolution information including situational level, key impact areas, and potential derivative risks, including: The fused features are input into a pre-trained spatiotemporal graph neural network model, and the propagation probability of abnormal events in the spatial topology of the venue is calculated through a node propagation algorithm. Based on propagation probability, the Monte Carlo tree search algorithm is used to simulate the evolution path of anomalous events at multiple time steps in the future, generating multiple potential evolution scenarios with different occurrence probabilities. For each potential evolution scenario, an impact assessment is conducted, and the corresponding situation level, key impact area range, and potential derivative risk type are calculated using a risk quantification model; By integrating the simulation results of all potential evolution scenarios, an evidence theory algorithm is used for fusion decision-making to generate situation evolution information including situation level prediction with confidence intervals, probability distribution of key impact areas, and derivative risk spectrum.

[0082] In this embodiment, the pre-trained spatiotemporal graph neural network model learns the propagation patterns in the spatial topology of a location by analyzing historical anomalous event data. The node propagation algorithm, based on a graph attention mechanism, calculates the propagation probability of an anomalous event along spatial connection paths, while also considering the influence of spatiotemporal characteristics such as functional zoning and personnel flow patterns on the propagation process. This graph-based propagation modeling can accurately reflect the diffusion characteristics of anomalous events in complex environments.

[0083] The Monte Carlo Tree Search algorithm simulates the possible evolution paths of anomaly events across multiple time steps through random sampling and policy evaluation. In each simulation, the algorithm selects the most probable direction based on the current propagation probability, generating potential evolutionary scenarios with varying probabilities through multiple independent simulations. This probability-based sampling simulation method comprehensively covers all possible evolutionary paths, avoiding the limitations of single-prediction approaches.

[0084] The risk quantification model employs a multi-indicator comprehensive evaluation method, combining factors such as event type, impact range, and duration to calculate the situation level. The critical impact area is determined through a spatial propagation model and population density, while potential derivative risk types are identified based on event chain analysis to pinpoint possible secondary events. This model can quantitatively assess the risk level of various evolution scenarios.

[0085] The evidence-based algorithm calculates the confidence interval for situational level prediction and the probability distribution of key influence areas by fusing simulation results from multiple evolutionary scenarios. The algorithm comprehensively considers the occurrence probability and risk assessment results of each scenario to generate situational evolution information with uncertainty metrics. This evidence-based fusion decision-making improves the reliability and practicality of the prediction results.

[0086] This embodiment achieves precise control over the evolution trend of abnormal events through a multi-layered prediction and evaluation mechanism. The spatiotemporal graph neural network accurately models the event propagation patterns, Monte Carlo tree search comprehensively explores possible development paths, the risk quantification model provides objective evaluation criteria, and evidence theory effectively integrates uncertain information. This comprehensive prediction method provides reliable situational awareness support for the system's dynamic decision-making.

[0087] In some embodiments, based on situational evolution information, the cooperative control instruction set is dynamically adjusted through a decision control algorithm until the handling of abnormal events is completed, including: The situation evolution information is quantitatively analyzed to extract the trend of situation level change, the diffusion speed of key affected areas, and the probability of potential derivative risks. The analyzed situation quantification indicators are compared with the preset handling target thresholds, and a handling strategy adjustment vector is generated through a fuzzy decision algorithm; Based on the adjustment vector of the handling strategy, the parameters of the currently executing collaborative control instruction set are optimized. The parameter optimization includes at least adjusting the monitoring focus area of ​​the video analysis agent, updating the alarm content priority of the broadcast guidance agent, and replanning the resource scheduling path of the security and fire linkage agent. Drive each functional agent to perform adjusted disposal operations based on the optimized collaborative control instruction set, and continue to collect a new round of environmental feedback data; By continuously monitoring the situation level indicators in the situation evolution information through the convergence judgment algorithm, when all situation level indicators are continuously lower than the preset threshold and reach a stable duration, the abnormal event handling is determined to be completed.

[0088] In this embodiment, the quantitative analysis process extracts key indicators from the situation evolution information using time series analysis. The trend of situation level changes is calculated using sliding window regression, the diffusion speed of key impact areas is assessed using the spatial boundary change rate, and the probability of potential derivative risks is based on the output of a risk propagation model. This quantitative analysis transforms complex situation descriptions into measurable decision-making criteria, providing accurate input data for subsequent adjustments.

[0089] The fuzzy decision-making algorithm employs a multi-input, single-output fuzzy inference system. It generates an adjustment vector by comparing situation quantification indicators with preset thresholds. The algorithm defines a fuzzy rule base to transform fuzzy concepts such as "high situation level" and "rapid spread" into specific adjustment directions and magnitudes. The response strategy adjustment vector contains parameter adjustment suggestions from each agent, ensuring that the response strategy maintains an optimal match with the current situation.

[0090] During parameter optimization, the monitoring focus area is dynamically adjusted based on the probability distribution of key impact areas, alarm content priorities are reordered based on the probability of derived risks, and resource scheduling paths are optimized by comprehensively considering the current equipment status and expected needs. This fine-grained parameter adjustment ensures that the execution strategies of each agent remain synchronized with changes in the situation.

[0091] The convergence determination algorithm identifies the stable state of the response effect by monitoring the time-series characteristics of situation level indicators. When all situation level indicators remain below the safety threshold for a preset period of time, and the fluctuation range is controlled within the allowable range, the system determines that the abnormal event has been effectively controlled. This determination mechanism based on continuous monitoring of multiple indicators ensures the accurate judgment of the completion of the response.

[0092] This embodiment achieves dynamic adaptive adjustment of the handling strategy through a closed-loop control mechanism of quantitative analysis, fuzzy decision-making, parameter optimization, and convergence determination. Quantitative analysis provides precise decision-making basis, fuzzy decision-making enables intelligent strategy generation, parameter optimization ensures execution effectiveness, and convergence determination provides accurate termination judgment. This closed-loop adjustment mechanism enables the system to continuously optimize the handling strategy according to real-time situation changes until the abnormal event is completely resolved.

[0093] By adopting the above technical solutions, this invention differs from existing technologies and possesses the following beneficial effects: By constructing a complete technical chain from environmental perception to intelligent response, it achieves rapid identification and proactive handling of abnormal behaviors in key locations. The system employs multimodal data fusion technology to improve recognition accuracy, enhances semantic understanding of complex abnormal scenarios through knowledge graphs, and utilizes large language models to achieve intelligent generation and optimization of response plans. With the help of a multi-agent collaborative execution mechanism, the system can transform response plans into specific control commands and drive coordinated responses from various functional units. Combining real-time environmental feedback data, the system dynamically adjusts response strategies through situational analysis and decision control algorithms, forming a closed-loop control of perception, decision-making, execution, and optimization. This technical solution effectively improves the timeliness, accuracy, and adaptability of abnormal event handling, overcoming the shortcomings of traditional systems in response lag and rigid handling in complex scenarios, and providing intelligent technical support for the safety management of key locations.

[0094] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A smart system for identifying and proactively handling abnormal behavior in key locations, characterized in that: include: Acquire multimodal environmental data from sensing devices deployed in key locations; The multimodal environmental data is subjected to abnormal behavior identification and analysis to generate a preliminary abnormal event sequence; The preliminary sequence of abnormal events is semantically fused with a preset scene knowledge graph to generate composite abnormal event description information. The composite abnormal event description information is input into the dynamic contingency plan generation engine, and reasoning calculation is performed based on the large language model to output a dynamic contingency plan containing multiple handling actions. According to the dynamic response plan, the central dispatching intelligent agent decomposes the tasks and generates a set of collaborative control instructions for different functional intelligent agents, including video analysis intelligent agents, broadcast guidance intelligent agents, and security and fire linkage intelligent agents. The system executes the collaborative control instruction set to drive the video analysis agent, the broadcast guidance agent, and the security and fire linkage agent to perform collaborative handling operations, and collects environmental feedback data in real time. The environmental feedback data is input into a shared event canvas for multi-source information fusion to generate situational evolution information; Based on the situational evolution information, the collaborative control instruction set is dynamically adjusted through a decision control algorithm until the abnormal event is handled. Specifically, the preliminary abnormal event sequence is semantically fused with a preset scenario knowledge graph to generate composite abnormal event description information, including: Based on the spatiotemporal attributes of each abnormal event in the preliminary abnormal event sequence, a graph neural network algorithm is used to mine the associated paths in the scene knowledge graph to identify combinations of abnormal events with causal relationships. The identified abnormal event combinations are semantically enhanced, and the event elements are associated with the location structure entities, equipment entities and personnel role entities in the knowledge graph through the entity linking algorithm to obtain the association mapping results; Based on the correlation mapping results, an event evolution reasoning model is used to predict the development trend of abnormal event combinations, and generate composite abnormal event description information including event correlation, evolution path and scope of influence. The environmental feedback data is input into a shared event canvas for multi-source information fusion to generate situational evolution information, including: Spatiotemporal alignment processing is performed on the environmental feedback data input to the shared event canvas to establish a spatiotemporal correlation between video analysis data, personnel flow response data, and environmental safety parameter change data. By using a multi-source information fusion algorithm to perform feature fusion on the spatiotemporally aligned data, fused features are obtained that include at least the trend of personnel aggregation, the spread range of abnormal behavior, and the equipment response efficiency. Based on the fusion features, a situational simulation model is used to predict the evolution trend of abnormal events, generating situational evolution information that includes situational level, key impact areas, and potential derivative risks. Furthermore, the situational evolution information is associated and mapped with the composite abnormal event description information to update the global situational awareness data in the shared event canvas.

2. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, The multimodal environment data is subjected to abnormal behavior identification and analysis to generate a preliminary abnormal event sequence, including: The video stream in the multimodal environment data is analyzed using video analysis algorithms to extract dynamic features, including at least human posture, movement trajectory, and object state. The audio stream in the multimodal environment data is parsed using an audio processing algorithm to extract audio features, including at least voiceprint features, sound source location, and anomalous phonemes. The dynamic features and audio features are input into a multimodal fusion detection model for joint analysis to identify initial abnormal events that include at least the behavior type, location of occurrence, and severity level. Cluster analysis is performed on continuously identified initial anomalous events based on time windows to generate a preliminary sequence of anomalous events with spatiotemporal correlation.

3. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, Based on the spatiotemporal attributes of each anomalous event in the preliminary anomalous event sequence, a graph neural network algorithm is used to mine association paths in the scene knowledge graph to identify combinations of anomalous events with causal relationships, including: Extract the spatial coordinates and timestamps of each abnormal event from the preliminary abnormal event sequence to construct a spatiotemporal feature vector; The spatiotemporal feature vector is input into a graph neural network algorithm to perform multi-hop neighbor node traversal in the preset scene knowledge graph and calculate the correlation strength between nodes of abnormal events. Based on the aforementioned correlation strength, anomalous event combinations that satisfy spatiotemporal proximity and logical correlation are identified using a causal reasoning algorithm. Semantic enhancement processing is performed on the identified combinations of abnormal events. Event elements are then linked to location structure entities, equipment entities, and personnel role entities in the knowledge graph through entity links, resulting in the following association mapping results: Analyze the event elements in the abnormal event combination and extract the type and function attributes of the event elements; The type and functional attributes of the event elements are matched with the location structure entities, equipment entities and personnel role entities in the preset scene knowledge graph using an entity linking algorithm. Establish semantic association mapping relationships between event elements and knowledge graph entities to form an enhanced semantic network, which is the association mapping result; Based on the correlation mapping results, an event evolution inference model is used to predict the development trend of abnormal event combinations, generating composite abnormal event description information that includes event correlations, evolution paths, and impact scope, including: An enhanced semantic network input event evolution reasoning model is used to simulate the diffusion path of abnormal events in the site structure based on a spatiotemporal propagation algorithm; By using an impact range calculation model, combined with the functional zoning of the venue and the population density, the impact range of a combination of abnormal events can be predicted. By combining the aforementioned diffusion paths and impact ranges, a composite description of the abnormal event is generated, which includes event correlations, evolution paths, and impact ranges.

4. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, The composite abnormal event description information is input into the dynamic contingency plan generation engine, which performs inference calculations based on a large language model and outputs a dynamic contingency plan containing multiple handling actions, including: The composite abnormal event description information is structured and parsed to extract core semantic elements, which include at least the event type, key location, involved objects, and current situation level. The core semantic elements are matched and retrieved with the contingency plan knowledge base to obtain matching results. The contingency plan knowledge base stores historical handling cases, standard operating procedures and expert experience rules, and an initial contingency plan framework is generated based on the matching results. The initial contingency plan framework and real-time environmental status data are input into the large language model. The real-time environmental status data includes at least the dynamic distribution of personnel, the availability of equipment, and the passage capacity of channels. The large language model is used to perform multi-round inference calculations based on reinforcement learning strategies to simulate the expected effects of different handling strategies, and the initial contingency plan framework is optimized and reconstructed to generate a dynamic contingency plan that includes resource scheduling schemes, action sequence logic and expected control objectives. The dynamic response plan is logically consistent, including checking for timing conflicts of response actions, resource allocation conflicts, and compliance with site management rules, and outputting a dynamic response plan that has passed the verification.

5. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, According to the aforementioned dynamic response plan, the central scheduling agent decomposes the tasks and generates a set of collaborative control instructions for different functional agents, including: The dynamic response plan is semantically parsed to extract multiple atomic response actions and their corresponding execution constraints. Based on the task planning algorithm, the atomic processing actions are sorted and grouped according to the spatiotemporal logical relationship to form a task chain with sequential dependencies; Based on the capability profiles of each functional intelligent agent, the actions in the task chain are assigned to the corresponding video analysis intelligent agent, broadcast guidance intelligent agent, and security and fire linkage intelligent agent. Execution parameters and triggering conditions are added to each assigned task to generate a machine-readable set of collaborative control instructions. The execution parameters include at least the action target, resource quota and timeout setting, and the triggering conditions include the event status threshold and the previous task completion signal.

6. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, The system executes the collaborative control instruction set to drive the video analysis agent, the broadcast guidance agent, and the security and fire linkage agent to perform collaborative handling operations, and collects environmental feedback data in real time, including: Based on the video control instructions received from the collaborative control instruction set, the video analysis agent adjusts the monitoring parameters and viewing angle of the video acquisition equipment, continuously tracks and analyzes the target area, and generates video analysis data that includes the target's movement trajectory, changes in personnel density, and the continuous state of abnormal behavior. The broadcast guidance intelligent agent dynamically synthesizes scene-adaptive voice alarm content through a text generation algorithm based on the guidance strategy parameters in the collaborative control instruction set, and controls the broadcasting equipment to broadcast in a designated area. At the same time, it collects the on-site sound pressure characteristics and personnel flow response data after the voice broadcast. Based on the equipment operation logic in the collaborative control instruction set, the safety and fire protection linkage intelligent agent automatically triggers the pre-activation state of fire protection facilities, adjusts the access permissions of emergency passages, and generates resource scheduling plans, while collecting equipment execution status signals and environmental safety parameter change data. The video analysis data, personnel flow response data, and environmental safety parameter change data are used as environmental feedback data. After timestamp alignment and data fusion, the data is output to the shared event canvas.

7. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, Based on the fusion features, a situational simulation model is used to predict the evolution trend of abnormal events, generating situational evolution information including situational level, key impact areas, and potential derivative risks, including: The fused features are input into a pre-trained spatiotemporal graph neural network model, and the propagation probability of abnormal events in the spatial topology of the location is calculated through a node propagation algorithm. Based on the propagation probability, the Monte Carlo tree search algorithm is used to simulate the evolution path of the anomalous event at multiple time steps in the future, generating multiple potential evolution scenarios with different occurrence probabilities. For each potential evolution scenario, an impact assessment is conducted, and the corresponding situation level, key impact area range, and potential derivative risk type are calculated using a risk quantification model; By integrating the simulation results of all potential evolution scenarios, an evidence theory algorithm is used for fusion decision-making to generate the situation evolution information, which includes a situation level prediction with confidence intervals, a probability distribution of key impact areas, and a derived risk spectrum.

8. The intelligent identification and proactive handling system for abnormal behavior in key locations according to claim 1, characterized in that, Based on the aforementioned situational evolution information, the coordinated control instruction set is dynamically adjusted using a decision control algorithm until the handling of abnormal events is completed, including: The situation evolution information is quantitatively analyzed to extract the situation level change trend, the diffusion speed of key affected areas, and the probability of potential derivative risks. The analyzed situation quantification indicators are compared with the preset handling target thresholds, and a handling strategy adjustment vector is generated through a fuzzy decision algorithm; Based on the adjustment vector of the handling strategy, the parameters of the currently executing collaborative control instruction set are optimized. The parameter optimization includes at least adjusting the monitoring focus area of ​​the video analysis agent, updating the alarm content priority of the broadcast guidance agent, and replanning the resource scheduling path of the security and fire linkage agent. Drive each functional agent to perform adjusted disposal operations based on the optimized collaborative control instruction set, and continue to collect a new round of environmental feedback data; The situation level indicators in the situation evolution information are continuously monitored by the convergence judgment algorithm. When all situation level indicators are continuously lower than the preset threshold and reach a stable duration, the abnormal event handling is determined to be completed.

Images