Park public security event linkage disposal method and system based on multi-modal perception

Abstract

The present application relates to the technical field of wisdom security and multi-modal artificial intelligence, and discloses a park public security event linkage disposal method based on multi-modal sensing, which can improve the sensing robustness of a security system, reduce the false alarm and missed alarm rate, improve the event cognitive depth, linkage intelligence and disposal closed loop, and comprises the following steps: S1, collecting data of various sensing devices in the park, and preliminarily aligning all the data according to a unified space-time coordinate to obtain different modal characteristics; S2, using a multi-modal Transformer to align and fuse different modal characteristics; S3, introducing an event ontology to structurally model a security event, and outputting a structured event description; S4, reasoning the structured event through context sensing to grade the risk, and generating a disposal strategy; S5, processing the security event according to the disposal strategy and feeding back the execution effect. The system for implementing the above-mentioned park public security event linkage disposal method is also disclosed.

Description

Technical Field

[0001] This invention relates to the field of smart security and multimodal artificial intelligence, and in particular to a method and system for coordinated handling of public safety incidents in a park based on multimodal perception. Background Technology

[0002] As parks expand in scale and become increasingly complex, the demand for park security and emergency management continues to rise. Park security systems have become core infrastructure for ensuring the safety of personnel and property and maintaining normal operational order. Currently, park security systems typically consist of multiple independent subsystems, including video surveillance systems, fire alarm systems, access control and personnel positioning systems, and broadcasting and emergency lighting systems. Video surveillance systems utilize computer vision (CV) algorithms for facial recognition and area intrusion detection; fire alarm systems use smoke and heat sensors to detect environmental anomalies and trigger fire alarms; access control and personnel positioning systems record personnel entry and exit and their real-time location; and broadcasting and emergency lighting systems are used for evacuation guidance and lighting assistance in emergency scenarios to ensure orderly evacuation. However, these subsystems operate independently and do not interact with each other. Single sensors (such as smoke sensors) are susceptible to environmental interference, and the accuracy of single video surveillance systems drops significantly in low-light or obstructed environments, resulting in problems such as limited perception dimensions and high false alarm and false negative rates. Secondly, the system can only identify the surface symptoms of events (such as "smoke" or "intrusion"), and cannot comprehensively assess the actual threat by combining the park scenario and business logic. It struggles to distinguish between normal laboratory smoke exhaust and smoke from an electrical fire, or between unintentional tailing and malicious intrusion by employees who forgot their cards, resulting in a superficial understanding of events. Furthermore, the existing system's linkage rules are preset and fixed, failing to dynamically adjust the response process based on contextual information such as event type, severity, time, location, and personnel distribution. For example, a small-scale fire at night should trigger completely different emergency plans than a fire during a large gathering in the daytime, resulting in insufficient flexibility and rigid linkage, lack of coordination. In addition, after an alarm is issued, the system lacks tracking and feedback on the response process (such as whether firefighters have arrived and whether personnel have evacuated), making it impossible to assess the response effectiveness and ensure the quality of the response, resulting in a broken response process and a missing closed-loop response mechanism. Summary of the Invention

[0003] Therefore, it is necessary to address the above-mentioned shortcomings by providing a multimodal perception-based method and system for coordinated handling of public safety incidents in parks, which can improve the perception robustness of security systems, reduce false alarm and missed alarm rates, and enhance the depth of event cognition, linkage intelligence, and closed-loop handling.

[0004] A collaborative response method for public safety incidents in a park based on multimodal perception includes the following steps: S1. Collect data from various sensing devices in the park and initially align all data according to a unified spatiotemporal coordinate to obtain different modal features; S2. Employ a multimodal Transformer to align and fuse features from different modalities; S3. Introduce an event ontology to perform structured modeling of security events and output a structured event description; S4. Use context awareness to reason about structured events to classify risks and generate response strategies; S5. Handle security incidents according to the aforementioned handling strategy and provide feedback on the execution results.

[0005] In one embodiment, the various sensing device data include camera video streams, microphone audio streams, various sensor data streams that sense changes in the environment, preset personnel positioning data, and park business data.

[0006] In one embodiment, in step S1, based on the BIM model and NTP time synchronization, all data are initially aligned according to a unified spatiotemporal coordinate.

[0007] In one embodiment, in step S3, structured modeling includes defining event types, attributes, and causal relationships. The event types include fire, intrusion, fall, and gathering. The attributes include location, time, involved personnel, and severity level. The causal relationships include the order in which multiple different modal characteristics corresponding to the same security event occur.

[0008] In one embodiment, in step S3, the structured event description includes event type, confidence level, key evidence, and preliminary risk level.

[0009] In one embodiment, step S4, which involves reasoning about structured events through context awareness to determine risk levels and generate a response strategy, includes: S41. A static rule base for pre-setting national standards and park emergency plans; S42. Train a dynamic policy model based on the static rule base; S43. Input the structured event description, current context, and resource availability into the dynamic strategy model; S44. Output a set of processing instructions based on the structured event description, the current context, and resource availability.

[0010] In one embodiment, the current context includes time, weather, population density, and status of important areas; the resource availability includes the location of security personnel and the status of fire-fighting equipment; and the set of disposal instructions includes controlling preset equipment to operate and sending instructions to preset security personnel.

[0011] In one embodiment, step S5, which involves processing the security incident according to the handling strategy and providing feedback on the execution effect, includes: S51. Distribute the processing instructions in the processing instruction set to each execution subsystem for collaborative execution; S52. Continuously monitor the processing effect through various sensing devices and generate feedback signals; S53. Optimize the dynamic strategy model based on the feedback signal.

[0012] In one embodiment, step S51, the coordinated execution includes activating sprinkler or gas fire suppression, locking or opening specific passages, playing customized evacuation voice prompts, illuminating emergency routes, and dispatching orders to the nearest patrol personnel.

[0013] This invention also discloses a joint response system for public safety incidents in industrial parks, used to implement the aforementioned joint response method for public safety incidents in industrial parks, comprising: The multimodal sensing access module is used to collect data from various sensing devices in the park and initially align all data according to a unified spatiotemporal coordinate to obtain different modal features. The cross-modal event understanding module is used to align and fuse different modal features using a multimodal Transformer, introduce event ontology to perform structured modeling of security events, and output structured event descriptions. The context-aware decision-making module is used to reason about structured events through context awareness to classify risks and generate response strategies. The collaborative execution and closed-loop feedback module is used to process security events according to the handling strategy and provide feedback on the execution effect.

[0014] Implementing the method and system for the collaborative disposal of public security incidents in the park based on multi-modal perception of the present invention, through context awareness for structured event reasoning, realizes multi-modal cross-verification, improves the perception robustness of the security system. Compared with the false alarm rate of more than 30% of the conventional single smoke sensor, this solution can increase the fire recognition accuracy rate to more than 98%, significantly reducing the false alarm rate; supports complex scene reasoning, can distinguish easily confused scenes such as "real fire" and "cooking fume", "malicious intrusion" and "employee tailing", etc., improving the depth of event cognition; dynamically generates strategies according to the event type and context, avoids "making a mountain out of a molehill" or "insufficient response", realizes precise hierarchical disposal, and improves the intelligence of the linkage; from perception to execution, there is no need for manual intervention, the response speed is less than 10s, realizing a full-automatic closed-loop response; it realizes high-precision recognition of public security incidents, context-aware grading evaluation, and multi-system adaptive collaborative disposal by constructing a five-in-one intelligent security closed-loop of "perception - understanding - decision - execution - feedback", greatly improving the timeliness, accuracy and effectiveness of the park's security response; after testing, in a typical park scenario, this solution shortens the average response time of security incidents from 3 - 5 minutes to within 15s, and reduces the ineffective police dispatch caused by false alarms by 70%. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the method for the collaborative disposal of public security incidents in the park based on multi-modal perception in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be given with reference to the accompanying drawings. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0017] Embodiment 1 Please refer to Figure 1 , the present invention discloses a method for the collaborative disposal of public security incidents in the park based on multi-modal perception, which can improve the perception robustness of the security system, reduce the false alarm and missed alarm rates, improve the depth of event cognition, the intelligence of the linkage and the closed-loop nature of the disposal. The method for the collaborative disposal of public security incidents in the park includes the following steps: S1. Collect data of various perception devices in the park, and preliminarily align all the data according to the unified space-time coordinates to obtain different modal features, realizing real-time collection and space-time alignment of multi-modal data in the park.

[0018] In this embodiment, real-time access to data streams from various sensing devices within the park is required to obtain various real-time data related to security incidents within the park. The data from these sensing devices includes camera video streams, microphone audio streams, data streams from various sensors detecting environmental changes, preset personnel positioning data, and park business data. Specifically, the camera video streams include video streams from various visible light and infrared cameras deployed within the park, i.e., various video streams acquired within the park, thereby obtaining video modal data of the park. The microphone audio streams are directional microphone array audio streams, which can capture sound signals such as broken glass, cries for help, and explosions, thereby obtaining auditory modal data of the park. The various sensor data streams detecting environmental changes include data streams detected by sensors such as smoke sensors, temperature sensors, CO sensors, and combustible gas sensors used to detect environmental changes, thereby obtaining environmental sensing modal data of the park. The preset personnel positioning data includes personnel whose locations are determined using UWB (Ultra-Wideband) and Bluetooth beacon technologies; these personnel are capable of responding to security incidents within the park, thereby obtaining personnel location modal data of the park. The park's business data includes access control card swipe records, meeting reservation system data, visitor registration information, and other data, thereby obtaining business modality data within the park.

[0019] In step S1, based on the BIM model and NTP time synchronization, all data are initially aligned according to a unified spatiotemporal coordinate system. That is, using the BIM model as the spatial reference and NTP time synchronization as the time reference, data from different sources and of different types (such as video, sensor, and personnel positioning data) are uniformly calibrated into the same time and space system, achieving preliminary spatiotemporal synchronization and laying the foundation for subsequent cross-modal data fusion and event analysis. In this embodiment, the BIM model is a digital 3D model of the park, equivalent to drawing a precise digital map of the park, marking the specific spatial coordinates (such as which building, floor, and location) of each area and device (such as cameras, sensors, and Bluetooth beacons). In this way, the BIM model provides a spatial reference for all data, allowing each piece of data to correspond to a specific physical location. NTP is a precise time synchronization technology used to ensure that the time of all devices (such as cameras, sensors, and Bluetooth beacons) is completely unified, with errors controlled within a very small range. For example, the images captured by cameras, the alarm signals from smoke sensors, and the location data of personnel positioning are all tagged with a "unified timestamp" to ensure that they record events that occurred at the same time. In this way, by initially aligning the data using unified spatiotemporal coordinates, all data (i.e., visual modal data, auditory modal data, environmental sensor modal data, location modal data, and business modal data) are calibrated according to the spatial coordinates of the BIM model and the unified time of NTP. For example, "smoke sensor alarm at 10:00 AM (NTP unified time), 2nd floor of Building 3 (BIM spatial coordinates)," "video footage at the same time and location," and "personnel location information at the same location at the same time" will be accurately matched, achieving initial alignment with consistent time and corresponding location. This allows the dispersed multimodal data to be unified in the same time and space before more accurate fusion analysis can be performed.

[0020] S2. Employ a multimodal Transformer to align and fuse features from different modalities, thereby achieving cross-modal feature fusion and preliminary event identification.

[0021] Multimodal Transformer is an intelligent algorithm framework. Its core is to use alignment and fusion—two key steps—to transform different types of data (or modalities), such as video, sensor data (e.g., smoke or temperature sensors), and personnel location data, into unified and valuable comprehensive features. This allows the system to make more accurate judgments and breaks down the barriers between individual subsystems. Multimodal refers to multiple types of data sources (e.g., in park security, video surveillance image data, fire system sensor data, and access control personnel information data). These data types and formats are different and originally could not be directly coordinated. Alignment allows data from different modalities to correspond. For example, it precisely matches "smoke sensor alarm (value) in a certain area" with "video footage (image) of the same area at the same time" and "personnel location (location) in the area" in time and space, ensuring that the three describe "an event at the same time and in the same area," avoiding data discrepancies. Then, through fusion, using Transformer... The algorithm integrates aligned data from different modalities (such as smoke sensor readings, video footage, and personnel locations) and outputs a comprehensive feature. For example, it can determine a real fire based on the combination of "smoke sensor readings exceeding the limit, video showing open flames but no personnel location" and "smoke sensor readings exceeding the limit, video showing laboratory smoke exhaust and personnel location" and determine it as a false alarm. This enables in-depth analysis of the nature of the event. In this way, various scattered data can "match each other and information can be merged," no longer operating independently. This allows the system to make accurate judgments by combining multiple aspects of information, just like a human, solving the problems of "single perception, high false alarm rate, and shallow event understanding."

[0022] S3. Introduce Event Ontology to perform structured modeling of security events and output structured event descriptions, thereby generating structured descriptions based on Event Ontology.

[0023] In step S3, structured modeling includes defining event types, attributes, and causal relationships. Event types include fire, intrusion, falls, and gatherings. Attributes include the location, time, involved personnel, and severity level of the event. Causal relationships include the order in which multiple modal characteristics corresponding to the same safety event occur. For example, in the case of a fire caused by a short circuit, the causal relationship is generally that the short circuit causes smoke and temperature rise, which in turn triggers sparks or open flames. Furthermore, in step S3, the structured event description includes event type, confidence level, key evidence (such as "flame area in video, explosion sound in audio, smoke alarm"), and preliminary risk level. For example, the output structured event description could be: a small fire in a laboratory with a confidence level of 92%, indicating a medium risk as the camera in laboratory 302 detected sparks and the smoke alarm detected smoke exceeding preset values.

[0024] S4. Risk assessment is performed by reasoning about structured events using context awareness, generating response strategies. This step employs a context-aware intelligent decision-making engine to achieve dynamic intelligent decision-making.

[0025] In step S4, risk assessment is performed by reasoning about structured events through context awareness, and a response strategy is generated, including: S41. A static rule base for pre-set national standards and park emergency plans. Specifically, this static rule base for national standards and park emergency plans can be "GB 50116 Code for Design of Automatic Fire Alarm Systems" or other fire management codes that meet national standards.

[0026] S42. Training a dynamic strategy model based on a static rule base: When establishing a dynamic strategy model, the model is trained and established by feeding information such as event descriptions, handling instructions, and handling effects that have already occurred in response to fires or other fire safety incidents into the model and using reinforcement learning.

[0027] S43. Input structured event descriptions, current context, and resource availability into the dynamic strategy model to feed model information. The current context includes time, weather, population density, and the status of important areas; resource availability includes the location of security personnel and the status of fire-fighting equipment.

[0028] S44. Based on the structured event description, current context, and resource availability, output a set of handling instructions. The set of handling instructions includes controlling the operation of preset equipment and sending instructions to preset security personnel.

[0029] For example, the structured event input to the dynamic strategy model is described as a small fire in the laboratory (confidence level 92%); the current context is 23:00 at night, no personnel are present, and there is a hazardous chemical cabinet nearby; the set of instructions is: 1. Automatically shut off the ventilation in the area, 2. Activate the gas extinguishing system, 3. Lock the adjacent fire doors, 4. Notify the on-duty engineer for remote confirmation.

[0030] In other words, this solution employs a hybrid reasoning mechanism based on rules and learning. Rules ensure compliant, fixed, and mandatory business logic (such as fire thresholds, access boundaries, and emergency procedures), guaranteeing the system is error-free and does not violate regulations. Learning (model training) handles complex scenarios, ambiguous events, and interfering factors (such as smoke / steam, lighting changes, and occlusion), improving the system's anti-interference capabilities and generalization. This combination balances determinism and robustness, avoiding misjudgments and omissions. Secondly, pure deep learning is a "black box," unable to explain the basis for determining fire or intrusion. With hybrid reasoning based on rules, rules are traceable, and learning can assist; the decision-making process is transparent, auditable, and reviewable, meeting the regulatory and accountability requirements of high-security scenarios such as parks, fire protection, and security. Furthermore, traditional fixed rules cannot adapt to different scenarios such as day / night, dense / open spaces, normal operations / real emergencies. Hybrid reasoning can automatically adjust strategies based on environment, time, personnel distribution, and event severity, achieving flexible, adaptive, and context-aware intelligent decision-making, rather than rigid linkage. It can dynamically adapt to complex scenarios without relying on manual presets. In addition, rules can be quickly configured, put into use immediately, and are easy to maintain. Learning is automatically optimized from historical data, reducing manual debugging. The hybrid mechanism does not require training the model from scratch or hardcoding thousands of rules, reducing deployment costs and improving iteration efficiency.

[0031] S5. Handle security incidents according to the handling strategy and provide feedback on the implementation results to obtain information on the handling status of security incidents.

[0032] In step S5, handling security incidents according to the handling strategy and providing feedback on the execution results includes: S51. Distribute the disposal instructions from the disposal instruction set to each execution subsystem for coordinated execution. These execution subsystems include the fire protection subsystem, access control subsystem, broadcasting subsystem, lighting subsystem, and security dispatch subsystem. In step S51, coordinated execution includes activating sprinkler or gas extinguishing systems, locking or opening specific passageways, playing customized evacuation audio, illuminating emergency routes, and dispatching orders to the nearest patrol personnel. Specifically, the fire protection subsystem activates cold water sprinklers or non-combustible and non-flammable gas extinguishing; the access control subsystem locks or opens specific passages to facilitate personnel evacuation and movement; the broadcast subsystem plays customized evacuation messages to guide personnel to the appropriate emergency exits, such as "Please proceed to fire door No. 5 via the stairs for evacuation"; the lighting subsystem illuminates emergency paths so that personnel can use them as needed, preventing falls, bumps, or stampedes; and the security dispatch subsystem dispatches orders to the nearest patrol personnel so that they can organize evacuations and intervene in relevant hazardous areas in a timely manner, thus comprehensively addressing safety incidents from aspects such as fire control and personnel evacuation.

[0033] S52. Continuously monitor the processing effect through various sensing devices and generate feedback signals.

[0034] In other words, cameras capture images of the location where a security incident occurred; microphones capture sound signals such as broken glass, cries for help, or explosions; various sensors detect signals such as smoke, gas, and temperature at the location of the incident to determine whether the resulting changes in smoke, gas, and temperature have been controlled or resolved; preset personnel location detection determines whether relevant personnel have arrived at or left the location of the incident; and access control or visitor registration systems track the entry and exit of relevant personnel into the park. For example, detecting whether "the temperature has dropped" or "whether personnel have left the danger zone" allows for obtaining the results or effects of the security incident handling, enabling park personnel to continue monitoring the progress of the incident.

[0035] S53. Optimize the dynamic strategy model based on feedback signals.

[0036] In other words, the received feedback signals are input into the dynamic strategy model and matched with the structured event description, current context, and handling instructions. The processing instructions corresponding to the structured event description and current context are then optimized based on the feedback signals. For example, when the received feedback signal indicates a temperature drop but the rate of drop does not meet the preset requirements, the optimization plan can increase ventilation at the location of the safety incident and increase the gas or liquid flow rate of the fire extinguishing system. When the received feedback signal indicates that personnel are moving but have not left the danger zone, the optimization plan can control the opening of more doors (to prevent personnel from being trapped in the fire and unable to reach the preset safety doors) and increase the volume of the voice evacuation content, so as to continuously optimize the dynamic strategy model and improve the handling effect of safety incidents.

[0037] The following example, using the handling of a suspected fire incident in a park laboratory, illustrates the entire process of the park's joint response to public safety incidents.

[0038] The scenario is set as follows: a chemical laboratory on the third floor of a research and development building in a smart park, equipped with visible light and infrared dual-light cameras (capable of shooting under visible light and infrared light conditions), high-sensitivity smoke sensors, temperature sensors, CO sensors, directional microphones covering the experimental area to detect sound signals, UWB (ultra-wideband) personnel positioning tags worn by laboratory personnel, a gas fire extinguishing system, electric fire dampers, and emergency broadcasting.

[0039] During multimodal sensing and initial triggering, at t=14:05:00 (2:05 PM), the smoke sensor alarmed (smoke concentration > threshold concentration); the infrared camera detected a sudden rise in local temperature to 80℃ on the experimental platform; the microphone captured a short "crackling" sound (suspected electric arc); personnel location showed that the laboratory was currently unoccupied. At this time, the system aligned the above data according to BIM coordinates (room R305) and timestamp, triggering a "suspected fire" candidate event.

[0040] In cross-modal event understanding, the multimodal Transformer model processes inputs including: video features: ViT (Vision Transformer model, a deep learning model for extracting image features) extracts flame texture and high-temperature areas from thermal imaging; audio features: 1D-CNN (one-dimensional convolutional neural network) identifies the acoustic spectrum features of electric arc discharge; sensor features: triple verification using smoke, temperature, and CO sensors; event ontology matching: conforming to the "electrical fire" subclass (feature combination: electric arc sound + localized high temperature + smoke + video without open flame). The output structured event is as follows: json edit { "event_type": "Electrical fire", "location": "R305", "confidence": 0.94, "evidence": ["Smoke sensor triggered", "Infrared temperature > 80℃", "Audio detection of electric arc sound"], "context": {"time":"14:05", "occupancy":0, "nearby_hazard":"Hazardous Chemicals Cabinet A"} } During intelligent decision-making and strategy generation, the decision engine integrates the following information: Rule base matching: Complies with the emergency plan for "electrical fire in unmanned laboratory"; Contextual analysis: There are hazardous chemical containers nearby, so it is necessary to prevent the fire from spreading; Resource status: Gas fire suppression system is ready; nearest security personnel are on the 2nd floor.

[0041] Then, a handling strategy is generated, and the following content is executed immediately: Close the supply and exhaust valves of R305 (to prevent smoke diffusion); activate the heptafluoropropane gas fire extinguishing system (for electrical fires, it leaves no residue, no ablation marks, and no slag); lock the fire doors of R305 (the laboratory where the safety incident occurred) and the adjacent R304 and R306 laboratories; broadcast: "Fire extinguishing procedure activated in R305 area on the 3rd floor, do not approach." Execute with a 5-second delay to avoid system misjudgment; notify the on-duty engineer for remote video confirmation via mobile APP; dispatch security personnel from the 2nd floor with fire extinguishers to the 3rd floor to stand by.

[0042] During collaborative execution and feedback, instructions are automatically sent to each subsystem and executed within 1 second. The system continuously monitors the following results: at 14:05:10, the temperature of laboratory R305 drops to 40℃ and the smoke concentration decreases, indicating that the fire is under control. Subsequently, the security dispatch is automatically canceled, and a "the incident has been automatically handled" notification is sent to the engineer. Finally, the event log is stored in the knowledge base for optimizing future fire handling strategies for "unmanned laboratories".

[0043] Similarly, when intrusion occurs, images or video information at the park entrance can be collected via cameras to determine if there are any people at the entrance. Voice information of people at the park entrance can be collected via microphones to determine if there is any dispute between the intruder and security personnel. Access control card swipe records, meeting appointment systems, and visitor registration information can be used to determine if the people at the park entrance are legitimate entrants. After spatiotemporal alignment and multimodal Transformer alignment and fusion of the above information, the information is input into a dynamic strategy model for processing, and finally, corresponding disposal instructions are output, such as closing the gate and driving away the intruder via voice playback, while notifying the relevant security personnel to go to the park entrance for coordination.

[0044] When a person falls, images or videos of the fallen person can be captured by a camera to determine if anyone is present. Sound information, such as groans, screams, or the sound of the person hitting objects when falling, can be collected by a microphone. Bluetooth beacons can be used to check if there are any staff members nearby. After spatiotemporal alignment and multimodal Transformer alignment and fusion of the above information, it is input into a dynamic policy model for processing. Finally, corresponding handling instructions are output, such as notifying the nearest staff to go to the accident site for rescue, and simultaneously playing a voice message to comfort the fallen person.

[0045] When people gather, cameras can be used to collect images or video information of people to determine whether a large number of people are gathered in a certain place. Microphones can be used to collect sound information, such as loud shouting or emotionally agitated sounds. Bluetooth beacons can be used to check whether there are patrol personnel nearby. After the above information is spatiotemporally aligned and fused with multimodal Transformer alignment, it is input into a dynamic policy model for processing. Finally, corresponding disposal instructions are output, such as notifying patrol personnel to go to the gathering point to evacuate the crowd, and at the same time, voice playback can be used to calm and guide the crowd.

[0046] Of course, depending on the actual situation, the event ontology and rule base of the above-mentioned safety events can be flexibly expanded according to the safety needs of the park, such as adding the "hazardous chemical leakage" event type, etc., which will not be elaborated here.

[0047] Example 2 This invention also discloses a joint response system for public safety incidents in industrial parks. This system is used to implement the aforementioned joint response method for public safety incidents in industrial parks. The system includes: The multimodal sensing access module is used to collect data from various sensing devices in the park and initially align all data according to a unified spatiotemporal coordinate to obtain different modal features.

[0048] The cross-modal event understanding module is used to align and fuse different modal features using a multimodal Transformer, introduce event ontology to perform structured modeling of security events, and output structured event descriptions.

[0049] The context-aware decision-making module is used to reason about structured events through context awareness to classify risks and generate response strategies.

[0050] The collaborative execution and closed-loop feedback module is used to handle security incidents according to the handling strategy and provide feedback on the execution results.

[0051] In this embodiment, video modal data of the park is obtained by acquiring various video streams from visible light cameras and infrared cameras deployed within the park; auditory modal data of the park is obtained by capturing sound signals such as broken glass, cries for help, and explosions using a directional microphone array; environmental sensing modal data of the park is obtained by detecting environmental changes using smoke sensors, temperature sensors, CO sensors, and combustible gas sensors; personnel location modal data of the park is obtained by locating staff positions using UWB (Ultra-Wideband) and Bluetooth beacon technologies, enabling these personnel to respond to park security incidents; and business modal data of the park is obtained by acquiring visitor information through access control card swipe records, meeting reservation systems, and visitor registration information.

[0052] Based on the BIM model and NTP time synchronization, all data is initially aligned according to a unified spatiotemporal coordinate system. That is, using the BIM model as the spatial reference and NTP time synchronization as the time reference, data from different sources and of different types (such as video, sensor data, and personnel positioning data) are uniformly calibrated into the same time and space system, achieving preliminary spatiotemporal synchronization and laying the foundation for subsequent cross-modal data fusion and event analysis. In this embodiment, the BIM model is a digital 3D model of the park, equivalent to drawing a precise digital map of the park, marking the specific spatial coordinates (e.g., which building, floor, location, etc.) of each area and device (such as cameras, sensors, and Bluetooth beacons). Thus, the BIM model provides a spatial reference for all data, allowing each piece of data to correspond to a specific physical location. NTP is a precise time synchronization technology used to completely unify the time of all devices (such as cameras, sensors, and Bluetooth beacons), controlling the error to a very small range. For example, images captured by cameras, alarm signals from smoke sensors, and location data from personnel positioning are all tagged with a "unified timestamp," ensuring that they record events occurring at the same time. In this way, by initially aligning the data using unified spatiotemporal coordinates, all data (i.e., visual modal data, auditory modal data, environmental sensor modal data, location modal data, and business modal data) are calibrated according to the spatial coordinates of the BIM model and the unified time of NTP. For example, "smoke sensor alarm at 10:00 AM (NTP unified time), 2nd floor of Building 3 (BIM spatial coordinates)," "video footage at the same time and location," and "personnel location information at the same location at the same time" will be accurately matched, achieving initial alignment with consistent time and corresponding location. This allows the dispersed multimodal data to be unified in the same time and space before more accurate fusion analysis can be performed.

[0053] Multimodal Transformer is an intelligent algorithm framework. Its core is to use alignment and fusion—two key steps—to transform different types of data (or modalities), such as video, sensor data (e.g., smoke or temperature sensors), and personnel location data, into unified and valuable comprehensive features. This allows the system to make more accurate judgments and breaks down the barriers between individual subsystems. Multimodal refers to multiple types of data sources (e.g., in park security, video surveillance image data, fire system sensor data, and access control personnel information data). These data types and formats are different and originally could not be directly coordinated. Alignment allows data from different modalities to correspond. For example, it precisely matches "smoke sensor alarm (value) in a certain area" with "video footage (image) of the same area at the same time" and "personnel location (location) in the area" in time and space, ensuring that the three describe "an event at the same time and in the same area," avoiding data discrepancies. Then, through fusion, using Transformer... The algorithm integrates aligned data from different modalities (such as smoke sensor readings, video footage, and personnel locations) and outputs a comprehensive feature. For example, it can determine a real fire based on the combination of "smoke sensor readings exceeding the limit, video showing open flames but no personnel location" and "smoke sensor readings exceeding the limit, video showing laboratory smoke exhaust and personnel location" and determine it as a false alarm. This enables in-depth analysis of the nature of the event. In this way, various scattered data can "match each other and information can be merged," no longer operating independently. This allows the system to make accurate judgments by combining multiple aspects of information, just like a human, solving the problems of "single perception, high false alarm rate, and shallow event understanding."

[0054] Structured modeling includes defining event types, attributes, and causal relationships. Event types include fire, intrusion, falls, and gatherings. Attributes include the location, time, personnel involved, and severity level of the event. Causal relationships include the order in which multiple modalities corresponding to the same safety event occur. For example, in the case of a fire caused by a short circuit, the causal relationship is generally that the short circuit causes smoke and temperature rise, which in turn ignites a spark or open flame. In step S3, the structured event description includes event type, confidence level, key evidence (such as "flame area in video, explosion sound in audio, smoke alarm"), and preliminary risk level. For example, the output structured event description could be a small fire in a laboratory with a confidence level of 92%, indicating a medium risk as the camera in laboratory 302 detected a spark and the smoke alarm detected smoke exceeding a preset value.

[0055] The context-aware decision-making module, when reasoning about structured events through context awareness to determine risk levels and generate response strategies, first pre-sets a static rule base of national standards and park emergency plans. This static rule base can be based on the "GB 50116 Code for Design of Automatic Fire Alarm Systems" or other fire management standards that meet national standards. Secondly, it trains a dynamic strategy model based on the static rule base. This is done by feeding the model with information such as event descriptions, response instructions, and handling effects for fires or other fire safety incidents, using reinforcement learning to train and build the dynamic strategy model. Thirdly, it inputs structured event descriptions, current context, and resource availability into the dynamic strategy model, feeding the model information. The current context includes time, weather, personnel density, and the status of important areas; resource availability includes the location of security personnel and the status of fire-fighting equipment. Finally, based on the structured event descriptions, current context, and resource availability, it outputs a set of response instructions, including controlling preset equipment and sending instructions to preset security personnel. For example, the structured event input to the dynamic strategy model is described as a small fire in the laboratory (confidence level 92%); the current context is 23:00 at night, no personnel are present, and there is a hazardous chemical cabinet nearby; the set of instructions is: 1. Automatically shut off the ventilation in the area, 2. Activate the gas extinguishing system, 3. Lock the adjacent fire doors, 4. Notify the on-duty engineer for remote confirmation.

[0056] The collaborative execution and closed-loop feedback module handles safety incidents according to the response strategy and provides feedback on the execution results. This process includes: First, activating cold water sprinklers or non-combustible and non-flammable gas extinguishing through the fire protection subsystem; locking or opening specific passages through the access control subsystem to facilitate personnel evacuation and movement; playing customized evacuation messages through the broadcast subsystem to guide personnel to the appropriate emergency exits, such as "Please proceed to fire door No. 5 via the stairs for evacuation"; illuminating emergency paths through the lighting subsystem to ensure personnel can use emergency exits as needed, preventing falls, bumps, or stampedes; and dispatching orders to the nearest patrol personnel through the security dispatch subsystem to organize evacuation and intervene promptly at potential hazards, thus comprehensively addressing safety incidents from aspects such as fire control and personnel evacuation. Secondly, cameras capture images of the location where the security incident occurred; microphones capture sound signals such as broken glass, cries for help, or explosions; various sensors detect signals such as smoke, gas, and temperature at the location of the security incident to determine whether the changes in smoke, gas, and temperature caused by the incident have been controlled or resolved; preset personnel location detection determines whether relevant personnel have arrived at or left the location of the security incident; and access control systems or visitor registration systems track the entry and exit of relevant personnel into the park. For example, detecting whether "the temperature has dropped" or "whether personnel have left the danger zone" allows for obtaining the results or effects of the security incident handling, enabling park personnel to continue monitoring the progress of the incident. Finally, the received feedback signals are input into the dynamic strategy model and matched with the structured event description, current context, and handling instructions. Based on the feedback signals, the handling instructions corresponding to the structured event description and current context are optimized. For example, when the received feedback signal is that the temperature is decreasing but the rate of decrease is not meeting the preset requirements, the optimization plan can increase the ventilation at the location of the safety incident and increase the gas or liquid flow rate of the fire extinguishing system. When the received feedback signal is that personnel are moving but have not left the danger zone, the optimization plan can control more doors to be opened (to prevent personnel from being trapped in the fire and unable to reach the preset safety door) and increase the volume of the voice evacuation content, so as to continuously optimize the dynamic strategy model and improve the handling effect of safety incidents.

[0057] Implementing the method and system for the linkage disposal of public security incidents in the park based on multi-modal perception, through context awareness for structured event reasoning, multi-modal cross-verification is achieved, and the perception robustness of the security system is improved. Compared with the false alarm rate of more than 30% of the conventional single smoke sensor, the accuracy rate of fire recognition in this solution can be increased to more than 98%, significantly reducing the false alarm rate; it supports complex scene reasoning and can distinguish easily confused scenes such as "real fire" and "cooking fume", "malicious intrusion" and "employee tailing", improving the depth of event cognition; according to the event type and context, strategies are dynamically generated to avoid "making a fuss" or "underreacting", achieving precise hierarchical disposal and improving the intelligence of the linkage; there is no need for manual intervention from perception to execution, and the response speed is less than 10s, achieving a full-automatic closed-loop response; by constructing an intelligent security closed-loop of "perception - understanding - decision - execution - feedback", it realizes high-precision recognition of public security incidents, context-aware grading evaluation, and multi-system adaptive collaborative disposal, greatly improving the timeliness, accuracy, and effectiveness of the park's security response; after testing, in a typical park scenario, this solution shortens the average response time of security incidents from 3 - 5 minutes to within 15 seconds, and reduces the ineffective police dispatch caused by false alarms by 70%.

[0058] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0059] The above-described embodiments only express several implementation manners of the present invention, and the description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A method for coordinated handling of public safety incidents in a park based on multimodal perception, characterized in that, Includes the following steps: S1. Collect data from various sensing devices in the park and initially align all data according to a unified spatiotemporal coordinate to obtain different modal features; S2. Employ a multimodal Transformer to align and fuse features from different modalities; S3. Introduce an event ontology to perform structured modeling of security events and output a structured event description; S4. Use context awareness to reason about structured events to classify risks and generate response strategies; S5. Handle security incidents according to the aforementioned handling strategy and provide feedback on the execution results.

2. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, The data from various sensing devices include camera video streams, microphone audio streams, various sensor data streams that sense changes in the environment, preset personnel positioning data, and park business data.

3. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, In step S1, based on the BIM model and NTP time synchronization, all data are initially aligned according to a unified spatiotemporal coordinate system.

4. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, In step S3, structured modeling includes defining event types, attributes, and causal relationships. The event types include fire, intrusion, fall, and gathering. The attributes include location, time, involved personnel, and severity level. The causal relationships include the order in which multiple different modal characteristics corresponding to the same security event occur.

5. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, In step S3, the structured event description includes event type, confidence level, key evidence, and preliminary risk level.

6. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, In step S4, the step of reasoning about structured events through context awareness to determine risk levels and generate a response strategy includes: S41. A static rule base for pre-setting national standards and park emergency plans; S42. Train a dynamic policy model based on the static rule base; S43. Input the structured event description, current context, and resource availability into the dynamic strategy model; S44. Output a set of processing instructions based on the structured event description, the current context, and resource availability.

7. The method for joint handling of public safety incidents in industrial parks according to claim 6, characterized in that, The current context includes time, weather, population density, and status of important areas; the resource availability includes the location of security personnel and the status of fire-fighting equipment; and the set of disposal instructions includes controlling preset equipment and sending instructions to preset security personnel.

8. The method for joint handling of public safety incidents in industrial parks according to claim 1, characterized in that, In step S5, processing the security incident according to the handling strategy and providing feedback on the execution effect includes: S51. Distribute the processing instructions in the processing instruction set to each execution subsystem for collaborative execution; S52. Continuously monitor the processing effect through various sensing devices and generate feedback signals; S53. Optimize the dynamic strategy model based on the feedback signal.

9. The method for joint handling of public safety incidents in industrial parks according to claim 8, characterized in that, In step S51, the coordinated execution includes activating sprinkler or gas fire extinguishing, locking or opening specific passages, playing customized evacuation voice prompts, illuminating emergency routes, and dispatching orders to the nearest patrol personnel.

10. A park public safety incident joint response system, used to implement the park public safety incident joint response method according to any one of claims 1-9, characterized in that, include: The multimodal sensing access module is used to collect data from various sensing devices in the park and initially align all data according to a unified spatiotemporal coordinate to obtain different modal features. The cross-modal event understanding module is used to align and fuse different modal features using a multimodal Transformer, introduce event ontology to perform structured modeling of security events, and output structured event descriptions. The context-aware decision-making module is used to reason about structured events through context awareness to classify risks and generate response strategies. The collaborative execution and closed-loop feedback module is used to process security events according to the handling strategy and provide feedback on the execution effect.

Images