An intelligent security risk prediction method and system based on a multi-modal large model

Abstract

This invention provides an intelligent security risk prediction method and system based on a multimodal large model. The method includes the following steps: collecting video, audio, and sensor data, and outputting a unified event token sequence; generating natural language causal descriptions for the event token sequence, constructing an event causal graph with event tokens as nodes, causal descriptions as edges, and outputting logits as edge weights, and dynamically updating the graph structure; obtaining the prior probability of node risk using a Bayesian neural network, fusing the correlation information between nodes through a fusion graph neural network message passing mechanism, and using Do-Calculus intervention loss calibration to obtain the node risk probability; using a Continuous-Time Markov Network to infer future risk trends; outputting a decision report and executing a tiered response; and updating the end-to-end model under privacy protection through federated learning. The beneficial effects of this invention are: achieving cognitive-level risk identification and proactive intervention in complex scenarios; significantly reducing false alarm and false negative rates, and enabling early warning.

Description

Technical Field

[0001] This invention belongs to the field of security risk prediction technology, and in particular relates to an intelligent security risk prediction method and system based on a multimodal large model. Background Technology

[0002] With the widespread adoption of ultra-high-definition video surveillance, IoT sensors, and 5G / 6G communication technologies, the data generated by security systems has expanded from a single visual modality to multimodal heterogeneous big data encompassing vision, hearing, spatiotemporal sensing, and environmental physical quantities. Traditional security systems typically employ a linear "perception-rule-response" architecture, but it still suffers from the following drawbacks: The false alarm rate is high. Alarms are triggered based on shallow features or fixed thresholds, and the semantic difference between "birds flying by" and "people climbing over" at the pixel level cannot be distinguished. It has a high false negative rate and lacks contextual awareness of camouflaged, slow-changing, and collusive threats. Without causal cognition, events are simply superimposed using "AND" or "OR" logic, making it difficult to depict the multi-level causal chain of "drinking → arguing → carrying weapons → injury or death"; Lacking forward-looking capabilities, the system can only issue alerts for events that have already occurred, and lacks the ability to predict risk evolution on a "minute-hour" scale. Data silos, where video streams, audio streams, and IoT sensor streams are not interconnected at the protocol, timing, and semantic levels, lead to information loss.

[0003] In recent years, the Transformer architecture's large-scale foundation model has achieved breakthroughs in areas such as unified vision-language-speech modeling, chain-of-thought reasoning, and instruction tuning, providing a technological foundation for security systems to move from "perceptual intelligence" to "cognitive intelligence." However, current technologies remain at the level of "single-modal large-scale model + traditional backend rules," and have not yet revealed how to use large-scale models to perform causal modeling, risk quantification, iterative deduction, and generate explainable decisions for cross-modal events. Therefore, there is an urgent need for a systematic, engineered, and implementable "cognitive-level" security solution. Summary of the Invention

[0004] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes an intelligent security risk prediction method and system based on a multimodal large model.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: The first aspect of this invention provides an intelligent security risk prediction method based on a multimodal large model, comprising the following steps: S1. Collect video, audio, and sensor data, and output a unified event token sequence; S2. Generate natural language causal descriptions for the event token sequence, construct an event causal graph with event tokens as nodes and causal descriptions as edges, output logits as edge weights, and dynamically update the graph structure; S3. Use a Bayesian neural network to obtain the prior probability of node risk, fuse the correlation information between nodes through the fusion graph neural network message passing, and use Do-Calculus intervention loss calibration to obtain the node risk probability; S4. Use Continuous-Time Markov Network to extrapolate future risk scenarios; S5. Output a decision report and implement a tiered response; S6. Update the end-to-end model with privacy protection through federated learning.

[0006] Furthermore, in step S1, the event token includes modality source, semantic tag, time sequence, spatial coordinates, and confidence level.

[0007] Furthermore, the causal description of the event causal graph in step S2 includes the correlation logic between events.

[0008] Furthermore, step S4 is as follows: Define a four-level risk state space, namely safe, low risk, medium risk, and high risk. First, map the event causal graph nodes to the four-level risk state space. Then, learn the state transition strength matrix Qθ through Continuous-Time Markov Network parameterization. Iteratively solve the risk probability curve for the next 0-10 minutes based on Kolmogorov Forward Equation. The parameter θ is obtained by fine-tuning. Finally, output the risk probability confidence interval through a deep ensemble method.

[0009] Furthermore, the graded response in step S5 includes silent monitoring, pop-up notification, dispatching PTZ cameras and activating broadcast deterrence, and linking with external security platforms and locking the target trajectory. The trigger thresholds for each level of response are as follows: silent monitoring is executed when R < 0.3, pop-up notification is triggered when 0.3 ≤ R < 0.5, PTZ cameras are automatically dispatched and broadcast deterrence is activated when 0.5 ≤ R < 0.7, and external security platforms are linked and the target trajectory is locked when R ≥ 0.7.

[0010] A second aspect of this invention provides an intelligent security risk prediction system based on a multimodal large model, comprising: The multimodal sensing module is used to collect video, audio, and sensor data, and output a unified event token sequence. The causal hint and event causal graph construction module is used to generate natural language causal descriptions for event token sequences, construct an event causal graph with event tokens as nodes and causal descriptions as edges, output logits as edge weights, and dynamically update the graph structure. The risk quantification and probabilistic reasoning module is used to obtain the prior probability of node risk by using a Bayesian neural network, fuse the correlation information between nodes by fusing graph neural network message passing, and use Do-Calculus intervention loss calibration to obtain the node risk probability. The threat scenario simulation module is used to predict future risk situations using a Continuous-Time Markov Network; The interpretable decision and response module is used to output decision reports and execute tiered responses; The optimization module is used for end-to-end model updates with privacy protection through federated learning.

[0011] Furthermore, the multimodal perception module includes a visual subnet, an audio subnet, a sensing subnet, and a cross-modal alignment unit. The visual subnet uses Swin-Transformer V2 to extract spatiotemporal tokens, the audio subnet uses the BEATs pre-trained model to extract semantic tokens, the sensing subnet uses TimesNet combined with Transformer to process multidimensional temporal sensing data, and the cross-modal alignment unit generates a unified event token sequence through Cross-Modal Attention Pooling.

[0012] Compared with the prior art, the present invention has the following advantages: The intelligent security risk prediction method based on a multimodal large model described in this invention achieves cognitive-level risk identification and proactive intervention in complex scenarios through multimodal perception, large model causal prompting, event causal graph construction, GNN-Bayesian risk reasoning, Continuous-Time Markov Network situational inference, and interpretable decision response. This method significantly reduces the false alarm rate and false negative rate and enables early warning. It can be widely applied to scenarios such as urban squares, subway hubs, and important infrastructure, significantly improving the level of intelligent public safety. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the intelligent security risk prediction method based on a multimodal large model as described in Embodiment 1 of the present invention; Figure 2 The following is a flowchart of the intelligent security risk prediction method based on a multimodal large model as described in Embodiment 1 of the present invention; Figure 3 This is a flowchart of the event cause-effect graph construction process described in Embodiment 1 of the present invention. Detailed Implementation

[0014] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0015] In the description of this invention, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0016] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0017] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0018] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] Example 1 like Figure 1-3 As shown, an intelligent security risk prediction method based on a multimodal large model includes the following steps: S1. Multimodal data acquisition and event token extraction: Acquire video, audio and IoT sensor data, and output a unified event token sequence. Specifically, extract semantic tokens through each modal subnet, and then generate a unified event token sequence through cross-modal alignment. S2. The large model generates causal descriptions and constructs an event causal graph. The event token sequence is input into the large model through a cross-modal attention pooling alignment mechanism to generate natural language causal descriptions for the event token sequence. The event tokens are used as nodes and the causal descriptions are used as edges. The large model outputs logits as edge weights to construct the event causal graph. The graph structure is dynamically updated through an online incremental TemporalGraph Transformer with an update latency of less than 50ms. In this embodiment, the large model used is the DeepSeek-R1 large model developed by Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co., Ltd. or the QWQ-32B large model developed by Alibaba Group. S3. Based on GNN-Bayesian joint inference of node risk probabilities, the event causal graph obtained in step S2 is used as input data. A Bayesian neural network (BNN) is used to obtain the prior probability of node risk. The association information between nodes is fused through message passing in a fused graph neural network (GNN), and Do-Calculus (interventional calculus) is used to calibrate the intervention loss, thus obtaining the node risk probability. Specifically, the dynamic event causal graph constructed in step S2 (including event token nodes, causal description edges, and logits edge weights) is used as the core input data. A Bayesian neural network is used in conjunction with training data such as historical security risk logs and risk event samples from similar scenarios to model the probability distribution and output the node risk probabilities. The system effectively preserves the uncertainty information in risk prediction by identifying the prior probability of risk at each node. Furthermore, it introduces a message passing mechanism from a graph neural network, quantifying the correlation strength between nodes based on the edge weights of the event causal graph. This dynamically fuses the risk characteristics of adjacent nodes with the characteristics of the current node, achieving effective interaction and integration of cross-node risk information. Finally, addressing the issue of traditional probabilistic inference being susceptible to interference from data correlation, a Do-Calculus intervention loss function is constructed. By simulating virtual interventions on key causal paths (such as excluding the influence of "non-causally related nodes"), the system calibrates the deviation in node risk probability calculation, ultimately outputting accurate posterior node risk probabilities that balance causal logic and data correlation, providing a reliable quantitative basis for subsequent risk situation projection. S4. Using the node risk probability obtained in step S3 as the basic state data, the future risk situation is deduced using Continuous-Time Markov Network. S5. Based on the future risk situation results obtained in step S4, output an interpretable decision report and execute a tiered response. Calculate a risk score based on a regularized linear weighted decision function, match four-level response strategies, generate an interpretable decision report, and execute response actions; whereby the regularized linear weighted decision function is R=Σ i w i P(e i )+Σ ijw ij P(e i →e j ), where R is the final decision result, w i The first-order weight corresponds to a single risk event e. i The importance of P(e) i ) is event e i The probability of occurrence, w i P(e i ) represents the contribution of a single event to the decision; w ij The weights are second-order, corresponding to two risk events e. i e j The importance of the relationship between them, P(e i →e j ) for event e i The incident e j The probability, w ij P(e i →e j The weight w represents the contribution of the event association to the decision-making process. i and w ij It is obtained through regularization optimization based on security specifications, ensuring interpretability and compliance; S6. Online model optimization: Based on response feedback, the response execution status of step S5 is used as feedback data. The entire-link model is updated under privacy protection through federated learning. Counterfactual data augmentation is used to complete long-tail samples. Continuous learning is carried out using a rolling window to prevent catastrophic forgetting.

[0020] In step S1, the event token includes modality source, semantic label, time sequence, spatial coordinates, and confidence level.

[0021] In step S2, the causal description of the event causal graph includes the correlation logic between events, such as "people gathering ∧ loud arguments → risk of physical conflict ↑", and the Temporal Graph Transformer supports millisecond-level incremental updates to ensure the real-time performance of the graph structure.

[0022] Step S4 is as follows: Define a four-level risk state space, namely safe, low-risk, medium-risk, and high-risk. First, map the event causal graph nodes to the four-level risk state space. Then, parametrically learn the state transition strength matrix Qθ using a Continuous-Time Markov Network and iteratively solve it based on the Kolmogorov Forward Equation. π(t+Δt)=π(t)e^(QθΔt), outputting the risk probability curve for the next 0-10 minutes, where the parameter θ is obtained by fine-tuning the large model, and π(t) is the risk state probability distribution vector at time t, for example, corresponding to the state space {safe, low risk, medium risk, high risk}. π(t) = [0.8, 0.1, 0.08, 0.02] represents a "safe" probability of 80%, a "low-risk" probability of 10%, a "medium-risk" probability of 8%, and a "high-risk" probability of 2%. π(t+Δt) is the risk state probability distribution vector at time t+Δt. This formula represents the risk probability distribution π(t) at the current time t. After a time step Δt, the risk probability distribution π(t+Δt) at the next time t+Δt is obtained through the exponential operation of the state transition intensity matrix. The risk probability confidence interval is output through the deep ensemble method for decision-making reference.

[0023] Step S5 involves a tiered response, including silent monitoring, pop-up notifications, dispatching PTZ cameras and activating broadcast deterrence, and linking with external security platforms to lock the target's trajectory. The trigger thresholds for each level of response are as follows: silent monitoring is executed when R < 0.3; pop-up notifications are triggered when 0.3 ≤ R < 0.5; PTZ cameras are automatically dispatched and broadcast deterrence is activated when 0.5 ≤ R < 0.7; and external security platforms are linked and the target's trajectory is locked when R ≥ 0.7. Furthermore, a natural language report is automatically generated using a triplet format consisting of causal chains, evidence screenshots, and probability curves, and one-click PDF export is supported.

[0024] In step S6, federated learning only uploads gradient hashes at edge nodes, not the original video, to protect citizens' privacy.

[0025] Figure 2 This is a detailed execution flowchart based on the method, presenting a complete closed-loop process from data acquisition to model optimization. It can be divided into six core stages: data preprocessing, causal modeling, risk reasoning, situational analysis, decision response, and model optimization. Each stage is interconnected and contains detailed sub-steps. 1. Multimodal data acquisition and preprocessing First, data acquisition is performed to obtain heterogeneous data from multiple sources, such as video, audio, and IoT sensors; Then, preprocessing is performed to complete operations such as data denoising, normalization, and timestamp alignment, in preparation for subsequent token extraction.

[0026] 2. Event token extraction and cross-modal alignment For the preprocessed data, semantic tokens are extracted from each modal subnet (the visual subnet uses Swin-Transformer V2 to extract spatiotemporal tokens, the audio subnet uses BEATs to extract semantic tokens, and the sensor subnet uses TimesNet combined with Transformer to process temporal data). Then, through the Cross-Modal Attention Pooling mechanism of the cross-modal alignment unit, a unified event token sequence containing five tuples of "modality, semantics, temporality, space, and confidence" is generated.

[0027] 3. Causal Implication Engineering and Causal Description Generation Input the unified event token sequence into the large model and optimize the causal hints by combining it with the Causal Prompt template; The large model outputs natural language causal descriptions, such as causal statements with clear logical connections, like "people gathering ∧ loud arguments → risk of physical conflict ↑".

[0028] 4. Construction and dynamic updating of event cause-effect graph An initial event causal graph is constructed using event tokens as nodes, causal descriptions generated by the large model as edges, and logits output by the large model as edge weights. By leveraging Temporal GNN, millisecond-level incremental updates are achieved, ensuring that the graph structure can adapt to new event data in real time, with an update latency of less than 50ms.

[0029] 5. Risk Probability of GNN-Bayesian Joint Inference First, a Bayesian neural network is used in conjunction with historical log training data to obtain the prior probability of node risk. Then, the message passing mechanism of graph neural networks (GNN) is used to fuse the correlation information between nodes, and finally, the loss is calibrated by Do-Calculus intervention to obtain the accurate posterior probability of node risk.

[0030] 6. Continuous-Time Markov Network Situational Simulation Based on node risk probability as the fundamental state data, the core is to solve the formula. π(t+Δt)=π(t)e^(QθΔt), iteratively deduce the risk probability curve for the next 0-10 minutes; It also outputs a 95% confidence interval, providing a probability range reference for decision-making.

[0031] 7. Explainable decision-making and hierarchical response First, calculate the risk score R using a regularized linear weighted decision function. The function formula is as follows: R=Σ i wi P(e i )+Σ ij w ij P(e i →e j ); Then match the four-level response strategy (R<0.3 silent monitoring, 0.3≤R<0.5 pop-up prompt, 0.5≤R<0.7 dispatch PTZ camera + broadcast deterrence, R≥0.7 link external security platform + lock target trajectory); The final output is a triplet-based interpretable decision report containing causal chains, evidence screenshots, and probability curves, which triggers corresponding hardware / platform response actions.

[0032] 8. Response Feedback and Online Model Optimization Collect execution feedback data of response actions; Federated learning enables end-to-end model updates with privacy protection (edge ​​nodes only upload gradient hashes, not the original video), while counterfactual data is used to augment long-tail samples, and continuous learning through a rolling window avoids catastrophic forgetting.

[0033] Figure 3 This is a flowchart illustrating the construction and dynamic updating of an event cause-effect graph. It fully demonstrates the entire process from inputting multimodal event tokens to generating the final dynamic cause-effect graph, specifically including: 1. Input layer: Multimodal event token sequence The input token is shown as a specific example, containing two core event tokens. e1:<Visual, crowd gathering, 14:32, Plaza A, 0.92>, meaning that at 14:32, a crowd gathering was identified in Plaza A through visual modality, with a confidence level of 0.92; e2:<Audio + Visual, Loud Argument, 14:34, Square A, 0.88>, meaning that at 14:34, a loud argument was detected in Square A using audio and video dual-mode recognition, with a confidence level of 0.88.

[0034] 2. Intermediate Layer: Output of causal hints and causal descriptions for the large model. Input the Prompt command into the large model: "Analyze the causal relationships and strengths between events based on e1 and e2"; The large model outputs a causal description: "e1 (gathering) ∧ e2 (argument) → e3 (risk of physical conflict ↑), causal strength logits=0.91", and infers information about the newly added node e3: <Inference, 14:36-38, Square A, 0.83>, that is, it predicts that there is a risk of physical conflict in Square A from 14:36 ​​to 14:38, with a confidence level of 0.83.

[0035] 3. Construction Layer: Definition of Nodes and Edges in the Cause-Effect Graph Nodes: The nodes of the cause-effect graph are the existing events e1 and e2, and the inferred risk event e3; Edges and weights: Define two causal edges, e1→e3 and e2→e3, respectively. The weights of both edges are the causal strength logits value of 0.91 output by the large model, corresponding to the association strength of "aggregation → conflict risk" and "dispute → conflict risk".

[0036] 4. Update layer: Dynamically update the cause-effect graph. The Temporal Graph Transformer is called to incrementally update the initial causal graph, adding node e3 and two related edges, with the update delay controlled within 50ms. Simultaneously, the update rule is marked: the edge weights are recalculated every 100ms to ensure the timeliness of the causal graph; The final output is a dynamic event cause-effect graph in GraphML format, which can be directly used by downstream risk inference modules.

[0037] Example 2 An intelligent security risk prediction system based on a multimodal large model includes: The multimodal sensing module is used to collect video, audio, and sensor data, and output a unified event token sequence. The causal hint and event causal graph construction module is used to generate natural language causal descriptions for event token sequences, construct an event causal graph with event tokens as nodes and causal descriptions as edges, output logits as edge weights, and dynamically update the graph structure. The risk quantification and probabilistic reasoning module is used to obtain the prior probability of node risk by using a Bayesian neural network, fuse the correlation information between nodes by fusing graph neural network message passing, and use Do-Calculus intervention loss calibration to obtain the node risk probability. The threat scenario simulation module is used to predict future risk situations using a Continuous-Time Markov Network; The interpretable decision and response module is used to output decision reports and execute tiered responses; The optimization module is used for end-to-end model updates with privacy protection through federated learning.

[0038] The multimodal perception module includes a visual subnet, an audio subnet, a sensing subnet, and a cross-modal alignment unit. The visual subnet uses Swin-Transformer V2 to extract spatiotemporal tokens, the audio subnet uses the BEATs pre-trained model to extract semantic tokens, the sensing subnet uses TimesNet combined with Transformer to process multidimensional temporal sensing data, and the cross-modal alignment unit generates a unified event token sequence through Cross-Modal Attention Pooling.

[0039] Two examples are given below to illustrate the use of the present invention.

[0040] Example 1, deployed in Huigongmao Park.

[0041] Scenario characteristics: Targeting the four core needs of industrial and trade parks, namely "production safety management and control, material storage protection, personnel and vehicle access control, and fire hazard early warning", the scenario covers 16 production workshops, 8 storage warehouses, and 3 entrance and exit gates. It is characterized by dense equipment, complex personnel flow, storage of hazardous chemicals (such as paint and solvents), and high risks of mechanical operation. It is necessary to balance the continuity of production with the real-time nature of safety protection.

[0042] Hardware configuration: (1) Edge side: 32 NVIDIA Jetson Xavier NX edge computing boxes are deployed (suitable for distributed deployment in workshops / warehouses). Each box is connected to 8 4K industrial cameras (supporting dustproof and waterproof), 4 gas sensors (detecting combustible gas concentration), and 2 vibration sensors (monitoring equipment operating status); (2) Entrance and exit: 8 sets of multimodal access control all-in-one machines are configured, integrating face liveness detection, ID card chip reading, and gait recognition modules; (3) Cloud: A100×4 GPU cluster is adopted, equipped with a dedicated security computing power pool for the park, supporting lightweight deployment of large models with 1 billion parameters; (4) Security enhancement: the entire link is encrypted with the national cryptographic SM4, and the equipment operation and maintenance data is stored in the TEE trusted environment, which complies with industrial data security specifications.

[0043] Software adaptation measures: (1) Multimodal perception optimization: For workshop dust and strong light environment, the visual subnet (Swin-Transformer V2) is normalized for illumination and fine-tuned for noise suppression; the audio subnet (BEATs) adds a special feature library of "mechanical abnormal noise and valve leakage"; (2) Causal rule customization: Construct a special causal rule library for the park, such as "the temperature of the hazardous chemical warehouse exceeds the standard ∧ the gas concentration is abnormal → the risk of explosion will increase" and "not wearing a safety helmet ∧ approaching the stamping equipment → the risk of work injury will increase"; (3) Response strategy adaptation: Link the park PLC system. When "abnormal equipment vibration + operator violation" is detected, the equipment will be automatically stopped (response delay <150ms); the fire hazard warning of the storage area will be pushed to the fire sprinkler system at the same time.

[0044] Operational Results: The system operated without failure for 60 consecutive days, collecting a total of 327,000 multimodal data points and identifying 489 valid risk events, including 23 early warnings of equipment failures, 17 fire hazards, and 356 violations of regulations. The industrial park's work-related accident rate decreased by 82% year-on-year, the efficiency of handling hazardous chemical-related hazards increased by 70%, and there were zero incidents of theft of stored materials. The system passed the Level 1 certification for safety production standardization in industrial and commercial enterprises.

[0045] Example 2, deployed at border crossings and along the border control line.

[0046] Scenario characteristics: Covering a 20-kilometer border and one land port, the terrain includes complex landforms such as mountains, rivers, and grasslands. It faces risks such as "illegal border crossing, smuggling of contraband, and interference from severe weather (low temperature, sandstorms, and rainstorms)". It needs to achieve the security requirements of "full coverage, long-distance identification, all-weather operation, and rapid linkage", and areas without power supply need to rely on solar power.

[0047] Hardware configuration: (1) Perception terminal: 40 infrared thermal imaging cameras (detection distance ≥3km), 15 sets of millimeter-wave radar (covering areas without light), 8 tethered inspection drones (endurance ≥8 hours), and 30 vibration fiber optic sensors (buried in the border fence); (2) Edge computing: low-power edge server (solar power supply) is used, supporting wide temperature operation from -40℃ to 60℃, and a lightweight large model is deployed (parameters compressed to 280 million) to achieve local real-time identification; (3) Cloud linkage: access the cloud platform of the border control command center, and realize edge-cloud data transmission through 5G slicing network (latency <20ms), supporting cross-regional multi-site model collaboration; (4) Linkage equipment: equipped with drone countermeasure device, border patrol vehicle positioning terminal, and port intelligent inspection channel to form a "identification-tracking-interception" closed loop.

[0048] Software adaptation measures: (1) Multimodal fusion optimization: Infrared camera and millimeter-wave radar data are deeply aligned through Cross-Modal Attention Pooling to solve the problem of target recognition in sandstorm and foggy weather; the sensor subnet adds vibration feature recognition of "fence touch and vehicle run-over"; (2) Causal inference customization: construct border-specific causal chains such as "personnel loitering ∧ carrying large bags → smuggling risk ↑" and "nighttime infrared heat source movement ∧ no customs clearance record → illegal border crossing risk ↑", and adjust the risk weights in combination with terrain data (such as rivers and mountain passes); (3) Privacy protection: adopt federated learning to realize cross-port model updates, and edge nodes only upload risk feature gradients and do not upload original monitoring videos, which meets the border data confidentiality requirements.

[0049] Operational results: The system maintained an accuracy rate of over 93% under complex weather conditions such as high temperatures, sandstorms, and heavy rain. It issued warnings for 37 illegal border crossing risks and 12 signs of smuggling of contraband, with an average advance warning time of 5.8 minutes, providing sufficient interception time for border patrol teams. The efficiency of port inspections increased by 45%, and the false detection rate dropped from 18% in the traditional system to 2.7%. It successfully assisted in the seizure of 23 batches of various contraband, achieving "zero missed reports and low false reports" in border control.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent security risk prediction based on a multimodal large model, characterized in that, Includes the following steps: S1. Collect video, audio, and sensor data, and output a unified event token sequence; S2. Generate natural language causal descriptions for the event token sequence, construct an event causal graph with event tokens as nodes and causal descriptions as edges, and use the logits output by the large model as edge weights, and dynamically update the graph structure. S3. Use a Bayesian neural network to obtain the prior probability of node risk, fuse the correlation information between nodes through the fusion graph neural network message passing, and use Do-Calculus intervention loss calibration to obtain the node risk probability; S4. Use Continuous-Time Markov Network to extrapolate future risk scenarios; S5. Output a decision report and implement a tiered response; S6. Update the end-to-end model with privacy protection through federated learning.

2. The intelligent security risk prediction method based on a multimodal large model according to claim 1, characterized in that: In step S1, the event token includes modality source, semantic label, time sequence, spatial coordinates, and confidence level.

3. The intelligent security risk prediction method based on a multimodal large model according to claim 1, characterized in that: The causal description of the event causal graph in step S2 includes the relationship logic between events.

4. The intelligent security risk prediction method based on a multimodal large model according to claim 1, characterized in that, Step S4 is as follows: Define a four-level risk state space, namely safe, low risk, medium risk, and high risk. First, map the event causal graph nodes to the four-level risk state space. Then, learn the state transition intensity matrix Qθ through Continuous-Time Markov Network parameterization. Iteratively solve the risk probability curve for the next 0-10 minutes based on Kolmogorov Forward Equation. The parameter θ is obtained through fine-tuning. Finally, output the risk probability confidence interval through a deep ensemble method.

5. The intelligent security risk prediction method based on a multimodal large model according to claim 1, characterized in that: The graded response in step S5 includes silent monitoring, pop-up notification, dispatching PTZ cameras and activating broadcast deterrence, and linking with external security platforms and locking the target trajectory. The trigger thresholds for each level of response are as follows: silent monitoring is performed when R < 0.3, pop-up notification is triggered when 0.3 ≤ R < 0.5, PTZ cameras are automatically dispatched and broadcast deterrence is activated when 0.5 ≤ R < 0.7, and external security platforms are linked and the target trajectory is locked when R ≥ 0.

7.

6. A smart security risk prediction system based on a multimodal large model, used to implement the smart security risk prediction method based on a multimodal large model as described in any one of claims 1-5, characterized in that, include: The multimodal sensing module is used to collect video, audio, and sensor data, and output a unified event token sequence. The causal hint and event causal graph construction module is used to generate natural language causal descriptions for event token sequences, construct an event causal graph with event tokens as nodes and causal descriptions as edges, output logits as edge weights, and dynamically update the graph structure. The risk quantification and probabilistic reasoning module is used to obtain the prior probability of node risk by using a Bayesian neural network, fuse the correlation information between nodes by fusing graph neural network message passing, and use Do-Calculus intervention loss calibration to obtain the node risk probability. The threat scenario simulation module is used to predict future risk situations using a Continuous-Time Markov Network; The interpretable decision and response module is used to output decision reports and execute tiered responses; The optimization module is used for end-to-end model updates with privacy protection through federated learning.

7. The intelligent security risk prediction system based on a multimodal large model according to claim 6, characterized in that: The multimodal perception module includes a visual subnet, an audio subnet, a sensing subnet, and a cross-modal alignment unit. The visual subnet uses Swin-Transformer V2 to extract spatiotemporal tokens, the audio subnet uses the BEATs pre-trained model to extract semantic tokens, the sensing subnet uses TimesNet combined with Transformer to process multidimensional temporal sensing data, and the cross-modal alignment unit generates a unified event token sequence through Cross-Modal Attention Pooling.

Images