Community intelligent security decision-making method based on multi-modal data

Abstract

The invention provides a community intelligent security decision-making method based on multi-modal data. The community intelligent security decision-making method comprises the following steps of (a) a multi-source heterogeneous data fusion stage, (b) a dynamic risk assessment stage, (c) a self-adaptive decision-making stage and (d) a co-processing stage. Compared with the prior art, the method has the following beneficial effects: through multi-dimensional technical innovation, the pain points of data isolation, response lag, high false alarm rate, difficult coordination and the like in traditional community security are systematically solved, and a technical closed loop from accurate perception to intelligent disposal is formed.

Description

Technical Field

[0001] This invention belongs to the field of intelligent decision-making for community security, and specifically relates to a method for intelligent decision-making for community security based on multimodal data. Background Technology

[0002] Current community security systems generally suffer from defects such as single perception dimension, lagging risk assessment, low decision-making efficiency, and insufficient cross-system collaboration. The core problems stem from the traditional technical architecture's insufficient ability to fuse multimodal data, lack of dynamic risk modeling, and rigid response mechanism.

[0003] Because existing systems mostly rely on a single video surveillance data source and lack multi-dimensional perception capabilities such as infrared, soundprint, and environmental parameters, the recognition rate of abnormal behavior is low (especially at night or in severe weather). This problem stems from the limitations of sensor selection and the lack of data fusion algorithms. Conventional solutions attempt to improve coverage by increasing the number of sensors, but this can lead to problems such as the accumulation of spatiotemporal alignment errors of multi-source data (typical error > 1.5 meters), feature semantic gap (difference between video and soundprint feature dimensions > 80%), and a surge in computational load (processing latency increases by 3.8 times when the amount of data increases by 5 times).

[0004] Meanwhile, risk assessment often uses static threshold methods (such as preset intrusion zones), which cannot dynamically adapt to spatiotemporal changes and complex event correlations. This is mainly due to the lack of four-dimensional risk modeling capabilities and knowledge graph reasoning mechanisms. Existing improvement methods attempt to introduce time series analysis or simple rule engines, but they face drawbacks such as high false alarm rates (false alarm rate >35% at night) and significant response delays (average time from event occurrence to alarm >8 seconds). Furthermore, resource scheduling relies on manual decision-making, resulting in low emergency response efficiency (average response time >5 minutes). Although some systems have introduced path planning algorithms, the actual scheduling success rate is less than 70% because they do not consider dynamic obstacles (such as crowd flow) and multi-objective optimization (multi-dimensional conflicts such as time, cost, and security).

[0005] In addition, cross-community joint defense suffers from serious data silo problems. Traditional solutions achieve information sharing through centralized data pools, but face challenges such as privacy leakage risks (data transmission in plaintext accounts for more than 90%), poor model generalization ability (cross-community identification accuracy drops by more than 40%), and high collaboration latency (cross-system command transmission time is more than 15 seconds).

[0006] Existing security mechanisms mostly employ passive defense measures such as periodic key updates. Under the threat of quantum computing (such as Shor's algorithm which can crack 2048-bit RSA keys), traditional encryption systems face the risk of systemic failure. Although some schemes attempt to increase the key length (such as upgrading to 3072 bits), this leads to a surge in storage overhead (key size increases by 2.5 times) and a decrease in processing speed (signature verification time increases by 3 times).

[0007] These shortcomings collectively lead to bottlenecks in existing community security systems, such as large perception blind spots, high response delays, weak joint defense capabilities, and prominent security risks. There is an urgent need to build a new generation of technology system that integrates multimodal functions, real-time risk assessment, intelligent decision optimization, and cross-domain collaboration. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a community intelligent security decision-making method based on multimodal data, thereby solving the problems mentioned in the background section.

[0009] This invention is achieved through the following technical solution: a community intelligent security decision-making method based on multimodal data, comprising the following steps:

[0010] (a) Multi-source heterogeneous data fusion stage:

[0011] a1 collects three-dimensional spatial data of the community through a multimodal sensor array. The three-dimensional spatial data includes video, infrared, acoustic fingerprint, temperature and humidity data.

[0012] a2, Construct a unified spatiotemporal coordinate system to achieve heterogeneous data alignment;

[0013] a3 uses an attention mechanism to fuse feature vectors from multiple sources;

[0014] (b) Dynamic Risk Assessment Phase:

[0015] b1. Establish a four-dimensional risk assessment model, which includes time, space, event type, and personnel density.

[0016] b2, Constructing an abnormal behavior knowledge graph based on transfer learning;

[0017] b3, calculates regional risk index and generates heat map in real time;

[0018] (c) Adaptive decision-making phase:

[0019] c1, Deploy a mixed-integer programming model to optimize security resource scheduling.

[0020] c2 generates a multi-level response plan, which includes early warning, response, and post-disaster management.

[0021] c3, verify the feasibility of the decision-making scheme through a digital twin system;

[0022] (d) Collaborative handling phase:

[0023] d1, Achieving cross-community data sharing based on federated learning;

[0024] d2, Construct a multi-agent collaborative operation framework;

[0025] d3, key data in the data processing process is stored and processed through blockchain.

[0026] In a preferred embodiment, the multimodal sensor array includes:

[0027] The panoramic camera features both visible light and infrared dual-spectrum imaging capabilities; the MEMS acoustic sensor array operates within a frequency range of 20Hz-20kHz; the millimeter-wave radar boasts a detection accuracy of ±0.1m; and the environmental monitoring terminal is used to detect PM2.5, temperature, humidity, and harmful gas data.

[0028] As a preferred embodiment, in step (a) the multi-source heterogeneous data fusion stage, specifically:

[0029] Video data is transmitted to edge nodes via H.265 encoding and compression using dual-spectrum panoramic cameras to capture high-definition images in the visible light range and infrared sensors to detect human body heat radiation. Voiceprint data is collected from 20Hz-20kHz frequency bands using MEMS microphone arrays, and beamforming technology is used to locate the direction of the sound source and extract MFCC features. Environmental data is obtained by detecting target distance and speed using millimeter-wave radar, and environmental sensors monitor PM2.5, temperature, humidity, and VOCs in real time.

[0030] Using the community's 3D electronic map as the reference coordinate system, a coordinate system is constructed, and the data is aligned using a rotation matrix R and a translation vector T:

[0031] Where R is a 3×3 rotation matrix used to correct for differences in sensor viewing angles (such as the installation angle deviation between the camera and the radar).

[0032] T: 3×1 translation vector, used to compensate for the physical position offset between sensors (unit: meters).

[0033] Target coordinates in the original sensor coordinate system.

[0034] Coordinates after alignment to a unified spatial coordinate system.

[0035] We used an improved YOLOv7 model to extract video target features, a 1D-CNN classification network to extract MFCC features from speaker prints, and designed a cross-modal attention mechanism.

[0036]

[0037] in: The query vector for the i-th modality;

[0038] K j : The key vector of the j-th mode;

[0039] A ij The association weights between modal i and j are used for weighted fusion of multi-source features;

[0040] Weighted fusion generates 256-dimensional joint features, resolving semantic biases in multi-source data.

[0041] As a preferred implementation, in step (b) of the dynamic risk assessment stage, specifically:

[0042] Construct a four-dimensional model: R = 0.3T + 0.4S + 0.2E + 0.1D

[0043] in:

[0044] Time factor T: Nighttime (20:00-6:00) coefficient × 1.5, Holidays × 1.3;

[0045] Spatial factor S: Calculates the topological distance d from the target to the power distribution room / entrance, mapped as S = 1 / (1 + ... 5 e -0.5 d);

[0046] Event Factor E: Identifies 32 types of events through transfer learning using a ResNet-50 domain adaptation network, outputting probability P. e E = 1.5P e ;

[0047] Density factor D: Population density grading based on YOLO+DeepSORT:

[0048]

[0049] Combining a knowledge graph with 2000+ nodes (stored in Neo4j), a path ranking algorithm is used to mine risk chains, and a 1m grid heatmap is generated through inverse distance weighted interpolation.

[0050]

[0051] in:

[0052] R i Risk values ​​of the four nearest grid points;

[0053] d: The Euclidean distance from the target point (x,y) to the i-th grid point;

[0054] Spatial smooth interpolation of risk values ​​is achieved by using the inverse square of the distance as a weighting factor.

[0055] As a preferred implementation, step (c), the adaptive decision-making stage, specifically includes:

[0056] c1, Establish a mixed-integer programming model:

[0057] min(0.6∑t j +0.4∑c i x ij )

[0058] Where: t j The time when security personnel arrive at area j;

[0059] c i The unit time cost of security guard i;

[0060] x ij A binary variable (0 or 1) indicates whether personnel i is assigned to region j;

[0061] The constraint condition is: Skill matching: ∑x ij s ik ≥r jk , where s ij For personnel i skills k,r jk (For regional needs)

[0062] Response time limit: t j ≤5min (Level 2 Warning)

[0063] c2, develop a three-level response mechanism:

[0064] c21. Warning: Adjust the camera to 60fps and issue a broadcast alert (sound pressure ≥ 75dB).

[0065] c22. Solution: Improve the A* algorithm for path planning:

[0066] f(n) = g(n) + h(n) + 0.3P obs (n)

[0067] Where: g(n): the actual path cost from the starting point to node n.

[0068] h(n): The estimated cost (Manhattan distance) from node n to the destination.

[0069] P obs (n): The probability that there is a dynamic obstacle at node n (predicted from historical data).

[0070] $0.3$: Risk aversion coefficient, balancing path length and safety.

[0071] c23. Aftermath: Automatically generate reports and verify them through Monte Carlo simulation using digital twins to ensure a success rate of >92%.

[0072] As a preferred implementation, in the (d) collaborative processing phase, specifically:

[0073] d1, Constructing a Federated Learning Framework:

[0074] Gradient-encrypted transmission: Paillier homomorphic encryption is used.

[0075] Model aggregation:

[0076]

[0077] Where: w k : The weight parameters of the local model of the Kth community.

[0078] n k : The amount of data in the Kth community.

[0079] n: Total data volume (n = ∑n) k ).

[0080] Federated model fusion is achieved through weighted averaging, protecting data privacy.

[0081] d2, Design a multi-agent collaborative system (patrol robot + drone)

[0082] d3, the blockchain notarization uses the PBFT consensus.

[0083] After adopting the above technical solution, the beneficial effects of the present invention are as follows: Compared with the prior art, the present invention achieves breakthrough improvements in the following five dimensions:

[0084] Multimodal perception leap: By fusing visible light / infrared dual spectra (traditional solutions only use single-spectrum video analysis), and combining millimeter-wave radar point clouds and acoustic MFCC features (traditional solutions only rely on video), a cross-modal attention fusion mechanism is constructed (the false alarm rate is reduced from 32.7% to 6.3% in the traditional solution), achieving an intrusion detection accuracy of >95% (traditional solutions ≤80%).

[0085] Risk perception is proactive: An innovative four-dimensional dynamic model (adding a time factor, nighttime risk weight of 1.5, and spatial topological distance calculation) combined with a knowledge graph reasoning of 2000+ nodes (the traditional static rule base has only 200 rules) improves the risk prediction accuracy from 50% to 93.2%, and provides early warnings 3-5 minutes in advance.

[0086] Innovation in decision-making efficiency: The mixed integer programming model (MILP) is used to optimize resource scheduling (traditional manual scheduling takes more than 5 minutes, while this solution takes ≤120 seconds). Combined with the improved A* algorithm to introduce an obstacle probability term (path planning efficiency is improved by 35%), the success rate of handling is 94.7% (traditional manual scheduling ≤70%).

[0087] System openness and collaboration: Privacy-preserving model sharing based on federated learning (traditional cross-community data cannot be shared) increases the iteration speed of anomaly detection model by 3 times. At the same time, blockchain notarization ensures full traceability (notarization and verification time is <2 seconds, while traditional centralized systems take >10 seconds).

[0088] Three-dimensional defense system: Integrating Beidou elevation positioning (traditional two-dimensional geofencing is vulnerable to drone relay attacks) and environmental sensing data (traditional systems ignore the impact of temperature and humidity on sensor performance), a four-dimensional spatiotemporal verification cube is constructed, improving the ability to defend against positioning deception attacks by 40 times (false alarm rate reduced from 1.2 times / hour to 0.03 times / hour). Detailed Implementation

[0089] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0090] This invention provides a technical solution: a community intelligent security decision-making method based on multimodal data, comprising the following steps:

[0091] (a) Multi-source heterogeneous data fusion stage:

[0092] a1 collects three-dimensional spatial data of the community through a multimodal sensor array. The three-dimensional spatial data includes video, infrared, acoustic fingerprint, temperature and humidity data.

[0093] a2, Construct a unified spatiotemporal coordinate system to achieve heterogeneous data alignment;

[0094] a3 uses an attention mechanism to fuse feature vectors from multiple sources;

[0095] (b) Dynamic Risk Assessment Phase:

[0096] b1. Establish a four-dimensional risk assessment model, which includes time, space, event type, and personnel density.

[0097] b2, Constructing an abnormal behavior knowledge graph based on transfer learning;

[0098] b3, calculates regional risk index and generates heat map in real time;

[0099] (c) Adaptive decision-making phase:

[0100] c1, Deploy a mixed-integer programming model to optimize security resource scheduling.

[0101] c2 generates a multi-level response plan, which includes early warning, response, and post-disaster management.

[0102] c3, verify the feasibility of the decision-making scheme through a digital twin system;

[0103] (d) Collaborative handling phase:

[0104] d1, Achieving cross-community data sharing based on federated learning;

[0105] d2, Construct a multi-agent collaborative operation framework;

[0106] d3, key data in the data processing process is stored and processed through blockchain.

[0107] The multimodal sensor array includes:

[0108] The panoramic camera features both visible light and infrared dual-spectrum imaging capabilities; the MEMS acoustic sensor array operates within a frequency range of 20Hz-20kHz; the millimeter-wave radar boasts a detection accuracy of ±0.1m; and the environmental monitoring terminal is used to detect PM2.5, temperature, humidity, and harmful gas data.

[0109] In step (a), the multi-source heterogeneous data fusion stage, specifically:

[0110] Video data is transmitted to edge nodes via H.265 encoding and compression using dual-spectrum panoramic cameras to capture high-definition images in the visible light range and infrared sensors to detect human body heat radiation. Voiceprint data is collected from 20Hz-20kHz frequency bands using MEMS microphone arrays, and beamforming technology is used to locate the direction of the sound source and extract MFCC features. Environmental data is obtained by detecting target distance and speed using millimeter-wave radar, and environmental sensors monitor PM2.5, temperature, humidity, and VOCs in real time.

[0111] Using the community's 3D electronic map as the reference coordinate system, a coordinate system is constructed, and the data is aligned using a rotation matrix R and a translation vector T:

[0112] Where R is a 3×3 rotation matrix used to correct for differences in sensor viewing angles (such as the installation angle deviation between the camera and the radar).

[0113] T: 3×1 translation vector, used to compensate for the physical position offset between sensors (unit: meters).

[0114] Target coordinates in the original sensor coordinate system.

[0115] Coordinates after alignment to a unified spatial coordinate system.

[0116] We used an improved YOLOv7 model to extract video target features, a 1D-CNN classification network to extract MFCC features from speaker prints, and designed a cross-modal attention mechanism.

[0117]

[0118] in: The query vector for the i-th modality;

[0119] K j : The key vector of the j-th mode;

[0120] A ij The association weights between modal i and j are used for weighted fusion of multi-source features;

[0121] Weighted fusion generates 256-dimensional joint features, resolving semantic biases in multi-source data.

[0122] In step (b) of the dynamic risk assessment phase, specifically:

[0123] Construct a four-dimensional model: R = 0.3T + 0.4S + 0.2E + 0.1D

[0124] in:

[0125] Time factor T: Nighttime (20:00-6:00) coefficient × 1.5, Holidays × 1.3;

[0126] Spatial factor S: Calculates the topological distance d from the target to the power distribution room / entrance, mapped as S = 1 / (1 + ... 5 e -0.5 d);

[0127] Event Factor E: Identifies 32 types of events through transfer learning using a ResNet-50 domain adaptation network, outputting probability P. e E = 1.5P e ;

[0128] Density factor D: Population density grading based on YOLO+DeepSORT:

[0129]

[0130] Combining a knowledge graph with 2000+ nodes (stored in Neo4j), a path ranking algorithm is used to mine risk chains, and a 1m grid heatmap is generated through inverse distance weighted interpolation.

[0131]

[0132] in:

[0133] R i Risk values ​​of the four nearest grid points;

[0134] d: The Euclidean distance from the target point (x,y) to the i-th grid point;

[0135] Spatial smooth interpolation of risk values ​​is achieved by using the inverse square of the distance as a weighting factor.

[0136] Step (c) Adaptive decision-making phase, specifically:

[0137] c1, Establish a mixed-integer programming model:

[0138] min(0.6∑t j +0.4∑c i x ij )

[0139] Where: t j The time when security personnel arrive at area j;

[0140] c i The unit time cost of security guard i;

[0141] x ij A binary variable (0 or 1) indicates whether personnel i is assigned to region j;

[0142] The constraint condition is: Skill matching: ∑x ij s ik ≥r jk , where s ik For personnel i skills k,r jk (For regional needs)

[0143] Response time limit: t j ≤5min (Level 2 Warning)

[0144] c2, develop a three-level response mechanism:

[0145] c21. Warning: Adjust the camera to 60fps and issue a broadcast alert (sound pressure ≥ 75dB).

[0146] c22. Solution: Improve the A* algorithm for path planning:

[0147] f(n) = g(n) + h(n) + 0.3P obs (n)

[0148] Where: g(n): the actual path cost from the starting point to node n.

[0149] h(n): The estimated cost (Manhattan distance) from node n to the destination.

[0150] P obs (n): The probability that there is a dynamic obstacle at node n (predicted from historical data).

[0151] $0.3$: Risk aversion coefficient, balancing path length and safety.

[0152] c23. Aftermath: Automatically generate reports and verify them through Monte Carlo simulation using digital twins to ensure a success rate of >92%.

[0153] In the (d) collaborative handling phase, specifically:

[0154] d1, Constructing a Federated Learning Framework:

[0155] Gradient-encrypted transmission: Paillier homomorphic encryption is used.

[0156] Model aggregation:

[0157]

[0158] Where: w k : The weight parameters of the local model of the Kth community.

[0159] n k : The amount of data in the Kth community.

[0160] n: Total data volume (n = ∑n) k ).

[0161] Federated model fusion is achieved through weighted averaging, protecting data privacy.

[0162] d2, Design a multi-agent collaborative system (patrol robot + drone)

[0163] d3, the blockchain notarization uses the PBFT consensus.

[0164] As an embodiment of the present invention:

[0165] Example 1,

[0166] If the smart community perimeter security system detects an unauthorized intrusion at night.

[0167] Step (a) Multi-source heterogeneous data fusion

[0168] Hardware Deployment: 1. Visible / infrared dual-spectrum cameras: deployed along the community fence (50-meter interval); 2. Millimeter-wave radar array: installed at a high point (detection radius 200 meters); 3. Distributed acoustic signature sensors: arranged in the green belt (20-meter interval); 4. Environmental monitoring terminal: collects temperature, humidity, and wind speed data in real time.

[0169] Data fusion process:

[0170] 1. Spatiotemporal Alignment: The BeiDou positioning module provides a unified coordinate system (116.4034°E, 39.9155°N) for all devices, synchronizes the clocks of each device via a time server (error < 1 millisecond), and aligns the data using a coordinate transformation matrix.

[0171]

[0172] 2. Feature extraction and fusion:

[0173] Video analysis: Human target detected climbing over fence (92% confidence); Radar tracking: Target trajectory captured (speed 2.3 m / s, direction northwest); Voiceprint recognition: Analysis of climbing metallic sound characteristics (frequency band 3-5 kHz, matching degree 87%); Environmental parameters: Temperature 25℃, wind speed 2.5 m / s (affecting sound propagation); Cross-modal attention mechanism calculates feature weights.

[0174] A video =0.62,A radar =0.28,A audio =0.10

[0175] Generate the fused feature vector: [0.92×0.62,2.3×0.28,0.87×0.10]=[0.570,0.644,0.087].

[0176] Step (b) Dynamic Risk Assessment

[0177] Risk assessment model calculation: 1. Factor calculation:

[0178] Time factor T: 22:30 (nighttime coefficient × 1.5) → T = 0.85

[0179] Spatial factor S: Intrusion point 80 meters from the power distribution room →

[0180] Event Factor E: Identified as "Unauthorized Intrusion" (Preset Risk Value 0.95)

[0181] Density factor D: Surrounding population density 0.8 people / m² → D = 0.52. Overall risk value:

[0182] R = 0.3 × 0.85 + 0.4 × 0.98 + 0.2 × 0.95 + 0.1 × 0.5 = 0.887 (exceeds the level 2 warning threshold of 0.7)

[0183] 3. Knowledge Graph Reasoning:

[0184] Matching historical cases to the risk chain of "nighttime intrusion → equipment theft → power supply disruption"

[0185] A risk heat map was generated, with high-risk areas covering the power distribution room and a 30-meter radius around it.

[0186] Step (c) Adaptive decision-making:

[0187] Resource scheduling optimization:

[0188] 1. Mixed-integer programming model: Decision variables: Dispatch patrol robot (ID:R2), security personnel (ID:G5)

[0189] Objective function:

[0190] min(0.6×(120s)+0.4×(150 yuan))=72+60=132

[0191] (R2 arrival time: 120 seconds; G5 labor cost: 150 yuan)

[0192] 2. Level 3 response execution:

[0193] Warning: Perimeter glare warning triggered (10,000 lumens), warning voice broadcast.

[0194] Solution: Optimal path for robot R2:

[0195] f(n) = 350m + estimated 120m + 0.3 × 0.15 = 470.045

[0196] Dispatch G5, equipped with riot control gear, to provide support from guard post number 3.

[0197] Aftermath: Automatically generate an event report (including 14 key time points).

[0198] Step (d) Collaborative processing

[0199] Cross-system collaboration:

[0200] 1. Federated learning model update: Encrypt the uploaded feature vector of this event (Paillier homomorphic encryption), update the intrusion detection model (accuracy improved to 93.8%).

[0201] 2. Multi-agent collaboration: Patrol robot R2 performs target tracking (UWB positioning error ±8cm), drone U3 takes off for monitoring (RTK positioning error ±2cm), transmits thermal images in real time, and intelligent access control automatically locks high-risk areas.

[0202] 3. Blockchain Evidence Storage: Generate evidence storage blocks.

[0203] Compared to existing technologies, this invention achieves breakthrough improvements in the following five dimensions:

[0204] Multimodal perception leap: By fusing visible light / infrared dual spectra (traditional solutions only use single-spectrum video analysis), and combining millimeter-wave radar point clouds and acoustic MFCC features (traditional solutions only rely on video), a cross-modal attention fusion mechanism is constructed (the false alarm rate is reduced from 32.7% to 6.3% in the traditional solution), achieving an intrusion detection accuracy of >95% (traditional solutions ≤80%).

[0205] Risk perception is proactive: An innovative four-dimensional dynamic model (adding a time factor, nighttime risk weight of 1.5, and spatial topological distance calculation) combined with a knowledge graph reasoning of 2000+ nodes (the traditional static rule base has only 200 rules) improves the risk prediction accuracy from 50% to 93.2%, and provides early warnings 3-5 minutes in advance.

[0206] Innovation in decision-making efficiency: The mixed integer programming model (MILP) is used to optimize resource scheduling (traditional manual scheduling takes more than 5 minutes, while this solution takes ≤120 seconds). Combined with the improved A* algorithm to introduce an obstacle probability term (path planning efficiency is improved by 35%), the success rate of handling is 94.7% (traditional manual scheduling ≤70%).

[0207] System openness and collaboration: Privacy-preserving model sharing based on federated learning (traditional cross-community data cannot be shared) increases the iteration speed of anomaly detection model by 3 times. At the same time, blockchain notarization ensures full traceability (notarization and verification time is <2 seconds, while traditional centralized systems take >10 seconds).

[0208] Three-dimensional defense system: Integrating Beidou elevation positioning (traditional two-dimensional geofencing is vulnerable to drone relay attacks) and environmental sensing data (traditional systems ignore the impact of temperature and humidity on sensor performance), a four-dimensional spatiotemporal verification cube is constructed, improving the ability to defend against positioning deception attacks by 40 times (false alarm rate reduced from 1.2 times / hour to 0.03 times / hour).

[0209] This invention systematically solves the pain points of traditional community security, such as data isolation, delayed response, high false alarm rate, and difficulty in coordination, through multi-dimensional technological innovation, forming a technological closed loop from accurate perception to intelligent handling.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A community intelligent security decision-making method based on multimodal data, characterized in that, Includes the following steps: (a) Multi-source heterogeneous data fusion stage: a1 collects three-dimensional spatial data of the community through a multimodal sensor array. The three-dimensional spatial data includes video, infrared, acoustic fingerprint, temperature and humidity data. a2, Construct a unified spatiotemporal coordinate system to achieve heterogeneous data alignment; a3 uses an attention mechanism to fuse feature vectors from multiple sources; (b) Dynamic Risk Assessment Phase: b1. Establish a four-dimensional risk assessment model, which includes time, space, event type, and personnel density. b2, Constructing an abnormal behavior knowledge graph based on transfer learning; b3, calculates regional risk index and generates heat map in real time; (c) Adaptive decision-making phase: c1, Deploy a mixed-integer programming model to optimize security resource scheduling. c2 generates a multi-level response plan, which includes early warning, response, and aftermath. c3, verify the feasibility of the decision-making scheme through a digital twin system; (d) Collaborative handling phase: d1, Achieving cross-community data sharing based on federated learning; d2, Construct a multi-agent collaborative operation framework; d3, key data in the data processing process is stored and processed through blockchain.

2. The community intelligent security decision-making method based on multimodal data as described in claim 1, characterized in that: The multimodal sensor array includes: The panoramic camera features both visible light and infrared dual-spectrum imaging capabilities; the MEMS acoustic sensor array operates within a frequency range of 20Hz-20kHz; the millimeter-wave radar boasts a detection accuracy of ±0.1m; and the environmental monitoring terminal is used to detect PM2.5, temperature, humidity, and harmful gas data.

3. The community intelligent security decision-making method based on multimodal data as described in claim 2, characterized in that: In step (a), the multi-source heterogeneous data fusion stage, specifically: Video data is transmitted to edge nodes via H.265 encoding and compression using dual-spectrum panoramic cameras to capture high-definition images in the visible light range and infrared sensors to detect human body heat radiation. Voiceprint data is collected from 20Hz-20kHz frequency bands using MEMS microphone arrays, and beamforming technology is used to locate the direction of the sound source and extract MFCC features. Environmental data is obtained by detecting target distance and speed using millimeter-wave radar, and environmental sensors monitor PM2.5, temperature, humidity, and VOCs in real time. Using the community's 3D electronic map as the reference coordinate system, a coordinate system is constructed, and the data is aligned using a rotation matrix R and a translation vector T: Where R is a 3×3 rotation matrix used to correct for differences in sensor viewing angles (such as the installation angle deviation between the camera and the radar). T: 3×1 translation vector, used to compensate for the physical positional offset between sensors (unit: meters). Target coordinates in the original sensor coordinate system. Coordinates after alignment to a unified spatial coordinate system. We used an improved YOLOv7 model to extract video target features, a 1D-CNN classification network to extract MFCC features from speaker prints, and designed a cross-modal attention mechanism. in: The query vector for the i-th modality; K j : The key vector of the j-th mode; A ij The association weights between modal i and j are used for weighted fusion of multi-source features; Weighted fusion generates 256-dimensional joint features to resolve semantic biases in multi-source data.

4. The community intelligent security decision-making method based on multimodal data as described in claim 1, characterized in that: In step (b) of the dynamic risk assessment phase, specifically: Construct a four-dimensional model: R = 0.3T + 0.4S + 0.2E + 0.1D in: Time factor T: Nighttime (20:00-6:00) coefficient × 1.5, Holidays × 1.3; Spatial factor S: Calculates the topological distance d from the target to the power distribution room / entrance, mapped as S = 1 / (1 + ... 5 e -0.5 d); Event Factor E: Identifies 32 types of events through transfer learning using a ResNet-50 domain adaptation network, outputting probability P. e E = 1.5P e ; Density factor D: Population density grading based on YOLO+DeepSORT: Combining a knowledge graph with 2000+ nodes (stored in Neo4j), a path ranking algorithm is used to mine risk chains, and a 1m grid heatmap is generated through inverse distance weighted interpolation. in: R i Risk values ​​of the four nearest grid points; d: The Euclidean distance from the target point (x,y) to the i-th grid point; Spatial smooth interpolation of risk values ​​is achieved by using the inverse square of the distance as a weighting factor.

5. The community intelligent security decision-making method based on multimodal data as described in claim 1, characterized in that: Step (c) Adaptive decision-making phase, specifically: c1, Establish a mixed-integer programming model: min(0.6∑t j +0.4∑c i x ij ) Where: t j The time when security personnel arrive at area j; c i The unit time cost of security guard i; x ij A binary variable (0 or 1) indicates whether personnel i is assigned to region j; Where the constraint is: Skill matching: ∑x ij s ik ≥r jk , where s ik For personnel i skills k,r jk (For regional needs) Response time limit: t j ≤5min (Level 2 Warning) c2, develop a three-level response mechanism: c21. Warning: Adjust the camera to 60fps and issue a broadcast alert (sound pressure ≥ 75dB). c22. Solution: Improve the A* algorithm for path planning: f(n)=g(n)+h(n)+0.3P obs (n) Where: g(n): the actual path cost from the starting point to node n. h(n): The estimated cost (Manhattan distance) from node n to the destination. P obs (n): The probability that there is a dynamic obstacle at node n (predicted from historical data). $0.3$: Risk aversion coefficient, balancing path length and safety. c23. Aftermath: Automatically generate reports and verify them through Monte Carlo simulation using digital twins to ensure a success rate of >92%.

6. The community intelligent security decision-making method based on multimodal data as described in claim 1, characterized in that: In the (d) collaborative handling phase, specifically: d1, Constructing a Federated Learning Framework: Gradient-encrypted transmission: Paillier homomorphic encryption is used. Model aggregation: Where: w k : The weight parameters of the local model of the Kth community. n k : The amount of data in the Kth community. n: Total data volume (n = ∑n) k ). Federated model fusion is achieved through weighted averaging, protecting data privacy. d2, Design a multi-agent collaborative system (patrol robot + drone) d3, the blockchain notarization uses the PBFT consensus.

Images