Public safety situation prediction method and system based on multi-modal data fusion

Abstract

The application relates to the technical field of public safety monitoring, in particular to a public safety situation prediction method and system based on multi-modal data fusion. The method comprises the following steps: continuously monitoring a public place through video and performing non-susceptible discovery to obtain non-susceptible discovery data of the public place; performing multi-modal data feature extraction on the non-susceptible discovery data of the public place to obtain safety threat multi-modal feature data; performing situation assessment on the safety threat multi-modal feature data to obtain safety threat event level data, and performing event level division on the safety threat multi-modal feature data according to the safety threat event level data to obtain safety threat risk event data; performing automatic response control according to the safety threat risk event data to obtain response control early warning data, and performing multi-view angle assessment optimization according to the response control early warning data to obtain response control early warning optimization data. The application can predict potential safety situations and improves the level of public safety management.

Description

Technical Field

[0001] This invention relates to the field of public safety monitoring technology, and in particular to a public safety situation prediction method and system based on multimodal data fusion. Background Technology

[0002] With the rapid development of internet technology and mobile terminal products, video surveillance systems have been widely deployed in locations such as home security, public transportation, banks, ATM centers, city centers, airports, and public roads, playing a vital role in protecting public safety. Meanwhile, video surveillance technology is constantly evolving, from traditional analog surveillance and digital surveillance to the current video surveillance systems. As an important means of modern social security, the security situation awareness capabilities of video surveillance systems are crucial for crime prevention, ensuring public safety, and national security. However, current video surveillance systems face challenges such as difficulty in data acquisition, difficulty in assessing security situations, frequent security vulnerabilities, and difficulties in technology implementation. Therefore, how to effectively assess and predict public safety situations through video surveillance has become an urgent problem to be solved. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a public safety situation prediction method and system based on multimodal data fusion, thereby resolving at least one of the aforementioned technical issues.

[0004] This application provides a public safety situation prediction method based on multimodal data fusion, including the following steps:

[0005] Step S1: Conduct continuous video surveillance of public places and perform seamless detection to obtain seamless detection data of public places;

[0006] Step S2: Perform multimodal data feature extraction on the non-intrusive detection data in public places to obtain multimodal feature data of security threats;

[0007] Step S3: Perform situation assessment on the multimodal feature data of security threats to obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data;

[0008] Step S4: Perform automatic response control based on security threat risk event data to obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimized data, so as to assist in public safety situation prediction.

[0009] This invention utilizes an inductive enhancement algorithm to supplement and infer information loss in videos caused by environmental factors such as occlusion and insufficient lighting, thereby improving the accuracy of security incident identification. Simultaneously, a multimodal data fusion method enhances the system's adaptability and efficiency in changing environments. Furthermore, the system can predict potential security situations based on historical data and classify these situations into four levels (Level 1 to Level 4), designing corresponding response mechanisms for each level, further improving the level of public safety management.

[0010] Preferably, step S1 specifically includes:

[0011] Step S11: Capture a continuous video stream using a camera to obtain video stream matrix data;

[0012] Step S12: Extract abnormal behavior events based on the video stream matrix data to obtain abnormal behavior event data;

[0013] Step S13: Perform extrapolation and enhancement based on abnormal behavior event data to obtain data on seamless detection in public places.

[0014] This invention utilizes cameras to continuously capture video streams of public places, enabling uninterrupted real-time monitoring and ensuring the capture of every subtle change within the area. The real-time captured video stream matrix data rapidly provides the latest scene information, laying the foundation for abnormal behavior detection and response, and improving the overall system response speed and monitoring coverage. Extracting abnormal behavior events from the video stream matrix data allows for the rapid identification of potential security threats within the area. By focusing on abnormal event characteristics, the system can separate normal and abnormal activities, effectively avoiding false alarms and missed detections. The abnormal behavior event extraction module automatically identifies abnormal behavior characteristics such as gathering, running, and arguing, significantly improving the accuracy of threat detection and helping to identify security risks at the first opportunity. Inference enhancement can perform inference and completion when video data is obstructed or in poor lighting conditions, enhancing the completeness of event detection. Time-series prediction restores missing behavioral information, making the monitoring data more continuous and complete. Inference enhancement compensates for the incompleteness of abnormal behavior, enabling the system to accurately identify threat events even when monitoring is obstructed or data is missing, thereby improving monitoring reliability. Through non-intrusive detection technology, continuous monitoring is achieved without interfering with the activities of people in public places. Compared to traditional overt monitoring methods, non-intrusive detection reduces interference with people in the premises and is suitable for high-traffic or sensitive areas.

[0015] Preferably, the abnormal behavior event extraction in step S12 is performed using a preset abnormal behavior event identification model, and the specific steps for constructing the preset abnormal behavior event identification model are as follows:

[0016] Obtain historical video stream matrix data and historical abnormal behavior event tags;

[0017] Clustering calculations are performed on the historical video stream matrix data to obtain historical video stream matrix cluster data;

[0018] Balanced classification is performed based on the clustering data of the historical video stream matrix to obtain historical video stream test data and historical video stream verification data.

[0019] Preliminary convolution calculations are performed based on historical video stream test data to obtain preliminary video stream feature data;

[0020] Pooling and multi-layer self-attention calculations are performed on the initial video stream feature data to obtain video stream pooled data and video stream weighted feature data, respectively.

[0021] Fully connected video stream data is obtained by performing fully connected encoding calculations based on the weighted feature data and pooled data of the video stream.

[0022] Preliminary abnormal behavior event identification data is obtained by calculating the index based on the fully connected video stream data and mapping it through historical abnormal behavior event labels.

[0023] By using historical video stream verification data to iteratively train the preliminary abnormal behavior event identification data, an abnormal behavior event identification model is obtained.

[0024] In this invention, by acquiring historical video stream matrix data and abnormal behavior event labels, the model can learn a large number of known behavioral features and event types during the construction process. Supervised learning of abnormal behavior features using labeled data enables the model to more accurately identify and distinguish different types of abnormal behavior in practical applications, improving detection accuracy. The clustering calculation step groups the historical video stream data, enabling a better understanding of the video data's inherent structure and helping the model capture different categories of abnormal behavior features. The test and validation data generated through balanced classification are representative, helping to address the problem of imbalanced data distribution, allowing the model to perform better when handling minority-class abnormal behavior events, thus improving the model's generalization ability. The preliminary convolution calculation extracts the basic features of the video stream data, simplifying the original data into more representative feature data, thereby effectively reducing computational complexity.

[0025] Pooling further reduces the dimensionality of feature data, retaining important feature information and reducing model computation, thus improving training and prediction efficiency while maintaining accuracy. Multi-layer self-attention computation can capture long-term dependencies between video frames, helping the model identify anomalous events in complex behavioral scenarios. The self-attention mechanism helps the model dynamically focus on important information at different time points, especially giving higher weight to anomalous changes in certain keyframes, improving the model's sensitivity to subtle anomalous behaviors. The fully connected encoding layer integrates the weighted feature data and pooled data of the video stream, forming a higher-dimensional feature space, enabling the model to more comprehensively understand the features of anomalous behavioral events, enhancing the model's expressive power and ensuring accuracy in complex behavioral scenarios.

[0026] By performing exponential calculations on fully connected data, anomalous behavior feature data can be transformed into different probability distributions, making the model output more interpretable and continuous. Label mapping associates the calculation results with specific anomalous behavior types, enabling the model to identify clear anomalous behavior types. Iterative training on initial identification data using validation data progressively optimizes model parameters, improving recognition accuracy and model stability. This iterative training approach ensures more consistent model performance on both training and validation sets, enhancing robustness across different scenarios and reducing overfitting. The trained model can then run on real-time video streams, quickly identifying anomalous events and supporting automated public safety responses.

[0027] Preferably, the multi-layer self-attention calculation specifically involves:

[0028] Multi-scale hierarchical convolution and spatiotemporal feature segmentation are performed on the preliminary video stream feature data to obtain multi-scale video stream feature data;

[0029] Local multi-head hierarchical self-attention calculation and global multi-head hierarchical self-attention calculation are performed on the multi-scale feature data of the video stream to obtain the local self-attention feature data and the global self-attention feature data of the video stream, respectively.

[0030] The relative position matrix is ​​constructed by combining the local self-attention feature data and the global self-attention feature data of the video stream to obtain the multi-head self-attention feature matrix data of the video stream.

[0031] Multi-layer graphs are constructed based on the multi-head self-attention feature matrix data of the video stream to obtain multi-layer graph data of the video stream;

[0032] Global-local graph convolution is performed on the multi-layer graph data of the video stream to obtain the global-local graph convolution data of the video stream;

[0033] Multi-scale feature fusion is performed on the global-local graph convolutional data of the video stream to obtain weighted feature data of the video stream.

[0034] This invention employs multi-scale hierarchical convolution and spatiotemporal feature segmentation to extract features from video streams at different scales, capturing both subtle and large-scale behavioral changes. Local multi-head self-attention (MOA) captures detailed behavioral changes within the video stream, while global MOA captures overall behavioral trends. The combination of these two approaches enhances the model's sensitivity to both details and the overall picture. A relative position matrix incorporates positional information into the local and global MOA feature data of the video stream, enabling the model to capture behavioral changes at different times and spatial locations. Through position matrix construction, the model can combine specific temporal and spatial positional relationships during analysis, thereby achieving more precise localization and understanding of abnormal behavioral events. Multi-layer graph construction transforms the MOA feature matrix of the video stream into graph-structured data, allowing for the capture of long-term and short-term dependencies between nodes through graph convolution. Global and local graph convolution calculations within the multi-layer graph enable the model to simultaneously handle global dependencies and local differences in behavioral features. Global-local graph convolution operations enhance the dynamic feature representation in the video, enabling the model to accurately identify relatively stable abnormal behavioral features (such as long-term clustering) and transient, sudden behaviors (such as running and collisions). By weighted fusion of multi-scale features, the model can simultaneously analyze both large-scale collective behavior and small-scale individual behavior in videos, achieving a comprehensive understanding of anomalous behavior. Weighted feature data possesses strong expressiveness and discriminative power in anomalous behavior classification, contributing to more accurate anomalous behavior identification and response.

[0035] Preferably, the multi-layer graph construction specifically involves:

[0036] Hierarchical multi-scale graph deconstruction is performed based on the multi-head self-attention feature matrix data of the video stream to obtain the hierarchical multi-scale graph data of the video stream;

[0037] Cross-layer association nodes are constructed from the hierarchical multi-scale graph data of the video stream to obtain cross-layer association node data of the video stream.

[0038] Based on the cross-layer association node data of the video stream, association features are extracted to obtain cross-layer association feature data of the video stream;

[0039] Gaussian convolution kernel mapping is performed on the cross-layer correlation feature data of the video stream to obtain the Gaussian convolution kernel data of the video stream;

[0040] Based on the Gaussian convolution kernel data of the video stream, graph convolution processing is performed on the cross-layer related node data of the video stream to obtain the cross-layer convolutional layer data of the video stream;

[0041] Cross-layer transfer processing is performed on the video stream convolutional layer data to obtain multi-layer graph data of the video stream.

[0042] In this invention, hierarchical multi-scale graph deconstruction transforms the self-attention feature matrix of a video stream into graph structures at different levels, enabling the model to focus on different behavioral features in the video hierarchically and at different scales. Cross-layer association node construction introduces cross-layer association nodes into the hierarchical graph structure, allowing the model to express the relationship between short-term and long-term behaviors through data exchange between different layers. Through the extraction of association features from cross-layer association node data, the model can automatically extract the information dependencies between cross-layer nodes, forming graph-based spatiotemporal association features. Gaussian convolution kernel mapping, through smoothing and weighting, makes the distribution of cross-layer association feature data more uniform and continuous, ensuring a smoother feature extraction process in the graph structure. Graph convolution processing based on Gaussian convolution kernel data can not only capture global features in the graph structure but also identify local differential behavioral features of nodes. The transmission of cross-layer convolutional layer data allows the feature information after each graph convolution to continuously propagate and share across different layers, ensuring that the model maintains information consistency across all graph layers. Multi-layer graph data encompasses multi-dimensional spatiotemporal information, enabling the model to have spatiotemporal representation capabilities across layers and scales, enhancing the model's adaptability, and making it suitable for long-term monitoring and detection of various abnormal behaviors in complex environments.

[0043] Preferably, step S2 specifically includes:

[0044] Step S21: Extract dynamic perception features of visual crowds based on the non-intrusive detection data in public places to obtain dynamic perception feature data of visual crowds;

[0045] Step S22: Extract audio synchronization data based on the visual crowd dynamic perception feature data and video stream matrix data to obtain audio synchronization data;

[0046] Step S23: Perform audio emotion analysis based on the audio synchronization data to obtain audio emotion analysis data;

[0047] Step S24: Perform multimodal fusion of visual crowd dynamic perception feature data and audio emotion analysis data to obtain multimodal feature data of security threats.

[0048] This invention extracts dynamic crowd perception features from visual information in non-sensory detection data, accurately capturing behavioral characteristics such as crowd gathering, dispersal, and movement, and analyzing the dynamic changes of crowds in space and time. Audio synchronization data extracted from visual crowd dynamic perception features and video stream matrix data allows the model to establish temporal and scene consistency between visual and audio data. Emotional analysis based on audio synchronization data can capture emotional changes in the crowd (such as anxiety, panic, and anger), providing additional emotional information for judging abnormal events. Multimodal fusion of visual crowd dynamic features and audio emotional analysis data helps the model comprehensively analyze the features of video and audio, thereby more accurately judging potential threats in public places. Spatiotemporal correlation enables the model to identify behavioral patterns and emotional fluctuations across different time periods, establishing a stronger correlation between events occurring at a specific moment and the background environment, improving the ability to identify persistent threats. Multimodal fusion of visual and audio provides more refined event features, helping the model determine real threats in specific situations, such as sudden panic in crowds or abnormal individual behavior.

[0049] Preferably, the audio emotion analysis is processed through a preset audio emotion analysis model, and the construction of the audio emotion analysis model includes the following steps:

[0050] Acquire historical audio synchronization data and historical audio emotion tag data;

[0051] The historical audio frequency domain data is obtained by performing frequency domain conversion based on the historical audio synchronization data;

[0052] The historical audio frequency domain data is divided to obtain historical audio test data and historical audio verification data;

[0053] The initial state is constructed based on historical audio test data to obtain historical audio initial state data;

[0054] Based on the historical audio initial state data, the input layer and forgetting layer are calculated to obtain the audio input layer data and audio forgetting layer data, respectively.

[0055] Multi-head emotion candidate memory state processing is performed based on audio input layer data and audio forgetting layer data to obtain audio candidate memory state data.

[0056] The audio candidate memory state data is processed by the output layer to obtain the audio hidden state data;

[0057] By iteratively training the audio hidden state data using historical audio verification data and historical audio emotion label data, an audio emotion parsing model is obtained.

[0058] The processing of multi-headed emotion candidate memory states includes:

[0059] Based on the audio input layer data and the audio forgetting layer data, multi-dimensional candidate memory state processing is performed to obtain the initial audio candidate memory state data.

[0060] Multi-head emotion attention calculation is performed on the initial audio candidate memory state data to obtain audio multi-head emotion attention matrix data;

[0061] Emotional memory enhancement was performed on audio multi-head emotion attention matrix data to obtain audio enhanced memory state data;

[0062] The initial audio candidate memory state data is fed back to the memory state data by using the audio-enhanced memory state data, and the feedback memory state data is obtained.

[0063] Based on the feedback memory state data, multi-level candidate state aggregation is performed to obtain audio candidate memory state data.

[0064] This invention utilizes historical audio synchronization data and emotion tags to train the model on a large amount of labeled emotion data, enabling accurate emotion recognition. Frequency domain transformation extracts frequency components from the audio signal, capturing emotional information in different frequency bands, such as low-frequency tone and emotional fluctuations, and high-frequency tension and anxiety signals. Initial state construction based on test data provides the model with a preliminary emotional information baseline before input and forgetting calculations. Input layer and forgetting layer calculations control which emotional features need updating and which can be forgotten, establishing a dynamic management mechanism for changes in emotional state. Through multi-head emotion candidate memory state processing, the model can simultaneously focus on features of different emotional dimensions (such as anger, anxiety, calmness, etc.). The hidden state, as the model's output, reflects the overall emotional state of the audio, making emotion prediction clearer and more expressive, providing higher-quality data for emotion tag matching. Iterative training gradually improves the model's performance on the validation set, enhancing its adaptability to different environments and audio scenarios, and ensuring the reliability of emotion parsing.

[0065] Multi-dimensional candidate memory state processing decomposes audio input and forgetting layer data into multi-dimensional emotional features, enabling the model to analyze the expression of different emotions (such as anger, anxiety, and calmness) in audio signals. A multi-head emotion attention mechanism allows the model to focus on parts of candidate memory states with important emotional information, capturing subtle emotional features through different attention heads. Emotional memory enhancement focuses on enhancing emotional features in the multi-head emotion attention matrix, making the model more sensitive to abnormal emotional changes. A memory state feedback mechanism feeds the enhanced emotional state back into the initial candidate memory state data, achieving secondary optimization of emotional features. Aggregation operations allow the model to extract core emotional features from emotional data of different dimensions, avoiding loss of emotional information during transmission and improving the overall accuracy of emotion analysis. During emotional memory enhancement and feedback, the model can retain and strengthen high-intensity emotional information in emotional features, making it suitable for detecting intense emotional fluctuations. The cyclical processing of feedback memory state data constructs a feedback loop, enabling the model to dynamically adjust the expression of emotional features, making it more adaptable to changing emotional environments. The multi-level aggregated audio candidate memory state data retains multi-layered emotional feature information, enabling the output emotional state data to have rich emotional expression.

[0066] Preferably, step S3 specifically includes:

[0067] Step S31: Perform preliminary screening of security threat multimodal features based on the security threat multimodal feature data to obtain threat preliminary screening index data;

[0068] Step S32: Allocate risk weights based on the initial threat screening index data to obtain threat weight matrix data;

[0069] Step S33: Construct a contextualized risk model based on the threat weight matrix data to obtain a public safety contextualized risk model;

[0070] Step S34: Perform risk classification based on the public safety contextualized risk model to obtain security threat event level data;

[0071] Step S35: Determine the event level threshold based on the security threat event level data to obtain the event level determination table data;

[0072] Step S36: Classify the multimodal feature data of security threats into event levels based on the event level determination table data to obtain integrated event level data;

[0073] Step S37: Aggregate risk events from the event level integration data to obtain security threat risk event data.

[0074] In this invention, by initially screening multimodal feature data of security threats, the system can eliminate irrelevant or low-risk features, focusing on the key features constituting security threats. Risk weights are assigned based on the initial screening results; by assigning weights to different features, the system can distinguish and quantify different types of threats. A contextualized risk model constructed based on the weight matrix data links threat features to actual scenarios, enabling the model to dynamically adjust risk assessment standards according to the characteristics of different scenarios. By classifying different threats through the contextualized risk model, the system can distinguish threats of different levels, thereby achieving hierarchical management. Event level threshold determination, by setting clear threat level standards, allows the system to adjust the flexibility of risk classification according to specific situational conditions or security strategies. The structured representation of the classified data helps improve the system's efficiency in processing different risk events, providing a clear risk level basis for event aggregation and early warning strategies. Through the aggregation processing of integrated event level data, events of the same or similar levels are grouped together, reducing redundant data and simplifying the system's response process.

[0075] Preferably, step S4 specifically includes:

[0076] Step S41: Classify and schedule risk events based on security threat risk event data to obtain risk event scheduling mapping data;

[0077] Step S42: Match response strategies based on risk event scheduling mapping data to obtain response strategy data;

[0078] Step S43: Dynamically trigger early warning based on response strategy data to obtain response control early warning data;

[0079] Step S44: Perform multi-perspective dynamic risk assessment based on response control and early warning data to obtain perspective risk assessment matrix data;

[0080] Step S45: Predict response consequences based on the perspective risk assessment matrix data to obtain response consequence prediction data;

[0081] Step S46: Optimize the corresponding consequence prediction data to obtain optimized response control and early warning data for auxiliary operations in public safety situation prediction.

[0082] This invention categorizes and schedules risk events based on their level, type, and scope of impact, enabling the system to intelligently determine the priority of various events and allocate resources accordingly. Response strategy matching allows the system to select the most appropriate intervention measures based on the specific nature of the event, thereby achieving targeted early warning and control and enhancing the effectiveness of the response. Dynamic early warning ensures the system can flexibly respond to rapid changes in events, triggering early warning measures in real time, allowing security managers to obtain timely event updates and take prompt action. Multi-perspective dynamic risk assessment analyzes security threat events from multiple perspectives (such as security personnel, emergency managers, and ordinary users), generating multi-dimensional assessment matrix data. Response consequence prediction can estimate the effectiveness and impact of different strategies, helping managers select the best solution before implementing a response, thereby reducing risks and adverse effects. Optimized response control early warning data includes evaluated and improved response strategies and early warning information, more accurately reflecting the actual risks of the event. The system can automatically complete the entire process from event identification to response, greatly improving the efficiency and intelligence level of public safety management, reducing the burden on management personnel, and improving the overall level of safety management.

[0083] Preferably, this application also provides a public security situation prediction system based on multimodal data fusion, used to execute the public security situation prediction method based on multimodal data fusion as described above. The public security situation prediction system based on multimodal data fusion includes:

[0084] The Public Places Seamless Detection Module is used for continuous video surveillance of public places and seamless detection, and to obtain seamless detection data of public places.

[0085] The security threat multimodal feature extraction module is used to extract multimodal data features from non-intrusive detection data in public places to obtain security threat multimodal feature data.

[0086] The security threat risk event classification module is used to perform situational assessment on multimodal feature data of security threats, obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data.

[0087] The public safety situation prediction auxiliary module is used to automatically respond to and control security threat and risk event data, obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimized data, so as to assist in public safety situation prediction operations.

[0088] The beneficial effects of this invention are as follows:

[0089] 1. By combining the visual crowd dynamic perception data captured by the camera with the audio emotion analysis engine captured by the sensor, anomalies in the video can be further identified, and security levels can be classified according to the detected features, and corresponding responses can be made.

[0090] 2. Based on the characteristics of different security incidents, the system classifies them into four security posture levels (Level 1 to Level 4). A corresponding response mechanism is designed for each level, from incident recording to emergency response, ensuring appropriate action is taken for security incidents of varying risk levels.

[0091] 3. Design a security posture indicator system based on different roles to better assess and manage the security status and improve the security and reliability of the system.

[0092] 4. Construct a comprehensive security situation awareness and prediction model that integrates dense crowd detection and voice emotion analysis. This model can learn and identify specific security risk patterns, predict security situations, and improve the assessment, prediction, and response capabilities of public safety monitoring systems. Attached Figure Description

[0093] Other features, objects, and advantages of this application will become more apparent from the following detailed description of the non-limiting embodiments, taken with reference to the accompanying drawings:

[0094] Figure 1 A flowchart illustrating the steps of a public safety situation prediction method based on multimodal data fusion according to one embodiment is shown.

[0095] Figure 2 A flowchart illustrating the steps of a method for seamless detection in public places according to one embodiment is shown.

[0096] Figure 3 A flowchart illustrating the steps of a method for extracting multimodal features of security threats according to an embodiment is shown.

[0097] Figure 4 A flowchart illustrating the steps of a security threat risk event classification method according to an embodiment is shown;

[0098] Figure 5 A flowchart illustrating the steps of a security threat risk event classification method according to an embodiment is shown. Detailed Implementation

[0099] The technical method of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0100] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor methods and / or microcontroller methods.

[0101] It should be understood that although the terms "first," "second," etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are used merely to distinguish one unit from another. For example, without departing from the scope of the exemplary embodiments, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0102] This invention proposes a public safety situation assessment and prediction system based on deep learning and multimodal data fusion, which consists of six parts: research on non-intrusive detection technology, analysis of monitoring equipment, development of security assessment technology, design of security situation indicator system, construction of security situation perception model, and integration and optimization of control system.

[0103] First, using non-intrusive detection technology, a continuous video stream is captured via a camera. The video stream can be represented using a matrix, and the sequence of video frames is as follows: Among them I t Let represent the image at frame t. Then, image recognition algorithms such as convolutional neural networks are used to analyze this video data and identify abnormal behaviors or events. Specifically, the convolution operation can be represented as:

[0104]

[0105] Among them I t Given the input image, W is the convolution kernel, and (x,y) are the pixel coordinates.

[0106] Then, inference enhancement algorithms are used to complete and infer information loss in the video caused by occlusion, insufficient lighting, or other environmental factors. The inference enhancement algorithm can be represented as a prediction model f:

[0107]

[0108] in, For the predicted future frames, I t ,I t-1 ,…,I t-nFor past frames t to n

[0109] On the one hand, this technology can accurately identify real security threats from massive amounts of video data, effectively avoiding false alarms and missed detections. On the other hand, the algorithm must be able to adapt to different monitoring environments, such as changing lighting conditions, complex backgrounds, and changes in camera angle. This adaptability can be achieved by enhancing the model's robustness, for example, by introducing a lighting variation model L during model training.

[0110]

[0111] Among them, I′ t The image is processed by the illumination change model, where L is the illumination change model and θ represents the illumination change parameters.

[0112] Through these technical means, the system can effectively detect anomalies in complex environments, improving the reliability and accuracy of security monitoring systems.

[0113] Next, the visual crowd dynamic perception data captured by the camera, combined with the audio emotion analysis engine captured by the sensor, further identifies anomalies in the video and classifies them into security levels based on the detected features, then takes appropriate responses. The video frame sequence is as follows: The audio frame sequence is

[0114] Video crowd dynamic perception can be processed using convolutional neural networks (CNNs), as described in the convolution operation above:

[0115]

[0116] Audio emotion analysis engines can analyze the emotional fluctuations of audio signals using Long Short-Term Memory (LSTM) networks.

[0117] h t =σ(W h ·A t +U h ·h t-1 +b n );

[0118] Among them, h t Indicates the hidden state at the current moment, A t For audio signals, W h and U h Let b be the weight matrix. n σ is the bias, and σ is the activation function.

[0119] Multimodal fusion of visual and audio data can be achieved through weighted fusion methods:

[0120] Ft =α·f v (I t )+β·f a (A t );

[0121] Among them, F t f represents the features after fusion. v f represents the visual feature extraction function. a This represents the audio feature extraction function, where α and β are the fusion weights.

[0122] To achieve security level classification, video and audio data are continuously monitored over a period of time. By continuously monitoring and recording event data, the system can learn and identify specific security risk patterns and compare them with previous security event data to identify important security event nodes, namely the thresholds N1, N2, N3, E1, E2, E3 for each risk level. Assume we monitor data within the time period [0, T] and statistically analyze the distribution of each parameter:

[0123] N1 = max(n c |n c (is low risk);

[0124] N2 = max(n c |n c (is medium risk);

[0125] N3 = max(n c |n c (is high risk);

[0126]

[0127] Where n C This represents the population size, where N1 is the low-risk threshold, N2 is the medium-risk threshold, and N3 is the high-risk threshold. E(A) t E1 represents the emotional fluctuation value, E2 represents the low-risk emotional fluctuation threshold, and E3 represents the high-risk emotional fluctuation threshold.

[0128] Based on the detected feature F t The system will classify security levels and respond accordingly. The security levels are classified as follows:

[0129] The Level 1 safety incident (low risk) RISK1 involved a crowd gathering and then dispersing; no significant emotional fluctuations were detected in the voice analysis. This can be expressed as:

[0130] RISK1 = {n C ≤N1,|E(A t )|≤E1};

[0131] Where n C This represents the number of people, N1 is the low-risk threshold, and E(A) t E1 represents the emotional fluctuation value, and E2 represents the low-risk emotional fluctuation threshold. At this point, the system will record the event and continue monitoring, without requiring immediate action.

[0132] A Level 2 safety incident (medium risk) RISK2 is characterized by a sustained and increasing gathering of people, with minor emotional fluctuations such as small-scale arguments being detected. This can be expressed as:

[0133] RISK2 = {N1≤n C ≤N2,E1≤|E(A t )|≤E2};

[0134] At this point, the system should increase the monitoring frequency and notify security personnel to prepare for intervention.

[0135] Level 3 safety incidents (high risk) are characterized by significantly abnormal behavior, such as a person falling to the ground or physical conflict, accompanied by obvious tension or panic. This can be expressed as:

[0136] RISK3 = {N2≤n C ≤N3,E2≤|E(A t )|≤E3};

[0137] At this point, the system must immediately notify on-site safety personnel and the emergency response team to intervene.

[0138] Level 4 security incidents (serious risks) are the most urgent situations, such as large-scale brawls or life-threatening medical emergencies, characterized by sustained high-intensity emotional reactions such as screaming, crying, and cries for help. This can be expressed as:

[0139] RISK3 = {n C >N3,|E(A t )|>E3};

[0140] The system will trigger the highest level of alert, immediately dispatch security and medical emergency services, and notify management to take further action.

[0141] By combining dense crowd detection and voice emotion analysis, public safety monitoring systems can more accurately assess and respond to potential threats, effectively enhancing security in public places. Furthermore, through continuous monitoring and the acquisition of large amounts of audio and video data, future security incidents can be effectively predicted, improving response efficiency. Through multimodal data fusion methods, this technology also aims to improve the adaptability and effectiveness of artificial intelligence systems in changing environments.

[0142] Table 1 Classification of Security Incident Levels

[0143]

[0144] Then, security assessment technology is developed, and a security posture indicator system is designed to establish a capability verification-based framework for systematically assessing and managing security risks. This assessment technology is mainly subdivided into three core dimensions: access awareness capability, threat blocking capability, and vulnerability identification capability. Access awareness capability focuses on how quickly and accurately the system can identify and respond to access attempts from external devices or users. Assume the access request time is t. a The system response time is t. r Access perception capability C a It can be represented as:

[0145]

[0146] Among them, the accuracy of access sensing A a This can be represented by the ratio of correctly identified access requests:

[0147]

[0148] Here, N correct N is the number of correctly identified access requests. total This is the total number of access requests.

[0149] Threat blocking capability assesses a system's ability to defend against potential attacks, including how effectively it can identify, isolate, and disrupt malicious activity. Threat blocking time t b It can be expressed by the following formula:

[0150] t b =t detect +t isolate +t mitigate ;

[0151] Among them, t detect Threat detection time, t isolate It is a threat to the isolation time, t mitigate This is a dangerous interruption time.

[0152] Threat blocking capability C b It can be represented as:

[0153]

[0154] Similarly, the accuracy of threat blocking A b This can be expressed as the ratio of correct identification to blocking:

[0155]

[0156] Where, N blockedN is the number of threats successfully blocked. total threats That is the total number of threats.

[0157] Vulnerability identification capability refers to a system's ability to discover and patch security vulnerabilities. (Vulnerability discovery time t) v and repair time t f It can be represented as:

[0158]

[0159] Among them, t detect，i t is the detection time for the i-th vulnerability. fix，i N is the fix time for the i-th vulnerability. v That is the total number of vulnerabilities.

[0160] Vulnerability identification capability C v It can be represented as:

[0161]

[0162] The system is designed with a set of security posture indicators for different roles, including system administrators, security analysts and ordinary users, in order to better assess the security status.

[0163] System administrators focus on the overall security posture of the system, including access awareness, threat blocking, and vulnerability identification capabilities. Therefore, comprehensive indicator C... admin The formula is:

[0164] C admin =α a C a +α b C b +α v C v ;

[0165] Where, α a α b α v It represents the weights of each capability dimension.

[0166] Security analysts focus on specific threat intelligence and response results, primarily assessing threat blocking capabilities and vulnerability identification capabilities.

[0167] Comprehensive Index C analyst The formula is

[0168] C analyst =β b C b +β v C v ;

[0169] Where, βb ,β v It is the weight of threat blocking capability and vulnerability identification capability.

[0170] System users are concerned about the system's access awareness capabilities and response speed.

[0171]

[0172] Where γ a γ r It is the weight of access sensing capability and system response speed.

[0173] Through these formulas and indicator systems, different users can systematically assess and manage security risks, ensuring that the system can respond and handle various security threats quickly and accurately.

[0174] Finally, the above is integrated into a control system, which enables real-time collection and processing of monitoring data. This data is then processed according to the safety level (Risk(F)). t Design an automatic response mechanism. The response mechanism can be represented as:

[0175]

[0176] When the system detects abnormal behavior or security threats, it can automatically trigger an alarm or take other measures. The alarm system's trigger function A(RISK) can be expressed as:

[0177]

[0178] By capturing image data through real-time monitoring cameras, the controller processes this data to identify potential security issues and takes action through the alarm system.

[0179] Please see Figures 1 to 5 This application provides a public safety situation prediction method based on multimodal data fusion, including the following steps:

[0180] Step S1: Conduct continuous video surveillance of public places and perform seamless detection to obtain seamless detection data of public places;

[0181] Specifically, multiple high-definition cameras are deployed in public areas, and video data is continuously monitored through a cloud control center. After capturing video, the cameras transmit the images to the processing module. By monitoring crowd flow, abnormal behavior, and the movement trajectories of specific objects in real time, the non-intrusive detection module filters out abnormal situations using predefined behavior or feature detection models. Without interfering with activities in public areas, AI detection technology is used to identify security threat characteristics such as crowding and fighting, and the identified non-intrusive data is transmitted to the feature extraction module.

[0182] Step S2: Perform multimodal data feature extraction on the non-intrusive detection data in public places to obtain multimodal feature data of security threats;

[0183] Specifically, the data from the non-sensory detection system enters the multimodal feature extraction stage. This stage first decomposes the video data into frames and converts them into image format. The image data, along with data from other sensing devices (such as thermal imagers and sound monitors), is input into the feature extraction network. Then, specific rules or thresholds (such as sudden temperature changes or volume exceeding a set value) are used to extract multimodal features. Combining target tracking and region segmentation techniques, images from different angles in the video and other modal data are combined to form a dataset. This data undergoes computation by the feature extraction algorithm to extract spatial features, temporal features, and complementary features between modalities, providing multi-dimensional input data for subsequent situation assessment.

[0184] Step S3: Perform situation assessment on the multimodal feature data of security threats to obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data;

[0185] Specifically, in situation assessment, the system receives multimodal feature data and analyzes its impact on the level of security threats. First, the input feature data is quantified to create risk scores, such as assessing the severity of events through dimensions like frequency, location, and time of occurrence. Next, through historical risk data and pattern matching, similar events and potential risk levels are identified, and new events are assigned levels, outputting security threat event level data. To achieve accurate event level classification, event association rules are introduced; for example, if screams are accompanied by rapid movement in a densely populated area, the event level is raised. This event level data is labeled with different risk levels, generating the final security threat risk event data, ensuring that risk classification conforms to public safety standards.

[0186] Step S4: Perform automatic response control based on security threat risk event data to obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimized data, so as to assist in public safety situation prediction.

[0187] Specifically, based on risk event data, the system automatically generates response plans. First, different response measures are triggered based on the event level, such as sending notifications to nearby security guards or police personnel, or automatically initiating area broadcasts. To ensure the effectiveness of the response, the system adjusts control strategies based on real-time feedback. For example, if a high-risk event is quickly brought under control, the event level is reduced through feedback. On the other hand, the system utilizes multi-view assessments from other cameras and sensors to optimize the initial response. Through multi-view analysis, the event perspectives provided by each camera are combined into a global view, optimizing the initial response. Finally, the system stores optimized warning data to aid in future situation prediction.

[0188] Preferably, step S1 specifically includes:

[0189] Step S11: Capture a continuous video stream using a camera to obtain video stream matrix data;

[0190] Specifically, the camera continuously captures a continuous video stream of public spaces. The video stream is a collection of time-series images, which the system converts into video stream matrix data. Video frames are sampled at a fixed frame rate, such as 30 frames per second. Each frame is processed into grayscale and converted into pixel matrix data. With an image size of 1920x1080 pixels, each frame is represented as a matrix containing the grayscale value of each pixel. Throughout the video stream, the matrix data of each frame forms a three-dimensional array with the time axis as its depth, creating a complete spatiotemporal video matrix. The system uses pixel changes in the matrix to determine the movement of actions or objects in the scene, thereby capturing dynamic information in a time series.

[0191] Step S12: Extract abnormal behavior events based on the video stream matrix data to obtain abnormal behavior event data;

[0192] Specifically, after obtaining the video stream matrix data, the system begins analyzing abnormal behavior events within the video stream. First, motion detection is used to extract dynamic objects appearing in the video stream; for example, object movement is detected by calculating pixel differences between adjacent frames. If, in a specific frame, the movement pattern of a crowd or the speed of an individual abnormally increases, the system will mark these areas. Then, using a behavior classification model, the system analyzes the movement trajectory, speed, direction, and other features of the marked areas to detect abnormal behavior. For example, if the density of a certain area suddenly increases, or a large-scale crowd movement is detected, the system will record this as abnormal behavior event data. For each detected abnormal behavior event, the system generates information such as event type (e.g., running, crowding), location coordinates, and time of occurrence.

[0193] Step S13: Perform extrapolation and enhancement based on abnormal behavior event data to obtain data on seamless detection in public places.

[0194] Specifically, based on abnormal behavior event data, the system further performs inductive enhancement processing. By comparing current event data with historical abnormal event patterns, such as the area, time, and type of the event, the system assesses the potential risk of the event. For example, if the system detects congestion events in entrance / exit areas, and combines historical data analysis, such events have historically been accompanied by more severe congestion risks, so the system will raise the risk level. Furthermore, inductive enhancement incorporates environmental factors, such as trends in on-site temperature and volume data, to predict the event's development direction. The seamless detection data includes event type, location, risk assessment level, and event development prediction. Alternatively, inductive enhancement algorithms can be used to supplement and infer information missing from videos due to obstructions, insufficient lighting, or other environmental factors.

[0195] Preferably, the abnormal behavior event extraction in step S12 is performed using a preset abnormal behavior event identification model, and the specific steps for constructing the preset abnormal behavior event identification model are as follows:

[0196] Obtain historical video stream matrix data and historical abnormal behavior event tags;

[0197] Specifically, historical video stream data is collected, ensuring that this data comes from different environments and contexts. The video stream data includes multiple scenes and behavior types. When collecting video stream data, each video stream sample needs to be labeled with a specific anomalous behavioral event (such as "running," "fighting," or "crowding"), which will serve as the label for that video stream. Anomalous behaviors corresponding to each video stream segment are identified through manual annotation or rule-based detection.

[0198] Clustering calculations are performed on the historical video stream matrix data to obtain historical video stream matrix cluster data;

[0199] Specifically, after obtaining the historical video stream matrix data, the next step is to perform clustering on this data. The goal of the clustering process is to group similar video stream samples into the same category, reducing the complexity of the data space. Using a pixel-feature-based clustering method, the video streams are divided into different groups by comparing the similarity of image features (such as color distribution, edge information, etc.) between different video streams. The key to this clustering process is to identify video segments with similar behaviors or scenes.

[0200] Balanced classification is performed based on the clustering data of the historical video stream matrix to obtain historical video stream test data and historical video stream verification data.

[0201] Specifically, data balancing techniques (such as oversampling, undersampling, or data augmentation) are used to ensure that the number of samples in different categories is similar, thereby avoiding model bias towards frequently occurring behavior types. These balancing processes generate test and validation data.

[0202] Preliminary convolution calculations are performed based on historical video stream test data to obtain preliminary video stream feature data;

[0203] Specifically, the initial convolution calculation is the first step in feature extraction for the model, aiming to identify important features in the video through convolution operations. Through convolution operations, key low-level features (such as edges and textures) are extracted from each frame of the image.

[0204] Pooling and multi-layer self-attention calculations are performed on the initial video stream feature data to obtain video stream pooled data and video stream weighted feature data, respectively.

[0205] Specifically, pooling layers reduce the spatial dimensionality of feature maps, extracting more important features while reducing computational cost. For example, max pooling selects the maximum value in each small region, thus preserving the most salient features. Self-attention mechanisms are used to capture long-range dependencies between video frames. When processing a continuous video stream, each frame is correlated with other frames. Self-attention mechanisms enhance more correlated frames by weighted summation of different frames, while ignoring less important parts.

[0206] Fully connected video stream data is obtained by performing fully connected encoding calculations based on the weighted feature data and pooled data of the video stream.

[0207] Specifically, after pooling and self-attention calculations, the resulting feature data undergoes further processing through fully connected layers. The role of fully connected layers is to map the extracted features to a higher-dimensional space, helping the model learn more complex patterns. Through fully connected layers, the model can adjust the weights of different behavioral features, optimizing model performance.

[0208] Preliminary abnormal behavior event identification data is obtained by calculating the index based on the fully connected video stream data and mapping it through historical abnormal behavior event labels.

[0209] Specifically, after obtaining the fully connected data, the purpose of performing exponential calculation is to assess the importance of behavioral patterns. By exponentially weighting the fully connected data, more important features are given higher weights. Following this, the system maps the features output by the model to preset abnormal behavior types (such as running, fighting, etc.) based on historical abnormal behavior event labels.

[0210] By using historical video stream verification data to iteratively train the preliminary abnormal behavior event identification data, an abnormal behavior event identification model is obtained.

[0211] Specifically, the model's performance is optimized by training the initial identification results on a validation dataset using either the perfect squared difference function or the information entropy loss function. During training, the model iteratively adjusts its weights based on incorrectly predicted samples to reduce errors, resulting in a model capable of accurately identifying anomalous behavioral events.

[0212] Preferably, the multi-layer self-attention calculation specifically involves:

[0213] Multi-scale hierarchical convolution and spatiotemporal feature segmentation are performed on the preliminary video stream feature data to obtain multi-scale video stream feature data;

[0214] Specifically, the initial video stream feature data is first processed through multi-scale hierarchical convolution to extract details and global information under different behavioral patterns. Through multi-scale convolution, the system can identify features of different sizes and distances. For example, 3x3, 5x5, and 7x7 convolution kernels are used to scan the video stream features to obtain feature maps at different scales. Next, these feature data are segmented based on spatiotemporal dimensions (time axis and spatial location) to capture features with different spatiotemporal distributions.

[0215] Local multi-head hierarchical self-attention calculation and global multi-head hierarchical self-attention calculation are performed on the multi-scale feature data of the video stream to obtain the local self-attention feature data and the global self-attention feature data of the video stream, respectively.

[0216] Specifically, local multi-head self-attention computation is performed on multi-scale feature data, with each head focusing on features of a specific spatial region to ensure that detailed features of each local region are highlighted. The features of each local region are weighted through a self-attention mechanism to emphasize key features. After local computation, the system performs global self-attention computation on the feature data of the entire video stream, capturing global information through a multi-head approach. Each head focuses on different feature combinations to ensure that the model can identify spatiotemporal relationships within the global scope of video frames.

[0217] The relative position matrix is ​​constructed by combining the local self-attention feature data and the global self-attention feature data of the video stream to obtain the multi-head self-attention feature matrix data of the video stream.

[0218] Specifically, local and global self-attention feature data are combined to construct a relative position matrix. This matrix calculates the relative positional relationship between each feature point and other feature points, ensuring the self-attention mechanism can effectively perceive spatial relationships. During construction, local and global features are fused with different weights and encoded according to their relative positions to form a multi-head self-attention feature matrix.

[0219] Multi-layer graphs are constructed based on the multi-head self-attention feature matrix data of the video stream to obtain multi-layer graph data of the video stream;

[0220] Specifically, a multi-layer graph is constructed using a multi-head self-attention feature matrix. Each layer represents the spatial feature connections at a given time point, with nodes representing each location feature and edges representing the weights of relative positions. The layers from each time point are overlaid to generate a multi-layer spatiotemporal graph to capture the evolution of features in the time series. For example, if a video stream contains 5 time slices, and the graph generated for each time slice has 100 nodes and 1000 edges (relative positions), the multi-layer graph contains 5 layers, with each layer storing features from different time points.

[0221] Global-local graph convolution is performed on the multi-layer graph data of the video stream to obtain the global-local graph convolution data of the video stream;

[0222] Specifically, global-local graph convolution is performed on multi-layer graphs to capture the spatiotemporal relationships between nodes. Local graph convolution propagates features among local neighbors in each layer, enhancing local correlations. Global convolution integrates global features across graph layers, ensuring that cross-temporal dependencies are captured. Through this computation, the model obtains deep correlations between spatiotemporal features.

[0223] Multi-scale feature fusion is performed on the global-local graph convolutional data of the video stream to obtain weighted feature data of the video stream.

[0224] Specifically, the features after global-local graph convolution are fused at multiple scales to integrate details and global information at different scales. Multi-scale feature fusion combines local and global features along the time dimension using weighted averaging and stacked features to generate weighted feature data. The weight of local features is 0.7, and the weight of global features is 0.3, resulting in a 20-dimensional feature vector that encompasses multi-scale information from the video stream.

[0225] Preferably, the multi-layer graph construction specifically involves:

[0226] Hierarchical multi-scale graph deconstruction is performed based on the multi-head self-attention feature matrix data of the video stream to obtain the hierarchical multi-scale graph data of the video stream;

[0227] Specifically, the multi-head self-attention feature matrix of a video stream contains spatial and temporal features. First, a hierarchical multi-scale graph deconstruction is required to understand these features at different scales. The feature matrix is ​​divided into multiple scales, such as near-range, mid-range, and far-range levels. Features at each scale are partitioned according to their distances, ensuring that each scale graph structure represents the feature relationships within a specific distance range. Specifically, the system forms three hierarchical subgraphs by partitioning the relative distances between nodes, each subgraph containing nodes and edges within a certain range.

[0228] Cross-layer association nodes are constructed from the hierarchical multi-scale graph data of the video stream to obtain cross-layer association node data of the video stream.

[0229] Specifically, based on a hierarchical multi-scale graph, cross-layer associative nodes are constructed to connect nodes at different scales, helping the model capture relationships between different spatiotemporal scales. Each pair of cross-layer associative nodes represents a potential relationship at different spatiotemporal scales; for example, a node at a near-range scale is associated with a node at a medium-range scale. The weights of cross-layer node pairs are calculated based on the feature similarity and distance between nodes; higher weights indicate stronger node associations.

[0230] Based on the cross-layer association node data of the video stream, association features are extracted to obtain cross-layer association feature data of the video stream;

[0231] Specifically, based on the cross-layer associated node data, the system extracts features from these associated node pairs to obtain cross-layer feature relationships. The feature differences, similarities, and distance relationships of each pair of associated nodes are extracted to help the model understand the connections between cross-layer nodes. Cross-layer features are processed using methods such as weighted averaging and distance normalization, enabling the model to effectively perceive cross-layer information.

[0232] Gaussian convolution kernel mapping is performed on the cross-layer correlation feature data of the video stream to obtain the Gaussian convolution kernel data of the video stream;

[0233] Specifically, Gaussian convolution kernels are applied to the cross-layer correlation feature data of the video stream. Gaussian convolution kernels apply smooth weights to neighboring nodes, making the correlation between adjacent nodes stronger. Through the Gaussian function, weighted values ​​are generated based on the distance between related nodes; closer nodes receive higher weights, while farther nodes receive lower weights. This ensures that the model focuses on important local feature relationships while preserving key spatiotemporal features.

[0234] Based on the Gaussian convolution kernel data of the video stream, graph convolution processing is performed on the cross-layer related node data of the video stream to obtain the cross-layer convolutional layer data of the video stream;

[0235] Specifically, graph convolution is performed on the weighted data generated by Gaussian convolution kernel mapping to integrate the features of cross-layer related nodes into a holistic representation. The graph convolution operation aggregates the features of each node with the features of its neighboring nodes to generate a feature representation that includes neighborhood information. This ensures that the features of each node not only contain its own information but also reflect the feature relationships with nodes at other layers. The initial features of node A are [0.2, 0.5, 0.1], and the features of its neighbor node B, after Gaussian convolution mapping, are [0.072, 0.072, -0.072]. The graph convolution operation performs a weighted sum of the features of A and B, such as 0.6 × [0.2, 0.5, 0.1] + 0.4 × [0.072, 0.072, -0.072], resulting in a new feature representation [0.143, 0.3, 0.04].

[0236] Cross-layer transfer processing is performed on the video stream convolutional layer data to obtain multi-layer graph data of the video stream.

[0237] Specifically, the data from the cross-layer convolutional layers after graph convolution undergoes cross-layer propagation to spread information between different layers, forming integrated multi-layer graph data. The system transmits node information through cross-layer paths and weights, ensuring that the node features of each layer can interact with the node features of other layers. Through this cross-layer propagation, the model can integrate features from all spatiotemporal levels into a unified representation, obtaining multi-layer graph data from the video stream.

[0238] Preferably, step S2 specifically includes:

[0239] Step S21: Extract dynamic perception features of visual crowds based on the non-intrusive detection data in public places to obtain dynamic perception feature data of visual crowds;

[0240] Specifically, the system identifies people in each video frame using object detection algorithms and calculates the population density per unit area. For example, it calculates the population density per square meter. Changes in density can indicate the state of crowd gathering or dispersal. The system tracks the movement trajectory of each individual or group of people, calculating their positional changes in each frame to obtain the movement speed and direction of movement for each person or group. For example, it calculates the average speed difference between individuals to determine if there is a large-scale crowd movement or congestion. If the system detects a sudden increase in density or abnormal flow speed in a certain area, it marks these features as anomalies and generates visual crowd dynamic perception feature data.

[0241] Step S22: Extract audio synchronization data based on the visual crowd dynamic perception feature data and video stream matrix data to obtain audio synchronization data;

[0242] Specifically, by aligning the timestamps of video and audio, each video frame is matched with its corresponding audio segment to achieve multimodal synchronization. For example, in a video at 30 frames per second, the audio segment per second is divided into 30 segments, synchronized with each video frame. After alignment, the system extracts features from the audio signal, such as volume, frequency, and amplitude changes, focusing on abnormal audio such as screams and rapid sounds. The system detects abnormal fluctuations in volume and frequency; if there is a significant increase in volume or a sharp change in frequency in the audio, it is marked as audio emotion analysis data.

[0243] Step S23: Perform audio emotion analysis based on the audio synchronization data to obtain audio emotion analysis data;

[0244] Specifically, the system first extracts emotion-related features from the audio signal, including pitch, volume variation rate, and timbre. Rapid or high-frequency audio fluctuations typically represent anxiety or tension. These audio features are then compared to a pre-trained emotion recognition model to classify the audio segment by emotion. For example, a rapid rise in pitch and high volume fluctuations are often classified as "anger" or "tension." The intensity of the emotion is then rated (from 0 to 1), with higher intensity indicating a greater potential threat.

[0245] Step S24: Perform multimodal fusion of visual crowd dynamic perception feature data and audio emotion analysis data to obtain multimodal feature data of security threats.

[0246] Specifically, the system further aligns the visual and audio time segments and assigns different weights based on the nature of the event. Typically, the identification of crowding behavior relies primarily on visual features, while emotional events depend more on audio emotional data. Features such as crowd density, speed, volume, and emotional category are combined to form a complete multimodal feature vector. For example, when high crowd density is combined with high emotional intensity, the system can identify the possibility of an "emergency situation." On the multimodal feature vector, the system classifies the potential threat level. If visual features indicate high crowd density and the audio emotion is interpreted as "anger," the event is marked as "high risk."

[0247] Preferably, the audio emotion analysis is processed through a preset audio emotion analysis model, and the construction of the audio emotion analysis model includes the following steps:

[0248] Acquire historical audio synchronization data and historical audio emotion tag data;

[0249] Specifically, the system needs to acquire a large amount of historical audio data and corresponding emotion tags. The historical audio data includes audio clips under various emotions, such as anger, anxiety, joy, and calmness, and each audio clip is already labeled with an emotion tag.

[0250] The historical audio frequency domain data is obtained by performing frequency domain conversion based on the historical audio synchronization data;

[0251] Specifically, the system performs a short-time Fourier transform on each audio segment, converting the audio waveform data into a spectrum, where the horizontal axis represents frequency, the vertical axis represents time, and the brightness of each point represents the intensity of a specific frequency at that time point.

[0252] The historical audio frequency domain data is divided to obtain historical audio test data and historical audio verification data;

[0253] Specifically, for model training and validation, the frequency domain data is divided into test data and validation data. 80% of the data is used as test data and 20% as validation data.

[0254] The initial state is constructed based on historical audio test data to obtain historical audio initial state data;

[0255] Specifically, the initial state is usually set to zero or initialized randomly to give the model a consistent starting point.

[0256] Based on the historical audio initial state data, the input layer and forgetting layer are calculated to obtain the audio input layer data and audio forgetting layer data, respectively.

[0257] Specifically, in the input layer of the model, the system inputs the frequency-domain converted audio data into the model. The computation of the input layer is mainly responsible for receiving and processing audio features, transforming them into feature vectors for sentiment analysis. Next, the forgetting layer is used to filter out unimportant audio features, retaining information valuable for sentiment judgment. Based on the feature vectors output by the input layer, the forgetting layer uses weighted or gating mechanisms to filter out key feature information.

[0258] Multi-head emotion candidate memory state processing is performed based on audio input layer data and audio forgetting layer data to obtain audio candidate memory state data.

[0259] Specifically, audio features are processed in parallel by multiple heads (each handling specific emotional features), with each head focusing on specific emotional information. For example, one head might focus on high-frequency variations, while another focuses on low-frequency features. Through this multi-head mechanism, the system generates candidate memory state data containing different emotional candidate features.

[0260] The audio candidate memory state data is processed by the output layer to obtain the audio hidden state data;

[0261] Specifically, based on the candidate memory states, the model performs output layer calculations, aggregating multi-headed emotion features into a latent feature vector, namely "audio hidden state data." This latent feature represents a high-level abstraction of the audio emotion and possesses representativeness of the emotion. During the output layer calculation process, the system generates emotion feature data through weighted summation or gating mechanisms.

[0262] By iteratively training the audio hidden state data using historical audio verification data and historical audio emotion label data, an audio emotion parsing model is obtained.

[0263] Specifically, after constructing the initial model for audio emotion analysis, the system uses historical audio validation data and emotion labels for iterative training. In each iteration, the model provides feedback and adjustments based on the discrepancy between the predicted results and the actual labels, optimizing the computational weights of each layer. Through multiple iterations of training, the model gradually learns emotional features, achieving accurate emotion analysis.

[0264] The processing of multi-headed emotion candidate memory states includes:

[0265] Based on the audio input layer data and the audio forgetting layer data, multi-dimensional candidate memory state processing is performed to obtain the initial audio candidate memory state data.

[0266] Specifically, a multi-dimensional memory state data structure is created by analyzing the feature vectors of the audio input layer and the weighted features of the forgetting layer. This structure contains information on dimensions such as different frequencies, volume, and pitch variations. In detail, the model decomposes the audio input features, and each dimension (such as frequency range or specific pitch) yields a separate candidate state.

[0267] Multi-head emotion attention calculation is performed on the initial audio candidate memory state data to obtain audio multi-head emotion attention matrix data;

[0268] Specifically, the goal of multi-head emotion attention computation is to perform in-depth processing on the initial audio candidate memory state by having multiple "heads" focus on different emotional feature dimensions. Each "head" will focus on different audio features, such as one head focusing on high-frequency emotion features and another head focusing on low-frequency features.

[0269] Emotional memory enhancement was performed on audio multi-head emotion attention matrix data to obtain audio enhanced memory state data;

[0270] Specifically, based on the multi-headed emotion attention matrix, the system performs emotion memory enhancement processing to amplify key emotional features and reduce unimportant information. Memory enhancement weights the emotion attention matrix, focusing on amplifying the parts with greater emotional fluctuations, such as high-frequency fluctuations in anger or low-frequency fluctuations in anxiety, making the model pay more attention to the intense expression of emotions.

[0271] The initial audio candidate memory state data is fed back to the memory state data by using the audio-enhanced memory state data, and the feedback memory state data is obtained.

[0272] Specifically, the memory state is further optimized by enhancing features, making important emotional features more prominent. The feedback mechanism, through a weighted combination of the enhanced and initial states, enables the model to more accurately capture emotional changes during iteration. The enhanced and initial memory states are weighted and combined to generate the feedback-adjusted memory state. The combined features are then updated to ensure greater accuracy of the memory state under multi-headed attention.

[0273] Based on the feedback memory state data, multi-level candidate state aggregation is performed to obtain audio candidate memory state data.

[0274] Specifically, different emotional features of the feedback memory state are aggregated hierarchically, merging the emotional information from all the heads into a single overall emotional state. During the aggregation process, the weights of different features are adjusted to prevent any single feature from excessively influencing emotion analysis.

[0275] Preferably, step S3 specifically includes:

[0276] Step S31: Perform preliminary screening of security threat multimodal features based on the security threat multimodal feature data to obtain threat preliminary screening index data;

[0277] Specifically, by setting preliminary screening criteria (such as feature values ​​exceeding a specific threshold), the system filters out noisy or unimportant feature data, retaining only features closely related to security risks. This is based on multiple indicators, such as crowd density, audio emotional intensity, and temperature.

[0278] Step S32: Allocate risk weights based on the initial threat screening index data to obtain threat weight matrix data;

[0279] Specifically, after obtaining the initial threat indicator data, the system assigns a risk weight to each threat feature. The weight allocation takes into account the degree of impact of each feature on the security threat; for example, crowd density and the intensity of emotional tone in audio are more indicative of an emergency situation than temperature changes. Therefore, the system presets the weight ratio for each feature based on historical data and risk models. The risk weight allocation can also be dynamically adjusted, making the weighting more flexible. For example, features with high-density crowds and negative audio emotions receive higher weights.

[0280] Step S33: Construct a contextualized risk model based on the threat weight matrix data to obtain a public safety contextualized risk model;

[0281] Specifically, after obtaining the threat weight matrix, the system begins to construct a contextualized risk model. This model analyzes multimodal data within a specific context, combining the surrounding environment and historical data to generate a risk assessment model that better reflects the actual situation. The contextualized model integrates the weights of each feature to form a specific model for that location or event.

[0282] Step S34: Perform risk classification based on the public safety contextualized risk model to obtain security threat event level data;

[0283] Specifically, through the output of the contextualized risk model, the system classifies each feature into risk levels, categorizing different feature data into low, medium, and high risk levels. The risk level classification is based on the importance of the features output by the contextual model. For example, if the population density feature is given a higher weight in the contextual model, population density data will be more likely to be classified as high-risk than other features.

[0284] Step S35: Determine the event level threshold based on the security threat event level data to obtain the event level determination table data;

[0285] Specifically, after obtaining the risk level of each feature, the system performs a threshold determination for the event level. The threshold determination is based on specific level classification standards (e.g., a population density exceeding 5 people / square meter is considered high risk) to determine the level of each event and generate an event level determination table. The thresholds set by the system are based on historical data and the specific conditions of the current scenario to ensure the accuracy of event level classification.

[0286] Step S36: Classify the multimodal feature data of security threats into event levels based on the event level determination table data to obtain integrated event level data;

[0287] Specifically, based on the event level determination table, the system classifies each feature data into an event level so as to integrate features of different risk levels into a complete event level integrated data.

[0288] Step S37: Aggregate risk events from the event level integration data to obtain security threat risk event data.

[0289] Specifically, the system performs an overall analysis of the risk level of each feature, integrates the correlations between different features, and forms an overall risk event assessment. For example, in the context of highly emotional audio in a densely populated area, the event is marked as a "high-risk event".

[0290] Preferably, step S4 specifically includes:

[0291] Step S41: Classify and schedule risk events based on security threat risk event data to obtain risk event scheduling mapping data;

[0292] Specifically, the system first classifies and schedules risk events based on security threat risk event data, identifying different types of risk events (such as congestion, riots, fires, etc.), and generating scheduling mapping data according to event type, level, and location. The goal of scheduling is to map different risk events to corresponding emergency response resources or departments.

[0293] Step S42: Match response strategies based on risk event scheduling mapping data to obtain response strategy data;

[0294] Specifically, after event scheduling is completed, the system performs response strategy matching for each scheduled event. The system searches the policy database for response strategies that match the event type and risk level.

[0295] Step S43: Dynamically trigger early warning based on response strategy data to obtain response control early warning data;

[0296] Specifically, the system triggers dynamic early warnings based on response strategy data and controls the response process in real time. Early warning triggering includes the system issuing evacuation orders, sending alarm notifications to relevant departments and personnel, and conducting on-site control through broadcasts, screens, and other media. Early warning triggering also automatically adjusts response resources, such as mobilizing nearby security personnel or automatically unlocking access control systems to ensure smooth evacuation.

[0297] Step S44: Perform multi-perspective dynamic risk assessment based on response control and early warning data to obtain perspective risk assessment matrix data;

[0298] Specifically, the system utilizes multi-view monitoring data to conduct dynamic risk assessments of events. This multi-view data comes from various surveillance cameras, sensors, and data sources, such as population density and volume changes in different areas. The system integrates this data into an assessment matrix to quantify the real-time risk status of the event scene.

[0299] Step S45: Predict response consequences based on the perspective risk assessment matrix data to obtain response consequence prediction data;

[0300] Specifically, based on the perspective risk assessment matrix data, the system predicts the consequences of current response measures (e.g., through linear regression or vector machine models) and analyzes the effects after the warning is triggered. Consequence prediction involves inputting the risk assessment matrix data into the prediction model to analyze the direction of event progression.

[0301] Step S46: Optimize the corresponding consequence prediction data to obtain optimized response control and early warning data for auxiliary operations in public safety situation prediction.

[0302] Specifically, based on the predicted consequences of the response, the system optimizes the current response measures to ensure that the incident is brought under control as quickly as possible. The optimization process adjusts the response strategy according to the consequences prediction results, such as increasing the frequency of evacuations or increasing the allocation of personnel resources.

[0303] Preferably, this application also provides a public security situation prediction system based on multimodal data fusion, used to execute the public security situation prediction method based on multimodal data fusion as described above. The public security situation prediction system based on multimodal data fusion includes:

[0304] The Public Places Seamless Detection Module is used for continuous video surveillance of public places and seamless detection, and to obtain seamless detection data of public places.

[0305] The security threat multimodal feature extraction module is used to extract multimodal data features from non-intrusive detection data in public places to obtain security threat multimodal feature data.

[0306] The security threat risk event classification module is used to perform situational assessment on multimodal feature data of security threats, obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data.

[0307] The public safety situation prediction auxiliary module is used to automatically respond to and control security threat and risk event data, obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimized data, so as to assist in public safety situation prediction operations.

[0308] Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended application documents rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of the equivalents of the application documents be incorporated into the invention.

[0309] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A public safety situation prediction method based on multimodal data fusion, characterized in that, Includes the following steps: Step S1: Conduct continuous video surveillance of public places and perform seamless detection to obtain seamless detection data of public places; Step S2: Extract multimodal data features from the non-intrusive detection data in public places to obtain multimodal feature data of security threats; Step S3: Perform situation assessment on the multimodal feature data of security threats to obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data; Step S4: Perform automatic response control based on security threat risk event data to obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimization data, so as to assist in public safety situation prediction. Step S1 is as follows: Step S11: Capture a continuous video stream using a camera to obtain video stream matrix data; Step S12: Extract abnormal behavior events based on the video stream matrix data to obtain abnormal behavior event data; Step S13: Perform inference and enhancement based on the abnormal behavior event data to obtain seamless public place detection data. In step S12, the abnormal behavior event extraction is performed using a preset abnormal behavior event identification model. The specific steps for constructing the preset abnormal behavior event identification model are as follows: Obtain historical video stream matrix data and historical abnormal behavior event labels; perform clustering calculations on the historical video stream matrix data to obtain historical video stream matrix clustering data; perform balanced classification based on the historical video stream matrix clustering data to obtain historical video stream test data and historical video stream verification data; perform preliminary convolution calculations based on the historical video stream test data to obtain preliminary video stream feature data; Pooling and multi-layer self-attention calculations are performed on the initial video stream feature data to obtain video stream pooled data and video stream weighted feature data, respectively. Fully connected encoding calculations are then performed on the video stream weighted feature data and video stream pooled data to obtain fully connected video stream data. An exponential calculation is performed on the fully connected video stream data, and mapping is performed using historical abnormal behavior event labels to obtain initial abnormal behavior event identification data. The initial abnormal behavior event identification data is then iteratively trained using historical video stream verification data to obtain the abnormal behavior event identification model. The multi-layer self-attention calculation is specifically as follows: The process involves performing multi-scale hierarchical convolution and spatiotemporal feature segmentation on the initial video stream feature data to obtain multi-scale feature data. Local and global multi-head hierarchical self-attention calculations are then performed on the multi-scale feature data to obtain local and global self-attention feature data, respectively. A relative position matrix is ​​constructed from the local and global self-attention feature data to obtain a multi-head self-attention feature matrix. Multi-layer graphs are then constructed based on the multi-head self-attention feature matrix to obtain multi-layer graph data. Global-local graph convolution calculations are performed on the multi-layer graph data to obtain global-local graph convolution data. Finally, multi-scale feature fusion is performed on the global-local graph convolution data to obtain weighted feature data. The construction of multi-layer graphs is specifically as follows: The video stream is deconstructed into a hierarchical multi-scale graph based on its multi-head self-attention feature matrix data, resulting in hierarchical multi-scale graph data. Cross-layer association nodes are constructed from this graph data, yielding cross-layer association node data. Association features are extracted from these nodes, resulting in cross-layer association feature data. Gaussian convolution kernels are mapped from this feature data, resulting in Gaussian convolution kernel data. Graph convolution is then applied to the cross-layer association node data based on the Gaussian convolution kernel data, resulting in cross-layer convolutional layer data. Finally, cross-layer propagation processing is performed on the cross-layer convolutional layer data, resulting in multi-layer graph data.

2. The method according to claim 1, characterized in that, Step S2 is as follows: Step S21: Extract dynamic perception features of visual crowds based on the non-intrusive detection data in public places to obtain dynamic perception feature data of visual crowds; Step S22: Extract audio synchronization data based on the visual crowd dynamic perception feature data and video stream matrix data to obtain audio synchronization data; Step S23: Perform audio emotion analysis based on the audio synchronization data to obtain audio emotion analysis data; Step S24: Perform multimodal fusion of visual crowd dynamic perception feature data and audio emotion analysis data to obtain multimodal feature data of security threats.

3. The method according to claim 2, characterized in that, The audio emotion analysis is processed using a pre-defined audio emotion analysis model, which is constructed using the following steps: Acquire historical audio synchronization data and historical audio emotion tag data; The historical audio frequency domain data is obtained by performing frequency domain conversion based on the historical audio synchronization data; The historical audio frequency domain data is divided to obtain historical audio test data and historical audio verification data; The initial state is constructed based on historical audio test data to obtain historical audio initial state data; Based on the historical audio initial state data, the input layer and forgetting layer are calculated to obtain the audio input layer data and audio forgetting layer data, respectively. Multi-head emotion candidate memory state processing is performed based on audio input layer data and audio forgetting layer data to obtain audio candidate memory state data. The audio candidate memory state data is processed by the output layer to obtain the audio hidden state data; By iteratively training the audio hidden state data using historical audio verification data and historical audio emotion label data, an audio emotion parsing model is obtained. The processing of multi-headed emotion candidate memory states includes: Based on the audio input layer data and the audio forgetting layer data, multi-dimensional candidate memory state processing is performed to obtain the initial audio candidate memory state data. Multi-head emotion attention calculation is performed on the initial audio candidate memory state data to obtain audio multi-head emotion attention matrix data; Emotional memory enhancement was performed on audio multi-head emotion attention matrix data to obtain audio enhanced memory state data; The initial audio candidate memory state data is fed back to the memory state data by using the audio-enhanced memory state data, and the feedback memory state data is obtained. Based on the feedback memory state data, multi-level candidate state aggregation is performed to obtain audio candidate memory state data.

4. The method according to claim 1, characterized in that, Step S3 is as follows: Step S31: Perform preliminary screening of security threat multimodal features based on the security threat multimodal feature data to obtain threat preliminary screening index data; Step S32: Allocate risk weights based on the initial threat screening index data to obtain threat weight matrix data; Step S33: Construct a contextualized risk model based on the threat weight matrix data to obtain a public safety contextualized risk model; Step S34: Perform risk classification based on the public safety contextualized risk model to obtain security threat event level data; Step S35: Determine the event level threshold based on the security threat event level data to obtain the event level determination table data; Step S36: Classify the multimodal feature data of security threats into event levels based on the event level determination table data to obtain integrated event level data; Step S37: Aggregate risk events from the event level integration data to obtain security threat risk event data.

5. The method according to claim 1, characterized in that, Step S4 is as follows: Step S41: Classify and schedule risk events based on security threat risk event data to obtain risk event scheduling mapping data; Step S42: Match response strategies based on risk event scheduling mapping data to obtain response strategy data; Step S43: Dynamically trigger early warning based on response strategy data to obtain response control early warning data; Step S44: Perform multi-perspective dynamic risk assessment based on response control and early warning data to obtain perspective risk assessment matrix data; Step S45: Predict response consequences based on the perspective risk assessment matrix data to obtain response consequence prediction data; Step S46: Optimize the corresponding consequence prediction data to obtain optimized response control and early warning data for auxiliary operations in public safety situation prediction.

6. A public safety situation prediction system based on multimodal data fusion, characterized in that, For executing the public security situation prediction method based on multimodal data fusion as described in claim 1, the public security situation prediction system based on multimodal data fusion includes: The Public Places Seamless Detection Module is used for continuous video surveillance of public places and seamless detection, and to obtain seamless detection data of public places. The security threat multimodal feature extraction module is used to extract multimodal data features from non-intrusive detection data in public places to obtain security threat multimodal feature data. The security threat risk event classification module is used to perform situational assessment on multimodal feature data of security threats, obtain security threat event level data, and classify the multimodal feature data of security threats into event levels based on the security threat event level data to obtain security threat risk event data. The public safety situation prediction auxiliary module is used to automatically respond to and control security threat and risk event data, obtain response control early warning data, and perform multi-perspective evaluation and optimization based on the response control early warning data to obtain response control early warning optimized data, so as to assist in public safety situation prediction operations.

Images