Public security situation prediction method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of public security monitoring, in particular to a public security situation prediction method and system based on multi-modal data fusion. The method comprises the following steps: performing continuous video monitoring and non-inductive discovery on a public place to obtain non-inductive discovery data of the public place; performing multi-modal data feature extraction on the public place non-inductive discovery data to obtain security threat multi-modal feature data; performing situation assessment on the security threat multi-modal feature data to obtain security threat event level data, and performing event level division on the security threat multi-modal feature data according to the security threat event level data to obtain security threat risk event data; and performing automatic response control according to the security threat risk event data to obtain response control early warning data, and performing multi-view evaluation optimization according to the response control early warning data to obtain response control early warning optimization data. According to the invention, the potential security situation can be predicted, and the level of public security management is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of public safety monitoring, and in particular to a public safety situation prediction method and system based on multimodal data fusion. Background Art

[0002] With the rapid development of Internet technology and mobile terminal products, video surveillance systems have been widely deployed in strategic locations such as home security, public transportation, banks, ATM centers, city centers, airports and public roads, playing an important role in protecting public safety. At the same time, video surveillance technology is also constantly improving, from traditional analog monitoring and digital monitoring to the current video surveillance system. Today, video surveillance systems have problems such as difficulty in obtaining data, difficulty in assessing security trends, frequent security vulnerabilities, and difficulty in implementing technology. Therefore, how to effectively evaluate and predict public security trends through video surveillance has become an urgent problem to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a public safety situation prediction method and system based on multimodal data fusion to solve at least one of the above technical problems.

[0004] The present application provides a public safety situation prediction method based on multimodal data fusion, comprising the following steps:

[0005] Step S1: Continuously monitor the public places through video and perform non-sensing detection to obtain non-sensing detection data of public places;

[0006] Step S2: extracting multimodal data features from the public place non-sensing discovery data to obtain multimodal feature data of security threats;

[0007] Step S3: Perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data;

[0008] Step S4: Automatically respond to and control the security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to assist in public security situation prediction operations.

[0009] The present invention uses a deduction enhancement algorithm to supplement and infer information missing in the video due to environmental factors such as occlusion and insufficient light, thereby improving the accuracy of security event identification. At the same time, the multimodal data fusion method improves the adaptability and effectiveness of the system in a changing environment. The system can also predict potential security situations based on historical data, and divide the security situation into four levels (level one to level four), and design corresponding response mechanisms for each level, further improving the level of public safety management.

[0010] Preferably, step S1 specifically includes:

[0011] Step S11: Capture a continuous video stream through a camera to obtain video stream matrix data;

[0012] Step S12: extracting abnormal behavior events according to the video stream matrix data to obtain abnormal behavior event data;

[0013] Step S13: Perform deduction and enhancement based on the abnormal behavior event data to obtain unconscious discovery data in public places.

[0014] In the present invention, the system can continuously monitor the real-time dynamics by capturing the video stream of the public place through the camera, and ensure that every subtle change in the place is captured. The real-time captured video stream matrix data can quickly provide the latest scene information, lay the foundation for abnormal behavior detection and response, and improve the overall response speed and monitoring coverage of the system. According to the video stream matrix data, abnormal behavior events can be extracted to quickly identify potential security threats in the place. By focusing on the characteristics of abnormal events, the system can separate normal activities from abnormal activities, effectively avoiding false alarms and missed reports. The abnormal behavior event extraction module can automatically identify abnormal behavior characteristics such as gathering, running, and quarreling, significantly improve the accuracy of threat detection, and help identify safety hazards in the first time. Deduction enhancement can be used to infer and complete the video data when it is blocked or the light is poor, thereby enhancing the integrity of event detection. The missing behavior information is restored through time series prediction, making the monitoring data more continuous and complete. Deduction enhancement compensates for the integrity of abnormal behavior, so that the system can still accurately identify threat events when monitoring is blocked or data is missing, thereby improving the reliability of monitoring. Through the senseless discovery technology, continuous monitoring is achieved without interfering with the activities of people in public places. Compared with traditional explicit monitoring methods, non-contact detection reduces interference to people in the venue and is suitable for high-traffic or sensitive areas.

[0015] Preferably, the abnormal behavior event extraction in step S12 is performed by using a preset abnormal behavior event recognition model, and the steps of constructing the preset abnormal behavior event recognition model are specifically as follows:

[0016] Obtain historical video stream matrix data and historical abnormal behavior event labels;

[0017] Perform clustering calculation on the historical video stream matrix data to obtain historical video stream matrix clustering data;

[0018] Perform balanced classification based on historical video stream matrix clustering data to obtain historical video stream test data and historical video stream verification data;

[0019] Perform preliminary convolution calculations based on historical video stream test data to obtain preliminary video stream feature data;

[0020] Perform pooling calculation and multi-layer self-attention calculation on the preliminary video stream feature data to obtain video stream pooling data and video stream weighted feature data respectively;

[0021] Perform encoding full connection calculation according to the weighted feature data of the video stream and the pooled data of the video stream to obtain the full connection data of the video stream;

[0022] Perform index calculation based on the fully connected data of the video stream and map it through the historical abnormal behavior event labels to obtain preliminary abnormal behavior event recognition data;

[0023] The historical video stream verification data is used to iteratively train the preliminary abnormal behavior event recognition data to obtain the abnormal behavior event recognition model.

[0024] In the present invention, by obtaining historical video stream matrix data and abnormal behavior event labels, the model can learn a large number of known behavior features and event types during the construction process. The abnormal behavior features are supervised by using label data, so that the model can more accurately identify and distinguish different types of abnormal behaviors in practical applications, thereby improving the accuracy of detection. The clustering calculation step groups the historical video stream data, which can better understand the inherent structure of the video data and help the model capture abnormal behavior features of different categories. The test data and verification data generated by balanced classification are representative, which helps to solve the problem of uneven data distribution, so that the model performs better when dealing with minority abnormal behavior events and improves the generalization ability of the model. The preliminary convolution calculation extracts the basic features of the video stream data and simplifies the original data into more representative feature data, thereby effectively reducing the computational complexity.

[0025] Pooling calculations further reduce the dimension of feature data, retain important feature information, and reduce the amount of model calculations, so that the model can improve training and prediction efficiency while ensuring accuracy. Multi-layer self-attention calculations can capture long-term dependencies between video frames, helping the model identify abnormal events in complex behavioral scenarios. The self-attention mechanism helps the model dynamically focus on important information at different time points, especially giving higher weights to abnormal changes in certain key frames, which improves the model's sensitivity to subtle abnormal behaviors. The encoding fully connected layer integrates the weighted feature data and pooled data of the video stream to form a higher-dimensional feature space, allowing the model to more comprehensively understand the characteristics of abnormal behavior events, enhance the model's expressiveness, and ensure accuracy in complex behavioral scenarios.

[0026] By calculating the exponentials of the fully connected data, the abnormal behavior feature data can be converted into different probability distributions, making the model output more interpretable and continuous. Label mapping associates the calculation results with the specific abnormal behavior type, so that the model can give a clear abnormal behavior type. The preliminary recognition data is iteratively trained using the validation data to gradually optimize the model parameters and improve the recognition accuracy and model stability. The iterative training method ensures that the performance of the model on the training set and the validation set is more consistent, enhances the robustness of the model in different scenarios, and reduces overfitting. The trained model can be run on real-time video streams to quickly identify abnormal events and provide support for automated public safety response.

[0027] Preferably, the multi-layer self-attention calculation is specifically as follows:

[0028] Perform multi-scale layered convolution and spatiotemporal feature segmentation on the preliminary video stream feature data to obtain multi-scale feature data of the video stream;

[0029] Perform local multi-head hierarchical self-attention calculation and global multi-head hierarchical self-attention calculation on the multi-scale feature data of the video stream to obtain local self-attention feature data of the video stream and global self-attention feature data of the video stream respectively;

[0030] The local self-attention feature data of the video stream and the global self-attention feature data of the video stream are used to construct a relative position matrix to obtain the multi-head self-attention feature matrix data of the video stream;

[0031] Construct a multi-layer graph based on the multi-head self-attention feature matrix data of the video stream to obtain the multi-layer graph data of the video stream;

[0032] Perform global and local graph convolution calculations based on multi-layer graph data of the video stream to obtain global and local graph convolution data of the video stream;

[0033] Multi-scale feature fusion is performed based on the global and local graph convolution data of the video stream to obtain the weighted feature data of the video stream.

[0034] In the present invention, multi-scale layered convolution and spatiotemporal feature segmentation are used to extract different scale features of the video stream at multiple levels, capturing subtle and large-scale behavioral changes in the video. Local multi-head self-attention calculation captures detailed behavioral changes in the video stream, while global multi-head self-attention captures overall behavioral trends. The combination of the two improves the model's sensitivity to details and the whole. The relative position matrix adds position information to the local and global self-attention feature data of the video stream, allowing the model to capture behavioral changes at different time and space positions. Through the construction of the position matrix, the model can combine the specific position relationship of time and space during analysis, thereby more accurately locating and understanding abnormal behavior events. Based on multi-layer graph construction, the multi-head self-attention feature matrix of the video stream is converted into graph structure data, and the long-term and short-term dependencies between nodes can be captured through graph convolution. Global and local graph convolution calculations are performed in the multi-layer graph, so that the model can simultaneously process global dependencies and local differences in behavioral features. The global and local graph convolution operation strengthens the dynamic feature expression in the video, allowing the model to accurately identify relatively stable abnormal behavior features (such as long-term aggregation) and short-term sudden behaviors (such as running, collision, etc.). By weighted fusion of multi-scale features, the model can simultaneously analyze large-scale collective behaviors and small-scale individual behaviors in the video, achieving a comprehensive understanding of abnormal behaviors. Weighted feature data has strong expressiveness and discrimination in abnormal behavior classification, which helps to achieve more accurate abnormal behavior identification and response.

[0035] Preferably, the multi-layer graph is constructed as follows:

[0036] According to the multi-head self-attention feature matrix data of the video stream, a hierarchical multi-scale graph is deconstructed to obtain the hierarchical multi-scale graph data of the video stream;

[0037] Constructing cross-layer associated nodes for hierarchical multi-scale graph data of video streams to obtain cross-layer associated node data of video streams;

[0038] Extracting correlation features based on cross-layer correlation node data of the video stream to obtain cross-layer correlation feature data of the video stream;

[0039] Gaussian convolution kernel mapping is performed according to the cross-layer correlation feature data of the video stream to obtain Gaussian convolution kernel data of the video stream;

[0040] Perform graph convolution processing on the cross-layer associated node data of the video stream according to the Gaussian convolution kernel data of the video stream to obtain the cross-layer convolution layer data of the video stream;

[0041] The cross-layer convolutional layer data of the video stream is processed across layers to obtain multi-layer graph data of the video stream.

[0042] The hierarchical multi-scale graph deconstruction in the present invention converts the self-attention feature matrix of the video stream into a graph structure of different levels, so that the model can pay attention to different behavioral features in the video in a hierarchical and scaled manner. The cross-layer association node construction introduces cross-layer association nodes in the hierarchical graph structure, so that the model can express the relationship between long-term and short-term behaviors through the data communication of different layers. Through the correlation feature extraction of cross-layer association node data, the model can automatically extract the information dependency relationship between cross-layer nodes, forming a graph-based spatiotemporal correlation feature. The Gaussian convolution kernel mapping makes the distribution of cross-layer association feature data more uniform and continuous through smoothing and weighting processing, ensuring that the feature extraction process in the graph structure is smoother. The graph convolution processing based on Gaussian convolution kernel data can not only capture the global features in the graph structure, but also identify the local differential behavioral characteristics of the nodes. The transmission of cross-layer convolution layer data enables the feature information after each layer of graph convolution to be continuously propagated and shared in different layers, ensuring that the model maintains information consistency in each layer of graph. Multi-layer graph data covers multi-dimensional spatiotemporal information, which enables the model to have cross-layer and cross-scale spatiotemporal expression capabilities, enhances the adaptability of the model, and is suitable for long-term monitoring and detection of various abnormal behaviors in complex environments.

[0043] Preferably, step S2 specifically includes:

[0044] Step S21: extracting visual crowd dynamic perception features based on the public place non-sensing discovery data to obtain visual crowd dynamic perception feature data;

[0045] Step S22: extracting audio synchronization data according to the visual crowd dynamic perception feature data and the video stream matrix data to obtain audio synchronization data;

[0046] Step S23: performing audio emotion analysis according to the audio synchronization data to obtain audio emotion analysis data;

[0047] Step S24: Perform multimodal fusion on the visual crowd dynamic perception feature data and the audio emotion analysis data to obtain security threat multimodal feature data.

[0048] In the present invention, by extracting crowd dynamic perception features from visual information in the senseless discovery data, the behavioral features of the crowd such as gathering, dispersing, and moving can be accurately captured, and the dynamic changes of the crowd in space and time can be analyzed. According to the audio synchronization data extracted from the visual crowd dynamic perception features and the video stream matrix data, the model can establish the consistency of time and scene between the visual and audio data. Emotional analysis based on audio synchronization data can capture emotional changes in the crowd (such as anxiety, panic, anger, etc.), and provide additional emotional information for the judgment of abnormal events. Multimodal fusion of visual crowd dynamic features and audio emotional analysis data helps the model to comprehensively analyze the features of video and audio, so as to more accurately judge potential threats in public places. The spatiotemporal correlation enables the model to identify behavioral patterns and emotional fluctuations across different time periods, thereby establishing a stronger association between events occurring at a specific moment and the background environment, and improving the ability to identify persistent threats. The multimodal fusion of vision and audio provides more refined event features, which helps the model to judge the real threats in specific situations, such as sudden panic among the crowd or abnormal activities of individuals.

[0049] Preferably, the audio emotion analysis is processed by a preset audio emotion analysis model, and the construction method of the audio emotion analysis model includes the following steps:

[0050] Obtain historical audio synchronization data and historical audio emotion label data;

[0051] Perform frequency domain conversion according to historical audio synchronization data to obtain historical audio frequency domain data;

[0052] Perform data division on the historical audio frequency domain data to obtain historical audio test data and historical audio verification data;

[0053] Construct an initial state according to historical audio test data to obtain historical audio initial state data;

[0054] Perform input layer calculation and forgetting layer calculation according to the historical audio initial state data to obtain audio input layer data and audio forgetting layer data respectively;

[0055] Perform multi-head emotion candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain audio candidate memory state data;

[0056] Perform output layer calculation on the audio candidate memory state data to obtain the audio hidden state data;

[0057] The audio hidden state data is iteratively trained using historical audio verification data and historical audio emotion label data to obtain an audio emotion parsing model;

[0058] The multi-head emotion candidate memory state processing includes:

[0059] Perform multi-dimensional candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain initial audio candidate memory state data;

[0060] Perform multi-head emotional attention calculation on the initial audio candidate memory state data to obtain audio multi-head emotional attention matrix data;

[0061] Perform emotional memory enhancement on audio multi-head emotional attention matrix data to obtain audio enhanced memory state data;

[0062] Performing memory state feedback on the initial audio candidate memory state data using the audio enhancement memory state data to obtain feedback memory state data;

[0063] Multi-level candidate state aggregation is performed according to the feedback memory state data to obtain audio candidate memory state data.

[0064] In the present invention, by acquiring historical audio synchronization data and emotion tags, the model can be trained based on a large amount of labeled emotion data, so that it has accurate emotion recognition capabilities. Frequency domain conversion captures emotional information in different frequency bands by extracting the frequency components of the audio signal, such as low-frequency intonation, emotional fluctuations and high-frequency tension and anxiety signals. The initial state is constructed according to the test data, so that the model has a preliminary emotional information baseline before entering the input and forgetting calculations. The input layer calculation and the forgetting layer calculation respectively control which emotional features need to be updated and which can be forgotten, and establish a dynamic management mechanism for changes in emotional states. Through multi-head emotion candidate memory state processing, the model can simultaneously focus on the characteristics of different emotional dimensions (such as anger, anxiety, calmness, etc.). The hidden state, as the output of the model, reflects the overall emotional state of the audio, making the emotion prediction clearer and more expressive, and providing higher quality data for emotion tag matching. Iterative training gradually improves the performance of the model on the validation set, helps to enhance the adaptability of the model in different environments and audio scenes, and ensures the reliability of emotion analysis.

[0065] The multi-dimensional candidate memory state processing decomposes the audio input layer and the forgetting layer data into multi-dimensional emotional features, so that the model can analyze the performance of different emotions (such as anger, anxiety, calmness, etc.) in the audio signal separately. The multi-head emotional attention mechanism enables the model to focus on the parts with important emotional information in the candidate memory state and capture subtle emotional features through different attention heads. The emotional memory enhancement focuses on enhancing the emotional features in the multi-head emotional attention matrix, making the model more sensitive to abnormal emotional changes. The memory state feedback mechanism feeds back the enhanced emotional state to the initial candidate memory state data, realizing the secondary optimization of the emotional features. The aggregation operation enables the model to extract core emotional features from emotional data of different dimensions, avoiding the loss of emotional information during the transmission process and improving the integrity of emotional analysis. In the process of emotional memory enhancement and feedback, the model can retain and strengthen the high-intensity emotional information in the emotional features, which is suitable for detecting drastic emotional fluctuations. The cyclic processing of feedback memory state data builds a feedback loop, which enables the model to dynamically adjust the expression of emotional features and make it more adaptable to the changing emotional environment. The audio candidate memory state data after multi-level aggregation retains multi-level emotional feature information, so that the output emotional state data has rich emotional expression.

[0066] Preferably, step S3 specifically includes:

[0067] Step S31: Performing a preliminary screening of the multimodal features of security threats according to the multimodal feature data of security threats to obtain preliminary screening indicator data of threats;

[0068] Step S32: Allocate risk weights according to the threat initial screening indicator data to obtain threat weight matrix data;

[0069] Step S33: constructing a situational risk model according to the threat weight matrix data to obtain a public safety situational risk model;

[0070] Step S34: Perform risk classification according to the public security scenario risk model to obtain security threat event level data;

[0071] Step S35: performing event level threshold determination according to the security threat event level data to obtain event level determination table data;

[0072] Step S36: classifying the security threat multimodal feature data into event levels according to the event level determination table data to obtain event level integrated data;

[0073] Step S37: Aggregate the event level integrated data into risk events to obtain security threat risk event data.

[0074] In the present invention, by preliminarily screening the multimodal feature data of security threats, the system can eliminate irrelevant or low-risk features and focus on the key features that constitute security threats. Risk weights are allocated based on the results of the initial threat screening. By assigning weights to different features, the system can distinguish and quantify different types of threats. The situational risk model constructed based on the weight matrix data can link threat features with actual situations, so that the model can dynamically adjust the risk assessment criteria according to the characteristics of different scenarios. Different threats are graded by the situational risk model, so that the system can distinguish threats of different levels, thereby achieving hierarchical management. Event level threshold determination sets clear threat level standards, so that the system can adjust the flexibility of risk grading according to specific situational conditions or security policies. The structured expression of data after level division helps to improve the system's processing efficiency for different risk events, and provides a clear risk level basis for event aggregation and early warning strategies. By aggregating the event level integrated data, events of the same or similar levels are integrated together, reducing redundant data and simplifying the system's response process.

[0075] Preferably, step S4 is specifically:

[0076] Step S41: classify and schedule risk events according to the security threat risk event data to obtain risk event scheduling mapping data;

[0077] Step S42: matching response strategies according to the risk event scheduling mapping data to obtain response strategy data;

[0078] Step S43: Dynamically triggering early warning according to the response strategy data to obtain response control early warning data;

[0079] Step S44: performing multi-view dynamic risk assessment according to the response control warning data to obtain view risk assessment matrix data;

[0080] Step S45: performing response consequence prediction according to the perspective risk assessment matrix data to obtain response consequence prediction data;

[0081] Step S46: Response optimization is performed on the corresponding consequence prediction data to obtain response control warning optimization data to perform public safety situation prediction auxiliary operations.

[0082] The present invention classifies and dispatches risk events according to their level, type and scope of impact, so that the system can intelligently determine the priority of various events and allocate resources accordingly. Response strategy matching enables the system to select the most appropriate intervention measures according to the specific nature of the event, thereby achieving targeted early warning and control, and enhancing the effectiveness of the response. Dynamic early warning ensures that the system can flexibly respond to the rapid changes of events, trigger early warning measures in real time, and enable security managers to obtain event update information in a timely manner, so as to take timely actions. Multi-perspective dynamic risk assessment analyzes security threat events from multiple perspectives (such as security personnel, emergency managers, and ordinary users) to generate multi-dimensional evaluation matrix data. Response consequence prediction can estimate the effectiveness and impact of different strategies, helping managers to choose the best solution before executing the response, thereby reducing risks and adverse effects. The optimized response control early warning data contains evaluated and improved response strategies and early warning information, which can more accurately reflect the actual risks of the event. The system can automatically complete the entire process from event identification to response, greatly improving the efficiency and intelligence level of public safety management, reducing the burden on managers, and improving the overall level of safety management.

[0083] Preferably, the present application also provides a public security situation prediction system based on multimodal data fusion, which is used to execute the public security situation prediction method based on multimodal data fusion as described above, and the public security situation prediction system based on multimodal data fusion includes:

[0084] The public place non-sensing detection module is used to continuously monitor public places through video and perform non-sensing detection to obtain the public place non-sensing detection data;

[0085] The security threat multimodal feature extraction module is used to extract multimodal data features from the public place non-sensing discovery data to obtain security threat multimodal feature data;

[0086] The security threat risk event classification module is used to perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data;

[0087] The public security situation prediction auxiliary module is used to perform automatic response control based on security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to perform public security situation prediction auxiliary operations.

[0088] The beneficial effects of the present invention are:

[0089] 1. Combine the visual crowd dynamic perception data captured by the camera with the audio emotion analysis engine captured by the sensor to further identify anomalies in the video, divide the security level according to the detected features, and make corresponding responses.

[0090] 2. According to the characteristics of different security incidents, the system divides them into four levels (level 1 to level 4) of security situation. A corresponding response mechanism is designed for each level, from recording incidents to emergency response, to ensure that appropriate actions are taken for security incidents of different risk levels.

[0091] 3. Design a security situation indicator system based on different roles to better evaluate and manage the security status and improve the security and reliability of the system.

[0092] 4. Build a security situation awareness and prediction model that integrates dense crowd detection and voice sentiment analysis, which can learn and identify specific security risk patterns, predict security situations, and improve the assessment, prediction, and response capabilities of public security monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting implementations made with reference to the following drawings:

[0094] Figure 1 A flowchart of a method for public safety situation prediction based on multimodal data fusion according to an embodiment is shown;

[0095] Figure 2 A flowchart showing a method for detecting a public place without any sense of presence according to an embodiment of the present invention is shown;

[0096] Figure 3 A flowchart of a method for extracting multimodal features of security threats according to an embodiment of the present invention is shown;

[0097] Figure 4 A flowchart showing a method for classifying security threat risk events according to an embodiment of the present invention is provided;

[0098] Figure 5 A flowchart of the steps of a method for classifying security threat risk events according to an embodiment is shown. DETAILED DESCRIPTION

[0099] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0100] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0101] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0102] The present invention proposes a public safety situation assessment and prediction system based on deep learning and multimodal data fusion, which is divided into six parts: research on sensorless discovery technology, analysis of monitoring equipment, development of security assessment technology, design of security situation indicator system, construction of security situation awareness model, and integration and optimization of control system.

[0103] First, we use the sensorless detection technology to capture a continuous video stream through the camera. The video stream can be represented by a matrix, and the sequence of video frames is Among them I t Represents the tth frame image. Then, image recognition algorithms such as convolutional neural networks are used to analyze these video data to identify abnormal behaviors or events. Specifically, the convolution operation can be expressed as:

[0104]

[0105] Among them I t is the input image, W is the convolution kernel, and (x, y) is the pixel coordinate.

[0106] The inference enhancement algorithm is then used to complete and infer the missing information in the video due to occlusion, insufficient light or other environmental factors. The inference enhancement algorithm can be expressed as a prediction model f:

[0107]

[0108] in, is the predicted future frame, I t ,I t-1 ,…,I t-nThe image is the past t to n frames

[0109] On the one hand, the technology can accurately identify real security threats from massive video data and effectively avoid false positives and false negatives. On the other hand, the algorithm must be able to adapt to different monitoring environments, such as changing lighting conditions, complex backgrounds, and camera angle changes. The ability to adapt to different monitoring environments can be achieved by enhancing the robustness of the model, for example, by introducing the illumination change model L when training the model:

[0110]

[0111] Among them, I' t is the image processed by the illumination change model, L is the illumination change model, and θ represents the illumination change parameter.

[0112] Through these technical means, the system can effectively detect anomalies in complex environments and improve the reliability and accuracy of the security monitoring system.

[0113] Then, the visual crowd dynamic perception data captured by the camera is combined with the audio emotion analysis engine captured by the sensor to further identify anomalies in the video, and classify the security level according to the detected features and make corresponding responses. The video frame sequence is The audio frame sequence is

[0114] The crowd dynamics perception in videos can be processed by convolutional neural networks (CNN), such as the convolution operation mentioned above:

[0115]

[0116] The audio emotion analysis engine can analyze the emotional fluctuations of audio signals through the long short-term memory network (LSTM):

[0117] h t =σ(W h ·A t +U h ·h t-1 +b n );

[0118] Among them, h t Represents the hidden state at the current moment, A t is the audio signal, W h and U h is the weight matrix, b n is the bias and σ is the activation function.

[0119] Multimodal fusion of visual and audio data can be achieved through weighted fusion:

[0120] Ft =α·f v (I t )+β·f a (A t );

[0121] Among them, F t represents the fused features, f v represents the visual feature extraction function, f a represents the audio feature extraction function, α and β are the fusion weights.

[0122] In order to achieve security level classification, the video and audio data are continuously monitored for a period of time. By continuously monitoring and recording event data, the system can learn and identify specific security risk patterns and compare them with previous security event data to identify important security event nodes, that is, the thresholds N1, N2, N3, E1, E2, and E3 of each risk level. Assume that we monitor data in the time period [0, T] and count the distribution of each parameter:

[0123] N1=max(n c |n c is low risk);

[0124] N2=max(n c |n c is medium risk);

[0125] N3=max(n c |n c is high risk);

[0126]

[0127] where n C Indicates the number of people, N1 is the low risk threshold, N2 is the medium risk threshold, and N3 is the high risk threshold. t ) is the emotional fluctuation value, E1 is the low-risk emotional fluctuation threshold, E2 is the medium-risk emotional fluctuation threshold, and E3 is the high-risk emotional fluctuation threshold.

[0128] According to the detected feature F t , the system will classify the security level and make corresponding responses. The security level is divided as follows:

[0129] Level 1 safety incident (low risk) RISK1 involves a crowd of people watching and then dispersing, and no significant emotional fluctuations are detected in the voice analysis. It can be expressed as:

[0130] RISK1={n C ≤N1,|E(A t)|≤E1};

[0131] where n C represents the number of people, N1 is the low risk threshold, E(A t ) is the emotional fluctuation value, and E1 is the low-risk emotional fluctuation threshold. At this point, the system will record the event and continue monitoring, and no immediate action is required.

[0132] Level 2 security incident (medium risk) RISK2 is manifested by the continued gathering of people and the increase in number, and slight emotional fluctuations such as small-scale quarrels are detected. It can be expressed as:

[0133] RISK2={N1≤n C ≤N2,E1≤|E(A t )|≤E2};

[0134] At this time, the system should increase the monitoring frequency and notify security personnel to prepare for intervention.

[0135] Level 3 safety incident (high risk) RISK3: Significant abnormal behavior occurs, such as people falling to the ground or physical conflict, accompanied by obvious tension or panic. It can be expressed as:

[0136] RISK3={N2≤n C ≤N3,E2≤|E(A t )|≤E3};

[0137] At this time, the system needs to immediately notify on-site security personnel and emergency response teams to intervene.

[0138] Level 4 safety incidents (serious risks) are the most urgent situations, such as large-scale fights or life-threatening medical emergencies, with sustained high-intensity emotional reactions such as screaming, crying, and cries for help. It can be expressed as:

[0139] RISK3={n C >N3,|E(A t )|>E3};

[0140] The system will trigger the highest level of alarm, immediately dispatch security and medical emergency services, and notify management for further action.

[0141] Through this method of combining dense crowd detection and voice emotion analysis, the public security monitoring system can more accurately assess and respond to potential threats, effectively enhancing the security of public places. In addition, through continuous monitoring and obtaining a large amount of audio and video data, it can effectively predict future security incidents and improve response efficiency. Through multimodal data fusion methods, this technology is also committed to improving the adaptability and effectiveness of artificial intelligence systems in changing environments.

[0142] Table 1 Security incident level classification table

[0143]

[0144] Then, we develop security assessment technology and design a security situation indicator system to establish a capability verification-based framework to systematically assess and manage security risks. This assessment technology is mainly divided into three core dimensions: access awareness capability, threat blocking capability, and vulnerability identification capability. Access awareness capability focuses on how quickly and accurately the system can identify and respond to access attempts by external devices or users. Assume that the access request time is t a , the system response time is t r , access perception capability C a It can be expressed as:

[0145]

[0146] Among them, the accuracy of access perception A a It can be expressed as the ratio of correctly identified access requests:

[0147]

[0148] Here, N correct is the number of correctly identified access requests, N total is the total number of access requests.

[0149] Threat blocking capability evaluates the system's ability to defend against potential attacks, including how to effectively identify, isolate, and interrupt malicious activities. Threat blocking time t b It can be expressed by the following formula:

[0150] t b =t detect +t isolate +t mitigate ;

[0151] Among them, t detect is the threat detection time, t isolate is the threat isolation time, t mitigate It’s dangerous interruption time.

[0152] Threat blocking capability C b It can be expressed as:

[0153]

[0154] Similarly, the accuracy of threat blocking A b It can be expressed as the ratio of correct identification and blocking:

[0155]

[0156] Among them, Nblocked is the number of threats successfully blocked, N total threats is the total number of threats.

[0157] Vulnerability identification capability refers to the system's ability to discover and repair security vulnerabilities. Vulnerability discovery time t v and repair time t f It can be expressed as:

[0158]

[0159] Among them, t detect , i is the detection time of the ith vulnerability, t fix，i is the repair time of the ith vulnerability, N v is the total number of vulnerabilities.

[0160] Vulnerability identification ability C v It can be expressed as:

[0161]

[0162] The system designs a security situation indicator system for different roles including system administrators, security analysts and ordinary users to better evaluate the security status.

[0163] System administrators are concerned about the overall system security situation, including access perception, threat blocking, and vulnerability identification capabilities. admin The formula is:

[0164] C admin =α a C a +α b C b +α v C v ;

[0165] Among them, α a , α b , α v is the weight of each capability dimension.

[0166] Security analysts focus on specific threat intelligence and response results, mainly evaluating threat blocking capabilities and vulnerability identification capabilities. Comprehensive indicator C analyst The formula is

[0167] C analyst =β b C b +β b C b ;

[0168] Among them, β b , βv It is the weight of threat blocking capability and vulnerability identification capability.

[0169] System users are concerned about the system's access awareness capability and the system's response speed.

[0170]

[0171] where γ a , γ r It is the weight of access perception capability and system response speed.

[0172] Through these formulas and indicator systems, different users can systematically evaluate and manage security risks, ensuring that the system can respond and handle various security threats quickly and accurately.

[0173] Finally, the above data are integrated into a control system, and the control system is used to collect and process the monitoring data in real time. t ) Design an automatic response mechanism. The response mechanism can be expressed as:

[0174]

[0175] When the system detects abnormal behavior or security threats, it can automatically trigger an alarm or take other measures. The trigger function A(RISK) of the alarm system can be expressed as:

[0176]

[0177] The image data is captured by real-time surveillance cameras, and the controller processes the data to identify potential safety issues and take action through the alarm system.

[0178] See also Figures 1 to 5 , the present application provides a public safety situation prediction method based on multimodal data fusion, comprising the following steps:

[0179] Step S1: Continuously monitor the public places through video and perform non-sensing detection to obtain non-sensing detection data of public places;

[0180] Specifically, multiple high-definition cameras are deployed in public places, and video data is continuously monitored through the cloud control center. After the camera captures the video, the image is transmitted to the processing module. By real-time monitoring of crowd flow, abnormal behavior and the movement trajectory of specific objects, the sensorless detection module screens abnormal situations through pre-defined behavior or feature detection models. Without interfering with activities in public places, AI detection technology is used to identify security threat features such as crowding and fighting, and the identified sensorless data is transmitted to the feature extraction module.

[0181] Step S2: extracting multimodal data features from the public place non-sensing discovery data to obtain multimodal feature data of security threats;

[0182] Specifically, the sensorless discovery data enters the multimodal feature extraction stage. This stage first decomposes the video data into frames and converts them into image format. The image data is input into the feature extraction network together with the data from other sensing devices (such as thermal imagers, sound monitors, etc.). Then, multimodal features are extracted using specific rules or thresholds (such as sudden temperature changes, volume exceeding a set value). Combining target tracking and area division techniques, images from different angles in the video and other modal data are combined to form a data set. These data will be calculated by the feature extraction algorithm to extract spatial features, temporal features, and complementary features between modalities, providing multi-dimensional input data for subsequent situation assessment.

[0183] Step S3: Perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data;

[0184] Specifically, in situation assessment, the system receives multimodal feature data and analyzes its impact on the level of security threats. First, the input feature data is quantified to create a risk score, such as evaluating the severity of the incident through dimensions such as the frequency, location, and time of behavior. Next, through historical risk data and pattern matching, similar events and potential risk levels are identified, and levels are assigned to new events, and security threat event level data is output. In order to achieve accurate event level classification, event association rules are also introduced. For example, if screams accompanied by rapid movement are found in crowded areas, the event level will be increased. These event level data are labeled and marked with different risk levels to generate the final security threat risk event data to ensure that the risk classification meets public safety standards.

[0185] Step S4: Automatically respond to and control the security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to assist in public security situation prediction operations.

[0186] Specifically, based on the risk event data, the system automatically generates a response plan. First, different response measures are triggered based on the event level, such as sending notifications to nearby security or police officers, or automatically starting regional broadcasts. To ensure the effectiveness of the response, the system will adjust the control strategy based on real-time feedback. For example, if a high-risk event is quickly controlled, the event level is reduced through feedback. On the other hand, the system uses multi-view evaluations from other cameras and sensors to optimize the initial response. Through multi-view analysis, the event perspectives provided by each camera are combined into a global view to optimize the initial response. Finally, the system stores the early warning optimization data to assist in future situation predictions.

[0187] Preferably, step S1 specifically includes:

[0188] Step S11: Capture a continuous video stream through a camera to obtain video stream matrix data;

[0189] Specifically, the camera continuously captures a continuous video stream of a public place. A video stream is a collection of time-series images, which the system converts into video stream matrix data. Video frames are sampled at a fixed frame rate, such as 30 frames per second, and each frame is converted into pixel matrix data after grayscale processing. If the image size is 1920x1080 pixels, each frame is represented as a matrix containing the grayscale value of each pixel. In the entire video stream, the matrix data of each frame forms a three-dimensional array with the time axis as the depth to form a complete spatiotemporal video matrix. The system judges the action or movement of the object in the picture by the pixel changes in the matrix, thereby capturing the dynamic information of the time series.

[0190] Step S12: extracting abnormal behavior events according to the video stream matrix data to obtain abnormal behavior event data;

[0191] Specifically, after obtaining the video stream matrix data, the system begins to analyze the video stream for abnormal behavior events. First, through motion detection, dynamic objects appearing in the video stream are extracted, such as detecting object movement by calculating the pixel differences between adjacent frames. Assuming that in a specific frame, the movement pattern of the crowd or the speed of an individual increases abnormally, the system will mark these areas. After that, through the behavior classification model, the movement trajectory, speed, direction and other characteristics of the marked area are analyzed to detect abnormal behavior. For example, if the density of a certain area suddenly increases, or a large-scale crowd flow is detected, the system will record it as abnormal behavior event data. For each detected abnormal behavior event, the system will generate information such as event type (such as running, crowding), location coordinates, and time of occurrence.

[0192] Step S13: Perform deduction and enhancement based on the abnormal behavior event data to obtain unconscious discovery data in public places.

[0193] Specifically, based on the abnormal behavior event data, the system further performs deduction enhancement processing. By comparing the current event data with historical abnormal event patterns, such as the area, time, event type and other information where the event occurred, the system assesses the potential risk of the event. For example, the system detects a congestion event in the entrance and exit area. Combined with historical data analysis, such events were usually accompanied by more serious congestion risks in the past, so the system will increase the risk level. In addition, deduction enhancement will introduce environmental factors, such as trends in on-site temperature and volume data, to predict the direction of event development. Unconscious discovery data includes event type, location, risk assessment level and event development prediction. Alternatively, use a deduction enhancement algorithm to complete and infer missing information in the video due to occlusion, insufficient light or other environmental factors.

[0194] Preferably, the abnormal behavior event extraction in step S12 is performed by using a preset abnormal behavior event recognition model, and the steps of constructing the preset abnormal behavior event recognition model are specifically as follows:

[0195] Obtain historical video stream matrix data and historical abnormal behavior event labels;

[0196] Specifically, collect historical video stream data and ensure that the data comes from different environments and situations. Video stream data includes multiple scenes and behavior types. When collecting video stream data, each video stream sample needs to be marked as a specific abnormal behavior event (such as "running", "fighting", "crowding"), which will serve as the label of the video stream. Through manual annotation or rule-based detection, mark the abnormal behavior corresponding to each video stream segment.

[0197] Perform clustering calculation on the historical video stream matrix data to obtain historical video stream matrix clustering data;

[0198] Specifically, after obtaining the historical video stream matrix data, the data is clustered. The goal of the clustering process is to classify similar video stream samples into the same category to reduce the complexity of the data space. Using a clustering method based on pixel features, the video streams are divided into different groups by comparing the similarity of image features (such as color distribution, edge information, etc.) between different video streams. The key to the clustering process is to identify video clips with similar behaviors or scenes.

[0199] Perform balanced classification based on historical video stream matrix clustering data to obtain historical video stream test data and historical video stream verification data;

[0200] Specifically, data balancing techniques (such as oversampling, undersampling, or data augmentation) are used to ensure that the number of samples of different categories is close, thereby avoiding the model from being biased towards frequently occurring behavior types. Through these balancing processes, test data and validation data are generated.

[0201] Perform preliminary convolution calculations based on historical video stream test data to obtain preliminary video stream feature data;

[0202] Specifically, the initial convolution calculation is the first step of feature extraction in the model, which aims to identify important features in the video through convolution operations. Through convolution operations, key low-level features (such as edges, textures, etc.) are extracted from each frame of the image.

[0203] Perform pooling calculation and multi-layer self-attention calculation on the preliminary video stream feature data to obtain video stream pooling data and video stream weighted feature data respectively;

[0204] Specifically, the role of the pooling layer is to extract more important features while reducing the amount of computation by reducing the spatial dimension of the feature map. For example, the maximum pooling operation selects the maximum value in each small area to retain the most significant features. The self-attention mechanism is used to capture the long-distance dependencies between video frames. When processing a video stream of continuous frames, each frame has some association with other frames. The self-attention mechanism enhances frames with stronger correlation by weighted summation between different frames, while ignoring unimportant parts.

[0205] Perform encoding full connection calculation according to the weighted feature data of the video stream and the pooled data of the video stream to obtain the full connection data of the video stream;

[0206] Specifically, after pooling and self-attention calculation, the feature data obtained will be further processed by the fully connected layer. The function of the fully connected layer is to map the extracted features to a higher-dimensional space to help the model learn more complex patterns. Through the fully connected layer, the model can adjust the weights of different behavioral features to optimize the performance of the model.

[0207] Perform index calculation based on the fully connected data of the video stream and map it through the historical abnormal behavior event labels to obtain preliminary abnormal behavior event recognition data;

[0208] Specifically, after obtaining the fully connected data, the purpose of performing index calculation is to evaluate the importance of the behavior pattern. By weighting the fully connected data exponentially, more important features can obtain higher weights. After that, the system maps the features output by the model according to the historical abnormal behavior event labels, and matches them with the preset abnormal behavior types (such as running, fighting, etc.).

[0209] The historical video stream verification data is used to iteratively train the preliminary abnormal behavior event recognition data to obtain the abnormal behavior event recognition model.

[0210] Specifically, the preliminary recognition results are trained through the validation data set through the complete square error function or information entropy loss function to optimize the performance of the model. During the training process, the model will iteratively adjust the weights according to the samples with prediction errors to reduce the error, and obtain a model that can accurately identify abnormal behavior events.

[0211] Preferably, the multi-layer self-attention calculation is specifically as follows:

[0212] Perform multi-scale layered convolution and spatiotemporal feature segmentation on the preliminary video stream feature data to obtain multi-scale feature data of the video stream;

[0213] Specifically, the preliminary video stream feature data is first processed by multi-scale hierarchical convolution to extract details and global information under different behavior modes. Through multi-scale convolution, the system can identify features of different sizes and distances. For example, 3x3, 5x5, and 7x7 convolution kernels are used to scan the video stream features respectively to obtain feature maps of different scales. Next, these feature data are segmented based on the spatiotemporal dimensions (time axis and spatial position) to capture features of different spatiotemporal distributions.

[0214] Perform local multi-head hierarchical self-attention calculation and global multi-head hierarchical self-attention calculation on the multi-scale feature data of the video stream to obtain local self-attention feature data of the video stream and global self-attention feature data of the video stream respectively;

[0215] Specifically, a local multi-head self-attention calculation is performed on the multi-scale feature data, and each head focuses on the features of a specific spatial region to ensure that the detailed features of each local region are highlighted. The features of each local region are weighted through the self-attention mechanism to highlight the key features. After the local calculation, the system performs a global self-attention calculation on the feature data of the entire video stream, capturing global information in a multi-head manner. Each head focuses on different feature combinations to ensure that the model can identify spatiotemporal associations in the global range of the video frame.

[0216] The local self-attention feature data of the video stream and the global self-attention feature data of the video stream are used to construct a relative position matrix to obtain the multi-head self-attention feature matrix data of the video stream;

[0217] Specifically, the local and global self-attention feature data are combined to construct a relative position matrix. The relative position matrix calculates the relative position relationship between each feature point and other feature points to ensure that the self-attention mechanism can effectively perceive spatial relationships. During the construction process, local and global features are fused with different weights and encoded by relative position to form a multi-head self-attention feature matrix.

[0218] Construct a multi-layer graph based on the multi-head self-attention feature matrix data of the video stream to obtain the multi-layer graph data of the video stream;

[0219] Specifically, a multi-layer graph is constructed through a multi-head self-attention feature matrix. Each layer represents the spatial feature connection at a time point. The node is each position feature, and the edge is the weight of the relative position. The layers of each time point are superimposed to generate a multi-layer spatiotemporal graph to capture the evolution of features in the time series. For example, a video stream contains 5 time slices, and the graph generated for each time slice is 100 nodes and 1000 edges (relative position). The multi-layer graph contains 5 layers, and the graph structure of each layer stores the features of different time points.

[0220] Perform global and local graph convolution calculations based on multi-layer graph data of the video stream to obtain global and local graph convolution data of the video stream;

[0221] Specifically, global local graph convolution is performed on multi-layer graphs to capture the spatiotemporal correlation of nodes. Local graph convolution propagates features between local neighbors in each layer to enhance local correlation. Global convolution integrates global features between graph layers to ensure that cross-temporal dependencies are captured. Through this calculation, the model obtains deep correlation of spatiotemporal features.

[0222] Multi-scale feature fusion is performed based on the global and local graph convolution data of the video stream to obtain the weighted feature data of the video stream.

[0223] Specifically, the features after global-local graph convolution are fused at multiple scales to integrate details and global information at different scales. Multi-scale feature fusion combines local and global features in the time dimension by weighted averaging and stacking features to generate weighted feature data. The weight of local features is 0.7, and that of global features is 0.3. The generated weighted feature data is a 20-dimensional feature vector, covering the multi-scale information of the video stream.

[0224] Preferably, the multi-layer graph is constructed as follows:

[0225] According to the multi-head self-attention feature matrix data of the video stream, a hierarchical multi-scale graph is deconstructed to obtain the hierarchical multi-scale graph data of the video stream;

[0226] Specifically, the multi-head self-attention feature matrix of the video stream contains spatial and temporal features. First, a hierarchical multi-scale graph deconstruction is required to understand these features at different scales. The feature matrix is ​​divided into multiple scales, such as three levels of close distance, medium distance and long distance. The features of each scale are divided according to their distance, so that each scale graph structure can represent the feature relationship of a specific distance range. Specifically, the system divides the relative distances between nodes to form three hierarchical subgraphs, each of which contains nodes and edges within a certain range.

[0227] Constructing cross-layer associated nodes for hierarchical multi-scale graph data of video streams to obtain cross-layer associated node data of video streams;

[0228] Specifically, on the basis of hierarchical multi-scale graphs, by constructing cross-layer association nodes, the nodes of graphs of different scales are connected to help the model capture the relationship between different spatiotemporal scales. Each pair of cross-layer association nodes represents the potential association relationship in different spatiotemporal scales, for example, a node in the close distance scale is associated with a node in the medium distance scale. The weight of the cross-layer node pair is calculated based on the feature similarity and distance between the nodes. The higher the weight, the stronger the node association.

[0229] Extracting correlation features based on cross-layer correlation node data of the video stream to obtain cross-layer correlation feature data of the video stream;

[0230] Specifically, based on the cross-layer associated node data, the system extracts features from these associated node pairs to obtain cross-layer feature relationships. The feature differences, similarities, and distance relationships of each pair of associated nodes are extracted to help the model understand the connections between cross-layer nodes. Cross-layer features are processed through weighted averaging, distance normalization, etc., so that the model can effectively perceive cross-layer information.

[0231] Gaussian convolution kernel mapping is performed according to the cross-layer correlation feature data of the video stream to obtain Gaussian convolution kernel data of the video stream;

[0232] Specifically, Gaussian convolution kernel mapping is applied to the cross-layer correlation feature data of the video stream. The Gaussian convolution kernel applies smooth weights to neighboring nodes, making the association between adjacent nodes closer. Through the Gaussian function, weighted values ​​are generated according to the distance between the associated nodes. Nodes with closer distances will receive higher weights, while nodes with farther distances will receive lower weights. This ensures that the model focuses on important local feature relationships and retains key spatiotemporal features.

[0233] Perform graph convolution processing on the cross-layer associated node data of the video stream according to the Gaussian convolution kernel data of the video stream to obtain the cross-layer convolution layer data of the video stream;

[0234] Specifically, graph convolution is performed on the weight data generated by Gaussian convolution kernel mapping to integrate the features of cross-layer related nodes into an overall representation. The graph convolution operation aggregates the features of each node with the features of its neighboring nodes to generate a feature representation containing neighborhood information. Ensure that the features of each node not only contain its own information, but also reflect the feature connection with other hierarchical nodes. The initial feature of node A is [0.2, 0.5, 0.1], and the feature of neighbor node B is [0.072, 0.072, -0.072] after Gaussian convolution mapping. The graph convolution operation performs a weighted summation of the features of A and B, such as 0.6×[0.2, 0.5, 0.1]+0.4×[0.072, 0.072, -0.072], to obtain a new feature representation [0.143, 0.3, 0.04].

[0235] The cross-layer convolutional layer data of the video stream is processed across layers to obtain multi-layer graph data of the video stream.

[0236] Specifically, the cross-convolution layer data after graph convolution will be processed by cross-layer transfer to propagate information between different layers to form integrated multi-layer graph data. The system transfers node information through cross-layer paths and weights to ensure that the node features of each layer can interact with the node features of other layers. Through this cross-layer transfer, the model can integrate the features of all spatiotemporal layers into a unified representation to obtain multi-layer graph data of the video stream.

[0237] Preferably, step S2 specifically includes:

[0238] Step S21: extracting visual crowd dynamic perception features based on the public place non-sensing discovery data to obtain visual crowd dynamic perception feature data;

[0239] Specifically, the system uses an object detection algorithm to identify people in each video frame and calculates the population density per unit area. For example, the population density per square meter is calculated. Changes in density can indicate the state of crowd gathering or evacuation. The system tracks the movement trajectory of each individual or crowd area and calculates its position change in each frame to obtain the movement speed and flow direction of each person or crowd. For example, the average speed difference between individuals is calculated to determine whether there is large-scale crowd flow or congestion. If the system detects a sudden increase in density or an abnormal flow speed in a certain area, it will mark these features as abnormal and generate visual crowd dynamic perception feature data.

[0240] Step S22: extracting audio synchronization data according to the visual crowd dynamic perception feature data and the video stream matrix data to obtain audio synchronization data;

[0241] Specifically, by aligning the timestamps of the video and audio, each video frame is matched with the corresponding audio clip to achieve multimodal synchronization. For example, in a 30-fps video, each second of audio clip will be divided into 30 segments, synchronized with each frame of video. After alignment, the system extracts features from the audio signal, such as volume, frequency, amplitude changes, etc., focusing on abnormal audio such as screams and rapid sounds. The system detects abnormal fluctuations in volume and frequency, and if there is a significant increase in volume or a sharp change in frequency in the audio, it is marked as audio emotion analysis data.

[0242] Step S23: performing audio emotion analysis according to the audio synchronization data to obtain audio emotion analysis data;

[0243] Specifically, the system first extracts emotion-related features from the audio signal, including pitch, volume change rate, timbre, etc. Rapid or high-frequency audio fluctuations usually represent anxiety or nervousness. The audio features are compared with a pre-trained emotion recognition model to classify the audio clips by emotion. For example, rapid pitch increases and high volume fluctuations are often classified as "angry" or "nervous." The intensity of the emotion is scored (0 to 1), and the higher the intensity of the emotion, the greater the potential threat.

[0244] Step S24: Perform multimodal fusion on the visual crowd dynamic perception feature data and the audio emotion analysis data to obtain security threat multimodal feature data.

[0245] Specifically, the system further aligns the visual and audio time segments and assigns different weights according to the nature of the event. Typically, the recognition of crowding behavior is based on visual features, while emotional events rely more on audio emotion data. Features such as crowd density, speed, volume, and emotion category are combined to form a complete multimodal feature vector. For example, when high crowd density is combined with high emotion intensity, the system can identify the possibility of an "emergency situation." On the multimodal feature vector, the system classifies the potential threat level. If the visual features show high crowd density and the audio emotion is interpreted as "angry," the event is marked as "high risk."

[0246] Preferably, the audio emotion analysis is processed by a preset audio emotion analysis model, and the construction method of the audio emotion analysis model includes the following steps:

[0247] Obtain historical audio synchronization data and historical audio emotion label data;

[0248] Specifically, the system needs to obtain a large amount of historical audio data and corresponding emotion labels. The historical audio data includes audio clips of various emotions, such as anger, anxiety, joy, calmness, etc., and each audio clip has an emotional label.

[0249] Perform frequency domain conversion according to historical audio synchronization data to obtain historical audio frequency domain data;

[0250] Specifically, the system performs a short-time Fourier transform on each audio segment, converting the audio waveform data into a spectrogram, where the horizontal axis represents frequency, the vertical axis represents time, and the brightness of each point represents the intensity of the specific frequency at that point in time.

[0251] Perform data division on the historical audio frequency domain data to obtain historical audio test data and historical audio verification data;

[0252] Specifically, in order to train and verify the model, the frequency domain data is divided into test data and verification data, with 80% of the data used as test data and 20% as verification data.

[0253] Construct an initial state according to historical audio test data to obtain historical audio initial state data;

[0254] Specifically, the initial state is usually set to the zero state or initialized randomly so that the model has a consistent starting point.

[0255] Perform input layer calculation and forgetting layer calculation according to the historical audio initial state data to obtain audio input layer data and audio forgetting layer data respectively;

[0256] Specifically, at the input layer of the model, the system inputs the audio data after frequency domain conversion into the model. The calculation of the input layer is mainly responsible for receiving and processing audio features and converting them into feature vectors for emotion analysis. Next, the forgetting layer calculation is used to filter out unimportant audio features and retain information that is valuable for emotion judgment. The forgetting layer filters out key feature information based on the feature vector output by the input layer through a weighted or gating mechanism.

[0257] Perform multi-head emotion candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain audio candidate memory state data;

[0258] Specifically, the audio features are processed in parallel through multiple heads (each head processes a specific emotional feature), and each head focuses on specific emotional information. For example, one head pays more attention to high-frequency changes, while another head focuses on low-frequency features. Through the multi-head mechanism, the system generates candidate memory state data that contains candidate features of different emotions.

[0259] Perform output layer calculation on the audio candidate memory state data to obtain the audio hidden state data;

[0260] Specifically, based on the candidate memory states, the model performs output layer calculations to aggregate multiple emotion features into an implicit feature vector, namely "audio hidden state data". This implicit feature represents a high-level abstraction of audio emotions and is representative of emotions. During the output layer calculation process, the system generates emotion feature data through weighted summation or gating mechanisms.

[0261] The audio hidden state data is iteratively trained using historical audio verification data and historical audio emotion label data to obtain an audio emotion parsing model;

[0262] Specifically, after building the initial model for audio emotion analysis, the system uses historical audio verification data and emotion labels for iterative training. In each iteration, the model will provide feedback and adjustments on the gap between the predicted results and the actual labels, and optimize the calculation weights of each layer. Through multiple iterative training, the model gradually learns emotional features and achieves the effect of accurately analyzing emotions.

[0263] The multi-head emotion candidate memory state processing includes:

[0264] Perform multi-dimensional candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain initial audio candidate memory state data;

[0265] Specifically, by analyzing the feature vectors of the audio input layer and the weighted features of the forgetting layer, a multi-dimensional memory state data structure is created. This structure contains information about dimensions such as different frequencies, volumes, and pitch changes. Specifically, the model decomposes the audio input features, and each dimension (such as frequency range or specific pitch) will get a separate candidate state.

[0266] Perform multi-head emotional attention calculation on the initial audio candidate memory state data to obtain audio multi-head emotional attention matrix data;

[0267] Specifically, the goal of multi-head emotion attention calculation is to deeply process the initial audio candidate memory state through multiple "heads" focusing on different emotion feature dimensions. Each "head" will focus on different audio features, such as one head focusing on high-frequency emotion features and another head focusing on low-frequency features.

[0268] Perform emotional memory enhancement on audio multi-head emotional attention matrix data to obtain audio enhanced memory state data;

[0269] Specifically, based on the multi-head emotion attention matrix, the system performs emotion memory enhancement processing, with the goal of amplifying key emotion features and reducing unimportant information. Memory enhancement weights the emotion attention matrix, focusing on amplifying parts with large emotion fluctuations, such as high-frequency fluctuations in anger or low-frequency fluctuations in anxiety, so that the model pays more attention to strong expressions of emotions.

[0270] Performing memory state feedback on the initial audio candidate memory state data using the audio enhancement memory state data to obtain feedback memory state data;

[0271] Specifically, the memory state is further optimized by enhancing features, making important emotional features more prominent. The feedback mechanism enables the model to more accurately capture emotional changes during the iteration process by weighted combination of the enhanced state and the initial state. The enhanced memory state is weighted combined with the initial memory state to generate the memory state after feedback. The combined features are updated so that the memory state under multi-head attention is more accurate.

[0272] Multi-level candidate state aggregation is performed according to the feedback memory state data to obtain audio candidate memory state data.

[0273] Specifically, the different emotional features of the feedback memory state are aggregated step by step, so that the emotional information of all heads is merged into an overall emotional state. During the aggregation process, the weights of different features are adjusted to avoid a certain feature from excessively affecting the emotional analysis.

[0274] Preferably, step S3 specifically includes:

[0275] Step S31: Performing a preliminary screening of the multimodal features of security threats according to the multimodal feature data of security threats to obtain preliminary screening indicator data of threats;

[0276] Specifically, by setting preliminary screening conditions (such as feature values ​​exceeding a certain threshold), the system filters out noise or unimportant feature data and only retains features that are closely related to security risks based on multiple indicators such as crowd density, audio emotion intensity, temperature, etc.

[0277] Step S32: Allocate risk weights according to the threat initial screening indicator data to obtain threat weight matrix data;

[0278] Specifically, after obtaining the threat indicator data after the initial screening, the system assigns a risk weight to each threat feature. The weight allocation takes into account the degree of impact of each feature on the security threat. For example, crowd density and audio emotion intensity can better reflect emergency situations than temperature changes. Therefore, the system will preset the weight ratio of each feature based on historical data and risk models. The risk weight allocation can also be adjusted dynamically to make the weight allocation more flexible. For example, features with high-density crowds and negative audio emotions are given higher weights.

[0279] Step S33: constructing a situational risk model according to the threat weight matrix data to obtain a public safety situational risk model;

[0280] Specifically, after obtaining the threat weight matrix, the system begins to build a contextualized risk model. This model analyzes multimodal data in a specific context, that is, combining the background of the surrounding environment and historical data to generate a risk assessment model that is more in line with the actual situation. The contextualized model integrates the weight of each feature to form a specific model for the place or event.

[0281] Step S34: Perform risk classification according to the public security scenario risk model to obtain security threat event level data;

[0282] Specifically, through the output of the contextualized risk model, the system classifies each feature into three risk levels: low, medium, and high. The risk level division is based on the importance of the features output by the contextual model. For example, if the crowd density feature is given a higher weight in the contextual model, the crowd density data will be more likely to be classified as high risk than other features.

[0283] Step S35: performing event level threshold determination according to the security threat event level data to obtain event level determination table data;

[0284] Specifically, after obtaining the risk level of each feature, the system will perform a threshold determination of the event level. The threshold determination determines the level of each event based on specific level classification standards (such as a crowd density exceeding 5 people / square meter is high risk) and generates an event level determination table. The threshold set by the system is based on historical data and the specific conditions of the current scene to ensure the accuracy of event level classification.

[0285] Step S36: classifying the security threat multimodal feature data into event levels according to the event level determination table data to obtain event level integrated data;

[0286] Specifically, based on the event level determination table, the system classifies each feature data into event levels so as to integrate features of different risk levels into a complete event level integrated data.

[0287] Step S37: Aggregate the event level integrated data into risk events to obtain security threat risk event data.

[0288] Specifically, the system conducts an overall analysis of the risk level of each feature, integrates the correlation between different features, and forms an overall risk event assessment. For example, in the context of high-emotion audio in a crowded area, the event is marked as a "high-risk event."

[0289] Preferably, step S4 is specifically:

[0290] Step S41: classify and schedule risk events according to the security threat risk event data to obtain risk event scheduling mapping data;

[0291] Specifically, the system first classifies and dispatches risk events based on security threat risk event data, identifies different risk event types (such as crowding, riots, fires, etc.), and generates dispatch mapping data based on event type, level, and location. The goal of the dispatch is to map different risk events to corresponding emergency response resources or departments.

[0292] Step S42: matching response strategies according to the risk event scheduling mapping data to obtain response strategy data;

[0293] Specifically, after completing the event scheduling, the system will match the response strategy for each scheduled event. The system searches the strategy database for a response strategy that matches the event type and risk level.

[0294] Step S43: Dynamically triggering early warning according to the response strategy data to obtain response control early warning data;

[0295] Specifically, the system triggers dynamic warnings based on response strategy data and controls the response process in real time. Warning triggers include the system issuing evacuation instructions, sending alarm notifications to relevant departments and personnel, and controlling the site through broadcasts, screens and other media. Warning triggers will also automatically adjust response resources, such as mobilizing nearby security personnel or automatically unlocking the venue access control to ensure smooth evacuation of personnel.

[0296] Step S44: performing multi-view dynamic risk assessment according to the response control warning data to obtain view risk assessment matrix data;

[0297] Specifically, the system uses multi-view surveillance data to dynamically assess the risk of an incident. The multi-view data comes from different surveillance cameras, sensors, and data sources, such as the density of people in different areas, volume changes, etc. The system integrates this data into an assessment matrix to quantify the real-time risk status of the incident site.

[0298] Step S45: performing response consequence prediction according to the perspective risk assessment matrix data to obtain response consequence prediction data;

[0299] Specifically, based on the perspective risk assessment matrix data, the system predicts the consequences of the current response measures (such as predictive calculations through linear regression or vector machine models) and analyzes the effects after the warning is triggered. The consequence prediction analyzes the direction of the event by inputting the data of the risk assessment matrix into the prediction model.

[0300] Step S46: Response optimization is performed on the corresponding consequence prediction data to obtain response control warning optimization data to perform public safety situation prediction auxiliary operations.

[0301] Specifically, based on the response consequence prediction, the system optimizes the current response measures to ensure that the incident is under control as soon as possible. The optimization process adjusts the response strategy based on the consequence prediction results, such as increasing the frequency of evacuations or increasing the input of human resources.

[0302] Preferably, the present application also provides a public security situation prediction system based on multimodal data fusion, which is used to execute the public security situation prediction method based on multimodal data fusion as described above, and the public security situation prediction system based on multimodal data fusion includes:

[0303] The public place non-sensing detection module is used to continuously monitor public places through video and perform non-sensing detection to obtain the public place non-sensing detection data;

[0304] The security threat multimodal feature extraction module is used to extract multimodal data features from the public place non-sensing discovery data to obtain security threat multimodal feature data;

[0305] The security threat risk event classification module is used to perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data;

[0306] The public security situation prediction auxiliary module is used to perform automatic response control based on security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to perform public security situation prediction auxiliary operations.

[0307] Therefore, from any point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes falling within the meaning and scope of equivalent elements of the application documents are included in the present invention.

[0308] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A public safety situation prediction method based on multimodal data fusion, characterized in that: The following steps are involved: Step S1: Continuously monitor the public places through video and perform non-sensing detection to obtain non-sensing detection data of public places; Step S2: extracting multimodal data features from the public place non-sensing discovery data to obtain security threat multimodal feature data; Step S3: Perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data; Step S4: Automatically respond to and control the security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to assist in public security situation prediction operations.

2. The method according to claim 1, characterized in that Step S1 is specifically as follows: Step S11: Capture a continuous video stream through a camera to obtain video stream matrix data; Step S12: extracting abnormal behavior events according to the video stream matrix data to obtain abnormal behavior event data; Step S13: Perform deduction and enhancement based on the abnormal behavior event data to obtain unconscious discovery data in public places.

3. The method according to claim 2, characterized in that The abnormal behavior event extraction in step S12 is performed by using a preset abnormal behavior event recognition model. The steps for constructing the preset abnormal behavior event recognition model are as follows: Obtain historical video stream matrix data and historical abnormal behavior event labels; Perform clustering calculation on the historical video stream matrix data to obtain historical video stream matrix clustering data; Perform balanced classification based on historical video stream matrix clustering data to obtain historical video stream test data and historical video stream verification data; Perform preliminary convolution calculations based on historical video stream test data to obtain preliminary video stream feature data; Perform pooling calculation and multi-layer self-attention calculation on the preliminary video stream feature data to obtain video stream pooling data and video stream weighted feature data respectively; Perform encoding full connection calculation according to the weighted feature data of the video stream and the pooled data of the video stream to obtain the full connection data of the video stream; Perform index calculation based on the fully connected data of the video stream and map it through the historical abnormal behavior event labels to obtain preliminary abnormal behavior event recognition data; The historical video stream verification data is used to iteratively train the preliminary abnormal behavior event recognition data to obtain the abnormal behavior event recognition model.

4. The method according to claim 3, characterized in that The multi-layer self-attention calculation is specifically as follows: Perform multi-scale layered convolution and spatiotemporal feature segmentation on the preliminary video stream feature data to obtain multi-scale feature data of the video stream; Perform local multi-head hierarchical self-attention calculation and global multi-head hierarchical self-attention calculation on the multi-scale feature data of the video stream to obtain local self-attention feature data of the video stream and global self-attention feature data of the video stream respectively; The local self-attention feature data of the video stream and the global self-attention feature data of the video stream are used to construct a relative position matrix to obtain the multi-head self-attention feature matrix data of the video stream; Construct a multi-layer graph based on the multi-head self-attention feature matrix data of the video stream to obtain the multi-layer graph data of the video stream; Perform global and local graph convolution calculations based on multi-layer graph data of the video stream to obtain global and local graph convolution data of the video stream; Multi-scale feature fusion is performed based on the global and local graph convolution data of the video stream to obtain the weighted feature data of the video stream.

5. The method according to claim 4, characterized in that The multi-layer graph construction is as follows: According to the multi-head self-attention feature matrix data of the video stream, a hierarchical multi-scale graph is deconstructed to obtain the hierarchical multi-scale graph data of the video stream; Constructing cross-layer associated nodes for hierarchical multi-scale graph data of video streams to obtain cross-layer associated node data of video streams; Extracting correlation features based on cross-layer correlation node data of the video stream to obtain cross-layer correlation feature data of the video stream; Gaussian convolution kernel mapping is performed according to the cross-layer correlation feature data of the video stream to obtain Gaussian convolution kernel data of the video stream; Perform graph convolution processing on the cross-layer associated node data of the video stream according to the Gaussian convolution kernel data of the video stream to obtain the cross-layer convolution layer data of the video stream; The cross-layer convolutional layer data of the video stream is processed across layers to obtain multi-layer graph data of the video stream.

6. The method according to claim 1, characterized in that Step S2 is specifically as follows: Step S21: extracting visual crowd dynamic perception features based on the public place non-sensing discovery data to obtain visual crowd dynamic perception feature data; Step S22: extracting audio synchronization data according to the visual crowd dynamic perception feature data and the video stream matrix data to obtain audio synchronization data; Step S23: performing audio emotion analysis according to the audio synchronization data to obtain audio emotion analysis data; Step S24: Perform multimodal fusion on the visual crowd dynamic perception feature data and the audio emotion analysis data to obtain security threat multimodal feature data.

7. The method according to claim 6, characterized in that The audio emotion analysis is processed by a preset audio emotion analysis model. The construction method of the audio emotion analysis model includes the following steps: Obtain historical audio synchronization data and historical audio emotion label data; Perform frequency domain conversion according to historical audio synchronization data to obtain historical audio frequency domain data; Perform data division on the historical audio frequency domain data to obtain historical audio test data and historical audio verification data; Construct an initial state according to historical audio test data to obtain historical audio initial state data; Perform input layer calculation and forgetting layer calculation according to the historical audio initial state data to obtain audio input layer data and audio forgetting layer data respectively; Perform multi-head emotion candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain audio candidate memory state data; Perform output layer calculation on the audio candidate memory state data to obtain the audio hidden state data; The audio hidden state data is iteratively trained using historical audio verification data and historical audio emotion label data to obtain an audio emotion parsing model; The multi-head emotion candidate memory state processing includes: Perform multi-dimensional candidate memory state processing according to the audio input layer data and the audio forgetting layer data to obtain initial audio candidate memory state data; Perform multi-head emotional attention calculation on the initial audio candidate memory state data to obtain audio multi-head emotional attention matrix data; Perform emotional memory enhancement on audio multi-head emotional attention matrix data to obtain audio enhanced memory state data; Performing memory state feedback on the initial audio candidate memory state data using the audio enhancement memory state data to obtain feedback memory state data; Multi-level candidate state aggregation is performed according to the feedback memory state data to obtain audio candidate memory state data.

8. The method according to claim 1, characterized in that Step S3 is specifically as follows: Step S31: Performing a preliminary screening of the multimodal features of security threats according to the multimodal feature data of security threats to obtain preliminary screening indicator data of threats; Step S32: Allocate risk weights according to the threat initial screening indicator data to obtain threat weight matrix data; Step S33: constructing a situational risk model according to the threat weight matrix data to obtain a public safety situational risk model; Step S34: Perform risk classification according to the public security scenario risk model to obtain security threat event level data; Step S35: performing event level threshold determination according to the security threat event level data to obtain event level determination table data; Step S36: classifying the security threat multimodal feature data into event levels according to the event level determination table data to obtain event level integrated data; Step S37: Aggregate the event level integrated data into risk events to obtain security threat risk event data.

9. The method according to claim 1, characterized in that: Step S4 is specifically as follows: Step S41: classify and schedule risk events according to the security threat risk event data to obtain risk event scheduling mapping data; Step S42: matching response strategies according to the risk event scheduling mapping data to obtain response strategy data; Step S43: Dynamically triggering early warning according to the response strategy data to obtain response control early warning data; Step S44: performing multi-view dynamic risk assessment according to the response control warning data to obtain view risk assessment matrix data; Step S45: performing response consequence prediction according to the perspective risk assessment matrix data to obtain response consequence prediction data; Step S46: Response optimization is performed on the corresponding consequence prediction data to obtain response control warning optimization data to perform public safety situation prediction auxiliary operations.

10. A public safety situation prediction system based on multimodal data fusion, characterized in that: Used to execute the public security situation prediction method based on multimodal data fusion as claimed in claim 1, the public security situation prediction system based on multimodal data fusion comprises: The public place non-sensing detection module is used to continuously monitor public places through video and perform non-sensing detection to obtain the public place non-sensing detection data; The security threat multimodal feature extraction module is used to extract multimodal data features from the public place non-sensing discovery data to obtain security threat multimodal feature data; The security threat risk event classification module is used to perform situation assessment on the security threat multimodal feature data to obtain security threat event level data, and classify the security threat multimodal feature data into event levels according to the security threat event level data to obtain security threat risk event data; The public security situation prediction auxiliary module is used to perform automatic response control based on security threat risk event data to obtain response control warning data, and perform multi-perspective evaluation and optimization based on the response control warning data to obtain response control warning optimization data to perform public security situation prediction auxiliary operations.

Images