A security risk analysis system and method based on multimodal data processing

Abstract

The present invention discloses a safety risk analysis system and method based on multimodal data processing; it relates to the field of industrial automation and information technology; the purpose of the present invention is to specifically target high-risk scenarios of hazardous chemical enterprises, and provide a safety risk analysis system and method based on multimodal data processing, which is based on deep learning technology, comprehensively considers the changes in spatiotemporal characteristics, can monitor and evaluate the safety risks of hazardous chemical enterprises in real time, and automatically integrates multimodal data such as monitored videos, sensor data, and operation logs. The spatial features are extracted through convolutional neural networks, and the time series data is processed using recurrent neural networks and long short-term memory networks, and further feature extraction and fusion are performed. The key risk factors are also identified through the attention mechanism, thereby achieving a comprehensive and accurate safety risk assessment, thereby providing real-time warnings for safety managers to ensure that potential safety hazards are discovered in a timely manner.

Description

Technical Field

[0001] The present invention relates to the field of industrial automation and informatization, and in particular to a security risk analysis system and method based on multimodal data processing. Background Art

[0002] With the acceleration of industrialization, especially the continuous growth in the number and scale of hazardous chemical enterprises, their safety risk management has become increasingly important. In the production process of these enterprises, due to the involvement of multiple risk factors, the traditional safety risk assessment method relies on manual inspection and empirical judgment, which is not only inefficient, but also difficult to fully cover and monitor in real time. Therefore, the traditional method has many limitations, mainly reflected in the inability to conduct real-time and comprehensive monitoring of large-scale production processes, and is prone to omissions and judgment bias. Due to the strong subjectivity and judgment differences in manual inspections, and the lack of unified and objective standards, the evaluation results of different personnel may vary greatly. These problems make it inevitable that safety hazards will arise in the production process of hazardous chemical enterprises, resulting in a decline in the overall performance of the system, and may even pose a serious threat to the production safety or transportation safety of hazardous chemical enterprises.

[0003] Traditional manual inspection and empirical judgment methods often have difficulty handling complex production environments and massive amounts of real-time data. Traditional safety risk analysis methods are mainly based on a single data source for monitoring and evaluation. When applied to high-risk scenarios in hazardous chemical companies, there are large errors, which may affect the accuracy and timeliness of safety work, especially in scenarios with extreme environmental conditions and complex operating processes. Traditional methods often rely on manual inspections, scheduled data collection and other means, which easily ignore potential dynamic risks and complex correlations, leading to accidents.

[0004] In the production environment of hazardous chemical enterprises, if there is no real-time and effective safety risk analysis mechanism, it may lead to catastrophic consequences, seriously affecting the safety, stability and sustainability of the enterprise, and may even endanger the lives of employees. Therefore, a safety risk assessment method is particularly necessary. Through in-depth analysis of various data in the production process, the intelligent system can monitor safety risks in real time, and make accurate predictions and assessments based on real-time data. When disasters, emergencies or major public events occur, important information can be quickly conveyed to relevant personnel, thereby ensuring emergency response and personnel safety. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a safety risk analysis system and method based on multimodal data processing. The technical problem to be solved is: the traditional safety risk analysis method is mainly based on a single data source for monitoring and evaluation, and there are large errors when applied to high-risk scenarios of hazardous chemical enterprises; the purpose of the present invention is to specifically target high-risk scenarios of hazardous chemical enterprises and provide a safety risk analysis system and method based on multimodal data processing. Based on deep learning technology, it comprehensively considers the changes in temporal and spatial characteristics, and considers the interactions and influences between different types of data in a dynamic environment. It comprehensively considers multimodal data such as video surveillance, sensor data, and operation logs in the risk assessment process, reduces misjudgments and missed judgments caused by human factors, and provides an intelligent risk assessment method suitable for complex security environments.

[0006] The purpose of the present invention is mainly achieved through the following technical solutions: a security risk analysis system based on multimodal data processing, including a data acquisition module, a data processing module, a feature vector extraction module, a time-space fusion module, a risk assessment module, and a risk classification module connected in sequence.

[0007] in:

[0008] The data acquisition module is electrically connected to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; it collects the monitoring video data, various sensor data and text data of the operation log of the hazardous chemical enterprise in real time as raw data, and transmits the raw data to the data processing module.

[0009] The data processing module cleans, formats and normalizes the collected raw data in turn, and extracts the spatiotemporal feature vectors in the raw data, the spatiotemporal feature vectors including the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data; transmits the normalized data to the feature extraction module, and transmits the spatiotemporal feature vectors to the spatiotemporal fusion module.

[0010] The feature extraction module extracts feature vectors from the normalized data through a neural network structure, outputs visual feature vectors, physical feature vectors and text feature vectors, and transmits the visual feature vectors, physical feature vectors and text feature vectors as multimodal feature vectors to the spatiotemporal fusion module.

[0011] The spatiotemporal fusion module fuses the multimodal feature vector with the spatiotemporal feature vector to output a comprehensive spatiotemporal feature vector.

[0012] The risk assessment module uses the long short-term memory LSTM network to process the fused comprehensive spatiotemporal feature vector, captures the feature vector of the changing trend of the output security risk over time as a dynamic risk assessment; combines the attention mechanism to identify risk factors, calculates the attention weight, and obtains the feature vector output by the attention mechanism.

[0013] The risk classification module uses the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divides the risks into three categories: high risk, medium risk, and low risk according to the probability distribution.

[0014] The present invention also provides a security risk analysis method based on multimodal data processing, which is implemented based on the above system and includes the following steps:

[0015] Step S1, perform data collection; electrically connect to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; collect the monitoring video data, various sensor data, and operation log text data of the hazardous chemical enterprise in real time as raw data.

[0016] Step S2, perform data processing; clean, format and normalize the collected raw data in turn, and extract the spatiotemporal feature vector in the raw data, the spatiotemporal feature vector includes the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data.

[0017] Step S3, extracting feature vectors from the normalized data through a neural network structure, outputting visual feature vectors, physical feature vectors and text feature vectors, and using the visual feature vectors, physical feature vectors and text feature vectors as multimodal feature vectors.

[0018] Step S4, performing spatiotemporal fusion of the multimodal feature vector; fusing the multimodal feature vector with the spatiotemporal feature vector, outputting a comprehensive spatiotemporal feature vector, and using the comprehensive spatiotemporal feature vector for risk assessment.

[0019] Step S5, perform risk assessment; use the LSTM network to process the integrated spatiotemporal feature vector after fusion, capture the changing trend of security risks over time, calculate the attention weight, and obtain the feature vector output by the attention mechanism; output the key risk factor feature vector based on the dynamic risk assessment combined with the attention weight.

[0020] Step S6: classify the risk assessment results; use the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divide the risks into three categories: high risk, medium risk, and low risk according to the probability distribution.

[0021] How this solution works:

[0022] By working together, each module can monitor and evaluate the safety risks of hazardous chemical enterprises in real time, and automatically integrate multimodal data such as monitored videos, sensor data, and operation logs. The spatial features are extracted through convolutional neural networks, and time series data are processed using recurrent neural networks and long short-term memory networks. Further feature extraction and fusion are performed to achieve a comprehensive and accurate safety risk assessment. At the same time, the system also identifies key risk factors through the attention mechanism, and classifies and outputs risks through the softmax function, thereby providing real-time warnings for safety managers to ensure that potential safety hazards are discovered in a timely manner.

[0023] In order to further optimize the above technical solution, the data processing module cleans the collected raw data, including: cleaning the video frames, sensor data and text data; using the median filtering image processing algorithm to detect and remove noise and abnormal points in the video frames; identifying and deleting abnormal data based on the set threshold range of the sensor data; and removing irrelevant characters, garbled characters and format errors from the text data through regular expression technology.

[0024] The formatting processing of the data processing module includes: uniformly representing video frames of different monitoring video data in a visual format; uniformly representing physical quantities of different types of sensor data in a physical format; uniformly representing text data of the operation log in a text format; and uniformly converting the timestamps of the video frames of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data into a specific time format and location format.

[0025] The normalization processing of the data processing module adopts linear normalization processing, and normalizes the video frame, each sensor data and text data respectively, and scales the data to the interval [0,1]. The normalization processing calculation is expressed as

[0026]

[0027] X is the original data, X min is the minimum value in the original data, X max is the maximum value in the original data, X norm It is the normalized data.

[0028] The data processing module extracts the spatiotemporal feature vector from the original data, including:

[0029] For video data, the timestamp of the video frame is extracted as the time feature vector, and combined with the installation location information of the monitoring equipment as the spatial feature vector, the spatiotemporal feature vector is output; the installation location information refers to the area number and coordinates of the monitoring equipment in a specific enterprise.

[0030] For sensor data, the collection time is used as the time feature vector, and combined with the sensor's installation location identifier as the space feature vector, the spatiotemporal feature vector is output; wherein the installation location identifier is the specific location coordinates or area name in the enterprise.

[0031] For text data, the time information is used as the time feature vector, and combined with the location information as the spatial feature vector, the spatiotemporal feature vector is output.

[0032] In order to further optimize the above technical solution, the feature extraction module uses a convolutional neural network to process the video data, and extracts visual feature vectors using a recurrent neural network to process the sensor data and text data respectively. The recurrent neural network extracts physical feature vectors from the sensor data, and the recurrent neural network extracts text feature vectors from the text data, mining the temporal and semantic information in the multimodal feature vectors; the feature extraction module calculation includes:

[0033] Visual Features v Expressed as: F v =CNN(X v ),

[0034] Physical characteristicsF a Expressed as: F a =RNN(X a ),

[0035] Text feature F t Expressed as: F t =RNN(X t ),

[0036] Where: X v is the normalized video data, X a is the normalized sensor data, X t It is the normalized text data.

[0037] In order to further optimize the above technical solution, the visual feature vector is extracted by using a convolutional neural network (CNN) model to perform transfer learning on video data to extract the visual feature vector in the video data; the video frame is input into the CNN model, and after multiple layers of convolution and pooling operations, a feature vector representing the key information of the image is obtained, and the visual feature vector is output.

[0038] The physical feature vector is extracted by using a recurrent neural network (RNN) model, and the sensor data is input into the RNN model in chronological order. The RNN model extracts the physical feature vector that can reflect the changing trend of the physical process by learning the time sequence in the data, and outputs the physical feature vector.

[0039] The text feature vector is extracted by using an RNN model to process text data, and the words in the text are converted into vector representations using word vector technology to obtain a text sequence, and then the text sequence is input into the RNN model. The RNN model learns the semantic information in the text and extracts text feature vectors that can represent operation behaviors, event descriptions, time and place.

[0040] In order to further optimize the above technical solution, the spatiotemporal fusion module fuses the multimodal feature vector with the spatiotemporal feature vector, specifically including: setting the weights of different modal feature vectors in the fusion process, weighted summing the visual feature vector, the physical feature vector and the text feature vector according to their respective set weights, and obtaining the fused spatiotemporal feature vector as the comprehensive spatiotemporal feature vector; using the fusion algorithm to calculate the comprehensive spatiotemporal feature vector S t , expressed as: S t =Fuse(F v ,F a ,F t ), where F v is the visual feature vector, F a is the physical eigenvector, F t is the text feature vector, S t It is a comprehensive spatiotemporal feature vector, and the Fuse function is a weighted summation fusion algorithm.

[0041] In order to further optimize the above technical solution, the risk assessment module performs dynamic risk assessment using a long short-term memory LSTM network to process the fused spatiotemporal feature vector, specifically including the following steps: first, setting the number of layers and hidden units of the long short-term memory LSTM network to determine the network structure; then setting the input and output dimensions; and finally, inputting the comprehensive spatiotemporal feature vector to obtain a risk level assessment.

[0042] The input dimension should be the same as the dimension of the comprehensive spatiotemporal feature vector output by the spatiotemporal fusion module; assuming that the dimension of the feature vector output by the spatiotemporal fusion module is d, the input dimension of the LSTM network should also be d; among them, the comprehensive spatiotemporal feature vector after spatiotemporal fusion includes visual feature vector, physical feature vector and text feature vector; the visual feature vector in the monitored video is assumed to have a dimension of m; the physical feature vector in the sensor data is assumed to have a dimension of n, and the text feature vector of the operation log is assumed to have a dimension of p; the dimension of the spatiotemporal feature vector obtained after the fusion operation is d=m+n+p, where d is the input dimension of the LSTM network.

[0043] The output dimension is determined according to the number of risk classification categories, which are divided into three categories: high risk, medium risk, and low risk. The output dimension is 3.

[0044] The risk level assessment is obtained by inputting the comprehensive spatiotemporal feature vector sequence into the established long short-term memory LSTM network.

[0045] Input the comprehensive spatiotemporal feature vector sequence The comprehensive spatiotemporal feature vector sequence generated by the spatiotemporal fusion module is input into the LSTM network sequentially according to the time step.

[0046] The length of the time series of hazardous chemical enterprise data collected by processing the comprehensive spatiotemporal feature vector is set to T. At each time step t (1≤t≤T), the corresponding comprehensive spatiotemporal feature vector S t Input into the LSTM network to obtain the characteristic vector H of the trend of security risk over time t , denoted as H t =LSTM(S t ).

[0047] At each time step, the LSTM network generates a new feature vector based on the current input feature vector and the hidden state h of the previous time step. t-1 And cell state c t-1 To calculate the hidden state h of the current time step t and cell state c t , the calculation process involves the gating mechanism of the LSTM network, which includes the forget gate, input gate, cell state update and output gate. The specific process includes:

[0048] First calculate the input gate, which includes the input gate activation function and the input modulation gate.

[0049] Input gate activation function i t , denoted as i t =σ(W i ·[h t-1 ,S t ]+b i ),

[0050] Where: i t is the input gate activation value at time t; σ is the sigmoid function, whose input range is (0,1); W i is the comprehensive spatiotemporal feature vector S of the input t To the weight matrix set by the input gate, the comprehensive spatiotemporal feature vector S t It is also the input vector at time t; h t-1 is the hidden state of the previous time step; b i is the bias vector for the input gate.

[0051] Input modulation gate Expressed as

[0052] in: is the candidate cell state, tanh is the hyperbolic tangent function, and its output range is (-1,1); W c is the comprehensive spatiotemporal feature vector S of the input t to the weight matrix set for the candidate cell state; b c is the bias vector of the candidate cell state.

[0053] Then the forget gate activation value f t The calculation is expressed as f t =σ(W f ·[h t-1 ,S t ]+b f ),

[0054] Among them, W f is the comprehensive spatiotemporal feature vector S of the input t To the weight matrix of the forget gate, b f is the bias vector of the forget gate.

[0055] Then continue with cell state c t The updated calculation is expressed as:

[0056] Among them, c t-1 is the cell state at the previous time step, and ⊙ represents element-wise multiplication.

[0057] The input gate activation value i t The role of controlling how many new candidate cell states there are Can be updated to the current cell state c t Finally, the output gate is calculated, where the output gate activation value o t , expressed as: o t =σ(W o ·[h t-1 ,S t ]+b o ),

[0058] Where W o is the comprehensive spatiotemporal feature vector S of the input t The weight matrix to the output gate; b o is the bias vector for the output gate.

[0059] Hidden state h t The update formula is expressed as h t =o t ⊙tanh(ct ), the hidden state is the output of the cell state after the function is controlled by the output gate. It will be used as one of the inputs of the next time step, and can also be used as the output of the LSTM network. The characteristic vector H of the trend of security risk over time t , H t =[h 1 ,h 2 ,…,h t ].

[0060] The risk level assessment is to judge the changing trend of the risk by observing the changes in the hidden states output by the LSTM network at different time steps, calculating the difference or similarity index between the hidden states of adjacent time steps, and calculating the Euclidean distance. If it gradually increases, it means that the risk is increasing; if it gradually decreases, it means that the risk is decreasing; if it is maintained within the set threshold range, the risk may remain stable.

[0061] In order to further optimize the above technical solution, the risk assessment module combines the attention mechanism to identify risk factors, combines the LSTM network to obtain the characteristic vector of the trend of security risk over time with the attention mechanism to calculate the attention weight, and obtains the characteristic vector A output by the attention mechanism t , expressed as: A t =Attention(H t ), wherein the calculation of attention weight specifically includes:

[0062] Set the dimension of the query and key matrix to d q =d k =d h , the dimension of the value matrix is ​​set to the same as the hidden state dimension d as needed v =d h , the specific calculation attention weight of the attention mechanism is: where Q, K, V are query, key and value matrices, d k is the dimension of the key.

[0063] According to the calculated attention weight, the characteristic vector matrix A output by the attention mechanism is obtained t , A t =[a 1 ,a 2 ,…,a t ].

[0064] In order to further optimize the above technical solution, the risk assessment module also includes calculating the key risk factor feature vector based on the dynamic risk assessment combined with the attention weight to visualize the attention weight; the key risk factor feature vector is the weighted sum of the feature vector of the calculated trend of change of security risk over time and the attention weight to obtain the attention-weighted feature vector.

[0065] Let the attention weight be a t , the hidden state of the LSTM network at time step is h t , then the key risk factor feature vector after attention weighting is

[0066] In order to further optimize the above technical solution, the risk classification module uses a softmax function, specifically including:

[0067] First, build a softmax classification layer and use the feature vector output by the LSTM network combined with the attention mechanism as input; the softmax function converts the input vector into a probability distribution, and each element represents the probability of the corresponding risk category.

[0068] The Softmax classification module algorithm is expressed as: t =Softmax(A t ),

[0069] The Softmax function is expressed as:

[0070] Among them, e is a natural constant, The feature vector matrix A output by the attention mechanism t Each element in is exponentially operated separately; i is the risk category, It is the sum of the index values ​​corresponding to the three risk categories of high risk, medium risk and low risk; t is the probability distribution of the corresponding risk category, indicating the characteristic vector matrix A output by the attention mechanism at time t t The probability of belonging to the i-th category.

[0071] The final risk classification result is determined based on the preset probability threshold; when the value of the probability distribution of the risk category exceeds the set high threshold, the risk is judged as high risk; when the value of the probability distribution of the risk category is less than the set low threshold, the risk is judged as low risk; when the value of the probability distribution of the risk category is between the set high threshold and the low threshold, the risk is judged as medium risk.

[0072] In summary, the present invention has the following beneficial effects compared with the prior art:

[0073] 1. Multimodal data integration, key risk factor identification, and improved risk assessment accuracy. By integrating video, sensor data, and text logs, comprehensive monitoring of security risks is achieved; by using the attention mechanism to identify and focus on key risk factors, the response speed and accuracy of the early warning system is improved; by integrating multimodal data, using deep learning models to extract deep features, and combining spatiotemporal fusion and attention mechanisms, security risks can be identified and assessed more comprehensively and accurately, reducing misjudgments and missed judgments.

[0074] 2. Fusion of spatiotemporal features, dynamic risk assessment, and real-time dynamic monitoring. Using spatiotemporal fusion technology, combined with the temporal and spatial characteristics of data, potential risks can be accurately analyzed; LSTM networks are used to process time series data to assess dynamic changes in risks in real time; data is collected and processed in real time, risk status is continuously assessed, and potential risk changes are discovered in a timely manner, so that enterprises can respond quickly and take effective measures to reduce the probability of accidents.

[0075] 3. Enhance adaptability and flexibility, real-time warning and classification, and improve safety management efficiency. Users can customize risk assessment parameters and warning thresholds according to the actual situation of the enterprise, so that the system can adapt to the production process, environment and safety management requirements of different enterprises, and improve the practicality and applicability of the system; classify risk levels through the softmax function, issue warnings in time, and ensure that safety hazards are quickly dealt with; automated risk analysis and warning mechanisms reduce the burden of manual inspections, improve the efficiency of safety management work, and provide a scientific basis for enterprise safety decisions, which helps to optimize safety management strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0077] Figure 1 It is a flow chart of a security risk analysis system based on multimodal data processing;

[0078] Figure 2 A step-by-step diagram of a security risk analysis method based on multimodal data processing. DETAILED DESCRIPTION

[0079] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0080] Traditional safety risk analysis methods are mainly based on a single data source for monitoring and evaluation. When applied to high-risk scenarios in hazardous chemical companies, there are large errors, which may affect the accuracy and timeliness of safety work, especially in scenarios with extreme environmental conditions and complex operating processes. In view of this, the present invention provides the following embodiments to solve the above-mentioned technical problems.

[0081] In a specific embodiment, Figure 1 As shown, the present invention proposes a security risk analysis system for multimodal data processing, which is mainly composed of a data acquisition module, a data processing module, a feature extraction module, a spatiotemporal fusion module, a risk assessment module and a risk classification module, and also includes an alarm module and a human-computer interaction module.

[0082] A safety risk analysis system based on multimodal data processing includes a sequentially connected data acquisition module, a data processing module, a feature extraction module, a time-space fusion module, a risk assessment module, and a risk classification module; each module works together to achieve comprehensive monitoring and analysis of safety risks applied to hazardous chemical enterprises, including:

[0083] The data acquisition module is electrically connected to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; it collects the monitoring video data, various sensor data and text data of the operation log of the hazardous chemical enterprise as raw data in real time, and transmits the raw data to the data processing module in real time.

[0084] The data processing module cleans, formats and normalizes the collected raw data in turn, and extracts the spatiotemporal feature vectors in the raw data, the spatiotemporal feature vectors including the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data; transmits the normalized data to the feature extraction module, and transmits the spatiotemporal feature vectors to the spatiotemporal fusion module.

[0085] The feature extraction module extracts feature vectors from the normalized data through a neural network structure, outputs visual feature vectors, physical feature vectors and text feature vectors, and transmits the visual feature vectors, physical feature vectors and text feature vectors as multimodal feature vectors to the spatiotemporal fusion module.

[0086] The space-time fusion module fuses the multimodal feature vector with the space-time feature vector, outputs a comprehensive space-time feature vector, and uses the comprehensive space-time feature vector to perform risk assessment.

[0087] The risk assessment module uses a long short-term memory LSTM network to process the fused comprehensive spatiotemporal feature vector, captures the feature vector of the changing trend of the output security risk over time, captures the dependency on the time series, and evaluates the risk that changes over time as a dynamic risk assessment; combined with the attention mechanism, it dynamically allocates the feature vectors of different modes and the attention of time steps, identifies risk factors, calculates the attention weight, and obtains the feature vector output by the attention mechanism.

[0088] The risk classification module uses the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divides the risks into three categories: high risk, medium risk, and low risk according to the probability distribution, providing enterprises with intuitive and clear risk assessment results, making it easier for enterprises to take corresponding risk management measures.

[0089] The data acquisition module is electrically connected to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise, and collects the monitoring video data, various sensor data, and text data of the operation log of the hazardous chemical enterprise in real time as raw data, ensuring the integrity, diversity and timeliness of the data, and providing a rich data foundation for subsequent analysis; and establishes a stable data transmission channel, and transmits the collected raw data to the data processing module in real time.

[0090] The data acquisition module transmits the comprehensively collected monitoring video, sensor and text data within the hazardous chemical enterprise to the data processing module. The data processing module cleans, formats and normalizes the collected raw data, including video frame extraction of monitoring video data, cleaning of video, sensor and log data, and formatting of log data to ensure data quality and availability; finally, the three types of data are normalized separately to facilitate subsequent feature vector extraction and analysis.

[0091] Video frame extraction is to extract key frames from the video through motion detection technology. Video frame extraction of surveillance video data uses motion detection technology to extract key frames. Motion detection technology identifies active areas by analyzing the motion changes of objects in the video and extracts frames related to motion. Video frame extraction compares continuous video data to detect motion differences in the picture, thereby saving frames containing important information as output video frames.

[0092] The data processing module performs cleaning processing on the collected raw data, including: cleaning video frames, sensor data and text data; using median filtering image processing algorithm to detect and remove noise and abnormal points in video frames to improve image quality; identifying and deleting abnormal data according to the set threshold range of sensor data, such as the temperature sensor suddenly having a maximum or minimum value beyond the normal range, and repairing the data through data interpolation; using regular expression technology to remove irrelevant characters, garbled codes and format errors from text data. Regular expression technology is a tool for pattern matching and text processing, which can find, replace or verify the content in the text by defining specific character patterns. Cleaning text data is to identify and delete irrelevant characters, garbled codes or format errors through regular expressions to ensure the standardization and accuracy of the data.

[0093] The formatting processing of the data processing module includes: uniformly representing video frames of different monitoring video data in a visual format; uniformly representing physical quantities of different types of sensor data in a physical format, and ensuring the numerical comparability of the data according to the same unit system and data accuracy; uniformly representing text data of the operation log in a text format; uniformly converting the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data into a specific time format and location format. This embodiment adopts the ISO8601 format to facilitate subsequent time series analysis.

[0094] Sensor data cleaning removes outliers. The cleaning operation removes noise, outliers and erroneous data in the data. Log data is formatted into a unified data format to facilitate subsequent processing, and key information is extracted through the word vector model, laying the foundation for feature vector extraction.

[0095] The normalization processing of the data processing module adopts linear normalization processing, and normalizes the video frame, each sensor data and text data respectively, and scales the data to the interval [0,1]. The normalization processing calculation is expressed as

[0096] X is the original data, X min is the minimum value in the original data, X max is the maximum value in the original data, X norm It is the normalized data.

[0097] After normalization, the data is mapped to the set range to facilitate subsequent data analysis and machine learning algorithm processing, improve data processing efficiency and subsequent network training effects, and improve the convergence speed and stability of model training. During the normalization process, the original range of the data and the normalization processing parameters are recorded so that the true value of the data can be restored in subsequent analysis.

[0098] The data processing module extracts the spatiotemporal feature vector from the original data, including:

[0099] For video data, the timestamp of the video frame is extracted as the time feature vector, and combined with the installation location information of the monitoring equipment as the spatial feature vector, the spatiotemporal feature vector is output; the installation location information refers to the area number and coordinates of the monitoring equipment in a specific enterprise.

[0100] For sensor data, the collection time is used as the time feature vector, and combined with the sensor's installation location identifier as the space feature vector, the spatiotemporal feature vector is output; wherein the installation location identifier is the specific location coordinates or area name in the enterprise.

[0101] For text data, the time information is used as the time feature vector, and combined with the location information as the spatial feature vector, the spatiotemporal feature vector is output.

[0102] Extract key spatiotemporal feature vectors to provide important basis for subsequent spatiotemporal fusion and risk assessment.

[0103] The feature extraction module uses different convolutional neural networks to mine key information in feature vectors of different modalities. Visual feature vector extraction is achieved through convolutional neural networks, audio feature vector extraction is achieved through recurrent neural networks, and text feature vector extraction is also achieved through recurrent neural networks, respectively extracting key feature vectors of visual, audio, and text data. Use convolutional neural networks to process video data, extract visual feature vectors, and capture key information in images; use recurrent neural networks to process sensor data and text data respectively, recurrent neural networks extract physical feature vectors from sensor data, and recurrent neural networks extract text feature vectors from text data, mining the temporal and semantic information in multi-modal feature vectors; feature extraction module calculations include:

[0104] Visual Features v Expressed as: F v =CNN(X v ),

[0105] Physical characteristicsF a Expressed as: F a =RNN(X a ),

[0106] Text feature Ft Expressed as: F t =RNN(X t ),

[0107] Where: X v is the normalized video data, X a is the normalized sensor data, X t It is the normalized text data.

[0108] The method for extracting visual feature vectors adopts a convolutional neural network (CNN) model to perform transfer learning on video data related to hazardous chemical enterprises to extract visual feature vectors from the video data; the video frame is input into the CNN model, and after multi-layer convolution and pooling operations, a feature vector representing key information of the image is obtained, including a personnel behavior feature vector, an equipment status feature vector, etc., and a visual feature vector is output.

[0109] The physical feature vector is extracted by adopting a recurrent neural network (RNN) model according to the time series characteristics of the sensor data. The sensor data is input into the RNN model in chronological order. The RNN model extracts the physical feature vector that can reflect the change trend of the physical process, such as the temperature change trend and the pressure fluctuation feature vector, after multi-layer convolution and pooling operations by learning the time series in the data, and outputs the physical feature vector.

[0110] The text feature vector is extracted, and the text data is processed by the RNN model. The words in the text are converted into vector representations using word vector technology to obtain a text sequence, and then the text sequence is input into the RNN model. After multiple layers of convolution and pooling operations, the RNN model learns the semantic information in the text and extracts text feature vectors that can represent operation behaviors, event descriptions, time and place, such as identifying key operation steps in operation logs, abnormal event descriptions, etc.

[0111] The spatiotemporal fusion module fuses the multimodal feature vector with the spatiotemporal feature vector, specifically including:

[0112] The extracted visual feature vectors, physical feature vectors and text feature vectors are aligned with the spatiotemporal feature vectors and then fused. Considering the timestamp and spatial position of the data, the spatiotemporal correlation between the data can be reflected. In the fusion layer, the visual, audio and text feature vectors are fused in the spatiotemporal dimension to form a spatiotemporal feature vector that comprehensively reflects the security risk status.

[0113] In a specific embodiment, spatiotemporal fusion can also be implemented through a weighted summation or attention mechanism algorithm, which comprehensively considers the temporal and spatial correlation of data, integrates feature vectors of different modalities, and forms a spatiotemporal feature vector that can comprehensively reflect the security risk status, so that the model can understand and evaluate risks from multiple dimensions.

[0114] This embodiment adopts a weighted summation algorithm, sets the weights of different modal feature vectors in the fusion process, and performs weighted summation on the visual feature vector, the physical feature vector, and the text feature vector according to their respective set weights to obtain a fused spatiotemporal feature vector as a comprehensive spatiotemporal feature vector; the fusion algorithm is used to calculate the comprehensive spatiotemporal feature vector S t , expressed as: S t =Fuse(F v ,F a ,F t ),

[0115] where F v is the visual feature vector, F a is the physical eigenvector, F t is the text feature vector, S t It is a comprehensive spatiotemporal feature vector, and the Fuse function is a weighted summation fusion algorithm.

[0116] The LSTM network with long short-term memory can effectively capture the dependencies in the time series, thereby evaluating the changing trend of security risks over time and providing a dynamic perspective for risk assessment. The LSTM layer processes the fused comprehensive spatiotemporal feature vector to evaluate the changing trend of security risks over time.

[0117] The risk assessment module performs dynamic risk assessment by using a long short-term memory (LSTM) network to process the fused spatiotemporal feature vectors, specifically including the following steps:

[0118] First, the number of layers and hidden units of the long short-term memory LSTM network is set to determine the network structure; then the input and output dimensions are set; finally, the comprehensive spatiotemporal feature vector is input to obtain the risk level assessment.

[0119] According to the complexity of the problem and the time series characteristics of the data, the appropriate number of LSTM layers is selected through experiments. Generally speaking, 1-3 layers of LSTM networks can achieve good results when processing most time series data. For the safety risk assessment of hazardous chemical enterprises, if the data has a long time dependency or complex dynamic changes, 2-3 layers of LSTM may be required to capture deep information; if the data is relatively simple, 1 layer of LSTM may be sufficient.

[0120] In an application of risk assessment, experiments have shown that a two-layer LSTM can better learn how parameters such as temperature and pressure change over time during a reaction, thereby accurately assessing risks.

[0121] The number of hidden units determines the expressive power of the LSTM network. Too few hidden units may cause the model to fail to fully learn the patterns in the data, while too many hidden units may cause overfitting. Start with a smaller value, 32, 64, and gradually increase the number of hidden units while evaluating the model performance on the validation set.

[0122] When assessing the risk of storage areas in hazardous chemical enterprises, it was found that when the number of hidden units was 128, the model had the highest accuracy on the validation set and could better capture the impact of changes in various factors in the storage environment on risk.

[0123] The input dimension should be the same as the dimension of the comprehensive spatiotemporal feature vector output by the spatiotemporal fusion module; assuming that the dimension of the feature vector output by the spatiotemporal fusion module is d, the input dimension of the LSTM network should also be d; this ensures that the LSTM network can correctly receive and process the fused feature vector data.

[0124] Among them, the comprehensive spatiotemporal feature vector after spatiotemporal fusion includes visual feature vector, physical feature vector and text feature vector; the visual feature vector in the monitored video, the dimension is set to m; the physical feature vector in the sensor data, the dimension is set to n, and the text feature vector of the operation log, the dimension is set to p; the dimension of the spatiotemporal feature vector obtained after the fusion operation is d = m + n + p, d is the input dimension of the LSTM network.

[0125] The output dimension is determined according to the number of risk classification categories, which are divided into three categories: high risk, medium risk, and low risk. The output dimension is 3.

[0126] In the specific implementation, if the risk categories are further subdivided, such as adding extremely high risk and extremely low risk categories, the output dimension is adjusted accordingly. For example, in a fine risk classification scenario, the risks are divided into five categories, and the output dimension of the LSTM network is set to 5. Each output is a node representing the probability estimate of the corresponding risk category.

[0127] The risk level assessment is obtained by inputting the comprehensive spatiotemporal feature vector sequence into the established long short-term memory LSTM network.

[0128] Input the comprehensive spatiotemporal feature vector sequence The comprehensive spatiotemporal feature vector sequence generated by the spatiotemporal fusion module is input into the LSTM network sequentially according to the time step.

[0129] A time step is a time point or a position of data in a sequence. When processing a sequence of comprehensive spatiotemporal feature vectors, the data is collected in real time, so it is arranged in chronological order. Each unit in this order can be regarded as a time step. When the sequence of comprehensive spatiotemporal feature vectors is input into the LSTM network, the time step represents the time position corresponding to a feature vector in the sequence of comprehensive spatiotemporal feature vectors.

[0130] The length of the time series of the collected hazardous chemical enterprise data is set to T. At each time step t (1≤t≤T), the corresponding comprehensive spatiotemporal feature vector S t Input into the LSTM network to obtain the characteristic vector H of the trend of security risk over time t , denoted as H t =LSTM(S t ).

[0131] LSTM learns long-term dependencies in a sequence by processing the input vector at each time step. At each time step, the LSTM unit updates its state based on the current input vector, the hidden state of the previous time step, and the cell state. In this way, it can use the information of the past time steps to process the current time step, thereby effectively processing the temporal characteristics of the sequence data.

[0132] At each time step, the LSTM network generates a new feature vector based on the current input feature vector and the hidden state h of the previous time step. t-1 And cell state c t-1 To calculate the hidden state h of the current time step t and cell state c t , the calculation process involves the gating mechanism of the LSTM network, which includes the forget gate, input gate, cell state update and output gate. The specific process includes:

[0133] First calculate the input gate, which determines how much new information can be stored in the cell state. The input gate includes the input gate activation function and the input modulation gate.

[0134] Input gate activation function i t , denoted as i t =σ(W i ·[h t-1 ,S t ]+b i ),

[0135] Where: i t is the input gate activation value at time t; σ is the sigmoid function, whose input range is (0,1); W i is the comprehensive spatiotemporal feature vector S of the input tTo the weight matrix set by the input gate, the comprehensive spatiotemporal feature vector S t It is also the input vector at time t; h t-1 is the hidden state of the previous time step; b i is the bias vector for the input gate.

[0136] Input modulation gate Expressed as

[0137] in: is the candidate cell state, tanh is the hyperbolic tangent function, and its output range is (-1,1); W c is the comprehensive spatiotemporal feature vector S of the input t to the weight matrix set for the candidate cell state; b c is the bias vector of the candidate cell state.

[0138] Then the forget gate activation value f t The forget gate determines how much information is discarded from the cell state, denoted as f t =σ(W f ·[h t-1 ,S t ]+b f ),

[0139] Among them, W f is the comprehensive spatiotemporal feature vector S of the input t To the weight matrix of the forget gate, b f is the bias vector of the forget gate.

[0140] Then continue with cell state c t The updated calculation is expressed as:

[0141] Among them, c t-1 is the cell state at the previous time step, and ⊙ represents element-wise multiplication.

[0142] The input gate activation value i t The role of controlling how many new candidate cell states there are Can be updated to the current cell state c t , and the value range is (0,1), because the input gate activation value i t Calculated through a sigmoid function.

[0143] when i t When the value is close to 1, it means that more new information, that is, cell state, is allowed. Influx cell state c t On the contrary, when i tWhen the value is close to 0, it means that the flow of new information into the cell state is restricted, thus effectively controlling the update of information.

[0144] Finally, the output gate is calculated, where the output gate activation value o t , expressed as: o t =σ(W o ·[h t-1 ,S t ]+b o ),

[0145] Where W o is the comprehensive spatiotemporal feature vector S of the input t The weight matrix to the output gate; b o is the bias vector for the output gate.

[0146] Hidden state h t The update formula is expressed as h t =o t ⊙tanh(c t ), the hidden state is the output of the cell state after the function is controlled by the output gate. It will be used as one of the inputs of the next time step, and can also be used as the output of the LSTM network. The characteristic vector H of the trend of security risk over time t , H t =[h 1 ,h 2 ,…,h t ].

[0147] The risk level assessment is to judge the changing trend of the risk by observing the changes in the hidden states output by the LSTM network at different time steps, calculate the difference or similarity index between the hidden states of adjacent time steps, and calculate the Euclidean distance. If it gradually increases, it means that the risk is increasing; if it gradually decreases, it means that the risk is decreasing; if it remains within the set threshold range, that is, relatively stable, the risk may remain stable; in addition, indicators such as cosine similarity are also used to measure the changing direction of the hidden state, so as to judge the risk trend.

[0148] The risk assessment module combines the attention mechanism to identify risk factors, combines the LSTM network to obtain the characteristic vector of the trend of security risk over time with the attention mechanism to calculate the attention weight, and obtains the characteristic vector A output by the attention mechanism t , expressed as: A t =Attention(H t ), wherein the calculation of attention weight specifically includes:

[0149] The attention mechanism dynamically allocates attention to different modes and time steps, identifies key risk factors, enhances the model's ability to capture important information, and improves the accuracy of risk assessment. The attention mechanism layer identifies the most critical factors for security risk assessment by allocating different attention levels, thereby improving the response speed and accuracy of the risk assessment module.

[0150] The dimensions of the query, key, and value matrices are related to the hidden state dimensions of the LSTM network. Since the hidden state dimension of the LSTM network is d h , set the dimension of the query and key matrix to d q =d k =d h , the dimension of the value matrix is ​​set to the same as the hidden state dimension d as needed v =d h ,This setting enables the attention mechanism to effectively calculate the attention weights in the feature vector space output by the LSTM network.

[0151] The specific calculation attention weight of the attention mechanism is: where Q, K, V are query, key and value matrices, d k is the dimension of the key, where Q is the query matrix, which is obtained by transforming the hidden state of the LSTM network through a fully connected layer; K is the key matrix, which is also transformed by the hidden state of the LSTM network through another fully connected layer; V is the value matrix, which directly uses the hidden state of the LSTM network or undergoes a simple transformation. By calculating the attention weights, the importance weight of each time step and each feature vector dimension in the current risk assessment can be determined.

[0152] According to the calculated attention weight, the characteristic vector matrix A output by the attention mechanism is obtained t , A t =[a 1 ,a 2 ,…,a t ], the attention weight is calculated by the scaled dot product attention formula.

[0153] The risk assessment module also includes calculating a key risk factor feature vector based on dynamic risk assessment combined with attention weights to visualize the attention weights; the key risk factor feature vector is a weighted summation of the calculated characteristic vector of the trend of changes in security risks over time and the attention weights to obtain an attention-weighted feature vector.

[0154] Let the attention weight be a t , the hidden state of the LSTM network at time step is h t , then the key risk factor feature vector after attention weighting is This weighted eigenvector highlights the eigenvectors and time steps that contribute more to risk assessment, enabling the system to focus more on key risk factors.

[0155] In order to better understand the key risk factors, the attention weights are visualized. A heat map can be drawn and displayed in the human-computer interaction module, where the horizontal axis represents the time step, the vertical axis represents the feature vector dimension, and the color depth in the heat map represents the size of the attention weight. Through this visualization method, security managers can intuitively see which feature vectors play a key role in risk assessment at different time points, so as to better understand the decision-making process of the model and take corresponding risk management measures based on these key factors. At the same time, further analysis and interpretation of key risk factors are also carried out, such as calculating the importance score of feature vectors through the back-propagation algorithm, determining which sensor data, behaviors in monitored videos, or events in text logs have a decisive impact on risk assessment, so as to optimize the monitoring system and improve the safety management process in a targeted manner.

[0156] Through the above steps, the risk assessment module can effectively use the LSTM network and attention mechanism to dynamically assess the safety risks of hazardous chemical enterprises, accurately identify risk trends and key risk factors, provide enterprises with timely and reliable risk assessment results, and support enterprises to make scientific and reasonable safety decisions. In the actual implementation process, it is necessary to continuously optimize the model parameters and adjust the design of the attention mechanism to adapt to the complex environment and diversified risk feature vectors of different hazardous chemical enterprises.

[0157] The risk classification module uses the softmax function to convert the output into the probability distribution of risk categories and determine the risk level. The risk classification layer divides the output into three categories: high risk, medium risk, and low risk according to the model training results, providing enterprises with clear risk assessment results, including:

[0158] First, build a softmax classification layer and use the feature vector output by the LSTM network combined with the attention mechanism as input; the softmax function converts the input vector into a probability distribution, and each element represents the probability of the corresponding risk category.

[0159] The Softmax classification module algorithm is expressed as: t =Softmax(A t ),

[0160] The Softmax function is expressed as:

[0161] Among them, e is a natural constant, The feature vector matrix A output by the attention mechanism tEach element in is exponentially operated separately, mapping the input value to a non-negative range and amplifying the difference between larger and smaller values; i is the risk category, It is the sum of the index values ​​corresponding to the three risk categories of high risk, medium risk and low risk; t is the probability distribution of the corresponding risk category, indicating the characteristic vector matrix A output by the attention mechanism at time t t The probability of belonging to the i-th category.

[0162] The Softmax function converts the output of the model into an intuitively understandable probability distribution of risk categories, making it easier for companies to judge risk levels based on probability values ​​and take corresponding risk management measures.

[0163] The final risk classification result is determined based on the preset probability threshold; when the value of the probability distribution of the risk category exceeds the set high threshold, the risk is judged as high risk; when the value of the probability distribution of the risk category is less than the set low threshold, the risk is judged as low risk; when the value of the probability distribution of the risk category is between the set high threshold and the low threshold, the risk is judged as medium risk.

[0164] According to the calculation of the sequentially connected data acquisition module, data processing module, feature extraction module, spatiotemporal fusion module, risk assessment module and risk classification module, the overall algorithm model of the security risk analysis system based on multimodal data processing of the present invention can be obtained as follows:

[0165] O t =Softmax(A t )

[0166] =Softmax(Attention(H t ))

[0167] =Softmax(Attention(LSTM(S t )))

[0168] =Softmax(Attention(LSTM(Fuse(F v ,F a ,F t ))))

[0169] In this embodiment, the system further includes an alarm module and a human-computer interaction module.

[0170] When the risk classification module identifies a high-risk hidden danger, the alarm module is automatically triggered to send an alarm to the enterprise administrator or relevant personnel and generate a detailed risk assessment report. The report content includes risk category, risk factors, time and location of risk occurrence, so that the enterprise can take emergency measures in time to reduce the possibility of safety accidents.

[0171] Establish interfaces with enterprise communication systems, including email servers, SMS gateways or internal instant messaging tools. When the risk assessment module identifies high-risk hazards, it automatically generates a risk assessment report. The report includes information such as risk category, risk factors, time and location of risk occurrence, etc.

[0172] The key feature vectors and time steps of risk factors are obtained through attention mechanism analysis. The time of risk occurrence is determined according to the data collection time, and the location is determined according to the location information of sensors or monitoring devices.

[0173] Use the selected communication method to send alarm information and risk assessment reports to enterprise administrators or relevant responsible personnel to ensure that relevant personnel are notified in time to take emergency measures.

[0174] The human-computer interaction module provides a user-friendly operation interface, where enterprise personnel can view risk reports, adjust risk assessment parameters and alarm thresholds to meet the specific security management needs of the enterprise.

[0175] The human-computer interaction module allows users to customize risk assessment parameters and alarm thresholds, providing a user-friendly operation interface through which enterprise managers and security personnel can view risk assessment reports and monitor risk status in real time. They can also customize risk assessment parameters according to the actual needs of the enterprise, such as adjusting the weights of different risk factors, and configure alarm module thresholds to make the system more in line with the enterprise's personalized security management needs.

[0176] Develop web applications or desktop applications as human-computer interaction modules, and use front-end technology to build user-friendly interface layouts, including risk report display areas, real-time risk monitoring charts, parameter setting panels, etc.

[0177] Appropriate server-side technologies are used on the back end to interact with the system, including obtaining risk assessment results, querying historical data, receiving user-set parameters, such as risk assessment parameter adjustment, alarm threshold setting, etc., and updating data in real time on the front-end interface to provide a real-time and intuitive user experience.

[0178] In this embodiment, a security risk analysis method based on multimodal data processing is also included, and the method is implemented based on the system, such as Figure 2 , including the following steps:

[0179] Step S1, perform data collection; electrically connect to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; collect the monitoring video data, various sensor data, and operation log text data of the hazardous chemical enterprise in real time as raw data.

[0180] Step S2, perform data processing; clean, format and normalize the collected raw data in turn, and extract the spatiotemporal feature vector in the raw data, the spatiotemporal feature vector includes the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data.

[0181] Step S3, extracting feature vectors from the normalized data through a neural network structure, outputting visual feature vectors, physical feature vectors and text feature vectors, and using the visual feature vectors, physical feature vectors and text feature vectors as multimodal feature vectors.

[0182] Step S4, performing spatiotemporal fusion of the multimodal feature vector; fusing the multimodal feature vector with the spatiotemporal feature vector, outputting a comprehensive spatiotemporal feature vector, and using the comprehensive spatiotemporal feature vector for risk assessment.

[0183] Step S5, perform risk assessment; use the LSTM network to process the integrated spatiotemporal feature vector after fusion, capture the changing trend of security risks over time, calculate the attention weight, and obtain the feature vector output by the attention mechanism; output the key risk factor feature vector based on the dynamic risk assessment combined with the attention weight.

[0184] Step S6: classify the risk assessment results; use the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divide the risks into three categories: high risk, medium risk, and low risk according to the probability distribution.

[0185] In this embodiment, the method applied to the system further includes step S7 and step S8.

[0186] Step S7: When the model identifies a high-risk hidden danger, it will automatically alert the administrator and provide a risk assessment report.

[0187] Step S8: Provide a user-friendly interface that allows users to view risk reports, adjust risk assessment parameters, and configure alarm modules.

[0188] By working together, each module can realize comprehensive monitoring and assessment of safety risks of hazardous chemical enterprises. Compared with the traditional method that relies on manual inspection, the present invention can integrate various types of data, extract features using convolutional neural network CNN and recurrent neural network RNN ​​methods, and conduct comprehensive analysis of these features in the spatiotemporal fusion layer, thereby capturing the spatiotemporal correlation between features of different modes and achieving more accurate and real-time safety risk assessment.

[0189] It can monitor and evaluate the safety risks of hazardous chemical enterprises in real time, and automatically integrate multimodal data such as monitored videos, sensor data, and operation logs. It extracts spatial features through convolutional neural networks, processes time series data using recurrent neural networks and long short-term memory networks, and further extracts and fuses features to achieve comprehensive and accurate safety risk assessment. At the same time, the system also identifies key risk factors through the attention mechanism, and classifies and outputs risks through the softmax function, thereby providing real-time warnings for safety managers to ensure timely discovery of potential safety hazards.

[0190] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A security risk analysis system based on multimodal data processing, characterized in that: It includes a data acquisition module, a data processing module, a feature extraction module, a time-space fusion module, a risk assessment module, and a risk classification module which are sequentially connected; wherein: The data acquisition module is electrically connected to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; it collects the monitoring video data, various sensor data and text data of the operation log of the hazardous chemical enterprise in real time as raw data, and transmits the raw data to the data processing module; The data processing module sequentially cleans, formats and normalizes the collected raw data, and extracts the spatiotemporal feature vectors in the raw data, the spatiotemporal feature vectors including the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data; transmits the normalized data to the feature extraction module, and transmits the spatiotemporal feature vectors to the spatiotemporal fusion module; The feature extraction module extracts feature vectors from the normalized data through a neural network structure, outputs visual feature vectors, physical feature vectors, and text feature vectors, and transmits the visual feature vectors, physical feature vectors, and text feature vectors as multimodal feature vectors to the spatiotemporal fusion module; The spatiotemporal fusion module fuses the multimodal feature vector with the spatiotemporal feature vector to output a comprehensive spatiotemporal feature vector; The risk assessment module uses the LSTM network to process the integrated spatiotemporal feature vector after fusion, captures the feature vector of the changing trend of the security risk over time as a dynamic risk assessment; combines the attention mechanism to identify the risk factors, calculates the attention weight, and obtains the feature vector output by the attention mechanism; The risk classification module uses the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divides the risks into three categories: high risk, medium risk, and low risk according to the probability distribution.

2. A security risk analysis system based on multimodal data processing according to claim 1, characterized in that: The data processing module performs cleaning processing on the collected raw data, including: cleaning video frames, sensor data and text data; using the median filter image processing algorithm to detect and remove noise and abnormal points in the video frame; identifying and deleting abnormal data according to the set threshold range of the sensor data; and removing irrelevant characters, garbled characters and format errors from the text data through the regular expression technology; The formatting processing of the data processing module includes: uniformly representing video frames of different monitoring video data in a visual format; uniformly representing physical quantities of different types of sensor data in a physical format; uniformly representing text data of the operation log in a text format; uniformly converting the timestamp of the video frames of the monitoring video data and the installation location information of the monitoring device, the acquisition time and installation location identifier of the sensor data, and the time information and location information of the text data into a specific time format and location format; The normalization processing of the data processing module adopts linear normalization processing, and normalizes the video frame, each sensor data and text data respectively, and scales the data to the interval [0,1]. The normalization processing calculation is expressed as X is the original data, X min is the minimum value in the original data, X max is the maximum value in the original data, X norm It is the data after normalization; The data processing module extracts the spatiotemporal feature vector from the original data, including: For video data, extract the timestamp of the video frame as the time feature vector, and combine it with the installation location information of the monitoring equipment as the spatial feature vector, and output the spatiotemporal feature vector; the installation location information refers to the area number and coordinates of the monitoring equipment in a specific enterprise; For sensor data, the collection time is used as the time feature vector, and combined with the sensor installation location identifier as the space feature vector, the spatiotemporal feature vector is output; wherein the installation location identifier is the specific location coordinates or area name in the enterprise; For text data, the time information is used as the time feature vector, and combined with the location information as the spatial feature vector, the spatiotemporal feature vector is output.

3. The security risk analysis system based on multimodal data processing according to claim 1, characterized in that: The feature extraction module uses a convolutional neural network to process video data, extracts visual feature vectors, and uses a recurrent neural network to process sensor data and text data respectively. The recurrent neural network extracts physical feature vectors from sensor data, and the recurrent neural network extracts text feature vectors from text data, mining the time series and semantic information in multimodal feature vectors; the feature extraction module calculation includes: The visual feature Fv is expressed as: F v =CNN(X v ), The physical characteristic Fa is expressed as: F a =RNN(X a ), The text feature Ft is expressed as: F t =RNN(X t ), Where: X v is the normalized video data, X a is the normalized sensor data, X t It is the normalized text data.

4. A security risk analysis system based on multimodal data processing according to claim 3, characterized in that: The visual feature vector is extracted by using a convolutional neural network (CNN) model to perform transfer learning on the video data to extract the visual feature vector in the video data; the video frame is input into the CNN model, and after multi-layer convolution and pooling operations, a feature vector representing key information of the image is obtained, and the visual feature vector is output; The physical feature vector is extracted by using a recurrent neural network (RNN) model, and the sensor data is input into the RNN model in chronological order. The RNN model extracts the physical feature vector that can reflect the change trend of the physical process by learning the time sequence in the data, and outputs the physical feature vector; The text feature vector is extracted by using an RNN model to process text data, and the words in the text are converted into vector representations using word vector technology to obtain a text sequence, and then the text sequence is input into the RNN model. The RNN model learns the semantic information in the text and extracts text feature vectors that can represent operation behaviors, event descriptions, time and place.

5. The security risk analysis system based on multimodal data processing according to claim 1, characterized in that: The spatiotemporal fusion module fuses the multimodal feature vector with the spatiotemporal feature vector, specifically including: setting weights of different modal feature vectors in the fusion process, weighted summing the visual feature vector, the physical feature vector and the text feature vector according to their respective set weights, and obtaining the fused spatiotemporal feature vector as the comprehensive spatiotemporal feature vector; using the fusion algorithm to calculate the comprehensive spatiotemporal feature vector S t , expressed as: St = Fuse (Fv, Fa, Ft), where F v is the visual feature vector, F a is the physical eigenvector, F t is the text feature vector, S t It is a comprehensive spatiotemporal feature vector, and the Fuse function is a weighted summation fusion algorithm.

6. A security risk analysis system based on multimodal data processing according to claim 1, characterized in that: The risk assessment module uses a long short-term memory LSTM network to process the fused spatiotemporal feature vector for dynamic risk assessment, specifically including the following steps: firstly, the number of layers and the number of hidden units of the long short-term memory LSTM network are set to determine the network structure; then, the input and output dimensions are set; finally, the comprehensive spatiotemporal feature vector is input to obtain the risk level assessment; The input dimension should be the same as the dimension of the comprehensive spatiotemporal feature vector output by the spatiotemporal fusion module; if the dimension of the feature vector output by the spatiotemporal fusion module is d, then the input dimension of the LSTM network is d; the comprehensive spatiotemporal feature vector after spatiotemporal fusion includes a visual feature vector, a physical feature vector and a text feature vector; the visual feature vector in the monitored video has a dimension of m; the physical feature vector in the sensor data has a dimension of n, and the text feature vector of the operation log has a dimension of p; the dimension of the spatiotemporal feature vector obtained after the fusion operation is d=m+n+p, where d is the input dimension of the LSTM network; The output dimension is determined according to the number of risk classification categories, which are divided into three categories: high risk, medium risk, and low risk. The output dimension is 3; The risk level assessment is obtained by inputting the comprehensive spatiotemporal feature vector sequence into the established long short-term memory LSTM network; Input the comprehensive spatiotemporal feature vector sequence. Input the comprehensive spatiotemporal feature vector sequence generated by the spatiotemporal fusion module into the LSTM network in sequence according to the time step; The length of the time series of hazardous chemical enterprise data collected by processing the comprehensive spatiotemporal feature vector is set to T. At each time step t (1≤t≤T), the corresponding comprehensive spatiotemporal feature vector S t Input into the LSTM network to obtain the characteristic vector Ht of the trend of security risk over time, expressed as H t =LSTM(S t ); At each time step, the LSTM network generates a new feature vector based on the current input feature vector and the hidden state h of the previous time step. t-1 And cell state c t-1 To calculate the hidden state h of the current time step t and cell state c t , the calculation process involves the gating mechanism of the LSTM network, which includes the forget gate, input gate, cell state update and output gate. The specific process includes: First calculate the input gate, which includes the input gate activation function and the input modulation gate; Input gate activation function i t , denoted as i t =σ(W i ·[h t-1 ,S t ]+b i ), Where: i t is the input gate activation value at time t; σ is the sigmoid function, whose input range is (0,1); W i is the comprehensive spatiotemporal feature vector S of the input t To the weight matrix set by the input gate, the comprehensive spatiotemporal feature vector S t It is also the input vector at time t; h t-1 is the hidden state of the previous time step; b i is the bias vector of the input gate; Input modulation gate Expressed as in: is the candidate cell state, tanh is the hyperbolic tangent function, and its output range is (-1,1); W c is the comprehensive spatiotemporal feature vector S of the input t to the weight matrix set for the candidate cell state; b c is the bias vector of the candidate cell state; Then the forget gate activation value f t The calculation of f t =σ(W f ·[h t-1 ,S t ]+b f ), Among them, W f is the comprehensive spatiotemporal feature vector S of the input t To the weight matrix of the forget gate, b f is the bias vector of the forget gate; Then continue with cell state c t The updated calculation is expressed as: Among them, c t-1 is the cell state at the previous time step, ⊙ represents element-wise multiplication; The input gate activation value i t The role of is to control how many new candidate cell states there are Can be updated to the current cell state c t Finally, the output gate is calculated, where the output gate activation value o t , expressed as: o t =σ(W o ·[h t-1 ,S t ]+b o ), Where W o is the comprehensive spatiotemporal feature vector S of the input t The weight matrix to the output gate; b o is the bias vector of the output gate; Hidden state h t The update formula is expressed as ht = ot⊙tanh(ct). The hidden state is the output of the cell state after the function is controlled by the output gate. It will be used as one of the inputs of the next time step and can also be used as the output feature vector H of the LSTM network. t , H t =[h1,h2,…,h t ]; The risk level assessment is to judge the changing trend of the risk by observing the changes in the hidden states output by the LSTM network at different time steps, calculating the difference or similarity index between the hidden states of adjacent time steps, and calculating the Euclidean distance. If it gradually increases, it means that the risk is increasing; if it gradually decreases, it means that the risk is decreasing; if it is maintained within the set threshold range, the risk may remain stable.

7. A security risk analysis system based on multimodal data processing according to any one of claims 1 or 6, characterized in that: The risk assessment module combines the attention mechanism to identify risk factors, combines the LSTM network to obtain the characteristic vector of the trend of security risk over time with the attention mechanism to calculate the attention weight, and obtains the characteristic vector A output by the attention mechanism t , expressed as: A t =Attention(H t ), wherein the calculation of attention weight specifically includes: Set the dimension of the query and key matrix to d q =d k =d h , the dimension of the value matrix is ​​set to the same as the hidden state dimension d as needed v =d h , the specific calculation attention weight of the attention mechanism is: where Q, K, V are query, key and value matrices, d k is the dimension of the key; According to the calculated attention weight, the characteristic vector matrix A output by the attention mechanism is obtained t , A t =[a1,a2,…,a t ].

8. A security risk analysis system based on multimodal data processing according to any one of claim 1, characterized in that: The risk assessment module also includes calculating a key risk factor feature vector based on the dynamic risk assessment combined with the attention weight to visualize the attention weight; the key risk factor feature vector is obtained by weighted summing the calculated security risk change trend feature vector over time and the attention weight to obtain the attention-weighted feature vector; assuming that the attention weight is a t , the hidden state of the LSTM network at time step is h t , then the key risk factor feature vector after attention weighting is 9. A security risk analysis system based on multimodal data processing according to any one of claims 1 or 7, characterized in that: The risk classification module uses a softmax function, specifically including: First, a softmax classification layer is constructed, and the feature vector output by the LSTM network combined with the attention mechanism is used as input; the softmax function converts the input vector into a probability distribution, and each element represents the probability of the corresponding risk category; The Softmax classification module algorithm is expressed as: t =Softmax(A t ), The Softmax function is expressed as: Among them, e is a natural constant, The feature vector matrix A output by the attention mechanism t Each element in is exponentially calculated separately; i is the risk category, It is the sum of the index values ​​corresponding to the three risk categories of high risk, medium risk and low risk; t is the probability distribution of the corresponding risk category, indicating the characteristic vector matrix A output by the attention mechanism at time t t The probability of belonging to the i-th category; The final risk classification result is determined based on the preset probability threshold; when the value of the probability distribution of the risk category exceeds the set high threshold, the risk is judged as high risk; when the value of the probability distribution of the risk category is less than the set low threshold, the risk is judged as low risk; when the value of the probability distribution of the risk category is between the set high threshold and the low threshold, the risk is judged as medium risk.

10. A security risk analysis method based on multimodal data processing, characterized in that: The method is implemented based on a security risk analysis system based on multimodal data processing according to any one of claims 1 to 9, and comprises the following steps: Step S1, data collection; electrically connected to the monitoring system, sensor and operation log database interface of the hazardous chemical enterprise; real-time collection of monitoring video data, various sensor data, and text data of the operation log of the hazardous chemical enterprise as raw data; Step S2, performing data processing; cleaning, formatting and normalizing the collected raw data in turn, and extracting the spatiotemporal feature vector in the raw data, the spatiotemporal feature vector includes the timestamp of the video frame of the monitoring video data and the installation location information of the monitoring device, the collection time and installation location identifier of the sensor data, and the time information and location information of the text data; Step S3, extracting feature vectors from the normalized data through a neural network structure, outputting visual feature vectors, physical feature vectors, and text feature vectors, and using the visual feature vectors, physical feature vectors, and text feature vectors as multimodal feature vectors; Step S4, performing spatiotemporal fusion of the multimodal feature vector; fusing the multimodal feature vector with the spatiotemporal feature vector, outputting a comprehensive spatiotemporal feature vector, and using the comprehensive spatiotemporal feature vector for risk assessment; Step S5, perform risk assessment; use the LSTM network to process the integrated spatiotemporal feature vector after fusion, capture the changing trend of security risks over time, calculate the attention weight, and obtain the feature vector output by the attention mechanism; output the key risk factor feature vector based on the dynamic risk assessment combined with the attention weight; Step S6: classify the risk assessment results; use the softmax function to calculate the probability distribution of risk categories based on the feature vector output by the attention mechanism, and divide the risks into three categories: high risk, medium risk, and low risk according to the probability distribution.

Images