A knowledge-data dual-driven association modeling method and system

Abstract

The present invention relates to a knowledge-data dual-driven association modeling method and system. The method first acquires video surveillance data, sensor data, and domain safety specification knowledge, and performs frame extraction and feature extraction, time series normalization, and structured representation and encoding. Visual features, time series features, and semantic features are then extracted to obtain security knowledge features. A hypergraph neural network is used to model multimodal data and its high-order associations with security knowledge. The multimodal features and security knowledge features are input into the hypergraph neural network, and hypergraph convolution operations are used to achieve deep fusion of multimodal data and security knowledge, generating a unified representation space. A safety hazard identification model is constructed, and the system is used to accurately identify and monitor safety hazards in complex scenarios in real time. This method addresses the heterogeneity of multimodal data and the semantic gap between modalities, improving the accuracy and real-time performance of safety hazard identification.

Description

Technical Field

[0001] The present invention belongs to the technical field of data modeling and safety hazard identification, and in particular relates to a knowledge-data dual-driven association modeling method and system. Background Art

[0002] With the continuous advancement of industrial automation and intelligence, video surveillance, sensor networks, and safety knowledge have become crucial sources of information for ensuring safe operations. In complex scenarios, a single data source often fails to fully describe potential safety hazards. Traditional single-data analysis methods struggle to provide effective safety identification and real-time monitoring, especially when complex relationships exist between multimodal data and safety knowledge. Therefore, integrating multimodal data with domain knowledge to improve the efficiency and accuracy of identifying safety hazards has become a key challenge in the current security monitoring field.

[0003] In recent years, hypergraph neural networks (HGNNs), a novel deep learning method, have achieved remarkable results in fields such as image recognition and recommendation systems due to their ability to handle complex relationships and high-order associations. By defining hyperedges to connect different types of nodes, HGNNs can effectively model the relationships between multimodal data, providing a new approach for identifying safety hazards. However, existing research has largely focused on processing single-modal data and lacks a systematic approach to deeply integrating multimodal data with safety regulatory knowledge.

[0004] To address this issue, this paper proposes a knowledge-data dual-driven association modeling approach. Using a hypergraph neural network, this approach deeply integrates video surveillance data, sensor data, and safety regulations, enabling accurate identification and real-time monitoring of safety hazards. While improving the comprehensive utilization of multimodal data, it also effectively captures the potential threats of safety hazards, providing a more intelligent solution for security management in complex scenarios. Summary of the Invention

[0005] The purpose of the present invention is to provide a knowledge-data dual-driven association modeling method and system for heterogeneous processing of multimodal data in complex scenarios, elimination of semantic gaps between modalities, and integration of domain safety specification knowledge with real-time multimodal data, thereby improving the accuracy and real-time performance of safety hazard identification.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, a knowledge-data dual-driven association modeling method is provided, comprising the following steps:

[0008] S1: Acquire video surveillance data, sensor data, and domain safety specification knowledge, perform frame extraction and feature extraction on video surveillance data, perform time series normalization on sensor data, and perform structured representation and encoding on safety specification knowledge;

[0009] S2: Extract visual features of specified dimensions from video surveillance data at specified time intervals, extract time series features of specified dimensions from sensor data to obtain multimodal features, and extract semantic features of specified dimensions from safety specification knowledge to obtain safety knowledge features;

[0010] S3: Build a hypergraph neural network and use it to model multimodal data and its high-order associations with security knowledge;

[0011] S4: Inputting the multimodal features and security knowledge features into the hypergraph neural network, achieving deep fusion of multimodal data and security knowledge through hypergraph convolution operations, and generating a unified representation space;

[0012] S5: Based on the fused unified representation space, a safety hazard identification model is constructed, and safety hazards in complex scenarios are accurately identified and monitored in real time through the safety hazard identification model.

[0013] Preferably, the specific process of performing frame extraction and feature extraction on video surveillance data, performing time series normalization processing on sensor data, and performing structured representation and encoding on safety specification knowledge in step S1 is as follows:

[0014] S11: acquiring video data by sampling at fixed time intervals and extracting video frames, performing feature extraction on the video frames, and extracting objects and their behavior features in each frame using a target detection algorithm;

[0015] S12: Unify the timestamp format and build a time series prediction model to fill in long-term missing data. Give more weight to recent data based on the timestamp, and give historical data an exponential decay weight according to preset rules to achieve noise suppression. Through timestamp synchronization and spatial coordinate conversion, unify the multi-source data benchmark and perform standardization.

[0016] S13: Split the text of safety specification knowledge into atomic knowledge units, bind risk levels and protection requirements based on hazard source identification results, output quadruple of labeled knowledge fragments, define entities and relationships, add attributes to entities, build attribute graph model, nodes represent entities, edges represent relationships, both edges and nodes can carry attributes, extract entity instances from preprocessed data, identify relationships between entities, handle entity conflicts in data from different sources, store entities, relationships and attributes in a graph database, establish efficient index, and build a safety specification knowledge base.

[0017] Preferably, the specific process of step S2 is as follows:

[0018] S21: Normalize all video frame images to obtain images of a specified resolution and standardize their pixel values, load the preset model, remove the top classifier, retain the convolutional base, and extract the visual feature vector of a specified dimension;

[0019] S22: Construct a two-layer LSTM network model. The input layer accepts n×d dimensional data, where n is the time step and d is the sensor dimension. The last time step output is used as the feature representation to finally obtain the time series features.

[0020] S23: Add [CLS] and [SEP] tags to the atomized knowledge unit, load the Chinese BERT-base model, and extract the specified dimension vector corresponding to the [CLS] tag as the semantic feature of the safety specification knowledge.

[0021] Preferably, the hypergraph neural network includes multiple nodes and hyperedges, the multiple nodes include video frames, sensor data points, and security knowledge entities, and the hyperedges are used to represent the complex associations between multimodal data and knowledge;

[0022] The hypergraph neural network extracts features from nodes and hyperedges, including attribute features of nodes or attribute features of hyperedges, classifies or clusters nodes based on the features of nodes, and finds K nearest neighbor nodes for each node;

[0023] The features of the nodes of the hypergraph neural network are transmitted through hyperedges to represent the relationship between the nodes; the features of the hyperedges are transmitted through the nodes to represent the relationship between the hyperedges;

[0024] The fully connected layer of the hypergraph neural network integrates the features of nodes and hyperedges, performs linear transformation through the weight matrix, and extracts higher-level features;

[0025] The features of the aggregated nodes and hyperedges of the hypergraph neural network are further extracted through convolution operations to capture the global connection pattern.

[0026] Preferably, the specific process of constructing a hyperedge using a preset algorithm is as follows:

[0027] S31: Calculate similarity based on the improved similarity calculation formula. The specific formula is as follows:

[0028] ;

[0029] in, u , v are the feature vectors of the two nodes, ||·||2 is the L2 norm;

[0030] S32: Perform hypergraph convolution operation. The specific formula is as follows:

[0031] ;

[0032] S31: Calculate similarity based on the improved similarity calculation formula. The specific formula is as follows:

[0033] ;

[0034] in, u , v are the feature vectors of the two nodes, ||·||2 is the L2 norm;

[0035] S32: Perform hypergraph convolution operation. The specific formula is as follows:

[0036] ;

[0037] in, H is the hypergraph association matrix, which represents the connection relationship between vertices or nodes and hyperedges. v i Belong to the super edge e j ,but H ij =1, otherwise 0; D v is the vertex degree matrix, the diagonal elements D v ( i , i ) indicates the i The number of hyperedges connecting vertices, D e is the hyperedge degree matrix, the diagonal elements D e ( j , j ) indicates the j The number of nodes connected by a hyperedge; W e is the weight matrix of the hyperedge, is a diagonal matrix, indicating the importance or trust of each hyperedge; For the l Layer to l +1 trainable weight matrix for feature dimension transformation; represents a nonlinear activation function; H (l) For the l Node feature representation of the layer.

[0038] Preferably, in step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and the deep fusion of multimodal data and security knowledge is achieved through the hypergraph convolution operation to generate a 512-dimensional unified representation space. The specific process is as follows:

[0039] The input image is first processed by a 7×7 convolution layer and a 3×3 maximum pooling layer, and the output size is 112×112;

[0040] Entering the backbone stage consisting of four stacked Bottleneck modules, the output feature maps are 256, 512, 1024, and 2048 channels respectively. The corresponding spatial resolution is halved layer by layer until it is finally reduced to 7×7.

[0041] In each Bottleneck module, a three-layer convolution structure of "1×1→3×3→1×1" is used to reduce the dimension first and then increase the dimension to reduce the computational cost while maintaining the expressive power;

[0042] Each Bottleneck module has a main path and a shortcut path. If the output dimensions of the two are different, a 1×1 convolution is used to align the shortcut dimensions.

[0043] At the end of the network, the 7×7×2048 feature map is compressed into a 2048-dimensional vector through global average pooling.

[0044] Preferably, in step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and a deep fusion of multimodal data and security knowledge is achieved through a hypergraph convolution operation. The specific formula for generating a unified representation space is as follows:

[0045] ;

[0046] in, represents the visual feature vector extracted from the video data, h t represents the state feature vector from the time series, e k Represents the structured security knowledge vector extracted through the knowledge graph; W v 、W s 、W k are the corresponding linear mapping weight matrices, which are used to map the features of each modality to the same 512-dimensional space; || represents the vector-level splicing operation.

[0047] Preferably, the safety hazard identification model in step S5 adopts a two-layer perceptron, the input is the fused unified representation space, and the output is the probability distribution of the safety hazard:

[0048] ;

[0049] in,W 1 is the weight matrix of the first layer of perceptron, which compresses the high-dimensional input to 256 dimensions; b 1 is the first layer bias term; W 2 is the weight matrix of the second layer perceptron, ,in C The total number of categories representing safety hazards; b 2 is the second layer bias term; As the activation function, the softmax function normalizes the output value into a probability distribution.

[0050] In a second aspect, a knowledge-data dual-driven association modeling system is provided, which is used to implement the knowledge-data dual-driven association modeling method, including a multimodal data acquisition and preprocessing unit, a multimodal data feature extraction unit, a hypergraph neural network construction unit, a multimodal data and knowledge fusion unit, and a safety hazard identification and real-time monitoring unit;

[0051] The multimodal data acquisition and preprocessing unit is used to acquire video surveillance data, sensor data, and domain safety specification knowledge, and perform preprocessing;

[0052] The multimodal data feature extraction unit is used to extract visual features of video surveillance data, time series features of sensor data, and semantic features of safety specification knowledge;

[0053] The hypergraph neural network construction unit is used to construct a hypergraph neural network to model multimodal data and its high-order association with security knowledge;

[0054] The multimodal data and knowledge fusion unit is used to input multimodal features and security knowledge features into the hypergraph neural network to generate a unified representation space;

[0055] The safety hazard identification and real-time monitoring unit is used to construct a safety hazard identification model based on the fused unified representation space to achieve accurate identification and real-time monitoring of safety hazards.

[0056] The beneficial effects of the present invention include:

[0057] The knowledge-data dual-driven association modeling method and system provided by the present invention include multimodal data acquisition and preprocessing, which acquires video surveillance data, sensor data, and domain knowledge and performs preprocessing; multimodal data feature extraction, which extracts the visual features of video data, the time series features of sensor data, and the semantic features of safety specification knowledge; constructing a hypergraph neural network, which uses the hypergraph neural network to model multimodal data and its high-order associations with knowledge; fusion of multimodal data and knowledge, which inputs multimodal data and knowledge features into the hypergraph neural network and achieves deep fusion through hypergraph convolution operations; safety hazard identification and real-time monitoring, which builds a model based on the unified representation space after fusion to achieve accurate identification and real-time monitoring of safety hazards. By deeply integrating multimodal data and knowledge, an efficient association modeling solution is provided, which can effectively improve the accuracy and effectiveness of multi-source information analysis.

[0058] First, a hypergraph neural network unifies video, sensor data, and safety knowledge, resolving the fragmentation of multi-source data in traditional methods. The fused feature space significantly improves the accuracy of identifying safety hazards. Dynamic weighting using exponentially decaying weights based on time series data increases the contribution of recent key data, and noise suppression is significantly superior to traditional sliding window methods.

[0059] Second, the quadruple knowledge representation transforms safety regulations into machine-understandable semantic features, enabling the model's decision-making process to be traceable to regulatory compliance. An attribute graph model explicitly associates hazards with protective measures, significantly reducing false alarm rates in testing scenarios.

[0060] Third, the improved similarity calculation and hyperedge construction mechanism, as well as the application of functions in the hypergraph convolution formula, solve the gradient vanishing problem, accelerate the model convergence speed, and effectively improve the efficiency of modeling complex "multi-person-device-environment" relationships.

[0061] Fourth, hypergraph neural networks can effectively process high-order associations between multimodal data and security knowledge. Compared to traditional graph neural networks, they can capture the complex relationships between data and knowledge more comprehensively and deeply. Through hypergraph convolution operations, multimodal features and security knowledge features are input into the hypergraph neural network, achieving a deep fusion of multimodal data and security knowledge, generating a unified representation space. This allows different types of data and knowledge to interact and merge in the same space, fully leveraging their respective strengths, improving the model's ability to capture complex relationships, and providing more powerful model support for subsequent safety hazard identification.

[0062] Fifth, a safety hazard identification model is constructed based on the fused unified representation space, enabling accurate identification and real-time monitoring of safety hazards in complex scenarios. The unified representation space deeply integrates multimodal data and safety knowledge, enabling the model to comprehensively consider various factors and analyze safety hazards more comprehensively. The two-layer perceptron structure further processes and analyzes the fused features, enabling a quantitative assessment of safety hazards. This provides a scientific basis for safety decision-making, enables more accurate identification of safety hazards, and promptly detects potential safety issues. This improves the efficiency and accuracy of safety monitoring, effectively reducing the occurrence of safety accidents in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Schematic diagram of the process of the knowledge-data dual-driven association modeling method of the present invention.

[0064] Figure 2 Schematic diagram of the structure of the hypergraph neural network of the present invention.

[0065] Figure 3 Schematic diagram of the fusion principle of multimodal data and knowledge of the present invention.

[0066] Figure 4 Schematic diagram of the convolutional neural network structure of the present invention.

[0067] Figure 5 Schematic diagram of the BERT network structure of the present invention. DETAILED DESCRIPTION

[0068] The following is combined with Figures 1 to 5 The present invention is described in further detail:

[0069] Example 1

[0070] See attached Figure 1 As shown, a knowledge-data dual-driven association modeling method includes the following steps:

[0071] S1: Acquire video surveillance data, sensor data, and domain safety specification knowledge, perform frame extraction and feature extraction on video surveillance data, perform time series normalization on sensor data, and perform structured representation and encoding on safety specification knowledge;

[0072] S2: Extract visual features of specified dimensions from video surveillance data at specified time intervals, extract time series features of specified dimensions from sensor data to obtain multimodal features, and extract semantic features of specified dimensions from safety specification knowledge to obtain safety knowledge features;

[0073] S3: Build a hypergraph neural network and use it to model multimodal data and its high-order associations with security knowledge;

[0074] S4: Inputting the multimodal features and security knowledge features into the hypergraph neural network, achieving deep fusion of multimodal data and security knowledge through hypergraph convolution operations, and generating a unified representation space;

[0075] S5: Based on the fused unified representation space, a safety hazard identification model is constructed, and safety hazards in complex scenarios are accurately identified and monitored in real time through the safety hazard identification model.

[0076] For example, safety monitoring in a complex, large-scale chemical plant industrial scenario presents a critical need. The plant's production processes are complex, involving the storage, transportation, and processing of a wide range of hazardous chemicals. This presents numerous potential safety hazards, such as equipment failure, chemical leaks, and human misconduct. To achieve safety monitoring in this complex industrial scenario, this knowledge- and data-driven, dual-driven, associative modeling approach is employed.

[0077] Video surveillance data from the chemical plant is collected. Video surveillance covers key areas such as the main production workshops, storage tank areas, and transportation corridors. Video data is sampled at regular intervals and frames are extracted. Data from various sensors, including temperature, pressure, and gas concentration sensors, is also collected. These sensors are located throughout various equipment and areas, collecting real-time data on temperature, pressure, and gas concentration. Furthermore, domain safety regulations are acquired, such as chemical industry safety regulations, equipment operating procedures, and hazardous chemical management regulations. Frame extraction and feature extraction are performed on the video surveillance data. Object detection algorithms are used to extract object and behavioral features from each frame. For example, these algorithms can identify objects such as storage tanks, pipelines, valves, and personnel, as well as personnel behavior and equipment operating status. The sensor data is time-series normalized to a unified timestamp format. Because sensor failures or communication issues in industrial environments can lead to data loss, a time series prediction model is constructed to fill in missing data for extended periods. Recent data is given a higher weight based on the timestamp, and historical data is given an exponential decay weight according to preset rules to achieve noise suppression. The multi-source data benchmark is unified through timestamp synchronization and spatial coordinate conversion, and normalization is performed to convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0078] Safety specification knowledge is structured and encoded, breaking the text of safety specification knowledge into atomic knowledge units, such as "Immediately implement pressure reduction measures when storage tank pressure exceeds the threshold" and "Personnel entering hazardous areas must wear protective equipment." Based on the hazard source identification results, the risk level and protection requirements are bound together, and a labeled quadruple of knowledge fragments (entity, relationship, attribute, risk level) is output. Entities and relationships are defined, attributes are added to entities, and an attribute graph model is constructed. Nodes represent entities, including equipment, personnel, chemicals, etc., and edges represent relationships, including "belongs to," "connects," and "operates." Both edges and nodes can carry attributes. Entity instances are extracted from preprocessed data, and relationships between entities are identified. Entity conflicts in data from different sources are resolved. Entities, relationships, and attributes are stored in a graph database, and an efficient index is established to build a safety specification knowledge base.

[0079] Perform feature extraction on processed video surveillance data, sensor data, and safety regulations. Normalize all video frames to a specified resolution and standardize pixel values. Load the pre-set model, remove the top-level classifier, and retain the convolutional basis to extract a visual feature vector of a specified dimension. This feature vector effectively represents the objects and scene information in the image.

[0080] A two-layer LSTM network model is constructed to extract the time series features of sensor data. The input layer accepts n×d-dimensional data, where n is the time step, which is set to 100, and d is the sensor dimension, which is determined according to the actual number of sensors. The last time step output is used as the feature representation, and finally the time series feature is obtained. This feature can capture the temporal dependency and change trend in the sensor data.

[0081] The [CLS] and [SEP] tags are added to the atomic knowledge units, and the Chinese BERT-base model is loaded. The vector of the specified dimension corresponding to the [CLS] tag is extracted as the semantic feature of the safety specification knowledge. This feature can deeply understand the semantic connotation of safety specification knowledge. Multimodal features and safety knowledge features, including visual features and time series features, are obtained, providing high-quality input for subsequent modeling and fusion.

[0082] A hypergraph neural network is constructed, consisting of multiple nodes and hyperedges. Nodes include video frame nodes, sensor data point nodes, and security knowledge entity nodes. Hyperedges are used to represent complex associations between multimodal data and knowledge, such as associations between objects in video frames and security knowledge entities, or between sensor data points and security knowledge entities. The hypergraph neural network extracts features from nodes and hyperedges. These features include node attributes, such as visual features of video frames, temporal features of sensor data points, semantic features of security knowledge entities, or hyperedge attributes, such as the strength and type of associations. Classification or clustering is performed based on node features, finding the K nearest neighbors of each node. Node features in the hypergraph neural network are transferred through hyperedges, representing relationships between nodes; hyperedge features are transferred through nodes, representing relationships between hyperedges. The fully connected layer of the hypergraph neural network integrates node and hyperedge features, performing linear transformations using a weight matrix to extract higher-level features. The hypergraph neural network aggregates node and hyperedge features, further extracting features through convolution operations to capture global connectivity patterns. Using hypergraph neural networks to model multimodal data and its high-order associations with security knowledge can effectively process the complex relationship between multimodal data and security knowledge and capture high-order association information in data and knowledge.

[0083] Multimodal features and security knowledge features are input into a hypergraph neural network. Hypergraph convolution operations are used to achieve deep fusion of multimodal data and security knowledge, generating a unified representation space. A specific formula is used to map multimodal features and security knowledge features into the same 512-dimensional space, achieving unified representation and deep fusion. The visual feature vector, the time series state feature vector, and the security knowledge vector are mapped to the 512-dimensional space via a linear mapping weight matrix. These three are then concatenated via a vector-level concatenation operation to generate a unified 512-dimensional representation space.

[0084] Based on the fused unified representation space, a safety hazard identification model is constructed using a two-layer perceptron. The input is the fused unified representation space. The weight matrix W1∈R^(512×256) of the first-layer perceptron compresses the high-dimensional input to 256 dimensions, and the bias term b1∈R^(1×256). The weight matrix W2∈R^(256×C) of the second-layer perceptron is C, where C represents the total number of safety hazard categories. This is determined based on the actual safety hazard types of the chemical plant, such as equipment failure, chemical leak, and human misconduct. Assuming C=10, the bias term b2∈R^(1×10). The output values ​​are normalized into a probability distribution using the ReLU activation function and the softmax function, outputting the probability distribution of the safety hazard. In the actual safety monitoring of the chemical plant, real-time video surveillance data, sensor data, and safety regulations are processed and analyzed according to the above steps. The resulting unified representation space is input into the safety hazard identification model, which outputs the probability distribution of each safety hazard category. Based on the preset probability threshold, the existence of a safety hazard, its type, and its severity are determined. For example, when the probability of a "chemical leak" category exceeds a preset threshold, a potential chemical leak is identified and an alarm is issued. This information is also sent to safety managers so they can take timely action. This method enables precise identification and real-time monitoring of safety hazards in this complex industrial scenario, improving the efficiency and accuracy of safety monitoring and enabling the timely detection of potential safety issues.

[0085] In this embodiment, the specific process of performing frame extraction and feature extraction on video surveillance data, performing time series normalization processing on sensor data, and performing structured representation and encoding on safety specification knowledge in step S1 is as follows:

[0086] S11: acquiring video data by sampling at fixed time intervals and extracting video frames, performing feature extraction on the video frames, and extracting objects and their behavior features in each frame using a target detection algorithm;

[0087] S12: Unify the timestamp format and build a time series prediction model to fill in long-missing data. Recent data is given a higher weight based on the timestamp, and historical data is given an exponentially decaying weight according to preset rules to achieve noise suppression. Timestamp synchronization and spatial coordinate conversion are used to unify and standardize multi-source data. The chemical plant deploys sensors for temperature, pressure, flow, and gas concentration. The reactor temperature sensor has a sampling frequency of 10Hz, the tank pressure sensor has a sampling frequency of 5Hz, and the gas concentration sensor has a sampling frequency of 1Hz. For multi-source sensor data, timestamps are first calibrated through the Industrial IoT gateway, using the IEEE 1588 precision clock synchronization protocol to ensure millisecond-level timestamp accuracy for all sensor data. To fill in long-missing data, a time series prediction model based on a bidirectional LSTM is constructed. Taking reactor temperature data as an example, the model input is the temperature series for the previous 24 hours (time steps n = 1440, dimension d = 1). A two-layer bidirectional LSTM network captures bidirectional dependencies within the time series and automatically focuses on key time features using a self-attention mechanism. For continuous missing data segments exceeding 4 hours, the model prediction value is used to fill in the gaps; for short-term missing data <4 hours, linear interpolation of adjacent moments is used.

[0088] S13: Split the text of safety specification knowledge into atomic knowledge units, bind risk levels and protection requirements based on the hazard source identification results, and output labeled knowledge fragments: a four-tuple of "operation object, behavior constraint, risk type, consequence". Define entities and relationships, add attributes to entities, and build an attribute graph model. Nodes represent entities, edges represent relationships, and both edges and nodes can carry attributes. Through NLP tools or manual annotation, extract entity instances from preprocessed data, identify relationships between entities, handle entity conflicts in data from different sources, store entities, relationships, and attributes in a graph database, establish an efficient index, and support the construction of a safety specification knowledge base.

[0089] Example 2

[0090] Based on Example 1, the specific process of step S2 is as follows:

[0091] S21: Normalize all video frame images to obtain images of specified resolution and standardize their pixel values. Load the preset convolutional neural network model, remove the top classifier and retain the convolution base to extract the specified dimension visual feature vector. The convolutional neural network uses ResNet50, see Figure 4As shown in Figure 1, its structure can be divided into five stages. The input image is first processed by a 7×7 convolutional layer and a 3×3 max pooling layer, resulting in an output size of 112×112. It then enters a backbone stage consisting of four stacked Bottleneck modules, which output feature maps of 256, 512, 1024, and 2048 channels, respectively. The spatial resolution is halved layer by layer until it is finally reduced to 7×7. Within each Bottleneck module, a three-layer convolutional structure (1×1→3×3→1×1) is used to reduce dimensionality before increasing it, reducing computational cost while maintaining expressiveness. Each Bottleneck module contains a main path and a shortcut path. If the output dimensions of the two differ, a 1×1 convolution is used to align the shortcut's dimensions. Finally, the network uses global average pooling to compress the 7×7×2048 feature map into a 2048-dimensional vector.

[0092] S22: Build a two-layer LSTM network model. The input layer accepts n×d dimensional data, where n is the time step and d is the sensor dimension. The last time step output is used as the feature representation to finally obtain the time series features.

[0093] A two-layer LSTM network model was constructed to extract features from time series data from several chemical plant sensors, including reactor temperature and tank pressure. For example, a reactor temperature sensor with a sampling frequency of 10 Hz and a 10-minute time window was used. The model input was n × d-dimensional data, where n = 600 (time steps) and d = 1 (temperature dimension).

[0094] The first LSTM layer has 128 neurons and returns sequence output (return_sequences=True). It uses the ReLU activation function and dropout=0.2 to prevent overfitting. The second LSTM layer has 64 neurons and does not return sequence output (return_sequences=False). The activation function is Tanh. The output layer directly uses the last time step output of the second LSTM layer as the time series feature vector.

[0095] Test results on reactor temperature data show that the features extracted by the model can effectively characterize temperature change trends: under normal operating conditions, the feature vectors form compact clusters in the two-dimensional space after PCA dimensionality reduction. In abnormal temperature rise scenarios, the Euclidean distance between the feature vectors and the normal clusters increases significantly.

[0096] In actual deployment, a sliding window mechanism is used to process real-time data streams, ensuring the continuity and real-time performance of feature extraction. After each new time window is input into the model, a feature vector is output. This vector is then packaged with information such as the sensor ID and timestamp into a feature data packet and sent to the subsequent hypergraph neural network module.

[0097] S23: Add [CLS] and [SEP] tags to the atomized knowledge unit, load the Chinese BERT-base model, and extract the specified dimension vector corresponding to the [CLS] tag as the semantic feature of the safety specification knowledge.

[0098] For atomic knowledge units of chemical plant safety regulations, such as "Class A and B liquid storage tank areas, fire barriers are set up and the effective capacity is ≥ the maximum storage tank capacity, fire and explosion, and fire spread caused by tank leakage", first add the [CLS] and [SEP] tags. Figure 5 , based on the BERT model with a 12-layer Transformer encoder, 768-dimensional hidden state, and 12-head attention, semantic features are extracted. The dimension of each head is 64. The multi-head mechanism can learn different attention subspaces in parallel. Each Transformer layer also includes a feed-forward fully connected network with an intermediate dimension of 3072 and GELU as the activation function.

[0099] Convert the quadruple of atomic knowledge units into a natural language sentence. For example, "The operation object is the Class A and B liquid storage tank area. The behavioral constraint is to set up fire dikes and the effective capacity of the fire dikes must not be less than the capacity of the largest storage tank. The possible risk type is fire and explosion, and the consequence is tank leakage causing fire spread." Add the [CLS] and [SEP] tags at the beginning and end of the sentence respectively to form the input format: "[CLS] The operation object is the Class A and B liquid storage tank area. The behavioral constraint is to set up fire dikes and the effective capacity of the fire dikes must not be less than the capacity of the largest storage tank. The possible risk type is fire and explosion, and the consequence is tank leakage causing fire spread [SEP]."

[0100] Word segmentation: BERT's built-in WordPiece tokenizer is used to segment sentences into subword units, such as: "[CLS] The operation targets Class A and Class B liquid storage tank areas. The behavioral constraint is to establish fire barriers with an effective capacity no less than the capacity of the largest tank. The possible risk type is fire and explosion, and the consequence is a tank leak causing the spread of fire [SEP]." Encoding: The segmented subwords are converted into the input ID, segment ID, and attention mask required by the BERT model. Model inference: The encoded input data is loaded into the Chinese BERT-base model, and forward propagation is performed to obtain the hidden states of each layer. Feature extraction: The vector corresponding to the first-layer [CLS] token is used as the semantic feature of the safety specification knowledge.

[0101] The extracted semantic features are stored in a graph database and indexed with the corresponding knowledge quadruple. When a device's safety specifications are needed, a Cypher statement is used to retrieve the relevant knowledge quadruple and obtain the corresponding semantic feature vector, which is then used for subsequent fusion analysis with video and sensor features. For example, the fusion of the safety specification semantic features of a benzene tank with the tank's appearance features in real-time video and the time series features of the pressure sensor can achieve a comprehensive assessment of the tank's safety status.

[0102] See also Figure 2 ,The hypergraph neural network includes a plurality of nodes and hyperedges, where the multiple nodes include video frames, sensor data points, and security knowledge entities, and the hyperedges are used to represent the complex associations between multimodal data and knowledge;

[0103] The hypergraph neural network extracts features from nodes and hyperedges. The extracted features include attribute features of nodes or attribute features of hyperedges. Classification or clustering is performed based on the node features to find the K nearest neighbor nodes of each node, which helps to understand the local relationship between nodes.

[0104] The features of the nodes of the hypergraph neural network are transmitted through hyperedges to represent the relationship between the nodes; the features of the hyperedges are transmitted through the nodes to represent the relationship between the hyperedges;

[0105] The fully connected layer of the hypergraph neural network integrates the features of nodes and hyperedges, performs linear transformation through the weight matrix, and extracts higher-level features.

[0106] The features of the aggregated nodes and hyperedges of the hypergraph neural network are further extracted through convolution operations to capture the global connection pattern.

[0107] The specific process of hyperedge construction using the preset algorithm is as follows:

[0108] S31: Calculate similarity based on the improved similarity calculation formula. The specific formula is as follows:

[0109] ;

[0110] in, u , v are the feature vectors of the two nodes, ||·||2 is the L2 norm;

[0111] S32: Perform hypergraph convolution operation. The specific formula is as follows:

[0112] ;

[0113] in, H is the hypergraph incidence matrix, , represents the connection relationship between a vertex or node and a hyperedge. If the node vi Belong to the super edge e j ,but H ij =1, otherwise 0; is the vertex degree matrix, the diagonal elements Indicates the i The number of hyperedges connecting vertices, is the hyperedge degree matrix, the diagonal elements Indicates the j The number of nodes connected by a hyperedge; is the weight matrix of the hyperedge, which is generally a diagonal matrix, indicating the importance or trust of each hyperedge; For the l Layer to l+ 1 layer of trainable weight matrix for feature dimension transformation; Represents a nonlinear activation function. Here, the LeakyReLU function is selected, which is defined as follows: α =0.2 controls the negative slope; For the l Node feature representation of the layer.

[0114] Example 3

[0115] On the basis of Example 1 or Example 2, see Figure 3 In step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and the deep fusion of multimodal data and security knowledge is achieved through the hypergraph convolution operation to generate a 512-dimensional unified representation space. The specific process is as follows:

[0116] The input image is first processed by a 7×7 convolution layer and a 3×3 maximum pooling layer, and the output size is 112×112;

[0117] Entering the backbone stage consisting of four stacked Bottleneck modules, the output feature maps are 256, 512, 1024, and 2048 channels respectively. The corresponding spatial resolution is halved layer by layer until it is finally reduced to 7×7.

[0118] In each Bottleneck module, a three-layer convolution structure of "1×1→3×3→1×1" is used to reduce the dimension first and then increase the dimension to reduce the computational cost while maintaining the expressive power;

[0119] Each Bottleneck module has a main path and a shortcut path. If the output dimensions of the two are different, a 1×1 convolution is used to align the shortcut dimensions.

[0120] At the end of the network, the 7×7×2048 feature map is compressed into a 2048-dimensional vector through global average pooling.

[0121] In step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and the deep fusion of multimodal data and security knowledge is achieved through the hypergraph convolution operation. The specific formula for generating a unified representation space is as follows:

[0122] ;

[0123] in, represents the visual feature vector extracted from the video data, h t represents the state feature vector from the time series, e k Represents the structured security knowledge vector extracted through the knowledge graph; W v 、W s 、W k is the corresponding linear mapping weight matrix, which is used to map each modal feature to the same 512-dimensional space; || represents the vector-level splicing operation.

[0124] The safety hazard identification model in step S5 uses a two-layer perceptron, with the input being the fused unified representation space and the output being the probability distribution of safety hazards:

[0125] ;

[0126] in, W 1 is the weight matrix of the first layer of perceptron, which compresses the high-dimensional input to 256 dimensions; b 1 is the first layer bias term; W 2 is the weight matrix of the second layer perceptron, ,in C The total number of categories representing safety hazards; b 2 is the second layer bias term; As the activation function, the softmax function normalizes the output value into a probability distribution.

[0127] A knowledge- and data-driven association modeling system, used to implement a knowledge- and data-driven association modeling method, comprises a multimodal data acquisition and preprocessing unit, a multimodal data feature extraction unit, a hypergraph neural network construction unit, a multimodal data and knowledge fusion unit, and a safety hazard identification and real-time monitoring unit. The multimodal data acquisition and preprocessing unit is used to acquire and preprocess video surveillance data, sensor data, and domain safety specification knowledge. The multimodal data feature extraction unit is used to extract visual features of video surveillance data, time series features of sensor data, and semantic features of safety specification knowledge. The hypergraph neural network construction unit is used to construct a hypergraph neural network to model multimodal data and its high-order associations with safety knowledge. The multimodal data and knowledge fusion unit is used to input multimodal features and safety knowledge features into the hypergraph neural network to generate a unified representation space. The safety hazard identification and real-time monitoring unit is used to construct a safety hazard identification model based on the fused unified representation space to achieve accurate identification and real-time monitoring of safety hazards.

[0128] In summary, the association modeling method and system based on knowledge-data dual drive provided by the present invention include multimodal data acquisition and preprocessing, acquiring video surveillance data, sensor data and domain knowledge, and performing preprocessing; multimodal data feature extraction, extracting visual features of video data, time series features of sensor data and semantic features of safety specification knowledge; constructing a hypergraph neural network, using the hypergraph neural network to model multimodal data and its high-order association with knowledge; fusion of multimodal data and knowledge, inputting multimodal data and knowledge features into the hypergraph neural network, and achieving deep fusion through hypergraph convolution operations; safety hazard identification and real-time monitoring, based on the unified representation space after fusion, constructing a model to achieve accurate identification and real-time monitoring of safety hazards. By deeply fusing multimodal data and knowledge, an efficient association modeling solution is provided, which can effectively improve the accuracy and effect of multi-source information analysis.

[0129] A hypergraph neural network unifies the representation of video, sensor data, and security knowledge, resolving the fragmentation of multi-source data in traditional methods. The fused feature space significantly improves the accuracy of safety hazard identification. Quadruple knowledge representation transforms safety regulations into machine-understandable semantic features, enabling the model's decision-making process to be traceable to regulatory compliance. Improved similarity calculation and hyperedge construction mechanisms, along with the application of functions within the hypergraph convolution formula, address the vanishing gradient problem, accelerate model convergence, and effectively improve the efficiency of modeling complex "person-device-environment" relationships. Hypergraph neural networks effectively handle high-order relationships between multimodal data and safety knowledge. Hypergraph convolution operations integrate multimodal features with safety knowledge features, enabling a deep fusion of multimodal data and safety knowledge, generating a unified representation space. This enhances the model's ability to capture complex relationships and provides more robust model support for subsequent safety hazard identification. Building a safety hazard identification model based on this fused unified representation space enables accurate identification and real-time monitoring of safety hazards in complex scenarios, enabling the model to comprehensively consider various factors for a more comprehensive safety hazard analysis. The two-layer perceptron structure can further process and analyze the fused features, identify safety hazards more accurately, discover potential safety issues in a timely manner, and improve the efficiency and accuracy of safety monitoring.

Claims

1. A knowledge-data dual-driven association modeling method, characterized in that: The following steps are involved: S1: Acquire video surveillance data, sensor data, and domain safety specification knowledge, perform frame extraction and feature extraction on video surveillance data, perform time series normalization on sensor data, and perform structured representation and encoding on safety specification knowledge; S2: Extract visual features of specified dimensions from video surveillance data at specified time intervals, extract time series features of specified dimensions from sensor data to obtain multimodal features, and extract semantic features of specified dimensions from safety specification knowledge to obtain safety knowledge features; S3: Build a hypergraph neural network and use it to model multimodal data and its high-order associations with security knowledge; S4: Inputting the multimodal features and security knowledge features into the hypergraph neural network, achieving deep fusion of multimodal data and security knowledge through hypergraph convolution operations, and generating a unified representation space; S5: Based on the fused unified representation space, a safety hazard identification model is constructed, and safety hazards in complex scenarios are accurately identified and monitored in real time through the safety hazard identification model; The hypergraph neural network includes multiple nodes and hyperedges, wherein the multiple nodes include video frames, sensor data points, and security knowledge entities, and the hyperedges are used to represent the complex associations between multimodal data and knowledge; The hypergraph neural network extracts features from nodes and hyperedges, including attribute features of nodes or attribute features of hyperedges, classifies or clusters nodes based on the features of nodes, and finds K nearest neighbor nodes for each node; The features of the nodes of the hypergraph neural network are transmitted through hyperedges to represent the relationship between the nodes; the features of the hyperedges are transmitted through the nodes to represent the relationship between the hyperedges; The fully connected layer of the hypergraph neural network integrates the features of nodes and hyperedges, performs linear transformation through the weight matrix, and extracts higher-level features; The features of the aggregated nodes and hyperedges of the hypergraph neural network are further extracted through convolution operations to capture the global connection pattern.

2. The method for association modeling based on knowledge-data dual drive according to claim 1, characterized in that: The specific process of performing frame extraction and feature extraction on video surveillance data, normalizing sensor data in time series, and structurally representing and encoding safety specification knowledge in step S1 is as follows: S11: acquiring video data by sampling at fixed time intervals and extracting video frames, performing feature extraction on the video frames, and extracting objects and their behavior features in each frame using a target detection algorithm; S12: Unify the timestamp format and build a time series prediction model to fill in long-term missing data. Give more weight to recent data based on the timestamp, and give historical data an exponential decay weight according to preset rules to achieve noise suppression. Through timestamp synchronization and spatial coordinate conversion, unify the multi-source data benchmark and perform standardization. S13: Split the text of safety specification knowledge into atomic knowledge units, bind risk levels and protection requirements based on hazard source identification results, output quadruple of labeled knowledge fragments, define entities and relationships, add attributes to entities, build attribute graph model, nodes represent entities, edges represent relationships, both edges and nodes can carry attributes, extract entity instances from preprocessed data, identify relationships between entities, handle entity conflicts in data from different sources, store entities, relationships and attributes in a graph database, establish efficient index, and build a safety specification knowledge base.

3. The method for association modeling based on knowledge-data dual drive according to claim 2, characterized in that: The specific process of step S2 is as follows: S21: Normalize all video frame images to obtain images of a specified resolution and standardize their pixel values, load the preset model, remove the top classifier, retain the convolutional base, and extract the visual feature vector of a specified dimension; S22: Construct a two-layer LSTM network model. The input layer accepts n×d dimensional data, where n is the time step and d is the sensor dimension. The last time step output is used as the feature representation to finally obtain the time series features. S23: Add [CLS] and [SEP] tags to the atomized knowledge unit, load the Chinese BERT-base model, and extract the specified dimension vector corresponding to the [CLS] tag as the semantic feature of the safety specification knowledge.

4. The method for association modeling based on knowledge-data dual drive according to claim 1, characterized in that: The specific process of hyperedge construction using the preset algorithm is as follows: S31: Calculate similarity based on the improved similarity calculation formula. The specific formula is as follows: ； in, u , v are the feature vectors of the two nodes, ||·||2 is the L2 norm; S32: Perform hypergraph convolution operation. The specific formula is as follows: ； in, H is the hypergraph association matrix, which represents the connection relationship between vertices or nodes and hyperedges. v i Belong to the super edge e j ,but H ij =1, otherwise 0; D v is the vertex degree matrix, the diagonal elements D v ( i , i ) indicates the i The number of hyperedges connecting vertices, D e is the hyperedge degree matrix, the diagonal elements D e ( j , j ) indicates the j The number of nodes connected by a hyperedge; W e is the weight matrix of the hyperedge, is a diagonal matrix, indicating the importance or trust of each hyperedge; For the l Layer to l +1 trainable weight matrix for feature dimension transformation; represents a nonlinear activation function; H (l) For the l Node feature representation of the layer.

5. The method for association modeling based on knowledge-data dual drive according to claim 1, characterized in that: In step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and the deep fusion of multimodal data and security knowledge is achieved through the hypergraph convolution operation to generate a unified representation space. The specific process is as follows: The input image is first processed by a 7×7 convolution layer and a 3×3 maximum pooling layer, and the output size is 112×112; Entering the backbone stage consisting of four stacked Bottleneck modules, the output feature maps are 256, 512, 1024, and 2048 channels respectively. The corresponding spatial resolution is halved layer by layer until it is finally reduced to 7×7. In each Bottleneck module, a three-layer convolution structure of "1×1→3×3→1×1" is used to reduce the dimension first and then increase the dimension to reduce the computational cost while maintaining the expressive power. Each Bottleneck module has a main path and a shortcut path. If the output dimensions of the two are different, a 1×1 convolution is used to align the shortcut dimensions. Finally, the 7×7×2048 feature map is compressed into a 2048-dimensional vector through global average pooling.

6. The method for association modeling based on knowledge-data dual drive according to claim 5, characterized in that: In step S4, the multimodal features and security knowledge features are input into the hypergraph neural network, and the deep fusion of multimodal data and security knowledge is achieved through the hypergraph convolution operation. The specific formula for generating a unified representation space is as follows: ； in, represents the visual feature vector extracted from the video data, h t represents the state feature vector from the time series, e k Represents the structured security knowledge vector extracted through the knowledge graph; W v 、W s 、W k are the corresponding linear mapping weight matrices, which are used to map the features of each modality to the same 512-dimensional space; || represents the vector-level splicing operation.

7. The method for association modeling based on knowledge-data dual drive according to claim 1, characterized in that: The safety hazard identification model in step S5 uses a two-layer perceptron, with the input being the fused unified representation space and the output being the probability distribution of safety hazards: ； in, W 1 is the weight matrix of the first layer of perceptron, which compresses the high-dimensional input to 256 dimensions; b 1 is the first layer bias term; W 2 is the weight matrix of the second layer perceptron, ,in C The total number of categories representing safety hazards; b 2 is the second layer bias term; As the activation function, the softmax function normalizes the output value into a probability distribution.

8. A knowledge-data dual-driven association modeling system, used to implement the knowledge-data dual-driven association modeling method according to any one of claims 1 to 7, characterized in that: It includes multimodal data acquisition and preprocessing unit, multimodal data feature extraction unit, hypergraph neural network construction unit, multimodal data and knowledge fusion unit, safety hazard identification and real-time monitoring unit; The multimodal data acquisition and preprocessing unit is used to acquire video surveillance data, sensor data, and domain safety specification knowledge, and perform preprocessing; The multimodal data feature extraction unit is used to extract visual features of video surveillance data, time series features of sensor data, and semantic features of safety specification knowledge; The hypergraph neural network construction unit is used to construct a hypergraph neural network to model multimodal data and its high-order association with security knowledge; The multimodal data and knowledge fusion unit is used to input multimodal features and security knowledge features into the hypergraph neural network to generate a unified representation space; The safety hazard identification and real-time monitoring unit is used to construct a safety hazard identification model based on the fused unified representation space to achieve accurate identification and real-time monitoring of safety hazards.

Images