Multi-source multi-modal data fusion and association method based on massive complex spatio-temporal data

Abstract

The invention discloses a multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data, which comprises the following steps of: firstly, acquiring multi-source and multi-modal data in real time through a security element multi-modal data real-time acquisition technology, and then, providing a low-quality data dynamic multi-modal fusion technology. By automatically sensing dynamic changes of different modal data quality and combining an uncertainty sensing model and a dynamic Bayesian network fusion algorithm, the problems of accuracy and stability of multi-modal data fusion in a complex and uncertain environment are solved; and finally, an industrial safety production knowledge base is constructed through an industrial safety production knowledge base modeling technology, and a knowledge representation basis is provided for subsequent functional modules such as knowledge questions and answers, processing suggestions and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of massive spatio-temporal data acquisition and processing, and particularly relates to a multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data. Background Art

[0002] The multi-modal data real-time acquisition technology obtains heterogeneous data from different data sources (such as sensors, cameras, GPS, etc.), and supports real-time processing and decision-making; this kind of data acquisition not only needs to acquire data of different modalities, but also requires data synchronization, fusion and low-latency transmission, so as to ensure that the system can respond to complex environmental changes in a timely manner.

[0003] Currently, technologies such as deep learning and graph neural networks (GNNs) have greatly promoted the development of multi-modal data real-time acquisition. For example, deep neural networks (DNNs) are used to extract features from visual, audio and sensor data, and these heterogeneous data are integrated through fusion algorithms. Time synchronization technologies such as Network Time Protocol (NTP) and Global Positioning System (GPS) clock synchronization ensure that data of different modalities can be accurately aligned, so as to achieve efficient data processing and decision-making. In addition, with the application of 5G and edge computing, the low-latency transmission and real-time processing capabilities of multi-modal data have been further improved.

[0004] In practical applications, autonomous vehicles achieve precise perception of the surrounding environment by real-time collecting data from cameras, lidar, GPS and other sensors, and make timely driving decisions. Similarly, telemedicine uses multi-modal data real-time acquisition technology to combine patients' health data with images, providing doctors with more comprehensive diagnostic information. However, despite significant technological progress, multi-modal data real-time acquisition still faces challenges such as synchronization, data transmission stability and data fusion complexity; future research directions will continue to optimize these problems, and use emerging technologies such as reinforcement learning and transfer learning to further improve the performance and application breadth of multi-modal data real-time acquisition systems.

[0005] Multi-modal data acquisition and processing is a relatively new technical field, where data acquisition methods, the design of each modal architecture, and data fusion methods are very different, and the lack of labeled data in the corresponding fields has also become a major problem in this field. Selecting an optimal technical route, as well as real-time and efficient data acquisition and data fusion have become the most important problems in this field. So far, the general technical drawbacks of multi-modal data acquisition and processing are summarized as follows: 1. Bandwidth and latency problems of data transmission.

[0006] Real-time multimodal data collection relies on high-bandwidth, low-latency transmission technologies such as 5G networks, especially in scenarios involving large-scale, high-frequency data (such as the real-time transmission of high-definition camera and lidar data). However, even in a 5G environment, the transmission network may still be affected by signal interference, bandwidth limitations, or physical obstacles, resulting in reduced transmission speeds or delays. In remote or isolated areas, insufficient network coverage and poor signal stability are particularly prominent, hindering the timely acquisition and processing of multimodal data.

[0007] 2. Complexity and computational load of data fusion.

[0008] With the diversification of multimodal data, data fusion algorithms are becoming increasingly complex. In practical applications, data from different modalities have different physical properties and noise, and accurately and effectively fusing them remains a technical challenge. For example, the correlation between visual data (such as images and videos) and non-visual data (such as sensor, text, or audio data) is complex, requiring advanced technologies such as deep learning or graph neural networks for efficient fusion. This not only increases the computational load but also places higher demands on real-time performance, making it difficult for systems to operate in resource-constrained environments.

[0009] 3. Insufficient robustness and environmental adaptability.

[0010] While multimodal data acquisition performs well in experimental environments, its performance can significantly degrade in complex real-world scenarios, such as extreme weather, low-light conditions, and signal interference. The operating status of hardware devices like sensors and cameras can be affected by the environment, leading to reduced data quality. Data fusion algorithms can also struggle with incomplete or noisy data, impacting the robustness of the entire system. Summary of the Invention

[0011] In view of the above, the present invention provides a multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data. Through efficient data fusion and association technology, it can integrate and mine the value behind these complex data, and provide strong data support for decision support, process optimization, fault prediction, etc. of industrial systems, thereby promoting the in-depth development of industrial intelligence and digital transformation.

[0012] A multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data includes the following steps: (1) Real-time collection and data transmission of multi-source and multi-modal data through real-time collection technology of multi-modal data of security factors; (2) Fusion and correlation of collected data using low-quality data dynamic multimodal fusion technology; (3) Based on the fused data, construct an industrial safety production knowledge base through industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules including knowledge Q&A and disposal suggestions.

[0013] Further, the specific implementation method of the step (1) is as follows: 1.1 Design edge nodes for real-time data acquisition that support multi-modal fusion, and form these edge nodes into a WSN (Wireless Sensor Networks) to collect data from industrial production sites; 1.2 Adopt a data fusion method based on model assistance to compress and remove redundancy from the collected data. By establishing a spatio-temporal correlation model, use model assistance to reduce the node sampling frequency and fuse the data of adjacent nodes, minimizing the amount of data transmitted, reducing power consumption, and extending the service life of the nodes; 1.3 Adopt an adaptive encryption transmission method suitable for low signal-to-noise ratio, that is, according to the changes in the network environment, use adaptive coding technology and packet size for communication, tolerate an appropriate amount of packet transmission errors, and pre-distribute keys in a probabilistic manner in the WSN to ensure that adjacent nodes can share keys with a certain probability, thereby establishing a secure communication link; 1.4 Adopt a data forwarding strategy based on geographical location, that is, adjacent nodes exchange information regularly, use stateless single-hop delay to ensure the real-time performance of end-to-end data transmission; adopt a weighted multi-routing forwarding strategy, that is, when congestion is detected somewhere in the network, automatically switch to an alternative path to reduce the delay caused by re-routing backward; according to the spatial change characteristics of the network environment, make the data routing line avoid areas with high signal-to-noise ratio as much as possible.

[0014] Further, the edge nodes have a variety of data interfaces, supporting the access of multi-modal data acquired by various methods including DCS (Distributed Control System), PLC (Programmable Logic Controller), SCADA (Supervisory Control And Data Acquisition), wireless sensor network, and video acquisition; at the same time, through the modular design of the hardware interface, the edge nodes can be compatible with a variety of communication protocols, realizing the real-time acquisition and local processing of industrial safety production element data, effectively reducing data transmission delay. Different from traditional single-mode data acquisition systems, the edge nodes of the present invention not only have the function of multi-interface fusion, but also have preliminary data fusion processing capabilities, and can preprocess data locally to reduce the subsequent transmission load.

[0015] Furthermore, the real-time multi-modal data acquisition technology for safety elements adopts an anti-interference strategy based on inter-network and inter-cluster for data transmission to avoid interference in industrial safety production monitoring. At the same time, in step 1.2, the existing WSN fusion protocols and algorithms are extended, and a dynamic fusion algorithm based on energy awareness is designed to dynamically adjust the nodes for data fusion and the node caching strategy during network operation to reduce the energy consumption of nodes during data transmission.

[0016] Furthermore, the specific implementation method of step (2) is as follows: 2.1 Use the Dempster-Shafer framework to construct an uncertainty perception model. Regard the data of each node as an evidence body, and each evidence body has its support degree and importance. These two metrics are used to evaluate the uncertainty of the evidence body. Use historical data or prior knowledge to assign weights, that is, importance, to each node, calculate the uncertainty of each evidence body, and then integrate the evidence bodies of nodes in the WSN to obtain the uncertainty of the entire WSN dataset; 2.2 Before data fusion, preprocess the data through a fuzzy logic system. The fuzzy logic system maps the data input value to a fuzzy set through a membership function, and outputs a result with a higher membership degree after applying fuzzy rule reasoning; 2.3 Use DBN (Dynamic Bayesian Networks) for data fusion. DBN captures the state changes of data over time and dynamically adjusts its weight in the fusion according to the uncertainty of different modal data. For data with higher uncertainty, DBN assigns it a lower weight; 2.4 Classify and organize the fused data, conduct theme classification according to five basic elements: event, video, personnel, location, and equipment, and conduct data layering according to ODS (Operational Data Store), DWD (Data Warehouse Detail), DWS (Data Warehouse Service), and ADS (Application Data Service) to provide a structured data framework.

[0017] Furthermore, in step 2.1, the uncertainty of each evidence body and the uncertainty of the entire WSN data are calculated through the following expressions;

[0018]

[0019] Where: U i is the evidence body E i is the uncertainty of E i represents the node iThe evidence body, m i ( E ) is the support degree of the node i The support degree, w i is the importance degree of the node i which reflects the credibility of the evidence body. α i is the weight of the node i in the WSN. n is the number of nodes in the WSN. U total is the uncertainty of the entire WSN dataset.

[0020] Furthermore, the specific implementation method of the step (3) is as follows: 3.1 Based on the semantic Web specifications and standards, widely collect information related to industrial safety production through multimodal channels including domain experts, books, network information, and literature, so as to establish a domain ontology for industrial safety production, and standardize the description of concepts related to the domain and their interrelationships; 3.2 Construct a structured knowledge base model based on the semantic representation of the domain ontology for integrating and managing data from different sources; this knowledge base model uniformly integrates and represents heterogeneous industrial safety production information by dynamically generating semantic templates, converts unstructured information into formal structured knowledge records, and supports efficient storage and read / write operations of large-scale domain knowledge; 3.3 Adopt a knowledge base incremental expansion mechanism, and allow domain experts to update and expand knowledge at any time during use through a multimodal human-computer operation interface; at the same time, design an automated knowledge scraping mechanism to extract new knowledge from the Internet and other sources and integrate it into the knowledge base; 3.4 Design an access and query interface and an inference module for the knowledge base, support users to efficiently access knowledge in the field of industrial safety production through keyword queries, dynamically associate and expand relevant knowledge according to query requests using the inference module through semantic analysis, and provide support for upper-layer intelligent analysis and decision control.

[0021] Furthermore, the content of the domain ontology in the step 3.1 includes: A basic classification system with industry characteristics, constructing a classification framework that conforms to industrial safety production covering infrastructure, equipment, and environmental conditions; Basic concept definitions with industry characteristics, defining key safety production elements including hazard sources, production equipment, and emergency plans; Descriptions of the interrelationships between concepts with industry characteristics, describing the hierarchical structure and association relationships between different concepts including equipment and maintenance requirements, risks and response measures.

[0022] A computer device includes a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the computer program to implement the above-mentioned multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data.

[0023] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data.

[0024] To solve the tasks of collecting and fusing complex spatio-temporal data in an industrial environment, the present invention first performs real-time collection of multi-source and multi-modal data through the real-time multi-modal data collection technology for safety elements, solves the problems of the data transmission rate and data quality degradation of each device in complex or even extreme industrial environments, completes local processing and data fusion operations of the data, and reduces data transmission and latency. Then, through the dynamic multi-modal fusion technology for low-quality data, it fuses and associates the collected data, solves the problem of dependence on high-quality data in multi-modal data fusion, and adopts a fusion strategy that can be dynamically adjusted to adapt to changes in data uncertainty and improve the fusion effect between different modalities. Finally, through the industrial safety production knowledge base modeling technology, it constructs an industrial safety production knowledge base to provide a knowledge representation basis for subsequent functional modules such as knowledge Q&A and disposal suggestions. Based on the above, the present invention has the following beneficial technical effects: 1. The efficiency of real-time multi-modal data collection. Traditional industrial data collection systems can usually only process single-type data and are difficult to handle multiple data sources in complex industrial scenarios. By designing edge collection nodes that support multi-modal data fusion, the present invention can integrate multiple data sources such as DCS, PLC, SCADA, and video monitoring, realize the efficient collection and real-time processing of multi-modal data in industrial safety production, significantly improve the speed and accuracy of data collection, and reduce the delay of data transmission.

[0025] 2. The accuracy and robustness of low-quality data fusion. In the prior art, the processing of low-quality data is often simplified, and the spatio-temporal correlation and dynamic changes of the data cannot be effectively distinguished. By introducing an uncertainty perception model and a dynamic Bayesian network, the present invention dynamically adjusts the weights of different data sources, solves the problem of processing low-quality data in complex industrial scenarios, and not only improves the accuracy of data fusion compared with the prior art, but also enhances the robustness of the system in a multi-interference environment.

[0026] 3. Dynamic Expansion and Intelligent Reasoning of Knowledge Base. Most traditional industrial safety production knowledge bases rely on manual updates, with poor scalability and insufficient performance in the face of multi-modal data and complex reasoning requirements. Through an automated incremental expansion mechanism and semantic reasoning function, the present invention not only realizes the dynamic expansion of the knowledge base but also supports intelligent cross-domain query and reasoning functions, enabling the knowledge base to be continuously updated and optimized, providing strong support for intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the technical framework of the multi-source multi-modal data fusion and association method based on massive complex spatio-temporal data of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] To describe the present invention more specifically, the technical solutions of the present invention will be described in detail below in conjunction with the drawings and specific embodiments.

[0029] To solve the tasks of collecting and fusing complex spatio-temporal data in industrial environments, the present invention aims to solve problems such as the dependence on high-quality data in multi-modal data fusion, the lack of robustness and environmental adaptability in data collection, and the bandwidth and latency in data transmission. Specifically, it solves the problem of how to more efficiently fuse the differences between different modalities of multi-modal data (including low-quality data) in industrial scenarios, and how to improve the real-time performance of data transmission and enhance the data quality of each data source (such as sensors, cameras, GPS, etc.) in extreme environments. Therefore, the present invention aims to provide a strong data foundation for fields such as intelligent decision-making support and large-scale data analysis by constructing a multi-modal data fusion framework to integrate massive spatio-temporal data from different sources (including images, texts, videos, sensor data, etc.).

[0030] The present invention first collects multi-source multi-modal data in real time through the real-time collection technology of multi-modal data of safety elements, then fuses and associates the collected data through the dynamic multi-modal fusion technology of low-quality data, and finally constructs an industrial safety production knowledge base through the industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules such as knowledge Q&A and disposal suggestions. The specific technical implementation is as Figure 1 shown.

[0031] (1) Real-time Collection Technology of Multi-modal Data of Safety Elements This technical point aims to solve the complexity, real-time performance, and reliability problems of collecting and fusing various data types related to industrial safety production in complex or even extreme environments. By designing edge collection nodes that support multi-modal data fusion and combining key technologies of WSN, efficient collection, processing, and transmission of diverse safety production data are achieved.

[0032] 1.1 Research and Design of Edge Nodes for Real-time Acquisition of Data Supporting Multimodal Fusion The elements affecting the safe production of industrial enterprises are very complex and diverse. Therefore, there is a wide variety of data on safe production elements that need to be collected. Different data requires different real-time acquisition methods, including the DCS / PLC / SCADA acquisition method built into the system, the wireless sensor network acquisition method, and video acquisition methods. For this reason, it is necessary to research and design an edge node for real-time acquisition of data supporting multimodal fusion to be compatible with various data input interfaces and formats, and have a certain computing power to complete local data processing and data fusion operations, reducing data transmission and latency.

[0033] Different from traditional single-mode data acquisition systems, the edge nodes in the present invention not only have the function of multi-interface fusion, but also have preliminary data fusion processing capabilities, and can preprocess data locally to reduce the subsequent transmission load.

[0034] Wireless sensor networks are formed by sensor nodes to collect data in industrial production sites. The key technologies involved include data acquisition, encrypted transmission, fusion, etc. In particular, the external environment of industrial production sites is relatively harsh, complex, and full of uncertainties, which puts higher requirements on the deployment and operation of wireless sensor network systems.

[0035] 1.2 Model-assisted Low-power Data Collection Method for Sensor Networks with Data Fusion Function In view of the characteristics of time correlation, space correlation, and spatio-temporal correlation of industrial enterprise safety production data (such as equipment parameters such as pressure, liquid level, and flow rate in the tank area and environmental parameters such as temperature, wind force, smoke, and concentration of toxic gases), it is necessary to research a model-assisted data collection method for sensor networks with data fusion function to minimize the amount of network data transmission and reduce the power consumption of network data transmission. Key research includes: ① The time-domain change model of industrial enterprise safety production data and its data collection method based on this model; ② The space-domain change model of industrial enterprise safety production data and its data collection method based on this model; ③ The data collection method based on the spatio-temporal correlation of industrial enterprise safety production data.

[0036] The present invention designs a model-assisted data fusion method to compress and remove redundancy from data. By establishing change models in the time domain and space domain, it reduces the sensor sampling frequency and fuses data from adjacent nodes, minimizing the amount of data transmitted to the greatest extent. By reducing the amount of data transmitted in the sensor network, power consumption is reduced, and the service life of wireless sensor network nodes is extended, especially suitable for energy-sensitive industrial environments. In existing sensor network technologies such as LEACH (Low Energy Adaptive Clustering Hierarchy) technology, it mainly relies on the simple fusion of data by cluster heads, lacks in-depth analysis of the spatio-temporal correlation of data, and fails to minimize the amount of data transmission to the greatest extent. The present invention reduces data transmission by introducing a spatio-temporal correlation model, greatly improving the energy efficiency of the network in data compression and fusion.

[0037] 1.3 Data Encryption Transmission Technology under Low Signal-to-Noise Ratio Industrial production environments often have characteristics such as complexity, diversity, drastic changes, and lack of safety. The existence of dust and extreme working environments, drastic temperature changes, and complex electromagnetic environments pose very severe challenges to the reliability of WSN data communication, reducing the signal-to-noise ratio during WSN data transmission and increasing the probability of unreliable data transmission. To address the above problems, the following are proposed for research: ① Coding technology considering network error correction to tolerate an appropriate amount of packet transmission errors; ② Research on packet splitting technology for environmental perception, adaptively adopting appropriate coding technologies and packet sizes for communication according to changes in the network environment to improve the error correction ability; ③ Intelligent routing technology for network environment perception, making the data routing line most likely bypass areas with high signal-to-noise ratio according to the spatial change characteristics of the network environment; ④ Random key distribution technology, pre-distributing keys in a probabilistic manner in the wireless sensor network to ensure that adjacent nodes can share keys with a certain probability, thereby establishing a secure communication link.

[0038] The present invention designs an adaptive encryption transmission technology suitable for low signal-to-noise ratio, ensuring the reliability and security of data transmission in the case of low signal-to-noise ratio through an adaptive error correction coding and packet splitting mechanism. This technology combines the strategies of network error correction coding and adaptive adjustment of packet size, and has stronger anti-interference ability and enhanced data security in complex environments compared with traditional wireless transmission methods.

[0039] 1.4 Routing Technology for Providing Real-Time Data Transmission Complex environments also pose requirements for the real-time and effectiveness of WSN in transmitting monitoring data. How to effectively transmit and process the original monitored data is the key research and problem to be solved in this invention. To this end, it is planned to combine the environmental characteristics of the monitoring area to achieve the real-time transmission of monitoring data, and focus on researching a real-time routing protocol. This routing protocol includes: ① Using location-based data forwarding, neighboring sensor nodes exchange information regularly, and using a stateless single-hop delay guarantee forwarding strategy to ensure the real-time guarantee of end-to-end data transmission; ② When congestion is detected somewhere in the network, through a weighted multi-routing forwarding strategy, reduce the delay caused by backtracking and re-routing to ensure that the end-to-end data transmission has a definite upper limit of delay. Ensure the real-time transmission of data in complex industrial environments, especially in scenarios where the environment changes rapidly, and can quickly adjust the transmission path to reduce delays.

[0040] Different from traditional routing, this invention combines geographical location and multi-path weighting strategies to ensure the real-time and stability of data transmission, especially suitable for the transmission of critical safety monitoring data in industrial production.

[0041] 1.5 Anti-interference strategies for wireless sensor networks Since WSN operates on an open wireless communication channel, communication interference is relatively serious, affecting data transmission between nodes. In complex environments, multiple monitoring networks with the same or different channels may coexist, and WSN nodes in the monitoring network are usually relatively dense. Therefore, WSN operating in complex environments will inevitably be interfered by multiple interference sources, thus affecting the normal monitoring of production safety data. Aiming at the possible interference in WSN for production safety monitoring, research an anti-interference strategy for WSN based on inter-network and inter-cluster to ensure the reliable transmission of data in WSN. Focus on researching: ① The inter-network interference problem of WSN; ② The inter-cluster interference problem of WSN; ③ The MAC access method applicable to three-dimensional WSN.

[0042] 1.6 Energy-aware sensing data fusion method Since the nodes in the WSN have limited energy, reducing the amount of data transmitted by the nodes through data fusion and thus reducing the total energy consumption in the network is an effective method. The present invention extends the existing wireless sensor data fusion protocols and algorithms, researches and implements an energy-efficient data fusion method for WSN, and focuses on the following aspects: ① Dynamically generating and selecting fusion nodes; ② The caching strategy of WSN nodes; ③ Heterogeneous sensing data fusion of sink nodes. A dynamic fusion algorithm based on energy awareness is designed to dynamically adjust the data fusion nodes and caching strategy during the operation of the network, so as to reduce the energy consumption of data transmission, ensure the reliability and energy utilization efficiency of WSN data transmission in a multi-interference environment, and extend the network lifetime. The data fusion strategy based on energy awareness in the present invention not only reduces the repeated transmission of data, but also optimizes the energy use of the network by adjusting the positions of the fusion nodes. Compared with the traditional WSN data fusion scheme, it has better energy utilization effect and anti-interference ability.

[0043] The above-mentioned real-time multi-modal data acquisition technology for security elements theoretically supports all mainstream acquisition methods on the market and is expected to support no less than 4,000 Internet of Things sensing protocols.

[0044] (2) Dynamic multi-modal fusion technology for low-quality data The motivation for multi-modal fusion is to jointly utilize the effective information from different modalities to improve the accuracy and stability of downstream tasks. Traditional multi-modal fusion methods often rely on high-quality data. However, in real-world scenarios, due to the complex diversity of application environments, the quality of modalities has dynamic variation characteristics in different samples, different times and spaces, and the appearance of low-quality modal data is often difficult to predict in advance. Therefore, the present invention conducts research on complex low-quality multi-modal data in real-world applications to form a dynamic multi-modal fusion technology for low-quality data.

[0045] Dynamic multimodal data refers to the quality of modalities that changes dynamically with different input samples and scenarios. For example, in the scenario of industrial safety production, the road system obtains road surface and target information through RGB (Red, Green, Blue color model) and infrared sensors. In well-lit conditions, the RGB camera can better support the decision-making of intelligent systems because it can capture rich texture and color information of the target. However, at night with insufficient lighting, the perception information provided by infrared sensors is more reliable. How to enable the model to automatically perceive the changes in the quality of different modalities and thus perform precise and stable fusion is the core task of dynamic multimodal fusion methods. Dynamic multimodal fusion methods can process data from different sources and of different types and integrate them to provide more comprehensive and accurate information. In the scenario of industrial safety production, due to the large variety of data types involved and the characteristics and requirements of these data may change over time, a dynamic approach is needed for data fusion. In addition, considering the possible uncertainties in the data fusion process, adopting a method that can dynamically adjust the fusion strategy to adapt to the changes in data uncertainty is the key to improving the fusion effect.

[0046] Specifically, the present invention first uses the Dempster-Shafer framework to construct an uncertainty perception model, regarding each sensor report as an evidence body. Under this framework, each evidence body has its degree of support and importance, and these two metrics can be used to evaluate the uncertainty of the evidence body. By analyzing the data set, an overall assessment of the uncertainty of the entire data set can be obtained. Considering the possible uncertainties in the data fusion process, the present invention adopts a fusion algorithm based on DBN, which can dynamically adjust the fusion strategy according to the uncertainty of the current data, thereby improving the accuracy and reliability of the fusion result. That is, first use fuzzy logic to preprocess the underlying data, and then use a dynamic Bayesian network to fuse these preprocessed data. In this way, the uncertainties in the data can be effectively processed, and at the same time, the data information from different sources can be fully utilized. Finally, the processed data is stratified according to the five basic element theme libraries of events, videos, personnel, locations, and devices.

[0047] 2.1 Construction of the Uncertainty Perception Model Use the Dempster-Shafer evidence theory to construct an uncertainty perception model. This framework allows us to analyze the data provided by different sensors as evidence bodies. Each evidence body has a degree of support and importance to evaluate its uncertainty.

[0048] Let E i denote the evidence body of sensor i , m i (E ) is its support and importance w i reflects the credibility of the evidence body. For each sensor, its uncertainty U i , is calculated by the following formula:

[0049] The Dempster-Shafer evidence theory can effectively handle the uncertainty in multi-modal data and enhance the system's perception ability of low-quality data. Different from the traditional data fusion method based on fixed quality assumptions, the present invention enables the system to dynamically perceive and adjust the weights of low-quality data through an uncertainty perception framework.

[0050] 2.2 Overall evaluation of data uncertainty For the uncertainty of the entire data set, it is calculated by integrating the evidence bodies of all sensors:

[0051] Where: α i is the weight of sensor i , n is the number of nodes in the WSN.

[0052] Based on the weights of each sensor, the overall uncertainty evaluation of the data set is obtained by combining all evidence bodies, providing a basis for subsequent data fusion and ensuring that low-quality data is not overly relied on in the fusion decision-making; by combining the overall evaluation of the uncertainty of multiple modal data, the present invention can achieve dynamic regulation of the fusion accuracy of the entire system.

[0053] 2.3 Fuzzy logic preprocessing Before data fusion, fuzzy logic is used to preprocess the underlying data to handle the uncertainty in the data. A fuzzy logic system usually maps the input value to a fuzzy set through a membership function and then applies fuzzy rules for reasoning to output a result with a higher membership degree.

[0054] For a certain input value x , its membership function μ ( x ) is defined as:

[0055] Where: c is the central value, σ is the expansion parameter used to control the range of fuzziness.

[0056] The present invention preprocesses modal data of different qualities, reduces data noise and errors, improves the fusion effect, and can flexibly cope with data quality changes in different environments by dynamically adjusting data input through fuzzy logic.

[0057] 2.4 Dynamic Bayesian Network Fusion Algorithm The dynamic Bayesian network is used to dynamically adjust the fusion strategy according to uncertainty. The DBN can capture changes in data states in time series and update the model through Bayesian inference. During the fusion process, modal data with higher uncertainty will be assigned lower weights, thereby reducing its impact on the final decision and ensuring the accuracy and robustness of the fusion result.

[0058] The present invention dynamically adjusts the data fusion strategy according to changes in modal data quality to achieve adaptive fusion in different environments. The introduction of the dynamic Bayesian network enables the system to automatically adjust the fusion strategy in a time-series environment, significantly improving the stability and accuracy of multi-modal data fusion.

[0059] 2.5 Data Stratification and Thematic Library Construction.

[0060] After completing data fusion, the processed data is stratified into a thematic library according to five basic elements such as events, videos, personnel, locations, and devices. This step involves classifying and organizing the fused data, providing a structured data framework, which is helpful for subsequent downstream tasks (such as data analysis, event prediction), and supports various intelligent applications in industrial production. The construction of data stratification and thematic library in the present invention can effectively support multi-dimensional data management in complex industrial scenarios, improving data utilization efficiency and operability.

[0061] (3)Industrial Safety Production Knowledge Base Modeling Technology The present invention aims to establish a safety production knowledge base for complex industrial sites, model and develop the knowledge base based on ontology to support intelligent decision-making and knowledge expression. According to the knowledge in the industrial production field, fully considering the production characteristics of complex industrial sites, especially chemical enterprises, a domain ontology for safety production in complex industrial sites is established. Based on the industrial safety production ontology, a large-scale safety production knowledge base for complex industrial sites is realized. Through cross-domain data and view fusion technology, the knowledge base integrates multiple modal data sources (such as text, pictures, expert experience, etc.), contains various basic knowledge required for safety production in complex industrial sites and safety production models. Finally, through an automated incremental expansion mechanism and query inference function, comprehensive knowledge representation and management of safety production in complex industrial sites are realized, providing a knowledge representation basis for upper-layer intelligent decision-making support. The specific research content is described in detail as follows: 3.1 Ontology Modeling of Industrial Site Safety Production with Cross-Domain Data and Video Fusion Supporting Multi-Modal Fusion Based on general Semantic Web specifications and standards, information related to the safe production of industrial enterprises in a specific industry is widely collected through various channels (domain experts, books, online information, literature, etc.). A relatively complete domain ontology for industrial enterprises is established through a multimodal approach to standardize the description of domain concepts and their interrelationships. The domain ontology mainly includes the following aspects: ① A basic classification system for the safe production of industrial enterprises with industry characteristics, constructing a classification framework that conforms to the safe production of industrial enterprises, covering infrastructure, equipment, environmental conditions, etc.; ② The definition and description of basic concepts related to the safe production of industrial enterprises with industry characteristics, defining key safe production elements, such as hazard sources, production equipment, emergency plans, etc.; ③ The definition and description of the interrelationships between concepts, describing the hierarchical structure and association relationships between different concepts, such as the association between equipment and its maintenance requirements, and between risks and countermeasures.

[0062] Through the structured modeling of the safe production in complex industrial sites, the present invention establishes a comprehensive domain ontology, providing a semantic basis for the development of knowledge bases and knowledge reasoning.

[0063] 3.2 Structured Knowledge Base Model Based on Ontology Based on the semantic representation of the domain ontology, a structured knowledge base model is constructed to integrate and manage information from different sources; by dynamically generating semantic templates, heterogeneous industrial enterprise safe production information (expert experience, literature, pictures, industry data, safe production models, etc.) is uniformly integrated and represented, converting unstructured or natural language-described information (such as text, pictures, etc.) into formal and structured knowledge records. At the same time, a specific knowledge storage model is studied and designed to support the efficient storage of large-scale industry and domain knowledge, as well as the efficient reading and writing operations of knowledge records.

[0064] The structured knowledge base model of the present invention makes it possible to integrate and manage heterogeneous data, ensuring the systematic and standardized representation of knowledge related to industrial safety production; based on the unified knowledge integration of semantic representation, it breaks through the bottleneck that traditional knowledge bases are difficult to process heterogeneous information, ensuring knowledge consistency.

[0065] 3.3 Knowledge Base Incremental Expansion Mechanism Supporting Multimodal Fusion In addition to centrally defining and collecting knowledge about the safe production of industrial enterprises, which is the core content of the knowledge base for the safe production of industrial enterprises, new domain knowledge also needs to be continuously added and expanded during the use of the knowledge base. For this purpose, it is necessary to study the incremental expansion mechanism of the knowledge base, provide a simple and effective multimodal human-machine operation interface, allowing domain experts to dynamically enter knowledge in a multimodal manner, and automatically grab or extract knowledge from various media (such as the Internet).

[0066] The dynamic expansion and update mechanism of the knowledge base ensures its persistence and adaptability in applications and can track the latest developments in the field of industrial enterprise work safety; the present invention designs an expansion mechanism that combines multi-modal input and automated knowledge extraction, reduces the complexity of manual updates, and improves the dynamic response ability of the knowledge base.

[0067] 3.4 Query and Inference of the Knowledge Base Research and implement access and query interfaces for the work safety knowledge base of industrial sites for cross-domain digital-visual fusion, support keyword-based knowledge queries and searches, and at the same time implement basic knowledge inference to support reasoning about the most basic semantic relationships, dynamically associate and expand knowledge query requests; on the public platform for external display, the query and inference mechanism of the knowledge base can be used to achieve semantic-based intelligent analysis and control. The present invention provides intelligent query and inference interfaces, supports cross-domain digital-visual fusion industrial work safety applications, and enhances the practical value of the knowledge base; through the semantic inference mechanism, the knowledge base not only supports static queries, but also can dynamically expand query results, enhancing the intelligent level and response ability of the system.

[0068] The technical solution of the present invention is widely applied to various data types including ground sensor data, video surveillance streams, Internet of Things device data, etc. These data not only have a wide range of sources and diverse forms, covering various modalities such as text, images, videos, and sounds, but also are applied to multiple core application scenarios such as smart cities, intelligent transportation, environmental monitoring, industrial automation, and other fields. Through efficient data fusion and association technologies, the value behind these complex data can be integrated and mined, providing strong data support for decision-making support, process optimization, and fault prediction of industrial systems, thus promoting the in-depth development of industrial intelligent and digital transformation.

[0069] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and apply the present invention. Those skilled in the art can obviously make various modifications to the above embodiments easily and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data, characterized in that, It includes the following steps: (1) Real-time collect and transmit multi-source and multi-modal data through the real-time collection technology of safety factor multi-modal data; (2) Fuse and correlate the collected data through the dynamic multi-modal fusion technology of low-quality data; (3) Based on the fused data, construct an industrial safety production knowledge base through the industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules including knowledge Q&A and disposal suggestions.

2. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 1, characterized in that: The specific implementation method of step (1) is as follows: 1.1 Design edge nodes for real-time data collection that support multi-modal fusion, and form a WSN with these edge nodes to collect data from industrial production sites; 1.2 Adopt a data fusion method based on model assistance to compress and remove redundancy from the collected data. By establishing a spatio-temporal correlation model, use model assistance to reduce the node sampling frequency and fuse the data of adjacent nodes, minimizing the amount of data transmitted, reducing power consumption, and extending the service life of the nodes; 1.3 Adopt an adaptive encryption transmission method suitable for low signal-to-noise ratio, that is, according to the changes in the network environment, use adaptive coding technology and packet size for communication, tolerate a certain amount of packet transmission errors, and pre-distribute keys in a probabilistic manner in the WSN to ensure that adjacent nodes can share keys with a certain probability, thereby establishing a secure communication link; 1.4 Adopt a data forwarding strategy based on geographical location, that is, adjacent nodes exchange information regularly, and use stateless single-hop delay to ensure the real-time performance of end-to-end data transmission; adopt a weighted multi-routing forwarding strategy, that is, when congestion is detected somewhere in the network, automatically switch to an alternative path to reduce the delay caused by backtracking and re-routing; according to the spatial change characteristics of the network environment, make the data routing line avoid areas with high signal-to-noise ratio as much as possible.

3. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 2, characterized in that: The edge nodes have a variety of data interfaces, supporting the access of multi-modal data collected in various ways including DCS, PLC, SCADA, wireless sensor network, and video collection; at the same time, through the modular design of the hardware interface, the edge nodes can be compatible with a variety of communication protocols, realizing the real-time collection and local processing of industrial safety production factor data, and effectively reducing data transmission delay.

4. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 2, wherein: The real-time collection technology of safety factor multi-modal data adopts an anti-interference strategy based on inter-network and inter-cluster for data transmission to avoid interference in industrial safety production monitoring. At the same time, in step 1.2, the existing WSN fusion protocols and algorithms are extended, and a dynamic fusion algorithm based on energy awareness is designed to dynamically adjust the nodes for data fusion and the node cache strategy during the network operation process to reduce the node energy consumption during data transmission.

5. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 1, characterized in that: The specific implementation method of step (2) is as follows: 2.1 Construct an uncertainty-aware model using the Dempster-Shafer framework. Consider the data of each node as an evidence body, and each evidence body has its support degree and importance degree. These two metrics are used to evaluate the uncertainty of the evidence body. Use historical data or prior knowledge to assign weights (i.e., importance degrees) to each node, calculate the uncertainty of each evidence body, and then integrate the evidence bodies of the nodes in the WSN to obtain the uncertainty of the entire WSN dataset. 2.2 Before data fusion, preprocess the data through a fuzzy logic system. The fuzzy logic system maps the data input value to a fuzzy set through a membership function, and outputs a result with a higher membership degree after applying fuzzy rule reasoning. 2.3 Use DBN for data fusion. DBN captures the state changes of data over time and dynamically adjusts its weight in the fusion according to the uncertainty of different modal data. For data with higher uncertainty, DBN assigns it a lower weight. 2.4 Classify and organize the fused data. Conduct topic classification according to five basic elements: event, video, person, location, and device, and conduct data layering according to ODS, DWD, DWS, and ADS, so as to provide a structured data framework.

6. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 5, characterized in that: In step 2.1, calculate the uncertainty of each evidence body and the uncertainty of the entire WSN data through the following expression; ; ; Wherein: U i is the evidence body E i of uncertainty E i represents the evidence body of the node i of m i ( E ) is the support degree of the node i of w i is the importance degree of the node i which reflects the credibility of the evidence body α i is the weight of the node i in the WSN n is the number of nodes in the WSN U total is the uncertainty of the entire WSN dataset 7. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 1, characterized in that: The specific implementation method of step (3) is as follows: 3.1 Based on the Semantic Web specifications and standards, widely collect information related to industrial safety production through multimodal channels including domain experts, books, network information, and literature, so as to establish a domain ontology for industrial safety production, and standardize the description of domain-related concepts and their interrelationships. 3.2 Construct a structured knowledge base model based on the semantic representation of the domain ontology for integrating and managing data from different sources; this knowledge base model uniformly integrates and represents heterogeneous industrial safety production information by dynamically generating semantic templates, converts unstructured information into formal structured knowledge records, and supports efficient storage, reading, and writing operations of large-scale domain knowledge. 3.3 Adopt a knowledge base incremental expansion mechanism, and allow domain experts to update and expand knowledge at any time during use through a multimodal human-computer operation interface; at the same time, design an automated knowledge extraction mechanism to extract new knowledge from the Internet and other sources and integrate it into the knowledge base. 3.4 Design the access and query interfaces and inference modules of the knowledge base, support users to efficiently access the knowledge in the industrial safety production field through keyword queries, and use the inference module to dynamically associate and expand relevant knowledge through semantic analysis according to the query request, providing support for upper-layer intelligent analysis and decision control.

8. The multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data according to claim 7, wherein: The content of the domain ontology in step 3.1 includes: A basic classification system with industry characteristics, constructing a classification framework that covers infrastructure, equipment, and environmental conditions and conforms to industrial safety production; Basic concept definitions with industry characteristics, defining key safety production elements including hazard sources, production equipment, and emergency plans. Description of the interrelationships between concepts with industry characteristics, including the hierarchical structure and association relationships between different concepts such as equipment and maintenance requirements, risks and countermeasures.

9. A computer device, including a memory and a processor, wherein a computer program is stored in the memory, characterized in that: The processor is used to execute the computer program to implement the multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the multi-source and multi-modal data fusion and association method based on massive complex spatio-temporal data as described in any one of claims 1 to 8.

Images