Multi-source and multi-modal data fusion and association method based on massive complex spatiotemporal data

Abstract

The present invention discloses a multi-source multi-modal data fusion and association method based on massive complex spatiotemporal data. The method first collects multi-source multi-modal data in real time through real-time collection technology of multi-modal data of safety factors, and then proposes a dynamic multi-modal fusion technology for low-quality data. By automatically sensing the dynamic changes in the quality of data of different modalities, combined with the uncertainty perception model and dynamic Bayesian network fusion algorithm, the accuracy and stability problems of multi-modal data fusion in complex and uncertain environments are solved; finally, an industrial safety production knowledge base is constructed through industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules such as knowledge question and answer and disposal suggestions.

Description

Technical Field

[0001] The present invention belongs to the technical field of massive spatiotemporal data acquisition and processing, and specifically relates to a multi-source and multi-modal data fusion and association method based on massive complex spatiotemporal data. Background Art

[0002] Multimodal data real-time acquisition technology supports real-time processing and decision-making by acquiring heterogeneous data from different data sources (such as sensors, cameras, GPS, etc.); this data acquisition not only requires the collection of data in different modalities, but also requires data synchronization, fusion and low-latency transmission to ensure that the system can respond to complex environmental changes in a timely manner.

[0003] Currently, technologies such as deep learning and graph neural networks (GNNs) have greatly promoted the development of real-time multimodal data acquisition. For example, deep neural networks (DNNs) are used to extract features from visual, audio, and sensor data and integrate these heterogeneous data through fusion algorithms. Time synchronization technologies such as Network Time Protocol (NTP) and Global Positioning System (GPS) clock synchronization ensure that data from different modalities can be precisely aligned, enabling efficient data processing and decision-making. In addition, with the application of 5G and edge computing, the low-latency transmission and real-time processing capabilities of multimodal data have been further improved.

[0004] In practical applications, autonomous vehicles achieve accurate perception of their surroundings and make timely driving decisions by collecting real-time data from cameras, lidar, GPS, and other sensors. Similarly, telemedicine utilizes multimodal data collection technology to combine patient health data with images, providing doctors with more comprehensive diagnostic information. However, despite significant technological advances, multimodal data collection still faces challenges such as synchronization, data transmission stability, and data fusion complexity. Future research will continue to optimize these issues and leverage emerging technologies such as reinforcement learning and transfer learning to further enhance the performance and application breadth of multimodal data collection systems.

[0005] Multimodal data acquisition and processing is a relatively new field, characterized by significant variations in data acquisition methods, the design of modal architectures, and data fusion approaches. Insufficient labeled data in the corresponding domains has become a major challenge. Choosing the optimal technical approach and ensuring real-time and efficient data acquisition and fusion are key challenges in this field. To date, the following are some common technical shortcomings of multimodal data acquisition and processing:

[0006] 1. Bandwidth and latency issues in data transmission.

[0007] Real-time multimodal data collection relies on high-bandwidth, low-latency transmission technologies such as 5G networks, especially in scenarios involving large-scale, high-frequency data (such as the real-time transmission of high-definition camera and lidar data). However, even in a 5G environment, the transmission network may still be affected by signal interference, bandwidth limitations, or physical obstacles, resulting in reduced transmission speeds or delays. In remote or isolated areas, insufficient network coverage and poor signal stability are particularly prominent, hindering the timely acquisition and processing of multimodal data.

[0008] 2. Complexity and computational load of data fusion.

[0009] With the diversification of multimodal data, data fusion algorithms are becoming increasingly complex. In practical applications, data from different modalities have different physical properties and noise, and accurately and effectively fusing them remains a technical challenge. For example, the correlation between visual data (such as images and videos) and non-visual data (such as sensor, text, or audio data) is complex, requiring advanced technologies such as deep learning or graph neural networks for efficient fusion. This not only increases the computational load but also places higher demands on real-time performance, making it difficult for systems to operate in resource-constrained environments.

[0010] 3. Insufficient robustness and environmental adaptability.

[0011] While multimodal data acquisition performs well in experimental environments, its performance can significantly degrade in complex real-world scenarios, such as extreme weather, low-light conditions, and signal interference. The operating status of hardware devices like sensors and cameras can be affected by the environment, leading to reduced data quality. Data fusion algorithms can also struggle with incomplete or noisy data, impacting the robustness of the entire system. Summary of the Invention

[0012] In view of the above, the present invention provides a multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data. Through efficient data fusion and association technology, it can integrate and mine the value behind these complex data, and provide strong data support for decision support, process optimization, fault prediction, etc. of industrial systems, thereby promoting the in-depth development of industrial intelligence and digital transformation.

[0013] A multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data includes the following steps:

[0014] (1) Real-time collection and data transmission of multi-source and multi-modal data through real-time collection technology of multi-modal data of security factors;

[0015] (2) Fusion and correlation of collected data using low-quality data dynamic multimodal fusion technology;

[0016] (3) Based on the fused data, an industrial safety production knowledge base is constructed through industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules including knowledge question and answer and disposal suggestions.

[0017] Furthermore, the specific implementation of step (1) is as follows:

[0018] 1.1 Design edge nodes that support real-time data collection with multimodal fusion, and form WSNs (Wireless Sensor Networks) with these edge nodes to collect data from industrial production sites;

[0019] 1.2 Adopt a model-assisted data fusion method to compress and remove redundancy from the collected data. By establishing a spatiotemporal correlation model, the model is used to assist in reducing the node sampling frequency and fusing the data of adjacent nodes, thereby minimizing the amount of transmitted data, reducing power consumption, and extending the service life of the node.

[0020] 1.3 Adopt an adaptive encryption transmission method that adapts to low signal-to-noise ratios. That is, according to changes in the network environment, adaptive coding technology and data packet size are used for communication, and a moderate amount of data packet transmission errors are tolerated. In the WSN, keys are pre-distributed in a probabilistic manner to ensure that adjacent nodes can share keys with a certain probability, thereby establishing a secure communication link.

[0021] 1.4 Adopt a data forwarding strategy based on geographic location, that is, adjacent nodes exchange information regularly and use stateless single-hop delay to ensure the real-time performance of end-to-end data transmission; adopt a weighted multi-route forwarding strategy, that is, when congestion is detected somewhere in the network, automatically switch to an alternative path to reduce the delay caused by fallback rerouting; based on the spatial variation characteristics of the network environment, make the data routing lines bypass areas with high signal-to-noise ratio as much as possible.

[0022] Furthermore, the edge node has multiple data interfaces, supporting multimodal data access collected and acquired in various ways, including DCS (Distributed Control System), PLC (Programmable Logic Controller), SCADA (Supervisory Control And Data Acquisition), wireless sensor networks, and video acquisition; at the same time, through the modular design of the hardware interface, the edge node is compatible with multiple communication protocols, realizing real-time collection and local processing of industrial safety production factor data, and effectively reducing data transmission delays. Different from the traditional single-mode data acquisition system, the edge node of the present invention not only has the function of multi-interface fusion, but also has preliminary data fusion processing capabilities, which can pre-process the data locally and reduce the subsequent transmission load.

[0023] Furthermore, the real-time acquisition technology of multimodal data of safety factors adopts an anti-interference strategy based on inter-network and inter-cluster for data transmission to avoid interference in industrial production safety monitoring. At the same time, in step 1.2, the existing WSN fusion protocol and algorithm are expanded, and a dynamic fusion algorithm based on energy perception is designed to dynamically adjust the data fusion nodes and node caching strategies during network operation to reduce node energy consumption during data transmission.

[0024] Furthermore, the specific implementation of step (2) is as follows:

[0025] 2.1 An uncertainty perception model is constructed using the Dempster-Shafer framework. The data of each node is regarded as an evidence body. Each evidence body has its support and importance. These two metrics are used to evaluate the uncertainty of the evidence body. Historical data or prior knowledge are used to assign a weight, i.e., importance, to each node. The uncertainty of each evidence body is calculated, and then the evidence bodies of the nodes in the WSN are integrated to obtain the uncertainty of the entire WSN dataset.

[0026] 2.2 Before data fusion, the data is pre-processed by the fuzzy logic system. The fuzzy logic system maps the data input value to the fuzzy set through the membership function, and outputs the result with higher membership after applying fuzzy rule reasoning;

[0027] 2.3 Use DBN (Dynamic Bayesian Networks) for data fusion. DBN captures the state changes of data over time and dynamically adjusts the weight of different modal data in the fusion according to their uncertainty. For data with high uncertainty, DBN assigns it a lower weight.

[0028] 2.4 Classify and organize the integrated data, classify the data by the five basic elements of events, videos, people, locations, and equipment, and layer the data by ODS (operational data storage layer), DWD (data warehouse detail layer), DWS (data warehouse service layer), and ADS (application data service layer), thereby providing a structured data framework.

[0029] Furthermore, in step 2.1, the uncertainty of each evidence body and the uncertainty of the entire WSN data are calculated by the following expression:

[0030]

[0031]

[0032] in: U i For the body of evidence E i uncertainty, E i Representation node i body of evidence, m i ( E ) is a node i support, w i For nodes i The importance of the evidence, which reflects the credibility of the evidence. α i For nodes i The weight in WSN, n is the number of nodes in WSN, U total is the uncertainty of the entire WSN dataset.

[0033] Furthermore, the specific implementation of step (3) is as follows:

[0034] 3.1 Based on the semantic web specifications and standards, extensive information related to industrial safety production is collected through multimodal channels including domain experts, books, online information, and literature, so as to establish a domain ontology for industrial safety production and standardize the description of domain-related concepts and their interrelationships;

[0035] 3.2 Based on the semantic representation of domain ontology, a structured knowledge base model is constructed to integrate and manage data from different sources. This knowledge base model dynamically generates semantic templates to uniformly integrate and represent heterogeneous industrial safety production information, transforming unstructured information into formalized structured knowledge records, supporting the efficient storage and read-write operations of large-scale domain knowledge.

[0036] 3.3 Adopt a knowledge base incremental expansion mechanism, allowing domain experts to update and expand knowledge at any time during use through a multimodal human-computer interface; at the same time, design an automated knowledge capture mechanism to extract new knowledge from the Internet and other sources and integrate it into the knowledge base;

[0037] 3.4 Design the access and query interface and reasoning module of the knowledge base to support users to efficiently access knowledge in the field of industrial safety production through keyword queries. According to the query request, the reasoning module is used to dynamically associate and expand related knowledge through semantic analysis to provide support for upper-level intelligent analysis and decision-making control.

[0038] Furthermore, the content of the domain ontology in step 3.1 includes:

[0039] A basic classification system with industry characteristics, building a classification framework that covers infrastructure, equipment, and environmental conditions that are consistent with industrial safety production;

[0040] Definitions of basic concepts with industry characteristics, including key production safety factors including hazard sources, production equipment, and emergency plans;

[0041] A description of the relationships between industry-specific concepts, including the hierarchical structure and associations between different concepts, including equipment and maintenance requirements, risks and countermeasures.

[0042] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned multi-source and multi-modal data fusion and association method based on massive complex spatiotemporal data.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data.

[0044] In order to solve the task of collecting and fusing complex spatiotemporal data in industrial environments, the present invention first collects multi-source multimodal data in real time through the real-time collection technology of multimodal data of safety factors, solves the problem of reduced data transmission rate and data quality of each device in complex and even extreme industrial environments, completes local data processing and data fusion operations, and reduces data transmission and delay; then, the collected data is fused and associated through the dynamic multimodal fusion technology of low-quality data, solves the dependency problem of high-quality data in multimodal data fusion, and adopts a fusion strategy that can be dynamically adjusted to adapt to changes in data uncertainty and improve the fusion effect of data between different modalities; finally, an industrial safety production knowledge base is constructed through the industrial safety production knowledge base modeling technology to provide a knowledge representation basis for subsequent functional modules such as knowledge questions and answers and disposal suggestions. Based on the above, the present invention has the following beneficial technical effects:

[0045] 1. Efficient real-time acquisition of multimodal data. Traditional industrial data acquisition systems can typically only process a single type of data and struggle to cope with the diverse data sources found in complex industrial scenarios. This invention, by designing edge acquisition nodes that support multimodal data fusion, can integrate multiple data sources, such as DCS, PLC, SCADA, and video surveillance, enabling efficient acquisition and real-time processing of multimodal data in industrial safety production. This significantly improves the speed and accuracy of data acquisition and reduces data transmission latency.

[0046] 2. Accuracy and robustness of low-quality data fusion. Existing technologies often oversimplify the processing of low-quality data and fail to effectively distinguish the temporal and spatial correlations and dynamic changes of the data. This invention addresses the problem of processing low-quality data in complex industrial scenarios by introducing an uncertainty-aware model and a dynamic Bayesian network to dynamically adjust the weights of different data sources. Compared to existing technologies, this invention not only improves the accuracy of data fusion but also enhances the system's robustness in multi-interference environments.

[0047] 3. Dynamic expansion and intelligent reasoning of the knowledge base. Traditional industrial safety production knowledge bases mostly rely on manual updates, have poor scalability, and perform insufficiently when dealing with multimodal data and complex reasoning requirements. This invention, through an automated incremental expansion mechanism and semantic reasoning capabilities, not only enables dynamic expansion of the knowledge base but also supports intelligent cross-domain query and reasoning capabilities, enabling continuous updating and optimization of the knowledge base and providing strong support for intelligent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the technical framework of the multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data of the present invention. DETAILED DESCRIPTION

[0049] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] In order to solve the task of collecting and fusing complex spatiotemporal data in industrial environments, the present invention aims to solve the problems of dependence on high-quality data in multimodal data fusion, insufficient robustness and environmental adaptability of data collection, and bandwidth and delay of data transmission; specifically, it aims to solve the problem of how to more efficiently fuse the differences between different modalities of multimodal data (including low-quality data) in complex spatiotemporal environments in industrial scenarios, and how to improve the real-time performance of data transmission and enhance data quality of various data sources (such as sensors, cameras, GPS, etc.) in extreme environments. Therefore, the present invention aims to integrate massive spatiotemporal data (including images, text, videos, sensor data, etc.) from different sources by constructing a multimodal data fusion framework, thereby providing a strong data foundation for fields such as intelligent decision support and large-scale data analysis.

[0051] The present invention first collects multi-source multi-modal data in real time through the real-time collection technology of multi-modal data of safety factors, then fuses and associates the collected data through the dynamic multi-modal fusion technology of low-quality data, and finally constructs the industrial safety production knowledge base through the industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules such as knowledge question and answer and disposal suggestions. The specific technical implementation is as follows: Figure 1 shown.

[0052] (1) Real-time collection technology of multimodal data of security factors

[0053] This technology aims to address the complexity of collecting and integrating various data types related to industrial safety production in complex and even extreme environments, as well as real-time and reliability issues. By designing edge collection nodes that support multimodal data fusion and combining key WSN technologies, it can achieve efficient collection, processing and transmission of diverse safety production data.

[0054] 1.1 Research and design of edge nodes supporting real-time data collection for multimodal fusion

[0055] The factors affecting industrial enterprise safety production are complex and diverse, resulting in a wide variety of data collection. Different data types require different real-time collection methods, ranging from those built into the system's DCS / PLC / SCADA system to wireless sensor networks and video. Therefore, it is necessary to research and design a real-time data collection edge node that supports multimodal fusion, compatible with various data input interfaces and formats, and possesses sufficient computing power to perform local data processing and data fusion operations, thereby reducing data transmission and latency.

[0056] Different from the traditional single-mode data acquisition system, the edge node in the present invention not only has the function of multi-interface fusion, but also has preliminary data fusion processing capabilities, which can pre-process the data locally and reduce the subsequent transmission load.

[0057] Sensor nodes are organized into wireless sensor networks to collect data from industrial production sites. The key technologies involved include data collection, encrypted transmission, fusion, etc. In particular, the external environment of industrial production sites is relatively harsh, complex, and full of uncertainty, which puts higher requirements on the deployment and operation of wireless sensor network systems.

[0058] 1.2 Model-assisted low-cost data collection method for sensor networks with data fusion function

[0059] Given the temporal, spatial, and spatiotemporal correlations of industrial enterprise production safety data (e.g., equipment parameters such as tank farm pressure, liquid level, and flow, and environmental parameters such as temperature, wind speed, smoke, and toxic gas concentrations), it is necessary to develop a model-assisted sensor network data collection method with data fusion capabilities to minimize network data transmission volume and power consumption. Key research areas include: 1. a temporal variation model for industrial enterprise production safety data and a data collection method based on this model; 2. a spatial variation model for industrial enterprise production safety data and a data collection method based on this model; and 3. a data collection method based on the spatiotemporal correlations of industrial enterprise production safety data.

[0060] This paper designs a model-assisted data fusion method that compresses and removes redundancy from data. By establishing temporal and spatial domain change models, it reduces sensor sampling frequency and fuses data from adjacent nodes, minimizing the amount of data transmitted. By reducing the amount of data transmitted by the sensor network, power consumption is reduced, and the service life of wireless sensor network nodes is extended, making it particularly suitable for energy-sensitive industrial environments. Existing sensor network technologies, such as LEACH (Low Energy Adaptive Clustering Hierarchy), primarily rely on simple data fusion by cluster heads, lacking in-depth analysis of data temporal and spatial correlations, and failing to minimize data transmission. This paper introduces a temporal and spatial correlation model to reduce data transmission, significantly improving network energy efficiency in terms of data compression and fusion.

[0061] 1.3 Data encryption transmission technology under low signal-to-noise ratio

[0062] Industrial production environments are often complex, diverse, volatile, and unsafe. The presence of dust, extreme working conditions, drastic temperature fluctuations, and complex electromagnetic environments pose significant challenges to the reliability of WSN data communications. This can reduce the signal-to-noise ratio (SNR) during WSN data transmission, increasing the probability of unreliable data transmission. To address these issues, we propose the following: ① Coding techniques that incorporate network error correction to tolerate a moderate amount of packet transmission errors; ② Research on environmentally aware packet segmentation techniques, which adaptively adopt appropriate coding techniques and packet sizes for communication based on changes in the network environment to improve error correction capabilities; ③ Intelligent routing techniques that are aware of network conditions, which maximize the possibility of data routing routes bypassing areas with high SNRs based on the spatial variations of the network environment; and ④ Random key distribution techniques, which probabilistically pre-distribute keys in wireless sensor networks to ensure that adjacent nodes can share keys with a certain probability, thereby establishing secure communication links.

[0063] This paper designs an adaptive encryption transmission technology that adapts to low signal-to-noise ratios. Through adaptive error correction coding and packet segmentation, it ensures reliable and secure data transmission in low signal-to-noise ratio conditions. This technology combines network error correction coding with adaptive packet size adjustment strategies. Compared to traditional wireless transmission methods, this technology has stronger anti-interference capabilities in complex environments and enhances data security.

[0064] 1.4 Routing technology that provides real-time data transmission

[0065] Complex environments also place demands on the real-time and effectiveness of WSN transmission of monitoring data. How to effectively transmit and process the raw data obtained from monitoring is the key issue studied and addressed in this invention. To this end, it is proposed to combine the environmental characteristics of the monitoring area to achieve real-time monitoring data transmission, focusing on a real-time routing protocol. The routing protocol includes: ① using data forwarding based on geographic location, periodically exchanging information between adjacent sensor nodes, and using a stateless single-hop delay-guaranteed forwarding strategy to achieve real-time end-to-end data transmission; ② when congestion is detected somewhere in the network, a weighted multi-route forwarding strategy is used to reduce the delay caused by fallback rerouting to ensure that end-to-end data transmission has a certain delay upper limit. To ensure the real-time performance of data transmission in complex industrial environments, especially in scenarios where the environment changes rapidly, the transmission path can be quickly adjusted to reduce delays.

[0066] Different from traditional routing, the present invention combines geographic location with multi-path weighting strategy to ensure the real-time and stability of data transmission, and is particularly suitable for key safety monitoring data transmission in industrial production.

[0067] 1.5 Anti-interference strategies for wireless sensor networks

[0068] Because WSNs operate on open wireless communication channels, communication interference is severe, affecting data transmission between nodes. In complex environments, multiple monitoring networks using the same or different channels may coexist, and WSN nodes in monitoring networks are often densely populated. Therefore, WSNs operating in complex environments are inevitably subject to interference from multiple sources, affecting the normal monitoring of production safety data. To address the potential interference in WSN production safety monitoring, this paper studies a WSN anti-interference strategy based on inter-network and inter-cluster methods to ensure reliable data transmission within WSNs. Key research areas include: ① inter-network interference in WSNs; ② inter-cluster interference in WSNs; and ③ a MAC access method suitable for three-dimensional WSNs.

[0069] 1.6 Energy-aware sensor data fusion method

[0070] Since WSN nodes have limited energy, it is an effective method to reduce the amount of data transmitted by nodes through data fusion, thereby reducing the total energy consumption in the network. The present invention expands the existing wireless sensor data fusion protocol and algorithm, studies and implements an energy-efficient WSN data fusion method, focusing on: ① dynamic generation and selection of fusion nodes; ② WSN node caching strategy; ③ heterogeneous sensor data fusion of aggregation nodes, and designs a dynamic fusion algorithm based on energy perception to dynamically adjust data fusion nodes and caching strategies during network operation to reduce the energy consumption of data transmission, ensure the reliability and energy utilization efficiency of WSN data transmission in multiple interference environments, and extend the network life cycle. The data fusion strategy based on energy perception in the present invention not only reduces the repeated transmission of data, but also optimizes the energy use of the network by adjusting the position of the fusion node. Compared with traditional WSN data fusion solutions, it has better energy utilization and anti-interference capabilities.

[0071] The above-mentioned real-time collection technology of multimodal security factor data theoretically supports all mainstream collection methods on the market and is expected to support no less than 4,000 IoT perception protocols.

[0072] (2) Dynamic multimodal fusion technology for low-quality data

[0073] The motivation for multimodal fusion is to jointly leverage effective information from different modalities to improve the accuracy and stability of downstream tasks. Traditional multimodal fusion methods often rely on high-quality data. However, in real-world scenarios, due to the complexity and diversity of application environments, different samples, and different time and space, modal quality has dynamic characteristics, and the emergence of low-quality modal data is often difficult to predict in advance. Therefore, this paper conducts research on complex and low-quality multimodal data in real-world applications to develop a dynamic multimodal fusion technology for low-quality data.

[0074] Dynamic multimodal data refers to data whose modal quality changes dynamically depending on the input sample and scenario. For example, in industrial safety production scenarios, road systems use RGB (red, green, and blue) and infrared sensors to obtain road surface and target information. In bright lighting conditions, RGB cameras, because they capture rich texture and color information of targets, can better support intelligent system decision-making. However, in low light conditions at night, infrared sensors provide more reliable perception information. Enabling models to automatically perceive changes in the quality of different modalities, thereby enabling accurate and stable fusion, is the core task of dynamic multimodal fusion methods. Dynamic multimodal fusion methods can process data from different sources and types and integrate them to provide more comprehensive and accurate information. In industrial safety production scenarios, due to the diverse data types involved and the fact that the characteristics and requirements of these data may change over time, a dynamic approach to data fusion is necessary. Furthermore, considering the potential uncertainty in the data fusion process, adopting a fusion strategy that can dynamically adjust to changing data uncertainty is key to improving fusion effectiveness.

[0075] Specifically, the present invention first uses the Dempster-Shafer framework to construct an uncertainty perception model, treating each sensor report as a body of evidence. In this framework, each body of evidence has its support and importance. These two metrics can be used to evaluate the uncertainty of the body of evidence. By analyzing the data set, an overall assessment of the uncertainty of the entire data set can be obtained. Taking into account the uncertainty that may arise during the data fusion process, the present invention adopts a DBN-based fusion algorithm that can dynamically adjust the fusion strategy according to the uncertainty of the current data, thereby improving the accuracy and reliability of the fusion results. That is, first, fuzzy logic is used to preprocess the underlying data, and then a dynamic Bayesian network is used to fuse these preprocessed data. In this way, the uncertainty in the data can be effectively handled, and data information from different sources can be fully utilized. Finally, the processed data is stratified according to the five basic element thematic libraries of events, videos, people, places, and equipment.

[0076] 2.1 Construction of uncertainty perception model

[0077] The Dempster-Shafer evidence theory is used to construct an uncertainty perception model. This framework allows us to analyze data provided by different sensors as evidence bodies. Each evidence body has support and importance to evaluate its uncertainty.

[0078] set up E i Indicates sensor i body of evidence, mi ( E ) is its support, importance w i It reflects the credibility of the evidence. For each sensor, its uncertainty U i , calculated using the following formula:

[0079]

[0080] The Dempster-Shafer evidence theory effectively handles uncertainty in multimodal data, enhancing the system's ability to perceive low-quality data. Unlike traditional data fusion methods that rely on fixed quality assumptions, this paper uses an uncertainty-aware framework to enable the system to dynamically perceive and adjust the weight of low-quality data.

[0081] 2.2 Overall assessment of data uncertainty

[0082] The uncertainty for the entire dataset is calculated by integrating the evidence from all sensors:

[0083]

[0084] in: α i It is a sensor i The weight of n is the number of nodes in the WSN.

[0085] Based on the weight of each sensor and combining all evidence, an overall uncertainty assessment of the data set is derived, providing a basis for subsequent data fusion and ensuring that low-quality data is not overly relied upon in fusion decisions; the present invention combines the overall uncertainty assessment of multiple modal data to achieve dynamic regulation of the fusion accuracy of the entire system.

[0086] 2.3 Fuzzy logic preprocessing

[0087] Before data fusion, fuzzy logic is used to pre-process the underlying data to deal with the uncertainty in the data. Fuzzy logic systems typically map input values ​​into fuzzy sets through membership functions, then apply fuzzy rules for reasoning and output results with a high degree of membership.

[0088] For an input value x , its membership function μ ( x ) is defined as:

[0089]

[0090] in: c is the center value, σis an extended parameter that controls the scope of fuzziness.

[0091] The present invention preprocesses modal data of different qualities, reduces data noise and errors, improves fusion effect, and dynamically adjusts data input through fuzzy logic, which can flexibly respond to changes in data quality in different environments.

[0092] 2.4 Dynamic Bayesian Network Fusion Algorithm

[0093] Dynamic Bayesian networks are used to dynamically adjust fusion strategies based on uncertainty. DBNs can capture changes in data states over time and update the model through Bayesian reasoning. During the fusion process, modal data with higher uncertainty is assigned lower weights, reducing its impact on the final decision and ensuring the accuracy and robustness of the fusion results.

[0094] The present invention dynamically adjusts the data fusion strategy according to the changes in the quality of the modal data to achieve adaptive fusion in different environments; the introduction of the dynamic Bayesian network enables the system to automatically adjust the fusion strategy in a time series environment, significantly improving the stability and accuracy of multimodal data fusion.

[0095] 2.5 Data stratification and subject database construction.

[0096] After data fusion is complete, the processed data is stratified into a thematic library based on five basic elements: events, videos, people, locations, and equipment. This step involves classifying and organizing the fused data, providing a structured data framework that facilitates subsequent downstream tasks (such as data analysis and event prediction) and supports various intelligent applications in industrial production. The data stratification and the construction of the thematic library in this invention can effectively support multidimensional data management in complex industrial scenarios, improving data utilization efficiency and operability.

[0097] (3) Industrial safety production knowledge base modeling technology

[0098] The present invention aims to establish a knowledge base for safe production in complex industrial sites, and to model and develop the knowledge base based on ontology to support intelligent decision-making and knowledge expression. Based on the knowledge in the field of industrial production, and taking full account of the production characteristics of complex industrial sites, especially chemical enterprises, a domain ontology for safe production in complex industrial sites is established; based on the industrial safe production ontology, a large-scale knowledge base for safe production in complex industrial sites is realized. Through cross-domain data and view fusion technology, the knowledge base integrates multimodal data sources (such as text, pictures, expert experience, etc.), and contains various basic knowledge and safe production models required for safe production in complex industrial sites; finally, through the automated incremental expansion mechanism and query reasoning function, comprehensive knowledge representation and management of safe production in complex industrial sites is realized, providing a knowledge representation basis for upper-level intelligent decision support. The specific research content is detailed as follows:

[0099] 3.1 Industrial Site Safety Production Ontology Modeling Supporting Multimodal Fusion and Cross-Domain Digital and Visual Fusion

[0100] Based on universal semantic web specifications and standards, we collect information related to industrial enterprise safety production in specific industries through various channels (domain experts, books, online information, and literature). We then establish a relatively complete industrial enterprise domain ontology through a multimodal approach, standardizing the description of domain concepts and their interrelationships. The domain ontology mainly includes the following aspects:

[0101] ① A basic classification system for industrial enterprise safety production with industry characteristics, building a classification framework that conforms to industrial enterprise safety production, covering infrastructure, equipment, environmental conditions, etc.;

[0102] ② Definition and description of basic concepts related to industrial safety production with industry characteristics, and definition of key safety production elements, such as hazard sources, production equipment, emergency plans, etc.;

[0103] ③ Definition and description of the relationships between concepts, describing the hierarchical structure and associations between different concepts, such as the relationship between equipment and its maintenance requirements, and risks and response measures.

[0104] The present invention establishes a comprehensive domain ontology through structured modeling of safe production in complex industrial sites, providing a semantic basis for the development of knowledge base and knowledge reasoning.

[0105] 3.2 Ontology-based structured knowledge base model

[0106] Based on the semantic representation of domain ontology, a structured knowledge base model is constructed to integrate and manage information from various sources. By dynamically generating semantic templates, heterogeneous industrial enterprise safety production information (expert experience, literature, images, industry data, safety production models, etc.) is uniformly integrated and represented, transforming unstructured information or natural language descriptions (such as text and images) into formalized, structured knowledge records. Furthermore, specific knowledge storage models are being researched and designed to support the efficient storage of large-scale industry and domain knowledge, as well as the efficient reading and writing of knowledge records.

[0107] The structured knowledge base model of the present invention makes it possible to integrate and manage heterogeneous data, ensuring the systematic and standardized representation of industrial safety production related knowledge; unified knowledge integration based on semantic representation breaks through the bottleneck of traditional knowledge bases that are difficult to handle heterogeneous and heterogeneous information, ensuring knowledge consistency.

[0108] 3.3 Knowledge Base Incremental Expansion Mechanism Supporting Multimodal Fusion

[0109] In addition to centrally defining and collecting industrial enterprise safety production knowledge as the core content of the industrial enterprise safety production knowledge base, it is also necessary to continuously add and expand new domain knowledge during the use of the knowledge base. To this end, it is necessary to study the incremental expansion mechanism of the knowledge base, provide a simple and effective multimodal human-computer interface, allow domain experts to dynamically enter knowledge in a multimodal manner, and automatically capture or extract knowledge from multiple media (such as the internet).

[0110] The dynamic expansion and update mechanism of the knowledge base ensures its persistence and adaptability in application and is able to track the latest developments in the field of safe production in industrial enterprises. The present invention designs an expansion mechanism that combines multimodal input with automated knowledge capture, which reduces the complexity of manual updates and improves the dynamic response capability of the knowledge base.

[0111] 3.4 Knowledge Base Query and Reasoning

[0112] Research and implement an access and query interface for an industrial workplace safety production knowledge base oriented towards cross-domain digital and visual integration, supporting keyword-based knowledge query and search, while implementing basic knowledge reasoning to support reasoning about the most basic semantic relationships, dynamically associating and expanding knowledge query requests; the query and reasoning mechanism of the knowledge base can be used on a public platform for external display to implement semantic-based intelligent analysis and control. The present invention provides an intelligent query and reasoning interface that supports industrial safety production applications with cross-domain digital and visual integration, enhancing the practical value of the knowledge base; through the semantic reasoning mechanism, the knowledge base not only supports static queries, but can also dynamically expand query results, improving the intelligence level and responsiveness of the system.

[0113] The technical solution of this invention is widely applicable to a variety of data types, including ground sensor data, video surveillance streams, and IoT device data. This data not only comes from a wide range of sources and is diverse in form, encompassing multiple modalities such as text, images, video, and sound, but is also used in core application scenarios such as smart cities, intelligent transportation, environmental monitoring, and industrial automation. Through efficient data fusion and correlation technologies, the value behind this complex data can be integrated and mined, providing strong data support for decision support, process optimization, and fault prediction in industrial systems, thereby promoting the in-depth development of industrial intelligence and digital transformation.

[0114] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It is apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments. Any improvements or modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data, characterized by: The steps include: (1) Real-time collection and data transmission of multi-source and multi-modal data through real-time collection technology of multi-modal data of security elements; (2) The collected data are fused and correlated through the low-quality data dynamic multimodal fusion technology. The specific implementation method is as follows: 2.1 An uncertainty perception model is constructed using the Dempster-Shafer framework. The data of each node is regarded as an evidence body. Each evidence body has its support and importance. These two metrics are used to evaluate the uncertainty of the evidence body. Historical data or prior knowledge are used to assign a weight, i.e., importance, to each node. The uncertainty of each evidence body is calculated, and then the evidence bodies of the nodes in the WSN are integrated to obtain the uncertainty of the entire WSN dataset. 2.2 Before data fusion, the data is pre-processed by the fuzzy logic system. The fuzzy logic system maps the data input value to the fuzzy set through the membership function, and outputs the result with higher membership after applying fuzzy rule reasoning; 2.3 Use dynamic Bayesian networks for data fusion. Dynamic Bayesian networks capture the state changes of data over time and dynamically adjust the weights of different modal data in the fusion according to their uncertainty. For data with higher uncertainty, the dynamic Bayesian network assigns it a lower weight. 2.4 Classify and organize the fused data by subject classification based on the five basic elements of events, videos, people, locations, and equipment, and stratify the data by ODS, DWD, DWS, and ADS, thereby providing a structured data framework; (3) Based on the fused data, an industrial safety production knowledge base is constructed through industrial safety production knowledge base modeling technology, providing a knowledge representation basis for subsequent functional modules including knowledge question and answer and disposal suggestions.

2. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 1 is characterized by: The specific implementation of step (1) is as follows: 1.1 Design edge nodes that support real-time data collection for multimodal fusion, and form these edge nodes into a WSN to collect data from industrial production sites; 1.2 Adopt a model-assisted data fusion method to compress and remove redundancy from the collected data. By establishing a spatiotemporal correlation model, the model is used to assist in reducing the node sampling frequency and fusing the data of adjacent nodes, thereby minimizing the amount of transmitted data, reducing power consumption, and extending the service life of the node. 1.3 Adopt an adaptive encryption transmission method that adapts to low signal-to-noise ratios. That is, according to changes in the network environment, adaptive coding technology and data packet size are used for communication, and a moderate amount of data packet transmission errors are tolerated. In the WSN, keys are pre-distributed in a probabilistic manner to ensure that adjacent nodes can share keys with a certain probability, thereby establishing a secure communication link. 1.4 Adopt a data forwarding strategy based on geographic location, that is, adjacent nodes exchange information regularly and use stateless single-hop delay to ensure the real-time performance of end-to-end data transmission; adopt a weighted multi-route forwarding strategy, that is, when congestion is detected somewhere in the network, automatically switch to an alternative path to reduce the delay caused by fallback rerouting; based on the spatial variation characteristics of the network environment, make the data routing lines bypass areas with high signal-to-noise ratio as much as possible.

3. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 2 is characterized by: The edge node has multiple data interfaces and supports access to multimodal data collected and acquired through various means including DCS, PLC, SCADA, wireless sensor networks and video acquisition. At the same time, through the modular design of the hardware interface, the edge node can be compatible with multiple communication protocols, realizing real-time collection and local processing of industrial safety production factor data, effectively reducing data transmission delays.

4. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 2, characterized in that: The real-time acquisition technology of multimodal data of safety factors adopts an anti-interference strategy based on inter-network and inter-cluster for data transmission to avoid interference in industrial production safety monitoring. At the same time, in step 1.2, the existing WSN fusion protocol and algorithm are expanded, and a dynamic fusion algorithm based on energy perception is designed. During the network operation, the data fusion nodes and node caching strategies are dynamically adjusted to reduce the node energy consumption during the data transmission process.

5. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 1 is characterized by: In step 2.1, the uncertainty of each evidence body and the uncertainty of the entire WSN data are calculated by the following expression: U i =1-w i m i (HAVE BEEN) Among them: U i Evidence Body E i The uncertainty of E i represents the evidence body of node i, m i (E) is the support of node i, w i is the importance of node i, which reflects the credibility of the evidence, α i is the weight of node i in WSN, n is the number of nodes in WSN, U total is the uncertainty of the entire WSN dataset.

6. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 1, characterized in that: The specific implementation of step (3) is as follows: 3.1 Based on the semantic web specifications and standards, extensive information related to industrial safety production is collected through multimodal channels including domain experts, books, online information, and literature, so as to establish a domain ontology for industrial safety production and standardize the description of domain-related concepts and their interrelationships; 3.2 Based on the semantic representation of domain ontology, a structured knowledge base model is constructed to integrate and manage data from different sources. This knowledge base model dynamically generates semantic templates to uniformly integrate and represent heterogeneous industrial safety production information, transforming unstructured information into formalized structured knowledge records, supporting the efficient storage and read-write operations of large-scale domain knowledge. 3.3 Adopt a knowledge base incremental expansion mechanism, allowing domain experts to update and expand knowledge at any time during use through a multimodal human-computer interface; at the same time, design an automated knowledge capture mechanism to extract new knowledge from the Internet and other sources and integrate it into the knowledge base; 3.4 Design the access and query interface and reasoning module of the knowledge base to support users to efficiently access knowledge in the field of industrial safety production through keyword queries. According to the query request, the reasoning module is used to dynamically associate and expand related knowledge through semantic analysis to provide support for upper-level intelligent analysis and decision-making control.

7. The multi-source and multi-modal data fusion and association method based on massive and complex spatiotemporal data according to claim 6, characterized in that: The contents of the domain ontology in step 3.1 include: A basic classification system with industry characteristics, building a classification framework that covers infrastructure, equipment, and environmental conditions that are consistent with industrial safety production; Definitions of basic concepts with industry characteristics, including key production safety factors including hazard sources, production equipment, and emergency plans; A description of the relationships between industry-specific concepts, including the hierarchical structure and associations between different concepts, including equipment and maintenance requirements, risks and countermeasures.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The processor is used to execute the computer program to implement the multi-source and multi-modal data fusion and association method based on massive complex spatiotemporal data as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the multi-source and multi-modal data fusion and association method based on massive complex spatiotemporal data as claimed in any one of claims 1 to 7.

Images