Deterministic network modeling method based on data driving and knowledge graph

Abstract

The invention discloses a deterministic network modeling method based on data driving and a knowledge graph, and the method comprises the steps: obtaining various real-time data of a system in operation, and carrying out the preprocessing of the data; constructing a knowledge graph based on the collected data, expert knowledge and a system design document, obtaining two preliminary models with different granularities, namely a system level and a node level, and obtaining a model of the system level and a model of the node level; identifying the preprocessed data and removing abnormal values by adopting a method based on machine learning to obtain cleaned high-quality data, namely data under the normal condition of the system; constructing a self-adaptive model threshold value based on a statistical algorithm by using the cleaned high-quality data; and the model receives the preprocessed data, screens the preprocessed data by using a self-adaptive model threshold, integrates the screened data in combination with the constructed knowledge graph, dynamically and accurately depicts the data of each node, and provides decision support based on the integrated information.

Description

Technical Field

[0001] The present invention relates to a deterministic network modeling method based on data-driven and knowledge graph, and belongs to the field of complex network modeling. Background Art

[0002] In today's highly information-based and globalized era, complex command and control system networks play a vital role in various fields. Whether it is military operations, emergency rescue, large-scale engineering construction, or urban management, efficient and reliable command and control are indispensable. With the development of society and the advancement of science and technology, the scenarios of command and control are becoming increasingly complex and changeable, and the requirements are becoming more diverse and demanding. In the military field, modern warfare presents the characteristics of multi-service coordinated operations and informationized operations. The battlefield situation changes rapidly, which puts forward extremely high requirements for the rapid response, accurate decision-making and efficient coordination of the command and control system. For example, in joint operations, it is necessary to integrate the information of multiple combat units such as land, sea, air, and space to achieve integrated command and control. This requires modeling of complex command and control system networks to optimize information transmission paths, improve command efficiency, and ensure victory in operations.

[0003] In terms of emergency rescue, such as responding to major natural disasters such as earthquakes, floods, and fires, multiple departments and institutions need to work together, including firefighting, medical care, transportation, civil affairs, etc. Complex command and control system network modeling can clarify the responsibilities and relationships between departments, achieve reasonable allocation of resources, improve the timeliness and effectiveness of rescue, and minimize casualties and property losses.

[0004] There are some significant defects in the existing methods of modeling complex command and control system network systems: First, the existing methods often have low computational efficiency when dealing with large-scale, highly dynamic command and control networks. Since actual command and control systems may involve numerous nodes and complex interactions, existing modeling algorithms may not be able to quickly provide accurate command decision support when dealing with massive data and frequent network changes. Second, existing modeling methods are insufficient in cross-domain integration. Complex command and control systems often involve knowledge and technology from multiple fields, such as information technology, management, psychology, etc., but existing methods are not perfect in integrating elements from these different fields, resulting in limited comprehensiveness and practicality of the model. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, to provide a deterministic network modeling method based on data-driven and knowledge graphs, to process data based on artificial intelligence and deep learning methods, to represent the knowledge in related fields in the form of graphs based on knowledge graphs, to systematically display each node and their mutual relationships, to obtain preliminary modeling at two different granularities, namely, the system level and the node level, and finally to determine the normal range of the data through statistical analysis, to identify key eigenvalues ​​and indicators for system level modeling and node level modeling respectively, to achieve better modeling effects.

[0006] The technical solution of the present invention is: a deterministic network modeling method based on data-driven and knowledge graph, including:

[0007] Obtain various real-time data of the running system and pre-process the data;

[0008] Construct a knowledge graph based on the collected data, expert knowledge, and system design documents, including preliminary modeling at two different granularities: system level and node level, to obtain a system level model and a node level model;

[0009] Use machine learning-based methods to identify and remove outliers from preprocessed data to obtain cleaned high-quality data, that is, data under normal system conditions;

[0010] Use cleaned high-quality data to build adaptive model thresholds based on statistical algorithms;

[0011] The model receives preprocessed data, filters it using an adaptive model threshold, integrates the filtered data with the constructed knowledge graph, accurately describes the dynamics of each node data, and provides decision support based on the integrated information.

[0012] Preferably, the various data in operation acquired in real time are time series data;

[0013] If there are fewer missing values ​​in the data, the corresponding missing values ​​are discarded, otherwise the data are interpolated; at the same time, the data are downsampled and smoothed.

[0014] Preferably, the method for constructing a knowledge graph based on collected data, expert knowledge and system design documents is:

[0015] Identify key entities based on system design documents, extract system architecture diagrams, module divisions, data flow diagrams, and interface design information from system design documents, and mark all modules, components, and services, as well as corresponding functional descriptions and technology stack information;

[0016] Create a relationship list to describe the interaction methods and data flows between modules. Use a graph database to represent entities and relationships. Create entity nodes for each module, component, and service, and create relationship edges for the interactions between modules.

[0017] Based on the above key entities and relationships, a topological diagram of the system is drawn based on the knowledge graph. Nodes are used to represent the modules or components of the system, and edges are used to represent the relationships and data flows between modules. In the topological diagram, detailed information labels are added to each node and edge to describe the module function, interface type, and data transmission protocol.

[0018] Preferably, the preliminary modeling at the system level is specifically as follows:

[0019] Identify the main functional modules, services and components from the system design documents, identify the interaction relationships at the system level, thereby determining the interaction relationships and data flows between the main modules, construct the entity nodes and relationship edges at the system level, design the topology model at the system level, and obtain the system level model;

[0020] The model can accept preprocessed data, filter the data through the threshold within the model, integrate the filtered data, and provide decision support based on the integrated information.

[0021] Preferably, the node level focuses on the internal structure of each module and the relationship between its subcomponents. The preliminary modeling at the node level is as follows:

[0022] Identify the subcomponents and services within each module from the system design documents, determine the interaction relationships and data flows between the subcomponents, build entity nodes at the node level, and obtain a node-level model;

[0023] The node-level model receives preprocessed data, filters the data through the threshold within the model, integrates the filtered data, and provides decision support based on the integrated information.

[0024] Preferably, a method based on artificial intelligence and deep learning is used to identify and remove outliers from the preprocessed data, including a rough recall stage and a fine recall stage, specifically:

[0025] In the rough recall stage, Kalman filtering, kurtosis, skewness, exponential moving average and STL time series data decomposition strategies are used to extract features from the preprocessed data. The strategy of integrating two machine learning methods, Isolation Forest and LODA, is used to quickly and preliminarily screen out normal values ​​and possible outliers based on the extracted features as suspicious data.

[0026] In the fine recall stage, normal data filtered by the coarse recall module are normalized and a training data set is constructed according to a fixed sliding window length; a variational autoencoder model based on semi-implicit distribution is trained based on the training data set; after the training is completed, the variational autoencoder model calculates the reconstruction probability value of the observed data at each moment in real time, and the smaller the reconstruction probability value, the greater the probability of abnormality; a threshold is set for the reconstruction probability value for screening: if the reconstruction probability of a sample is lower than the threshold, it means that the input data does not conform to the normal data distribution learned by the model and can be classified as an outlier; the identified outliers are removed to obtain the cleaned data.

[0027] Preferably, when setting a threshold for the reconstruction probability, an automatic threshold setting method based on extreme value theory is used, that is, the Pareto distribution function is used to model the distribution obeyed by extreme values, and the parameters in the distribution are estimated using the maximum likelihood for the reconstruction probability, and finally the corresponding threshold is quickly calculated.

[0028] Preferably, the cleaned high-quality data is used to construct an adaptive model threshold based on a statistical algorithm, specifically:

[0029] For the real-time data of the system that has been cleared of outliers, find the median X of the data in the window of length W with this point as the end of the data sequence for each data point. m ; Then calculate the absolute value sequence of the difference between the data in the window and the median {|X i -X m |} i＝1,...W , the median of this absolute value sequence is the median absolute deviation, that is,

[0030] MAD=Median({|X i -X m |} i＝1,...W )

[0031] If the observed data exceeds the absolute deviation range of the three medians above and below the median of the sequence, that is:

[0032] X m +3MAD <X obs

[0033] or:

[0034] X m -3MAD>X obs

[0035] It will be marked as abnormal and an alarm will be issued.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] (1) In view of the problems of high-dimensional and heterogeneous elements, complex and variable interaction relationships, and uncertain evaluation measures in modeling complex deterministic systems, a knowledge graph is constructed based on expert knowledge, the characteristics of collected data, and system design documents to model deterministic systems from both the system and node levels.

[0038] (2) In view of the highly dynamic characteristics of complex deterministic systems, a data-driven approach is used to detect and remove noise and anomalies from existing data. Based on statistical characteristics and data analysis, the thresholds of constraints such as the flow rate and delay size of each node are dynamically and adaptively set to achieve dynamic and accurate characterization. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a block diagram of the system composition of the present invention. DETAILED DESCRIPTION

[0040] A data-driven and knowledge graph-based deterministic network modeling method, the specific process is:

[0041] The first step is to collect sensor data of the deterministic system, obtain various operating data in real time through efficient data collection methods, and pre-process the data, including filling missing values ​​and normalization.

[0042] The second step is to build a knowledge graph based on the collected data, expert knowledge and system design documents. By organizing and integrating the existing knowledge system, the knowledge graph represents the knowledge in related fields in the form of a graph, systematically displays each node and their relationship, and obtains preliminary modeling at two different granularities: system level and node level.

[0043] The third step is to use artificial intelligence and deep learning-based methods to process relevant features and remove outliers and noise for dynamic characteristics such as flow restrictions and delay restrictions in system modeling.

[0044] The fourth step is to build an adaptive model threshold based on the statistical algorithm based on the cleaned high-quality data. The system continuously optimizes and adjusts the dynamic parameters in the model according to real-time data and historical data to ensure that the threshold built based on the statistical algorithm is always in a normal and stable state, thereby accurately characterizing the dynamics of each node data.

[0045] Furthermore, the specific process of step one of the present invention is:

[0046] The specific process is:

[0047] (1) The acquired system sensor data is time series data, such as traffic flow, delay, etc. If there are few missing values ​​in the data, the corresponding missing values ​​are discarded, otherwise the data needs to be interpolated.

[0048] (2) Since the sampling data frequency is high and contains a lot of noise, the data needs to be downsampled and smoothed.

[0049] Furthermore, the specific process of the second step of the present invention is:

[0050] (1) Build a knowledge graph based on system design documents, expert knowledge, and the data extracted in the first step:

[0051] Identify key entities based on system design documents, extract system architecture diagrams, module divisions, data flow diagrams, interface designs and other information from system design documents, mark all modules, components and services, as well as their functional descriptions and technology stack information.

[0052] Create a relationship list to describe the interaction methods and data flows between modules. Use a graph database to represent entities and relationships, create entity nodes for each module, component, and service, including specific functional attributes in the system design document, etc. In addition, create relationship edges for each interaction between modules, including attributes such as interface type and data transmission protocol.

[0053] Based on the above entities and relationships, the topology of the system is drawn based on the knowledge graph, using nodes to represent the modules or components of the system, and edges to represent the relationships and data flows between modules. In the topology diagram, detailed information labels are added to each node and edge to explain the module function, interface type, and data transmission protocol, etc. It is specifically divided into two levels: system and node, so that managers can understand and manage each part of the system in more detail.

[0054] (2) At the system level, we first extract the key entities and relationships at the system level, identify the main functional modules, services and components from the system design documents, such as user management module, data processing module, communication module, etc., and identify the interaction relationships at the system level to determine the interaction relationships and data flows between the main modules, such as data transmission protocols, interface types, etc., so as to construct the entity nodes and relationship edges at the system level, and then design the topological model at the system level to obtain the system level model. The model can accept preprocessed data, filter the data through the constraints (thresholds) within the model, integrate the filtered data, and provide decision support based on the integrated information.

[0055] (3) At the node level, the node level focuses on the internal structure of each module and the relationship between its subcomponents. The topological model at this level shows the internal details of each module and the interaction of its subcomponents. First, it is necessary to identify the subcomponents within the module, that is, to identify the subcomponents and services within each module from the system design document, such as database, cache, API processor, etc. In addition, it is necessary to determine the interaction relationship and data flow between subcomponents, such as database query, cache update, etc., to build the entity nodes at the node level, and obtain the node level model. The model can accept pre-processed data, filter the data through the constraints (thresholds) within the model, integrate the filtered data, and provide decision support based on the integrated information.

[0056] Furthermore, the specific process of the third step of the present invention is:

[0057] (1) Node-level constraints, such as traffic size, delay time, etc., need to be set dynamically based on the data. The preprocessed data collected in the first step is cleaned and trained.

[0058] (2) Using artificial intelligence and deep learning-based methods to detect noise and anomalies in the first-step processed data, the characteristics of the system under normal conditions can be obtained.

[0059] Specifically, in order to ensure real-time and accuracy, a two-stage recall strategy is used to detect and remove anomalies and noise points in historical data. Since the amount of system data collected is large and the real-time requirements are high, two modules, coarse recall and fine recall, are used to detect anomalies and clean data. The goal of coarse recall is to ensure that anomalies and noise points can be detected, and to use a fast anomaly detection algorithm to quickly filter out most of the normal data. The fine recall module uses a trained variational autoencoder model for the remaining small part of the data to further improve the detection accuracy.

[0060] The first is the rough recall module. The goal of this module is to quickly filter a large amount of normal data while ensuring the recall as much as possible, and screen out suspicious data for the second stage of recall. The specific implementation strategy is: first, feature engineering is performed, and features are extracted using Kalman filtering, kurtosis, skewness, exponential moving average method, and STL time series data decomposition strategy, so that normal data and anomalies and noise points are better separable after feature extraction; secondly, the strategy of integrating the two machine learning methods of isolation forest and LODA is used to quickly screen out normal data and possible anomalies and noise points based on the extracted features, and possible anomalies and noise points are regarded as suspicious data. Since the computational cost of feature engineering and the above two machine learning methods is low, the real-time performance is good, and suspicious data can be quickly screened out.

[0061] The second is the fine recall module, which aims to further improve the detection accuracy based on the suspicious data filtered by the coarse recall. The specific implementation strategy is: first, based on the normal data filtered by the coarse recall module, normalize and build a training data set according to a fixed sliding window length; then train a variational autoencoder model based on a semi-implicit distribution based on the data set. Compared with the general autoencoder model, the variational autoencoder model can model the distribution of normal data, so it is more robust to anomaly detection; in addition, compared with the standard variational autoencoder, the model used in this scheme is more flexible in modeling the distribution of normal data, so the modeling ability of the model is stronger. The encoder and decoder of the variational autoencoder are both implemented using a simple multi-layer perceptron model, so the number of parameters is small and the training process is fast. After the training is completed, the reconstruction probability value of the observed data at each moment can be calculated in real time. The smaller the reconstruction probability value, the greater the probability of its abnormality. Therefore, it is necessary to set a threshold for the reconstruction probability value for screening: if the reconstruction probability of a sample is lower than the threshold, it means that the input sample does not conform to the normal data distribution learned by the model and can be classified as an abnormal sample; the model adopts a regular update strategy so that the model can meet the usage requirements according to the data characteristics at different stages.

[0062] In addition, when setting the threshold for the reconstruction probability, an automatic threshold setting method based on extreme value theory is used to avoid the cumbersome threshold setting process. Specifically, the Pareto distribution function is used to model the distribution obeyed by extreme values, and the parameters in the distribution are estimated using the maximum likelihood for a given risk probability (i.e., reconstruction probability). Finally, the corresponding threshold is quickly calculated, avoiding cumbersome manual settings.

[0063] The abnormal points and noise points finally detected are the points that need to be removed from the original data, and the data after they are screened out is normal data.

[0064] Furthermore, the specific process of the fourth step of the present invention is:

[0065] (1) For the cleaned normal data, feature statistics tools are used to adaptively set node constraints, i.e., thresholds, such as flow size, delay constraints, etc., so as to accurately characterize the dynamic characteristics of the system;

[0066] Specifically, for the real-time data of the system that has been cleared of noise points and abnormal points, a simple statistical method such as Median Absolute Deviation (MAD) is used to set the threshold. Specifically, for the real-time data of the system that only has normal data (such as the traffic of each node, the delay size, etc.), for each data point, the median X of the data in the window with a length of W and the point as the end of the data sequence is found. m ; Then calculate the absolute value sequence of the difference between the data in the window and the median {|X i -Xm |} i＝1,...W , the median of this absolute value sequence is the median absolute deviation, that is,

[0067] MAD=Median({|X i -X m |}| i＝1，...W )

[0068] If the observed data exceeds the absolute deviation range of the three medians above and below the median of the sequence, that is,

[0069] X m +3MAD<X obs

[0070] or

[0071] X m -3MAD>X obs

[0072] , it will be marked as abnormal and an alarm will be issued.

[0073] As the data is constantly changing, the MAD and the threshold calculated based on the MAD also change accordingly, so that the threshold can always model the data distribution under normal conditions.

[0074] (2) Constraint settings are regularly updated based on the data after cleaning and removing anomalies, thus ensuring that the modeling system can adaptively adjust the thresholds based on the collected data, thereby accurately modeling the state of the system.

[0075] As time goes by, the data in the time window is constantly changing, and the corresponding median and median absolute deviation are also constantly changing, and the threshold also changes accordingly, thereby achieving the effect of the threshold changing dynamically with time and data.

[0076] for Figure 1 The system shown can be divided into three stages and continuously cycled. Different data is used in each stage. The specific data in chronological order are as follows:

[0077] Phase 1: Adaptive threshold setting: It is divided into three steps: preprocessing -> removing noise points and outliers based on machine learning methods -> setting adaptive thresholds based on statistical algorithms. The output of each step is the input of the next step, so the adaptive threshold setting based on statistical algorithms is based on the cleaned data;

[0078] Phase 2: Real-time monitoring: At this point, the threshold of the first phase is already in place. The real-time data of each node is received after preprocessing. The system is monitored by comparing the size of the threshold and the preprocessed data of the real-time data.

[0079] The third stage: threshold update: After the real-time data of the second stage is recorded, the threshold of the first stage can be updated. It also needs to go through three steps: preprocessing, removing noise and outliers, and adaptive threshold using statistical algorithm.

[0080] The above threshold setting and updating parts all use cleaned data; the real-time monitoring part receives pre-processed data.

[0081] The present invention can overcome the problem in existing system modeling tasks that existing methods are unable to model large-scale, highly dynamic deterministic network systems. In addition, the knowledge graph is constructed using knowledge from multiple fields to achieve high-precision, adaptive modeling of the system, which has certain reference significance for building an intelligent modeling system.

[0082] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A deterministic network modeling method based on data-driven and knowledge graph, characterized by include: Obtain various real-time data of the running system and pre-process the data; Construct a knowledge graph based on the collected data, expert knowledge, and system design documents, including preliminary modeling at two different granularities: system level and node level, to obtain a system level model and a node level model; Use machine learning-based methods to identify and remove outliers from preprocessed data to obtain cleaned high-quality data, that is, data under normal system conditions; Use cleaned high-quality data to build adaptive model thresholds based on statistical algorithms; The model receives preprocessed data, filters it using an adaptive model threshold, integrates the filtered data with the constructed knowledge graph, accurately describes the dynamics of each node data, and provides decision support based on the integrated information.

2. According to claim 1, a deterministic network modeling method based on data-driven and knowledge graph, characterized in that: The various data in operation acquired in real time are time series data; If there are fewer missing values ​​in the data, the corresponding missing values ​​are discarded, otherwise the data are interpolated; at the same time, the data are downsampled and smoothed.

3. According to claim 1, a deterministic network modeling method based on data-driven and knowledge graph, characterized in that: The method of building a knowledge graph based on collected data, expert knowledge and system design documents is: Identify key entities based on system design documents, extract system architecture diagrams, module divisions, data flow diagrams, and interface design information from system design documents, and mark all modules, components, and services, as well as corresponding functional descriptions and technology stack information; Create a relationship list to describe the interaction methods and data flows between modules. Use a graph database to represent entities and relationships. Create entity nodes for each module, component, and service, and create relationship edges for the interactions between modules. Based on the above key entities and relationships, draw the topology diagram of the system based on the knowledge graph, use nodes to represent the modules or components of the system, and use edges to represent the relationships and data flows between modules; In the topology diagram, detailed information labels are added to each node and edge to explain the module function, interface type and data transmission protocol.

4. According to claim 1, a deterministic network modeling method based on data-driven and knowledge graph, characterized in that: The preliminary modeling at the system level is as follows: Identify the main functional modules, services and components from the system design documents, identify the interaction relationships at the system level, thereby determining the interaction relationships and data flows between the main modules, construct the entity nodes and relationship edges at the system level, design the topology model at the system level, and obtain the system level model; The model can accept preprocessed data, filter the data through the threshold within the model, integrate the filtered data, and provide decision support based on the integrated information.

5. According to claim 1, a deterministic network modeling method based on data-driven and knowledge graph, characterized in that: The node level focuses on the internal structure of each module and the relationship between its subcomponents. The preliminary modeling at the node level is as follows: Identify the subcomponents and services within each module from the system design documents, determine the interaction relationships and data flows between the subcomponents, build entity nodes at the node level, and obtain a node-level model; The node-level model receives preprocessed data, filters the data through the threshold within the model, integrates the filtered data, and provides decision support based on the integrated information.

6. The method of deterministic network modeling based on data-driven and knowledge graph according to claim 1, characterized in that: The method based on artificial intelligence and deep learning is used to identify and remove outliers from the preprocessed data, including the coarse recall stage and the fine recall stage, specifically: In the rough recall stage, Kalman filtering, kurtosis, skewness, exponential moving average and STL time series data decomposition strategies are used to extract features from the preprocessed data. The strategy of integrating two machine learning methods, Isolation Forest and LODA, is used to quickly and preliminarily screen out normal values ​​and possible outliers based on the extracted features as suspicious data. In the fine recall stage, based on the normal data filtered by the coarse recall module, normalization is performed and a training data set is constructed with a fixed sliding window length; A variational autoencoder model based on semi-implicit distribution is trained based on the training data set. After the training is completed, the variational autoencoder model calculates the reconstruction probability value of the observed data at each moment in real time. The smaller the reconstruction probability value, the greater the probability of abnormality. A threshold is set for the reconstruction probability value for screening: if the reconstruction probability of a sample is lower than the threshold, it means that the input data does not conform to the normal data distribution learned by the model and can be classified as an outlier. The identified outliers are removed to obtain the cleaned data.

7. The method of deterministic network modeling based on data-driven and knowledge graph according to claim 6, characterized in that: When setting the threshold for the reconstruction probability, an automatic threshold setting method based on extreme value theory is used, that is, the Pareto distribution function is used to model the distribution obeyed by extreme values, and the maximum likelihood is used to estimate the parameters in the distribution for the reconstruction probability, and finally the corresponding threshold is quickly calculated.

8. The method of deterministic network modeling based on data-driven and knowledge graph according to claim 1, characterized in that: Using the cleaned high-quality data, we build an adaptive model threshold based on a statistical algorithm. Specifically: For the real-time data of the system that has been cleared of outliers, find the median X of the data in the window of length W with this point as the end of the data sequence for each data point. m ; Then calculate the absolute value sequence of the difference between the data in the window and the median {|X i -X m |} i＝1,...W , the median of this absolute value sequence is the median absolute deviation, that is, MAD=Median({|X i -X m |} i＝1,...W ) If the observed data exceeds the absolute deviation range of the three medians above and below the median of the sequence, that is: <h2 style=";text-align:left;direction:ltr">X<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> +3MAD <X <h2 style=";text-align:left;direction:ltr"> obm or: X m -3MAD>X obm It will be marked as abnormal and an alarm will be issued.

Images