A Risk Monitoring Method Based on Intelligent Association of Multi-Source Data and Global Situation

Abstract

This invention belongs to the field of network security technology, and particularly relates to a risk monitoring method based on intelligent correlation and global situational awareness using multi-source data. The method includes: acquiring a multi-source heterogeneous dataset; preprocessing the multi-source heterogeneous dataset to obtain preprocessed multi-source data; acquiring an attack behavior correlation graph based on the multi-source data, using a graph neural network to perform anomaly detection on the correlation graph, and acquiring explicit attack links and potential attack links; acquiring a dynamic network security situational awareness graph based on the explicit and potential attack links; generating a risk assessment report based on the dynamic network security situational awareness graph, triggering corresponding security policies, and performing risk monitoring based on the security policies. This invention significantly improves the accuracy and response speed of threat detection, enhances the global network security situational awareness capability, and provides an intelligent solution for security protection in complex network environments.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, and in particular relates to a risk monitoring method based on intelligent correlation and global situational awareness of multi-source data. Background Technology

[0002] With the rapid development of internet technology, cyberattack methods have become increasingly diverse and complex, making traditional cybersecurity protection methods inadequate for the modern threat environment. Current security systems typically rely on a single data source for threat detection, lacking the ability to comprehensively analyze multi-source heterogeneous data. This limitation of a single data source makes the system inadequate when facing complex attack chains, especially in scenarios such as advanced persistent threats (APTs), zero-day exploits, and large-scale distributed attacks.

[0003] While existing network security systems (such as Intrusion Detection Systems (IDS) and Security Information and Event Management Systems (SIEM) have improved security capabilities to some extent, they still have many shortcomings. For example, traditional security systems generate a large number of alerts, but due to the lack of contextual analysis, it is difficult for operations and maintenance personnel to quickly locate critical threats. Furthermore, existing systems mainly rely on rule matching or simple statistical analysis, which proves inadequate when facing complex attack chains. In terms of real-time performance, traditional systems perform poorly when processing massive amounts of real-time data, failing to meet the demands of real-time threat detection in modern network environments. At the same time, traditional systems typically only focus on localized threats, lacking dynamic monitoring and visualization of the overall network security status.

[0004] In recent years, the application of artificial intelligence and big data technologies in the field of cybersecurity has gradually emerged, providing new ideas for solving the aforementioned problems. For example, machine learning and deep learning technologies can improve the accuracy of threat detection by training models to identify abnormal behavior. However, these technologies still face problems such as poor data quality and insufficient model generalization ability in practical applications. Graph computing and knowledge graph technologies can uncover hidden attack chains by constructing attack behavior association graphs, but existing methods have high computational complexity when processing large-scale heterogeneous data and limited ability to process unstructured data. Streaming computing and distributed frameworks can improve the timeliness of systems by processing massive amounts of data in real time, but existing frameworks still need improvement in data fusion and dynamic analysis.

[0005] In modern cyber environments, single threat detection methods are no longer sufficient to guarantee network security. Decision-makers need to understand the overall security status of the network in order to take timely action. Global situational awareness technology, by fusing real-time and historical data, generates dynamic network security situation maps, helping decision-makers grasp the overall security situation and predict future trends. However, existing technologies still have significant room for improvement in terms of the real-time performance, accuracy, and visualization capabilities of situational awareness.

[0006] With the increasing frequency and complexity of cyberattacks, manual intervention is no longer sufficient to meet the demands for rapid response. Automated response mechanisms, by combining Bayesian networks and SOAR (Security Orchestration, Automation, and Response) technology, can automatically generate risk assessment reports and trigger security policies, significantly improving incident response efficiency. However, existing automated systems still need improvement in terms of the flexibility and effectiveness of policy orchestration.

[0007] In summary, existing cybersecurity technologies have significant shortcomings in areas such as multi-source data integration, intelligent analysis, global situational awareness, and automated response. To address these challenges, an intelligent analysis method capable of integrating heterogeneous multi-source data is urgently needed. Summary of the Invention

[0008] To address the aforementioned technical issues, this invention proposes a risk monitoring method based on intelligent correlation and global situational awareness using multi-source data. This method significantly improves the ability to reconstruct complex attack scenarios and the accuracy of prediction by introducing causal reasoning and an improved graph neural network (GNN), and supports high-concurrency data processing and dynamic strategy orchestration.

[0009] To achieve the above objectives, this invention provides a risk monitoring method based on intelligent correlation and global situational awareness using multi-source data, comprising:

[0010] Obtain multi-source heterogeneous datasets;

[0011] The multi-source heterogeneous dataset is preprocessed to obtain preprocessed multi-source data;

[0012] Based on the multi-source data, an attack behavior correlation graph is obtained, and a graph neural network is used to perform anomaly detection on the correlation graph to obtain explicit attack links and potential attack links.

[0013] Based on explicit and potential attack chains, obtain a dynamic map of network security situation;

[0014] A risk assessment report is generated based on the aforementioned network security situation dynamic map, and corresponding security policies are triggered. Risk monitoring is then performed based on these security policies.

[0015] Optionally, obtaining the multi-source heterogeneous dataset includes:

[0016] Log information, status information, local flash data, and threat intelligence data from network devices, security devices, and servers are collected through multiple protocols to form an initial multi-source heterogeneous dataset;

[0017] The initial multi-source heterogeneous dataset is partitioned using a partitioning strategy to obtain the multi-source heterogeneous dataset.

[0018] Optionally, preprocessing the multi-source heterogeneous dataset to obtain preprocessed multi-source data includes:

[0019] The multi-source heterogeneous dataset is cleaned, deduplicated, and format-standardized to obtain initially processed multi-source data, which includes structured and unstructured data.

[0020] The structured data is parsed using regularization to obtain the target data, and the target data is formatted to obtain the processed structured data.

[0021] The unstructured data is converted into vectors using word embedding to obtain the processed unstructured data;

[0022] The Isolation Forest method is used to remove outlier data points from both the processed structured and unstructured data to obtain preprocessed multi-source data.

[0023] Optionally, the structured data is parsed using regularization to obtain target data, and the target data is formatted to obtain processed structured data, including:

[0024] ;

[0025] ;

[0026] in, (*) is the regular expression parsing function. This is the original log. For the parsed structured data, For predefined standardized templates, For the processed structured data, (*) is a data format unification function.

[0027] Optionally, converting the unstructured data into vectors using word embedding includes:

[0028] ;

[0029] in, This is unstructured log text. This is a vector representation of unstructured log text. (*) is the word embedding function.

[0030] Optionally, obtaining the attack behavior correlation graph based on the multi-source data includes:

[0031] Based on the graph computing model, entities in the multi-source data are modeled as nodes, event relationships are modeled as edges, and edge weights are calculated to obtain the attack behavior association graph.

[0032] Optionally, a graph neural network can be used to perform anomaly detection on the association graph to obtain potential attack links, including:

[0033] The graph neural network is used to detect anomalies in the associated graph and obtain abnormal nodes.

[0034] The probability of attack paths in the paths formed by the abnormal nodes is calculated using a Bayesian model to obtain explicit attack links.

[0035] Frequent subgraph mining is performed on the association graph to obtain potential attack links.

[0036] Optionally, the method for obtaining a dynamic network security situation map based on the explicit and potential attack chains is as follows:

[0037] ;

[0038] in, For the merged data, This is real-time data at the current moment. For historical data, This refers to the weighting coefficient. Optionally, generating a risk assessment report based on the aforementioned network security situation dynamic map and triggering corresponding security policies includes:

[0039] Based on the aforementioned network security situation dynamic diagram, the risk probability is calculated using a Bayesian model.

[0040] An assessment report is generated based on the stated risk probabilities;

[0041] Based on the assessment report, the corresponding security policy is triggered.

[0042] This invention also provides a risk monitoring system based on intelligent correlation and global situational awareness using multi-source data, including: a data acquisition module, a data preprocessing module, an intelligent correlation analysis module, a global situational awareness module, and a coordinated response module;

[0043] The data acquisition module is used to collect log information, status information and external threat intelligence data from network devices, security devices and servers through multiple protocols to form a multi-source heterogeneous dataset.

[0044] The data preprocessing module is used to process the multi-source heterogeneous dataset to obtain preprocessed multi-source data;

[0045] The intelligent correlation analysis module is used to obtain an attack behavior correlation graph based on the multi-source data, use a graph neural network to perform anomaly detection on the correlation graph, and obtain explicit attack links and potential attack links.

[0046] The global situation awareness module is used to obtain a dynamic map of network security situation based on explicit attack links and potential attack links;

[0047] The coordinated response module is used to generate a risk assessment report based on the network security situation dynamic map, trigger corresponding security policies, and perform risk monitoring based on the security policies.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] This invention integrates multi-source heterogeneous data and utilizes machine learning algorithms (such as deep learning and graph neural networks) for comprehensive analysis, significantly reducing false positive and false negative rates and accurately identifying hidden patterns in complex attack chains. It employs distributed computing frameworks (such as Spark and Flink) and streaming computing technology to achieve real-time threat detection and situational updates, processing massive amounts of data in milliseconds to greatly improve system timeliness. Furthermore, it combines automated orchestration technology to automatically generate risk assessment reports and trigger emergency response strategies, shortening threat handling time. In addition, it integrates real-time and historical data to generate a dynamic network security situation map, providing a global perspective to comprehensively understand the security situation, quickly locate high-risk areas, and support long-term strategy formulation. It also combines natural language processing (NLP), knowledge graphs, and causal reasoning techniques to deeply mine unstructured data (such as social media discussions), improving... It provides early warnings of potential threats and reveals attack logic, enhancing its intelligence level. Through efficient integration and intelligent analysis of multi-source data, it significantly reduces invalid alarms, lowers the operational burden, and its automated response mechanism further reduces manual intervention, improving operational efficiency. It is widely applicable to enterprise network security, cloud security, industrial internet, smart cities, and other fields. It can monitor abnormal behavior in real time, predict faults, and ensure the security of critical infrastructure. It not only solves key technical problems in the current network security field but also provides guidance for future security hardening through causal reasoning and other technologies. It has high scalability to drive continuous innovation. In terms of economic benefits, it improves threat detection and response capabilities, reducing economic losses for enterprises. In terms of social benefits, it enhances the security of critical infrastructure and helps build a safer cyberspace. It is an important milestone in addressing the complex security challenges of the modern network environment and promoting the development of network security technology. Attached Figure Description

[0050] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a flowchart of the risk monitoring method based on intelligent association and global situational awareness using multi-source data, according to an embodiment of the present invention.

[0052] Figure 2 This is a flowchart of the multi-source data acquisition and preprocessing process according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram illustrating the working principle of the intelligent association analysis module in an embodiment of the present invention;

[0054] Figure 4 This is a flowchart of the strategy orchestration process of the linkage handling module in an embodiment of the present invention. Detailed Implementation

[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0057] Multi-source data integration not only improves the comprehensiveness and accuracy of data but also provides richer contextual information for subsequent intelligent analysis. Internal data includes network device logs, security device alarms, and server operating status; external data includes malicious IP address databases and malicious domain name lists provided by threat intelligence platforms; unstructured data covers user behavior descriptions and security-related discussions on social media. Through comprehensive analysis of this data, attack scenarios can be more fully reconstructed, and potential threats can be identified.

[0058] This embodiment proposes a risk monitoring method based on intelligent correlation and global situational awareness using multi-source data, such as... Figure 1 As shown, the specific steps include:

[0059] The multi-source heterogeneous dataset is preprocessed to obtain preprocessed multi-source data;

[0060] Based on the multi-source data, an attack behavior correlation graph is obtained, and a graph neural network is used to perform anomaly detection on the correlation graph to obtain explicit attack links and potential attack links.

[0061] Based on explicit and potential attack chains, obtain a dynamic map of network security situation;

[0062] A risk assessment report is generated based on the aforementioned network security situation dynamic map, and corresponding security policies are triggered. Risk monitoring is then performed based on these security policies.

[0063] Specifically, this embodiment integrates multi-source heterogeneous data (such as network device logs, security device alarms, and external threat intelligence), combines natural language processing and knowledge graph technologies to perform semantic extraction on unstructured data, and constructs a unified knowledge representation model. Based on this, advanced technologies such as graph neural networks, Bayesian inference, and frequent subgraph mining are used to uncover hidden attack chains and anomaly patterns. Furthermore, real-time and historical data are fused using big data streaming computing technology to generate a dynamic network security situation map. Simultaneously, visualization technology is used to display the distribution, trends, and impact range of attack events, and automated orchestration technology is used to trigger security policies, achieving efficient emergency response. This invention significantly improves the accuracy and response speed of threat detection, enhances global network security situation awareness capabilities, and provides an intelligent solution for security protection in complex network environments.

[0064] Furthermore, obtaining the multi-source heterogeneous dataset includes:

[0065] Log information, status information, local flash data, and threat intelligence data from network devices, security devices, and servers are collected through multiple protocols to form an initial multi-source heterogeneous dataset;

[0066] The initial multi-source heterogeneous dataset is partitioned using a partitioning strategy to obtain the multi-source heterogeneous dataset.

[0067] Specifically, the data acquisition module actively collects log information, status information, and local flash memory data from network devices (such as routers and switches), security devices (such as firewalls, IDS / IPS), and servers through various protocols (such as Telnet, SNMP, Syslog, NetFlow, etc.), while simultaneously obtaining the latest threat intelligence data (such as malicious IP address databases, malicious domain name lists, etc.) from external threat intelligence platforms, forming a multi-source heterogeneous data set. The data acquisition frequency employs a periodic polling mechanism. It can be dynamically adjusted according to needs:

[0068] ;

[0069] Where t represents the data collection interval in seconds. To optimize the storage and query performance of massive amounts of data, distributed storage technologies (such as HDFS and Cassandra) are used to store the data, ensuring high reliability and high throughput. The data partitioning strategy is as follows:

[0070] ;

[0071] in, It is a unique identifier used to determine which partition the data should be stored in. This is the identifier for the data source node. For timestamps, (*) represents the hash function. Furthermore, to reduce bandwidth consumption, the data acquisition module supports data compression and incremental transmission.

[0072] Furthermore, the multi-source heterogeneous dataset is preprocessed to obtain preprocessed multi-source data, including:

[0073] The multi-source heterogeneous dataset is cleaned, deduplicated, and format-standardized to obtain initially processed multi-source data, which includes structured and unstructured data.

[0074] The structured data is parsed using regularization to obtain the target data, and the target data is formatted to obtain the processed structured data.

[0075] The unstructured data is converted into vectors using word embedding to obtain the processed unstructured data;

[0076] The Isolation Forest method is used to remove outlier data points from both the processed structured and unstructured data to obtain preprocessed multi-source data.

[0077] Specifically, data preprocessing involves cleaning, deduplicating, and standardizing the format of the collected multi-source data to ensure data consistency and usability. Regular expressions are used to parse the raw logs and extract key fields (such as timestamps, source IPs, destination IPs, and protocol types). The formula is as follows:

[0078] ;

[0079] in, (*) is the regular expression parsing function. This is the original log. This is the parsed structured data. Data in different formats is converted into a unified JSON format:

[0080] ;

[0081] in, For the processed structured data, For predefined standardized templates, (*) is a standardized function.

[0082] For unstructured log text, word embedding techniques (such as Word2Vec and BERT) are used to convert it into a vector representation:

[0083] ;

[0084] in, Let V be the log text, and let V be the vector representation of the unstructured log text. This is the word embedding function. To detect outliers, the Isolation Forest algorithm is used, with the following formula:

[0085] ;

[0086] in, For abnormal scores, (*) represents the Isolation Forest algorithm. For the input dataset, This refers to the abnormal proportion parameter. Furthermore, the data preprocessing module also supports dimensionality reduction based on feature engineering to improve the efficiency of subsequent analysis. Specifically, It can be processed structured data It can also be processed unstructured data.

[0087] Furthermore, obtaining the attack behavior correlation graph based on the multi-source data includes:

[0088] Based on the graph computing model, entities in the multi-source data are modeled as nodes, event relationships are modeled as edges, and edge weights are calculated to obtain the attack behavior association graph.

[0089] Furthermore, graph neural networks are used to perform anomaly detection on the association graph to obtain explicit and potential attack links, including:

[0090] The graph neural network is used to detect anomalies in the associated graph and obtain abnormal nodes.

[0091] The probability of attack paths in the paths formed by the abnormal nodes is calculated using a Bayesian model to obtain explicit attack links.

[0092] Frequent subgraph mining is performed on the association graph to obtain potential attack links.

[0093] Specifically, the intelligent association analysis module, based on graph computing frameworks (such as Neo4j and GraphX), models entities (such as IP addresses, user accounts, and device IDs) in multi-source data as nodes and event relationships as edges, constructing an attack behavior association graph. Edge weights are also included. The calculation formula is as follows:

[0094] ;

[0095] in, For nodes and nodes similarity, To adjust the parameters, a graph neural network (GNN) is used to detect anomalies in the graph, and node anomaly scores are calculated. :

[0096] ;

[0097] in, For nodes The neighborhood group, For nodes eigenvectors.

[0098] To infer the attack path, a Bayesian inference model is used:

[0099] ;

[0100] in, Given the evidence, the probability of the attack path is... For the prior path probability, For the probability of evidence, Possible attack paths, For observed evidence, frequent subgraph mining algorithms (such as gSpan) are used to mine hidden attack links in the attack behavior correlation graph:

[0101] ;

[0102] in, For a set of frequent subgraphs, (*) represents the frequent subgraph mining algorithm. The input graph is the attack behavior association graph. The minimum support threshold is used. To reconstruct the attack scenario, causal inference is employed:

[0103] ;

[0104] in, For cause-and-effect diagrams, (*) represents the causal inference function. Using prior knowledge constraints, such as known attack patterns and security rules, causal reasoning analyzes the causal relationships in the graph to infer the temporal sequence and logical dependencies of attack behaviors. Combined with the constraints, it can more accurately reconstruct the attack scenario and help security analysts understand the attacker's intent and methods.

[0105] Furthermore, based on explicit and potential attack chains, the method for obtaining a dynamic network security situation map is as follows:

[0106] Global situational awareness, based on big data streaming computing technologies (such as Apache Flink and Kafka Streams), fuses and analyzes real-time and historical data to generate a dynamic network security situation map. A sliding window technique is used to fuse real-time and historical data, where historical data refers to data collected in the past and real-time data refers to data collected at the current moment. The collected data is multi-source and heterogeneous.

[0107] ;

[0108] in, For the merged data, This is real-time data at the current moment. For historical data, These are the weighting coefficients.

[0109] Calculate the network security posture score:

[0110] ;

[0111] in, As the indicator weight, For indicator values, The situation score is given by n, where n is the number of indicators.

[0112] Use time series analysis (such as ARIMA models, LSTM) to predict the evolution trend of the situation:

[0113] ;

[0114] in, This represents the current situation value. For regression coefficients, For noise terms, This represents the situation value at the previous moment. These are the situation values ​​for the first two moments.

[0115] Use heatmaps, topology maps, and other methods to illustrate the distribution and propagation paths of attack events:

[0116] ;

[0117] in, For heatmaps, (*) is the rendering function. For the spatial distribution of attack events, This refers to color mapping rules.

[0118] Specifically, the collected data includes situational awareness data, such as the number of malicious IP accesses, the frequency of attack behaviors, changes in network traffic, and the proportion of abnormal traffic. .

[0119] exist middle This is real-time data at the current moment. This is historical data (aggregated results within a sliding window). This is a weighting coefficient used to control the contribution ratio of real-time data and historical data.

[0120] From For the fused situational awareness data at time t, a weighted situational score at time t is calculated.

[0121] It can be modified to ,in, Assess the current situation. Assess the situation at the previous moment. The situation score for the next moment is calculated based on the situation scores of the current and past moments, and the situation score for the next moment is predicted.

[0122] Using heatmaps, topology maps, and other similar formats to display the distribution and propagation paths of attack events is unrelated to the preceding content; it is an abstract representation used to describe how to generate heatmaps through data rendering.

[0123] Heatmap: The final generated heatmap is a visualization used to visually show the spatial distribution of attack events.

[0124] Render: The rendering function represents the process of converting the input data (AttackDistribution) and color mapping rules (ColorMapping) into a heatmap.

[0125] AttackDistribution: Spatial distribution data of attack events, usually a two-dimensional or three-dimensional matrix, which records the number or intensity of attack events at different locations (such as network nodes, IP addresses, geographical locations, etc.).

[0126] ColorMapping: Color mapping rules define how to assign colors based on the intensity (such as quantity, frequency, etc.) of attack events.

[0127] Furthermore, a risk assessment report is generated based on the aforementioned network security situation dynamic diagram, and corresponding security policies are triggered, including:

[0128] Based on the aforementioned network security situation dynamic diagram, the risk probability is calculated using a Bayesian model.

[0129] An assessment report is generated based on the stated risk probabilities;

[0130] Based on the assessment report, the corresponding security policy is triggered.

[0131] Specifically, the coordinated response module automatically generates a risk assessment report based on the global situational awareness results and triggers corresponding security policies (such as blocking malicious IPs and isolating infected devices). Bayesian networks are used to calculate the risk probability. :

[0132] ;

[0133] in, For condition variables, Its parent node.

[0134] Trigger predefined security policies based on risk assessment results:

[0135] ;

[0136] in, As a risk threshold, Assess the current risk level. For specific response actions (such as blocking malicious IPs, isolating devices, etc.). (*) is the trigger function, which determines whether to take action based on RiskScore and Threshold.

[0137] Implement policy orchestration using SOAR (Secure Orchestration, Automation, and Response) technology:

[0138] ;

[0139] in, A set of executable actions. For contextual information, The final automated workflow is used to execute a series of security policies. (*) is an orchestration function used to combine ActionSet and Context to generate an automated workflow.

[0140] To verify the effectiveness of the strategy, an emergency response drill was conducted using a simulated attack scenario (Red Team vs. Blue Team):

[0141] ;

[0142] in, For simulation results, To reflect the true situation, The effectiveness of the strategy refers to the actual effect of the security strategy in simulated attack scenarios. (*) is the evaluation function used to compare the simulation results with the real situation and calculate the effectiveness of the strategy.

[0143] This embodiment also provides a risk monitoring system based on intelligent correlation and global situational awareness using multi-source data, including: a data acquisition module, a data preprocessing module, an intelligent correlation analysis module, a global situational awareness module, and a coordinated response module;

[0144] The data acquisition module is used to collect log information, status information and external threat intelligence data from network devices, security devices and servers through multiple protocols to form a multi-source heterogeneous dataset.

[0145] The data preprocessing module is used to process the multi-source heterogeneous dataset to obtain preprocessed multi-source data;

[0146] The intelligent correlation analysis module is used to obtain an attack behavior correlation graph based on the multi-source data, use a graph neural network to perform anomaly detection on the correlation graph, and obtain explicit attack links and potential attack links.

[0147] The global situation awareness module is used to obtain a dynamic map of network security situation based on explicit attack links and potential attack links;

[0148] The coordinated response module is used to generate a risk assessment report based on the network security situation dynamic map, trigger corresponding security policies, and perform risk monitoring based on the security policies.

[0149] The following is a detailed description of this embodiment with reference to the accompanying drawings:

[0150] like Figure 2As shown, data acquisition is the foundation of this embodiment, responsible for obtaining multi-source heterogeneous data from network devices, security devices, servers, and external threat intelligence platforms. In actual deployment, data acquisition agents first need to be installed on key network nodes (such as routers, switches, firewalls, and servers). These agents actively collect device status information and log information through various protocols (such as SNMP, Syslog, NetFlow, and API interfaces). For example, they collect router traffic statistics via the SNMP protocol, firewall alarm logs via the Syslog protocol, and the latest malicious IP address database and domain name list from external threat intelligence platforms via API interfaces. The collection frequency can be dynamically adjusted according to actual conditions; for example, a shorter collection interval (e.g., once per second) can be set during high-traffic periods, while the collection interval can be extended (e.g., once per minute) during low-traffic periods.

[0151] like Figure 2 As shown, the main task of data preprocessing is to clean, deduplicate, and standardize the format of the collected multi-source data to ensure data consistency and usability. For structured log data, regular expressions are used to extract key fields, such as timestamp, source IP, destination IP, and protocol type. Taking Syslog format logs as an example, the following regular expression is designed:

[0152] .

[0153] After parsing, the raw logs are converted into a unified JSON format for easier subsequent analysis. For unstructured log text (such as free text descriptions), natural language processing techniques are used for semantic extraction. For example, the BERT model is used to generate vector representations of the log text, transforming a log entry "User admin failed to log in" into a 1024-dimensional vector. Furthermore, to detect outliers, the Isolation Forest algorithm is applied, assuming an anomaly rate of 1%, constructing 100 isolation trees for anomaly detection.

[0154] like Figure 3 As shown, intelligent association analysis, based on graph computing frameworks (such as Neo4j and GraphX), models entities in multi-source data as nodes and event relationships as edges, constructing an association graph of attack behaviors. For example, one node can represent the IP address "192.168.1.1", another node can represent the user "admin", and the edge between them represents the login behavior. The edge weight is calculated using the formula:

[0155] ;

[0156] Building upon this foundation, graph neural networks (GNNs) are used to detect anomalies in the graph, identifying potential attack paths and anomalous patterns. For instance, by analyzing frequently occurring subgraphs, hidden attack paths are uncovered, and the gSpan algorithm is employed with a minimum support threshold of 0.05 to identify frequent subgraphs.

[0157] Global situational awareness, based on big data streaming computing technologies (such as Apache Flink and Kafka Streams), fuses and analyzes real-time and historical data to generate a dynamic network security situation map. A sliding window technique is used to fuse real-time and historical data; for example, setting the sliding window size to 5 minutes and the step size to 1 minute, the fusion formula is as follows:

[0158] ;

[0159] Define multiple security metrics (such as attack frequency, number of affected assets, and threat level) and assign different weights to calculate the overall situational awareness score.

[0160] To further predict the evolution of the situation, an LSTM model is used to predict situational changes within the next 30 minutes. Input features include historical situational scores from the past hour. Finally, the distribution, propagation paths, and impact range of attack events are displayed through heatmaps, topology maps, and other formats, providing decision-makers with an intuitive global view.

[0161] like Figure 4 As shown, the coordinated response automatically generates a risk assessment report based on the global situational awareness results and triggers corresponding security policies. Risk probabilities are calculated using Bayesian networks, for example, given conditional variables (such as attack type, scope of impact, and duration), to infer an overall risk score. When the risk score exceeds a preset threshold (such as 80 points), security policies are automatically triggered, such as blocking malicious IP addresses and isolating infected devices. Policy orchestration is implemented using SOAR (Security Orchestration, Automation, and Response) technology, defining the following action sequence:

[0162] 1. Block malicious IP addresses;

[0163] 2. Send alarm notifications to maintenance personnel;

[0164] 3. Launch a vulnerability scanning tool to check for affected devices.

[0165] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A risk monitoring method based on intelligent correlation and global situational awareness using multi-source data, characterized in that, include: Obtain multi-source heterogeneous datasets; Obtaining the multi-source heterogeneous dataset includes: Log information, status information, local flash data, and threat intelligence data from network devices, security devices, and servers are collected through multiple protocols to form an initial multi-source heterogeneous dataset; The initial multi-source heterogeneous dataset is partitioned using a partitioning strategy to obtain the multi-source heterogeneous dataset. The multi-source heterogeneous dataset is preprocessed to obtain preprocessed multi-source data; Preprocessing the multi-source heterogeneous dataset to obtain preprocessed multi-source data includes: The multi-source heterogeneous dataset is cleaned, deduplicated, and format-standardized to obtain initially processed multi-source data, which includes structured and unstructured data. The structured data is parsed using regularization to obtain the target data, and the target data is formatted to obtain the processed structured data. The unstructured data is converted into vectors using word embedding to obtain the processed unstructured data; The Isolation Forest method is used to remove outlier data points from both the processed structured and unstructured data to obtain preprocessed multi-source data. The structured data is parsed using regularization to obtain the target data, and the target data is then formatted to obtain the processed structured data, which includes: ； ； in, (*) is the regular expression parsing function. This is the original log. For the parsed structured data, For predefined standardized templates, For the processed structured data, (*) is a data format unification function; Converting the unstructured data into vectors using word embedding includes: ； in, This is unstructured log text. This is a vector representation of unstructured log text. (*) represents the word embedding function; Based on the multi-source data, an attack behavior correlation graph is obtained, and a graph neural network is used to perform anomaly detection on the correlation graph to obtain explicit attack links and potential attack links. Obtaining the attack behavior correlation graph based on the multi-source data includes: Based on the graph computing model, entities in the multi-source data are modeled as nodes, event relationships are modeled as edges, and edge weights are calculated to obtain the attack behavior association graph. Using graph neural networks to perform anomaly detection on the association graph and identify potential attack chains include: The graph neural network is used to detect anomalies in the associated graph and obtain abnormal nodes. The probability of attack paths in the paths formed by the abnormal nodes is calculated using a Bayesian model to obtain explicit attack links. Frequent subgraph mining is performed on the association graph to obtain potential attack links; Based on the explicit and potential attack chains, a dynamic network security situation map is obtained; The method for obtaining a dynamic network security situation map based on the explicit and potential attack chains is as follows: ； in, For the merged data, This is real-time data at the current moment. For historical data, These are the weighting coefficients; A risk assessment report is generated based on the aforementioned network security situation dynamic map, and corresponding security policies are triggered. Risk monitoring is then performed based on the aforementioned security policies. A risk assessment report is generated based on the aforementioned network security situation dynamic diagram, and corresponding security policies are triggered, including: Based on the aforementioned network security situation dynamic diagram, the risk probability is calculated using a Bayesian model. An assessment report is generated based on the stated risk probabilities; Based on the assessment report, the corresponding security policy is triggered.

2. The risk monitoring system based on intelligent correlation and global situational awareness using multi-source data, implemented according to the method described in claim 1, is characterized in that... include: The system includes a data acquisition module, a data preprocessing module, an intelligent correlation analysis module, a global situational awareness module, and a coordinated response module. The data acquisition module is used to collect log information, status information and external threat intelligence data from network devices, security devices and servers through multiple protocols to form a multi-source heterogeneous dataset. The data preprocessing module is used to process the multi-source heterogeneous dataset to obtain preprocessed multi-source data; The intelligent correlation analysis module is used to obtain an attack behavior correlation graph based on the multi-source data, use a graph neural network to perform anomaly detection on the correlation graph, and obtain explicit attack links and potential attack links. The global situation awareness module is used to obtain a dynamic map of network security situation based on explicit attack links and potential attack links; The coordinated response module is used to generate a risk assessment report based on the network security situation dynamic map, trigger corresponding security policies, and perform risk monitoring based on the security policies.

Images