A big data-based network security threat assessment system

Abstract

This invention discloses a big data-based network security threat assessment system, comprising: a multi-source data acquisition module, a big data processing platform module, an intelligent threat assessment engine module, and a service and response module. This invention relates to the field of network security technology. By integrating multi-dimensional data such as network traffic, terminal behavior, and asset status, it breaks down the data silos of traditional security devices, constructs a comprehensive security situation awareness view, significantly reduces detection blind spots, and adopts a batch-stream integrated architecture to collaboratively analyze real-time data and historical baselines. Combined with supervised and unsupervised machine learning models for fusion detection, it can effectively identify known attacks and unknown anomalies, reducing false alarms while improving the detection rate of advanced threats. It innovatively quantifies and integrates multiple factors such as threat signals, asset criticality, and vulnerability exploitability, outputting dynamic and interpretable risk scores, transforming security response from coarse alarm management to precise risk prioritization management.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a network security threat assessment system based on big data. Background Technology

[0002] Cybersecurity threat assessment refers to the process of identifying, quantifying, and issuing early warnings about potential attack risks to network systems by collecting and analyzing various types of security data. It is a core basis for decision-making and response in modern security operations centers. With the increasing complexity of cyberattack methods and the proliferation of advanced persistent threats, data in the network environment is characterized by massive volume, heterogeneity, and high speed.

[0003] Currently, mainstream cybersecurity threat assessment solutions primarily rely on Security Information and Event Management (SIEM) systems. These systems use predefined rules or signatures to perform correlation analysis on logs from single-point devices such as firewalls and intrusion detection systems. However, these traditional solutions have significant limitations in practical applications: First, their analytical capabilities heavily depend on rule bases based on known attack patterns, making it difficult to effectively identify new or variant unknown threats, leading to missed detections. Second, data from various security devices is isolated, lacking deep integration and contextual correlation analysis of multi-dimensional information such as user behavior, network traffic, and asset status, making it difficult to construct a complete attack chain view. Third, alert outputs are mostly presented as low, medium, and high risk levels, lacking dynamic and quantitative risk scoring that combines specific asset value, vulnerability status, and business context, making it difficult for security teams to determine handling priorities when faced with massive alerts, resulting in low response efficiency. Finally, the system's detection models and rules often require manual maintenance and updates by security experts, failing to adapt to the evolution and changes in normal network behavior, resulting in high false positive rates and huge maintenance costs.

[0004] Therefore, the cybersecurity field urgently needs a system that can integrate multi-source big data, has adaptive learning capabilities, and can output accurate, quantifiable, and interpretable dynamic threat assessment results to solve the problems of perception blind spots, rough assessments, and delayed responses in existing technologies. Summary of the Invention

[0005] (a) Technical problems to be solved The purpose of this invention is to provide a big data-based network security threat assessment system, which aims to solve the problems of perception blind spots, rough assessment, and delayed response in existing technologies.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: Firstly, embodiments of the present invention provide a big data-based network security threat assessment system, comprising: Multi-source data acquisition module, big data processing platform module, intelligent threat assessment engine module, and service and response module; The multi-source data acquisition module is used to collect network traffic data, terminal and host data, security device logs, external threat intelligence, and asset and vulnerability data. The big data processing platform module adopts a batch-stream integrated architecture, including a real-time stream processing pipeline for streaming processing and correlation of real-time data, and an offline batch processing pipeline for batch processing of historical data to establish behavioral baselines and train models. The intelligent threat assessment engine module is connected to the big data processing platform module, including a feature engineering submodule for generating real-time and historical feature sets, a multi-model fusion detection submodule that uses a fusion of unsupervised models, supervised models and threat intelligence rules to output the original threat signal, and a dynamic risk assessment submodule for fusing the original threat signal, asset value and vulnerability information to calculate the dynamic risk score. The service and response module is connected to the intelligent threat assessment engine module and is used to visualize risks, generate handling work orders, or trigger automated response actions based on the dynamic risk score.

[0007] This invention proposes a big data-based network security threat assessment system. When the system is running, a multi-source data acquisition module comprehensively gathers multi-dimensional data, including network traffic, terminal behavior, security logs, external intelligence, and asset vulnerabilities, solving the problem of isolated data sources. This data is input into a big data processing platform module employing a batch-stream integrated architecture. The real-time stream processing pipeline performs instant correlation and feature extraction on the data to support second-level threat discovery, while the offline batch processing pipeline deeply mines massive amounts of historical data to establish dynamic behavioral baselines and train detection models. The intelligent threat assessment engine module, based on real-time and historical features, uses a multi-model fusion strategy for comprehensive analysis. It not only identifies known attack patterns but also detects unknown abnormal behaviors. Furthermore, it combines asset criticality and vulnerability status to quantify the original threat signal into a dynamic risk score associated with a specific asset. Finally, the service and response module uses this dynamic risk score to achieve a closed loop from global risk visibility and priority work order generation to automated response. This solution effectively addresses the shortcomings of traditional solutions in terms of threat discovery comprehensiveness, assessment accuracy, and response efficiency through data fusion, real-time analysis, intelligent assessment, and coordinated response.

[0008] In some embodiments, the real-time stream processing pipeline is implemented based on a stream computing engine and is used to calculate at least one real-time feature among the following within a sliding time window: destination port entropy of the source IP, access geographic dispersion, frequency of matching with malicious intelligence, and abnormality of host process creation.

[0009] By calculating these dynamic characteristics in real time, the system can instantly capture transient abnormal behaviors of networks and hosts, providing key signals for discovering threats such as zero-day attacks or rapid lateral movement, and significantly improving the timeliness of threat perception.

[0010] In some embodiments, the offline batch processing pipeline is configured to process historical communication data based on graph algorithms, construct a relationship graph between hosts and hosts, and between users and resources, and identify abnormal subgraphs that deviate from the historical normal communication pattern based on the relationship graph.

[0011] By mining deep relationships between entities through graph computing, it is possible to effectively discover covert lateral movement or internal infiltration behaviors that exploit normal trust relationships, making up for the shortcomings of single-point event detection and enhancing the detection capability against advanced persistent threats.

[0012] In some embodiments, the unsupervised learning model in the multi-model fusion detection submodule includes an isolated forest or an autoencoder for detecting traffic statistics anomalies or deviations from normal patterns; the supervised learning model includes a gradient boosting tree model for detecting known attack patterns.

[0013] By integrating supervised and unsupervised models, the system combines high-precision detection of known threats with the ability to discover unknown abnormal behaviors, achieving complementary detection coverage and effectively reducing false negatives and false positives caused by a single model.

[0014] In some embodiments, the dynamic risk assessment submodule employs a weighted scoring card algorithm, wherein the dynamic risk score is calculated by weighting a threat severity factor, an asset value factor, an attack credibility factor, and a vulnerability exposure factor.

[0015] By introducing multi-dimensional factors such as asset value and vulnerability status for quantitative integration, this solution transforms isolated threat alerts into dynamic risk values ​​that are related to specific business impacts. This enables security teams to respond based on clear numerical priorities, greatly improving the efficiency and accuracy of security operations and decision-making.

[0016] In some embodiments, the service and response module further includes a feedback interface for receiving confirmation or false alarm feedback from security operations personnel regarding alarm events, and for feeding the feedback data back to the offline batch processing pipeline for updating the behavior baseline or retraining the model.

[0017] This feedback loop design enables the system to continuously self-optimize, adapting to changes in the network environment and the evolution of attack methods, continuously reducing the false alarm rate and improving model accuracy, thus reducing the cost of manual maintenance in the later stages.

[0018] In some embodiments, the multi-source data acquisition module includes a hardware probe deployed on a network core node, used to collect network traffic metadata through port mirroring and encapsulate it into a standard IPFIX format log output.

[0019] Using dedicated hardware probes to process high-speed network traffic ensures the performance and stability of full traffic metadata collection, providing a high-quality data source for standardized subsequent processing and laying the foundation for real-time analysis in large-scale network environments.

[0020] Secondly, embodiments of the present invention provide a network security threat assessment method based on big data, comprising: S1 collects network traffic, endpoint behavior, security logs, threat intelligence, and asset vulnerability data through a multi-source data acquisition module; S2 utilizes the real-time stream processing pipeline of the big data processing platform module to perform streaming processing and feature calculation on the collected real-time data, while using its offline batch processing pipeline to batch process historical data to establish behavioral baselines. S3, through its intelligent threat assessment engine module, analyzes real-time and historical characteristics using a multi-model fusion strategy to generate raw threat signals. S4. Combining asset value and vulnerability information, perform dynamic risk assessment on the original threat signal and calculate the dynamic risk score associated with the specific asset. S5, through the service and response module, performs risk visualization display, generates handling work orders, or triggers automated response scripts based on the dynamic risk score.

[0021] (III) Beneficial Effects The beneficial effects of this invention are: 1. By integrating multi-dimensional data such as network traffic, terminal behavior, and asset status, it breaks down the data silos of traditional security devices, constructs a comprehensive security situation awareness view, and significantly reduces detection blind spots.

[0022] 2. By adopting a batch-stream integrated architecture to collaboratively analyze real-time data and historical baselines, and combining supervised and unsupervised machine learning models for fusion detection, it can effectively identify known attacks and unknown anomalies, reducing false alarms while improving the detection rate of advanced threats.

[0023] 3. Innovatively, it quantifies and integrates multiple factors such as threat signals, asset criticality, and vulnerability exploitability to output dynamic and interpretable risk scores, transforming security response from extensive alarm management to precise risk priority management.

[0024] 4. The system's built-in feedback loop and adaptive learning mechanism enable the model and baseline to automatically adjust as business and environmental conditions change, reducing reliance on manual maintenance of expert rules and improving the system's long-term applicability and operational efficiency.

[0025] 5. The structured risk scoring and API interfaces provided by the system can be seamlessly integrated with the automated orchestration and response platform, promoting the development of security operations towards intelligence and automation, and accelerating the speed of threat containment. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0027] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The big data-based network security threat assessment system and method proposed in this invention, when the system is running, comprehensively gathers multi-dimensional data such as network traffic, terminal behavior, security logs, external intelligence, and asset vulnerabilities through a multi-source data acquisition module, solving the problem of isolated data sources. This data is input into a big data processing platform module employing a batch-stream integrated architecture. The real-time stream processing pipeline performs instant correlation and feature extraction on the data to support second-level threat discovery, while the offline batch processing pipeline performs in-depth mining of massive historical data to establish dynamic behavioral baselines and train detection models. The intelligent threat assessment engine module, based on real-time and historical features, uses a multi-model fusion strategy for comprehensive analysis, not only identifying known attack patterns but also detecting unknown abnormal behaviors. Furthermore, it combines asset criticality and vulnerability status to quantify the original threat signal into a dynamic risk score associated with a specific asset. Finally, the service and response module, based on this dynamic risk score, achieves a closed loop from global risk visibility and priority work order generation to automated handling and response. This solution, through data fusion, real-time analysis, intelligent assessment, and coordinated response, effectively addresses the shortcomings of traditional solutions in terms of threat discovery comprehensiveness, assessment accuracy, and response efficiency.

[0029] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] See attached document Figure 1 A big data-based cybersecurity threat assessment system includes a multi-source data acquisition module, a big data processing platform module, an intelligent threat assessment engine module, and a service and response module connected in sequence.

[0031] The multi-source data acquisition module is responsible for collecting heterogeneous security data from different sources. Specifically, it deploys hardware probes at the core network switches via port mirroring to collect full traffic metadata and encapsulates it into standard IPFIX format for output. Lightweight agents are deployed on servers and terminal computers throughout the network to periodically collect host behavior data such as processes, files, and network connections. Logs and alerts from security devices such as firewalls, intrusion prevention systems, and web application firewalls are collected via system log protocols or dedicated APIs. Simultaneously, it accesses external commercial and open-source threat intelligence sources through subscription services to obtain the latest malicious IP, domain, and file hash intelligence. Furthermore, it integrates with the organization's configuration management database and vulnerability scanners to synchronize asset inventories and their corresponding business importance levels, as well as known vulnerability information.

[0032] The big data processing platform module adopts a batch-stream integrated architecture and serves as the data processing hub of the entire system. This module comprises a message queue, a stream processing engine, and a batch processing engine. All raw data from the acquisition module is first sent to a distributed message queue and cached according to different data types, such as network traffic topics and terminal event topics, to meet the requirements of high throughput and loose coupling.

[0033] The real-time stream processing pipeline is built on a stream computing engine and consumes real-time data from corresponding topics in the message queue. It primarily performs two tasks: first, it performs real-time event correlation, for example, associating network connection events from the same source IP within the same time period with abnormal process creation events discovered on the host corresponding to that source IP, forming richer contextual events; second, it calculates real-time features within a sliding time window, for example, calculating the entropy value of a source IP accessing different destination ports in the past five minutes, the number of geographical locations it connects to, and the communication frequency with malicious IPs in the threat intelligence database.

[0034] The offline batch processing pipeline periodically reads historical data from the data lake. The data lake stores all raw data and intermediate processing results from the message queue. The batch processing engine is used to perform computationally intensive tasks, including: performing statistical analysis on a daily or weekly basis on user login times, resource access habits, and inter-server communication traffic patterns to establish dynamic behavioral baselines; performing complex graph computations, constructing and analyzing "user-resource-host" relationship graphs, and using community detection algorithms to identify anomalous internal access subgraphs; and training and updating supervised machine learning models using historical data.

[0035] The intelligent threat assessment engine module is the core of the system's intelligent analysis. It consists of a feature engineering submodule, a multi-model fusion detection submodule, and a dynamic risk assessment submodule.

[0036] The feature engineering submodule receives real-time features from the real-time stream processing pipeline and historical baseline features and graph features from the offline batch processing pipeline, and combines and standardizes them to form a standardized feature vector for model inference.

[0037] The multi-model fusion detection submodule runs multiple detection models in parallel. Unsupervised detection models, such as Isolation Forest, are used to detect statistical anomalies in metrics such as traffic byte count and packet count; autoencoder models learn normal network access patterns and alert on sessions with high reconstruction errors. Supervised detection models, such as classifiers trained using the gradient boosting tree algorithm, are used to identify known attack patterns such as brute-force attacks and web attacks. Simultaneously, hard-coded threat intelligence matching rules serve as a fast track for immediate alerts on high-confidence threats. The outputs of these models and rules are unified into the original threat signal and its corresponding confidence score.

[0038] The dynamic risk assessment submodule receives raw threat signals and obtains the importance level of affected assets from the asset database (e.g., level 5 for core database servers and level 1 for ordinary office terminals), and the severity score of relevant vulnerabilities from the vulnerability database. This submodule incorporates a weighted scoring card algorithm that comprehensively calculates factors such as threat severity (normalized from multiple model outputs), asset value, attack credibility (e.g., high credibility if both network anomalies and terminal alerts are triggered simultaneously), and vulnerability exposure, according to preset weights. The final output is a dynamic risk score between 0 and 100, clearly indicating the specific affected asset.

[0039] The Service and Response module serves as the interface for interaction between the system and security operations personnel, as well as other systems. It provides a visual dashboard displaying the overall network risk situation, high-risk asset rankings, and a complete attack chain analysis view in the form of heatmaps, topology maps, etc. For high-risk events (e.g., risk scores greater than 80), this module automatically creates a handling work order in the work order system and assigns it to the appropriate responsible person. More importantly, it provides structured event data and risk scores through standard APIs, enabling seamless integration with security orchestration and automated response platforms to trigger predefined response scripts, such as automatically calling firewall APIs to isolate compromised hosts or calling endpoint protection software to perform virus scans.

[0040] When using this system to conduct network security threat assessments, the workflow is as follows: S1: All data collection components continue to run, pushing network traffic metadata, terminal behavior events, security device logs, etc., to the message queue of the big data processing platform in real time.

[0041] S2, the stream processing pipeline consumes data in real time, performs correlation analysis and real-time feature calculation; at the same time, the batch processing pipeline starts according to a predetermined cycle, updating the behavioral baseline and machine learning model based on historical data.

[0042] S3, the intelligent threat assessment engine, obtains real-time features and historical baselines from the data processing platform, generates original threat signals through multi-model fusion analysis, and calculates dynamic risk scores by combining asset vulnerability information.

[0043] S4, the service and response module, receives risk scores and event details and presents them visually. For low- to medium-risk events, it generates investigation work orders; for high-risk or confirmed attacks, it can automatically or after manual approval execute predefined response actions such as isolation and blocking.

[0044] In S5, security analysts input the results of their handling of system alarms (such as confirming a true positive or a false alarm) into the system through a feedback interface. This feedback data will serve as the data source for the next round of model training, driving continuous system optimization.

[0045] Through the above specific implementation methods, this system realizes a closed-loop process from multi-source data acquisition, real-time parallel processing, intelligent fusion analysis to dynamic risk assessment and automated response, providing an effective technical solution for building a proactive, intelligent, and adaptive next-generation network security defense system.

[0046] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0047] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A big data based cyber security threat assessment system, characterized in that, include: Multi-source data acquisition module, big data processing platform module, intelligent threat assessment engine module, and service and response module; The multi-source data acquisition module is used to collect network traffic data, terminal and host data, security device logs, external threat intelligence, and asset and vulnerability data, and send the collected data to the big data processing platform module. The big data processing platform module adopts a batch-stream integrated architecture, including a real-time stream processing pipeline and an offline batch processing pipeline. The real-time stream processing pipeline is used to perform streaming processing on the real-time data input from the multi-source data acquisition module, including event correlation and real-time feature calculation. The offline batch processing pipeline is used to batch process historical data to establish a baseline of historical behavior of users and entities and to train models. The intelligent threat assessment engine module is connected to the big data processing platform module and includes a feature engineering submodule, a multi-model fusion detection submodule, and a dynamic risk assessment submodule. The feature engineering submodule is configured to generate a feature set for threat assessment based on the real-time features output by the real-time stream processing pipeline and the historical baseline features output by the offline batch processing pipeline. The multi-model fusion detection submodule is configured to use a detection strategy that integrates unsupervised learning models, supervised learning models, and threat intelligence rules to analyze the feature set and output the original threat signal. The dynamic risk assessment submodule is configured to receive the original threat signal and, in conjunction with asset value information and vulnerability information, calculate a dynamic risk score associated with a specific asset through a risk assessment algorithm. The service and response module is connected to the intelligent threat assessment engine module and is used to visualize the dynamic risk score, generate security work orders, or trigger automated response actions.

2. The big data based cyber security threat assessment system as claimed in claim 1, wherein, The big data processing platform module also includes: The message queue module is used to receive and cache data from the multi-source data acquisition module, and provide it to the real-time stream processing pipeline and the offline batch processing pipeline by topic; The data lake storage module is used to store the processing results and historical data from the offline batch processing pipeline. 3.The big data based cyber security threat assessment system according to claim 1 or 2, characterized in that, The real-time stream processing pipeline is implemented based on a stream computing engine and is used to perform real-time feature calculations on network connections and host behavior within a sliding time window. The real-time features include at least one of the following: the destination port entropy of the source IP, the number of accesses to different geographical regions, the frequency of matching with malicious intelligence, and the abnormality of process creation.

4. The big data based cyber security threat assessment system according to claim 1 or 2, wherein, The offline batch processing pipeline is used to process historical communication data based on graph algorithms to construct a relationship graph between hosts and hosts, and between users and resources, and to identify abnormal communication subgraphs based on the relationship graph as part of the historical baseline features.

5. The big data based cyber security threat assessment system as claimed in claim 1, wherein, The unsupervised learning models in the multi-model fusion detection submodule include isolated forests or autoencoders, used to detect traffic statistics anomalies or deviations from normal patterns; the supervised learning models include gradient boosting tree models, used to detect known attack patterns.

6. The big data based cyber security threat assessment system of claim 1, wherein, The dynamic risk assessment submodule employs a weighted scoring card algorithm. The dynamic risk score is calculated by weighting at least the threat severity factor, asset value factor, attack credibility factor, and vulnerability exposure factor. The threat severity factor is obtained by normalizing the original threat signal output by the multi-model fusion detection submodule. The asset value factor comes from the configuration management database. The attack credibility factor is determined based on the aggregation result of multi-source alarm signals.

7. The big data-based network security threat assessment system according to claim 1, characterized in that, The service and response module also includes a feedback interface for receiving confirmation or false alarm feedback from security operations personnel regarding alarm events, and for feeding the feedback data back to the offline batch processing pipeline to update the historical behavior baseline or retrain the supervised learning model.

8. The big data-based network security threat assessment system according to claim 1, characterized in that, The multi-source data acquisition module includes hardware probes deployed on core network nodes, used to collect network traffic metadata via port mirroring and encapsulate it into standard IPFIX format log output.

9. The big data-based network security threat assessment system according to claim 1, characterized in that, The visualization provided by the service and response module includes an attack chain visualization view, which is used to correlate and graphically present discrete alarm events according to the attack lifecycle stages.

Images