Network security situation awareness and early warning method based on big data analysis

Abstract

The invention relates to the technical field of network security, in particular to a network security situation awareness and early warning method based on big data analysis, and the method comprises the following steps: collecting multi-modal security data, and carrying out the noise reduction and normalization processing; constructing a high-dimensional security feature vector by using feature engineering; establishing an anomaly detection model based on physical constraints and a deep residual network; a self-adaptive loss optimization strategy training model is adopted, so that the detection accuracy is improved; through cross-domain transfer learning, model adaptability is optimized, and unknown attacks can be detected; future attack trends are analyzed in combination with historical data, and defense strategies are dynamically adjusted. Through combination of deep learning and big data analysis, the accuracy, real-time performance and adaptive capability of network security situation awareness are improved, unknown attacks and variant attacks can be effectively detected, the false alarm rate is reduced, the security protection capability is enhanced, and the method is suitable for a security defense system in a complex network environment.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular, to a network security situation awareness and early warning method based on big data analysis. Background Art

[0002] With the rapid development of Internet technology, the means of network attacks have been continuously upgraded, and the attack methods have become increasingly complex and concealed, bringing huge challenges to traditional security defense systems. Especially in the context of the widespread application of cloud computing, big data, and the Internet of Things (IoT), the complexity of the network environment has been further enhanced, the attack surface has been continuously expanded, and the traditional security defense methods based on rule matching and signature detection have gradually revealed limitations.

[0003] At present, the existing network security situation awareness technologies mainly rely on intrusion detection systems (IDSs) based on rule matching and anomaly detection systems based on statistical analysis, but these methods have significant deficiencies. First, the detection method based on rule matching needs to rely on known attack signatures to build a signature library, and it cannot identify unknown attacks, and its detection ability is weak when facing variant attacks. Second, although the method based on statistical analysis can detect abnormal behaviors, for a complex network environment with high dimensions and multi-modalities, its detection accuracy is low, and it is easily affected by data noise, resulting in a high false alarm rate. In addition, when the existing situation awareness systems process large-scale network data, they often face problems such as low data processing efficiency, insufficient real-time performance, and poor model migration ability, making it difficult for the systems to adapt to the complex and changing network environment. Summary of the Invention

[0004] The present invention provides a network security situation awareness and early warning method based on big data analysis.

[0005] The network security situation awareness and early warning method based on big data analysis includes the following steps:

[0006] S1, preprocessing of security log data: collecting original security log data from distributed network devices, servers, and terminal systems, and performing data cleaning, format standardization, and time synchronization processing to generate a structured security log data set;

[0007] S2, feature correlation analysis: based on the structured security log data set, extracting security-related features such as network traffic, system calls, and user behaviors, and using an association rule mining algorithm to build an attack behavior feature library, where the attack behavior feature library is used to store the behavior patterns of different attack types;

[0008] S3, training of an anomaly detection model: based on the attack behavior feature library, training an anomaly detection model using an adaptive deep learning model, and optimizing the model parameters through a federated learning mechanism to generate a dynamically updatable security situation baseline;

[0009] S4, Real-time Security Situation Awareness: Input the structured security log data in the current network environment into the anomaly detection model, calculate the real-time security threat score, and conduct trend analysis based on historical situation data to form a network security situation map;

[0010] S5, Early Warning and Response Strategy Generation: According to the network security situation map, automatically generate attack early warning information based on the intelligent reasoning algorithm, and match the optimal defense strategy. The defense strategies include access control adjustment, traffic diversion, and isolation strategy deployment;

[0011] S6, Dynamic Feedback and Strategy Optimization: Based on the attack handling results and the execution effects of the defense strategies, update the attack behavior feature library and adjust the parameters of the anomaly detection model to improve the accuracy of future network security situation awareness.

[0012] Optionally, S1 includes:

[0013] S11, Data Collection: Obtain the original security logs from different security devices through a distributed log collection agent;

[0014] S12, Format Standardization: Unify different log formats, parse the key fields, and generate a standardized data format;

[0015] S13, Time Synchronization: Align the timestamps of all log data based on the Network Time Protocol to eliminate the time deviation of data from different sources;

[0016] S14, Data Denoising: Adopt an outlier detection method to remove invalid logs and improve the data quality.

[0017] Optionally, S2 includes:

[0018] S21, Network Traffic Feature Extraction: Based on traffic aggregation analysis, extract traffic statistical features and protocol distribution features;

[0019] S22, Behavior Pattern Analysis: Use machine learning methods to identify the normal behavior patterns of users and detect abnormal behavior patterns;

[0020] S23, Attack Pattern Matching: Based on association rule mining, construct an attack behavior feature library and store the feature patterns of different attack types.

[0021] Optionally, S3 includes:

[0022] S31, Data Preprocessing: Perform sample balancing and feature enhancement on the data in the attack behavior feature library;

[0023] S32, Model Training: Adopt an adaptive deep learning algorithm to train the anomaly detection model;

[0024] S33, Federated Learning Optimization: Optimize model parameters using a distributed training mechanism.

[0025] Optionally, S4 includes:

[0026] S41, Real-time Data Input: Obtain structured security log data of the current network environment;

[0027] S42, Threat Score Calculation: Calculate security threat scores based on an anomaly detection model;

[0028] S43, Situation Trend Analysis: Analyze the security situation trend in combination with historical data.

[0029] 6. The network security situation awareness and early warning method based on big data analysis according to claim 5, wherein S5 includes:

[0030] S51, Early Warning Information Generation: Set the early warning level based on the threat score;

[0031] S52, Defense Strategy Matching: Match the optimal defense strategy according to the attack type;

[0032] S53, Strategy Execution: Automatically execute the defense strategy on the target network environment.

[0033] Optionally, S6 includes:

[0034] S61, Disposal Result Collection: Record the attack disposal results and the execution situation of the defense strategy;

[0035] S62, Feature Library Update: Update the attack behavior feature library based on the disposal results;

[0036] S63, Model Parameter Optimization: Adjust the anomaly detection model parameters to improve the recognition accuracy.

[0037] Advantages of the present invention:

[0038] In the present invention, by constructing a multi-dimensional feature data fusion model, in the data preprocessing stage, through multi-source data collection (including network traffic, user behavior, system calls, etc.), and combined with data processing technologies such as dimensionality reduction, denoising, and normalization, the quality of the input data is ensured. In the feature extraction stage, using feature engineering methods, the original data is mapped to a high-dimensional feature space, effectively improving the distinguishability of abnormal behaviors. Compared with traditional security detection means based on single feature analysis, this method can greatly improve the data representation ability, enabling the anomaly detection model to accurately identify potential threats in a complex network environment.

[0039] The present invention uses an adaptive deep learning model for anomaly detection. In the model construction stage, a physical constraint layer is introduced to enhance the adaptability of the model to specific attack patterns, and deep features are extracted through a residual neural network to improve the generalization ability of the model. In the model training stage, a dynamic weight adjustment strategy is used to balance and optimize the physical constraint loss and the data fitting loss, enabling the model to improve the detection accuracy while reducing the false alarm rate. In addition, the present invention adopts a cross-domain transfer learning method during the training process to solve the problem of inconsistent data distributions in different network environments, so that the model can adapt to different network topologies and improve the robustness of detection. Brief Description of the Drawings

[0040] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only those of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flowchart of the method according to an embodiment of the present invention. Detailed Embodiments

[0042] The present invention will be described in detail below in conjunction with the drawings and specific embodiments. At the same time, it should be noted here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments. For some well-known technologies, those skilled in the art can also adopt other alternative methods for implementation; and the drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0043] It should be pointed out that when referring to "an embodiment", "embodiments", "exemplary embodiments", "some embodiments", etc. in the specification, the embodiments indicated may include specific features, structures or characteristics, but not necessarily every embodiment includes such specific features, structures or characteristics. In addition, when combining embodiments to describe specific features, structures or characteristics, implementing such features, structures or characteristics in combination with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the relevant art.

[0044] Generally, the terms can be understood at least in part from their use in the context. For example, at least in part depending on the context, the term "one or more" used herein can be used to describe any feature, structure or characteristic in a singular sense, or can be used to describe a combination of features, structures or characteristics in a plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey a set of exclusive factors, but rather, at least in part depending on the context, can allow for the existence of other factors that are not necessarily explicitly described.

[0045] As Figure 1 shown, the network security situation awareness and early warning method based on big data analysis includes the following steps:

[0046] S1, preprocessing of security log data: Collecting original security log data from distributed network devices, servers and terminal systems, and performing data cleaning, format standardization and time synchronization processing to generate a structured security log data set;

[0047] S2, feature correlation analysis: Extracting security-related features such as network traffic, system calls and user behavior based on the structured security log data set, and using the association rule mining algorithm to construct an attack behavior feature library, which is used to store the behavior patterns of different attack types;

[0048] S3, training of anomaly detection model: Based on the attack behavior feature library, using an adaptive deep learning model to train the anomaly detection model, and optimizing the model parameters through the federated learning mechanism to generate a dynamically updatable security situation baseline;

[0049] S4, real-time security situation awareness: Inputting the structured security log data in the current network environment into the anomaly detection model, calculating the real-time security threat score, and performing trend analysis based on historical situation data to form a network security situation map;

[0050] S5, generation of early warning and response strategies: According to the network security situation map, automatically generating attack early warning information based on the intelligent reasoning algorithm, and matching the optimal defense strategies, which include access control adjustment, traffic traction and isolation strategy deployment;

[0051] S6, dynamic feedback and strategy optimization: Based on the attack disposal results and the execution effects of the defense strategies, updating the attack behavior feature library and adjusting the parameters of the anomaly detection model to improve the accuracy of future network security situation awareness.

[0052] S1 includes:

[0053] S11, data collection: Obtaining original security logs from different security devices through a distributed log collection agent;

[0054] S12, format standardization: Unifying different log formats, parsing key fields, and generating a standardized data format;

[0055] S13, time synchronization: Aligning the timestamps of all log data based on the network time protocol to eliminate the time deviation of data from different sources;

[0056] S14, data denoising: Using the outlier detection method to remove invalid logs and improve the data quality.

[0057] The preprocessing of security log data realizes the standardization, synchronization, and noise reduction of security log data, providing high-quality structured data for subsequent feature correlation analysis.

[0058] S2 includes:

[0059] S21, network traffic feature extraction: Based on traffic aggregation analysis, extract traffic statistical features and protocol distribution features;

[0060] S22, behavior pattern analysis: Use machine learning methods to identify normal user behavior patterns and detect abnormal behavior patterns;

[0061] S23, attack pattern matching: Based on association rule mining, construct an attack behavior feature library to store feature patterns of different attack types;

[0062] Construct an attack behavior feature library through feature extraction and behavior analysis to provide data support for anomaly detection.

[0063] S3 includes:

[0064] S31, data preprocessing: Perform sample balancing and feature enhancement on the data in the attack behavior feature library;

[0065] S32, model training: Use an adaptive deep learning algorithm to train an anomaly detection model;

[0066] Use an adaptive deep learning algorithm to train an anomaly detection model. Let the input data set be X = {x1, x2,..., x n}, where each data point x i has a feature vector f i and a corresponding label y i (0 represents normal traffic, 1 represents abnormal traffic). Use the cross-entropy loss function to optimize the classification effect, expressed as:

[0067]

[0068] where N is the total number of training samples, y i is the true label (0 or 1) of sample x i , p(y i |f i ; θ) is the sample classification probability calculated based on the model parameters θ, θ is the trainable parameter of the deep learning model, and the goal of the model is to minimize the loss function by the gradient descent algorithm and continuously adjust the parameter θ to improve the classification accuracy;

[0069] S33, federated learning optimization: Use a distributed training mechanism to optimize the model parameters;

[0070] Improve the generalization ability of the anomaly detection model through data optimization and deep learning training.

[0071] S4 includes:

[0072] S41, real-time data input: Obtain structured security log data of the current network environment;

[0073] S42, threat score calculation: Calculate the security threat score based on the anomaly detection model;

[0074] S43, situation trend analysis: Analyze the security situation trend in combination with historical data;

[0075] Achieve the visualization of the network security situation through real-time threat assessment and trend analysis.

[0076] 6. The network security situation awareness and early warning method based on big data analysis according to claim 5, wherein S5 includes:

[0077] S51, early warning information generation: Set the early warning level based on the threat score;

[0078] S52, defense strategy matching: Match the optimal defense strategy according to the attack type;

[0079] S53, strategy execution: Automatically execute the defense strategy on the target network environment;

[0080] Generate early warning information through intelligent reasoning and execute the corresponding defense strategy to improve security.

[0081] S6 includes:

[0082] S61, disposal result collection: Record the attack disposal result and the execution situation of the defense strategy;

[0083] S62, feature library update: Update the attack behavior feature library based on the disposal result;

[0084] S63, model parameter optimization: Adjust the parameters of the anomaly detection model to improve the recognition accuracy;

[0085] Enhance the adaptive ability by dynamically feedback optimizing the model and the feature library.

[0086] The present invention covers any alternatives, modifications, equivalent methods and solutions made within the essence and scope of the present invention. To enable the public to have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, and those skilled in the art can fully understand the present invention without these detailed descriptions. In addition, well-known methods, processes, procedures, components and circuits are not described in detail to avoid unnecessary confusion to the essence of the present invention.

[0087] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A network security situation awareness and early warning method based on big data analysis, characterized in that, It includes the following steps: S1, Preprocessing of security log data: Collect raw security log data from distributed network devices, servers, and terminal systems, and perform data cleaning, format standardization, and time synchronization processing to generate a structured security log dataset; S2, Feature correlation analysis: Based on the structured security log dataset, extract security-related features such as network traffic, system calls, and user behavior, and use association rule mining algorithms to construct an attack behavior feature library, which is used to store the behavior patterns of different attack types; S3, Training of anomaly detection model: Based on the attack behavior feature library, use an adaptive deep learning model to train the anomaly detection model, and optimize the model parameters through a federated learning mechanism to generate a dynamically updatable security situation baseline; S4, Real-time security situation awareness: Input the structured security log data in the current network environment into the anomaly detection model, calculate the real-time security threat score, and perform trend analysis based on historical situation data to form a network security situation map; S5, Generation of early warning and response strategies: According to the network security situation map, automatically generate attack early warning information based on intelligent reasoning algorithms, and match the optimal defense strategies. The defense strategies include access control adjustment, traffic diversion, and isolation strategy deployment; S6, Dynamic feedback and strategy optimization: Based on the attack handling results and the execution effects of defense strategies, update the attack behavior feature library and adjust the parameters of the anomaly detection model to improve the accuracy of future network security situation awareness.

2. The network security situation awareness and early warning method based on big data analysis according to claim 1, characterized in that, S1 includes: S11, Data collection: Obtain raw security logs from different security devices through a distributed log collection agent; S12, Format standardization: Unify different log formats, parse key fields, and generate a standardized data format; S13, Time synchronization: Align the timestamps of all log data based on the Network Time Protocol to eliminate the time deviation of data from different sources; S14, Data denoising: Use outlier detection methods to remove invalid logs and improve data quality.

3. The network security situation awareness and early warning method based on big data analysis according to claim 2, characterized in that S2 It includes: S21, Extraction of network traffic features: Based on traffic aggregation analysis, extract traffic statistical features and protocol distribution features; S22, Behavior pattern analysis: Use machine learning methods to identify normal user behavior patterns and detect abnormal behavior patterns; S23, Attack pattern matching: Based on association rule mining, construct an attack behavior feature library to store the feature patterns of different attack types.

4. The network security situation awareness and early warning method based on big data analysis according to claim 3, characterized in that S3 It includes: S31, Data preprocessing: Perform sample balancing and feature enhancement on the data in the attack behavior feature library; S32, Model training: Use an adaptive deep learning algorithm to train the anomaly detection model; S33, Federated learning optimization: Use a distributed training mechanism to optimize the model parameters.

5. The network security situation awareness and early warning method based on big data analysis according to claim 4, characterized in that S4 It includes: S41, Real-time data input: Obtain the structured security log data of the current network environment; S42, Threat score calculation: Calculate the security threat score based on the anomaly detection model; S43, Situation trend analysis: Analyze the security situation trend in combination with historical data.

6. The network security situation awareness and early warning method based on big data analysis according to claim 5, characterized in that S5 It includes: S51, Early warning information generation: Set the early warning level based on the threat score; S52, Defense strategy matching: Match the optimal defense strategy according to the attack type; S53, Policy Execution: Automatically execute defense policies on the target network environment.

7. The network security situation awareness and early warning method based on big data analysis according to claim 6, characterized in that S6 Including: S61, Disposal Result Collection: Record the attack disposal results and the execution status of defense policies; S62, Feature Library Update: Update the attack behavior feature library based on the disposal results; S63, Model Parameter Optimization: Adjust the parameters of the anomaly detection model to improve the recognition accuracy.

Images