Network security analysis method and system based on big data

Abstract

The invention relates to the technical field of network security, in particular to a network security analysis method and system based on big data. Comprising the following steps: collecting related multi-source heterogeneous data of a network, and carrying out standardized processing such as cleaning and de-noising; network analysis is carried out based on the preprocessed data, network traffic is analyzed in real time by using machine learning and deep learning algorithms, and abnormal conditions are detected; constructing a risk prediction model according to a network analysis result and related information, and predicting a future network security risk level; if the risk level exceeds the threshold value, determining a security event source and a responsibility subject through data tracing; and finally, generating a safety response strategy according to risk prediction and data traceability results, and performing disposal. The corresponding system covers the modules of data acquisition, preprocessing, network analysis, risk prediction, data tracing, security response and disposal and the like, and all the modules work cooperatively to form a complete network security analysis and guarantee system, so that the stable operation of the network system is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and specifically provides a network security analysis method and system based on big data. Background Art

[0002] With the rapid development of information technology, the network scale is increasing day by day, the network structure is becoming more complex, and various network security threats are emerging in an endless stream. Traditional network security analysis means often rely on a single data source or simple rule matching, and it is difficult to cope with the current massive, multi-source heterogeneous data environment and complex and changeable network attack forms.

[0003] For example, relying solely on the log information of network devices for analysis may miss potential threat clues detected by other security devices; moreover, for new and highly concealed attack behaviors, conventional fixed-rule-based detection methods are difficult to identify in a timely and accurate manner. In addition, problems such as the lack of a systematic mechanism for quantitatively evaluating by combining multi-dimensional information in risk prediction, and the difficulty in quickly and accurately tracing the source and taking effective response and disposal measures after a security event occurs are also relatively prominent. Summary of the Invention

[0004] The present invention provides a network security analysis method and system based on big data. By comprehensively collecting and preprocessing multi-source heterogeneous data, advanced algorithms such as machine learning and deep learning are used for network analysis, risk prediction, data tracing, and security response and disposal, so as to accurately control the network security situation, timely discover and respond to potential network security risks, and ensure the stable operation of the network system.

[0005] A network security analysis method based on big data includes:

[0006] Data collection and preprocessing: Collect multi-source heterogeneous data from network devices, security devices, and threat intelligence platforms, and perform cleaning, denoising, format conversion, and standardization processing on the data;

[0007] Network analysis: Based on the preprocessed data, construct a network traffic feature library, and use machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic, and identify potential network attack behaviors, abnormal traffic patterns, and security vulnerabilities;

[0008] Risk prediction: Based on the results of network analysis, as well as network device asset information and threat intelligence information, extract feature data related to network security risks, construct a risk prediction model, predict future network security risks, quantitatively evaluate the risk level, and optimize the network analysis process based on the risk prediction results;

[0009] Data Traceability: If the risk level exceeds the preset risk threshold, based on the network analysis results, determine the traceability starting point and scope, apply data mining and correlation analysis techniques, track the data transfer path and operation records, establish a data traceability chain, determine the source and responsible entity of the security incident, and generate a data traceability report;

[0010] Security Response and Handling: According to the results of risk prediction and data traceability, generate corresponding security response strategies, automatically trigger security protection measures, record the processing process and results of security incidents, and form a security incident handling log.

[0011] Preferably, the real-time analysis and anomaly detection of network traffic using machine learning and deep learning algorithms specifically include:

[0012] Feature Extraction and Vectorization:

[0013] Extract multi-dimensional features from the preprocessed network traffic data, including but not limited to source IP address, destination IP address, port number, protocol type, packet size, transmission frequency, connection duration, and traffic burstiness;

[0014] Convert the extracted features into numerical vectors, perform one-hot encoding on discrete features, and perform standardization processing on continuous features;

[0015] Anomaly Detection Model Construction:

[0016] Anomaly Detection Based on Supervised Learning:

[0017] Use historical labeled normal traffic and attack traffic samples to train classification models, including support vector machines, random forests, and gradient boosting trees;

[0018] Among them, cross-validation and hyperparameter optimization are adopted during the model training process to improve the classification accuracy;

[0019] Anomaly Detection Based on Unsupervised Learning:

[0020] Use isolation forests and local outlier factor algorithms to identify traffic deviating from the normal pattern;

[0021] Use a deep learning model based on autoencoders to detect abnormal traffic through reconstruction error, where the dimension of the hidden layer of the autoencoder is set to 1 / 3 to 1 / 2 of the original feature dimension;

[0022] Anomaly Detection Based on Deep Learning:

[0023] Use long short-term memory networks to process time-series traffic data and capture the changing patterns of traffic over time;

[0024] Adopt a convolutional neural network to extract local features of traffic data and achieve real-time detection through a sliding window mechanism;

[0025] Real-time analysis and detection process:

[0026] Chunk the real-time network traffic data according to a preset time window, and generate a feature vector for each time window;

[0027] Input the feature vector into the trained anomaly detection model. At the same time, apply the ensemble learning strategy and adopt a voting mechanism to synthesize the prediction results of multiple models as the final prediction result;

[0028] For the traffic detected as abnormal, further analyze its behavior characteristics, including traffic peak, access frequency, and data transmission volume, compare with the known attack pattern library to determine the attack type, and generate anomaly information including anomaly time, source IP, destination IP, and attack type;

[0029] Model dynamic update:

[0030] Adopt an online learning mechanism. After processing a certain number of new samples, use the incremental learning algorithm to update the model parameters;

[0031] Regularly retrain the model with the latest collected samples to maintain the model's detection ability for new types of attacks.

[0032] Preferably, the construction of the risk prediction model specifically includes:

[0033] Multi-source data fusion:

[0034] Quantitatively fuse the anomaly detection results output by the network analysis, network device asset information, and threat intelligence information;

[0035] Standardize the asset information, assign a risk weight to each asset, and the weight calculation is based on the formula:

[0036] w v = V × v × I;

[0037] Where, V is the asset value, the value range is 1 - 5, v is the asset vulnerability, normalized based on the CVSS score, and I is the business impact degree, evaluated through the business interruption loss.

[0038] Feature engineering:

[0039] Extract key features from the fused data:

[0040] Time series features: Time series of the occurrence frequency of abnormal events, time patterns of attack behaviors;

[0041] Association features: Degree of association between abnormal events and specific assets;

[0042] Statistical features: statistical distribution of abnormal traffic, spatial distribution of asset risks;

[0043] Apply feature selection algorithms to screen the top 30% features with the highest contribution to risk prediction;

[0044] Model architecture design:

[0045] Adopt a hybrid model architecture, including:

[0046] Basic prediction layer: Use the gradient boosting tree XGBoost to establish a risk level prediction model and output the risk probability distribution;

[0047] Time series prediction layer: Use the long short-term memory network LSTM to capture the evolution trend of risks over time;

[0048] Attention mechanism layer: Dynamically adjust the importance of different features through attention weights;

[0049] Integration layer: Adopt a weighted voting mechanism to fuse the prediction results of multiple sub-models, and the weights are dynamically adjusted based on the performance of the models on the validation set;

[0050] Model training and optimization:

[0051] Obtain the security event records of the past 12 months as the dataset, with 70% as the training set and 30% as the validation set;

[0052] Adopt the following optimization strategies:

[0053] Apply the early stopping mechanism and learning rate decay to prevent overfitting, and stop training when the validation set loss does not decrease for 10 consecutive rounds;

[0054] Search for the optimal hyperparameters through Bayesian optimization, including the depth of the tree, learning rate, and LSTM hidden layer size.

[0055] Risk quantification and grading: The model outputs a comprehensive risk score, and the risk level is divided according to the comprehensive risk score, where: 0-20 points is low risk, 21-40 points is medium-low risk, 41-60 points is medium risk, 61-80 points is high risk, and 81-100 points is emergency risk.

[0056] Model update mechanism:

[0057] Adopt an incremental learning strategy and update the model parameters every 24 hours using newly collected data;

[0058] When a new attack pattern is detected, trigger the model retraining process.

[0059] Preferably, optimizing the network analysis process based on the risk prediction results specifically includes:

[0060] Dynamic feature weight adjustment:

[0061] According to the risk level output by the risk prediction model, the weight coefficient w of each feature in the network analysis is dynamically adjusted:

[0062]

[0063] Among them, w0 is the initial weight, α is the risk sensitivity coefficient, β is the correlation between the feature and high-risk assets, and R is the risk level.

[0064] Adaptive optimization of anomaly detection model parameters:

[0065] According to the time evolution trend of risk prediction, adjust the threshold parameter θ of the anomaly detection model:

[0066]

[0067] Among them, θ0 is the initial threshold, γ is the trend response coefficient, is the risk trend change rate, which is predicted by the LSTM model.

[0068] Dynamic adjustment of analysis cycle:

[0069] Automatically adjust the time window Δt of network analysis according to the risk level:

[0070] Δt=Δt0×e -λ×R ;

[0071] Among them, Δt0 is the basic time window and λ is the adjustment coefficient.

[0072] Preferably, determining the tracing starting point and scope based on the network analysis results specifically includes:

[0073] Determine the source IP address and target asset of traffic based on the abnormal traffic characteristics in the anomaly detection results;

[0074] Determine the time and space scope for traceability data collection based on the network topology and asset importance level;

[0075] Combined with threat intelligence information, identify suspicious attack paths and potential impact scope.

[0076] A network security analysis system based on big data, including a data acquisition module, a data preprocessing module, a network analysis module, a risk prediction module, a data tracing module and a security response and disposal module;

[0077] The data acquisition module is used to collect multi-source heterogeneous data from network devices, security devices and threat intelligence platforms;

[0078] The data preprocessing module is used to clean, denoise, convert the format, and standardize the multi-source heterogeneous data;

[0079] The network analysis module constructs a network traffic feature library based on the preprocessed data, and uses machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic; The risk prediction module constructs a risk prediction model based on the network analysis results, network device asset information, and threat intelligence information, and predicts and quantitatively evaluates future network security risks;

[0080] The data traceability module is used to perform data traceability based on the network analysis results when the risk level exceeds the preset threshold, and determine the source and responsible entity of the security incident;

[0081] The security response and handling module is used to generate security response strategies according to the risk prediction and data traceability results, and trigger security protection measures.

[0082] Compared with the prior art, the advantages of the present invention are as follows:

[0083] Comprehensive data analysis ability: By collecting multi-source heterogeneous data, information at all levels in the network is integrated, avoiding the limitations of single data source analysis, and being able to grasp the network security situation more comprehensively and accurately.

[0084] Accurate anomaly detection and risk prediction: Using a variety of machine learning and deep learning algorithms for network analysis and risk prediction, combining the advantages of different algorithms, and through means such as feature engineering and model optimization, the detection and prediction accuracy of abnormal traffic and network security risks are improved, potential threats can be discovered in time, and preventive measures can be taken in advance.

[0085] Efficient data traceability function: After a risk event occurs, the traceability starting point and scope can be quickly determined, and data mining and correlation analysis techniques are used to accurately trace the source and responsible entity of the security incident, providing a strong basis for subsequent liability investigation and vulnerability repair.

[0086] Strong dynamic adaptability: Whether it is the anomaly detection model, risk prediction model, or the overall analysis process, it has a dynamic update mechanism, can adapt to the changing network environment, new attack patterns, and business requirement changes, and always maintain effective control over network security. Detailed implementation manners

[0087] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0088] In one embodiment, a network security analysis method based on big data is provided, including:

[0089] Data collection: Collect multi-source heterogeneous data from network devices, security devices, and threat intelligence platforms, specifically including:

[0090] Collect from network devices: Through corresponding network management protocols (such as SNMP, etc.), regularly collect traffic data, connection status information, etc. on network devices. For example, obtain the incoming and outgoing traffic data of each port on the router, the source IP and destination IP of the connection, etc.

[0091] Collect from security devices: Utilize the interfaces or log output functions provided by security devices to collect access control logs of firewalls (recording information such as which IPs are allowed or prohibited from accessing, etc.), relevant records of suspected attack events detected by intrusion detection systems, etc.

[0092] Collect from threat intelligence platforms: According to the APIs or data subscription services provided by the platforms, obtain the latest threat intelligence information, such as a list of known malicious IP addresses, descriptions of newly emerging network attack techniques, etc.

[0093] Data preprocessing: Perform the following processing on the collected multi-source heterogeneous data:

[0094] Cleaning: Remove duplicate records, error records, and data that is clearly illogical in the data. For example, if multiple identical network connection records are found, only keep one; directly delete records that do not conform to the IP address specification.

[0095] Denoising: Through statistical analysis and other methods, identify and filter out noise data that is abnormal but not a real security threat due to reasons such as temporary network fluctuations. For example, occasionally occurring extremely small abnormal traffic peaks, if analyzed and found not to conform to the characteristics of common attack traffic and no similar situations occur subsequently, can be regarded as noise and removed.

[0096] Format conversion: Uniformly convert data from different sources into a format convenient for subsequent processing. For example, uniformly convert the time format in security device logs into the standard timestamp format, and convert text data output by various devices into a structured data format.

[0097] Standardization processing: Perform operations such as normalization on the data to make data of different magnitudes within the same magnitude range, facilitating subsequent algorithm processing. For example, map numerical data such as packet size and transmission frequency to a specific interval according to certain rules.

[0098] Network Analysis: Based on the preprocessed data, construct a network traffic feature library, and use machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic, identifying potential network attack behaviors, abnormal traffic patterns, and security vulnerabilities. Specifically, it includes:

[0099] Construct a network traffic feature library:

[0100] Extract features from the preprocessed data, extracting multi-dimensional features such as source IP address, destination IP address, port number, protocol type, packet size, transmission frequency, connection duration, traffic burstiness, etc. from network traffic data, and then organize and construct these features into a network traffic feature library for subsequent analysis and detection;

[0101] Use machine learning and deep learning algorithms for real-time analysis and anomaly detection:

[0102] Feature extraction and vectorization:

[0103] Extract from the preprocessed network traffic data according to the above-established feature dimensions. For the extracted discrete features (such as protocol type, etc.), use one-hot encoding to convert them into numerical vectors. For example, if the protocol types are TCP, UDP, ICMP, etc., encode them into different vector representations respectively; for continuous features (such as packet size, etc.), perform standardization processing to meet the model input requirements;

[0104] Anomaly detection model construction:

[0105] Anomaly detection based on supervised learning: Collect historical labeled normal traffic and attack traffic samples, and use classification models such as support vector machines, random forests, and gradient boosting trees for training. During the training process, use cross-validation to evaluate the model performance and improve the classification accuracy through hyperparameter optimization.

[0106] Anomaly detection based on unsupervised learning: Use the isolation forest algorithm to judge whether a data point is abnormal by constructing a random forest and based on the depth of the data point in the tree; use the local outlier factor algorithm to calculate indicators such as the local reachability density of each data point to identify traffic deviating from the normal pattern. At the same time, construct a deep learning model based on autoencoders, compress the input feature vector to the hidden layer (the dimension of the hidden layer is set to 1 / 3 to 1 / 2 of the original feature dimension) through the encoder, and then reconstruct the input vector through the decoder. Detect abnormal traffic according to the reconstruction error, and traffic with a large reconstruction error is determined to be abnormal traffic.

[0107] Anomaly Detection Based on Deep Learning: For time-series traffic data, the Long Short-Term Memory network (LSTM) is used for processing. Its memory cells can capture the changing patterns of traffic over time, such as the periodic changes in traffic during different time periods. The Convolutional Neural Network (CNN) is used to extract local features of traffic data through convolutional kernels, and the sliding window mechanism is used to detect real-time traffic. The window slides along the traffic data, continuously extracting features and determining whether it is abnormal.

[0108] Real-time Analysis and Detection:

[0109] The real-time network traffic data is processed in chunks according to a preset time window (e.g., every 5 minutes as a time window), and a feature vector is generated for the data within each time window.

[0110] These feature vectors are input into the above-mentioned various anomaly detection models that have been trained. At the same time, an ensemble learning strategy is applied, and a voting mechanism is used (for example, if the majority of models determine that a certain traffic is abnormal, then it is finally determined to be abnormal), and the prediction results of multiple models are combined as the final prediction result.

[0111] For the traffic detected as abnormal, further analyze its behavioral characteristics, such as counting the traffic peak, access frequency, data transmission volume, etc., compare with the known attack pattern library to determine the attack type, and generate anomaly information including the anomaly time, source IP, destination IP, and attack type.

[0112] Model Dynamic Update:

[0113] An online learning mechanism is adopted. After processing a certain number (e.g., after processing 1000 new traffic samples) of new samples, the incremental learning algorithm is used to update the model parameters, so that the model can adapt to the new data distribution in a timely manner.

[0114] The model is retrained regularly (e.g., weekly or monthly) using the latest collected samples to maintain the model's detection ability for new types of attacks and ensure that the model does not become obsolete due to changes in network attack patterns.

[0115] Risk Prediction: Based on the results of network analysis, as well as network device asset information and threat intelligence information, extract feature data related to network security risks, construct a risk prediction model, predict future network security risks, quantitatively evaluate the risk level, and optimize the network analysis process based on the risk prediction results. Specifically, it includes:

[0116] Multi-source Data Fusion: Quantitatively fuse the anomaly detection results output by the network analysis module with network device asset information and threat intelligence information. For asset information, first perform standardization processing according to the formula w v=V×v×I (where V is the asset value, with a value range of 1 - 5, v is the asset vulnerability, normalized based on the CVSS score, and I is the business impact, evaluated through business interruption losses) to assign risk weights to each asset, enabling the risk situations of different assets to be measured under the same standard, and then using this fused data for subsequent feature engineering.

[0117] Feature engineering:

[0118] Extract key features from the fused data, including:

[0119] Time series features: Analyze the time series of the occurrence frequency of abnormal events to observe whether there are periodic or trend changes; extract the time patterns of attack behaviors, such as whether certain attacks always occur during specific time periods, etc.

[0120] Association features: Calculate the association degree between abnormal events and specific assets to determine which assets are more vulnerable to attacks or which abnormal events are associated with important assets.

[0121] Statistical features: Statistically analyze the statistical distribution of abnormal traffic, such as the mean, variance of traffic size, etc.; analyze the spatial distribution of asset risks to understand the high and low levels of asset risks in different regions or network segments.

[0122] Apply feature selection algorithms to screen the top 30% of features with the highest contribution to risk prediction, and use the screened features to build a risk prediction model.

[0123] Model architecture design:

[0124] Adopt a hybrid model architecture:

[0125] Basic prediction layer: Use the gradient boosting tree XGBoost to establish a risk level prediction model, input the screened features, output the risk probability distribution, and make a preliminary judgment on the general situation of network security risks.

[0126] Time series prediction layer: Use the long short-term memory network LSTM to capture the evolution trend of risks over time, such as judging whether the risk is gradually increasing or decreasing, and the speed of risk change, etc.

[0127] Attention mechanism layer: Dynamically adjust the importance of different features through attention weights, enabling the model to focus on more critical features for prediction in different risk scenarios.

[0128] Integration layer: Adopt a weighted voting mechanism to fuse the prediction results of multiple sub-models, with the weights dynamically adjusted based on the performance of the models on the validation set, and comprehensively utilize the advantages of each sub-model to obtain a more accurate risk prediction result.

[0129] Model training and optimization:

[0130] Obtain the security incident records of the past 12 months as the dataset, and divide the data according to the ratio of 70% as the training set and 30% as the validation set. During the training process, adopt the following optimization strategies.

[0131] Apply early stopping mechanism and learning rate decay to prevent overfitting. Stop training when the validation set loss does not decrease for 10 consecutive rounds, avoiding the situation where the model performs well on the training set but has poor generalization ability on new data.

[0132] Search for the optimal hyperparameters through Bayesian optimization, including the depth of the tree, learning rate, LSTM hidden layer size, etc., to improve the model performance.

[0133] The risk prediction model outputs a comprehensive risk score: divide the risk levels according to the comprehensive risk score (0 - 20 points is low risk, 21 - 40 points is medium - low risk, 41 - 60 points is medium risk, 61 - 80 points is high risk, 81 - 100 points is emergency risk), and pass the risk levels and related prediction results to the data tracing module and for optimizing the network analysis process.

[0134] Model update:

[0135] Adopt an incremental learning strategy, update the model parameters every 24 hours using newly collected data, ensure that the model can timely reflect the latest network security situation. When a new attack pattern is detected, trigger the model retraining process, and reconstruct and train the risk prediction model to adapt to the new security threat situation.

[0136] Optimize the network analysis process based on the risk prediction results, specifically including:

[0137] Dynamic feature weight adjustment:

[0138] According to the risk level output by the risk prediction model, dynamically adjust the weight coefficients of each feature in the network analysis. Specifically, adjust according to the formula where \(w_0\) is the initial weight, \(\alpha\) is the risk sensitivity coefficient (the appropriate value can be determined according to experience or through experiments within a certain range), \(\beta\) is the correlation degree between the feature and high - risk assets (quantified by analyzing the correlation relationship between the feature and the determined high - risk assets, for example, the higher the correlation tightness, the larger the value), and \(R\) is the risk level. For example, if the risk level is high risk, for those features with high correlation with high - risk assets (such as the source IP address - related features of accessing high - risk assets, etc.), correspondingly increase their weight coefficients, so that in the subsequent network analysis process, the model pays more attention to these key features, and thus can more accurately detect abnormal traffic situations related to high - risk.

[0139] Adaptive optimization of the anomaly detection model parameters:

[0140] Adjust the threshold parameter of the anomaly detection model according to the time evolution trend of risk prediction. Specifically, according to the formula for adjustment, where θ0 is the initial threshold, γ is the trend response coefficient (which can be set and adjusted according to the actual situation), is the risk trend change rate (predicted by the LSTM model, reflecting how fast the risk changes over time). For example, when the risk trend change rate shows that the risk is rising rapidly, appropriately reduce the threshold of the anomaly detection model so that the model can more sensitively detect potential abnormal traffic and timely discover possible security threats; conversely, if the risk trend is stable or decreasing, the threshold can be appropriately increased to reduce false alarms and improve detection efficiency.

[0141] Dynamic adjustment of the analysis period:

[0142] Automatically adjust the time window of network analysis according to the risk level, according to the formula Δt = Δt0 × e -λ×R for adjustment, where Δt0 is the base time window (the preset initial time window length, such as every 5 minutes as a base time window), and λ is the adjustment coefficient (determine a suitable value according to the adjustment requirements corresponding to different risk levels). For example, in the case of a high risk level, shorten the time window, such as adjusting it to every 2 minutes as a time window, so that abnormal traffic changes within a short period of time can be captured more timely and a response can be made faster; while in the case of a low risk level, the time window can be appropriately extended to reduce the consumption of computing resources and at the same time avoid too many false alarms due to too short a time window.

[0143] Data tracing: If the risk level exceeds the preset risk threshold (the risk threshold score is set to 30 points in this embodiment), based on the network analysis results, determine the tracing starting point and scope, use data mining and correlation analysis techniques to track the data transfer path and operation records, establish a data tracing chain, determine the source and responsible entity of the security incident, and generate a data tracing report, specifically including:

[0144] Determine the tracing starting point and scope:

[0145] Based on the abnormal traffic characteristics in the anomaly detection results of the network analysis module, first determine the source IP address of the traffic and the target asset. For example, find the possible attacker and the corresponding target asset by analyzing the source IP of the abnormal traffic.

[0146] According to the network topology structure and asset importance level, delimit the time range (such as tracing back 1 hour or longer, specifically determined according to factors such as the risk situation and network scale) and spatial range (which network segments, devices, etc. are involved) for tracing data collection.

[0147] Combine threat intelligence information to determine suspicious attack paths (such as whether it has passed through known malicious relay nodes, etc.) and potential impact scopes (which other related assets may be affected, etc.).

[0148] Use data mining and correlation analysis techniques for tracing: Based on the determined tracing starting point, scope, and relevant information, use data mining algorithms to mine the correlation relationships in the data. For example, analyze which devices have had abnormal data interactions within the tracing time range, which operations are related to abnormal traffic, etc.; Through correlation analysis techniques, trace the data transfer path and operation records. For example, starting from the source IP, trace through each network node passed and operations such as access and transmission on these nodes, establish a data tracing chain, thereby determining the source and responsible entity of the security incident, and finally generate a data tracing report, which can be used for subsequent security response and handling as well as liability investigation and other work.

[0149] Security response and handling: According to the results of risk prediction and data tracing, generate corresponding security response strategies, automatically trigger security protection measures, record the processing process and results of security incidents, form a security incident handling log, and evaluate and optimize the effectiveness of security response strategies, specifically including:

[0150] Generate security response strategies: Based on the risk level and the source, impact scope, etc. determined by tracing, formulate corresponding security response strategies. For example, for low-risk situations, it may just issue a warning notice for the administrator to pay attention; for high-risk situations, it may immediately take measures such as blocking access to suspicious IPs and isolating related affected assets.

[0151] Trigger security protection measures: Automatically convert the generated security response strategies into specific operation instructions and send them to corresponding security devices (such as the firewall to execute access control policy adjustments, the intrusion prevention system to strengthen detection rules, etc.) to implement security protection measures and prevent the further deterioration of security incidents.

[0152] Record the processing process and results of security incidents: Record the entire process of the security incident from discovering the risk, tracing to taking response measures in detail to form a security incident handling log, including information such as the time, operation content, and effect of each operation, which is convenient for subsequent review and auditing.

[0153] In another embodiment, a network security analysis system based on big data is provided, including a data collection module, a data preprocessing module, a network analysis module, a risk prediction module, a data tracing module, and a security response and handling module;

[0154] The data collection module is used to collect multi-source heterogeneous data from network devices, security devices, and threat intelligence platforms;

[0155] The data preprocessing module is used to clean, denoise, convert the format, and standardize the multi-source heterogeneous data;

[0156] The network analysis module constructs a network traffic feature library based on the preprocessed data, and uses machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic; the risk prediction module constructs a risk prediction model based on the network analysis results, network device asset information, and threat intelligence information, and predicts and quantitatively evaluates future network security risks;

[0157] The data traceability module is used to trace the data based on the network analysis results when the risk level exceeds the preset threshold, and determine the source and responsible entity of the security incident;

[0158] The security response and handling module is used to generate a security response strategy according to the risk prediction and data traceability results, trigger security protection measures, and evaluate the effectiveness of the strategy.

[0159] Through the specific implementation of the above modules, the described network security analysis method based on big data is fully realized, comprehensively ensuring network security, timely responding to various network security threats, and continuously optimizing the network security protection system.

[0160] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0161] The above-disclosed preferred embodiments of the present invention are only used to help explain the present invention. The preferred embodiments do not elaborate all the details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art in the relevant technical field can understand and utilize the present invention well. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A network security analysis method based on big data, characterized in that, Including: Data collection and preprocessing: Collect multi-source heterogeneous data from network devices, security devices, and threat intelligence platforms, and perform cleaning, denoising, format conversion, and standardization on the data; Network analysis: Based on the preprocessed data, construct a network traffic feature library, and use machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic, identifying potential network attack behaviors, abnormal traffic patterns, and security vulnerabilities; Risk prediction: Based on the results of network analysis, as well as network device asset information and threat intelligence information, extract feature data related to network security risks, construct a risk prediction model, predict future network security risks, quantitatively evaluate the risk level, and optimize the network analysis process based on the risk prediction results; Data tracing: If the risk level exceeds the preset risk threshold, based on the results of network analysis, determine the tracing starting point and scope, use data mining and correlation analysis techniques to track the data transfer path and operation records, establish a data tracing chain, determine the source and responsible entity of the security incident, and generate a data tracing report; Security response and handling: According to the results of risk prediction and data tracing, generate corresponding security response strategies, automatically trigger security protection measures, record the processing process and results of security incidents, and form a security incident handling log.

2. The network security analysis method based on big data according to claim 1, wherein The use of machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic specifically includes: Feature extraction and vectorization: Extract multi-dimensional features from the preprocessed network traffic data, including but not limited to source IP address, destination IP address, port number, protocol type, packet size, transmission frequency, connection duration, traffic burstiness; Convert the extracted features into numerical vectors, perform one-hot encoding on discrete features, and perform standardization on continuous features; Anomaly detection model construction: Anomaly detection based on supervised learning: Use historical labeled normal traffic and attack traffic samples to train classification models, including support vector machines, random forests, and gradient boosting trees; Among them, cross-validation and hyperparameter optimization are adopted during the model training process to improve the classification accuracy; Anomaly detection based on unsupervised learning: Use isolation forests and local outlier factor algorithms to identify traffic deviating from the normal pattern; Use a deep learning model based on autoencoders to detect abnormal traffic through reconstruction error, where the dimension of the hidden layer of the autoencoder is set to 1 / 3 to 1 / 2 of the original feature dimension; Anomaly detection based on deep learning: Use long short-term memory networks to process time-series traffic data and capture the changing patterns of traffic over time; Adopt convolutional neural networks to extract local features of traffic data and achieve real-time detection through a sliding window mechanism; Real-time analysis and detection process: Chunk the real-time network traffic data according to a preset time window, and generate a feature vector for each time window; Input the feature vector into the trained anomaly detection model, and at the same time apply an ensemble learning strategy, adopt a voting mechanism, and comprehensively use the prediction results of multiple models as the final prediction result; For the traffic detected as abnormal, further analyze its behavioral characteristics, including traffic peak, access frequency, and data transmission volume, compare with the known attack pattern library to determine the attack type, and generate abnormal information including abnormal time, source IP, destination IP, and attack type; Model dynamic update: Adopt an online learning mechanism. After processing a certain number of new samples, use the incremental learning algorithm to update the model parameters; Regularly retrain the model with the latest collected samples to maintain the model's detection ability for new types of attacks.

3. A network security analysis method based on big data according to claim 1, characterized in that The construction of the risk prediction model specifically includes: Multi-source data fusion: Quantitatively fuse the abnormal detection results output by the network analysis, network device asset information, and threat intelligence information; Standardize the asset information, assign a risk weight to each asset, and the weight calculation is based on the formula: w v = V × v × I; Where V is the asset value, with a value range of 1-5, v is the asset vulnerability, normalized based on the CVSS score, and I is the business impact degree, evaluated through the business interruption loss. Feature engineering: Extract key features from the fused data: Temporal features: Time series of the occurrence frequency of abnormal events, time patterns of attack behaviors; Association features: Degree of association between abnormal events and specific assets; Statistical features: Statistical distribution of abnormal traffic, spatial distribution of asset risks; Apply feature selection algorithms to screen the top 30% features with the highest contribution to risk prediction; Model architecture design: Adopt a hybrid model architecture, including: Basic prediction layer: Use the gradient boosting tree XGBoost to establish a risk level prediction model and output the risk probability distribution; Temporal prediction layer: Use the long short-term memory network LSTM to capture the evolution trend of risk over time; Attention mechanism layer: Dynamically adjust the importance of different features through attention weights; Integration layer: Adopt a weighted voting mechanism to fuse the prediction results of multiple sub-models, and the weights are dynamically adjusted based on the performance of the models on the validation set; Model training and optimization: Obtain the security event records of the past 12 months as the data set, with 70% as the training set and 30% as the validation set; Adopt the following optimization strategies: Apply the early stopping mechanism and learning rate decay to prevent overfitting, and stop training when the validation set loss does not decrease for 10 consecutive rounds; Search for the optimal hyperparameters through Bayesian optimization, including the depth of the tree, learning rate, and LSTM hidden layer size. Risk quantification and level division: The model outputs a comprehensive risk score, and divides the risk levels according to the comprehensive risk score, where: 0-20 points is low risk, 21-40 points is medium-low risk, 41-60 points is medium risk, 61-80 points is high risk, and 81-100 points is critical risk. Model update mechanism: Adopt an incremental learning strategy and update the model parameters with newly collected data every 24 hours; When a new type of attack pattern is detected, trigger the model retraining process.

4. A network security analysis method based on big data according to claim 1, characterized in that, The optimization of the network analysis process based on the risk prediction results specifically includes: Dynamic feature weight adjustment: According to the risk level output by the risk prediction model, dynamically adjust the weight coefficient w of each feature in the network analysis: Where w0 is the initial weight, α is the risk sensitivity coefficient, β is the degree of association between the feature and high-risk assets, and R is the risk level. Adaptive Optimization of Anomaly Detection Model Parameters: Adjust the threshold parameter θ of the anomaly detection model according to the time evolution trend of risk prediction: Among them, θ0 is the initial threshold, and γ is the trend response coefficient. is the risk trend change rate, which is obtained by predicting through the LSTM model. Dynamic Adjustment of Analysis Period: Automatically adjust the time window Δt of network analysis according to the risk level: Δt = Δt0×e -λ×R ; Where, Δt0 is the basic time window and λ is the adjustment coefficient.

5. A method for network security analysis based on big data according to claim 1, characterized in that Determining the origin and scope of traceability based on the network analysis results specifically includes: Determine the source IP address and target asset of the traffic based on the abnormal traffic characteristics in the anomaly detection results; Define the time range and space range for traceability data collection according to the network topology structure and asset importance level; Combine threat intelligence information to determine the suspicious attack path and potential impact range.

6. A network security analysis system based on big data, characterized in that, Including a data collection module, a data preprocessing module, a network analysis module, a risk prediction module, a data traceability module, and a security response and handling module; The data collection module is used to collect multi-source heterogeneous data from network devices, security devices, and threat intelligence platforms; The data preprocessing module is used to clean, denoise, convert the format, and standardize the multi-source heterogeneous data; The network analysis module constructs a network traffic feature library based on the preprocessed data, and uses machine learning and deep learning algorithms to perform real-time analysis and anomaly detection on network traffic; The risk prediction module constructs a risk prediction model based on the network analysis results, network device asset information, and threat intelligence information, and predicts and quantitatively evaluates future network security risks; The data traceability module is used to perform data traceability based on the network analysis results when the risk level exceeds the preset threshold, and determine the source and responsible entity of the security incident; The security response and handling module is used to generate a security response strategy according to the risk prediction and data traceability results, and trigger security protection measures.

Images