Network security sensing method and device based on big data

Abstract

The invention is suitable for the technical field of network security, and provides a network security sensing method and device based on big data, and the method comprises the following steps: collecting multi-source data, carrying out the preprocessing of the collected data, and carrying out the data fusion; performing classification analysis, clustering analysis and association rule mining on the fused data, extracting feature information associated with security situation assessment from the processed data, and dividing the feature information into a training set and a test set; constructing a security situation awareness model based on a random forest algorithm and evaluation indexes, wherein the evaluation indexes comprise an intrusion detection rate, a false alarm rate, a missing report rate and response time; training the security situation awareness model based on the training set, evaluating the performance of the model through a cross validation method, and performing parameter tuning; and evaluating the security situation awareness model based on the test set. Compared with an existing network security situation awareness scheme, the network security situation awareness method has higher accuracy, robustness and practicability.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network security perception method and device based on big data. Background Art

[0002] Network security situational awareness refers to the collection, analysis and processing of various information in the network environment to understand the network security status in real time, and to promptly discover and respond to potential threats and attacks. It involves early warning, evaluation and prediction of security threats in the network system to ensure the security of the network system. The data collected by the existing network security awareness scheme is not comprehensive enough, the robustness is not high enough, and the verification and evaluation are not sufficient. Therefore, it is necessary to provide a network security awareness method and device based on big data to solve the above problems. Summary of the invention

[0003] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a network security perception method and device based on big data to solve the problems existing in the above-mentioned background technology.

[0004] The present invention is implemented as follows: a network security perception method based on big data, the method comprising the following steps: Collect multi-source data, pre-process the collected data, and perform data fusion. The fusion result = ∑(value of data source i × weight i), where i represents each data source; Perform classification analysis, cluster analysis, and association rule mining on the fused data, extract feature information associated with security situation assessment from the processed data, the feature information includes network traffic features, system log features, and user behavior features, and divide the feature information into a training set and a test set; Building a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; The security situation awareness model is trained based on the training set, the performance of the model is evaluated through cross-validation method, and the parameters are tuned; The security situation awareness model is evaluated based on the test set, and evaluation indicators are calculated. The evaluation indicators include accuracy, recall rate and F1 score.

[0005] As a further solution of the present invention: the steps of performing classification analysis, cluster analysis and association rule mining on the fused data specifically include: Training a classifier based on a decision tree, and performing classification analysis on the fused data through the classifier; The fused data is input into the clustering algorithm K-Means to obtain the clustering results, and the quality of the clustering results is evaluated based on the silhouette coefficient; The fused data is converted into shopping basket data, the Apriori algorithm is used to mine the association rules in the data, and the association rules are screened according to the support and confidence thresholds.

[0006] As a further solution of the present invention: the step of extracting characteristic information associated with security situation assessment from the processed data specifically includes: Extract network traffic features and identify abnormal traffic patterns, including sudden increases in traffic and unusual port accesses; Extract system log features and extract key information from system error logs, where the key information includes error type, error time, and error frequency; Extract user behavior features, identify abnormal behaviors such as logging in during non-working hours and accessing sensitive resources, and correlate abnormal user behaviors with network traffic and system logs for analysis.

[0007] As a further solution of the present invention: the step of constructing a security situation awareness model based on the random forest algorithm and the evaluation index specifically includes: Construct a random forest, randomly extract multiple sample subsets from the training set with replacement, and use each subset to train a decision tree. At the node of each decision tree, randomly select some features for node splitting to increase the diversity of the model. Using the selected features and sample subsets to train each decision tree until a predetermined stopping condition is reached, wherein the stopping condition is the maximum depth of the tree or the minimum number of node samples; Based on the output results of random forest, the security situation assessment logic is constructed to obtain the security situation awareness model.

[0008] As a further solution of the present invention: the step of evaluating the security situation awareness model based on the test set and calculating the evaluation index specifically includes: Input the data in the test set into the security situation awareness model, make predictions, and output the prediction results for each sample; Based on the prediction results, the accuracy, recall and F1 score are calculated to obtain the evaluation results, and the performance of the model is displayed by drawing the ROC curve and PR curve.

[0009] Another object of the present invention is to provide a network security perception device based on big data, the device comprising: The multi-source data acquisition module is used to collect multi-source data, pre-process the collected data, and perform data fusion. The fusion result = ∑ (value of data source i × weight i), where i represents each data source; The data in-depth analysis module is used to perform classification analysis, cluster analysis and association rule mining on the fused data, extract feature information associated with security situation assessment from the processed data, the feature information includes network traffic features, system log features and user behavior features, and divide the feature information into training sets and test sets; A perception model building module is used to build a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; The perception model training module is used to train the security situation awareness model based on the training set, evaluate the performance of the model through the cross-validation method, and perform parameter tuning; The perception model evaluation module is used to evaluate the security situation awareness model based on the test set and calculate evaluation indicators, which include accuracy, recall rate and F1 score.

[0010] As a further solution of the present invention: the data depth analysis module includes: A data classification and analysis unit, used to train a classifier based on a decision tree, and to perform classification and analysis on the fused data through the classifier; The data clustering analysis unit is used to input the fused data into the clustering algorithm K-Means to obtain the clustering results and evaluate the quality of the clustering results based on the silhouette coefficient; The association rule mining unit is used to convert the fused data into shopping basket data, use the Apriori algorithm to mine the association rules in the data, and screen the association rules according to the support and confidence thresholds.

[0011] As a further solution of the present invention: the data depth analysis module also includes: A network traffic feature unit, used to extract network traffic features and identify abnormal traffic patterns, including sudden increases in traffic and unusual port accesses; A system log feature unit, used to extract system log features and key information from system error logs, wherein the key information includes error type, error time, and error frequency; The user behavior feature unit is used to extract user behavior features, identify abnormal behaviors such as logging in during non-working hours and accessing sensitive resources, and correlate abnormal user behaviors with network traffic and system logs for analysis.

[0012] As a further solution of the present invention: the perception model building module includes: The random forest construction unit is used to construct a random forest. Multiple sample subsets are randomly extracted from the training set with replacement. Each subset is used to train a decision tree. At the node of each decision tree, some features are randomly selected for node splitting to increase the diversity of the model. A decision tree training unit, used to train each decision tree using the selected features and sample subsets until a predetermined stopping condition is reached, wherein the stopping condition is the maximum depth of the tree or the minimum number of node samples; The perception model building unit is used to build security situation assessment logic based on the output results of the random forest to obtain a security situation perception model.

[0013] As a further solution of the present invention: the perception model evaluation module includes: A prediction result determination unit is used to input the data in the test set into the security situation awareness model, make predictions, and output the prediction results of each sample; The perception model evaluation unit is used to calculate the accuracy, recall rate and F1 score based on the prediction results, obtain the evaluation results, and display the performance of the model by drawing the ROC curve and PR curve.

[0014] Compared with the prior art, the present invention has the following beneficial effects: Through the optimization of multi-source data collection and fusion, data preprocessing and feature extraction, model construction based on random forest algorithm, cross-validation and parameter tuning, and a comprehensive evaluation system, the present invention has higher accuracy, robustness and practicality than existing network security situation awareness solutions, and can better adapt to complex and changeable network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flow chart of a network security perception method based on big data.

[0016] Figure 2 The present invention is a flowchart for processing fused data in a network security perception method based on big data.

[0017] Figure 3 This is a flowchart for extracting feature information from a network security perception method based on big data.

[0018] Figure 4 The present invention is a flowchart for constructing a security situation awareness model in a network security awareness method based on big data.

[0019] Figure 5 The figure is a flowchart for evaluating the security situation awareness model in a network security awareness method based on big data.

[0020] Figure 6The figure is a schematic diagram of the structure of a network security perception device based on big data. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a network security perception method based on big data, and the method includes the following steps: S100, collect multi-source data, pre-process the collected data, and perform data fusion, where the fusion result = ∑ (value of data source i × weight i), where i represents each data source; S200, performing classification analysis, cluster analysis, and association rule mining on the fused data, extracting feature information associated with security situation assessment from the processed data, the feature information including network traffic features, system log features, and user behavior features, and dividing the feature information into a training set and a test set; S300, constructing a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate, and response time; S400, trains the security situation awareness model based on the training set, evaluates the performance of the model through cross-validation method, and performs parameter tuning; S500, evaluating the security situation awareness model based on the test set, and calculating evaluation indicators, wherein the evaluation indicators include accuracy, recall rate, and F1 score.

[0024] It should be noted that the data collected by existing network security awareness solutions is not comprehensive enough, the robustness is not high enough, and the verification and evaluation are not sufficient. The embodiments of the present invention aim to solve the above problems.

[0025] In an embodiment of the present invention, multi-source data will be collected, and network security-related data will be collected from multiple data sources. These data sources may include open source intelligence, business intelligence, social media data, network traffic data, log data, etc. The collected data is preprocessed, and duplicate data, noise data, and incomplete data are removed using tools and techniques such as deduplication algorithms, data normalization, and formatting. Then, data fusion is performed to weighted average the data from different data sources according to certain weights. The weight can be set according to the reliability, accuracy, or importance of the data source, and the fusion result = ∑ (value of data source i × weight i), where i represents each data source. Then, the fused data needs to be classified, clustered, and mined for association rules, and feature information associated with security situation assessment is extracted from the processed data. The feature information is divided into training sets and test sets to facilitate subsequent training and verification evaluation work. Then, a security situation awareness model can be constructed based on the random forest algorithm and evaluation indicators, and the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; the security situation awareness model is trained based on the training set, and the performance of the model is evaluated by the cross-validation method, and the parameters are tuned; finally, the embodiment of the present invention will also evaluate the security situation awareness model based on the test set, calculate the evaluation indicators, and the evaluation indicators include accuracy, recall rate and F1 score, and optimize the model according to the evaluation results. If the performance of the model does not meet the requirements, you can try to adjust the parameters of the model, change the algorithm or use ensemble learning and other methods to improve the performance of the model. In this way, the embodiment of the present invention has higher accuracy, robustness and practicality than the existing network security situation awareness scheme through multi-source data collection and fusion, data preprocessing and feature extraction, model construction based on the random forest algorithm, cross-validation and parameter tuning, and comprehensive evaluation system optimization, and can better adapt to complex and changing network environments.

[0026] like Figure 2 As shown, as a preferred embodiment of the present invention, the steps of performing classification analysis, cluster analysis and association rule mining on the fused data specifically include: S201, training a classifier based on a decision tree, and performing classification analysis on the fused data through the classifier; S202, inputting the fused data into the clustering algorithm K-Means to obtain the clustering result, and evaluating the quality of the clustering result based on the silhouette coefficient; S203, converting the fused data into shopping basket data, using the Apriori algorithm to mine association rules in the data, and screening the association rules according to support and confidence thresholds.

[0027] In an embodiment of the present invention, in order to deeply analyze the data, a classifier is trained based on a decision tree classification algorithm, and the fused data is classified and analyzed by the classifier. Then, it is necessary to determine an appropriate number of clusters, input the fused data into the clustering algorithm K-Means, obtain the clustering results, evaluate the quality of the clustering results based on the silhouette coefficient, analyze and interpret the data based on the clustering results, and extract valuable information. In addition, it is also necessary to convert the fused data into a format suitable for association rule mining (such as shopping basket data), set appropriate support and confidence thresholds according to business needs, use the Apriori algorithm to mine association rules in the data, and screen out valuable association rules based on support and confidence thresholds.

[0028] As a preferred embodiment of the present invention, the step of inputting the fused data into the clustering algorithm K-Means to obtain the clustering result, and evaluating the quality of the clustering result based on the silhouette coefficient specifically includes: The fused data is standardized to obtain standardized data; Each data point in the standardized data is calculated using the weighted Mahalanobis distance discriminant method to obtain the weighted Mahalanobis distance between each pair of data points; Randomly select a data point as the first center from the standardized data, and select several center points by calculating the probability of each data point becoming the next center point based on the Euclidean distance between the existing center point and all data points; According to the weighted Mahalanobis distance between each pair of data points, the data points are assigned to the clusters to which the adjacent center points belong for allocation operation; The mean of all data points in each cluster is used as the new center point to update the center point. If an empty cluster appears, a data point is randomly selected from the standardized data as the new center point. The allocation operation and the center point update operation are repeated in an iterative manner to obtain the clustering result; The difference between the distance between each data point and other points in its cluster and the distance from the data point to the nearest cluster is taken as the silhouette coefficient; the value of the silhouette coefficient is between -1 and 1, and a value close to 1 indicates a good clustering structure; By comparing the silhouette coefficients under each number of clusters, the number of clusters that can maximize the silhouette coefficient is selected as the optimal number of clusters to obtain the optimized clustering result.

[0029] In the embodiment of the present invention, in order to ensure a better clustering effect, the present invention adopts a weighted Mahalanobis distance, fully considers the covariance between features, and helps to obtain a more reasonable distance metric when the feature scales are different. In addition, in each iteration, the center point is dynamically recalculated and possible empty clusters are processed to ensure the clustering quality.

[0030] As a preferred embodiment of the present invention, the steps of converting the fused data into shopping basket data, mining association rules in the data using the Apriori algorithm, and screening the association rules according to support and confidence thresholds specifically include: Step S2021, dividing the shopping basket data into a plurality of transaction lists, each transaction list consisting of a plurality of commodities; Step S2022, traverse all transactions, and form the first candidate item set with the single commodities appearing in all transactions; Step S2023, counting the number of transactions of each item set in the first candidate item set, and dividing the number of transactions of a single item set by the total number of transactions to obtain the support of the corresponding item set; Step S2024, setting a support threshold, and screening each item set in the first candidate item set according to the support threshold to determine the first item set; Step S2025, combining any two itemsets in the first itemset according to the number of transactions of each itemset in the first itemset to generate candidate frequent itemsets; Step S2026, repeating the operations from step S2023 to step S2025 in an iterative manner until no new frequent item sets can be generated, so as to obtain frequent item sets; Step S2027, splitting the itemsets in the frequent itemsets into antecedent items and consequent items, defining the antecedent items as rule conditions, and defining the consequent items as rule conclusions, and calculating the confidence of the rule conclusions under the rule conditions to calculate the correlation between the multiple commodities; Step S2028, setting a confidence threshold, filtering the confidence according to the confidence threshold, and then sorting the filtering results to obtain association rules between products.

[0031] like Figure 3 As shown, as a preferred embodiment of the present invention, the step of extracting feature information associated with security situation assessment from the processed data specifically includes: S301, extracting network traffic features and identifying abnormal traffic patterns, wherein the abnormal traffic patterns include sudden increases in traffic and unusual port accesses; S302, extracting system log features, extracting key information from the system error log, wherein the key information includes error type, error time, and error frequency; S303, extracting user behavior features, identifying abnormal behaviors of logging in and accessing sensitive resources during non-working hours, and correlating and analyzing abnormal user behaviors with network traffic and system logs.

[0032] In the embodiment of the present invention, the characteristic data is determined in three parts. The first is network traffic feature extraction: traffic statistical features, such as total traffic, inbound traffic, outbound traffic, traffic peak, traffic fluctuation, etc. Protocol distribution, analyzing the traffic proportion of different protocols (such as HTTP, FTP, SMTP, etc.). Abnormal traffic detection, identifying abnormal traffic patterns, such as sudden increase in traffic, uncommon port access, etc. The second is system log feature extraction: error log, extracting key information from system error logs, such as error type, error time, error frequency, etc. Login log, analyzing the frequency, source IP, login time and other features of login activities. Operation log, extracting key operation information from system operation logs, such as file access, permission change, etc. The third is user behavior feature extraction: user activity pattern, analyzing the user's daily activity pattern, such as login time, operation frequency, access path, etc. Abnormal behavior detection, identifying abnormal user behavior, such as logging in during non-working hours, accessing sensitive resources, etc. Behavior correlation analysis, correlating user behavior with network traffic and system logs to discover potential security threats.

[0033] like Figure 4 As shown, as a preferred embodiment of the present invention, the step of constructing a security situation awareness model based on a random forest algorithm and evaluation indicators specifically includes: S304, constructing a random forest, randomly extracting multiple sample subsets from the training set with replacement, each subset is used to train a decision tree, and at the node of each decision tree, some features are randomly selected for node splitting to increase the diversity of the model; S305, training each decision tree using the selected features and sample subsets until a predetermined stopping condition is reached, wherein the stopping condition is the maximum depth of the tree or the minimum number of node samples; S306, based on the output results of the random forest, construct a security situation assessment logic to obtain a security situation awareness model.

[0034] In the embodiment of the present invention, a random forest is first constructed, and multiple sample subsets are randomly extracted from the training set with replacement, each subset is used to train a decision tree, and at the nodes of each decision tree, some features are randomly selected for node splitting to increase the diversity of the model; each decision tree is trained using the selected features and sample subsets until a predetermined stop condition (such as the maximum depth of the tree, the minimum number of node samples, etc.) is reached, and the parameters of the random forest, such as the number of trees, the number of feature selections, etc., are adjusted by cross-validation and other methods to optimize the performance of the model. Then the situation assessment logic is constructed: based on the output results of the random forest, the security situation assessment logic is constructed. This usually involves the integration and analysis of the prediction results of the random forest to form an overall judgment of the network security situation. And integrated evaluation indicators: intrusion detection rate, the ability of the evaluation model to correctly identify intrusion behavior. False alarm rate, the proportion of normal behavior misjudged as intrusion by the evaluation model. Missing rate, the proportion of actual intrusion behavior that the evaluation model fails to identify. Response time, the time from the detection of intrusion behavior to the generation of an alarm or the adoption of countermeasures by the evaluation model. These indicators can be obtained by evaluating the test set and used to adjust and optimize the model.

[0035] In a preferred embodiment of the present invention, the step of constructing a random forest, randomly extracting multiple sample subsets from a training set with replacement, each subset is used to train a decision tree, and at each node of each decision tree, randomly selecting some features for node splitting to increase the diversity of the model, specifically includes: Step S3041, obtaining a training set and a feature set corresponding to the training set, randomly extracting multiple sample subsets from the training set with replacement, and assigning the same initial weight to each sample subset; Step S3042, construct a decision tree for each sample subset, input the sample subset into a single decision tree, and randomly select some features from the feature set at the node of each decision tree to perform node splitting to perform single decision tree training; Step S3043, after the training of a single decision tree is completed, the sample classification error is obtained in the decision tree after the training without selecting the sample input; Step S3044, adjusting the initial weights of the corresponding sample subsets according to the sample classification errors to obtain adjusted sample weights, and extracting multiple new sample subsets from the training set again with replacement based on the adjusted sample weights; Step S3045, repeating steps S3042 to S3044 in an iterative manner to train the random forest.

[0036] In the embodiment of the present invention, in order to improve the robustness and stability of the model, the present invention continuously adjusts the sample weights in an iterative manner and re-extracts sample subsets for training, so that the model can be gradually optimized and adapted to the complexity of the data. And it can pay more attention to those samples that are misclassified in training, thereby enhancing the influence of these samples in subsequent training. This method of adjusting sample weights can improve the generalization ability of the model for difficult-to-classify samples.

[0037] like Figure 5 As shown, as a preferred embodiment of the present invention, the step of evaluating the security situation awareness model based on the test set and calculating the evaluation index specifically includes: S501, input the data in the test set into the security situation awareness model, perform prediction, and output the prediction result of each sample; S502, calculate the accuracy, recall and F1 score based on the prediction results, obtain the evaluation results, and display the performance of the model by drawing the ROC curve and PR curve.

[0038] In the embodiment of the present invention, it is necessary to ensure that the test set data is independent of the training set, that is, the test set data has not been used in the model training process. The test set should contain data related to security situation assessment, such as various network traffic, system logs, and user behaviors. The data in the test set is labeled to clarify whether each sample belongs to a security threat (positive class) or a normal behavior (negative class). The labeling should be performed by security experts with professional knowledge to ensure the accuracy of the labeling. Then the data in the test set is input into the security situation awareness model, predicted, and the prediction results of each sample are output; the accuracy, recall rate and F1 score are calculated based on the prediction results to obtain the evaluation results, and the performance of the model is displayed by drawing the ROC curve and PR curve. Among them, the accuracy rate measures the proportion of the number of samples predicted correctly by the model to the total number of samples, and the recall rate mainly focuses on the ability of the model to correctly identify the positive class, that is, the proportion of true positive examples to all actual positive examples. The F1 score is the harmonic mean of the precision rate and the recall rate, which aims to combine the performance of the two and provide a balanced indicator.

[0039] like Figure 6 As shown, an embodiment of the present invention further provides a network security perception device based on big data, the device comprising: The multi-source data collection module 100 is used to collect multi-source data, pre-process the collected data, and perform data fusion, where the fusion result = ∑ (value of data source i × weight i), where i represents each data source; The data in-depth analysis module 200 is used to perform classification analysis, cluster analysis and association rule mining on the fused data, extract feature information associated with security situation assessment from the processed data, the feature information includes network traffic features, system log features and user behavior features, and divide the feature information into a training set and a test set; A perception model building module 300 is used to build a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; A perception model training module 400 is used to train the security situation awareness model based on the training set, evaluate the performance of the model through a cross-validation method, and perform parameter tuning; The perception model evaluation module 500 is used to evaluate the security situation awareness model based on the test set and calculate evaluation indicators, which include accuracy, recall rate and F1 score.

[0040] As a preferred embodiment of the present invention, the data depth analysis module 200 includes: A data classification and analysis unit, used to train a classifier based on a decision tree, and to perform classification and analysis on the fused data through the classifier; The data clustering analysis unit is used to input the fused data into the clustering algorithm K-Means to obtain the clustering results and evaluate the quality of the clustering results based on the silhouette coefficient; The association rule mining unit is used to convert the fused data into shopping basket data, use the Apriori algorithm to mine the association rules in the data, and screen the association rules according to the support and confidence thresholds.

[0041] As a preferred embodiment of the present invention, the data depth analysis module 200 further includes: A network traffic feature unit, used to extract network traffic features and identify abnormal traffic patterns, including sudden increases in traffic and unusual port accesses; A system log feature unit, used to extract system log features and key information from system error logs, wherein the key information includes error type, error time, and error frequency; The user behavior feature unit is used to extract user behavior features, identify abnormal behaviors such as logging in during non-working hours and accessing sensitive resources, and correlate abnormal user behaviors with network traffic and system logs for analysis.

[0042] As a preferred embodiment of the present invention, the perception model building module 300 includes: The random forest construction unit is used to construct a random forest. Multiple sample subsets are randomly extracted from the training set with replacement. Each subset is used to train a decision tree. At the node of each decision tree, some features are randomly selected for node splitting to increase the diversity of the model. A decision tree training unit, used to train each decision tree using the selected features and sample subsets until a predetermined stopping condition is reached, wherein the stopping condition is the maximum depth of the tree or the minimum number of node samples; The perception model building unit is used to build security situation assessment logic based on the output results of the random forest to obtain a security situation perception model.

[0043] As a preferred embodiment of the present invention, the perception model evaluation module 500 includes: A prediction result determination unit is used to input the data in the test set into the security situation awareness model, make predictions, and output the prediction results of each sample; The perception model evaluation unit is used to calculate the accuracy, recall rate and F1 score based on the prediction results, obtain the evaluation results, and display the performance of the model by drawing the ROC curve and PR curve.

[0044] The above only describes in detail the preferred embodiments of the present invention, which is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0045] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0046] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0047] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A network security perception method based on big data, characterized in that: The method comprises the following steps: Collect multi-source data, pre-process the collected data, and perform data fusion. The fusion result = ∑(value of data source i × weight i), where i represents each data source; Perform classification analysis, cluster analysis, and association rule mining on the fused data, extract feature information associated with security situation assessment from the processed data, the feature information includes network traffic features, system log features, and user behavior features, and divide the feature information into a training set and a test set; Building a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; The security situation awareness model is trained based on the training set, the performance of the model is evaluated through cross-validation method, and the parameters are tuned; The security situation awareness model is evaluated based on the test set, and evaluation indicators are calculated. The evaluation indicators include accuracy, recall rate and F1 score.

2. The network security perception method based on big data according to claim 1 is characterized in that: The steps of performing classification analysis, cluster analysis and association rule mining on the fused data specifically include: Training a classifier based on a decision tree, and performing classification analysis on the fused data through the classifier; The fused data is input into the clustering algorithm K-Means to obtain the clustering results, and the quality of the clustering results is evaluated based on the silhouette coefficient; The fused data is converted into shopping basket data, the Apriori algorithm is used to mine the association rules in the data, and the association rules are screened according to the support and confidence thresholds.

3. The network security perception method based on big data according to claim 2 is characterized in that: The step of inputting the fused data into the clustering algorithm K-Means to obtain the clustering result, and evaluating the quality of the clustering result based on the silhouette coefficient specifically includes: The fused data is standardized to obtain standardized data; Each data point in the standardized data is calculated using the weighted Mahalanobis distance discriminant method to obtain the weighted Mahalanobis distance between each pair of data points; Randomly select a data point as the first center from the standardized data, and select several center points by calculating the probability of each data point becoming the next center point based on the Euclidean distance between the existing center point and all data points; According to the weighted Mahalanobis distance between each pair of data points, the data points are assigned to the cluster to which the adjacent center points belong to perform the assignment operation; The mean of all data points in each cluster is used as the new center point to update the center point. If an empty cluster appears, a data point is randomly selected from the standardized data as the new center point. The allocation operation and the center point update operation are repeated in an iterative manner to obtain the clustering result; The difference between the distance between each data point and other points in its cluster and the distance from the data point to the nearest cluster is taken as the silhouette coefficient; By comparing the silhouette coefficients under each number of clusters, the number of clusters that can maximize the silhouette coefficient is selected as the optimal number of clusters to obtain the optimized clustering result.

4. The network security perception method based on big data according to claim 3 is characterized in that: The steps of converting the fused data into shopping basket data, mining association rules in the data using the Apriori algorithm, and screening the association rules according to support and confidence thresholds specifically include: Step S2021, dividing the shopping basket data into a plurality of transaction lists, each transaction list consisting of a plurality of commodities; Step S2022, traverse all transactions, and form the first candidate item set with the single commodities appearing in all transactions; Step S2023, counting the number of transactions of each item set in the first candidate item set, and dividing the number of transactions of a single item set by the total number of transactions to obtain the support of the corresponding item set; Step S2024, setting a support threshold, and screening each item set in the first candidate item set according to the support threshold to determine the first item set; Step S2025, combining any two itemsets in the first itemset according to the number of transactions of each itemset in the first itemset to generate candidate frequent itemsets; Step S2026, repeating the operations from step S2023 to step S2025 in an iterative manner until no new frequent item sets can be generated, so as to obtain frequent item sets; Step S2027, splitting the itemsets in the frequent itemsets into antecedent items and consequent items, defining the antecedent items as rule conditions, and defining the consequent items as rule conclusions, and calculating the confidence of the rule conclusions under the rule conditions to calculate the correlation between the multiple commodities; Step S2028, setting a confidence threshold, filtering the confidence according to the confidence threshold, and then sorting the filtering results to obtain association rules between products.

5. The network security perception method based on big data according to claim 4 is characterized in that: The step of extracting characteristic information associated with security situation assessment from the processed data specifically includes: Extract network traffic features and identify abnormal traffic patterns, including sudden increases in traffic and unusual port accesses; Extract system log features and extract key information from system error logs, where the key information includes error type, error time, and error frequency; Extract user behavior features, identify abnormal behaviors such as logging in during non-working hours and accessing sensitive resources, and correlate abnormal user behaviors with network traffic and system logs for analysis.

6. The network security perception method based on big data according to claim 5 is characterized in that: The steps of constructing a security situation awareness model based on the random forest algorithm and evaluation indicators specifically include: Construct a random forest, randomly extract multiple sample subsets from the training set with replacement, and use each subset to train a decision tree. At the node of each decision tree, randomly select some features for node splitting to increase the diversity of the model. Using the selected features and sample subsets to train each decision tree until a predetermined stopping condition is reached, wherein the stopping condition is the maximum depth of the tree or the minimum number of node samples; Based on the output results of random forest, the security situation assessment logic is constructed to obtain the security situation awareness model.

7. The network security perception method based on big data according to claim 6 is characterized in that: The steps of constructing a random forest, randomly extracting multiple sample subsets from a training set with replacement, each subset is used to train a decision tree, and randomly selecting some features at the nodes of each decision tree for node splitting to increase the diversity of the model, specifically include: Step S3041, obtaining a training set and a feature set corresponding to the training set, randomly extracting multiple sample subsets from the training set with replacement, and assigning the same initial weight to each sample subset; Step S3042, construct a decision tree for each sample subset, input the sample subset into a single decision tree, and randomly select some features from the feature set at the node of each decision tree to perform node splitting to perform single decision tree training; Step S3043, after the training of a single decision tree is completed, the sample classification error is obtained in the decision tree after the training without selecting the sample input; Step S3044, adjusting the initial weights of the corresponding sample subsets according to the sample classification errors to obtain adjusted sample weights, and extracting multiple new sample subsets from the training set again with replacement based on the adjusted sample weights; Step S3045, repeating steps S3042 to S3044 in an iterative manner to train the random forest.

8. The network security perception method based on big data according to claim 7 is characterized in that: The step of evaluating the security situation awareness model based on the test set and calculating the evaluation index specifically includes: Input the data in the test set into the security situation awareness model, make predictions, and output the prediction results for each sample; Based on the prediction results, the accuracy, recall and F1 score are calculated to obtain the evaluation results, and the performance of the model is displayed by drawing the ROC curve and PR curve.

9. A network security perception device based on big data, the device applying the network security perception method based on big data according to any one of claims 1 to 8, characterized in that: The device comprises: The multi-source data acquisition module is used to collect multi-source data, pre-process the collected data, and perform data fusion. The fusion result = ∑ (value of data source i × weight i), where i represents each data source; The data in-depth analysis module is used to perform classification analysis, cluster analysis and association rule mining on the fused data, extract feature information associated with security situation assessment from the processed data, the feature information includes network traffic features, system log features and user behavior features, and divide the feature information into training sets and test sets; A perception model building module is used to build a security situation awareness model based on a random forest algorithm and evaluation indicators, wherein the evaluation indicators include intrusion detection rate, false alarm rate, missed alarm rate and response time; The perception model training module is used to train the security situation awareness model based on the training set, evaluate the performance of the model through the cross-validation method, and perform parameter tuning; The perception model evaluation module is used to evaluate the security situation awareness model based on the test set and calculate evaluation indicators, which include accuracy, recall rate and F1 score.

10. The network security perception device based on big data according to claim 9 is characterized in that: The data depth analysis module includes: A data classification and analysis unit, used to train a classifier based on a decision tree, and to perform classification and analysis on the fused data through the classifier; The data clustering analysis unit is used to input the fused data into the clustering algorithm K-Means to obtain the clustering results and evaluate the quality of the clustering results based on the silhouette coefficient; The association rule mining unit is used to convert the fused data into shopping basket data, use the Apriori algorithm to mine the association rules in the data, and screen the association rules according to the support and confidence thresholds.

Images