Network attack classification method based on network security data cleaning and fusion

Abstract

The application discloses a network attack classification method based on network security data cleaning and fusion, relates to the field of network security, and calculates information entropy of each sub attribute space as the weight of the sub attribute space by cleaning data of multi-source heterogeneous network security data; constructs a hidden Markov model for each sub attribute space and trains the hidden Markov model; takes the output results of k trained sub Markov models as k pieces of evidence; inputs attribute sequence values of to-be-tested data of a network attack into the Markov model, and obtains the probability of occurrence of each attack result; according to the probability of occurrence of each attack result, the weight of the above sub attribute space is used for weighted calculation, the k pieces of evidence are fused by using D-S evidence theory; the fused data are judged based on a trust function of the D-S evidence theory, and the network attack is classified according to the judgment result.

Description

Technical Field

[0001] This invention relates to the field of network security, and specifically to a method for identifying network attack categories based on the cleaning and fusion decision-making of multi-source heterogeneous data. Background Technology

[0002] In recent years, with the rapid development of mobile internet, cloud computing, information security, machine learning, and other technologies, hundreds of millions of users generate massive amounts of data every day. Through technologies such as data mining, this large-scale data can be applied to many different fields, bringing convenience to people's lives. However, massive amounts of data are also accompanied by the problem of multi-source heterogeneity, resulting in inconsistent data quality. Data quality is a bottleneck restricting data use; without high-quality data, there are no high-quality data mining results. As tools for processing multi-source heterogeneous data, data cleaning and data fusion are important technologies for improving data quality and have significant value and meaning. Traditional data cleaning and fusion methods are no longer suitable for the needs of modern technological society, especially as the data sources in today's large-scale cybersecurity projects increasingly exhibit multi-source characteristics, necessitating the updating of data cleaning and fusion technologies.

[0003] The cleaning and fusion process of multi-source data mainly includes the following steps: 1. Multi-source data acquisition process; 2. Multi-source data preprocessing process; 3. Multi-source data fusion calculation process.

[0004] Multi-source data in the cybersecurity field exhibits characteristics distinct from multi-source data in other fields. In cybersecurity projects, data typically needs to be acquired from security data, log data, and traffic data from different vendors. Due to vendor differences, the cybersecurity equipment used varies, and even the software settings of the same equipment differ, resulting in a lack of uniformity among multi-source data and each exhibiting its own unique characteristics. To utilize cybersecurity data, data preprocessing is necessary, tailored to the characteristics of different data sources. The quality of cybersecurity data is fundamental to various research projects within the cybersecurity field. Data preprocessing is the first step after obtaining heterogeneous multi-source cybersecurity data. Given the massive volume of data today, originating from diverse data sources, the possibility of data anomalies has greatly increased. High-quality data is the foundation for better results and performance; therefore, the importance of data cleaning and fusion is growing, and it is a fundamental step in all project and platform construction. Cybersecurity data, after cleaning and fusion, can be applied to statistical analysis, machine learning, artificial intelligence, and other fields to analyze the current cybersecurity landscape and improve the cybersecurity protection capabilities of individuals, businesses, and nations. During data collection, various factors may interfere with data quality, such as the accuracy of data collection, the completeness of the data itself, and the consistency of data during aggregation from multiple sources.

[0005] Existing data cleaning and fusion technologies include supplementing missing values, processing interfering data, and integrating heterogeneous data from multiple sources. Due to the unique characteristics of cybersecurity data—multiple data sources and a wide range of applications—it is an important task to conduct reasonable data fusion for cybersecurity data to classify cybersecurity events. This requires collecting and fusing cybersecurity status data from multiple heterogeneous sources for cybersecurity diagnosis, in order to grasp the overall cybersecurity situation. Summary of the Invention

[0006] The purpose of this invention is to propose a network attack classification method based on network security data cleaning and fusion. First, data cleaning is carried out in stages according to the characteristics of network security data. Then, using the traditional DS evidence theory combined with the Markov model, data in the field of network security is cleaned and fused to identify the classification of network attacks, thereby realizing overall network security situation awareness.

[0007] The technical solution adopted in this invention is as follows:

[0008] A network attack classification method based on network security data cleaning and fusion includes the following steps:

[0009] 1) Collect multi-source heterogeneous network security data and perform data cleaning, merging, and data transformation;

[0010] 2) Divide the attribute space of the transformed data into k sub-attribute spaces according to the source and nature of the data, calculate the information entropy of each sub-attribute space, and use the information entropy as the weight of the sub-attribute space;

[0011] 3) Construct a hidden Markov model (i.e. a Markov model with hidden unknown parameters) for each sub-attribute space, and train the Markov model for each sub-attribute space to obtain k trained Markov models.

[0012] 4) Use the outputs of k trained Markov models as k evidence bodies;

[0013] 5) Input the attribute sequence values ​​of the network attack test data into k trained Markov models to obtain the probability of each attack result.

[0014] 6) Based on the probability of each attack result, perform weighted calculation using the weights of the aforementioned sub-attribute space, and use DS evidence theory to perform data fusion on the k evidence bodies;

[0015] 7) The trust function based on DS evidence theory is used to judge the fused data, and the network attack is classified according to the judgment result.

[0016] Furthermore, data cleaning includes the following steps:

[0017] Fill in the missing data in the incomplete data;

[0018] Perform deduplication on duplicate data;

[0019] Error data processing methods include binning, clustering to remove outliers, and establishing regression functions to smooth the data, or one or more of these methods.

[0020] Integrate the data table structure and field types of multi-source heterogeneous data into a unified format;

[0021] Delete data that is not needed.

[0022] Furthermore, data transformation includes one or more of the following: data normalization, reduction, switching, rotation, and projection.

[0023] Furthermore, the attribute sequence of data refers to the sequence of all attributes of a single data point; the attribute space is a matrix composed of attribute sequences of multiple data points, denoted as A. m*n ={A1,…,A i A m}, where attribute sequence A i ={A i (1), A i (2), ...,A i (n)},A i (1), A i (2), ...,A i (n) represents A i The n attributes; m represents m pieces of network security source data.

[0024] Furthermore, each sub-attribute space is a power set consisting of several attributes that influence the classification of network security events.

[0025] Furthermore, the information entropy is calculated according to the following formula:

[0026] H(X)=∑ i P(x i )I(x i )=-∑ i P(x i )log b P(x i );

[0027] Where H(X) represents the information entropy of random event X, and P(x) represents the information entropy of random event X. i ) represents a random event as I(x) i The probability of (i) is given by (b) where i represents the number of random events and b is the base of the logarithm.

[0028] Furthermore, based on the definition of each parameter in each sub-attribute space, training samples for each Markov model are obtained from different sources, and each Markov model is trained.

[0029] Furthermore, a Markov model with optimal parameters is trained and tested on a test set to obtain a well-trained Markov model.

[0030] Furthermore, the DS evidence theory framework is expressed as F = {F1, F2, F3, ..., F...} k}, where k represents the number of sub-attribute spaces, F k This represents the attack category corresponding to the k-th sub-attribute space.

[0031] Furthermore, in step 5), the probability of the attack result occurring is obtained according to the following formula:

[0032]

[0033] Where, p i F represents the probability of the attack outcome occurring. j H represents the attack result. i (F j ) indicates that evidence H i Assigning attack result F j The probability, Indicates k pieces of evidence H i Assigning attack result F j The sum of probabilities. Attached Figure Description

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] Figure 1 This is a flowchart of the network attack classification method based on network security data cleaning and fusion of the present invention. Detailed Implementation

[0036] The network attack classification method based on network security data cleaning and fusion proposed in this invention mainly includes two parts: constructing and training a Markov model and classifying network attacks. Specifically, it includes the following steps:

[0037] I. Constructing and Training Markov Models

[0038] Step 1: Data Cleaning. This step involves preprocessing the multi-source data, including removing noisy and irrelevant data, addressing potential omissions and dirty data, and specifically handling incomplete or outlier data. This includes processing incomplete and outlier data, smoothing noisy data, identifying and removing outliers, and resolving data inconsistencies to improve the quality, completeness, and accuracy of the collected data. Data cleaning includes, but is not limited to, the following:

[0039] 1) It should support the handling of missing values ​​in the collected data, such as missing data due to equipment failure, or some data that was not entered because it was not taken seriously during input;

[0040] 2) It should support the handling of abnormal data collected. Abnormal data includes duplicate data and erroneous data. Duplicate data may occur when merging multiple data sources. Erroneous data may be due to the business system not being robust enough, and the input is written directly to the backend database without being judged. Random errors or deviations may be introduced into a measurement variable, such as inaccurate date format.

[0041] 3) It should support the cleaning of the format and content of the collected data. Data from multiple data sources may differ in the data table structure definition or field types. When integrating data from multiple sources, it is necessary to process them into a consistent format.

[0042] 4) Non-required data cleaning should be supported in the collected data. In order to avoid data redundancy, after a reasonable evaluation of the platform business, non-required data in the collected data needs to be deleted.

[0043] Step Two: Data Integration. This mainly involves merging heterogeneous data from multiple file or database environments to resolve semantic ambiguity. By logically or physically centralizing data from different sources, formats, and characteristics, comprehensive data sharing is provided to the platform. Data from multiple data sources often exhibits inconsistencies in structure, naming, units, and meaning. Data integration requires unifying and standardizing the data. This part primarily involves data selection, resolving data conflicts, and handling inconsistencies.

[0044] Step 3: The main task is to find the feature representation of the data. By calculating the attribute similarity, feature fusion is performed on the original data. The grey relational analysis method can be used to aggregate data with high attribute similarity, eliminate redundant data, and achieve effective compression of network security data.

[0045] Step 4: Divide the data attribute space of multi-source heterogeneous network security data into k sub-attribute spaces according to the source and nature of the data, and calculate the information entropy of each sub-space according to the information entropy calculation formula, and use it as the weight of the sub-attribute space.

[0046] Step 5: Construct a Hidden Markov Model for each sub-attribute space, and train each Markov Model to obtain k learnable Markov Models.

[0047] Step 6: Construct k evidence bodies using the outputs of k learnable Markov models, which will serve as inputs for evidence fusion.

[0048] II. Classification of Cyber ​​Attacks

[0049] Step 7: Input the attribute sequence of the network attack data to be tested into the Markov model and calculate the probability of each attack outcome;

[0050] Step 8: Based on the probability of each attack outcome, fuse these k pieces of evidence using the DS evidence theory, and then make a decision based on the fused data to classify the network attack.

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0052] I. Constructing and Training Markov Models

[0053] 1) Data cleaning stage

[0054] a) Regarding incomplete data, the data entered into the database is not always complete. There are many reasons for missing values, such as equipment malfunction, data that was not entered due to misunderstanding, or data that was not entered because it was not taken seriously. In these cases, the missing values ​​should be filled in after inference.

[0055] b) Regarding anomalous data, which includes duplicate and erroneous data, duplicate data may occur when merging multiple data sources. It is necessary to identify and delete duplicate data, retaining only unique data units. Erroneous data may be due to inadequate business systems, where input is directly written to the backend database without proper evaluation, introducing random errors or biases into a measurement variable, such as inaccurate date formats. Erroneous data can be handled using one or more methods, such as binning, clustering to remove outliers, and establishing regression functions to smooth the data.

[0056] c) Format and content cleaning: When integrating data from multiple data sources, the data from different sources may be inconsistent in terms of format and content. A semi-automatic and semi-manual verification method can be used to find potential problems.

[0057] d) Cleaning non-required data: To avoid data redundancy, after a reasonable assessment of the platform's business, it is necessary to delete non-required data from the collected data.

[0058] 2) Markov Model Construction and Training Stage

[0059] a) Suppose there are m data entries in the data network security source data, and each data entry has n attributes. A i Let A be the attribute sequence of the i-th data. Then the attribute sequence A of each data is... i ={A i (1), A i (2), ..., A i (n)}, where the elements within the parentheses are, for example, A. i (1) represents an attribute, and the attribute space is a matrix A. m*n ={A1,…,A i ,…,A m};

[0060] b) Based on the nature of multiple data sources, all attributes affecting the classification of network security incidents are divided into k sub-attribute spaces S = {S1, S2, S3, ..., S...} according to certain rules. k}, where the elements within the parentheses, such as S1, represent a sub-attribute space, which is a power set composed of several attributes that affect the classification of network security events;

[0061] c) According to the information entropy calculation formula H(X)=∑ i P(x i )I(x i )=-∑ i P(x i )log b P(x i (where P(x) i ) represents a random event X as I(x i The probability of ) is calculated; b is the base of the logarithm, which is chosen according to the actual situation; i represents the number of random events. The information entropy of each attribute in each sub-attribute space is calculated, and the information entropy is summed to obtain the sub-attribute space S. j The information entropy is used as the weight of the sub-attribute space, resulting in a weight vector W = {W1, W2, W3, ..., W...}. k}, where the elements within the parentheses, such as W1, are the weights corresponding to sub-attribute spaces such as S1;

[0062] d) Based on the definition of each parameter in each sub-attribute space, obtain training samples for each Markov model from different sources to form a training set;

[0063] e) Construct k Markov models and train each Markov model using the training set to obtain k Markov models H = {H1, H2, H3, ..., H...} with optimal parameters. k};

[0064] f) Validate each Markov model using the test set. Only models that pass validation can be used.

[0065] II. Classification of Cyber ​​Attacks

[0066] g) Establish a multi-category DS evidence theoretical framework F = {F1, F2, F3, ..., F...} k}, where k represents the number of sub-attribute spaces, F k This represents the attack category corresponding to the k-th sub-attribute space;

[0067] h) Use the outputs of k Markov models as k evidence bodies, and use them as inputs to the DS evidence theory framework;

[0068] i) By using the attribute sequence values ​​of the data to be tested as input to each Markov model, the probability of each attack outcome can be obtained. After standardization, the model H can be... i Assigning attack result F j The probability is Among them, the molecule represents the evidence H i Assigned to result F j The probability of H is given by the denominator, which represents the probability of all (k) pieces of evidence H. i Assigned to result F j The sum of probabilities.

[0069] j) Based on the probability of each outcome, use the previously calculated sub-attribute space weight vector W = {W1, W2, W3, ..., W...} k We perform weighted calculations and use the DS evidence theory framework to fuse the evidence bodies in each sub-attribute space (or evidence fusion).

[0070] k) The trust function based on DS evidence theory is used to make decisions on the fused data to obtain the final classification result for network security incidents.

[0071] The foregoing description illustrates the technical approach of this invention, intended to enable those skilled in the art to understand and implement the invention, and should not be construed as limiting the scope of protection of this invention. All equivalent modifications or alterations made in accordance with the spirit and essence of this invention should be included within the scope of protection of this invention.

Claims

1. A network attack classification method based on network security data cleaning and fusion, characterized in that, Includes the following steps: 1) Collect multi-source heterogeneous network security data and perform data cleaning, merging, and data transformation; 2) Divide the attribute space of the transformed data into k sub-attribute spaces according to the source and nature of the data, calculate the information entropy of each sub-attribute space, and use the information entropy as the weight of the sub-attribute space; 3) Construct a hidden Markov model for each sub-attribute space, and train the Markov model for each sub-attribute space to obtain k trained Markov models. 4) Use the outputs of k trained Markov models as k evidence bodies; 5) Input the attribute sequence values ​​of the network attack test data into k trained Markov models to obtain the probability of each attack result. 6) Based on the probability of each attack result, a weighted calculation is performed using the weights of the aforementioned sub-attribute space, and the k evidence bodies are fused using the DS evidence theory. 7) The trust function based on DS evidence theory is used to judge the fused data, and the network attack is classified according to the judgment result.

2. The method as described in claim 1, characterized in that, Data cleaning includes the following steps: Fill in the missing data in the incomplete data; Perform deduplication on duplicate data; Error data processing methods include binning, clustering to remove outliers, and establishing regression functions to smooth the data, or one or more of these methods. Integrate the data table structure and field types of multi-source heterogeneous data into a unified format; Delete data that is not needed.

3. The method as described in claim 1, characterized in that, Data transformation includes one or more of the following: data normalization, reduction, switching, rotation, and projection.

4. The method as described in claim 1, characterized in that, A data attribute sequence refers to the sequence of all attributes of a single data point; an attribute space is a matrix composed of attribute sequences of multiple data points, represented as follows: Among them, attribute sequence , express The n attributes; m represents m pieces of network security source data.

5. The method as described in claim 1, characterized in that, Each sub-attribute space is a power set consisting of several attributes that influence the classification of network security events.

6. The method as described in claim 1, characterized in that, Calculate information entropy using the following formula: ； in, The information entropy represents the random event X. Representing a random event as The probability is given by , where i represents the index of the possible values ​​of the random event. The base of the logarithm is .

7. The method as described in claim 1, characterized in that, Based on the definition of each parameter in each sub-attribute space, training samples for each Markov model are obtained from different sources, and each Markov model is trained.

8. The method as described in claim 7, characterized in that, A Markov model with optimal parameters is obtained through training, and the model that passes the test on the test set is considered a well-trained Markov model.

9. The method as described in claim 1, characterized in that, The DS evidence theory framework is expressed as follows: Where k represents the number of sub-attribute spaces, F k This represents the attack category corresponding to the k-th sub-attribute space.

10. The method as described in claim 1, characterized in that, In step 5), the probability of the attack result occurring is obtained according to the following formula: ； in, This indicates the probability of the attack outcome occurring. F j Indicates the result of the attack. Indicate that the evidence H i Assigned to attack results F j The probability, Indicates k pieces of evidence H i Assigned to attack results F j The sum of probabilities.

Images