Network traffic anomaly detection method and system based on knowledge graph

Abstract

The invention relates to the technical field of network security, in particular to a network traffic anomaly detection method and system based on a knowledge graph, and the method comprises the steps: extracting entities in a traffic data set, relationships between the entities and entity attributes, and constructing a traffic information knowledge graph based on the extracted entities, relationships between the entities and entity attributes; storing the flow information knowledge graph into a graph database; and obtaining a feature similarity score between the to-be-detected unknown traffic and the traffic in the stored traffic information knowledge graph by adopting a collaborative filtering algorithm, and determining the type of the to-be-detected unknown traffic according to the feature similarity score. According to the method, the traffic information is stored in the graph database Neo4j, so that the relationship of characteristic values among the traffic is clearer and clearer, the operation of security personnel is facilitated, and the visualization effect is improved; through matching of intervals where flow characteristic values in the knowledge graph are located, the flow range participating in calculation of an anomaly detection algorithm is narrowed, and the calculation efficiency is improved; and by fusing the weight coefficients and comprehensively considering the two similarity results, the accuracy of the detection result is improved.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network traffic anomaly detection method and system based on knowledge graph. Background Art

[0002] With the development of information technology, the Internet is becoming more and more closely connected with life, and individuals' demand for the use of the Internet is also increasing. As Internet traffic surges, the means of using malicious traffic to carry out network attacks are also constantly updated and iterated, which has become one of the issues that cannot be ignored in network security. How to identify and detect malicious traffic in massive data is also a hot topic that security departments in enterprises and institutions continue to pay attention to. Especially in recent years, the concealment, diversity, and systematic nature of malicious traffic have increased, and traditional data packet and data flow detection methods can no longer meet security needs.

[0003] At present, the main problems in the field of abnormal traffic detection are as follows: 1. Frequent false alarms and misjudgments. The means used by malicious attacks are becoming more and more diverse, and the types of malicious traffic generated are also increasing. The use of a single detection method may lead to misjudgments under certain attack methods, bringing loopholes to real security threats. 2. The problem of resource occupation is serious. In the actual operation process, due to the large amount of traffic information, abnormal traffic detection requires a lot of computing resources. If it is not screened, it may bring a burden to some low-configuration devices or systems. 3. Complex feature engineering. Abnormal data is often complex and difficult to describe using traditional feature representation methods. In addition, for traffic information, the feature dimension is large. How to select effective features and perform reasonable preprocessing is an important challenge in anomaly detection. Summary of the invention

[0004] To this end, the present invention provides a network traffic anomaly detection method and system based on knowledge graph to solve the problems of unsatisfactory accuracy and insufficient feature consideration of existing abnormal traffic detection.

[0005] According to the design scheme provided by the present invention, on the one hand, a network traffic anomaly detection method based on knowledge graph is provided, comprising:

[0006] Extract entities, relationships between entities and entity attributes from the traffic data set, build a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities using traffic numbers to distinguish traffic information in the data set, indicator entities using traffic feature indicators to determine whether traffic information is abnormal, and standard entities using traffic indicator value ranges to associate traffic entities;

[0007] A collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and the type of the unknown traffic to be measured is determined based on the feature similarity score.

[0008] As the network traffic anomaly detection method based on knowledge graph of the present invention, further, the traffic information knowledge graph is represented as G<L,I,S,P,R> , G represents the knowledge graph, L represents the label entity, I represents the traffic entity, S represents the indicator entity, P represents the standard entity, and R represents the relationship set between the entities.

[0009] As a network traffic anomaly detection method based on knowledge graph of the present invention, further, entities, relationships between entities and entity attributes in the traffic data set are extracted, including:

[0010] According to the flow data set, label entities, flow entities, indicator entities and standard entities are extracted. The attributes of the label entities include flow label attributes for determining flow types and attack type attributes for determining what kind of attack causes abnormal flow; the attributes of the flow entities include flow numbers for identifying flow and timestamps for recording force generation time; the attributes of the indicator entities include feature indicator names, feature indicator contents and feature indicator values; the attributes of the standard entities include discrete values ​​of flow indicators or continuous value intervals of flow indicators;

[0011] Extract the association relationship between entities based on the attribute information of different entities.

[0012] As a network traffic anomaly detection method based on knowledge graph of the present invention, further, a collaborative filtering algorithm is used to obtain a feature similarity score between the unknown traffic to be tested and the traffic in the stored traffic information knowledge graph, including:

[0013] Import the unknown traffic to be measured into the traffic information knowledge graph based on the entity attribute relationship;

[0014] According to the unknown traffic characteristics to be tested, any three traffic characteristics with values ​​in the same interval are extracted from the traffic information knowledge graph, and the extracted known traffic types are used as candidate matching traffic;

[0015] Based on the traffic feature similarity, a feature similarity score between the unknown traffic to be tested and the candidate matching traffic is obtained.

[0016] As the network traffic anomaly detection method based on knowledge graph of the present invention, further, obtaining the feature similarity score between traffics includes:

[0017] Obtaining spatial similarity and feature similarity between traffic flows, wherein the spatial similarity is calculated using Euclidean distance, and the feature similarity is calculated using cosine similarity;

[0018] The spatial similarity and feature similarity are fused using the weight coefficient to obtain the final similarity score between flows.

[0019] As the network traffic anomaly detection method based on knowledge graph of the present invention, further obtaining the spatial similarity and feature similarity between traffic also includes:

[0020] The features of traffic information are normalized using the sum of the values ​​of traffic features in the traffic information knowledge graph.

[0021] As a network traffic anomaly detection method based on knowledge graph of the present invention, further, the process of fusing spatial similarity and feature similarity using weight coefficient is expressed as:

[0022] sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j ), where sim KG and sim u The flow rate I i ,I j The spatial similarity and feature similarity between them, a and b are the fusion weight coefficients.

[0023] In another aspect, the present invention further provides a network traffic anomaly detection system based on a knowledge graph, comprising: a graph construction module and an anomaly detection module, wherein:

[0024] A graph construction module is used to extract entities, relationships between entities and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities using traffic numbers to distinguish traffic information in a data set, indicator entities using traffic feature indicators to determine whether traffic information is abnormal, and standard entities using traffic indicator value ranges to associate traffic entities;

[0025] The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be tested and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be tested based on the feature similarity score.

[0026] Beneficial effects of the present invention:

[0027] 1. The present invention utilizes the knowledge graph technology in the field of artificial intelligence to construct a complete and visualized traffic knowledge graph by analyzing known traffic information, so as to more clearly display the relationship between different entities, attributes, and entities, so as to better grasp the changes in traffic; when there is traffic input of unknown classification, it looks for the connection between it and the known traffic, and uses the similarity calculation idea in the knowledge graph recommendation algorithm to perform association calculation to obtain traffic information with a greater correlation with the unknown classification traffic, which is used as the basis for classification, providing important help for discovering and analyzing abnormal network traffic.

[0028] 2. By storing traffic information in the graph database Neo4j, the relationship between the characteristic values ​​of traffic is made clearer and more explicit, which is convenient for security personnel to operate and improves the visualization effect. By matching the intervals of traffic characteristic values ​​in the knowledge graph, the traffic range involved in the calculation of the anomaly detection algorithm is narrowed, improving the calculation efficiency. By integrating the weight coefficient, the two similarity results are comprehensively considered to improve the accuracy of the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the network traffic anomaly detection process based on the knowledge graph in the embodiment;

[0030] Figure 2 This is a schematic diagram of a tag entity in the embodiment;

[0031] Figure 3 This is a schematic diagram of a traffic entity in the embodiment;

[0032] Figure 4 This is a schematic diagram of an indicator entity in the embodiment;

[0033] Figure 5 This is a schematic diagram of a standard entity class for taking values ​​of discrete indicators in the embodiment;

[0034] Figure 6 Schematic diagram of the numerical distribution of flow information indicators in the embodiment;

[0035] Figure 7 This is a schematic diagram of a standard entity class for a continuous indicator interval in the embodiment;

[0036] Figure 8 This is a schematic diagram of the knowledge graph ontology of traffic information in the embodiment;

[0037] Fig. 9 This is a schematic diagram of the conversion relationship between the traffic information structure and the user commodity structure in the embodiment;

[0038] Fig.10 This is a diagram showing the relationship between the fusion weight coefficient value and the detection accuracy in the embodiment;

[0039] Fig.11This is a schematic diagram of the test data set after preprocessing in the embodiment;

[0040] Fig.12 Schematic diagram of the abnormality detection experiment results in the embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0042] In recent years, the means of using traffic to carry out network attacks have become increasingly complex and diverse, and the concealment and encryption methods of malicious traffic itself have also been continuously enhanced, posing a considerable challenge to the field of anomaly detection. Traditional abnormal traffic detection technology mainly determines whether the traffic is abnormal by comparing it with the local abnormal traffic feature library; after the introduction of machine learning algorithms, the accuracy of detection has been greatly improved, but there are still problems such as insufficient consideration of abnormal feature selection and poor learning effect. For this reason, the embodiments of the present invention, see Figure 1 As shown, a network traffic anomaly detection method based on knowledge graph is provided, which specifically includes:

[0043] S101. Extract entities, relationships between entities and entity attributes from the traffic data set, build a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a data set, indicator entities that use traffic feature indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator numerical ranges to associate traffic entities.

[0044] The traffic information knowledge graph mainly involves entities in four dimensions, namely G<L,I,S,P,R> , which are label entity L, flow entity I, indicator entity S, standard entity P and the relationship between each entity R. Label entity is used to represent the category of a flow information, including normal flow and abnormal flow. Among them, abnormal flow includes detailed classifications such as BForce, SFTP and SSH, DDOS LOIT, etc.; flow entity represents each flow information in the data set, which is distinguished by the flow number; indicator entity is used to judge whether the flow information is abnormal, including 7 indicators including two main features obtained by principal component analysis; standard entity is a numerical range. Through this entity, two flow information entities with the same indicator value can be associated. After extracting and fusing the above data, the construction of the flow information knowledge graph can be completed.

[0045] In the principal component analysis method, the generated principal components are weighted combinations of multiple original variables, which can reflect the impact of most variables in the original data on the results. Suppose the characteristic values ​​of p flows are x1, x2, …, x p The corresponding feature value of the i-th flow information is x i1 ,x i2 ,…,x ip (i=1,2,…,n), the original matrix obtained is

[0046]

[0047] First, since the range and size of different variable values ​​may vary greatly, the target matrix must be standardized first. The resulting matrix is ​​recorded as x*, and the calculation formula is:

[0048]

[0049] in, is a data in the matrix x*, For variable x j The average value of the observed values; s jj For variable x j The variance of the observations.

[0050] Then the correlation coefficient matrix R is calculated

[0051] R=(r op ) m*n (o,p=1,2,...,m)

[0052] r op and r po is a symmetric matrix, and the correlation coefficient between indicators is r op The calculation formula is

[0053]

[0054] Finally, the number of principal components is determined, usually taking the features with cumulative variance contribution greater than 85% and number greater than 1. The eigenvalue is recorded as Cumulative variance contribution rate v t The calculation formula is

[0055]

[0056] According to the algorithm principle, the research data set can be calculated by writing a Python algorithm. When the threshold of the impact on the result is 85%, a total of 15 principal components are obtained. Due to the limitation of personal computing resources, if all principal components are considered together, a lot of computing time will be required. For simplicity, in the embodiment of this case, the threshold range is narrowed and the first two principal components that dominate the 15 principal components are selected.

[0057] The principal component obtained by principal component analysis is the weighted calculation of the original components, which has no practical significance. If it is directly used as the main feature of traffic to construct the knowledge graph, the user's readability and the visualization performance of the graph will be greatly reduced. The purpose of principal component analysis is to reduce the dimension of traffic information and obtain a small number of factors that have a greater impact on whether it is abnormal among many features. Therefore, the original traffic features that account for the largest proportion of the above two principal components are used as the principal components of traffic information to construct the knowledge graph. The traffic information is compressed from the original 78 dimensions to 2 dimensions, namely Fwd PSH Flags and Fwd Pkts / s, to complete the dimensionality reduction of the traffic information. These two features will be used as key features to participate in the construction of the traffic information knowledge graph.

[0058] Neo4j is a NoSQL database based on graph storage written in Java. It is developed in Java and stores information in the form of a graph structure. The associated data itself is the data it contains, so it can directly display the characteristics of the associated data and the relationship between the data. It can also easily access highly connected data and has excellent performance, making it the preferred tool for storing knowledge graphs. Neo4j graph databases mainly include two types: nodes and edges, which represent entities or attribute values ​​and their relationships.

[0059] Among them, entities, relationships between entities and entity attributes extracted from the traffic data set can be designed to include:

[0060] According to the flow data set, label entities, flow entities, indicator entities and standard entities are extracted. The attributes of the label entities include flow label attributes for determining flow types and attack type attributes for determining what kind of attack causes abnormal flow; the attributes of the flow entities include flow numbers for identifying flow and timestamps for recording force generation time; the attributes of the indicator entities include feature indicator names, feature indicator contents and feature indicator values; the attributes of the standard entities include discrete values ​​of flow indicators or continuous value intervals of flow indicators;

[0061] Extract the association relationship between entities based on the attribute information of different entities.

[0062] The entities of the traffic information knowledge graph are divided into four parts according to the entity type:

[0063] The label entity can be extracted according to the traffic type contained in the research data set. There is a total of 1 normal traffic label (Benign) and 13 abnormal traffic labels. This entity has two attributes: traffic type and attack type. The traffic type is one of the 14 types mentioned above, which is used to determine the final traffic type; the attack type describes the type of attack that causes this type of traffic, and provides a basis for decision makers to further select defense strategies after detecting abnormal traffic. Figure 2 As shown in the figure, in the traffic information knowledge graph, label entities are represented by dark green circles, and the label name is displayed on the entity.

[0064] Each flow entity represents a piece of flow information. For the convenience of calculation, it is represented by the number of the flow information. Its attributes include flow ID and timestamp. The flow ID itself has no practical meaning and is only used to identify the flow to facilitate subsequent calculations; the timestamp records the time when the flow is generated. Since abnormal flow often appears in large quantities in a short period of time, the timestamp also has a certain reference value for the detection of abnormal flow. Figure 3 As shown, in the traffic information knowledge graph, traffic-related entities are represented by gray circles, and the traffic number is displayed on the entity for distinction.

[0065] There are 7 indicator entities, two of which are the main features Fwd PSHFlags and Fwd Pkts / s obtained by the principal component analysis method in Chapter 3. According to the best features of DoS attack research, another 6 features are determined as indicator entities, namely Dst Port, Protocol, Flow Duration, Bwd Pkt Len Std, Fwd IAT Std, and Pkt Size Avg. The attributes they contain are indicator name, indicator content, and indicator value. Among them, the indicator content explains the meaning of the indicator, which is convenient for users to understand and improves the readability of the traffic information knowledge graph. Figure 4 As shown in the figure, in the traffic information knowledge graph, indicator entities are represented by blue circles, and the indicator name is displayed on the entity.

[0066] The definition of standard entities is quite special. For discrete indicators, each value is a separate standard, such as the indicator Dst Port. Since port numbers are discrete values, each value of this indicator can generate a corresponding standard entity. Different indicators have different standard entities, which are distinguished by different colors in the traffic information knowledge graph. Figure 5 It is a standard class entity of the destination port. The number on the entity is the value of the destination port of the traffic information.

[0067] For continuous indicators, since their numerical values ​​fluctuate greatly, if the numerical intervals are divided evenly, most intervals contain very little or even no flow information. Therefore, the standard intervals can be divided in accordance with statistical rules. Taking the indicator Bwd Pkt Len Std as an example, Figure 6 As shown in the figure, the values ​​are statistically obtained. The horizontal axis of the statistical result is the number of the traffic information, and the vertical axis is the corresponding Bwd Pkt Len Std feature value. Figure 6 It can be seen that most of the values ​​of this feature fall within the range of less than 500, a small part falls within the range of 500-1500, and a very small part falls within the range of more than 1500. Therefore, in this embodiment, the standard interval of the Bwd Pkt Len Std feature can be defined as <500, 500-1500, and >1500. The traffic entities whose feature values ​​fall within the corresponding interval are constructed in the knowledge graph with the standard entities in the interval, thereby creating a connection between the traffic entities. Figure 7 It is a standard entity of flow duration, represented by a red circle in the flow information knowledge graph. The number on the entity identifies the interval represented by the entity.

[0068] By extracting information from different entities, we find that each dimension in the traffic information knowledge graph is closely related. For each piece of traffic information, there is a label indicating whether it is abnormal traffic and which type of abnormal traffic it belongs to; the label used to determine a piece of traffic is the value of each feature of the traffic itself. The values ​​of the features are all discrete or continuous values, which must fall within a certain interval, that is, there is a standard interval entity associated with it. Finally, the ontology diagram obtained is as follows: Figure 8 shown.

[0069] We selected CICIDS-2018, a relatively new data set for traffic research, and carried out knowledge extraction, knowledge fusion, knowledge representation and other steps on these data. They were stored in the graph database Neo4j in the form of entity, relationship and attribute triples. It can be obtained that there are currently 2632 traffic entities in the knowledge graph, and each traffic entity is associated with 8 indicator entities and 1 label entity. Each indicator entity is associated with a corresponding standard entity defined according to statistical laws to form the final traffic information knowledge graph. Different entities can be distinguished by different colors. For example, gray is used to represent traffic entities, blue is used to represent indicator entities, dark green is used to represent label entities, and the rest are standard entities of different indicators, which are distinguished by different colors.

[0070] S102. Use a collaborative filtering algorithm to obtain a feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be measured based on the feature similarity score.

[0071] Currently, newer anomaly detection methods mostly use machine learning or behavioral analysis-based technologies. The key is to extract the characteristics of abnormal or normal traffic through training with a large amount of abnormal traffic, encode its behavioral patterns into a model, compare it with the characteristic coding of unknown types of traffic, and determine which type of traffic it belongs to based on the degree of similarity.

[0072] The core of the recommendation algorithm also lies in feature extraction and similarity calculation. In recent years, with the rapid development of artificial intelligence technology, especially the introduction of knowledge graphs, connections have been built between users and users, and between users and products. Using association calculations, the results have been relatively accurate. Therefore, in the embodiments of this case, the collaborative filtering recommendation algorithm commonly used in knowledge graphs is used in the field of anomaly detection. Collaborative filtering algorithms are generally used for recommendations between users and products. How to correspond the relationship between traffic information and the user-product relationship becomes the key. The entities contained in the knowledge graph: traffic entities, indicator entities, and standard entities, correspond to product entities, product-related entities, and user entities in the user-product model, respectively. New label entities are added to identify the type of traffic. For example Fig. 9 As shown in the figure, taking the user-product structure of recommending movies to users as an example, in the user-product structure, user A likes movie A, actor A appears in movie A, and the actor also starred in movie B, then it can be considered that movie A and movie B have similarities. User B likes movie B, then it can be considered that there are similarities between user A and user B, and other movies that user B likes can be recommended to user A. Similarly, in the traffic information structure, if the flow duration values ​​of traffic A and traffic B are in the same range, it can be simply considered that traffic A and traffic B have similarities. If the type of traffic B is known, it can have a certain reference value for the type of A, that is, traffic A that may be similar to its type is recommended to traffic B. In the actual knowledge graph, there are multiple feature-type entities and standard-type entities, and the determination of the similarity between traffic will also be calculated through the improved collaborative filtering algorithm given later, which guarantees the accuracy of the results to a certain extent.

[0073] Among them, a collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, which can be designed to include:

[0074] Import the unknown traffic to be measured into the traffic information knowledge graph based on the entity attribute relationship;

[0075] According to the unknown traffic characteristics to be tested, any three traffic characteristics with values ​​in the same interval are extracted from the traffic information knowledge graph, and the extracted known traffic types are used as candidate matching traffic;

[0076] Based on the traffic feature similarity, a feature similarity score between the unknown traffic to be tested and the candidate matching traffic is obtained.

[0077] Collaborative filtering algorithm is an important similarity calculation algorithm. It simultaneously considers multiple similarity calculation results to filter out the key information needed from massive information. This algorithm is widely used to recommend items with high similarity to users. Its core is to analyze the user's historical behavior data on items to predict the user's preferences and provide relevant recommendations. These behavior data can be stored in matrix form, such as rating, browsing, and searching. Therefore, the accuracy of the algorithm's recommendations is directly affected by the accuracy and completeness of the massive user behavior data stored in the system.

[0078] With the rapid development of Internet technology, the interactive behaviors between hosts are increasing, and the traffic data generated is becoming more and more massive. On the other hand, among the many abnormal traffic, there will also be non-malicious abnormal traffic caused by hardware problems or improper operation, which will cause unnecessary burden on the calculation. The traditional matrix storage method can no longer effectively process and transmit such a large amount of data, resulting in reduced efficiency of matrix use. In addition, network traffic data is usually very sparse, and only a very small number of users will visit a large number of sites or applications. Traditional collaborative filtering algorithms may not be able to handle such sparse data, resulting in similarity scoring errors and affecting the recommendation effect. Because traditional collaborative filtering algorithms need to have enough data to discover similarities and differences between users.

[0079] In order to deal with the above problems, a new collaborative filtering algorithm is proposed in the embodiment of this case. By making full use of the characteristic that the knowledge graph has rich relationships, the algorithm's computational efficiency in processing big data is improved. Compared with the traditional collaborative filtering algorithm, this new algorithm can capture useful information more accurately, has a more reasonable similarity score calculation result, and effectively solves the challenges brought by data sparsity.

[0080] In order to solve the problem of low efficiency of big data operation faced by collaborative filtering algorithms, we combined the traffic information knowledge graph to solve it. After receiving the feature information of unknown type traffic, it is imported into the traffic information knowledge graph according to the defined entity attribute relationship. Since the existing traffic information knowledge graph contains a large amount of data, if the similarity score is calculated one by one with the unknown type traffic, the calculation amount is large and the efficiency is low. Therefore, the relationship between entities in the knowledge graph is used to narrow the range of data that needs to be involved in the calculation. According to all the features contained in the unknown type traffic, only the traffic with the same value interval as any three of its features is considered within the calculation range. In this way, it is equivalent to a preliminary screening of the traffic through the knowledge graph, which greatly improves the calculation efficiency of the subsequent collaborative filtering algorithm.

[0081] Aiming at the problem that traditional similarity calculation methods have errors when data is sparse, this paper corrects it by calculating the spatial similarity between traffic flows, and combines it with the feature value similarity based on traffic through a weight coefficient to obtain the final similarity score.

[0082] In the knowledge graph, each traffic entity is connected to several feature entities, and each feature entity has its specific value. If each feature entity is regarded as a dimension, a piece of traffic information can be represented by an m-dimensional vector:

[0083] H i =(E 1i ,E 2i ,…,E mi ) T

[0084] E di Representative flow H i The value of the mth feature of . Based on this, the spatial similarity between two flows in the flow information knowledge graph can be calculated by the Euclidean distance formula, that is, the Euclidean distance. As shown in the following formula.

[0085]

[0086] The Euclidean distance obtained here is a non-negative value, which can be constrained between (0,1] by the following formula:

[0087]

[0088] As the value increases, the spatial similarity between the two flows will continue to increase, up to 1; however, if this value is infinitely close to 0, it means that there is almost no commonality between them.

[0089] For actual traffic data sets, the numerical differences between the features of each dimension of traffic information are too large. If the calculation is performed directly, the result will be determined only by the features with larger values, thereby reducing or even ignoring the features with smaller value ranges. Therefore, before calculating the Euclidean similarity, the features of the traffic information need to be normalized separately, that is, the value of each feature is divided by the sum of the feature value in the traffic information knowledge graph. The sum of each feature in the traffic information knowledge graph can be obtained before the similarity calculation, and defined in the program in the form of a constant. It can be added to the corresponding feature value of the traffic to be detected and used as a divisor during calculation.

[0090] Secondly, since the Euclidean distance is mostly used to calculate continuous variables, the features with continuous flow values ​​can be used to participate in the calculation, namely, Flow Duration, Bwd Pkt Len Std, Fwd IAT Std, Pkt Size Avg, and Fwd Pkts / s.

[0091] The similarity calculation in traditional collaborative filtering algorithms is mostly based on the similarity of users' products, which corresponds to the similarity calculation in the traffic information knowledge graph through the various feature value data of the traffic information. Although this method has a large deviation when facing sparse matrices, it can also accurately capture the preference of a certain type of traffic for specific features, thereby achieving effective classification. Therefore, this similarity calculation method should also be considered.

[0092] Assume that there are m pieces of traffic information: I = (I1, I2, ..., Im), and each piece of traffic information has n features. Then we can use the matrix R m×n to express these data in a form that allows for a better understanding of their characteristics and properties.

[0093]

[0094] The traffic Ii is abstracted into an n-dimensional vector, where each dimension contains the eigenvalue of the corresponding feature.

[0095] I i =(R i1 ,R i2 ,…,R in )

[0096] By applying the definition of cosine similarity, we can calculate similarity-based values ​​as follows:

[0097]

[0098] According to this formula, the similarity between flows Ii and Ij can be measured by the calculated value. If this value is 1, it means that they are exactly the same; if this value is 0, it means that they are completely different.

[0099] Similar to calculating Euclidean similarity, before calculating this similarity, the same normalization processing as in the previous section is also required, and only features with continuously changing values ​​are considered.

[0100] The results of two different similarities are comprehensively considered and effectively merged together. The specific algorithm is as follows:

[0101] sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j )

[0102] The parameters a and b are defined as fusion weight coefficients. The values ​​of these two parameters should be between (0, 1) and their sum should be 1. a and b represent the proportion of the two similarity calculation results introduced above in the final similarity score. The final similarity score ranges from [0, 1]. The values ​​of the fusion weight coefficients a and b can be determined by enumerating and experimenting.

[0103] There are 2632 pieces of traffic information in the constructed traffic information knowledge graph. Another 263 pieces of traffic information are extracted from the original data set as the test set input. The experimental algorithm is as follows:

[0104]

[0105]

[0106] The experimental results are as follows Fig.10 As shown in the figure, when a takes the value of 0.53 and b takes the value of 0.47, the algorithm detection accuracy is the highest, so the value of the fusion weight coefficient can be determined.

[0107] Further, based on the above method, an embodiment of the present invention also provides a network traffic anomaly detection system based on knowledge graph, comprising: a graph construction module and an anomaly detection module, wherein:

[0108] A graph construction module is used to extract entities, relationships between entities and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities using traffic numbers to distinguish traffic information in a data set, indicator entities using traffic feature indicators to determine whether traffic information is abnormal, and standard entities using traffic indicator value ranges to associate traffic entities;

[0109] The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be tested and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be tested based on the feature similarity score.

[0110] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0111] The experimental environment configuration is shown in Table 1.

[0112] Table 1 Experimental environment

[0113]

[0114] The data set used in the experiment is to extract 10% of the existing traffic numbers in the traffic information knowledge graph from the original data set as the test data set, process the data into a format that is easy to import into the traffic information knowledge graph, and save it as a CSV file, such as Fig.11 In the experimental data set, there are 268 pieces of traffic information, and their type distribution is shown in Table 2:

[0115] Table 2 Traffic type statistics of experimental data sets

[0116]

[0117] The principle of principal component analysis algorithm is used to standardize the characteristic data matrix of traffic information. According to the standardized matrix, the coefficient correlation matrix and covariance matrix are solved, and the characteristic decomposition is performed. The contribution is obtained by calculating the influence of each original feature on the result. The contribution is accumulated, and the features before the cumulative value reaches 0.85 are taken as the principal component. According to the selected principal components, the corresponding eigenvector matrix is ​​further calculated. When the cumulative contribution threshold is set to 0.85, it can be found that the cumulative contribution of the first 15 components is about 0.8409, which is just within the threshold range. If the 16th component is considered, it exceeds the threshold. Therefore, the original 78 features of the traffic information can be reduced to 15 principal components. These 15 principal components are formed by adding the original 78 features according to certain weights.

[0118] Based on the traffic information in the CICIDS-2018 dataset, a knowledge graph is constructed to serve as a database for abnormal traffic detection. This graph is constructed before the user enters the traffic information to be detected. Based on the traffic ID, destination port value, and value range entered by the user, the MERGE function in the Cypher syntax is used to create non-repeating corresponding traffic entities, feature entities, and standard entities. Then, a relationship is created between the entities through the CREATE statement.

[0119] Calculate the similarity scores of all known types of traffic to unknown types of traffic in the traffic information knowledge graph. Sort the scores from high to low. The higher the ranking, the greater the similarity with the unknown type of traffic, that is, the more likely the unknown type of traffic belongs to this type. Therefore, based on the ranking results, the classification of unknown type of traffic can be completed, so as to detect whether it is abnormal traffic and which type of abnormal traffic it belongs to. The implementation process is as follows:

[0120] First, run the program and enter the values ​​of the various features of the traffic type you want to detect according to the prompts. According to the user input, add the traffic entity to the constructed traffic information knowledge graph. After importing the test data set into the traffic information knowledge graph, find the known type of traffic that has any three feature values ​​in the same range as the traffic information to be detected and store them in the array result.

[0121] The result array contains the flow range to be considered for similarity calculation. By looping through the flow information within the range, the Euclidean similarity osim and the flow-based feature value similarity ysim with the flow information to be detected are calculated, and then further fused according to the fusion weight coefficient to obtain the final similarity score pfsim. The ID of the flow information and the corresponding similarity score calculation result are stored in the final array. According to the final array, the similarity scores in the results are sorted to obtain the final similarity ranking.

[0122] The experimental data set is used for anomaly detection according to the above steps, and the results are statistically analyzed using Excel tables. Fig.12 The experimental results are shown in Table 3:

[0123] Table 3 Experimental results statistics

[0124]

[0125] The experimental statistical results show that this solution is effective in normal traffic (Benign) and some DOS traffic (DoS

[0126] It has a good effect in detecting DoS attacks-GoldenEye and DoS attacks-Slowloris.

[0127] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0128] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0129] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0130] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0131] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A network traffic anomaly detection method based on knowledge graph, characterized in that: Include: Extract entities, relationships between entities and entity attributes from the traffic data set, build a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities using traffic numbers to distinguish traffic information in the data set, indicator entities using traffic feature indicators to determine whether traffic information is abnormal, and standard entities using traffic indicator value ranges to associate traffic entities; A collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and the type of the unknown traffic to be measured is determined based on the feature similarity score.

2. The network traffic anomaly detection method based on knowledge graph according to claim 1 is characterized in that: The traffic information knowledge graph is represented as G<L,I,S,P,R> , G represents the knowledge graph, L represents the label entity, I represents the traffic entity, S represents the indicator entity, P represents the standard entity, and R represents the relationship set between the entities.

3. The network traffic anomaly detection method based on knowledge graph according to claim 1 or 2 is characterized in that: Extract entities, relationships between entities, and entity attributes from traffic datasets, including: According to the flow data set, label entities, flow entities, indicator entities and standard entities are extracted. The attributes of the label entities include flow label attributes for determining the flow type and attack type attributes for determining what kind of attack causes abnormal flow; the attributes of the flow entities include flow numbers for identifying flow and timestamps for recording force generation time; the attributes of the indicator entities include feature indicator names, feature indicator contents and feature indicator values; the attributes of the standard entities include discrete values ​​of flow indicators or continuous value intervals of flow indicators; Extract the association relationship between entities based on the attribute information of different entities.

4. The network traffic anomaly detection method based on knowledge graph according to claim 1 is characterized in that: A collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, including: importing the unknown traffic to be measured into the traffic information knowledge graph according to the entity attribute relationship; According to the unknown traffic characteristics to be tested, any three traffic characteristics with values ​​in the same interval are extracted from the traffic information knowledge graph, and the extracted known traffic types are used as candidate matching traffic; Based on the traffic feature similarity, the feature similarity score between the unknown traffic to be tested and the candidate matching traffic is obtained.

5. The network traffic anomaly detection method based on knowledge graph according to claim 1 or 4 is characterized in that: Get the feature similarity scores between traffic, including: Obtaining spatial similarity and feature similarity between traffic flows, wherein the spatial similarity is calculated using Euclidean distance, and the feature similarity is calculated using cosine similarity; The spatial similarity and feature similarity are fused using the weight coefficient to obtain the final similarity score between flows.

6. The network traffic anomaly detection method based on knowledge graph according to claim 5 is characterized in that: Obtaining the spatial similarity and feature similarity between traffic also includes: The features of traffic information are normalized using the sum of the values ​​of traffic features in the traffic information knowledge graph.

7. The network traffic anomaly detection method based on knowledge graph according to claim 5 is characterized in that: The process of fusing spatial similarity and feature similarity using weight coefficients is expressed as: sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j ), Among them, sim KG and sim u The flow rate I i ,I j The spatial similarity and feature similarity between them, a and b are the fusion weight coefficients.

8. A network traffic anomaly detection system based on knowledge graph, characterized in that: Contains: graph construction module and anomaly detection module, among which, A graph construction module is used to extract entities, relationships between entities and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, relationships between entities and entity attributes, and store the traffic information knowledge graph in a graph database, wherein the entities include: label entities representing normal or abnormal traffic information categories, traffic entities using traffic numbers to distinguish traffic information in a data set, indicator entities using traffic feature indicators to determine whether traffic information is abnormal, and standard entities using traffic indicator value ranges to associate traffic entities; The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be tested and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be tested based on the feature similarity score.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images