Network traffic anomaly detection method and system based on knowledge graph

Abstract

The present invention relates to the field of network security technology, and in particular to a network traffic anomaly detection method and system based on a knowledge graph, which extracts entities, relationships between entities, and entity attributes from a traffic data set, constructs a traffic information knowledge graph based on the extracted entities, relationships between entities, and entity attributes, and stores the traffic information knowledge graph in a graph database; uses a collaborative filtering algorithm to obtain a feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determines the type of the unknown traffic to be measured based on the feature similarity score. The present invention stores traffic information in a graph database Neo4j, making the relationship between the feature values ​​between traffic flows clearer and more explicit, facilitating operations for security personnel and improving visualization effects; by matching the intervals where traffic feature values ​​are located in the knowledge graph, the range of traffic involved in the calculation of the anomaly detection algorithm is narrowed, thereby improving computational efficiency; and by integrating weight coefficients, comprehensively considering the two similarity results, thereby improving the accuracy of the detection results.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network traffic anomaly detection method and system based on a knowledge graph. Background Art

[0002] With the development of information technology, the internet has become increasingly integrated into our daily lives, and individuals' demand for internet access is also increasing. As internet traffic surges, the methods used to exploit malicious traffic for cyberattacks are constantly evolving, becoming a critical issue in network security. Identifying and detecting malicious traffic within massive amounts of data is a constant focus for security departments within enterprises and institutions. In recent years, the stealth, diversity, and systematic nature of malicious traffic have increased, making traditional packet and data flow detection methods inadequate.

[0003] Currently, the main problems in the field of abnormal traffic detection are as follows: 1. Frequent false alarms and misjudgments. The means used by malicious attacks are becoming more and more diverse, and the types of malicious traffic generated are also increasing. The use of a single detection method may lead to misjudgments under certain attack methods, creating loopholes for real security threats. 2. Serious resource usage issues. In actual operation, due to the huge amount of traffic information, abnormal traffic detection requires a large amount of computing resources. If it is not screened, it may be a burden for some low-configuration devices or systems. 3. Complex feature engineering. Abnormal data is often complex and difficult to describe using traditional feature representation methods. In addition, for traffic information, the feature dimension is large. How to select effective features and perform reasonable preprocessing is an important challenge in anomaly detection. Summary of the Invention

[0004] To this end, the present invention provides a network traffic anomaly detection method and system based on knowledge graph to solve the problems of unsatisfactory accuracy and insufficient feature consideration in existing abnormal traffic detection.

[0005] According to the design scheme provided by the present invention, on the one hand, a method for detecting network traffic anomalies based on a knowledge graph is provided, comprising:

[0006] Extract entities, inter-entity relationships, and entity attributes from a traffic dataset, construct a traffic information knowledge graph based on the extracted entities, inter-entity relationships, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label entities that represent normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a dataset, indicator entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator value ranges to associate traffic entities.

[0007] A collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and the type of the unknown traffic to be measured is determined based on the feature similarity score.

[0008] As the network traffic anomaly detection method based on knowledge graph of the present invention, further, the traffic information knowledge graph is represented as G<L,I,S,P,R> , G represents the knowledge graph, L represents the label entity, I represents the traffic entity, S represents the indicator entity, P represents the standard entity, and R represents the relationship set between the entities.

[0009] As a network traffic anomaly detection method based on knowledge graph of the present invention, further extracting entities, relationships between entities and entity attributes in the traffic data set includes:

[0010] According to the traffic data set, label entities, traffic entities, indicator entities and standard entities are extracted. The attributes of the label entities include traffic label attributes for determining traffic types and attack type attributes for determining what kind of attack causes abnormal traffic; the attributes of the traffic entities include traffic numbers for identifying traffic and timestamps for recording traffic generation time; the attributes of the indicator entities include feature indicator names, feature indicator contents and feature indicator values; the attributes of the standard entities include discrete values ​​of traffic indicators or continuous value intervals of traffic indicators;

[0011] Extract the association relationship between entities based on the attribute information of different entities.

[0012] As a network traffic anomaly detection method based on knowledge graph of the present invention, a collaborative filtering algorithm is further used to obtain a feature similarity score between the unknown traffic to be tested and the traffic in the stored traffic information knowledge graph, including:

[0013] Import the unknown traffic to be measured into the traffic information knowledge graph based on the entity attribute relationship;

[0014] Based on the unknown traffic characteristics to be measured, any three known types of traffic whose traffic characteristics are within the same interval are extracted from the traffic information knowledge graph, and the extracted known types of traffic are used as candidate matching traffic;

[0015] Based on the traffic feature similarity, a feature similarity score is obtained between the unknown traffic to be tested and the candidate matching traffic.

[0016] As the network traffic anomaly detection method based on knowledge graph of the present invention, further obtaining the feature similarity score between traffic flows includes:

[0017] Obtaining spatial similarity and feature similarity between traffic flows, wherein the spatial similarity is calculated using Euclidean distance, and the feature similarity is calculated using cosine similarity;

[0018] The spatial similarity and feature similarity are fused using the weight coefficient to obtain the final similarity score between flows.

[0019] The network traffic anomaly detection method based on the knowledge graph of the present invention further obtains the spatial similarity and feature similarity between traffic flows, and further includes:

[0020] The features of traffic information are normalized using the sum of the values ​​of traffic features in the traffic information knowledge graph.

[0021] As the network traffic anomaly detection method based on knowledge graph of the present invention, the process of fusing spatial similarity and feature similarity using weight coefficient is further expressed as:

[0022] sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j ), where sim KG and sim u Flow I i , I j The spatial similarity and feature similarity between them, a and b are fusion weight coefficients.

[0023] On the other hand, the present invention also provides a network traffic anomaly detection system based on knowledge graph, comprising: a graph construction module and an anomaly detection module, wherein:

[0024] A graph construction module is configured to extract entities, inter-entity relationships, and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, inter-entity relationships, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label entities that represent normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a data set, indicator entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator value ranges to associate traffic entities;

[0025] The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be measured based on the feature similarity score.

[0026] Beneficial effects of the present invention:

[0027] 1. The present invention utilizes the knowledge graph technology in the field of artificial intelligence to construct a complete and visual traffic knowledge graph by analyzing known traffic information, so as to more clearly display the relationship between different entities, attributes, and entities, so as to better understand the changes in traffic. When there is traffic input of unknown classification, it looks for the connection between it and known traffic, and uses the similarity calculation idea in the knowledge graph recommendation algorithm to perform association calculation to obtain traffic information with a greater correlation with the unknown classification traffic. This is used as the basis for classification, providing important help for discovering and analyzing abnormal network traffic.

[0028] 2. By storing traffic information in the graph database Neo4j, the relationship between traffic feature values ​​is made clearer and more explicit, which facilitates operations for security personnel and improves visualization effects. By matching the intervals of traffic feature values ​​in the knowledge graph, the range of traffic involved in the anomaly detection algorithm is narrowed, improving computational efficiency. By fusing weight coefficients and comprehensively considering the two similarity results, the accuracy of the detection results is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the network traffic anomaly detection process based on the knowledge graph in the embodiment;

[0030] Figure 2 This is a schematic diagram of a tag entity in the embodiment;

[0031] Figure 3 This is a schematic diagram of a traffic entity in the embodiment;

[0032] Figure 4 This is a schematic diagram of the indicator entity in the embodiment;

[0033] Figure 5 This is a schematic diagram of a standard entity class for taking values ​​of discrete indicators in the embodiment;

[0034] Figure 6 Schematic diagram of the numerical distribution of flow information indicators in the embodiment;

[0035] Figure 7 This is a schematic diagram of a standard entity class for a continuous indicator interval in the embodiment;

[0036] Figure 8 This is a schematic diagram of the traffic information knowledge graph ontology in the embodiment;

[0037] Figure 9 This is a diagram illustrating the conversion relationship between the traffic information structure and the user product structure in the embodiment;

[0038] Figure 10 This is a diagram illustrating the relationship between the fusion weight coefficient value and the detection accuracy in the embodiment;

[0039] Figure 11This is a schematic diagram of the test data set after preprocessing in the embodiment;

[0040] Figure 12 Schematic diagram of the abnormality detection experiment results in the embodiment. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below with reference to the accompanying drawings and technical solutions.

[0042] In recent years, the means of using traffic to carry out network attacks have become increasingly complex and diverse, and the concealment and encryption methods of malicious traffic itself have also been continuously enhanced, posing a considerable challenge to the field of anomaly detection. Traditional abnormal traffic detection technology mainly determines whether the traffic is abnormal by comparing it with the local abnormal traffic feature library; after the introduction of machine learning algorithms, the accuracy of detection has been greatly improved, but there are still problems such as insufficient consideration of abnormal feature selection and poor learning effect. For this reason, the embodiments of the present invention, see Figure 1 As shown, a network traffic anomaly detection method based on knowledge graph is provided, which specifically includes:

[0043] S101. Extract entities, relationships between entities, and entity attributes from the traffic data set, build a traffic information knowledge graph based on the extracted entities, relationships between entities, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label-type entities that represent normal or abnormal traffic information categories, traffic-type entities that use traffic numbers to distinguish traffic information in a data set, indicator-type entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard-type entities that use traffic indicator numerical ranges to associate traffic-type entities.

[0044] Traffic information knowledge graph mainly involves entities in four dimensions, namely G<L,I,S,P,R> , which are label entity L, traffic entity I, indicator entity S, standard entity P and the relationship between each entity R. Label entity is used to represent the category of a piece of traffic information, including normal traffic and abnormal traffic, a total of two categories. Among them, abnormal traffic includes detailed classifications such as BForce, SFTP and SSH, DDOS LOIT, etc.; traffic entity represents each piece of traffic information in the data set, which is distinguished by the traffic number; indicator entity is used to judge whether the traffic information is abnormal, including 7 indicators including two main features obtained by principal component analysis; standard entity is a numerical range. Through this entity, two traffic information entities with a certain indicator value in the same range can be associated. After extracting and fusing the above data, the construction of the traffic information knowledge graph can be completed.

[0045] In the principal component analysis method, the principal components generated are weighted combinations of multiple original variables, which can reflect the influence of most variables in the original data on the results. Suppose the characteristic values ​​of p flows are x1, x2, ..., x p The corresponding feature value of the i-th flow information is x i1 ,x i2 ,…,x ip (i=1,2,…,n), the original matrix obtained is

[0046]

[0047] First, since the range and size of different variable values ​​may vary greatly, the target matrix must be standardized first. The resulting matrix is ​​recorded as x*, and the calculation formula is

[0048]

[0049] in, is a data in the matrix x*, is the variable x j The average value of the observations; s jj is the variable x j The variance of the observations.

[0050] Then the correlation coefficient matrix R is calculated

[0051] R=(r op ) m*n (o,p=1,2,...,m)

[0052] r op and r po is a symmetric matrix, and the correlation coefficient r between indicators op The calculation formula is

[0053]

[0054] Finally, the number of principal components is determined, usually taking the features with cumulative variance contribution greater than 85% and number greater than 1. The eigenvalue is recorded as Cumulative variance contribution rate v t The calculation formula is

[0055]

[0056] According to the algorithm principle, a Python algorithm can be written to calculate the research data set. When the threshold for the impact on the results is 85%, a total of 15 principal components are obtained. Due to the limitations of personal computing resources, if all principal components are considered together, it will take a lot of computing time. For simplicity, in this embodiment, the threshold range is narrowed and the top two principal components that dominate the 15 principal components are selected.

[0057] The principal components obtained by principal component analysis are the weighted sum of the original components and are not inherently meaningful. Directly using these as key traffic features for knowledge graph construction would significantly reduce user readability and graph visualization. The purpose of principal component analysis is to reduce the dimensionality of traffic information and identify a small number of factors that significantly influence whether a traffic flow is abnormal, among numerous features. Therefore, the original traffic features with the largest proportions in each of the two principal components mentioned above are used as the principal components of traffic information for practical purposes in knowledge graph construction. This dimensionality reduction is achieved by compressing traffic information from its original 78 dimensions to two: Fwd PSH Flags and Fwd Pkts / s. These two features will serve as key features in the construction of the traffic information knowledge graph.

[0058] Neo4j is a NoSQL database based on graph storage, written in Java. It stores information in a graph structure. The associated data itself is the data it contains, allowing it to directly display the characteristics of the associated data and the relationships between them. It also offers easy access to highly connected data and excellent performance, making it a preferred tool for storing knowledge graphs. Neo4j graph databases primarily consist of two types: nodes and edges, representing entities or attribute values ​​and their relationships, respectively.

[0059] The entities, relationships between entities, and entity attributes extracted from the traffic dataset can be designed to include:

[0060] According to the traffic data set, label entities, traffic entities, indicator entities and standard entities are extracted. The attributes of the label entities include traffic label attributes for determining traffic types and attack type attributes for determining what kind of attack causes abnormal traffic; the attributes of the traffic entities include traffic numbers for identifying traffic and timestamps for recording traffic generation time; the attributes of the indicator entities include feature indicator names, feature indicator contents and feature indicator values; the attributes of the standard entities include discrete values ​​of traffic indicators or continuous value intervals of traffic indicators;

[0061] Extract the association relationship between entities based on the attribute information of different entities.

[0062] The entities in the traffic information knowledge graph are divided into four parts according to the entity type:

[0063] The label entity can be extracted based on the traffic type contained in the research dataset. There is a total of 1 normal traffic label (Benign) and 13 abnormal traffic labels. This entity has two attributes: traffic type and attack type. Traffic type is one of the 14 types mentioned above and is used to determine the final traffic type; attack type describes the type of attack that causes this type of traffic, providing a basis for decision makers to further select defense strategies after detecting abnormal traffic. Figure 2 As shown in the figure, in the traffic information knowledge graph, label entities are represented by dark green circles, and the label name is displayed on the entity.

[0064] Each flow entity represents a piece of flow information. To facilitate calculation, it is represented by the flow information number. Its attributes include flow ID and timestamp. The flow ID itself has no practical meaning and is only used to identify the flow to facilitate subsequent calculations; the timestamp records the time when the flow was generated. Since abnormal flow often appears in large quantities in a short period of time, the timestamp is also of certain reference value for the detection of abnormal flow. Figure 3 As shown in the traffic information knowledge graph, traffic-related entities are represented by gray circles, and the traffic number is displayed on the entity for distinction.

[0065] There are 7 types of indicator entities, two of which are the main features Fwd PSHFlags and Fwd Pkts / s obtained by the principal component analysis method in Chapter 3. Based on the best features of DoS attack research, another 6 features are determined as indicator entities, namely Dst Port, Protocol, Flow Duration, Bwd Pkt Len Std, Fwd IAT Std, and Pkt Size Avg. The attributes they contain are indicator name, indicator content, and indicator value. Among them, the indicator content explains the meaning of the indicator, which is convenient for users to understand and improves the readability of the traffic information knowledge graph. Figure 4 As shown in the figure, in the traffic information knowledge graph, indicator-type entities are represented by blue circles, and the indicator name is displayed on the entity.

[0066] The definition of standard entities is quite unique. For discrete metrics, each value represents a separate standard, such as the Dst Port metric. Since port numbers are discrete values, each value of this metric generates a corresponding standard entity. Different metrics have different standard entities, which are distinguished by different colors in the traffic information knowledge graph. Figure 5 It is a standard entity of the destination port. The number on the entity is the value of the destination port of the traffic information.

[0067] For continuous indicators, since their numerical values ​​fluctuate greatly, if the numerical intervals are divided evenly, most intervals contain very little or even no flow information. Therefore, the standard interval division can be combined with statistical laws. Take the indicator Bwd Pkt Len Std as an example. Figure 6 As shown in the figure, the values ​​are statistically obtained. The horizontal axis of the statistical result is the number of traffic information, and the vertical axis is the corresponding Bwd Pkt Len Std feature value. Figure 6 It can be seen that most of the values ​​of this feature fall within the range of less than 500, a small part falls within the range of 500-1500, and a very small part falls within the range of more than 1500. Therefore, in this embodiment, the standard interval of the Bwd Pkt Len Std feature can be defined as <500, 500-1500, and >1500. The traffic entities whose feature values ​​fall within the corresponding intervals are constructed in the knowledge graph with the standard entities of the intervals, thereby creating connections between the traffic entities. Figure 7 This is a standard entity for flow duration, represented by a red circle in the flow information knowledge graph. The number on the entity identifies the interval it represents.

[0068] By extracting information from different entities, we found that each dimension in the traffic information knowledge graph is closely related. For each piece of traffic information, there is a label indicating whether it is abnormal traffic and which type of abnormal traffic it belongs to; the label used to determine a piece of traffic is the value of each feature of the traffic itself. The values ​​of the features are all discrete or continuous values, which must fall within a certain interval, that is, there is a standard interval entity associated with it. Finally, the ontology diagram obtained is as follows Figure 8 shown.

[0069] We selected the CICIDS-2018 dataset, a relatively recent dataset for traffic research, and performed knowledge extraction, knowledge fusion, and knowledge representation on it. These data were stored in the graph database Neo4j as entity, relationship, and attribute triples. The resulting knowledge graph currently contains 2,632 traffic entities, each of which is associated with eight indicator entities and one label entity. Each indicator entity is associated with a corresponding standard entity defined based on statistical laws, forming the final traffic information knowledge graph. Different entities are distinguished by different colors. For example, gray represents traffic entities, blue represents indicator entities, and dark green represents label entities. The remaining standard entities for different indicators are distinguished by different colors.

[0070] S102. Use a collaborative filtering algorithm to obtain a feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be measured based on the feature similarity score.

[0071] Currently, newer anomaly detection methods mostly use machine learning or behavioral analysis-based technologies. The key is to extract the characteristics of abnormal or normal traffic through training with a large amount of abnormal traffic, encode its behavioral patterns into a model, compare it with the characteristic coding of unknown types of traffic, and determine which type of traffic it belongs to based on the degree of similarity.

[0072] The core of the recommendation algorithm also lies in feature extraction and similarity calculation, and in recent years, through the rapid development of artificial intelligence technology, especially the introduction of knowledge graphs, connections have been established between users and users, and between users and products. Using association calculations, the results have been relatively accurate. Therefore, in the embodiment of this case, the collaborative filtering recommendation algorithm commonly used in knowledge graphs is used in the field of anomaly detection. Collaborative filtering algorithms are generally used for recommendations between users and products, and how to correspond the relationship between traffic information with the user-product relationship becomes the key. The entities contained in the knowledge graph: traffic-type entities, indicator-type entities, and standard-type entities, respectively correspond to product entities, product-related entities, and user entities in the user-product model. Newly added label-type entities are used to identify the type of traffic. For example Figure 9 As shown in the figure, taking the user-item structure for recommending movies to users as an example, in this user-item structure, if user A likes movie A, actor A appears in movie A, and the actor also stars in movie B, then movies A and B can be considered similar. If user B also likes movie B, then users A and B can be considered similar, and other movies that user B likes can be recommended to user A. Similarly, in the traffic information structure, if the flow duration values ​​of traffic A and traffic B are within the same range, it can be simply assumed that traffic A and traffic B are similar. If the type of traffic B is known, it can provide a certain reference value for the type of traffic A, that is, traffic A, which may be similar to traffic B, can be recommended to traffic B. In actual knowledge graphs, there are multiple feature-class entities and standard-class entities, and the similarity between traffic flows is also calculated using the improved collaborative filtering algorithm presented later, which ensures the accuracy of the results to a certain extent.

[0073] Among them, the collaborative filtering algorithm is used to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, which can be designed to include:

[0074] Import the unknown traffic to be measured into the traffic information knowledge graph based on the entity attribute relationship;

[0075] Based on the unknown traffic characteristics to be measured, any three known types of traffic whose traffic characteristics are within the same interval are extracted from the traffic information knowledge graph, and the extracted known types of traffic are used as candidate matching traffic;

[0076] Based on the traffic feature similarity, a feature similarity score is obtained between the unknown traffic to be tested and the candidate matching traffic.

[0077] The collaborative filtering algorithm is an important similarity calculation algorithm. It simultaneously considers multiple similarity calculation results to filter out key information from massive amounts of information. This algorithm is widely used to recommend items that are highly similar to users' preferences. Its core principle is to analyze historical user behavior data on items to predict user preferences and provide relevant recommendations. This behavior data can be stored in a matrix format, such as ratings, browsing, and searches. Therefore, the accuracy of the algorithm's recommendations is directly affected by the accuracy and completeness of the massive amount of user behavior data stored in the system.

[0078] With the rapid development of internet technology, interactions between hosts are increasing, generating ever-increasing amounts of traffic data. Furthermore, among these abnormal flows, non-malicious traffic caused by hardware issues or improper operation can also be found, placing an unnecessary burden on computation. Traditional matrix storage methods are no longer able to effectively process and transmit such large amounts of data, resulting in reduced matrix utilization efficiency. Furthermore, network traffic data is typically very sparse, with only a small number of users accessing a large number of sites or applications. Traditional collaborative filtering algorithms may be unable to process this sparse data, leading to similarity scoring errors and compromising recommendation effectiveness. This is because traditional collaborative filtering algorithms require sufficient data to detect similarities and differences between users.

[0079] To address these issues, this case study proposes a new collaborative filtering algorithm. By leveraging the rich relationships within knowledge graphs, this algorithm improves its computational efficiency in processing big data. Compared to traditional collaborative filtering algorithms, this new algorithm more accurately captures useful information, produces more reasonable similarity scores, and effectively addresses the challenges posed by data sparsity.

[0080] In order to solve the problem of low efficiency of big data operation faced by collaborative filtering algorithms, a traffic information knowledge graph is combined to solve it. After receiving the feature information of unknown type traffic, it is imported into the traffic information knowledge graph according to the defined entity attribute relationship. Since the existing traffic information knowledge graph contains a large amount of data, if the similarity score is calculated one by one with the unknown type traffic, the calculation amount is large and the efficiency is low. Therefore, the relationship between entities in the knowledge graph is used to narrow the range of data that needs to be involved in the calculation. According to all the features contained in the unknown type traffic, only the traffic within the same range as the values ​​of any three of its features is considered in the calculation range. In this way, it is equivalent to a preliminary screening of the traffic through the knowledge graph, which greatly improves the calculation efficiency of the subsequent collaborative filtering algorithm.

[0081] Aiming at the problem that traditional similarity calculation methods have errors when data is sparse, this paper corrects it by calculating the spatial similarity between traffic flows and combines it with the feature value similarity based on traffic flow through a weight coefficient to obtain the final similarity score.

[0082] In the knowledge graph, each traffic entity is connected to several feature entities, and each feature entity has its specific value. If each feature entity is regarded as a dimension, a piece of traffic information can be represented by an m-dimensional vector:

[0083] H i =(E 1i ,E 2i ,…,E mi ) T

[0084] E di Representative flow H i Based on this, the spatial similarity between two traffic flows in the traffic information knowledge graph can be calculated using the Euclidean distance formula, which is shown in the following formula.

[0085]

[0086] The Euclidean distance obtained here is a non-negative value, which can be constrained between (0,1] by the following formula:

[0087]

[0088] As the value increases, the spatial similarity between the two flows will continue to increase, up to 1; however, if this value is infinitely close to 0, it means that there is almost no commonality between them.

[0089] For actual traffic data sets, the numerical differences between the features of each dimension of traffic information are too large. If calculated directly, the result will be determined only by the features with larger values, thereby reducing or even ignoring the features with smaller value ranges. Therefore, before calculating Euclidean similarity, the features of the traffic information need to be normalized separately. That is, the value of each feature is divided by the sum of the feature values ​​in the traffic information knowledge graph. The sum of each feature in the traffic information knowledge graph can be obtained before the similarity calculation and defined in the program as a constant. During the calculation, it can be added to the corresponding feature value of the traffic to be detected and used as a divisor.

[0090] Secondly, since Euclidean distance is mostly used to calculate continuous variables, the features with continuous flow values ​​can be used in the calculation, namely Flow Duration, Bwd Pkt Len Std, Fwd IAT Std, Pkt Size Avg, and Fwd Pkts / s.

[0091] Traditional collaborative filtering algorithms often calculate similarity based on user-item similarity. This corresponds to calculating similarity using the individual feature values ​​of traffic information in the traffic information knowledge graph. Although this approach suffers from significant bias when dealing with sparse matrices, it can accurately capture the preference of certain types of traffic for specific features, enabling effective classification. Therefore, this similarity calculation method should also be considered.

[0092] Assume that there are m pieces of traffic information: I = (I1, I2, ..., Im), and each piece of traffic information has n features. Then we can use the matrix R m×n to express these data in a form that allows for a better understanding of their characteristics and properties.

[0093]

[0094] The traffic Ii is abstracted into an n-dimensional vector, where each dimension contains the eigenvalue of the corresponding feature.

[0095] I i =(R i1 ,R i2 ,…,R in )

[0096] By applying the definition of cosine similarity, we can calculate similarity-based values, which can be expressed as follows:

[0097]

[0098] According to this formula, the similarity between flows Ii and Ij can be measured by the calculated value. If this value is 1, it means that they are exactly the same; if this value is 0, it means that they are completely different.

[0099] Similar to calculating Euclidean similarity, before calculating this similarity, the same normalization process as in the previous section is also required, and only features with continuously changing values ​​are considered.

[0100] Taking into account the results of two different similarities, we can effectively merge them together. The specific algorithm is as follows:

[0101] sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j )

[0102] Parameters a and b are defined as fusion weight coefficients. These two parameters should range from (0, 1) and sum to 1. a and b represent the weights of the two similarity calculation results described above in the final similarity score, which ranges from [0, 1]. The values ​​of the fusion weight coefficients a and b can be determined through enumeration and experimentation.

[0103] There are 2632 pieces of traffic information in the constructed traffic information knowledge graph. 263 more pieces of traffic information are extracted from the original dataset as the test set input. The experimental algorithm is as follows:

[0104]

[0105]

[0106] The experimental results are as follows Figure 10 As shown in the figure, when a takes the value of 0.53 and b takes the value of 0.47, the algorithm detection accuracy is the highest, so the value of the fusion weight coefficient can be determined.

[0107] Furthermore, based on the above method, an embodiment of the present invention also provides a network traffic anomaly detection system based on a knowledge graph, comprising: a graph construction module and an anomaly detection module, wherein:

[0108] A graph construction module is configured to extract entities, inter-entity relationships, and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, inter-entity relationships, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label entities that represent normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a data set, indicator entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator value ranges to associate traffic entities;

[0109] The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be measured based on the feature similarity score.

[0110] To verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0111] The experimental environment configuration is shown in Table 1.

[0112] Table 1 Experimental environment

[0113]

[0114] The data set used in the experiment is to extract 10% of the existing traffic numbers in the traffic information knowledge graph from the original data set as the test data set, process the data into a format that is easy to import into the traffic information knowledge graph, and save it as a CSV file, such as Figure 11 In the experimental data set, there are 268 pieces of traffic information, and their type distribution is shown in Table 2:

[0115] Table 2 Traffic type statistics of experimental dataset

[0116]

[0117] The principal component analysis algorithm is used to standardize the traffic information feature data matrix. Based on the standardized matrix, the coefficient correlation matrix and covariance matrix are solved, and eigendecomposition is performed. The contribution is obtained by calculating the influence of each original feature on the result. The contribution is accumulated, and the features whose cumulative value reaches 0.85 are taken as the principal components. Based on the selected principal components, the corresponding eigenvector matrix is ​​further calculated. When the cumulative contribution threshold is set to 0.85, it can be found that the cumulative contribution of the first 15 components is approximately 0.8409, which is just within the threshold range. If the 16th component is considered, it exceeds the threshold. Therefore, the original 78 features of the traffic information can be reduced to 15 principal components. These 15 principal components are composed of the original 78 features added according to certain weights.

[0118] A knowledge graph, serving as a database for abnormal traffic detection, is constructed based on traffic information from the CICIDS-2018 dataset. This knowledge graph is constructed before the user enters the traffic information to be detected. Based on the user-entered traffic ID, destination port value, and value range, the MERGE function in Cypher syntax is used to create unique entities for traffic, feature, and standard categories. Relationships are then established between these entities using the CREATE statement.

[0119] Calculate the similarity scores of all known traffic types in the traffic information knowledge graph to the unknown traffic types. Sort the scores from high to low. The higher the ranking, the greater the similarity with the unknown traffic type, meaning the more likely the unknown traffic type is of that type. Therefore, based on the ranking results, the unknown traffic type can be classified, detecting whether it is abnormal traffic and the specific type of abnormal traffic it belongs to. The implementation process is as follows:

[0120] First, run the program and follow the prompts to enter the values ​​for the various features of the traffic type you want to detect. Based on the user input, the traffic entity is added to the constructed traffic information knowledge graph. After importing the test dataset into the traffic information knowledge graph, it searches for known traffic types whose values ​​fall within the same range as any three feature values ​​of the traffic information to be detected and stores them in the array result.

[0121] The result array contains the traffic range considered for the similarity calculation. By looping through the traffic information within the range, the Euclidean similarity (osim) and the traffic-based feature value similarity (ysim) with the traffic information to be tested are calculated. These similarity scores are then fused based on the fusion weight coefficient to obtain the final similarity score (pfsim). The traffic information ID and the corresponding similarity score calculation results are stored in the final array. The final array is used to sort the similarity scores to obtain the final similarity ranking.

[0122] The experimental data set is anomaly detected according to the above steps, and the results are statistically analyzed using Excel tables. Figure 12 The experimental results are shown in Table 3:

[0123] Table 3 Experimental results statistics

[0124]

[0125] The experimental statistical results show that this solution is effective in normal traffic (Benign) and some DOS traffic (DoS

[0126] It has a good effect in detecting attacks-GoldenEye and DoS attacks-Slowloris.

[0127] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0129] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0130] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0131] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A network traffic anomaly detection method based on knowledge graph, characterized in that: Include: Extract entities, inter-entity relationships, and entity attributes from a traffic dataset, construct a traffic information knowledge graph based on the extracted entities, inter-entity relationships, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label entities that represent normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a dataset, indicator entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator value ranges to associate traffic entities. The unknown traffic to be measured is imported into the traffic information knowledge graph based on the entity attribute relationship; based on the characteristics of the unknown traffic to be measured, any three known types of traffic with traffic feature values ​​in the same interval are extracted from the traffic information knowledge graph, and the extracted known types of traffic are used as candidate matching traffic; based on the traffic feature similarity, the feature similarity score between the unknown traffic to be measured and the candidate matching traffic is obtained, and the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph is obtained by using a collaborative filtering algorithm, and the type of the unknown traffic to be measured is determined based on the feature similarity score.

2. The network traffic anomaly detection method based on knowledge graph according to claim 1 is characterized in that: The traffic information knowledge graph is represented as G<L,I,S,P,R> , G represents the knowledge graph, L represents the label entity, I represents the traffic entity, S represents the indicator entity, P represents the standard entity, and R represents the relationship set between the entities.

3. The network traffic anomaly detection method based on knowledge graph according to claim 1 or 2 is characterized in that: Extract entities, relationships between entities, and entity attributes from traffic datasets, including: Extracting label entities, flow entities, indicator entities, and standard entities based on the traffic data set, wherein the attributes of the label entities include a flow label attribute for determining the flow type and an attack type attribute for determining the type of attack causing the abnormal flow; the attributes of the flow entities include a flow number for identifying the flow and a timestamp for recording the time when the flow was generated; The attributes of the indicator-type entity include the characteristic indicator name, characteristic indicator content and characteristic indicator value; the attributes of the standard-type entity include the discrete value of the flow indicator or the continuous value interval of the flow indicator; Extract the association relationship between entities based on the attribute information of different entities.

4. The network traffic anomaly detection method based on knowledge graph according to claim 1 is characterized in that: Obtain feature similarity scores between traffic flows, including: Obtaining spatial similarity and feature similarity between traffic flows, wherein the spatial similarity is calculated using Euclidean distance, and the feature similarity is calculated using cosine similarity; The spatial similarity and feature similarity are fused using the weight coefficient to obtain the final similarity score between flows.

5. The network traffic anomaly detection method based on knowledge graph according to claim 4 is characterized in that: Obtaining spatial similarity and feature similarity between traffic flows also includes: The features of traffic information are normalized using the sum of the values ​​of traffic features in the traffic information knowledge graph.

6. The network traffic anomaly detection method based on knowledge graph according to claim 4 is characterized in that: The process of fusing spatial similarity and feature similarity using weight coefficients is expressed as: sim(I i ,I j )=asim KG (I i ,I j )+bsim u (I i ,I j ), Among them, sim KG and sim u Flow I i , I j The spatial similarity and feature similarity between them, a and b are fusion weight coefficients.

7. A network traffic anomaly detection system based on knowledge graph, characterized in that: The method according to claim 1 is implemented, comprising: a graph construction module and an anomaly detection module, wherein: A graph construction module is configured to extract entities, inter-entity relationships, and entity attributes from a traffic data set, construct a traffic information knowledge graph based on the extracted entities, inter-entity relationships, and entity attributes, and store the traffic information knowledge graph in a graph database. The entities include: label entities that represent normal or abnormal traffic information categories, traffic entities that use traffic numbers to distinguish traffic information in a data set, indicator entities that use traffic characteristic indicators to determine whether traffic information is abnormal, and standard entities that use traffic indicator value ranges to associate traffic entities; The anomaly detection module is used to use a collaborative filtering algorithm to obtain the feature similarity score between the unknown traffic to be measured and the traffic in the stored traffic information knowledge graph, and determine the type of the unknown traffic to be measured based on the feature similarity score.

8. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 6 can be implemented.

Images