Network traffic anomaly detection method and device, terminal equipment, readable storage medium

Abstract

The application provides a network traffic anomaly detection method and device, terminal equipment and readable storage medium, the method comprises the following steps: constructing a k-neighbor graph corresponding to a network traffic dataset after obtaining the network traffic dataset to be detected; then determining the relative distance and global density corresponding to each network traffic data; determining the gamma value of each network traffic data based on the relative distance and global density of each network traffic data, and stretching the gamma value of each network traffic data to obtain the gamma transformation value of each network traffic data; determining a cluster center point set based on the gamma transformation value of each network traffic data, and performing step-by-step distribution on each non-cluster center point according to the cluster center point set to obtain a plurality of network traffic data clusters; and determining abnormal network traffic data based on the plurality of network traffic data clusters. The application can reduce the false detection rate of network traffic anomaly detection and improve the detection accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network traffic anomaly detection, and more particularly to a network traffic anomaly detection method and device, a terminal device, and a readable storage medium. BACKGROUND

[0002] With the popularity of the network, network security has also been concerned. Once network security problems occur, it may cause application service interruption, user personal information leakage, and even serious economic losses and national security problems. With the update and iteration of various network technologies and the increase of system structure complexity, it is more and more difficult to monitor traffic data. Network security problems have become one of the hot topics in today's society. Due to the diversification and complexity of network intrusion behaviors, static security protection technologies based on firewalls cannot meet the needs of contemporary network security. Network traffic anomaly detection, as a security protection technology for actively defending network anomalies, can identify illegal intrusion behaviors and protect network information in real time. On this basis, various network traffic anomaly detection technologies have emerged.

[0003] Among them, the network traffic anomaly detection technology based on unsupervised learning has been widely concerned due to its practicality, but the network traffic anomaly detection technology based on unsupervised learning has no prior knowledge, resulting in a high false detection rate of network traffic anomalies. For example, the network traffic anomaly detection algorithm based on K-means has achieved good clustering effect on convex spherical structure data set, but the parameter k needs to be specified in advance, and it is seriously dependent on the initial clustering center, cannot find clusters of arbitrary shape, and the clustering is easily affected by noise or outliers. The network traffic anomaly detection algorithm based on DBSCAN algorithm has good clustering results for irregular cluster, and is not easily affected by noise points and outliers, but the clustering result is poor for processing clusters with large density difference and high-dimensional data. In addition, the clustering result is sensitive to the selection of radius Eps and density threshold MinPts. The network traffic anomaly detection algorithm based on spectral clustering can cluster in any shape sample space and converge to the global optimal solution, but it depends on the initial similarity matrix and has high computational cost. The clustering algorithms used in these unsupervised anomaly detection methods cannot flexibly identify the number of clustering clusters in real-time networks, and most algorithms use Euclidean distance as the similarity measure between data objects. For high-dimensional data, Euclidean distance is not enough to characterize the similarity between data objects.

[0004] Therefore, how to reduce the false detection rate in the network traffic anomaly detection process has become a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The application aims to provide a network traffic anomaly detection method and device, a terminal device and a readable storage medium, so as to solve the problem of high false detection rate in the network traffic anomaly detection process in the prior art.

[0006] In a first aspect, the application provides a network traffic anomaly detection method, comprising:

[0007] Obtaining a network traffic data set to be detected, and constructing a k-neighbor graph corresponding to the network traffic data set by calculating the distance between each two network traffic data;

[0008] Calculating the similarity of each two network traffic data to obtain an initial similarity matrix corresponding to the network traffic data set;

[0009] Determining the local density of each network traffic data according to the k-neighbor graph and the initial similarity matrix, and determining the relative distance corresponding to each network traffic data according to the local density of each network traffic data;

[0010] Determining the edge weight of each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data, and determining the global density of each network traffic data based on the edge weight of each two network traffic data and the Markov chain idea;

[0011] Determining the gamma value of each network traffic data based on the relative distance of each network traffic data and the global density of each network traffic data, and stretching the gamma value of each network traffic data to obtain the gamma transformed value of each network traffic data;

[0012] Determining a cluster center point set based on the gamma transformed value of each network traffic data, and performing hierarchical assignment of each non-cluster center point according to the cluster center point set to obtain a plurality of network traffic data clusters; wherein the cluster center point set contains each cluster center point, the cluster center point refers to the network traffic data as a cluster center, and the non-cluster center point refers to the network traffic data not as a cluster center;

[0013] Determining the abnormal network traffic data based on the plurality of network traffic data clusters.

[0014] In a possible implementation, the calculation of the similarity of each two network traffic data to obtain the initial similarity matrix corresponding to the network traffic data set comprises:

[0015] Setting a similarity target function, wherein the similarity target function is:

[0016]

[0017] wherein x iFor the i-th network traffic data, x j For the j-th network traffic data, s ij The similarity between the i-th network traffic data and the j-th network traffic data is represented by α, which is a preset hyperparameter, and n is the number of network traffic data in the network traffic dataset.

[0018] After optimizing the similarity objective function, the initial similarity matrix corresponding to the network traffic dataset is obtained.

[0019] In one possible implementation, the local density of each network traffic data point is determined as follows:

[0020]

[0021] Where, ρ i Let x be the local density of the i-th network traffic data. i For the i-th network traffic data, d ij Represents the i-th network traffic data x i and the j-th network traffic data x j The distance between them, j∈KNN(x) i ) represents x j For x i Nearest neighbor points on the k-nearest neighbor graph The average distance, For x i x's k nearest neighbors j The variance of the distance between them Represents the i-th network traffic data x i and the j-th network traffic data x j Similarity between them x represents i with k nearest neighbors x j The sum of similarities; x represents i with k nearest neighbors x j The sum of distances.

[0022] In one possible implementation, determining the relative distance corresponding to each network traffic data based on the local density of each network traffic data includes:

[0023]

[0024] Where, δ i ρ represents the relative distance of the i-th network traffic data point. i Let ρ be the local density of the i-th network traffic data. j Let d be the local density of the j-th network traffic data. ijdistance between the i-th network traffic data x i and the j-th network traffic data x j i max(ρ) represents that the local density of the i-th network traffic data is the maximum value in all local densities of the network traffic data.

[0025] In a possible implementation, the determining of the edge weight between each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data comprises:

[0026]

[0027] wherein w ij is the edge weight between the i-th network traffic data and the j-th network traffic data, ρ i is the local density of the i-th network traffic data, ρ j is the local density of the j-th network traffic data, and s ij represents the similarity between the i-th network traffic data and the j-th network traffic data.

[0028] In a possible implementation, the determining of the global density of each network traffic data based on the edge weight between each two network traffic data and the Markov chain idea comprises:

[0029] determining a transition probability matrix of the walk of different step numbers based on the edge weight between each two network traffic data and the Markov chain idea;

[0030] obtaining a consistency transition probability matrix after weighted processing of the transition probability matrix of the walk of different step numbers;

[0031] determining the global trajectory similarity between each two network traffic data according to the consistency transition probability matrix, and determining the global density of each network traffic data based on the global trajectory similarity.

[0032] In a possible implementation, the determining of the cluster center point set based on the γ transformation value of each network traffic data comprises:

[0033] sorting the γ transformation value of each network traffic data to obtain a sorting serial number of each network traffic data;

[0034] determining a target network traffic data for truncation based on the γ transformation value of each network traffic data;

[0035] taking each network traffic data with a sorting serial number before the target network traffic data as a cluster center point to obtain a cluster center point set centers={cc1, cc2,..., cc m}.​

[0036] wherein, cc i represents the i-th cluster center point;

[0037] The step of assigning each non-cluster center point according to the set of cluster center points comprises:

[0038] The first level assignment: if cc p ∈centers, b∈KNN(cc p ) and cc p ∈KNN(b), then Clu[b]=Clu[cc p ]; wherein, b∈KNN(cc p ) means that the network traffic data b belongs to the nearest neighbor point of the cluster center point cc p , cc p ∈KNN(b) means that the cluster center point cc p belongs to the nearest neighbor point of the network traffic data b, Clu[cc p ] means the network traffic data cluster corresponding to the cluster cc p , and Clu[b]=Clu[cc p ] means that the network traffic data b is assigned to the cluster Clu[cc p ];

[0039] The second level assignment: the first network traffic data is assigned to the network traffic data cluster in which the most points of the nearest neighbor points of the first network traffic data are located; wherein, the first network traffic data refers to the network traffic data which has not been assigned after the first level assignment;

[0040] The third level assignment: the second network traffic data is assigned to the network traffic data cluster in which the relative point of the second network traffic data is located; wherein, the second network traffic data refers to the network traffic data which has not been assigned after the second level assignment, and the relative point of the second network traffic data refers to the network traffic data which has a higher local density and is closest to the second network traffic data.

[0041] The second aspect of the embodiment of the application provides a network traffic anomaly detection device, comprising:

[0042] a data clustering module, wherein the data clustering module is configured to perform the following steps:

[0043] obtain a network traffic data set to be detected, and construct a k-nearest neighbor graph corresponding to the network traffic data set by calculating the distance between each two network traffic data;

[0044] calculate the similarity between each two network traffic data to obtain an initial similarity matrix corresponding to the network traffic data set;

[0045] determine a local density of each network traffic data according to the k-neighbor graph and the initial similarity matrix, and determine a relative distance corresponding to each network traffic data according to the local density of each network traffic data;

[0046] determine an edge weight between each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data, and determine a global density of each network traffic data based on the edge weight between each two network traffic data and a Markov chain thought;

[0047] determine a gamma value of each network traffic data based on the relative distance of each network traffic data and the global density of each network traffic data, and stretch the gamma value of each network traffic data to obtain a gamma transformed value of each network traffic data;

[0048] determine a cluster center point set based on the gamma transformed value of each network traffic data, and perform a step-by-step assignment of each non-cluster center point according to the cluster center point set to obtain a plurality of network traffic data clusters; wherein the cluster center point set contains each cluster center point, the cluster center point refers to a network traffic data as a cluster center, and the non-cluster center point refers to a network traffic data not as a cluster center;

[0049] an anomaly detection module, the anomaly detection module being configured to:

[0050] determine an abnormal network traffic data based on the plurality of network traffic data clusters.

[0051] In a third aspect, the present application provides a terminal device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the network traffic anomaly detection method when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the network traffic anomaly detection method when executed by a processor.

[0053] The network traffic anomaly detection method and device, the terminal device, and the computer readable storage medium provided by the present application have the following advantages:

[0054] Different from the prior art, the network flow data is converted into graph data for processing, the global density of the data is better considered through the connectivity of the graph and the reachability between the network flow data, and the gamma value of each network flow data is stretched, so that the distinction degree of the cluster center and the non-cluster center is effectively increased. Therefore, the network flow data can be more reasonably clustered, and the abnormal network flow data can be more accurately screened out. That is, the false detection rate is effectively reduced, the accuracy of the network flow anomaly detection is improved, and the problems in the prior art are solved. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The flowchart of the network flow anomaly detection method provided by an embodiment of the present application is shown in the figure.

[0057] Figure 2 The flowchart of the network flow anomaly detection method provided by another embodiment of the present application is shown in the figure.

[0058] Figure 3 The structure block diagram of the network flow anomaly detection device provided by an embodiment of the present application is shown in the figure.

[0059] Figure 4 The schematic block diagram of the terminal device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0060] In the following description, specific details are set forth in order to provide a thorough understanding of embodiments of the present application. However, persons of ordinary skill in the art will readily recognize that embodiments of the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0061] In order to make the objects, technical solutions and advantages of the present application clearer, the following will be described with reference to the accompanying drawings through specific embodiments.

[0062] Reference should be made to Figure 1 , Figure 1 The flowchart of the network flow anomaly detection method provided by an embodiment of the present application is shown in the figure, and the method comprises:

[0063] S101: Obtain a network traffic data set to be detected, and construct a k-neighbor graph corresponding to the network traffic data set by calculating the distance between each two network traffic data.

[0064] In this embodiment, the network traffic data set can also be preprocessed before the k-neighbor graph of the network traffic data set is constructed. The preprocessing of the network traffic data includes data integrity, character data numericalization, data standardization, and data attribute dimension reduction.

[0065] In this embodiment, in order to fully consider the local information of the sample (wherein the sample refers to the network traffic data, and the sample set refers to the network traffic data set, which is similar in subsequent embodiments and will not be repeated), the value of k can be set, for example, the value of k can be set to 8. By calculating the distance between samples, the nearest 8 samples of each sample are constructed to form a sample neighbor graph, and the K-neighbor graph of the data set is obtained, and the graph model is represented as G=(V, E, W), V represents the sample set, that is, V={x1, x2,..., x n},<x i ,x j > represents that the sample x i is connected with x j , and E is the edge set. W represents the edge weight.

[0066] The sample distribution in the data set is independent. When studying the similarity between samples, the existing algorithm usually only relies on the Euclidean distance between samples, and only focuses on the local similarity of samples, without considering the neighbor information of samples and the connectivity between samples, which is easy to cause the distance between the edge points of two clusters to be small, and the similarity to be large, thereby reducing the final clustering accuracy. The embodiment of the application constructs a k-neighbor graph of the sample, realizes the connection between points of a single sample, converts the data into graph data for research, and through the edge connection between samples, it can also be explained that two samples are similar. The subsequent design of the objective function learns the global similarity matrix as the initial similarity matrix of the algorithm, fully considers the global density of the sample, and can more reasonably cluster.

[0067] S102: Calculate the similarity between each two network traffic data to obtain an initial similarity matrix corresponding to the network traffic data set.

[0068] In one possible implementation manner of the embodiment, the similarity between each two network traffic data is calculated to obtain an initial similarity matrix corresponding to the network traffic data set, including:

[0069] The similarity objective function is set, wherein the similarity objective function is:

[0070]

[0071] Where, x i For the i-th network traffic data, x j For the j-th network traffic data, s ij Let α represent the similarity between the i-th network traffic data and the j-th network traffic data, where α is a preset hyperparameter and n is the number of network traffic data in the network traffic dataset.

[0072] After optimizing the similarity objective function, the initial similarity matrix corresponding to the network traffic dataset is obtained.

[0073] This invention is based on manifold dataset interpretation. If two data points (i.e., two network traffic data points) are very close, then they are also very close to each other in the graph. Therefore, the initial similarity matrix S can be obtained through the above formula, where α is a hyperparameter. This formula not only focuses on the similarity between two samples in terms of distance, but also considers the distance between each sample and all other samples to obtain the initial similarity matrix. This matrix is ​​symmetric, i.e., s ij =s ji .

[0074] S103: Determine the local density of each network traffic data point based on the k-nearest neighbor graph and the initial similarity matrix, and determine the relative distance of each network traffic data point based on its local density. The relative distance of a given network traffic data point is the distance between that network traffic data point and the nearest point with a higher local density.

[0075] In this embodiment, based on the characteristics of dense and sparse sample distribution, and combining the definitions of sample variance and sample similarity, a method for measuring local density is given, namely:

[0076]

[0077] Where, ρ i Let x be the local density of the i-th network traffic data. i For the i-th network traffic data, d ij Represents the i-th network traffic data x i and the j-th network traffic data x j The distance between them, j∈KNN(x) i ) represents x j For x i Nearest neighbor points on the k-nearest neighbor graph The average distance, For x i x's k nearest neighbors j The variance of the distance between them Represents the i-th network traffic data x i and the j-th network traffic data xj the similarity between x represents x i the sum of the similarities between x j and k nearest neighbors x represents x i the sum of the distances between x j and k nearest neighbors x i .

[0078] The smaller the distance variance is, the smaller the dispersion degree between x j and k nearest neighbors x i is. The smaller the distance sum is, the more concentrated the data distribution around x i is, and the local density is larger. Therefore, the local density measurement method provided by the embodiment of the present application can more truly reflect the spatial features between samples.

[0079] In a possible implementation manner of the embodiment, the relative distance corresponding to each network traffic data is determined according to the local density of each network traffic data, and the method comprises the following steps:

[0080]

[0081] wherein, δ i represents the relative distance of the i th network traffic data, ρ j is the local density of the i th network traffic data, ρ ij represents the distance between the i th network traffic data x i and the j th network traffic data x j , and ρ i = max ( ρ ) represents that the local density of the i th network traffic data is the maximum value in the local densities of all network traffic data.

[0082] In the prior art, due to different distribution features and different scales of data sets, the local density measurement methods are not unified, that is, two methods of a truncated kernel and an exponential kernel function are used to calculate the local density. When the local density is calculated, the truncated distance is an important parameter for measuring the density, which needs to be set according to the characteristics of the data set. Different values are selected for different data sets, which directly affects the local density of the sample and further affects the final clustering result. Therefore, in the density calculation, it is particularly important to realize fixed parameters or non-parameters in the clustering algorithm to avoid the influence of the truncated distance. According to the similarity information and k-neighbor information between samples, the sample density measurement is redefined in the embodiment of the present application, the influence of the parameters is avoided, the local density measurement is unified, and the spatial distribution features of the samples are fully considered.

[0083] S104: determine the edge weight between each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data, and determine the global density of each network traffic data based on the edge weight between each two network traffic data and the Markov chain thought.

[0084] In the embodiment, the edge weight between each two network traffic data on the k-neighbor graph is determined based on the initial similarity matrix and the local density of each network traffic data, including:

[0085]

[0086] wherein, w ij is the edge weight between the i th network traffic data and the j th network traffic data, p i is the local density of the i th network traffic data, p j is the local density of the j th network traffic data, s ij represents the similarity between the i th network traffic data and the j th network traffic data.

[0087] In the above embodiment, a more reasonable symmetric similarity matrix (i.e. the initial similarity matrix) can be obtained through the objective function. However, in order to be more consistent with the idea of density peak clustering, i.e. each sample finds the point with larger density and closest distance when assigning, the embodiment of the application integrates this idea into the calculation of the edge weight between samples, so that the edge weight between the sample and the sample with larger density is larger when the sample walks on the k-neighbor graph. This means that the motion track with larger edge weight is more essential for the sample, which represents that the similarity between the samples is large. Therefore, the edge weight between samples is calculated through the above formula. As can be seen from the above formula, given two sample points, the edge weight between the sample with lower density and the sample with higher density is large, while the edge weight between the sample with higher density and the sample with lower density is small, which means that when walking and transferring on the k-neighbor graph, the sample with lower density is more inclined to the area where the sample with higher density is located, and the corresponding transition probability matrix value under the Markov chain thought is large, and the similarity between the two samples is large.

[0088] In a possible implementation, the global density of each network traffic data is determined based on the edge weight between each two network traffic data and the Markov chain thought, including:

[0089] The transition probability matrix of the walk with different steps is determined based on the edge weight between each two network traffic data and the Markov chain thought.

[0090] After weighting processing the transition probability matrix of the walk with different steps, a consistent transition probability matrix is obtained.

[0091] The global trajectory similarity between each two network traffic data is determined according to the consistent transition probability matrix, and the global density of each network traffic data is determined based on the global trajectory similarity.

[0092] In the embodiment, according to the Markov chain idea, P(y1, y2, y n-1 , y n ) = P(y i |y i-1 ), the embodiment of the application can define a directed one-step walk probability transition matrix P = D -1 W, D is a diagonal matrix, and the diagonal matrix is (where w ij is the edge weight of the edge between the i th network traffic data and the j th network traffic data, and d ii is the data in the i th row and the i th column of the diagonal matrix), in the embodiment, the walk is not random walk, but purposeful walk to the point with the largest weight. The transition probability matrix after t-step walk (t>1) is P (t) = P (t-1) P, since the transition probability matrices corresponding to different steps are different, corresponding to different trajectories in each step of walk, therefore, the embodiment can weight process different transition probability matrices, and the weight of each matrix is obtained by the following formula and the converged consistent transition probability matrix P * (also referred to as a consistent trajectory matrix). The matrix weight The iteration update formula is as follows. The embodiment of the application can solve the consistent trajectory matrix P * and the transition probability matrix weight of different step walks by alternating optimization iteration.

[0093]

[0094]

[0095] wherein, indicates the j th row of the consistent trajectory matrix P * , P j (t) indicates the j th row of the transition probability matrix P after t-step walk, indicates P j (t) corresponding weight, and β is a preset regularization term parameter, and F is a preset value.

[0096] In the embodiment, the global similarity between samples can be measured based on the consistent transition probability matrix. P * is further optimized on the local similarity measure S considering the connectivity of the graph and the reachability of the samples, and P *Each row represents the global trajectory of each sample point, and the similarity of the trajectories between sample points is calculated to measure the similarity between samples. i Similar to x j According to the edge weight metric defined above, x i Similar to x j The possibility of trajectory walking between two points is greater, and there are parent and child nodes, that is, points with small density walk to points with large density, at which time the two points can be considered as bound nodes, and the walking trajectories are basically the same. Therefore, the cosine similarity is used to measure the similarity of the walking trajectories of two samples, that is, And The similarity measure is as follows:

[0097]

[0098] wherein, P * represents the consistent trajectory matrix P Similarly.

[0099] On this basis, the sample points obtain a unified consistent trajectory matrix P * after walking on the neighborhood graph. By calculating the similarity of the walking trajectories of the sample points, the points that can be reached by the sample points within t steps are obtained, that is, the points with greater similarity, and the points that cannot be reached have minimal similarity. The global density of the sample points is measured by calculating the similarity of the sample points with the reachable points within t steps and summing them, that is, the calculation method of the global density of each network flow data (or the global density of each sample point) is:

[0100]

[0101] In this embodiment, in order to consider the connectivity of the graph and the reachability between samples, the Markov Chains idea is used to calculate the probability of walking to the next state between samples in the graph, and the next state is determined by the current state and is irrelevant to other time states. Different step walking obtains different transition probability matrices, and t-step walking means reaching the destination sample from the source sample through t steps, that is, the similarity is propagated to the destination sample through the intermediate samples, and the larger the t is, the more stable the transition probability matrix is. Since each transition probability matrix is obtained by walking at different steps, that is, the source sample reaches the destination sample through different samples and different motion trajectories, each transition probability matrix is different, which also indicates that the similarity between samples is also different at different steps, and the similarity matrix at different steps is also different. Therefore, the present application weights and fuses different transition probability matrices by designing a target function to obtain a consistent similarity matrix, so that the similarity matrix between samples considers the local structure of the samples and the reachability between samples, and is more reasonable.

[0102] S105: Determine the gamma value of each network traffic data based on the relative distance of each network traffic data and the global density of each network traffic data, and stretch the gamma value of each network traffic data to obtain the gamma transformed value of each network traffic data.

[0103] In this embodiment, the relative distance δ i and the product of the density p i is the gamma i , that is

[0104] In this embodiment, the gamma value can be transformed by a Sigmoid function to increase the distinction between the cluster center and the non-cluster center. The existing algorithm usually selects the cluster center point intuitively on the decision graph, which is easy to cause multiple selection and omission of the center point, and there is a problem that the large value is larger and the small value is smaller on the gamma distribution graph, which causes the center point to be easily omitted when selecting the center point. Therefore, the gamma value is transformed by a Sigmoid function in the present application to stretch the sample gamma value at a certain position. The transformation function is derived from the stretching and translation of The transformation Sigmoid function is as follows:

[0105]

[0106] Wherein, the purpose of lambda is to stretch the gamma value, and the purpose of c is to stretch some samples near x=c and increase the distinction between the cluster center and the non-cluster center. It is found through analysis of the synthetic data set, the UCI real data set and the KDD Cup99 data set that most samples are relatively dense at gamma<0.2. Therefore, when c=0.2gamma, some samples near the center can be stretched, and the cluster center can be accurately determined by using the decision function, and the performance of the algorithm is better. Therefore, the following transformation function can be used to determine the gamma transformed value.

[0107]

[0108] Wherein, x in the above formula refers to the gamma value, which is the gamma transformed value.

[0109] The existing algorithm draws a two-dimensional decision graph with the local density as the horizontal coordinate and the relative distance as the vertical coordinate when selecting the center point, and marks the sample points with high and as the cluster center. The human subjective influence is large, and it is difficult to intuitively select the center point for the data set with uneven and large difference in density distribution. The Sigmoid function can stretch some samples at a specific value in the present application embodiment, increase the distinction between the cluster center and the non-cluster center, make it more obvious on the decision graph, and determine the cluster center through the most obvious part of the transformed gamma distribution graph.

[0110] S106: Determine a cluster center point set based on the gamma transformation value of each network traffic data, and perform step-by-step assignment of each non-cluster center point according to the cluster center point set, to obtain a plurality of network traffic data clusters. The cluster center point set contains each cluster center point, and the cluster center point refers to network traffic data as a cluster center, and the non-cluster center point refers to network traffic data that is not a cluster center.

[0111] In this embodiment, in a possible implementation, determining a cluster center point set based on the gamma transformation value of each network traffic data includes:

[0112] Sort the gamma transformation value of each network traffic data to obtain a sorting sequence number of each network traffic data.

[0113] Determine a target network traffic data for truncation based on the gamma transformation value of each network traffic data.

[0114] Take each network traffic data before the sorting sequence number of the target network traffic data as a cluster center point to obtain a cluster center point set centers = {cc1, cc2,..., cc m}.

[0115] Wherein, cc i represents the i-th cluster center point.

[0116] Perform step-by-step assignment of each non-cluster center point according to the cluster center point set to obtain a plurality of network traffic data clusters, including:

[0117] First-level assignment: if cc p ∈centers, b∈KNN(cc p ) and cc p ∈KNN(b), then Clu[b] = Clu[cc p ]. Wherein, b∈KNN(cc p ) means that the network traffic data b belongs to the near neighbor point of the cluster center point cc p , cc p ∈KNN(b) means that the cluster center point cc p belongs to the near neighbor point of the network traffic data b, Clu[cc p ] means the network traffic data cluster corresponding to the cluster center cc p , and Clu[b] = Clu[cc p ] means that the network traffic data b is assigned to the cluster Clu[cc p ].

[0118] Second-level allocation: The first network traffic data is allocated to the network traffic data cluster containing the most neighboring nodes. Here, the first network traffic data refers to the network traffic data that has not yet been allocated after the first-level allocation.

[0119] Third-level allocation: The second network traffic data is allocated to the network traffic data cluster where its relative point is located. Here, the second network traffic data refers to the network traffic data that has not yet been allocated after the second-level allocation. The relative point of the second network traffic data refers to the network traffic data with a higher local density and the closest distance.

[0120] In this embodiment, it can be Sort in descending order and find two samples The point with the largest difference is used as the cutoff point, and the points before the i-th sample are determined as the cluster centers. The cutoff point (i.e., the target network traffic data) is the center. i The method for determining it is as follows:

[0121]

[0122] In this embodiment, after determining the cluster center set, cluster labels can be assigned to the cluster centers first, and then the k nearest neighbors of the cluster centers can be assigned. If c p ∈centers, b∈KNN(cc p And cc p If ∈KNN(b), then Clu[b]=Clu[cc] p ]; For other unassigned points, assign them to the cluster containing the most points among their k nearest neighbors; if there are still unassigned points, i.e., Clu[x i If ] = -1, then it is assigned to the cluster containing the point with a local density greater than its closest point.

[0123] Existing algorithms assign non-clustered centroids in a single step, directly placing them into clusters with higher density and closer proximity than the original centroids. This approach is prone to cascading errors, reducing clustering accuracy. This invention, however, considers the k-nearest neighbor information of a sample and assigns centroids step-by-step, thus avoiding cascading errors.

[0124] S107: Identify abnormal network traffic data based on multiple network traffic data clusters.

[0125] In this embodiment, in the network traffic anomaly detection system, the abnormal traffic in the network dataset is much smaller than the normal traffic; therefore, the smaller clusters can be defined as abnormal traffic clusters. Each abnormal traffic cluster contains only abnormal traffic. The larger traffic clusters are defined as normal clusters, which contain only normal traffic.

[0126] Specifically, the number threshold can be set according to the number of network traffic data in each network traffic data cluster, and if the number of network traffic data in a certain network traffic data cluster is less than the number threshold, each network traffic data in the network traffic data cluster is taken as abnormal network traffic data. Alternatively, each network traffic data cluster is sorted in ascending order according to the number of network traffic data in each network traffic data cluster, and the network traffic data in the network traffic data cluster ranked in the first q are taken as abnormal network traffic data. Wherein, q is a preset value.

[0127] The embodiments of the present application are tested on UCI, artificial data set and CICIDS2017 data set. The experimental results on the artificial data set and the UCI data set show that the algorithm can accurately select the clustering center adaptively, and to a certain extent, improve the clustering performance. The experimental results on the CICIDS2017 data set show that the present application improves the abnormal network traffic detection precision and reduces the false detection rate while ensuring the practicability of the unsupervised network traffic anomaly detection system.

[0128] From the above, it can be concluded that the application adopts the idea that the local density of the sample is greater than the local density of the neighbor samples around it, and the distance between the cluster centers of different class clusters is relatively far. The existing local density measurement method, the selection of the cluster center and the assignment of the non-cluster center point are improved, and a density peak clustering method suitable for network traffic anomaly detection is proposed by fusing the transition probability matrix, which improves the detection rate of abnormal network traffic. In this embodiment, the sample data is converted into graph data for research, and a k-neighbor graph of the sample is constructed, each sample being a graph node, the connection between the nodes being the weight of the two nodes, and the edge weight being defined according to the sample point density idea and being able to represent the initial similarity between the two samples, the similarity between the low-density node and the high-density node being large. Starting from any sample on the graph, random walk is performed, and each step obtains the multi-order structure between the samples. Since the transition probability matrix obtained by different step numbers of random walk on the sample is different, the transition probability matrix of different step numbers affects the sample similarity and also affects the final clustering result. The transition probability matrix of different step numbers is automatically weighted in the application, and a consistent transition probability matrix is obtained by fusing the multi-step transition probability matrix, the similarity of each node walk trajectory is calculated, and the similarity between the samples is more comprehensively measured, considering the connectivity of the graph and the connection relationship between the samples. Moreover, the sample density measurement is redefined in the application, so that the density more reflects the sample space distribution characteristics. The normalized gamma is transformed by using a Sigmoid function transformation to increase the discrimination degree of the cluster center and the non-cluster center. At the same time, the non-sample point assignment strategy is optimized. Through experiments on UCI and artificial data sets, the algorithm can accurately select the cluster center, improve the clustering performance to a certain extent, improve the detection rate of abnormal network traffic, and reduce the false detection rate. Compared with the existing supervised anomaly detection technology, the algorithm does not need prior knowledge such as sample label constraint, pair similarity constraint, etc., reducing the time cost of manual marking. Compared with the existing unsupervised network traffic anomaly detection technology, the algorithm retains the practicability of the network traffic anomaly detection system while improving the detection rate of abnormal network traffic and reducing the false detection rate.

[0129] Please refer to Figure 2 The application also gives a specific implementation example based on UCI data set and artificial data set, CICIDS2017 data set, wherein the specific steps are as follows:

[0130] Step one: read in the data.

[0131] (1) directly read in the UCI data set and artificial data set.

[0132] (2) when reading in the CICIDS data set:

[0133] Data sampling: The data capture of the CICIDS2017 dataset experienced 5 days (July 3-7, 2017, Monday to Friday), and only the normal network traffic dataset was captured on Monday, and the normal and abnormal network traffic dataset was captured from Tuesday to Friday, which contains 15 sample labels, of which 14 are abnormal classes, and the dataset has a total of 2830743 data. Due to the large amount of data, the present application adopts a sampling strategy, which is divided into 5 data sets, and the normal traffic is much more than the abnormal traffic, D1={normal traffic=2000, abnormal traffic 500}, D2={normal traffic=5000, abnormal traffic 1000}, D3={normal traffic=10000, abnormal traffic 2000}, D4={normal traffic=20000, abnormal traffic 5000}, D5={normal traffic=100000, abnormal traffic 20000}, to ensure that as much abnormal traffic type as possible is included and the amount of data of the same abnormal traffic type is also large.

[0134] (3) Data preprocessing. The UCI dataset and synthetic dataset used by the present application are standard clustering datasets and do not require data preprocessing. The network traffic dataset CICIDS2017 has the following normal traffic behavior as an example:

[0135] (53, 199, 2, 2, 88, 188, 44, 44, 44, 0, 94, 94, 94, 0, 1386934.673, 20100.50251, 66.33333333, 30.89228598, 102, 48, 49, 49, 0, 49, 49, 48, 48, 0, 48, 48, 0, 0, 0, 0, 64, 64, 10050.25126, 10050.25126, 44, 94, 64, 27.38612788, 750, 0, 0, 0, 0, 0, 0, 0, 0, 1, 80, 44, 94, 64, 0, 0, 0, 0, 0, 0, 2, 88, 2, 188, -1, -1, 1, 32, 0, 0, 0, 0, 0, 0, 0, 0)

[0136] (a) Data completion. In the network traffic dataset CICIDS2017, there are null values in the Flow Bytes / s and Flow Packets / s features, which are filled with the maximum value of this feature.

[0137] (b) Character data numericalization. The CICIDS2017 dataset has time stamp, source port number, destination port number, and packet size for each network traffic feature, and there is no character data, so character data numericalization is not needed.

[0138] (c) Data standardization. In order to eliminate the influence of different dimensions between different features, Max-Min normalization is performed on the CICIDS2017 dataset, as shown in the following formula, so that the data are comparable.

[0139]

[0140] (d) Data attribute dimension reduction. The CICIDS2017 dataset has 78-dimensional features. First, the repeated feature FwdHeaderLength is deleted. Then, mutual information is used as a feature selection method to measure the relationship between features and categories. The order of features is determined by sorting the mutual information, and feature selection is performed. The mutual information formula for calculating the features is as follows, where the feature set of the dataset is F = {d1, d2,..., d m}, m = 77, the value of feature d i is v i , |V i | is the value range of feature d i . The sample has |V c | categories. p(v i , v c ) is the probability that the value of feature d i is v i and the category value is v c . The greater the value of I(d i , c) indicates that the greater the association between feature d i and category c, and d i can be used as an important feature to distinguish different categories of network traffic.

[0141]

[0142] Step two: calculate the Euclidean distance between samples

[0143]

[0144] Step three: construct the sample K-nearest neighbor graph, and the value of k is set to 8. Through the Euclidean distance between samples, the nearest 8 samples of each sample are constructed to form a sample neighbor graph, and the K-nearest neighbor graph of the dataset is obtained. The graph model is represented as G = (V, E, W), V represents the sample set, i.e. V = {x1, x2,..., x n}, <x i , x j > represents that the sample x i is connected to x j , and E is the edge set. W represents the edge weight.

[0145] Step four: calculate the relative distance of the sample.

[0146]

[0147] Step five: Calculate the sample similarity. α is the regularization term parameter.

[0148]

[0149] Step six: Calculate the sample local density.

[0150]

[0151] Step eight: Calculate the sample node edge weight.

[0152]

[0153] Step nine: Weighted fusion transfer probability matrix, alternating optimization to the objective function convergence. β is the regularization term parameter.

[0154]

[0155]

[0156] Step ten: According to the stability of the transition probability matrix, the sample global trajectory similarity is calculated, and the global similarity between samples is measured.

[0157]

[0158] Step eleven: Calculate the sample global density ρ * .

[0159]

[0160] Step twelve: Calculate the product of the sample relative distance δ i and the density γ i .

[0161]

[0162] Step thirteen: Sigmoid function transformation is performed on γ. The samples γ are arranged in ascending order to stretch the sample γ value at a certain position. The transformation function is derived from the scaling and translation of .

[0163]

[0164] Step fourteen: Determine the clustering center point. Arrange in descending order, find the maximum difference between two samples as the cut-off, and determine the points before the ith sample as the clustering center point. Set the clustering center point set as centers = {cc1, cc2,..., ccm}。

[0165]

[0166] Step fifteen: optimizing the allocation of non-cluster center points. First, assign cluster labels to cluster centers, then assign k-nearest neighbors of cluster centers, if cc p ∈centers, b∈KNN(cc p ) and cc p ∈KNN(b), then Clu[b]=Clu[cc p ]; for other unassigned points, assign them to the cluster where the most points of their k-nearest neighbors are located; if there are still unassigned points, i.e., Clu[x i ]=-1, then assign the unassigned points to the cluster where the point with the largest density and the closest distance is located.

[0167] Step thirteen: output the final clustering result. The evaluation indexes adopted by the present application are: calculation accuracy (ACC), false detection rate (FAR), adjusted mutual information (AMI), adjusted Land index (ARI) and FM index (FMI). After index evaluation, the present application embodiment can obtain good clustering effect and abnormal flow detection effect.

[0168] Corresponding to the network flow anomaly detection method of the above embodiment, Figure 3 is a structural block diagram of a network flow anomaly detection device provided by an embodiment of the present application. For ease of illustration, only parts related to the embodiment of the present application are shown. For parts not related to the embodiment of the present application, refer to Figure 3 The network flow anomaly detection device 20 comprises a data clustering module 21 and an anomaly detection module 22.

[0169] The data clustering module 21 is configured to perform the following steps:

[0170] Obtain a network flow data set to be detected, and construct a k-nearest neighbor graph corresponding to the network flow data set by calculating the distance between each two network flow data.

[0171] Calculate the similarity of each two network flow data to obtain an initial similarity matrix corresponding to the network flow data set.

[0172] Determine the local density of each network flow data according to the k-nearest neighbor graph and the initial similarity matrix, and determine the relative distance corresponding to each network flow data according to the local density of each network flow data. The relative distance of a certain network flow data is the distance between the network flow data and the point with the highest local density and the closest distance.

[0173] An edge weight between each two network traffic data on the k-neighbor graph is determined based on the initial similarity matrix and the local density of each network traffic data, and a global density of each network traffic data is determined based on the edge weight between each two network traffic data and a Markov chain idea.

[0174] A γ value of each network traffic data is determined based on the relative distance of each network traffic data and the global density of each network traffic data, and the γ value of each network traffic data is stretched to obtain a γ transformed value of each network traffic data.

[0175] A cluster center point set is determined based on the γ transformed value of each network traffic data, and each non-cluster center point is assigned step by step according to the cluster center point set to obtain a plurality of network traffic data clusters. The cluster center point set contains each cluster center point, and the cluster center point refers to the network traffic data as a cluster center, and the non-cluster center point refers to the network traffic data not as a cluster center.

[0176] The anomaly detection module 22 is configured to:

[0177] An abnormal network traffic data is determined based on the plurality of network traffic data clusters.

[0178] In a possible implementation, the data clustering module 21 is specifically configured to:

[0179] A similarity target function is set, and the similarity target function is:

[0180]

[0181] wherein x i is the i th network traffic data, x j is the j th network traffic data, s ij represents the similarity between the i th network traffic data and the j th network traffic data, and α is a preset hyperparameter, and n is the number of network traffic data in the network traffic data set.

[0182] After the similarity target function is optimized and solved, an initial similarity matrix corresponding to the network traffic data set is obtained.

[0183] In a possible implementation, the data clustering module 21 is specifically configured to calculate the local density of each network traffic data by the following formula:

[0184]

[0185] wherein ρ i is the local density of the i th network traffic data, x i is the i th network traffic data, and d ijRepresents the i-th network traffic data x i and the j-th network traffic data x j The distance between them, j∈KNN(x) i ) represents x j For x i The nearest neighbor points on the k-nearest neighbor graph The average distance, For x i x's k nearest neighbors j The variance of the distance between them Represents the i-th network traffic data x i and the j-th network traffic data x j Similarity between them x represents i with k nearest neighbors x j The sum of similarities. x represents i with k nearest neighbors x j The sum of distances.

[0186] In one possible implementation, the data clustering module 21 is specifically used to determine the relative distance corresponding to each network traffic data point using the following formula:

[0187]

[0188] Where, δ i ρ represents the relative distance of the i-th network traffic data point. i Let ρ be the local density of the i-th network traffic data. j Let d be the local density of the j-th network traffic data. ij Represents the i-th network traffic data x i and the j-th network traffic data x j The distance between them, ρ i =max(ρ) indicates that the local density of the i-th network traffic data is the maximum local density among all network traffic data.

[0189] In one possible implementation, the data clustering module 21 is specifically used to determine the edge weights of every two network traffic data points on the k-nearest neighbor graph using the following formula:

[0190]

[0191] Among them, w ij Let ρ be the edge weight between the i-th network traffic data and the j-th network traffic data. i Let ρ be the local density of the i-th network traffic data. j Let s be the local density of the j-th network traffic data. ijrepresents the similarity between the ith network traffic data and the jth network traffic data.

[0192] In a possible implementation, the data clustering module 21 is specifically configured to:

[0193] The transition probability matrix of the different step walks is determined based on the edge weight of each two network traffic data and the Markov chain idea.

[0194] After the transition probability matrix of the different step walks is weighted, a consistent transition probability matrix is obtained.

[0195] The global trajectory similarity between each two network traffic data is determined according to the consistent transition probability matrix, and the global density of each network traffic data is determined based on the global trajectory similarity.

[0196] In a possible implementation, the data clustering module 21 is specifically configured to:

[0197] The γ transformation value of each network traffic data is sorted to obtain a sorting sequence number of each network traffic data.

[0198] The target network traffic data for truncation is determined based on the γ transformation value of each network traffic data.

[0199] The network traffic data before the sorting sequence number of the target network traffic data is taken as a clustering center point, and a clustering center point set centers = {cc1, cc2,..., cc m} is obtained.

[0200] Wherein, cc i represents the ith clustering center point.

[0201] The data clustering module 21 is specifically configured to perform the following steps:

[0202] First level allocation: if cc p ∈centers, b∈KNN(cc p ) and cc p ∈KNN(b), then Clu[b] = Clu[cc p ]. Wherein, b∈KNN(cc p ) represents that the network traffic data b belongs to the near neighbor point of the clustering center point cc p , cc p ∈KNN(b) represents that the clustering center point cc p belongs to the near neighbor point of the network traffic data b, Clu[cc p ] represents the network traffic data cluster corresponding to the clustering center cc p , Clu[b] = Clu[cc p ].This indicates that network traffic data b is allocated to cluster Clu[cc]. p ].

[0203] Second-level allocation: The first network traffic data is allocated to the network traffic data cluster containing the most neighboring nodes. Here, the first network traffic data refers to the network traffic data that has not yet been allocated after the first-level allocation.

[0204] Third-level allocation: The second network traffic data is allocated to the network traffic data cluster where its relative point is located. Here, the second network traffic data refers to the network traffic data that has not yet been allocated after the second-level allocation. The relative point of the second network traffic data refers to the network traffic data with a higher local density and the closest distance.

[0205] See Figure 4 , Figure 4 This is a schematic block diagram of a terminal device provided according to an embodiment of the present invention. Figure 4 The terminal 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules / units in the above-described device embodiments, such as... Figure 3 The functions of modules 21 and 22 shown.

[0206] It should be understood that, in this embodiment of the invention, the processor 301 may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0207] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0208] The memory 304 can include read-only memory and random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 can also include non-volatile random access memory. For example, the memory 304 can also store device type information.

[0209] In specific implementations, the processor 301, the input device 302, and the output device 303 described in the embodiments of the present application can execute the implementation manners described in the first embodiment and the second embodiment of the network traffic anomaly detection method provided by the embodiments of the present application, and can also execute the implementation manners of the terminal described in the embodiments of the present application, which will not be described here.

[0210] In another embodiment of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program includes program instructions, and the program instructions are executed by a processor to implement all or part of the processes of the above-mentioned embodiments. The computer program can also be executed by the related hardware to complete the computer program. The computer program can be stored in a computer readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0211] The computer readable storage medium can be an internal storage unit of the terminal of any of the preceding embodiments, such as a hard disk or a memory of the terminal. The computer readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit and an external storage device of the terminal. The computer readable storage medium is used to store computer programs and other programs and data required by the terminal. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0212] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal and the unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0213] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal and the unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0214] In several embodiments provided in the present application, it should be understood that the disclosed terminal and method can be implemented by other ways. For example, the apparatus embodiment described above is only schematic, for example, the division of the unit is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces or units, and can also be electrical, mechanical or other forms of connection.

[0215] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0216] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0217] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A network traffic anomaly detection method, characterized by, The method comprises the following steps: obtaining a network traffic data set to be detected, and constructing a k-neighbor graph corresponding to the network traffic data set by calculating the distance between each two network traffic data; calculating the similarity between each two network traffic data to obtain an initial similarity matrix corresponding to the network traffic data set; determining the local density of each network traffic data according to the k-neighbor graph and the initial similarity matrix, and determining the relative distance corresponding to each network traffic data according to the local density of each network traffic data; determining the edge weight of each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data, and determining the global density of each network traffic data based on the edge weight of each two network traffic data and the Markov chain idea, comprising: determining the transition probability matrix of different step number walks based on the edge weight of each two network traffic data and the Markov chain idea; obtaining a consistency transition probability matrix after weighted processing of the transition probability matrix of different step number walks; determining the global trajectory similarity between each two network traffic data according to the consistency transition probability matrix, and determining the global density of each network traffic data based on the global trajectory similarity; determining the γ value of each network traffic data based on the relative distance of each network traffic data and the global density of each network traffic data, and stretching the γ value of each network traffic data to obtain the γ transformed value of each network traffic data; The cluster center point set is determined based on the gamma transformation value of each network traffic data, including: sorting the gamma transformation value of each network traffic data to obtain a sorting serial number of each network traffic data; determining target network traffic data for truncation based on the gamma transformation value of each network traffic data; and taking each network traffic data with a sorting serial number before the target network traffic data as a cluster center point to obtain a cluster center point set ; wherein, represents the th cluster center point; And according to the cluster center point set, each non-cluster center point is assigned step by step to obtain a plurality of network traffic data clusters, including: first level assignment: if , and , then ; wherein, represents network traffic data belonging to the near neighbor point of the cluster center point, represents the near neighbor point of the cluster center point belonging to the network traffic data, represents the cluster center corresponding network traffic data cluster, represents the assignment of network traffic data to the cluster ; second level assignment: the first network traffic data is assigned to the network traffic data cluster where the most points of its near neighbor points are located; wherein, the first network traffic data refers to the network traffic data that has not been assigned after the first level assignment; third level assignment: the second network traffic data is assigned to the network traffic data cluster where its relative point is located; wherein, the second network traffic data refers to the network traffic data that has not been assigned after the second level assignment; the relative point of the second network traffic data refers to the network traffic data with higher local density and closest distance; wherein, the cluster center point set contains each cluster center point, the cluster center point refers to the network traffic data as the cluster center, and the non-cluster center point refers to the network traffic data that is not as the cluster center.​ determining the abnormal network traffic data based on the plurality of network traffic data clusters.

2. The network traffic anomaly detection method of claim 1, wherein, The calculation of the similarity between each two network traffic data to obtain the initial similarity matrix corresponding to the network traffic data set comprises: setting a similarity target function, wherein the similarity target function is: in, For the first Network traffic data, For the first Network traffic data, Indicates the first The network traffic data and the first Similarity between network traffic data These are preset hyperparameters. The number of network traffic data points in the network traffic dataset; obtaining the initial similarity matrix corresponding to the network traffic data set after optimizing and solving the similarity target function.

3. The network traffic anomaly detection method of claim 1, wherein, The determination method of the local density of each network traffic data comprises: in, For the first Local density of network traffic data For the first Network traffic data, Indicates the first Network traffic data and the Network traffic data The distance between them express for Nearest neighbor, The average distance, for k nearest neighbors The variance of the distance between them Indicates the first Network traffic data and the Network traffic data Similarity between them express with k nearest neighbors The sum of similarities; express with k nearest neighbors The sum of distances.

4. The network traffic anomaly detection method of claim 1, wherein, The determination of the relative distance corresponding to each network traffic data according to the local density of each network traffic data comprises: wherein, represents a relative distance of the th network traffic data, is a local density of the th network traffic data, is a local density of the th network traffic data, represents a distance between the th network traffic data and the th network traffic data , represents a local density of the th network traffic data is a maximum value among the local densities of all the network traffic data.

5. The network traffic anomaly detection method of claim 1, wherein, The determination of the edge weight of each two network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data comprises: in, For the first The network traffic data and the first Edge weights between network traffic data For the first Local density of network traffic data For the first Local density of network traffic data Indicates the first The network traffic data and the first Similarity between network traffic data.

6. A network traffic anomaly detection apparatus characterized by comprising: The method comprises the following steps: a data clustering module, wherein the data clustering module is used to perform the following steps: obtaining a network traffic data set to be detected, and constructing a k-neighbor graph corresponding to the network traffic data set by calculating the distance between each two network traffic data; calculating the similarity between each two network traffic data to obtain an initial similarity matrix corresponding to the network traffic data set; determining the local density of each network traffic data according to the k-neighbor graph and the initial similarity matrix, and determining the relative distance corresponding to each network traffic data according to the local density of each network traffic data; determining an edge weight between each pair of network traffic data on the k-neighbor graph based on the initial similarity matrix and the local density of each network traffic data, and determining a global density of each network traffic data based on the edge weight between each pair of network traffic data and a Markov chain idea, including: determining a transition probability matrix of asynchronous number of walks based on the edge weight between each pair of network traffic data and the Markov chain idea; obtaining a consistency transition probability matrix after weighted processing of the transition probability matrix of asynchronous number of walks; determining a global trajectory similarity between each pair of network traffic data according to the consistency transition probability matrix, and determining the global density of each network traffic data based on the global trajectory similarity; determining a γ value of each network traffic data based on the relative distance of each network traffic data and the global density of each network traffic data, and stretching the γ value of each network traffic data to obtain a γ transformed value of each network traffic data; The cluster center point set is determined based on the gamma transformation value of each network traffic data, including: sorting the gamma transformation value of each network traffic data to obtain a sorting serial number of each network traffic data; determining target network traffic data for truncation based on the gamma transformation value of each network traffic data; and taking each network traffic data with a sorting serial number before the target network traffic data as a cluster center point to obtain a cluster center point set ; wherein, represents the th cluster center point; And based on the cluster center point set, each non-cluster center point is allocated level by level to obtain multiple network traffic data clusters, including: First-level allocation: if , and ,but ;in, Represents network traffic data Cluster center The nearest neighbor, Represents cluster center points Network traffic data The nearest neighbor, Representing cluster centers The corresponding network traffic data clusters, This indicates that network traffic data Assigned to cluster Second-level allocation: Allocate the first network traffic data to the network traffic data cluster containing the most nearest neighbor points; wherein, the first network traffic data refers to the network traffic data that has not yet been allocated after the first-level allocation; Third-level allocation: Allocate the second network traffic data to the network traffic data cluster containing its relative point; wherein, the second network traffic data refers to the network traffic data that has not yet been allocated after the second-level allocation; the relative point of the second network traffic data refers to the network traffic data with a higher local density and the closest distance; wherein, the cluster center point set includes various cluster center points, the cluster center point refers to the network traffic data that serves as the cluster center, and the non-cluster center point refers to the network traffic data that does not serve as the cluster center; an anomaly detection module, the anomaly detection module being configured to: determine an abnormal network traffic data based on the plurality of network traffic data clusters.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images