A traffic anomaly detection method based on time series clustering

Abstract

The present invention relates to a flow anomaly detection method based on time series clustering, which belongs to the field of data mining. The method comprises the following steps: collecting system flow data, preprocessing the flow data, sampling flow sequence feature points, performing interval sampling in the area between the feature points, finding the maximum information entropy value and recording the sequence interval information; calculating the interval cumulative gradient in all sequence segments, extracting local feature information of the sequence segments, locating the direction interval and constructing a histogram of the corresponding feature sequence gradient; performing low-dimensional information processing on the sampled sequence segments; obtaining a feature vector description of the sequence segments; mapping the feature vector into a high-dimensional feature space to obtain the feature segment category information distribution under different categories; performing accurate feature frequency vector encoding on each sequence; clustering the sequence encoding vector representation to obtain cluster category information of each sequence; evaluating the original labeled information and the category information obtained by clustering and applying anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of data mining and relates to a flow anomaly detection method based on time series clustering. Background Art

[0002] Time series data is a ubiquitous form of data in everyday life, and is inextricably linked to numerous fields, such as finance, meteorology, and medicine. In recent years, with the continuous collection of data and information in the Internet of Things (IoT), an increasing amount of time series data has been generated and stored. Extracting valuable knowledge from this valuable data is a key research topic in data mining. Identifying data types within this vast amount of data is a highly sought-after issue. Extracting implicit features from data and identifying sequences based on these features are also worthy of exploration. Similarity calculation is an integral component of many data mining and machine learning algorithms, and the kernel function ultimately aims to calculate the similarity between any two data points. Kernel functions have been a hot research topic in recent years. In machine learning, kernel functions are effective similarity measures. Essentially, they map data into a new space and then calculate the dot product between the data points in this new space as the new similarity. This effectively addresses the problem of linear inseparability in low-dimensional data. This approach employs kernel techniques to transform low-dimensional data into a linearly separable high-dimensional feature space through a nonlinear transformation. Compared with other nonlinear algorithms, kernel functions not only have simple computational complexity, but also simple structure and good generalization ability, so kernel functions are a very practical machine learning method.

[0003] Existing methods for time series clustering research can be divided into three different categories: (1) Similarity-based clustering, which is to calculate the similarity of time series data during the clustering process, directly referencing the original data, and is therefore also called shape-based time series clustering methods. This type of method usually adjusts the distance metric of the traditional static clustering method to make it suitable for matching between time series data, but it is easy to produce extremely high time complexity and is only suitable for clustering small data sets; (2) Feature-based time series clustering, which converts the original data into low-dimensional feature vectors, and then uses traditional static clustering methods to cluster the feature vectors. This method focuses on the internal characteristics of the time series when processing time series, focusing on improving the interpretability of the features. While achieving data simplicity, it can also handle situations where there is missing data or unequal series length; (3) Model-based time series clustering, which considers the data set to be fitted by a certain model or probability distribution, and different time series are model expressions with different parameters. The original time series is converted into model parameters, and then the distance is calculated based on the parameters, and the appropriate traditional clustering method is selected for clustering.

[0004] The bag-of-words model is a research method based on feature clustering. It consists of three main steps: feature extraction, feature encoding, and feature clustering. This method first extracts features from the sequence, then uses clustering methods to encode the features based on the features. Finally, clustering is used to identify different categories of encoding. Current cluster generation methods mostly combine Gaussian mixture models, using a Gaussian mixture prior to guide the distribution of data in the feature space. However, due to the integration of Gaussian mixture models, these methods suffer from slow convergence and unstable training. Furthermore, because the collected time series features are unlabeled, the model cannot understand the conceptual information contained in the features. Although multiple Gaussian distributions can be learned, Gaussian mixture distributions are probabilistic descriptions, and it is impossible to determine which distribution corresponds to which category of feature. Furthermore, because the sampled features themselves are in an unknown distribution, labeling sequence features is impractical. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method for pre-processing and encoding the extracted features by combining a bag-of-words framework model with a kernel function, so as to achieve a process of generating new encoding features in a feature set without prior information and in the process of time series cluster recognition.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A traffic anomaly detection method based on time series clustering includes the following steps:

[0008] S1: Collecting traffic operation data from the IoT data collection terminal and performing data cleaning, including preprocessing the traffic operation data using a standardized method and performing feature sampling on the traffic operation data;

[0009] S2: Using the information entropy calculation method, perform interval search operations on all sequences in the feature data, calculate the maximum information entropy value and record the sequence interval information to ensure that the sequence identification information is sufficient;

[0010] S3: Using the feature gradient extraction algorithm, under the optimal interval information, the interval cumulative gradients in all sequence segments are calculated, from which the local feature information of the sequence segments is extracted, the direction interval is located, and the gradient is decomposed using trigonometric functions to construct a histogram of the corresponding feature sequence gradient;

[0011] S4: Use a dimensionality reduction algorithm with tensile and compressive stability to process the sampled sequence fragments into low-dimensional information, retain the distinguishability of information in low-dimensional information, filter out irrelevant dimensions with low feature distribution, and retain local information of the sequence;

[0012] S5: Based on the standardized fusion of the two features, the variance standard filtering method is used to prune the dimensions with lower information density, remove the meaningless computational consumption, and obtain the final feature vector description of the sequence segment;

[0013] S6: Use the kernel function to map the feature vector into a high-dimensional feature space, perform clustering and recognition based on similarity measurement under the high-dimensional representation of the features of all sequence segments within the specified category, and obtain the category information distribution of feature segments under different categories;

[0014] S7: Using the idea of the bag-of-words framework, for known feature segment distribution information, each sequence is encoded into a feature frequency vector according to the sampling order of the time series to ensure that each sequence can be converted into a feature coding vector;

[0015] S8: Use the spectral clustering algorithm to cluster the sequence's encoded vector representation. Graph theory's construction and cutting methods are used to ensure the effectiveness of the clustering results, so that similar data are close together in the projected feature space, while different data are separated in the projected feature space. Computational optimization is performed in the feature space to finally obtain cluster category information for each sequence.

[0016] S9: Use the adjusted mutual information method to evaluate the original labeled information and the category information obtained by clustering and apply it to the existing abnormal traffic data.

[0017] Furthermore, the feature sampling described in step S1 includes the following steps: distinguishing points with prominent features by setting parameters, extracting the point as a feature point if it is, and then performing uniform interval sampling in the remaining interval, and finally filling the sequence fragments around the point as local information; dividing the sampling results in the sequence into rich points and poor points, sampling them separately, and using the interval parameters in the feature point sampling interval to obtain the remaining sampling points therefrom in sequence.

[0018] The feature points are defined as follows:

[0019] Given a time series T(t1,t2,...,t n ), first the feature points need to satisfy (t i -t i-1 )*(t i+1 -t i )<0 inflection point rule, and then continue to judge in the filter set. The judgment process is as follows, and it is sufficient to meet one of the following conditions:

[0020] t i >min(t ＜i )+Δ and t i >t i+1

[0021] or ti >t i-1 And t i >min(t ＞i )+Δ

[0022] where t i Represents the sequence value of the i-th position, t <i Represents all sequence values before the i-th sequence point, and Δ represents the ratio specified by the peak-to-valley value, that is, the maximum and minimum difference of the sequence multiplied by a given value in the interval 0 and 1.

[0023] Furthermore, an information entropy algorithm is used to maximize the representation of feature information. Since there's no quantified horizontal span between time series samples, to better represent feature information such as gradients between sequences, the algorithm takes the entire dataset as input and, within a common parameter range, it iterates to find the maximum entropy value that ensures the greatest feature description for each sequence. The sequence interval parameters at which this maximum entropy value is achieved are also saved. Using a central derivative algorithm, the gradients in eight directional ranges are accumulated sequentially within a specific parameter range, and the distribution percentage of each directional range is calculated. Specifically, the intervals are divided into different intervals based on the range. The information entropy for each interval is then calculated and saved. The maximum entropy value among all entropy values in this iteration is then found. The maximum entropy parameter search is then repeated within the intervals immediately above the maximum entropy value until the desired number of iterations is met. This algorithm offers excellent stability, ensuring consistent results, and the corresponding parameter information for the sequence data can be obtained in a single calculation.

[0024] The maximum information entropy calculation formula is:

[0025]

[0026] Where X is the random variable of the source, p i Represents the probability information of the i-th variable.

[0027] Furthermore, in step S3, the HOG-1D algorithm is used for feature description. The central derivative between each sequence point is calculated as gradient information on the padded sequence segment centered at the sampling point. The gradient weights of each sequence are then accumulated in the horizontal interval to represent the local information of the current sequence. The HOG-1D algorithm is used for feature description because it excels at preserving local features in images. The HOG algorithm effectively represents local image information by constructing a histogram within the gradient interval, while the HOG-1D algorithm only retains gradient information in the horizontal direction. The resulting histogram features are more interpretable than the original sequence.

[0028] Furthermore, in step S4, the dimensionality reduction method mds-dtw is used to perform dimensionality reduction processing on the sequence fragments;

[0029] MDS is a global dimensionality reduction method that determines the representation of objects in a low-dimensional space given the similarity between objects and tries to roughly match the original similarity as much as possible; DTW is a dynamic programming metric for sequences;

[0030] MDS combines the DTW metric to calculate the distance matrix between sequences, and then iteratively calculates the distance between different features; the calculation formula of the DTW algorithm is as follows:

[0031] For two sequences Q(q1,q2,...,q n ), C(c1,c2,...,c n ), calculate the dtw distance between them as follows:

[0032] γ(i,j)=d(q i ,c j )+min{γ(i-1,j-1),γ(i-1,j),γ(i,j-1)}

[0033] Where γ(i,j) represents the distance from the origin to the end point (q i ,c j ) coordinates, where when i and j are n, this is the distance of the entire sequence obtained by the DTW algorithm; the MDS dimensionality reduction method solves the distribution of points in the Z-dimensional space, that is, the target dimension, by minimizing the value of the loss function. The loss function is as follows:

[0034]

[0035] Where N is the number of instances, z i represents the vector representation of the i-th instance under the target dimension, |||| represents the metric calculation, d ij Represents the distance between i and j in the original distance matrix.

[0036] Furthermore, in step S5, the standard deviation method is used to maintain the dimensions with obvious feature differences. The features are first standardized, and then the variance is calculated for all dimensional information. The variance calculation formula is as follows:

[0037]

[0038] Where n represents the number of sequences, x i represents the i-th sample instance, Represents the mean value of all samples.

[0039] Furthermore, in step S6, the feature vector is projected into a high-dimensional space using an isolated kernel function, and then cluster identification of the sequence in the mapping space is performed to obtain the category distribution of the feature vector under unknown prior distribution information; the calculation formula of the isolated kernel function is as follows:

[0040]

[0041] Where x and y are two sample points, K Ψ (x,y|D) represents the percentage of x and y falling into the same cell in t independent experiments, D represents the feature set, Ψ is the number of non-overlapping partitions, l is an indicator function, θ[z] is an isolated partition, and H is the set of all partitions;

[0042] Then use the k-means method to divide all samples into k clusters and iteratively calculate the minimum mean square error between all samples and the cluster center:

[0043]

[0044] Where E represents the minimum mean square error, k represents the number of clusters, and C i represents the i-th cluster, x represents the sample instance, μ i represents the cluster center sample of the i-th cluster, where μ i The calculation formula is as follows:

[0045]

[0046] where |C i | represents the number of instances in the i-th cluster.

[0047] Furthermore, in step S8, spectral clustering is used to cluster the code vectors. First, a similarity matrix is obtained, and then the matrix is normalized. Then, k eigenvectors are obtained, and finally, k-means is used for clustering. Specifically, the following steps are performed:

[0048] First, construct the similarity matrix S based on graph theory knowledge, then construct the adjacency matrix W and degree matrix D, and calculate the Laplace matrix L, calculate D -1 / 2 LD -1 / 2 The eigenvectors f corresponding to the minimized k1 eigenvalues are normalized row by row to form the matrix composed of the corresponding eigenvectors f, and finally form the feature matrix F. Then, each row in F is regarded as a k1-dimensional sample, and a total of n samples are clustered using the input clustering method. The clustering dimension is k2, and the final cluster is divided into C.

[0049] Furthermore, the formula for adjusting the mutual information in step S9 is calculated as follows:

[0050]

[0051] Where H(U) and H(V) represent the information entropy of the two distributions, MI is the original mutual information index, and E[MI] represents the expectation of the mutual information:

[0052]

[0053] where a i ,b j are the number of samples of the corresponding categories in the U and V distributions respectively.

[0054] First, a portion of data is collected from all traffic data as a training set, then cleaned, feature-constructed, and encoded. Finally, clustering is performed on the encoding, and the clustering model parameters with good evaluation results are retained as prior information for subsequent clustering. Then, training is performed on the entire training data set, and finally the results of the current data are returned to the system. The system will alarm the machines that need to be managed based on all the returned results, and the operation and maintenance personnel will analyze and process them based on the alarm results and the actual situation of the machines.

[0055] The present invention has the following advantages: In the context of IoT systems generating large amounts of traffic data without effective supervision, the present invention obtains data sequence point sampling information using a reasonable feature sampling method. The sequence points are then expanded and filled to obtain feature segments to describe local information. The local information of the features is then enhanced by using the HOG feature extraction algorithm and the MDS-DTW dimensionality reduction algorithm, which are adapted to sequence computations. The feature interpretability is then enhanced. Variance filtering is then used on the fused features to remove irrelevant information dimensions, thus preventing the impact on subsequent computational efficiency. Existing work rarely obtains the distribution information of feature data in an unsupervised manner. After mapping the feature data into a feature space using a kernel function, the feature data is clustered and identified using the kmeans method. The features are then vector-encoded using a bag-of-words model. Finally, spectral clustering is used to identify sequences and perform corresponding label evaluation. The trained parameters are then applied to real traffic data clustering to distinguish between normal and abnormal machines. Compared to traditional sequence clustering algorithms, this method adds a feature clustering and hard-coding process, resulting in a stronger interpretability of the local information of the sequence, making our model more adaptable to feature distributions without prior information and more robust. By applying kernel function processing to features and clustering them to obtain their distribution information, unsupervised encoding of clustered features is possible. This clustering approach is more accurate and converges faster than using a mixture of Gaussian priors. After processing the high-dimensional space of feature data and generating vector encodings, it is only necessary to select an efficient clustering algorithm. Clustering the generated encoding vector data yields excellent clustering results and good clustering metrics, which is unattainable with existing unsupervised generation methods.

[0056] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0058] Figure 1 This is an algorithm flow chart of the traffic anomaly detection method based on time series clustering described in the present invention.

[0059] Figure 2 Experimental process for engineering applications of experiments. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0062] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] The present invention is primarily used for clustering and identifying time series datasets of equal length. It involves extracting sequence point features using an effective sequence point sampling method, then filling the sampled sequence with information entropy to calculate the maximum entropy sequence spacing description. Local feature information is extracted using the one-dimensional HOG algorithm and the MDS dimensionality reduction method, and the features are complementary and fused. The processed features are filtered using a variance criterion to remove dimensions that express weak information, ensuring the interpretability of the dimensional information while reducing the computational complexity of subsequent experimental processes. The sampled features are then mapped into a high-dimensional feature space using an isolated kernel function within the sampled feature set, and the distribution of clustered features is clustered and identified within the feature space using the kmeans method. The features are then binned and encoded using the bag-of-words model to obtain a feature embedding representation of the time series data. Time series classification is achieved by performing spectral clustering on the feature embedding representation, enabling unsupervised identification of sequence categories in a weighted graph feature space. After obtaining the category information, an adjusted mutual information method is used to fairly evaluate the sequence labels.

[0064] Specifically, the present invention proposes a traffic anomaly detection method based on time series clustering, such as Figure 1 ,include:

[0065] S1: Collect the system's daily traffic operation data from the IoT data collection terminal, and then clean the collected traffic data, including preprocessing the sequence data using standardized methods and using appropriate sampling methods to perform feature sampling on the traffic data;

[0066] S2: Using the information entropy calculation method, perform interval search operations on all sequences in the feature data, calculate the maximum information entropy value and record the sequence interval information to ensure that the sequence identification information is sufficient;

[0067] S3: Using the feature gradient extraction algorithm, under the optimal interval information, the interval cumulative gradients in all sequence segments are calculated, from which the local feature information of the sequence segments is extracted, the direction interval is located, and the gradient is decomposed using trigonometric functions to construct a histogram of the corresponding feature sequence gradient;

[0068] S4: Use a dimensionality reduction algorithm with tensile and compressive stability to process the sampled sequence fragments into low-dimensional information, retain the distinguishability of information in low-dimensional information, filter out irrelevant dimensions with low feature distribution, and retain local information of the sequence;

[0069] S5: Based on the standardized fusion of the two features, the variance standard filtering method is used to prune the dimensions with lower information density, remove the meaningless computational consumption, and obtain the final feature vector description of the sequence segment;

[0070] S6: Use the kernel function to map the feature vector into a high-dimensional feature space, perform clustering and recognition based on similarity measurement under the high-dimensional representation of the features of all sequence segments within the specified category, and obtain the category information distribution of feature segments under different categories;

[0071] S7: Using the idea of the bag-of-words framework, for known feature segment distribution information, each sequence is encoded into a feature frequency vector according to the sampling order of the time series to ensure that each sequence can be converted into a feature coding vector;

[0072] S8: Use the spectral clustering algorithm to cluster the sequence's encoded vector representation. Graph theory's construction and cutting methods are used to ensure the effectiveness of the clustering results, so that similar data are close together in the projected feature space, while different data are separated in the projected feature space. Computational optimization is performed in the feature space to finally obtain cluster category information for each sequence.

[0073] S9: Use the adjusted mutual information method to evaluate the original labeled information and the category information obtained by clustering and apply it to the existing abnormal traffic data.

[0074] In this embodiment, the traffic data acquired from the IoT system is messy and noisy, requiring relevant processing. Abnormal and missing data are regulated, high levels of data collection noise are removed, and invalid data is eliminated. After cleaning the invalid data and filling the missing data, a hybrid sampling method is first used to sample feature points in the time series dataset. First, based on the parameter definitions, feature points that meet peak and valley values in the sequence are identified. Then, based on the sequence coordinates between each pair, uniform sampling intervals are used. The coordinate indexes of all points are then recorded as the basis for filling the feature sequence. Compared to other sampling methods, this method is more rational, avoiding the feature loss problems of random sampling and the uneven distribution of features in uniform sampling. It ensures the richness of feature sampling, allowing the sampling process to focus on the selection of prominent features while not neglecting regional features beyond the key features. The entire sequence is then filled with the sampling point as the center, and a sequence point filling method is set to ensure that the filled sequences near the sampling endpoints are not too abrupt. Using hybrid sampling of time series data demonstrates stable sampling and distinct features compared to random and uniform sampling methods. By dividing the sampling results in the sequence into rich points and poor points, sampling them separately to obtain better data features that adapt to the different distributions of data features and take into account the eigenvalues of important feature areas. The interval parameter is used in the feature point sampling interval to obtain the remaining sampling points in sequence. The definition of feature points is as follows:

[0075] Given a time series T(t1,t2,...,t n ), first the feature points need to satisfy (ti -t i-1 )*(t i+1 -t i )<0 inflection point rule, and then continue to judge in the filter set. The judgment process is as follows, and it is sufficient to meet one of the following conditions:

[0076] t i >min(t ＜i )+Δ and t i >t i+1 or t i >t i-1 And t i >min(t ＞i )+Δ

[0077] where t i Represents the sequence value of the i-th position, t <i Represents all sequence values before the i-th sequence point, and Δ represents the ratio specified by the peak-to-valley value, that is, the maximum and minimum difference of the sequence multiplied by a given value in the interval 0 and 1.

[0078] In this embodiment, the information entropy method is used to calculate the maximum entropy based on all sequence data to represent the interval between sequences. This method can effectively improve the local representation information of the feature. After continuous iterative calculation, the interval distance with the optimal information entropy in the default interval is found. The specific calculation method of the maximum information entropy is:

[0079]

[0080] Where X is the random variable of the source, p i Represents the probability information of the i-th variable.

[0081] In this embodiment, the HOG algorithm is used with interval information to further calculate the central derivative between sequence points, which is then used for gradient segmentation and accumulation. The histogram is divided into eight intervals, and the nine directional values are accumulated. For each point, the gradient of the upper and lower intervals is calculated, and its position in the specific interval range is calculated. The gradient is then decomposed and accumulated according to the gradient of adjacent vectors, and finally the histogram information vector of all feature sequences is obtained.

[0082] In this embodiment, the original time series is then subjected to the MDS dimensionality reduction method to perform a local feature description. Since MDS is a global dimensional feature representation, the DTW method is first used to calculate the distance matrix, followed by matrix inner product calculation and loss solution. Finally, the matrix distance between the sequences after dimensionality reduction should be as consistent as possible with the DTW distance matrix. A larger space needs to be constructed in DTW, and the least squares method is used to accelerate this calculation process. The calculation formula of the DTW algorithm is as follows:

[0083] For two sequences Q(q1,q2,...,q n ), C(c1,c2,...,c n ), then the dtw distance between them is calculated as follows: γ(i,j)=d(q i ,c j )+min{γ(i-1,j-1),γ(i-1,j),γ(i,j-1)}

[0084] Where γ(i,j) represents the distance from the origin to the end point (q i ,c j ) coordinates, where when i and j are n, this is the distance of the entire sequence obtained by the DTW algorithm. The MDS dimensionality reduction method solves the distribution of points in Z-dimensional space, i.e., the target dimension, by minimizing the value of the loss function. The loss function is as follows:

[0085]

[0086] Where N is the number of instances, z i represents the vector representation of the i-th instance under the target dimension, |||| represents the metric calculation, d ij Represents the distance between i and j in the original distance matrix.

[0087] In this embodiment, after obtaining two corresponding features, the features are first standardized and then dimensional fusion and splicing are performed to ensure that the information between the two dimensions can be more recognizable. We filter the standardized dimensions by calculating the variance of each dimension, and the dimensions with smaller changes are removed, which can effectively speed up the subsequent experimental steps. The variance calculation formula is as follows:

[0088]

[0089] Where n represents the number of sequences, x i represents the i-th sample instance, Represents the mean value of all samples.

[0090] In this embodiment, further, by utilizing the features after dimension pruning, we map the feature data to the corresponding feature space by sampling and dimensionality increase. First, we randomly select Ψ sample points from the training data as sub-sampling, and then divide the entire data space with these Ψ sample points to form a Voronoi diagram, that is, one sample point is the center of a cell, and the training is repeated t times to obtain t Voronoi diagrams. The two test data points are placed in each Voronoi diagram. If the two points fall in the same cell, their similarity is 1, otherwise the similarity is 0. Then the probability that they fall in the same cell is calculated, which is the final similarity. The denser the data distribution, the more it will be divided. If they are in these places, they are more likely to fall into different cell spaces and obtain smaller similarities. In the specific feature space, the similarity comparison of each dimension can perform very well, the same category is close to each other, and the information of different categories is far away from each other. It is guaranteed that in the absence of prior information, the category of the feature can be effectively identified based on the existing data. The calculation formula of the isolated kernel function is as follows:

[0091]

[0092] Where x and y are two sample points, K Ψ (x,y|D) represents the percentage of x and y falling into the same cell in t independent experiments, D represents the feature set, Ψ is the number of non-overlapping partitions, l is an indicator function, θ[z] is an isolated partition, and H is the set of all partitions. The goal of the kmeans method is to divide all samples into k clusters and iteratively calculate the minimum mean square error between all samples and the cluster center:

[0093]

[0094] Where E represents the minimum mean square error, k represents the number of clusters, and C i represents the i-th cluster, x represents the sample instance, μ i represents the cluster center sample of the i-th cluster, where μ i The calculation formula is as follows:

[0095]

[0096] where |C i | represents the number of instances in the i-th cluster, and the other parameters are consistent with the above formula.

[0097] In this embodiment, we perform type encoding on the features containing category information for each original sequence, construct a histogram of feature category sizes for each sequence, and then count the sampled features on each sequence in sequence to obtain the distribution status of the features in each sequence. Through these encoding vectors, we can intuitively observe the characteristic differences in the distribution of sequences in the same category and the distribution of sequences in different categories.

[0098] In this embodiment, the original feature sequence is accurately clustered using stable spectral clustering. Spectral clustering ensures the relationship between categories by constructing an undirected weighted graph and minimizing the loss function between clusters by cutting the graph. Compared with other clustering algorithms, it shows good performance. The implementation process of spectral clustering first constructs the similarity matrix S based on graph theory knowledge, then constructs the adjacency matrix W and the degree matrix D, and calculates the Laplace matrix L. Calculate D -1 / 2 LD -1 / 2 The eigenvectors f corresponding to the minimized k1 eigenvalues are normalized row by row to form the matrix composed of the corresponding eigenvectors f, and then each row in F is treated as a k1-dimensional sample, with a total of n samples. The input clustering method is used for clustering, and the clustering dimension is k2, and the final cluster is divided into C. The weight sum between the two feature sets is defined as follows:

[0099]

[0100] Where A and B are any two instance sample sets. The definition of the slice is as follows:

[0101]

[0102] in It's A i The complement of , k represents the number of segmentation categories.

[0103] In this embodiment, a popular method of adjusting mutual information evaluation is used to evaluate cluster labels and original labels. This method can penalize good clustering indicators caused by increasing the number of clusters, and can fairly evaluate the clustering results. First, assume that U and V are the distribution of N sample labels. The upper bounds of the two distributions are:

[0104]

[0105]

[0106] Where P(i) is the probability of a randomly selected object from U to a class, and P(j) is the probability of a randomly selected object from V to a class. The mutual information MI between U and V is defined as

[0107]

[0108] in

[0109]

[0110] Where P(i,j) is the sum of the probabilities that a randomly selected object falls into the two classes, and N is the total number of samples. The formula for adjusting mutual information is as follows:

[0111]

[0112] Where MI is the mutual information index, E[MI] represents the expectation of mutual information, and the expectation of E[MI] is calculated as follows:

[0113]

[0114] where a i ,b j are the number of samples of the corresponding categories in the U and V distributions respectively.

[0115] Due to a lack of understanding of the dataset, prior knowledge must first be acquired from the dataset. This means collecting some traffic data information, then efficiently encoding and ultimately clustering the collected data. For example, the clustering hyperparameter k must be identified based on the data size. k is an important initial parameter for clustering. Furthermore, the number of data iterations must be set in advance. Iteration is a crucial step in fitting the data distribution during clustering. This information is initially learned and useful information is retained as subsequent clustering information. Grouping is then performed based on the mined information, with machines divided into different categories according to different traffic sequences. Groups with abnormal changes are given additional alerts, and finally, operations and maintenance personnel analyze and address specific issues.

[0116] Currently available time series clustering methods rarely extract conceptual information from feature data without supervision, nor can they accurately map data dimensions, and most clustering generation models have problems of instability and slow convergence. The present invention extracts sequence features using a mixed sampling method, which enhances the stability of the model. It ensures that the feature information between sequences is sufficient through the information entropy method. It uses the hog and mds methods to extract local information of the sequence. It uses the feature fusion method to make the features more interpretable and more stable. It also uses the kernel function to project the high-dimensional feature space to ensure that the features can be effectively distinguished and successfully extract the category information contained in the data without supervision. It uses a reliable spectral clustering method to solve the effectiveness of the clustering experiment. Compared with ordinary clustering methods, our framework has an additional sampling and feature extraction process, so the feature space becomes larger, which makes our method more accurate and robust. By obtaining the distribution of feature data and completing sequence encoding, data in the feature space can be clustered unsupervised. This clustering method is more stable and converges faster than using a mixed Gaussian prior.

[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A traffic anomaly detection method based on time series clustering, characterized by: The following steps are involved: S1: Collecting traffic operation data from the IoT data collection terminal and performing data cleaning, including preprocessing the traffic operation data using a standardized method and performing feature sampling on the traffic operation data; S2: Using the information entropy calculation method, perform interval search operations on all sequences in the feature data, calculate the maximum information entropy value and record the sequence interval information; S3: Using the feature gradient extraction algorithm, under the optimal interval information, the interval cumulative gradients in all sequence segments are calculated, from which the local feature information of the sequence segments is extracted, the direction interval is located, and the gradient is decomposed using trigonometric functions to construct a histogram of the corresponding feature sequence gradient; S4: Use a dimensionality reduction algorithm with tensile and compressive stability to process the sampled sequence fragments into low-dimensional information, retain the distinguishability of information in low-dimensional information, filter out irrelevant dimensions with low feature distribution, and retain local information of the sequence; S5: Based on the standardized fusion of the two features, the variance standard filtering method is used to prune the dimensions with lower information density, remove the meaningless computational consumption, and obtain the final feature vector description of the sequence segment; S6: Use the kernel function to map the feature vector into a high-dimensional feature space, perform clustering and recognition based on similarity measurement under the high-dimensional representation of the features of all sequence segments within the specified category, and obtain the category information distribution of feature segments under different categories; S7: Using the idea of the bag-of-words framework, for known feature segment distribution information, each sequence is encoded into a feature frequency vector according to the sampling order of the time series to ensure that each sequence can be converted into a feature coding vector; S8: Use the spectral clustering algorithm to cluster the sequence's encoded vector representation. Graph theory's construction and cutting methods are used to ensure the effectiveness of the clustering results, so that similar data are close together in the projected feature space, while different data are separated in the projected feature space. Computational optimization is performed in the feature space to finally obtain cluster category information for each sequence. S9: Use the adjusted mutual information method to evaluate the original labeled information and the category information obtained by clustering, and apply it to the existing abnormal traffic data.

2. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: The feature points in step S1 are defined as follows: Given a time series T(t1,t2,...,t n ), first the feature points need to satisfy (t i -t i-1 )*(t i+1 -t i )<0 inflection point rule, and then continue to judge in the filter set. The judgment process is as follows, and it is sufficient to meet one of the following conditions: t i > min(t ＜i ) + Δ and t i > t i+1 or t i >t i-1 And t i >min(t ＞i )+Δ where t i Represents the sequence value of the i-th position, t <i Represents all sequence values before the i-th sequence point, and Δ represents the ratio specified by the peak-to-valley value, that is, the maximum and minimum difference of the sequence multiplied by a given value in the interval 0 and 1.

3. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: The maximum information entropy calculation formula is: Where X is the random variable of the source, p i Represents the probability information of the i-th variable.

4. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: In step S3, the Hog-1D algorithm is used for feature description. The central derivative between each sequence point is calculated as the gradient information on the padded sequence segment centered on the sampling point. Then, the gradient weight of each sequence is accumulated in the horizontal direction interval to represent the local information of the current sequence.

5. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: In step S4, the dimensionality reduction method mds-dtw is used to reduce the dimensionality of the sequence fragments; MDS is a global dimensionality reduction method that determines the representation of objects in a low-dimensional space given the similarity between objects and tries to roughly match the original similarity as much as possible; DTW is a dynamic programming metric for sequences; MDS combines the DTW metric to calculate the distance matrix between sequences, and then iteratively calculates the distance between different features; the calculation formula of the DTW algorithm is as follows: For two sequences Q(q1,q2,...,q n ), C(c1,c2,...,c n ), calculate the dtw distance between them as follows: γ(i,j)=d(q i ,c j )+min{γ(i-1,j-1),γ(i-1,j),γ(i,j-1)} Where γ(i,j) represents the distance from the origin to the end point (q i ,c j ) coordinates, where when i and j are n, this is the distance of the entire sequence obtained by the DTW algorithm; the MDS dimensionality reduction method solves the distribution of points in the Z-dimensional space, that is, the target dimension, by minimizing the value of the loss function. The loss function is as follows: Where N is the number of instances, z i represents the vector representation of the i-th instance under the target dimension, || || represents the metric calculation, d ij Represents the distance between i and j in the original distance matrix.

6. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: In step S5, the standard deviation method is used to maintain dimensions with obvious feature differences. The features are first standardized, and then the variance is calculated for all dimensional information. The variance calculation formula is as follows: Where n represents the number of sequences, x i represents the i-th sample instance, Represents the mean value of all samples.

7. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: In step S6, the isolated kernel function is used to project the feature vector into a high-dimensional space, and then cluster identification of the sequence in the mapping space is performed to obtain the category distribution of the feature vector under unknown prior distribution information; the calculation formula of the isolated kernel function is as follows: Where x and y are two sample points, K Ψ (x,y|D) represents the percentage of x and y falling into the same cell in t independent experiments, D represents the feature set, Ψ is the number of non-overlapping partitions, l is an indicator function, θ[z] is an isolated partition, and H is the set of all partitions; Then use the k-means method to divide all samples into k clusters and iteratively calculate the minimum mean square error between all samples and the cluster center: Where E represents the minimum mean square error, k represents the number of clusters, and C i represents the i-th cluster, x represents the sample instance, μ i represents the cluster center sample of the i-th cluster, where μ i The calculation formula is as follows: where |C i | represents the number of instances in the i-th cluster.

8. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: In step S8, spectral clustering is used to cluster the code vectors. First, the similarity matrix is obtained, then the matrix is normalized, and then k eigenvectors are obtained. Finally, k-means is used for clustering. Specifically, the following steps are performed: First, construct the similarity matrix S based on graph theory knowledge, then construct the adjacency matrix W and degree matrix D, and calculate the Laplace matrix L, calculate D -1 / 2 LD -1 / 2 The eigenvectors f corresponding to the minimized k1 eigenvalues are normalized row by row to form the matrix composed of the corresponding eigenvectors f, and finally form the feature matrix F. Then, each row in F is regarded as a k1-dimensional sample, and a total of n samples are clustered using the input clustering method. The clustering dimension is k2, and the final cluster is divided into C.

9. The traffic anomaly detection method based on time series clustering according to claim 1 is characterized in that: The formula for adjusting the mutual information in step S9 is calculated as follows: Where H(U) and H(V) represent the information entropy of the two distributions, MI is the original mutual information index, and E[MI] represents the expectation of the mutual information: where a i ,b j are the number of samples of the corresponding categories in the U and V distributions respectively; In the final model evaluation, the model is evaluated using real traffic data, the prediction error is calculated, and the clustering data is returned to the system, and the staff is notified to handle the traffic of abnormal machines.

Images