Data anomaly detection and classification method based on matrix eigenvector trajectory

Abstract

This invention provides a data anomaly detection and classification method based on matrix eigenvector trajectories. The method primarily involves constructing an original dataset, performing mean-processing on the dataset to remove normalized data, initializing minimal-amplitude eigenvectors, and adaptively updating the eigenvectors using the mean-processed data. Data anomaly detection and classification are then performed based on the angular distribution of the eigenvectors in N-dimensional data space. This algorithm is applicable to data from various application fields and can adaptively perform automatic anomaly detection and classification.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, in particular to a method for automatic data anomaly detection and classification based on matrix eigenvector trajectories. Background Art

[0002] There are many algorithms in the field of artificial intelligence that can detect or classify anomalies in data. They can be divided into two types: machine learning and deep learning.

[0003] Most current machine learning and deep learning algorithms have limitations. For example, commonly used algorithms extract knowledge limited to specific historical datasets and can only acquire and interpret information based on prior patterns learned from the data. Not only do they require explicit and relatively time-consuming training using large amounts of data, but they also terminate training when the conditions of the objective function are met or when a predefined convergence is reached. This training mechanism does not allow for automatic updating of model parameters with new incoming data, resulting in a lack of adaptive capabilities to continuously learn from new patterns. Furthermore, complex models such as deep learning algorithms lack interpretability because their decisions cannot be explained. This can limit their effectiveness in many industries and fields. Finally, most of these algorithms can only perform detection or classification at a time. If both tasks are required, it is likely necessary to train a separate model. Summary of the Invention

[0004] The purpose of the present invention is to solve one of the above technical problems and provide a method for automatic anomaly detection and automatic classification of data that is simple in calculation and fast.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for automatic anomaly detection and classification of data based on matrix eigenvector trajectories, comprising the following steps:

[0007] S1: Construct the original data set: the data set is a data matrix with m rows and n columns; divide the data set into two categories by row, with the 1st to sth rows defined as the first category of data and the s+1th to mth rows defined as the second category of data; define x i is the i-th row of data in the original data set;

[0008] S2: Data mean removal: Calculate the mean of each column in the data set by column; subtract the mean of the corresponding column from each original data in the original data set to obtain the data after mean removal;

[0009] S3: Initialize the feature vector: define the kth n-dimensional feature vector e k ,k=1,2,…,n;initialize the kth eigenvector with the smallest amplitude as

[0010] S4: Update the kth eigenvector:

[0011]

[0012] φ(y i ,x′ i )=y i x′ i ;

[0013]

[0014] Where: y i is the projection formed by mapping the i-th sample to the direction of the current k-th eigenvector; φ(y i ,x′ i ) is the Hebbian term; is the kth eigenvector after the i-1th update, is the kth eigenvector after the i-th update; x′ i is the i-th row data after removing the mean value;

[0015] S5: Remove the influence of the kth eigenvector:

[0016]

[0017] Among them, e k is the final k-th feature vector after updating with all samples in step S4, for e k The transposed vector of

[0018] S6: Use x″ i Replace x′ i , repeat steps S4 and S5 to update the next feature vector;

[0019] S7: Data classification: Classify the data based on the distribution trajectory of the feature vector.

[0020] In some embodiments of the present invention, in the step of data de-meaning processing: the data mean of each column in the first type of data set is calculated column by column, and the data de-meaning processing is performed based on the obtained data mean.

[0021] In some embodiments of the present invention, the step of updating the feature vector further includes:

[0022] Set angle intervals in the N-dimensional data space coordinate system, and each angle interval corresponds to a data classification;

[0023] According to the position of the feature vector in the N-dimensional data space coordinate system, determine the angle interval to which it belongs;

[0024] Determine the classification of the data based on the angle interval to which the eigenvector belongs.

[0025] In some embodiments of the present invention, the following steps are further included:

[0026] Set the reference trajectory e0 of the data class feature vector;

[0027] Calculate the eigenvector trajectory e x The angle with the reference trajectory e0:

[0028]

[0029] If the angle is within the set threshold range, the data and the data corresponding to the reference trajectory e0 are considered to belong to the same type of data.

[0030] In some embodiments of the present invention, in step S6, p feature vectors are updated, where p≤n.

[0031] In some embodiments of the present invention, the dataset originates from industry data, and in the step of constructing the original dataset, the data in the dataset is arranged row by row in chronological order. Industry fields include, but are not limited to, cybersecurity data, mobile data, business data, social media data, and health data.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] A novel adaptive automatic detection and classification algorithm using matrix eigenvector trajectories is proposed. Under online data processing, the algorithm monitors the evolution of the eigenvector trajectories and uses their direction as a key indicator for automatic detection and classification. This algorithm does not require explicit training using large amounts of historical data. It is lightweight, fast, and easy to implement as a batch or adaptive technique. By monitoring the evolution of the eigenvector trajectories of the feature matrix, the algorithm can potentially achieve simultaneous automatic detection and classification when fed with both normal and abnormal data.

[0034] The algorithm can be applied to various types of data, such as network security data, mobile data, business data, social media data, health data, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1This is a classification diagram of the automatic data anomaly detection and automatic classification method provided by the present invention.

[0037] Figure 2 This is a flow chart of the method for automatic data anomaly detection and automatic classification provided by the present invention. DETAILED DESCRIPTION

[0038] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] The present invention provides a method for automatic anomaly detection and automatic classification of data based on matrix eigenvector trajectories, which can realize automatic anomaly detection and automatic classification of data.

[0040] This method is suitable for online data analysis scenarios. Online data analysis refers to the continuous, real-time input of data into the model. Mathematically, this means that each row of a matrix is ​​continuously fed into the model. This method differs from offline analysis in that the entire data matrix must be available before analysis can begin.

[0041] The steps of this method are detailed below. Figure 2 .

[0042] S1: Construct the original data set: the data set X is a data matrix with m rows and n columns; divide the data set into two categories by row, the 1st to sth rows are defined as the first category of data (corresponding to normal data in some embodiments), and the s+1th to mth rows are defined as the second category of data (corresponding to abnormal data in some embodiments); define x i is the i-th row of data in the original dataset.

[0043] The source field of the original data set is not restricted and can be application data from various industries. Based on the actual situation of the application scenario, the data of the original data set is sorted in order of formation time in units of rows. Taking the status monitoring data of a certain industrial equipment as an example, since the data arranged in the front is usually normal data, the probability of faulty data in the data arranged in the back may increase. Therefore, when classifying the data, the data arranged in the front is defined as the first category data in units of rows. In actual applications, it can be normal data, and the data arranged in the back is defined as the second category data. In actual applications, it can be abnormal data. The position of s is determined according to the actual application, and the specific value range is not limited.

[0044] The first type of data: x i =(d i1 ,d i2 ,…,d in), i=1,2,…,s;

[0045] The second type of data: x i =(d i1 ,d i2 ,…,d in ), i=s+1,s+2,…,m;

[0046] x i represents the i-th sample in the dataset X.

[0047] S2: Data mean removal: Calculate the data mean of each column in the data set column by column; subtract the data mean of the corresponding column from each original data in the original data set to obtain the data after mean removal.

[0048] Furthermore, in some embodiments of the present invention, in the step of data de-meaning processing: the data mean of each column in the first type of data set is calculated column by column, and the data de-meaning processing is performed based on the obtained data mean.

[0049] Since the system is initially in a normal state, the first type of data is normal data. The data monitored later is in an unknown state. It is also possible to calculate the average of several rows of known normal state data without having to find the specific value of s corresponding to the normal data.

[0050] The implementation process is described as follows:

[0051] Taking the first row to the sth row as the first type of data as an example, when calculating the data mean, the mean of each column of the data constructed from the first row to the sth row is calculated.

[0052] Take the data in the first column as an example:

[0053]

[0054] The calculation methods for other columns are the same and will not be repeated here.

[0055] After removing the mean value, the mean value of each column is subtracted from each data element in a row to obtain the sample after mean processing.

[0056]

[0057] S3: Initialize the feature vector: define the kth n-dimensional feature vector e k ,k=1,2,…,n;initialize the kth eigenvector with the smallest amplitude as

[0058] S4: Update the kth eigenvector:

[0059]

[0060] φ(y i ,x i ′)=y i x′ i ;

[0061]

[0062] Where: y i is the projection formed by mapping the i-th sample to the direction of the current k-th eigenvector; φ(y i ,x i ′) is the Hebbian term; is the kth eigenvector after the i-1th update, is the kth eigenvector after the i-th update; x′ i is the i-th row data after removing the mean.

[0063] The above updating process can be performed adaptively.

[0064] S5: Remove the influence of the kth eigenvector:

[0065]

[0066] Among them, e k is the final k-th feature vector after updating with all samples in step S4, for e k The transposed vector of

[0067] S6: Use x″ i Replace x′ i , repeat steps S4 and S5 to update the next feature vector.

[0068] Specifically, in step S4, after traversing all samples, the kth eigenvector is obtained. To find the k+1th eigenvector, the equation in step S5 must be applied to all data samples to eliminate the influence of the kth eigenvector to prevent it from being found again. Step S5 solves the problem of the influence of the eigenvector. Specifically, this step is a step to prepare for the calculation of the k+1th eigenvector, removing the kth eigenvector in the data x′. i If this step is not done, S4 will repeatedly calculate the k-th eigenvector.

[0069] In practical applications, only the first p eigenvectors can be calculated, where p <= n. Statistically, the degree of separability of data decreases with each eigenvector direction. The first eigenvector is most likely to separate the data, and subsequent eigenvectors become less so. Therefore, in general, automatic anomaly detection and classification can be achieved without calculating all eigenvectors.

[0070] S7: Data classification: Classify the data based on the distribution trajectory of the feature vector.

[0071] Since abnormal data may still have common features, in some embodiments of the present invention, in order to further classify the abnormal data according to the features of the abnormal data, the step of updating the feature vector further includes:

[0072] Set angle intervals in the N-dimensional data space coordinate system, and each angle interval corresponds to a data classification;

[0073] According to the position of the feature vector in the N-dimensional data space coordinate system, determine the angle interval to which it belongs;

[0074] Determine the classification of the data based on the angle interval to which the eigenvector belongs.

[0075] Specific reference Figure 1 For ease of understanding, the present embodiment below uses a two-dimensional coordinate system as an example to illustrate the principle of data classification based on feature vectors.

[0076] Figure 1 In the original dataset, after the mean is removed in step S2, the normal sample class of data samples will be located within the black circle, indicating that this data is normal data. In online data analysis mode where the presence of abnormal data is uncertain, if the feature vector update trajectory extends beyond the black circle, it can be considered that data different from normal data has been input into the model, thus enabling automatic anomaly detection.

[0077] Specifically, as data from the second category of abnormal data (which can be category 1, category 2, or category 3 in the figure) enters, the feature vector (kth dimension) is updated and moves from the coordinate system origin O to the cluster of second category data samples used to update itself. If the data samples in the second category are very clustered, the feature vector trajectory (a dotted line corresponding to each category of data) can be approximated as a line pointing to the center of the second category.

[0078] by Figure 1 Taking the second category in as an example, the feature vector is updated and approximately forms a linear feature vector trajectory e0 towards the second category data cluster. We use a fan-shaped area to represent the range of the feature vector trajectory, which is the feature vector trajectory e0 of any future data of this category. x A certain tolerance is given. The tolerance can be predefined and quantified by an angle θ, which can be calculated as follows.

[0079] Further comprising the steps of:

[0080] Set the reference trajectory e0 of the data class feature vector;

[0081] Calculate the eigenvector trajectory ex The angle with the reference trajectory e0:

[0082]

[0083] If the angle is within the set threshold range, the data and the data corresponding to the reference trajectory e0 are considered to belong to the same type of data.

[0084] If the angle between the feature vector trajectory of this type of data and the pre-defined trajectory is less than the tolerance (falling within the fan-shaped area), it can be judged as the same type of data; otherwise, it is different type of data. This adaptive update of the feature vector trajectory enables automatic anomaly detection and classification of data samples.

[0085] The mathematical reasoning process of this method is as follows. According to the mathematical steps of this algorithm, we can infer that:

[0086] (1) Eigenvector e i The trajectory evolution will move toward the second category (abnormal data category). If the second category of data is more aggregated, it will be more linear.

[0087] (2)||e i ||>||e i-1 ||, i = s + 1, s + 2, ..., m;

[0088] Before normalization, the magnitude of the eigenvectors is increasing.

[0089] The mathematical proof is as follows:

[0090] After the mean removal process, the data in the normal data class are already near the origin O in the N-dimensional coordinate system.

[0091] x′ i =(d i1 ,d i2 ,…,d in ), i=1,2,…,s;

[0092] So there exists a small positive value of ε such that:

[0093] |d ij |<ε, i=1,2,…,s, j=1,2,…,n;

[0094] Because there is:

[0095] e i =Σ i φ(y i ,x′ i )=y i x′ i +e i-1 =Σ i yi x′ i =y i x′ i +y i-1 x′ i-1 +…+y1x′1

[0096]

[0097] So, we can get:

[0098]

[0099] Considering that ε is a small positive value, it can be approximately considered as a zero vector.

[0100] After the original dataset is processed by removing the mean in step S2, the samples in the first category contribute little to the amplified feature vector. Next, the algorithm will use the samples in the second category to iteratively update the feature vector.

[0101] x′ i =(d i1 ,d i2 ,…,d in ),i=s+1,s+2,…,m;

[0102] Because y i is x i ′ in the eigenvector e i-1 The projection in the direction, and at this time e i-1 =e s

[0103] there will be:

[0104] e m =∑ i φ(y m ,x m ′)=y m x m ′+…y s+2 x s+2 ′+y s+1 x s+1 ′+e s ;

[0105] Due to e s Very small:

[0106] e m ≈∑ i y i x′ i ,i=s+1,s+2,…,m;

[0107] Depends on s and x s+1The initial angle between ′, e i and x i ' have the same or opposite directions, and e i The modulus before normalization is always increasing.

[0108] ||e i ||>||e i-1 ||,i=s+1,s+2,…,m;

[0109] More importantly, in this N-dimensional space, e i The trajectory evolution will move towards the second category. The more aggregated the second category data is, the more the evolution trajectory will be a straight line.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data anomaly detection and classification method based on matrix eigenvector trajectory, characterized in that: The following steps are involved: S1: Construct the original data set: the data set is OK The data matrix of the columns is divided into two categories by row, from row 1 to row The row is defined as the first type of data, Go to Row definition is the second type of data; definition The first Row data; S2: Data mean removal: Calculate the mean of each column in the data set by column; subtract the mean of the corresponding column from each original data in the original data set to obtain the data after mean removal; S3: Initialize the feature vector: define the indivual dimensional feature vector , ; Initialize the The eigenvector with the smallest magnitude is ; S4: Update Eigenvectors: ； ； ； in: For the The sample is mapped to the current The projection formed in the direction of the eigenvector; is the Hebbian term; For the first After the update feature vectors, For the first After the update feature vectors; is the first value after removing the mean value Row data; S5: Remove the The influence of eigenvectors: ； in, is the final updated value after all samples are updated in step S4. feature vectors, for The transposed vector of S6: Update Eigenvectors: use replace , repeat steps S4 and S5 to update the next feature vector; S7: Data classification: Classify the data based on the trajectory distribution of the feature vector.

2. The data anomaly detection and classification method based on matrix eigenvector trajectory according to claim 1, characterized in that: In the step of data mean removal: the data mean of each column in the first type of data set is calculated column by column, and the data mean removal is performed based on the obtained data mean.

3. The data anomaly detection and classification method based on matrix eigenvector trajectory according to claim 1, characterized in that: The step of updating the feature vector further includes: Set angle intervals in the N-dimensional data space coordinate system, and each angle interval corresponds to a data classification; According to the position of the feature vector in the N-dimensional data space coordinate system, determine the angle interval to which it belongs; Determine the classification of the data based on the angle interval to which the eigenvector belongs.

4. The data anomaly detection and classification method based on matrix eigenvector trajectory according to claim 1 or 3, characterized in that: Further comprising the steps of: Set the reference trajectory of the data class feature vector ; Calculate eigenvector trajectory With reference trajectory The angle between: ； If the angle is within the set threshold range, the data is considered to be consistent with the reference trajectory. The corresponding data classes belong to the same type of data.

5. The data anomaly detection and classification method based on matrix eigenvector trajectory according to claim 1, characterized in that: In step S6, update feature vectors, .

6. The data anomaly detection and classification method based on matrix eigenvector trajectory according to claim 1, characterized in that: In the steps of constructing the original data set: the data in the data set are arranged in rows in the order of formation time.

Images