Massive power data anomaly detection method and detection system

Abstract

The application discloses a large-scale power data anomaly detection method, comprising the following steps: obtaining power data to be detected and processing the power data to obtain a standard data set; calculating the similarity between each data; obtaining the similarity data label of each data; obtaining the historical detection rule of power data anomaly detection and processing the historical detection rule to obtain a corresponding check table and check field; generating the detection rule of the power data to be detected according to the similarity data label of each data and the check table and check field; and performing actual anomaly detection by using the detection rule of the power data to be detected. The application further discloses a detection system for realizing the large-scale power data anomaly detection method. The application realizes the fusion of machine discovery and expert experience, so that the detection rule generated by the machine can be fed back and optimized by the expert experience, and the detection rule is generated by introducing a word vector segmentation algorithm and a natural language processing model, so that the discovery efficiency is improved. The application has higher reliability, better accuracy and higher efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of power safety, and particularly relates to a large-scale power data anomaly detection method and a detection system. BACKGROUND

[0002] With the development of economy and technology and the improvement of people's living standards, electric energy has become an indispensable secondary energy in people's production and life, bringing endless convenience to people's production and life. Therefore, guaranteeing the safe and stable supply of electric energy has become one of the most important tasks of the power system.

[0003] With the development of the power system, power data has become an important support for the power system to guarantee power supply. However, due to the typical big data characteristics of the power data of power enterprises, the power data covers all aspects of power generation, power transmission, power transformation, power distribution and dispatching in power production and power use, including power grid operation, equipment management, marketing service and enterprise management and other types of data, which contains extremely rich information. The richness of power data will directly lead to the increasing complexity, relevance and applicability of data; and in this process, a large amount of data quality problems will naturally occur, which increases the difficulty of power data anomaly detection.

[0004] The current power data anomaly detection adopts a verification method based on business rules. From the perspective of power business, data verification rules and conditions are proposed, and data auditors convert the business rule description into executable code. However, this method mainly has two problems, one is that the business knowledge threshold is high, resulting in low efficiency of auditing rule development. Based on the business description of business rules, a large amount of power business knowledge is required for data tracing and code development, and the power professional ability of data auditors is high, and the rule development is difficult. The second is the lack of the ability to actively find data problems. SUMMARY

[0005] One of the purposes of the present application is to provide a large-scale power data anomaly detection method with high reliability, good accuracy and high efficiency.

[0006] The second purpose of the present application is to provide a detection system for realizing the large-scale power data anomaly detection method.

[0007] The large-scale power data anomaly detection method provided by the present application comprises the following steps:

[0008] S1. Obtain the power data to be detected, and process to obtain a standard data set;

[0009] S2. Calculate the similarity between each data in the standard data set obtained in step S1;

[0010] S3. Obtain the similarity data label of each data according to the similarity calculation result obtained in step S2;

[0011] S4. Obtain the historical detection rule of power data anomaly detection, and process to obtain the corresponding verification table and verification field;

[0012] S5. Generate the detection rule of the to-be-detected power data according to the similarity data label of each data obtained in step S3 and the verification table and verification field obtained in step S4;

[0013] S6. Perform actual anomaly detection of the to-be-detected power data by using the detection rule of the to-be-detected power data obtained in step S6.

[0014] The step S1 of obtaining the to-be-detected power data and processing to obtain the standard data set specifically includes the following steps:

[0015] Obtain the to-be-detected power data, and input the data and the corresponding project data field to the specified path according to the specified data model; import the data to the specified data path according to the physical model; and input the corresponding relationship between the tables and the fields in the data and the detailed path of the tables according to the set data model.

[0016] The step S2 of calculating the similarity between each data in the standard data set obtained in step S1 specifically includes the following steps:

[0017] Obtain the to-be-detected data by using the data model;

[0018] Arbitrarily select two groups of to-be-detected data, and judge the data types between the data:

[0019] If the data types are the same, the subsequent similarity calculation is performed;

[0020] If the data types are different, it is directly determined that the data are not similar;

[0021] For the to-be-detected data with the same data type, further judgment is performed:

[0022] If the data type is numerical, the similarity between the data is calculated by using the following formula:

[0023]

[0024] In the formula, d xy is the similarity between the data x and the data y, the value of d xy is between 0 and 1, and the greater the value of d xy , the lower the similarity between the data x and the data y; x k is the n-dimensional numerical vector corresponding to the data x; and y kA n-dimensional numerical vector corresponding to the data y;

[0025] If the data type is literal, the data similarity is calculated by the following steps:

[0026] Tokenize the two data texts a and b to obtain the corresponding token sets {a i} and {b j};

[0027] Obtain the corresponding word vectors v{a i} and v{b j} according to the word vector matrix;

[0028] Obtain the word frequency data p{a i} and p{b j} by counting the word frequency of each word;

[0029] Perform weighted average on the word frequency data p{a i} and p{b j} to obtain the sentence vector v(a) of text a as The sentence vector v(b) of text b is

[0030] Calculate the cosine value of the sentence vector v(a) and the sentence vector v(b), so as to obtain the similarity d ab between data a and data b.

[0031] Finally, obtain the data vector with a similarity higher than a set value; after further confirmation, label the corresponding data with a similarity label.

[0032] According to the similarity calculation result obtained in step S2, obtain the similarity data label of each data, which specifically includes the following steps:

[0033] According to the similarity calculation result obtained in step S2, label the corresponding data with a similarity label;

[0034] After artificial confirmation of the generated data label, the confirmed similarity label is archived.

[0035] The history detection rule of power data anomaly detection is obtained in step S4, and the corresponding verification table and verification field are processed, which specifically includes the following steps:

[0036] Obtain the training data set;

[0037] Construct a BiLSTM-CRF model as a preliminary entity recognition model:

[0038] The entity recognition preliminary model comprises a bidirectional long short-term memory network BiLSTM and a conditional random field model CRF;

[0039] Each unit in the bidirectional long short-term memory network BiLSTM comprises an input gate, a forgetting gate, an output gate, a long memory structure and a short memory structure; for the input character x t , the hidden layer state h t-1 and the unit state C t-1 at the t moment, the calculation mode of each unit is as follows:

[0040] The input gate calculation mode is i t =σ(W i ·[h t-1 ,x t ]+b i ), wherein i t is the output of the input gate, σ is a sigmoid activation function, W i is a weight matrix of the input gate, [h t-1 ,x t ] is a matrix obtained by splicing h t-1 and x t , and b i is a bias vector of the input gate;

[0041] The forgetting gate calculation mode is f t =σ(W f ·[h t-1 ,x t ]+b f ), wherein f t is the output of the forgetting gate, W f is a weight matrix of the forgetting gate, and b f is a bias vector of the forgetting gate;

[0042] The output gate calculation mode is o t =σ(W o ·[h t-1 ,x t ]+b o ), wherein o t is the output of the output gate, W o is a weight matrix of the output gate, and b o is a bias vector of the output gate;

[0043] The long memory structure calculation mode is wherein C t is the unit state at the t moment, is the unit state of the new input information at the t moment, and W C is a weight matrix, and b C is a bias vector;

[0044] The calculation method of the short memory machine structure is h t = o t · tanh(C t ), wherein h t is the hidden layer state at the t time.

[0045] The conditional random field model CRF is used to fully extract the global information of the input text; the parameterized representation of the observation sequence X in the conditional random field model CRF is

[0046]

[0047] In the formula, P(y|x) is a conditional probability of a state sequence y under the condition of an observation sequence x, y is a state sequence, x is an observation sequence, Z(x) is a normalization item, and

[0048] Z(x) = ∑ y exp(∑ i,k t k (y i-1 ,y i ,x,i) + ∑ i,l u l s l (y i ,x,i)), t k is a transition feature function, y i is a state at the i time, i and l are subscript variables of all times, u l is a state feature coefficient at the l time, and s l is a state feature function.

[0049] The obtained training data set is used to train the constructed preliminary entity recognition model until the model converges, so as to obtain an entity recognition model.

[0050] The obtained historical detection rule of power data anomaly detection is input into the entity recognition model, and a check table and a check field are predicted.

[0051] According to the similarity data labels of each data obtained in step S3 and the check table and the check field obtained in step S4, a detection rule of the to-be-detected power data is generated, and the specific steps include the following steps:

[0052] The check table and the corresponding check field in the historical detection rule are replaced by a data table corresponding to the similar label and a data field corresponding to the table, so as to generate a new detection rule of the to-be-detected power data.

[0053] The detection rule of the to-be-detected power data obtained in step S6 is used to perform actual anomaly detection on the to-be-detected power data, and the specific steps include the following steps:

[0054] Deploy a new detection rule for the to-be-detected power data, and run to generate example data;

[0055] Manually judge the example data:

[0056] If the judgment is passed, the new detection rule for the to-be-detected power data is used for actual detection;

[0057] If the judgment is not passed, manual intervention is performed.

[0058] The application also provides a detection system for implementing the large-scale power data anomaly detection method, comprising a standard data set acquisition module, a similarity calculation module, a similarity data label calculation module, a check table and check field calculation module, a detection rule updating module and a detection module; the standard data set acquisition module, the similarity calculation module, the similarity data label calculation module, the check table and check field calculation module, the detection rule updating module and the detection module are sequentially connected; the standard data set acquisition module is used to acquire to-be-detected power data, process to obtain a standard data set, and upload the result to the similarity calculation module; the similarity calculation module is used to calculate the similarity between each data in the received standard data set, and upload the result to the similarity data label calculation module; the similarity data label calculation module is used to obtain the similarity data label of each data according to the received similarity calculation result, and upload the result to the check table and check field calculation module; the check table and check field calculation module is used to acquire a historical detection rule of power data anomaly detection, process to obtain a corresponding check table and check field, and upload the result to the detection rule updating module; the detection rule updating module is used to generate a detection rule for to-be-detected power data according to the similarity data label and the check table and check field, and upload the result to the detection module; the detection module is used to adopt the received detection rule for to-be-detected power data to perform actual anomaly detection of to-be-detected power data.

[0059] The large-scale power data anomaly detection method and the detection system provided by the application first generate data similarity labels by using a deep learning algorithm to calculate the similarity, and then confirm the labels by experts, so as to realize the fusion of machine discovery and expert experience; then, after generating new detection rules and example data based on natural language processing, the rules are judged according to expert experience, so that the detection rules generated by the machine are optimized, and the feedback of expert experience is also recorded in the detection rules; finally, the application intelligently generates detection rules by introducing a word vector segmentation algorithm and a natural language processing BiLSTM-CRF model, and improves the discovery efficiency of abnormal data; therefore, the application has higher reliability, better accuracy and higher efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The method flowchart of the method of the present application is shown.

[0061] Figure 2 The functional module diagram of the system of the present application is shown. DETAILED DESCRIPTION

[0062] As Figure 1 The method flowchart of the method of the present application is shown: the large-scale power data anomaly detection method provided by the present application includes the following steps:

[0063] S1. Obtain the power data to be detected, and process to obtain a standard data set; specifically including the following steps:

[0064] Obtain the power data to be detected (including PMS system data and CMS system data, data middle station source layer data, data middle station model layer data, and data market, etc.), input the data and corresponding project data fields to the specified path according to the specified data model; import the data to the specified data path according to the physical model; at the same time, input the corresponding relationship between the data table and the field and the detailed path of the table according to the set data model;

[0065] S2. Calculate the similarity between each data in the standard data set obtained in step S1; specifically including the following steps:

[0066] Obtain the data to be detected through the data model;

[0067] Arbitrarily select two groups of data to be detected, and judge the data type between the data:

[0068] If the data types are the same, the similarity calculation is carried out subsequently;

[0069] If the data types are different, it is directly determined as non-similar data;

[0070] For the data to be detected with the same data type, further judgment is carried out:

[0071] If the data type is numerical, the data similarity is calculated by the following formula:

[0072]

[0073] In the formula, d xy is the similarity between data x and data y, the value of d xy is between 0 and 1, and the greater the value of d xy , the lower the similarity between data x and data y; x k is the n-dimensional numerical vector corresponding to data x; y k is the n-dimensional numerical vector corresponding to data y;

[0074] If the data type is text, the following steps are used to calculate data similarity:

[0075] In practice, word vectors trained based on the word2vec model are used to calculate similarity.

[0076] The Word2Vec training model is essentially a neural network with only one hidden layer. It has two forms: CBOW and Skip-gram, which transform words into semantically meaningful numerical vectors, i.e., word vectors. This invention adopts the Skip-gram model, which takes the center word w(t) as input and the surrounding words w(t-1) and w(t+1) as output, i.e., predicting the surrounding words using the center word. Assuming there are V words in total, after convergence through the neural network training method, the large matrix W formed by the weight vectors from all input words to the hidden layer... V×N This is the word vector matrix, where word x i The word vectors are the matrix W. V×N The transpose of the i-th row; since semantically similar words usually have similar contexts, the word vectors trained by the word2vec model have semantic similarity; the specific steps include:

[0077] Segment the two data texts a and b into words, and obtain the corresponding word sets {a}. i} and {b j};

[0078] Obtain the corresponding word vector v{a from the word vector matrix. i} and v{b j};

[0079] By analyzing the word frequency of each word in the international standard, we obtain word frequency data p{a i} and p{b j};

[0080] For word frequency data p{a i} and p{b j The sentence vector v(a) of text a is obtained by performing weighted averages on each part. The sentence vector v(b) of text b is

[0081] Calculate the cosine values ​​of sentence vectors v(a) and v(b) to obtain the similarity d between data a and data b. ab for

[0082] Finally, obtain data vectors with similarity higher than a set value; after further confirmation (preferably by experts), label the corresponding data with similarity tags;

[0083] S3. Obtain the similarity data label of each data according to the similarity calculation result obtained in step S2; specifically including the following steps:

[0084] According to the similarity calculation result obtained in step S2, the corresponding data is labeled with a similarity label;

[0085] After the generated data label is confirmed by human (preferably by experts), the confirmed similarity label is archived;

[0086] S4. Obtain the historical detection rules of power data anomaly detection, and process to obtain the corresponding check table and check field; specifically including the following steps:

[0087] Obtain the training data set;

[0088] Build a BiLSTM-CRF model (which fully considers the correlation between feature words and the global information of the sentence) as a preliminary entity recognition model:

[0089] The preliminary entity recognition model includes a bidirectional long short-term memory network BiLSTM and a conditional random field model CRF;

[0090] Each unit in the bidirectional long short-term memory network BiLSTM includes an input gate, a forget gate, an output gate, a long memory structure, and a short memory structure; for the input character x t , the hidden layer state h t-1 and the unit state C t-1 at time t-1, the calculation method of each unit is:

[0091] The input gate calculation method is i t =σ(W i ·[h t-1 ,x t ]+b i ), wherein i t is the output of the input gate, σ is the sigmoid activation function, W i is the weight matrix of the input gate, [h t-1 ,x t ] is a matrix obtained by concatenating h t-1 and x t , and b i is the bias vector of the input gate;

[0092] The forget gate calculation method is f t =σ(W f ·[h t-1 ,x t ]+b f ), wherein f t is the output of the forget gate, Wf is a weight matrix of the forget gate, b f is a bias vector of the forget gate;

[0093] The output gate is calculated as o t = σ(W o · [h t-1 , x t ]+b o ), where o t is the output of the output gate, W o is a weight matrix of the output gate, b o is a bias vector of the output gate;

[0094] The long memory structure is calculated as where C t is the cell state at time t, is the cell state of the new input information at time t, and W C is a weight matrix, and b C is a bias vector;

[0095] The short memory machine structure is calculated as h t = o t · tanh(C t ), where h t is the hidden layer state at time t;

[0096] The conditional random field model CRF is used to fully extract the global information of the input text; the parameterized representation of the observation sequence X in the conditional random field model CRF is

[0097]

[0098] In the formula, P(y|x) is the conditional probability of the state sequence y under the condition of the observation sequence x, y is the state sequence, x is the observation sequence, Z(x) is the normalization item, and

[0099] Z(x) = ∑ y exp(∑ i,k t k (y i-1 , y i , x, i) + ∑ i,l u l s l (y i , x, i)), t k is a transition feature function, y i is the state at time i, i and l are subscript variables of all times, u l is a state feature coefficient at time l, and s l is a state feature function;

[0100] Train the constructed entity recognition preliminary model with the obtained training data set until the model converges to obtain an entity recognition model;

[0101] Input the obtained historical detection rules of power data anomaly detection into the entity recognition model to predict the check table and the check field;

[0102] S5. Generate the detection rule of the to-be-detected power data according to the similarity data label of each data obtained in step S3 and the check table and the check field obtained in step S4; specifically including the following steps:

[0103] Replace the check table and the corresponding check field in the historical detection rule with the data table corresponding to the similar label and the data field corresponding to the table to generate a new detection rule of the to-be-detected power data;

[0104] S6. Perform actual anomaly detection of the to-be-detected power data by using the detection rule of the to-be-detected power data obtained in step S6; specifically including the following steps:

[0105] Deploy the new detection rule of the to-be-detected power data and run the generated sample data;

[0106] Manually judge the sample data;

[0107] If the judgment is passed, perform actual detection by using the new detection rule of the to-be-detected power data; when performing actual detection, configure the running time and frequency to perform detection;

[0108] If the judgment is not passed, perform manual intervention.

[0109] For example, Figure 2The system of the application is shown in the functional module diagram: the detection system for realizing the large-scale power data anomaly detection method provided by the application comprises a standard data set acquisition module, a similarity calculation module, a similarity data label calculation module, a check table and check field calculation module, a detection rule updating module and a detection module; the standard data set acquisition module, the similarity calculation module, the similarity data label calculation module, the check table and check field calculation module, the detection rule updating module and the detection module are sequentially connected; the standard data set acquisition module is used for acquiring the power data to be detected, processing to obtain a standard data set, and uploading the result to the similarity calculation module; the similarity calculation module is used for calculating the similarity between each data in the received standard data set, and uploading the result to the similarity data label calculation module; the similarity data label calculation module is used for obtaining the similarity data label of each data according to the received similarity calculation result, and uploading the result to the check table and check field calculation module; the check table and check field calculation module is used for acquiring the historical detection rule of the power data anomaly detection, processing to obtain the corresponding check table and check field, and uploading the result to the detection rule updating module; the detection rule updating module is used for generating the detection rule of the power data to be detected according to the similarity data label and the check table and check field, and uploading the result to the detection module; the detection module is used for adopting the received detection rule of the power data to be detected to perform actual anomaly detection on the power data to be detected.

Claims

1. A large-scale power data anomaly detection method, comprising the following steps: S1. Obtain the power data to be detected, and process to obtain a standard data set; specifically comprising the following steps: Obtain the power data to be detected, and input the data and the corresponding item data field to the specified path according to the specified data model; import the data into the specified data path according to the physical model; at the same time, input the corresponding relationship between the tables and fields in the data and the detailed path of the tables according to the set data model; S2. Calculate the similarity between each data in the standard data set obtained in step S1; specifically comprising the following steps: Obtain the data to be detected through the data model; Arbitrarily select two groups of data to be detected, and judge the data type between the data: If the data types are the same, the subsequent similarity calculation is performed; If the data types are different, it is directly determined as non-similar data; For the data to be detected with the same data type, further judgment is performed: If the data type is numerical, the data similarity is calculated using the following formula: In the formula is the similarity between data x and data y, The value of is between 0 and 1, and The greater the value of is, the lower the similarity between data x and data y is. is an n-dimensional numerical vector corresponding to data x; is an n-dimensional numerical vector corresponding to data y; If the data type is literal, the data similarity is calculated using the following steps: Tokenize two data texts a and b to obtain corresponding token sets and ; According to the word vector matrix, a corresponding word vector is obtained and ; Obtain the word frequency data of each word in the international corpus and ; The word frequency data And The sentence vector of text a is The sentence vector of text b is ; Computing sentence vectors and cosine values of the sentence vectors , so that the similarity of data a and data b is is ; Finally, obtain the data vector with a similarity higher than the set value; after further confirmation, the corresponding data is labeled with a similarity label; S3. Obtain the similarity data label of each data according to the similarity calculation result obtained in step S2; specifically comprising the following steps: According to the similarity calculation result obtained in step S2, the corresponding data is labeled with a similarity label; After the generated data label is confirmed by artificial confirmation, the confirmed similarity label is archived; S4. Obtain the historical detection rule of power data anomaly detection, and process to obtain the corresponding check table and check field; S5. Generate the detection rule of the power data to be detected according to the similarity data label of each data obtained in step S3 and the check table and check field obtained in step S4; S6. Use the detection rule of the power data to be detected obtained in step S6 to perform actual anomaly detection of the power data to be detected.

2. The method of claim 1, wherein The step S4 of obtaining the historical detection rule of power data anomaly detection and processing to obtain the corresponding check table and check field, specifically comprising the following steps: Obtain the training data set; Build a BiLSTM-CRF model as a preliminary entity recognition model: The preliminary entity recognition model includes a bidirectional long short-term memory network BiLSTM and a conditional random field model CRF; Each unit in the bidirectional long short-term memory network BiLSTM comprises an input gate, a forget gate, an output gate, a long memory structure and a short memory structure; for the input character at time t , the hidden layer state at time t-1 and the unit state at time t-1 , the calculation of each unit is as follows: The input gate calculation method is as follows ,in The output of the input gate, It is the sigmoid activation function. Here is the weight matrix of the input gate. It is and The matrix formed by splicing together The bias vector for the input gate; The forget gate calculation is wherein is the output of the forget gate, is the weight matrix of the forget gate, is the bias vector of the forget gate; The output gate is calculated in the following manner wherein is the output of the output gate, is the weight matrix of the output gate, is the bias vector of the output gate; The long memory structure is calculated in the following manner wherein is the cell state at time t, is the cell state of the new input information at time t and , is a weight matrix, is a bias vector; The short memory machine structure calculates in the following manner wherein is the hidden layer state at time t. The conditional random field model CRF is used to fully extract the global information of the input text; the parameterized representation of the observation sequence X in the conditional random field model CRF is wherein is the conditional probability of the state sequence y given the observation sequence x, is the state sequence, is the observation sequence, is the normalization term and , For the transition characteristic function, Let i be the state at time i. , It is the subscript variable at all times. for State characteristic coefficients at time t. It is a state characteristic function; Train the constructed preliminary entity recognition model using the obtained training data set until the model converges to obtain an entity recognition model; Input the obtained historical detection rule of power data anomaly detection into the entity recognition model to predict the check table and check field.

3. The method of claim 2, wherein The step S5 of generating the detection rule of the power data to be detected according to the similarity data label of each data obtained in step S3 and the check table and check field obtained in step S4, specifically comprising the following steps: The check table and the corresponding check field in the historical detection rule are replaced by the data table corresponding to the similar label and the data field corresponding to the table, so that a new detection rule of the to-be-detected power data is generated.

4. The method of claim 3, wherein The step S6 adopts the detection rule of the to-be-detected power data obtained in the step S6 to perform actual abnormal detection of the to-be-detected power data, and specifically includes the following steps: Deploy the new detection rule of the to-be-detected power data and run the generated sample data; Manually judge the sample data: If the judgment is passed, the new detection rule of the to-be-detected power data is used for actual detection; If the judgment is not passed, manual intervention is performed.

5. A detection system for implementing the method of any one of claims 1 to 4, characterized in that The system comprises a standard data set acquisition module, a similarity calculation module, a similarity data label calculation module, a check table and check field calculation module, a detection rule updating module and a detection module; the standard data set acquisition module, the similarity calculation module, the similarity data label calculation module, the check table and check field calculation module, the detection rule updating module and the detection module are sequentially connected; The standard data set acquisition module is used to acquire the to-be-detected power data, process a standard data set, and upload the result to the similarity calculation module; the similarity calculation module is used to calculate the similarity between each data in the received standard data set, and upload the result to the similarity data label calculation module; the similarity data label calculation module is used to obtain the similarity data label of each data according to the received similarity calculation result, and upload the result to the check table and check field calculation module; the check table and check field calculation module is used to acquire the historical detection rule of the power data abnormal detection, process the corresponding check table and check field, and upload the result to the detection rule updating module; The detection rule updating module is used to generate the detection rule of the to-be-detected power data according to the similarity data label and the check table and check field, and upload the result to the detection module; The detection module is used to adopt the received detection rule of the to-be-detected power data to perform actual abnormal detection of the to-be-detected power data.

Images