A carbon emission abnormal data detection method, a terminal and a storage medium

Abstract

A kind of carbon emission abnormal data detection method, terminal and storage medium, after the data processing of carbon emission system, train a scene-based support vector machine, for analyzing the real-time data monitored by carbon emission system, detect whether there is abnormal data.The present application constructs the SVM classifier model for carbon emission abnormal data detection, collects carbon emission historical data, labels the label of normal or abnormal to data by step S2, trains the SVM model constructed, finally the carbon emission data generated in real time is detected by the abnormal classification of trained SVM, it is beneficial to the effective control of mass carbon emission data, and abnormal data is detected.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology and relates to the analysis and processing of carbon emission data, specifically a method for detecting abnormal carbon emission data. Background Technology

[0002] Due to the complexity of carbon emissions, especially urban carbon emissions, existing technologies cannot accurately quantify urban carbon emission levels. They also lack the ability to analyze anomalies in carbon emission data, making it impossible to accurately analyze carbon emission anomalies from both macro and micro perspectives.

[0003] In recent years, the detection of anomaly carbon emission data has attracted considerable attention. Currently, commonly used methods include neural network algorithms, support vector machine algorithms, random forest algorithms, and data mining-based anomaly detection algorithms. However, all of these algorithms have limitations and cannot simultaneously achieve both detection accuracy and efficiency. While these algorithms can perform identification when the amount of data is fixed, in practical applications, data lengths vary greatly, and the data exhibits high fluidity and diversity. Historical data has very low reliability, leading to significant errors in the judgment thresholds obtained by traditional algorithms, resulting in serious false positives and false negatives. To address these issues, this invention proposes a carbon emission anomaly data detection algorithm based on Condition-Based SVM (Scene-Based Support Vector Machine). Summary of the Invention

[0004] The problem this invention aims to solve is to propose a method for detecting abnormal carbon emission data in order to detect abnormal data in massive heterogeneous carbon emission data.

[0005] The technical solution of this invention is: a method for detecting abnormal carbon emission data, which uses a scenario-based support vector machine to analyze the data monitored by the carbon emission system and detect whether there is abnormal data, including the following steps:

[0006] Step S1: Data preprocessing, normalizing the data obtained from the carbon emission system monitoring;

[0007] Step S2: Extract feature vectors from the preprocessed data based on three scenarios with recursive relationships. The three scenarios are, in order, based on data value characteristics, correlation between variables, and data change characteristics. The recursive relationships are as follows:

[0008] The first-level feature vector judgment involves setting a range of data values ​​and determining whether the data contains that range or exists within a specific time period. If not, it is identified as abnormal data. If it does exist, the second-level feature vector judgment uses distance metric learning to further filter abnormal data and calculates the metric matrix for the normal dataset. For the data set with specific value ranges after the first-level judgment, the distance to the test samples is calculated using the metric matrix. The furthest distance of the normal data is taken as a threshold. If the sample distance exceeds the threshold, it is identified as abnormal data. If it is less than the threshold, the third-level feature vector judgment begins. At this point, the monitored data is a time series with equal time intervals. If there are abrupt changes in the data between consecutive time points, it is identified as abnormal data; if the changes are the same, it is identified as normal data.

[0009] Step S3: Select training samples and prediction samples: Based on the labels of the selected detection samples in step S2, divide them into training sample set and test sample set;

[0010] Step S4: Model selection: Select RBF as the kernel function, and use grid search and cross-validation to select the kernel parameter c and penalty factor g;

[0011] Step S5: Construct an SVM classifier model; use the kernel function and optimal parameters selected in step S4 to train the selected training sample set to construct an SVM classifier model for detecting abnormal carbon emission data.

[0012] Step S6: Use the trained SVM classifier model to identify and classify the predicted sample data, that is, classify the data into two categories: abnormal data or normal data, thereby realizing abnormal data detection.

[0013] The present invention also provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the above-described method for detecting abnormal carbon emission data when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-described method for detecting abnormal carbon emission data.

[0015] The beneficial effects of this invention are as follows: This invention constructs an SVM classifier model for detecting abnormal carbon emission data, collects historical carbon emission data, labels the data as normal or abnormal in step S2, trains the constructed SVM model, and finally uses the trained SVM to perform anomaly classification and detection on the real-time carbon emission data, which is beneficial for effectively controlling massive carbon emission data and detecting abnormal data. Attached Figure Description

[0016] Figure 1 This is a flowchart of the carbon emission anomaly data detection method of the present invention.

[0017] Figure 2 This is a flowchart of feature vector extraction in the method of the present invention. Detailed Implementation

[0018] like Figure 1 As shown, a scenario-based support vector machine is used to analyze the data monitored by the carbon emission system and detect whether there is abnormal data. The steps include the following steps.

[0019] Step S1: Data preprocessing; using the data obtained from carbon emission monitoring as the test sample, the test sample is normalized.

[0020] Step S2: Feature extraction; feature values ​​are extracted based on three scenarios: data value characteristics, correlation between variables, and data change characteristics. These three scenarios are recursively related.

[0021] Table 1 Description of Data Feature Vectors

[0022]

[0023] like Figure 2 As shown, the feature vector extraction process of this invention is as follows.

[0024] (1) The first-level feature vector is the range of values ​​for the data (within a specific time period), i.e., the threshold m. The threshold of the sample data is used to determine whether the data itself is within the specific range of values ​​or whether it is within the range of values ​​within a specific time period. If not, it can be identified as abnormal data.

[0025] (2) If so, proceed to the second-level feature vector value judgment. Use distance metric learning to filter out abnormal data. Through learning, obtain a metric matrix that can effectively reflect the distance or similarity between data samples, which is used to measure the spatial distance between individuals. The greater the distance, the greater the difference between individuals.

[0026] This invention utilizes a non-paired constraint distance metric learning algorithm based on maximizing the classification margin. It uses non-paired constraint information such as categories or triples as prior knowledge in normal data samples and learns a metric matrix by maximizing the distance between different categories.

[0027] The specific algorithm is as follows: Let x represent a training set consisting of n samples, where the input x is... i The corresponding category label is y i Using scalar y ij ∈{0,1} represents the category label y iand y j Whether it matches, and whether a linear transformation L:R needs to be learned. d →R d Perform the following distance measurement:

[0028] d M (x i ,x j )=(x i -x j ) T M(x i -x j )=||L(x i -x j )|| 2 (3-1)

[0029] Where M = L T L.

[0030] For any input sample x i Its target neighbors are defined as input samples that meet the following conditions: 1) and x i There are labels of the same category y i ;2) Calculate x using equation (3-1) i The input sample with the minimum distance. For sample x i You can specify k target neighbors. Use the scalar η. i,j ∈{0,1}, when η i,j When x = 1, it means x j It is x i Target neighbors; y i,j When x = 1, it means x i With x j They belong to the same category. The pair y i,j and η i,j These are all fixed, meaning they will not change during the learning process.

[0031] The objective function is obtained based on the idea of ​​maximizing classification editing:

[0032] ε(L)=∑ ij η i,j ||L(x i -x j )|| 2 +c∑ ijl (1-y il )×[1+||L(x i -x j )|| 2 -||L(x i -x l )|| 2 ] + (3-2)

[0033] Among them, [z] + =max(z,0) and c is some positive constant.

[0034] In equation (3-2), the first term on the right-hand side adjusts the distance between all input samples and the target neighbor, minimizing this term to make the distance between the input sample and the target neighbor as small as possible; the second term adjusts the marginal distance between different categories, minimizing this term to maximize the marginal distance.

[0035] To facilitate solving over a larger feasible region, a slack variable ξ is introduced. ijl Equation (3-2) is transformed into solving the following positive semidefinite programming problem:

[0036]

[0037]

[0038]

[0039] M≥0 (3-3)

[0040] The metric matrix M is obtained by solving the above equation using a standard positive semidefinite programming algorithm. The distance between samples is then calculated using the obtained metric matrix. The furthest distance in the normal dataset is taken as a threshold. If the distance exceeds this threshold, the data is considered outlier.

[0041] (3) If the data is within the threshold, then proceed to the third-level feature vector judgment. At this point, the monitored data, i.e., the data that has not been judged as abnormal after the first and second levels of judgment, is a time series with equal time intervals. Calculate the monitored time series V = {v1,…,v...} t-1 ,v t ,v t+1 ,…,v T The degree of data change at each point in the vector is denoted by I, and the third-level feature vector is taken as the degree of data change I.

[0042]

[0043] This represents the average of the absolute values ​​of the data changes between the current time and the two time points before and after it. If the data changes between the two time points exceed the maximum allowable threshold, it can be identified as abnormal data.

[0044] (4) If the changes in the data before and after are basically the same, then it is determined to be normal data.

[0045] Step S3: Select training samples and prediction samples; divide the selected detection samples into training sample set and test sample set.

[0046] Step S4: Construct the SVM classifier model. Select the Gaussian kernel function RBF and train it on the training samples.

[0047] This involves mapping points in two-dimensional space to three-dimensional or even higher-dimensional spaces, transforming problems in low-dimensional space into problems in high-dimensional space. The mapping from low-dimensional to high-dimensional space uses a mapping function Φ(x). To simplify the vector inner product operation after mapping to high-dimensional space, a Gaussian kernel function is introduced, such that K(x) = ... i ,y i )=[Φ(x i )·Φ(x j To solve the classification problem under nonlinear segmentation, the objective function expression is:

[0048]

[0049] Therefore, the decision function for the interface is:

[0050]

[0051] We use grid search and cross-validation to select the kernel parameter g and penalty coefficient C to construct the SVM classifier model. The current approach involves selecting values ​​for c and g within a certain range, using the training set as the original dataset, and employing k-fold cross-validation (K-CV) to obtain the classification accuracy of the validation set under the combination of c and g. Finally, we use grid search to select the optimal c and g combination that maximizes the classification accuracy of the training set.

[0052] Step S5: Using the kernel function and optimal parameters selected in the previous step, train the constructed SVM classifier model for detecting carbon emission anomalies on the selected training sample set.

[0053] Step S6: Use the trained SVM classifier model to identify and classify the predicted sample data, determine whether the data belongs to abnormal data or normal data, and output the classification result.

Claims

1. A method for detecting anomaly data in carbon emissions, characterized in that: Using scenario-based support vector machines, the data from the carbon emission system monitoring is analyzed to detect any abnormal data. This includes the following steps: Step S1: Data preprocessing, normalizing the data obtained from the carbon emission system monitoring; Step S2: Extract feature vectors from the preprocessed data based on three scenarios with recursive relationships. The three scenarios are, in order, data value characteristics, correlation between variables, and data change characteristics. The recursive relationships are as follows: The first level of feature vector judgment involves setting a range of values ​​for the data and determining whether the data contains that range or exists within a specific time period. If not, it is identified as abnormal data. If it does exist, the process proceeds to the second level of feature vector judgment, which uses distance metric learning to further filter abnormal data and calculates the metric matrix for the normal dataset. For data sets with specific value ranges after the first level of judgment, the distance to test samples is calculated using the metric matrix. The furthest distance of normal data is taken as a threshold. If the sample distance exceeds the threshold, it is identified as abnormal data. If it is less than the threshold, the process proceeds to the third level of feature vector judgment. At this point, the monitored data is a time series with equal time intervals. If there are abrupt changes in the data between consecutive time points, it is identified as abnormal data. If the changes in the data are the same before and after, it is identified as normal data. Step S3: Select training samples and prediction samples: Based on the labels of the selected detection samples in step S2, divide them into training sample set and test sample set; Step S4: Model selection: Select RBF as the kernel function, and use grid search and cross-validation to select the kernel parameter c and penalty factor g; Step S5: Construct an SVM classifier model; use the kernel function and optimal parameters selected in step S4 to train the selected training sample set to construct an SVM classifier model for detecting abnormal carbon emission data. Step S6: Use the trained SVM classifier model to identify and classify the predicted sample data, that is, classify the data into two categories: abnormal data or normal data, thereby realizing abnormal data detection; In step S2, the metric matrix for the normal dataset is solved based on a non-paired constraint distance metric learning algorithm that maximizes the classification margin. Non-paired constraint information is used as prior knowledge in the normal data samples, and the metric matrix is ​​learned by maximizing the distance between different categories, as detailed below: express The training set consists of samples, where the input... The corresponding category label is Use scalar Indicates category label and Whether it matches, and whether a linear transformation needs to be learned. Perform the following distance measurement: （3-1） in, ; For any input sample Its target neighbors are defined as input samples that meet the following conditions: 1) and There are the same category labels ;2) Calculate the result using equation (3-1) Input samples with minimum distance; For the sample Specify k target neighbors, using scalar ,when When =1, it means yes Target neighbors; When =1, it means and Belonging to the same category, binary tuple and All are fixed; The objective function is obtained based on the idea of ​​maximizing the classification margin: （3-2） in, c is a positive constant; In equation (3-2), the first term on the right-hand side adjusts the distance between all input samples and the target neighbor. Minimizing this term minimizes the distance between the input sample and the target neighbor. The second term adjusts the marginal distance between different categories. Minimizing this term maximizes the marginal distance. Introducing slack variables Equation (3-2) is transformed into solving the following positive semidefinite programming problem: （3-3） The metric matrix M is obtained by solving the above equation using the standard positive semidefinite programming algorithm. The distance between samples is then calculated using the obtained metric matrix, and the farthest distance of the normal dataset is taken as the threshold.

2. The carbon emission anomaly data detection method according to claim 1, characterized in that, in step S2, during the third-level feature vector judgment, the monitored data is a time series with equal time intervals, and the monitored time series is calculated. The degree of data change at each point in the data is expressed as follows: This indicates that the third-level feature vector is taken as the degree of data change. : , （3-4） It represents the average of the absolute values ​​of the data changes between the current time and the two times before and after. If the data changes between the two times before and after exceed the maximum allowable threshold, it is identified as abnormal data.

3. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the carbon emission anomaly data detection method as described in claim 1 or 2.

4. A computer-readable storage medium storing a computer program, characterized in that... When the computer program is executed by the processor, it implements the carbon emission anomaly data detection method as described in claim 1 or 2.

Images