Spot electric energy market settlement data anomaly detection method and related device
Through Spearman correlation coefficient screening features and combining random forest and extreme gradient enhancement models, RF-XGBoost fusion decision tree is constructed, which solves the problem of data acquisition error and overfitting in spot electricity energy market settlement data, and improves the accuracy and performance of abnormal detection .
Patent Information
- Application Number
- CN202411202953.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-03
AI Technical Summary
Spot electricity market settlement data, in the case of data acquisition errors and lack of sufficient training data, leads to overfitting of the model and hinders the abnormal detection performance.
Spearman correlation coefficient was used to analyze feature correlation, and strongly correlated feature data were screened out, and the RF-XGBoost fusion decision tree model was constructed through random forest and extreme gradient enhancement XGBoost model to perform anomaly detection.
It improves the accuracy and performance of abnormal detection, reduces the impact of overfitting, and enhances the model's abnormal data capture ability on the market settlement system.
Smart Images

Figure CN119067698A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of electric energy market settlement data detection, and in particular relates to a spot electric energy market settlement data anomaly detection method and related devices. Background Art
[0002] With the construction of the domestic spot electricity market, the market settlement mechanism has become the key to the stable and orderly operation of the spot electricity market. The number of market settlement entities continues to grow, the transaction cycle is gradually shortened, the settlement rules are becoming more diverse, and the settlement complexity is rapidly expanding. Accurate market settlement abnormal data detection is an important measure to address the performance stability of the power market settlement system. Current research focuses on the prediction of sufficient and high-quality data. However, the settlement entities in the spot electricity market are diverse, and the settlement data is continuously rolling. The settlement system inevitably has problems such as data leakage, data collection errors, and data deviations, which directly affect the detection performance of the system. Therefore, there is an urgent need for anomaly detection methods that can extract effective feature data to improve the accuracy of data detection based on a limited amount of data.
[0003] Various algorithms in the prior art usually have branches with poor classification effects of decision trees, and lack comprehensive, integrated and multi-angle analysis of resource-constrained space payloads, which leads to inaccurate anomaly detection results. To address this problem, a spatial payload anomaly detection method based on improved random forest is proposed. By improving the weighted voting random forest algorithm, the weights of decision tree branches with poor classification effects are reduced, and the sliding window step size and failure threshold are optimized. A lightweight anomaly detection model is established to identify and locate anomalies in resource-constrained space payload data.
[0004] Most of the existing methods for detecting anomalies in settlement of spot electricity energy market only focus on the transaction settlement data of the settlement system. Although they can alleviate the impact of multi-type data errors to a certain extent, they ignore the characteristics of single settlement data under the influence of multiple features, which will interfere with the anomaly detection of internal single settlement data. Only focusing on the data characteristics of the settlement system itself will make the model pay more attention to the adequacy and effectiveness of its own data, which will inevitably be affected by data collection errors. In the absence of sufficient effective training data, severe overfitting will hinder the model's anomaly detection performance. Summary of the invention
[0005] The purpose of the present invention is to provide a method and related device for detecting anomaly in settlement data of spot electric energy market, so as to solve the problem that severe overfitting will hinder the anomaly detection performance of the model in the absence of sufficient effective training data under the influence of data acquisition errors.
[0006] To achieve the above object, the present invention adopts the following technical solutions:
[0007] In a first aspect, the present invention provides a method for detecting abnormality in settlement data of a spot electric energy market, comprising:
[0008] Collect historical settlement data of the spot electric energy market, calculate the characteristic correlation information of the historical settlement data, and preprocess the historical settlement data to obtain a preprocessed settlement data set;
[0009] Select the preprocessed settlement data set and feature correlation information, and filter out settlement feature data with strong correlation;
[0010] Based on the random forest and extreme gradient boosting XGBoost models, an anomaly detection model of RF-XGBoost fusion decision tree is constructed, which takes the settlement feature data with strong correlation as input and outputs the classification results of abnormal settlement data.
[0011] Furthermore, historical settlement data of the spot electricity market is collected, and characteristic correlation information of the historical settlement data is calculated, including:
[0012] The Spearman correlation coefficient is used to analyze the characteristic correlation between historical settlement data and sort the elements in the characteristic quantity, including:
[0013] The Spearman correlation coefficient is specifically defined as follows:
[0014]
[0015] Where, ρ is the Spearman correlation coefficient; d i is the rank difference between variable X and variable Y; n is the sample size; the Spearman correlation coefficient ranges from [-1,1], ρ=1 indicates a perfect positive correlation between the two variables, ρ=-1 indicates a perfect negative correlation between the two variables, and ρ=0 indicates no correlation between the two variables;
[0016] Select the settlement features of historical settlement data as input, which contains w feature sequences, each of which contains n T Sampling points; let each settlement feature sequence of historical settlement data be F T+1 ={f T+1,1 ,f T+1,2 ,...,f T+1,w}, each feature sequence is
[0017] The main characteristic sequence F of the historical settlement data within the settlement system T+1 With history obs The main characteristic sequence F of the settlement data of the day i i∈[Tn obs,T] as the model input, calculate the Spearman correlation coefficient between the two types of input sequences, and obtain the correlation coefficient matrix of the two types of sequences, as shown in the following formula:
[0018]
[0019] In the formula, ξ ij ,i∈[1,w]j∈[1,n obs ] is the Spearman correlation coefficient between the i-th feature of the historical settlement data on the j-th day;
[0020] Yes JS Sort and select the largest element corresponding to the settlement data of the historical day as the feature correlation information, denoted as D data .
[0021] Furthermore, the historical settlement data is preprocessed, including:
[0022] Preprocess the collected historical settlement data, including correcting and filling the missing values in the data, to obtain a complete historical data set, specifically:
[0023] The missing values are filled using the nearest neighbor interpolation method, and the calculation method is as follows:
[0024]
[0025] Where, X i It is the value in the historical settlement data at time point i on a certain day. NaN is a missing value.
[0026] Furthermore, the preprocessed settlement data set and feature correlation information are selected to screen out settlement feature data with strong correlation, including:
[0027] The random forest regression algorithm is used to perform feature screening on the historical settlement data and feature correlation information of the sorted historical days, specifically:
[0028] Assume that the size of the input historical settlement data T is N, the feature dimension of the data sample is M, and the number of CART regression trees in the random forest regression algorithm is n CART , the number of different features corresponding to each CART regression tree in the random forest regression algorithm is m CART , the maximum depth of the CART regression tree in the random forest regression algorithm is d max ; The random forest regression algorithm randomly selects n in T samples_split The settlement data of the day is used as the backup input T for each CART regression tree samples_split , calculate the regression error e1 of each CART regression tree; from T samples_split Randomly select m fromCART m CART <M features are used as out-of-bag features of the CART regression tree. After adding them to the random seed, they are input into the CART regression tree together with other features to calculate the out-of-bag error e2 of the out-of-bag features. After obtaining e1 and e2, calculate the importance feature matrix K = [k1, k2,..., k W , as the settlement feature data with strong correlation; the calculation method is as follows:
[0029]
[0030] In the formula, n c is the number of rounds of input data; y i , y i ′ and y′ OBB,i are the actual output of the i-th round, the output of the CART regression tree without out-of-bag feature input, and the output of the CART regression tree with out-of-bag feature input, respectively.
[0031] Furthermore, an anomaly detection model of the RF-XGBoost fusion decision tree is constructed based on the random forest and the extreme gradient boosting XGBoost model, including:
[0032] According to the settlement feature data K = [k1, k2,..., k W output by the random forest regression algorithm, input it into the extreme gradient boosting XGBoost model for feature fusion to obtain the anomaly detection model of the RF-XGBoost fusion decision tree.
[0033] Furthermore, using the settlement feature data with strong correlation as the input, output the classification result of the abnormal settlement data, including:
[0034] The objective function of the XGBoost model is shown as follows:
[0035]
[0036] In the formula: z t and z′ t are the actual value and the predicted value of the t-th data sample of the model respectively; l(z t , z′ t ) is the t
[0037] In the formula: U is the number of leaf nodes of the decision tree; γ and λ are hyperparameters; ω g is the score of the leaf node;
[0038] After s rounds of iteration, for the s-th tree, its objective function is shown as follows:
[0039]
[0040] The second-order Taylor expansion formula is used to expand the regularization term, and the final objective function is simplified as shown below:
[0041]
[0042] Where: c t (·) and d t (·) respectively represent right The first and second derivatives of ; Definition
[0043] The optimal value of the objective function can be obtained by taking the first-order derivative of the objective function and setting it to 0, as shown in the following formula:
[0044]
[0045] Furthermore, after outputting the classification results of abnormal settlement data, the performance of the anomaly detection model of RF-XGBoost fusion decision tree is evaluated using four model metrics: accuracy, precision, recall, and F1-score, including:
[0046]
[0047]
[0048] In the formula, TP is the number of samples that are actually positive and correctly predicted to be positive; TN is the number of samples that are actually negative and correctly predicted to be negative; FP is the number of samples that are actually negative but incorrectly predicted to be positive; FN is the number of samples that are actually positive but incorrectly predicted to be negative; Accuracy represents the ratio of the number of correctly classified samples to the total number of samples; Precision represents the ratio of the number of samples that are actually positive among the samples predicted to be positive to the number of samples predicted to be positive; Recall represents the ratio of the number of samples that are correctly predicted to be positive among the samples that are actually positive to the total number of actual positive samples; F1-score represents the harmonic mean that comprehensively considers precision and recall; the value range of the four indicators, Accuracy, Precision, Recall and F1-score, are all [0,1]. The larger the value, the better the anomaly detection performance.
[0049] In a second aspect, the present invention provides a spot electric energy market settlement data anomaly detection system, comprising:
[0050] The data acquisition module is used to collect historical settlement data of the spot electric energy market, calculate the characteristic correlation information of the historical settlement data, and pre-process the historical settlement data;
[0051] The data screening module is used to select the pre-processed settlement data set and feature correlation information, and screen out settlement feature data with strong correlation;
[0052] The model building and output module is used to build an anomaly detection model of RF-XGBoost fusion decision tree based on random forest and extreme gradient boosting XGBoost models, taking highly correlated settlement feature data as input and outputting abnormal settlement data classification results.
[0053] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a method for detecting anomalies in settlement data of a spot electric energy market when executing the computer program.
[0054] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a method for detecting anomalies in settlement data of a spot electric energy market are implemented.
[0055] Compared with the prior art, the present invention has the following technical effects:
[0056] The present invention proposes a method for detecting anomaly in settlement data of spot electric energy market by integrating decision tree, so as to realize the identification requirement of abnormal data generated by settlement calculation and improve the accuracy of anomaly detection. Firstly, the Spearman correlation coefficient is used to perform feature correlation analysis and preliminarily screen key feature data; then, the features of settlement data are further mined by random forest algorithm, and features with strong correlation are screened as new settlement data sets; finally, the settlement feature data screened by random forest algorithm is used as input to construct a fusion decision tree anomaly detection model based on random forest and extreme gradient boosting, and the classification result of abnormal settlement data is used as output to realize accurate detection and classification of outliers in settlement data sets.
[0057] From the perspective of the overall settlement data of the spot electricity market, a feature selection method combining the Spearman correlation coefficient and the random forest algorithm is designed to improve the settlement data of the market settlement system to be detected by extracting various feature data. The extreme gradient boosting algorithm can accurately process large-scale data and quickly distinguish different categories of features in classification problems. It is very suitable for the settlement anomaly data detection task in the market settlement system. At the same time, in order to adapt to the different data feature quantities selected, a settlement anomaly detection framework based on the RF-XGBoost fusion decision tree is constructed, which effectively extracts the characteristics of the market settlement system, so that the model has a strong ability to capture abnormal data and further improves the anomaly detection performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 Flow chart of the present invention;
[0059] Figure 2 Flowchart of historical settlement data feature selection;
[0060] Figure 3 Flowchart of anomaly detection fused with decision trees;
[0061] Figure 4 Fusion decision tree classification graph;
[0062] Figure 5 Comparison chart of multi-model anomaly detection indicators;
[0063] Figure 6 Classification chart of anomaly types in abnormal settlement data. DETAILED DESCRIPTION
[0064] The present invention is further described below in conjunction with the accompanying drawings:
[0065] Example 1, please refer to Figure 1 The present invention discloses a method for detecting abnormality in settlement data of spot electric energy market, comprising:
[0066] Collect historical settlement data of the spot electricity market, calculate the characteristic correlation information of the historical settlement data, and pre-process the historical settlement data;
[0067] Select the preprocessed settlement data set and feature correlation information, and filter out settlement feature data with strong correlation;
[0068] Based on the random forest and extreme gradient boosting XGBoost models, an anomaly detection model of RF-XGBoost fusion decision tree is constructed, which takes the settlement feature data with strong correlation as input and outputs the classification results of abnormal settlement data.
[0069] The settlement anomaly detection framework based on RF-XGBoost fusion decision tree constructed in the present invention effectively extracts the characteristics of the market settlement system, so that the model has a powerful abnormal data capture capability and further improves the anomaly detection performance.
[0070] Example 2, please refer to Figures 2 to 6 The present invention discloses a method for detecting abnormality in settlement data of spot electric energy market, which specifically includes:
[0071] Step 1: Preprocess the collected settlement data by data cleaning and filling in missing values, and use the Spearman correlation coefficient to analyze the feature correlation to provide necessary correlation information for subsequent feature screening;
[0072] The specific steps include:
[0073] Preprocessing the collected historical settlement data mainly includes correcting and filling the missing values in the data to obtain a complete historical data set. Including:
[0074] The missing values are filled using the nearest neighbor interpolation method, and the calculation method is as follows:
[0075]
[0076] Where, X i is the value in the historical settlement data at time point i on a certain day, and NaN is a missing value;
[0077] The Spearman correlation coefficient is used to analyze the feature correlation between historical settlement data, and the elements in the feature quantity are further sorted, including:
[0078] The Spearman correlation coefficient usually calculates the correlation between variable rankings to evaluate the relationship between variables. Its specific definition is shown in the following formula:
[0079]
[0080] Where, ρ is the Spearman correlation coefficient; d i is the rank difference (rank difference) between variable X and variable Y; n is the sample size. The Spearman correlation coefficient ranges from [-1,1], ρ=1 indicates a perfect positive correlation between the two variables, ρ=-1 indicates a perfect negative correlation between the two variables, and ρ=0 indicates no correlation between the two variables;
[0081] Select the settlement features of historical settlement data as input, which contains w feature sequences, each of which contains n T Sampling points. Assume that the settlement feature sequences of historical settlement data are F T+1 ={f T+1,1 ,f T+1,2 ,...,f T+1,w}, each feature sequence is
[0082] The main characteristic sequence F of the historical settlement data within the settlement system T+1 With history obs The main characteristic sequence F of the settlement data of the day i i∈[Tn obs ,T] as the model input, calculate the Spearman correlation coefficient between the two types of input sequences, and obtain the correlation coefficient matrix of the two types of sequences, as shown in the following formula:
[0083]
[0084] In the formula, ξij ,i∈[1,w]j∈[1,n obs ] is the Spearman correlation coefficient between the i-th feature of the historical settlement data on the j-th day;
[0085] Yes JS Sort and select the largest element corresponding to the settlement data of the historical day as the initial data, denoted as D data ;
[0086] Step 2: Use the random forest (RF) algorithm to select the preprocessed settlement data set and feature correlation information, learn and characterize various types of features of the spot electricity market settlement system, and screen out settlement feature data with strong correlation;
[0087] The random forest regression algorithm is used to perform feature screening on the historical settlement data and feature correlation information of the sorted historical days, including:
[0088] Assume that the size of the input historical settlement data T is N, the feature dimension of the data sample is M, and the number of CART regression trees in the random forest regression algorithm is n CART , the number of different features corresponding to each CART regression tree in the random forest regression algorithm is m CART , the maximum depth of the CART regression tree in the random forest regression algorithm is d max The random forest regression algorithm randomly selects n from T. samples_split The settlement data of the day is used as the backup input T for each CART regression tree samples_split , calculate the regression error e1 of each CART regression tree; from T samples_split Randomly select m from CART (m CART <M) features are used as out-of-bag features of the CART regression tree. After adding them to the random seed, they are input into the CART regression tree together with other features to calculate the out-of-bag error e2 of the out-of-bag features. After obtaining e1 and e2, calculate the importance feature matrix K=[k1,k2,...,k W ]. The calculation method is as follows:
[0089]
[0090] Where n c is the number of rounds of input data; y i ,y i ′ and y′ OBB,i They are the actual output of the i-th round, the output of the CART regression tree without out-of-bag feature input, and the output of the CART regression tree with out-of-bag feature input;
[0091] Step 3: The settlement feature data obtained by RF algorithm screening is used as input to build an anomaly detection model of RF-XGBoost fusion decision tree, and the abnormal settlement data classification results are used as output to achieve accurate detection of settlement anomalies in the spot electricity market settlement system;
[0092] According to the settlement feature data K=[k1,k2,...,k W ], input it into the XGBoost model for feature fusion, and detect the abnormality of settlement data in the settlement system to be detected through model training;
[0093] The objective function of the XGBoost model is as follows:
[0094]
[0095] Where: z t and z′ t are the actual value and predicted value of the t-th data sample of the model respectively; l(z t ,z′ t ) is the loss function of the tth data sample, the smaller the error, the better; is the regularization term. The smaller the value, the better the generalization ability of the model. The regularization term is shown as follows:
[0096]
[0097] Where: U is the number of decision tree leaf nodes; γ and λ are hyperparameters; ω g is the score of the leaf node;
[0098] After s rounds of iterations, for the s-th tree, the objective function is as follows:
[0099]
[0100] The second-order Taylor expansion formula is used to expand the regularization term, and the final objective function is simplified as shown below:
[0101]
[0102] Where: c t (·) and d t (·) respectively represent right The first and second derivatives of ; Definition
[0103] The optimal value of the objective function can be obtained by taking the first-order derivative of the objective function and setting it to 0, as shown in the following formula:
[0104]
[0105] Step 4: Based on the anomaly detection results obtained in step 3, four model metrics, namely accuracy, precision, recall and F1-score, are used to compare and evaluate the proposed model with the traditional model.
[0106] In order to evaluate the performance of the model more accurately, four indicators, Accuracy, Precision, Recall and F1-score, are set to evaluate the anomaly detection performance of the model; the specific calculation method is as follows:
[0107]
[0108] In the formula, TP is the number of samples that are actually positive and correctly predicted as positive; TN is the number of samples that are actually negative and correctly predicted as negative; FP is the number of samples that are actually negative but incorrectly predicted as positive; FN is the number of samples that are actually positive but incorrectly predicted as negative. Accuracy is the ratio between the number of correctly classified samples and the total number of samples. Precision is the ratio between the number of samples that are actually positive and the number of samples that are predicted as positive. Recall is the ratio between the number of samples that are actually positive and the total number of samples that are actually positive. F1-score is the harmonic mean that takes into account precision and recall. The value range of the four indicators Accuracy, Precision, Recall and F1-score is [0,1]. The larger the value, the better the anomaly detection performance of the algorithm.
[0109] The present invention realizes the identification demand of abnormal data generated by settlement calculation and improves the accuracy of anomaly detection. First, the Spearman correlation coefficient is used to perform feature correlation analysis and preliminarily screen key feature data; then, the features of settlement data are further mined through the random forest algorithm, and features with strong correlation are screened as new settlement data sets; finally, the settlement feature data screened by the random forest algorithm is used as input to construct a fusion decision tree anomaly detection model based on random forest and extreme gradient boosting, and the classification results of abnormal settlement data are used as output to achieve accurate detection and classification of outliers in the settlement data set.
[0110] Embodiment 3,
[0111] This example adopts a method for detecting anomaly in the settlement data of the spot electric energy market by integrating a decision tree proposed in the present invention. In order to verify the effectiveness of the proposed algorithm for detecting anomaly in the settlement data of the spot electric energy market by integrating a decision tree, the historical settlement data of the spot electric energy market in a certain region is selected for experimental analysis. The sampling frequency of the data is 30 minutes / time, and the time span of the data is from May 1, 2024 to June 1, 2024. In this chapter, 80% of the data is selected as the training set and 20% of the data is selected as the validation set.
[0112] In order to verify the superiority of the model proposed in this chapter, a comparative experiment between single RF and single XGBoost was designed. The input data volume of each model was kept consistent. The various indicators of anomaly detection obtained were as follows: Figure 5 As shown. It can be seen that the proposed method is superior to other anomaly detection algorithms in terms of evaluation indicators Accuracy, Precision, Recall and F1-score. Compared with the RF algorithm, the evaluation indicators Accuracy of the proposed model increased by 0.01679, Precision increased by 0.04962, Recall increased by 0.00101, and F1-score increased by 0.02503; compared with the XGBoost algorithm, the evaluation indicators Accuracy of the proposed model increased by 0.00672, Precision increased by 0.01887, Recall increased by 0.01818, and F1-score increased by 0.01852. Therefore, the performance of the proposed model is better than other algorithms.
[0113] In yet another embodiment of the present invention, a spot electric energy market settlement data anomaly detection system is provided, which can be used to implement the above-mentioned spot electric energy market settlement data anomaly detection method. Specifically, the system includes:
[0114] The data acquisition module is used to collect historical settlement data of the spot electric energy market, calculate the characteristic correlation information of the historical settlement data, and pre-process the historical settlement data to obtain a pre-processed settlement data set;
[0115] The data screening module is used to select the pre-processed settlement data set and feature correlation information, and screen out settlement feature data with strong correlation;
[0116] The model building and output module is used to build an anomaly detection model of RF-XGBoost fusion decision tree based on random forest and extreme gradient boosting XGBoost models, taking highly correlated settlement feature data as input and outputting abnormal settlement data classification results.
[0117] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0118] In another embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory being used to store a computer program, the computer program comprising program instructions, and the processor being used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a method for detecting abnormality in settlement data of a spot electric energy market.
[0119] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the above-mentioned embodiment in a method for detecting abnormality in a spot electric energy market settlement data.
[0120] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0121] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for detecting abnormality in settlement data of spot electric energy market, characterized in that: include: Collect historical settlement data of the spot electricity market, calculate the historical settlement data to obtain feature correlation information, and preprocess the historical settlement data to obtain a preprocessed settlement data set; Select the preprocessed settlement data set and feature correlation information, and filter out settlement feature data with strong correlation; Based on the random forest and extreme gradient boosting XGBoost models, an anomaly detection model of RF-XGBoost fusion decision tree is constructed, which takes the settlement feature data with strong correlation as input and outputs the classification results of abnormal settlement data.
2. A method for detecting abnormality in settlement data of spot electric energy market according to claim 1, characterized in that: Collect historical settlement data of the spot electricity market and calculate the characteristic correlation information of the historical settlement data, including: The Spearman correlation coefficient is used to analyze the characteristic correlation between historical settlement data and sort the elements in the characteristic quantity, including: The Spearman correlation coefficient is specifically defined as follows: Where, ρ is the Spearman correlation coefficient; d i is the rank difference between variable X and variable Y; n is the sample size; the Spearman correlation coefficient ranges from [-1,1], ρ=1 indicates a perfect positive correlation between the two variables, ρ=-1 indicates a perfect negative correlation between the two variables, and ρ=0 indicates no correlation between the two variables; Select the settlement features of historical settlement data as input, which contains w feature sequences, each of which contains n T Sampling points; let each settlement feature sequence of historical settlement data be F T+1 ={f T+1,1 ,f T+1,2 ,...,f T+1,w }, each feature sequence is i∈[1,w]; The main characteristic sequence F of the historical settlement data within the settlement system T+1 With history obs The main characteristic sequence F of the settlement data of the day i i∈[Tn obs ,T] as the model input, calculate the Spearman correlation coefficient between the two types of input sequences, and obtain the correlation coefficient matrix of the two types of sequences, as shown in the following formula: In the formula, ξ ij ,i∈[1,w]j∈[1,n obs ] is the Spearman correlation coefficient between the i-th feature of the historical settlement data on the j-th day; Yes JS Sort and select the largest element corresponding to the settlement data of the historical day as the feature correlation information, denoted as D data .
3. A method for detecting abnormality in settlement data of spot electric energy market according to claim 1, characterized in that: Preprocess historical settlement data, including: Preprocess the collected historical settlement data, including correcting and filling the missing values in the data, to obtain a complete historical data set, specifically: The missing values are filled using the nearest neighbor interpolation method, and the calculation method is as follows: In the formula, X i It is the value in the historical settlement data at time point i on a certain day. NaN is a missing value.
4. A method for detecting abnormality in settlement data of spot electric energy market according to claim 1, characterized in that: Select the preprocessed settlement data set and feature correlation information, and filter out settlement feature data with strong correlation, including: The random forest regression algorithm is used to perform feature screening on the historical settlement data and feature correlation information of the sorted historical days, specifically: Assume that the size of the input historical settlement data T is N, the feature dimension of the data sample is M, and the number of CART regression trees in the random forest regression algorithm is n CART , and the number of different features corresponding to each CART regression tree in the random forest regression algorithm is m CART , and the maximum depth of the CART regression tree in the random forest regression algorithm is d max ; The random forest regression algorithm randomly extracts the settlement data of n samples_split days from T as the standby input T of each CART regression tree samples_split , and calculate the regression error e1 of each CART regression tree; randomly select m samples_split from T CART m CART <M features as the out-of-bag features of the CART regression tree, add them to the random seed and input them into the CART regression tree together with other features, and calculate the out-of-bag error e2 of the out-of-bag features; after obtaining e1 and e2, calculate the importance feature matrix K = [k1, k2,..., k W , as the settlement feature data with strong correlation; the calculation method is as follows: Where n c is the number of rounds of input data; y i ,y i ′ and y′ OBB,i They are respectively the actual output of the i-th round, the output of the CART regression tree without out-of-bag feature input, and the output of the CART regression tree with out-of-bag feature input.
5. A method for detecting abnormality in settlement data of spot electric energy market according to claim 1, characterized in that: Anomaly detection model of RF-XGBoost fusion decision tree is built based on random forest and extreme gradient boosting XGBoost model, including: According to the settlement feature data K=[k1,k2,...,k W ] and input it into the extreme gradient boosting XGBoost model for feature fusion to obtain the anomaly detection model of RF-XGBoost fusion decision tree.
6. A method for detecting abnormality in settlement data of spot electric energy market according to claim 5, characterized in that: Taking the settlement feature data with strong correlation as input, the classification results of abnormal settlement data are output, including: The objective function of the XGBoost model is as follows: Where: z t and z′ t are the actual value and predicted value of the t-th data sample of the model respectively; l(z t ,z′ t ) is the loss function of the tth data sample; is the regularization term, which is shown as follows: Where: U is the number of decision tree leaf nodes; γ and λ are hyperparameters; ω g is the score of the leaf node; After s rounds of iterations, for the s-th tree, the objective function is as follows: The second-order Taylor expansion formula is used to expand the regularization term, and the final objective function is simplified as shown below: Where: c t (·) and d t (·) respectively represent right The first and second derivatives of ; Definition The optimal value of the objective function can be obtained by taking the first-order derivative of the objective function and setting it to 0, as shown in the following formula:
7. A method for detecting abnormality in settlement data of spot electric energy market according to claim 6, characterized in that: After outputting the classification results of abnormal settlement data, the performance of the anomaly detection model of RF-XGBoost fusion decision tree is evaluated using four model metrics: accuracy, precision, recall, and F1-score, including: In the formula, TP is the number of samples that are actually positive and correctly predicted to be positive; TN is the number of samples that are actually negative and correctly predicted to be negative; FP is the number of samples that are actually negative but incorrectly predicted to be positive; FN is the number of samples that are actually positive but incorrectly predicted to be negative; Accuracy represents the ratio of the number of correctly classified samples to the total number of samples; Precision represents the ratio of the number of samples that are actually positive among the samples predicted to be positive to the number of samples predicted to be positive; Recall represents the ratio of the number of samples that are correctly predicted to be positive among the samples that are actually positive to the total number of actual positive samples; F1-score represents the harmonic mean that comprehensively considers precision and recall; the value range of the four indicators, Accuracy, Precision, Recall and F1-score, are all [0,1]. The larger the value, the better the anomaly detection performance.
8. A spot electricity market settlement data anomaly detection system, characterized in that: include: The data acquisition module is used to collect historical settlement data of the spot electric energy market, calculate the characteristic correlation information of the historical settlement data, and pre-process the historical settlement data to obtain a pre-processed settlement data set; The data screening module is used to select the pre-processed settlement data set and feature correlation information, and screen out settlement feature data with strong correlation; The model building and output module is used to build an anomaly detection model of RF-XGBoost fusion decision tree based on random forest and extreme gradient boosting XGBoost models, taking highly correlated settlement feature data as input and outputting abnormal settlement data classification results.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a method for detecting anomalies in settlement data of a spot electric energy market are implemented as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting anomalies in settlement data of a spot electric energy market as described in any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Commission settlement abnormal data detection method and system based on gradient boosting decision tree
CN122989330A
