Load investigation data preprocessing and feature extraction method

Abstract

The present invention relates to the field of big data processing technology, and specifically to a method for preprocessing and feature extraction of power load investigation data. The method for preprocessing and feature extraction of load investigation data includes the following processes: data preprocessing; feature extraction, extracting features useful for load analysis from the preprocessed data; and finally performing feature dimension reduction. Through the above-mentioned data preprocessing and feature extraction process, the load data analysis module can provide high-quality and highly relevant data input for subsequent data modeling and analysis, thereby improving the application value of the data asset management platform in the field of power load business.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method for preprocessing and extracting features of power load investigation data. Background Art

[0002] In today's society, with the rapid growth of the economy, the demand for power load continues to rise, and power load management has become an important issue facing the power industry. In this context, how to efficiently manage and use load data has become an urgent problem to be solved in the power load business field.

[0003] For a long time, companies in various provinces in my country have accumulated a large amount of load data, but due to the lack of a unified data platform, these data resources have not been effectively integrated and utilized. Traditional load data processing methods often have data islands, resulting in poor information sharing and difficulty in deep mining and analysis of load data. In addition, with the development of technologies such as big data and cloud computing, power load data has shown explosive growth, which has put forward higher requirements for data processing and analysis capabilities. Summary of the invention

[0004] The in-depth mining and analysis of data based on the proposed unified data management platform can provide a powerful decision-making basis for power market operation, power system planning, etc. The specific technical solution is as follows:

[0005] The load investigation data preprocessing and feature extraction method includes the following processes:

[0006] S100: Data preprocessing; specifically including:

[0007] S110: Data cleaning; the first step is to check whether there are missing values ​​in the data set, and fill or delete these missing values; the second step is to detect outliers in the data set through statistical or machine learning methods, and decide whether to correct, delete or retain them.

[0008] S120: De-noising; specific process: applying digital filtering technology to remove high-frequency noise in the data; using smoothing technology to reduce the impact of random fluctuations on the data;

[0009] S130: Formatting processing, converting the data into a unified format;

[0010] S200: Feature extraction: extracting features useful for load analysis from preprocessed data; specifically including:

[0011] S210: Feature selection, calculating the following statistics of load data: maximum value, minimum value, mean, variance, standard deviation, peak-to-valley difference; extracting the following time series characteristics of load data: periodicity, trend, seasonality, which can be achieved through time series analysis technology;

[0012] S220: Feature construction; construct new features based on professional knowledge such as user behavior characteristics, weather impact characteristics, and holiday effects; use machine learning models to automatically select and construct features.

[0013] S300: Feature dimensionality reduction; specifically includes: using feature importance scoring method to evaluate the contribution of features to the model's predictive ability; based on the feature importance score, screening out the feature subset that contributes the most to the model to reduce the number of features and improve analysis efficiency.

[0014] Preferably, the filling method in S110 is to use the average value, median, mode or model-based predicted value.

[0015] Preferably, the statistical method in S110 is a box plot analysis method.

[0016] Preferably, the machine learning method in S110 is Isolation Forest.

[0017] Preferably, the digital filtering technology in S120 is sliding average filtering or low-pass filtering.

[0018] Preferably, the smoothing technique in S120 is exponential smoothing or sample smoothing.

[0019] Preferably, in S130, the date and time formats are unified, and load data in different units are converted into standard units.

[0020] Preferably, the machine learning model in S220 is a decision tree model or a random forest model.

[0021] Preferably, the feature importance scoring method in S300 is a model-based feature importance scoring method.

[0022] Through the above-mentioned data preprocessing and feature extraction process, the load data analysis module can provide high-quality and highly relevant data input for subsequent data modeling and analysis, thereby improving the application value of the data asset management platform in the power load business field. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the load troubleshooting data preprocessing and feature extraction method of the present invention. DETAILED DESCRIPTION

[0024] The load investigation data preprocessing and feature extraction method includes the following processes:

[0025] S100: Data preprocessing; specifically including:

[0026] First, data cleaning is required. The first step is to check whether there are missing values ​​in the data set and fill or delete these missing values. The filling method can be to use the mean, median, mode or model-based predicted value. The second step is to detect outliers in the data set through statistical methods (such as box plot analysis) or machine learning methods (such as Isolation Forest) and decide whether to correct, delete or retain them.

[0027] The second is denoising, which uses digital filtering techniques, such as sliding average filtering and low-pass filtering, to remove high-frequency noise in the data; and uses smoothing techniques, such as exponential smoothing or spline smoothing, to reduce the impact of random fluctuations on the data.

[0028] Finally, formatting is performed to convert the data into a unified format, such as unifying the date and time format and converting load data in different units into standard units. Dimensionality reduction is performed on the data, such as reducing the feature dimensions of the data through principal component analysis (PCA) while retaining most of the information.

[0029] S200: Extract features useful for load analysis from preprocessed data. First, feature selection is performed to calculate the statistics of load data, such as maximum value, minimum value, mean, variance, standard deviation, peak-to-valley difference, etc.; extract the time series features of load data, such as periodicity, trend, seasonality, etc., which can be achieved through time series analysis technology;

[0030] Feature construction: construct new features based on professional knowledge, such as user behavior features, weather impact features, holiday effect features, etc.; use machine learning models (such as decision trees and random forests) to automatically select and construct features.

[0031] S300: Perform feature dimensionality reduction and use feature importance scoring methods (such as model-based feature importance scoring) to evaluate the contribution of features to the model's predictive ability. Based on the feature importance scores, select the feature subset that contributes most to the model to reduce the number of features and improve analysis efficiency.

Claims

1. Load troubleshooting data preprocessing and feature extraction method, characterized in that: The process includes the following: S100: Data preprocessing; specifically including: S110: Data cleaning; the first step is to check whether there are missing values ​​in the data set, and fill or delete these missing values; the second step is to detect outliers in the data set through statistical methods or machine learning methods, and decide whether to correct, delete or retain them; S120: De-noising; specific process: applying digital filtering technology to remove high-frequency noise in the data; using smoothing technology to reduce the impact of random fluctuations on the data; S130: Formatting processing, converting the data into a unified format; S200: Feature extraction: extracting features useful for load analysis from preprocessed data; specifically including: S210: Perform feature selection and calculate the following statistics of the load data: maximum value, minimum value, mean, variance, standard deviation, peak-to-valley difference; extract the following time series features of the load data: periodicity, trend, and seasonality; S220: Feature construction: construct new features based on professional knowledge such as user behavior features, weather impact features, and holiday effects; use machine learning models to automatically select and construct features; S300: Feature dimensionality reduction; specifically includes: using feature importance scoring method to evaluate the contribution of features to the model's predictive ability; based on the feature importance score, screening out the feature subset that contributes the most to the model to reduce the number of features and improve analysis efficiency.

2. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The filling method in S110 is to use the average value, median, mode or a predicted value based on a model.

3. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The statistical method in S110 is a box plot analysis method.

4. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The machine learning method in S110 is Isolation Forest.

5. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The digital filtering technology in S120 is sliding average filtering or low-pass filtering.

6. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The smoothing technique in S120 is exponential smoothing or sample smoothing.

7. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: In the S130, the date and time formats are unified and the load data in different units are converted into standard units.

8. The load troubleshooting data preprocessing and feature extraction method according to claim 1 is characterized in that: The machine learning model in S220 is a decision tree model or a random forest model.

9. The load troubleshooting data preprocessing and feature extraction method according to claim 1, characterized in that: The feature importance scoring method in S300 is a model-based feature importance scoring method.

Images