Meteorological data cleaning method and system based on electric power environment

By generating data distribution images to detect and process outliers and missing values ​​in meteorological data, combined with repeated processing, the problems of meteorological data quality and reliability are solved, the accuracy and consistency of data are improved, and the power system is provided with high quality and reliability. Quality data support.

CN118939949APending Publication Date: 2024-11-12SOUTHERN POWER GRID DIGITAL GRID RESEARCH INSTITUTE CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410914339.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the power environment, there are problems of outliers, missing values ​​and repeated values ​​in meteorological data, which affects the quality and reliability of the data, and thus affects the accuracy of decisions.

Method used

Outliers are detected by generating data distribution images, and outliers are processed using Z-score comparison method, quantile detection method or autoencoder method; missing values ​​are processed using deletion method, interpolation method or constant filling method; finally The data set is deduplicated to improve the quality and reliability of the data.

Benefits of technology

It effectively improves the accuracy and reliability of meteorological data, ensures the consistency and integrity of data, and improves the support capabilities of power system operation and energy management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939949A_ABST
    Figure CN118939949A_ABST
Patent Text Reader

Abstract

The invention provides a meteorological data cleaning method and system based on an electric power environment. The method comprises the following steps: acquiring an original meteorological data set; generating a corresponding data distribution image according to the distribution condition of each data point in the original meteorological data set; whether an abnormal value exists in the original meteorological data set or not is judged according to the data distribution image, if the abnormal value exists, a preset abnormal value processing method is used for processing the abnormal value in the original meteorological data set, and a first meteorological data set is obtained, the abnormal value processing method is a Z-score comparison method, a quantile detection method or an auto-encoder method; detecting missing values in the first meteorological data set, and processing the missing values in the first meteorological data set by using a corresponding missing value processing method according to a detection result to obtain a second meteorological data set; and performing deduplication processing on the second meteorological data set to obtain the meteorological data set after data cleaning, thereby improving the quality and reliability of the data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a meteorological data cleaning method and system based on an electric power environment. Background Art

[0002] Meteorological data refers to data that records and measures various meteorological elements of the atmospheric environment, mainly including information such as temperature, humidity, air pressure, wind direction, wind speed, and precipitation. These data are collected through meteorological observation equipment such as meteorological stations, satellites, and radars, and are used to predict weather and climate conditions and support decision-making and planning in various industries. In the power environment, meteorological data is also widely used, such as in energy forecasting for wind power generation and solar power generation, and load forecasting for power systems. By analyzing and using meteorological data, the efficiency, accuracy, and reliability of the power system can be improved.

[0003] In the power environment, meteorological data is of great significance to the operation of the power system, equipment maintenance, and energy management. However, due to various reasons, the original meteorological data may have problems such as outliers, missing values, and duplicate values. These data problems will directly affect the quality and reliability of the data, and thus affect the accuracy of related decisions; meteorological data from different sources may need to be integrated and processed to ensure the consistency and integrity of the data. In addition, data integration may involve data in different formats, resolutions, and time intervals, which requires specialized technologies and tools to process. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a meteorological data cleaning method and system based on the power environment, which effectively improves the quality and reliability of the data by cleaning the original meteorological data, and provides data support for subsequent power system operation, equipment maintenance, energy management and other aspects.

[0005] In a first aspect, the present invention provides a meteorological data cleaning method based on power environment, comprising:

[0006] Get the original meteorological dataset;

[0007] Generate a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set;

[0008] Determine whether the original meteorological data set has an outlier according to the data distribution image; if an outlier exists, use a preset outlier processing method to process the outlier in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method, or an autoencoder method;

[0009] Detect missing values ​​in the first meteorological data set, and process the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method;

[0010] Deduplication processing is performed on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

[0011] The embodiment of the present invention provides a meteorological data cleaning method based on the power environment. First, a data distribution image is established for the original meteorological data set, which can more intuitively display the distribution status of the original meteorological data and quickly find abnormal data in the original meteorological data set. Further, when it is found that there are abnormal values ​​in the original meteorological data, the abnormal values ​​are processed using a preset abnormal value processing method to improve the accuracy of the meteorological data and obtain a first meteorological data set. Then, the missing values ​​in the first meteorological data set are detected, and the missing values ​​are processed using a deletion method, an interpolation method, or a constant filling method according to the detection result of the missing values ​​to obtain a second meteorological data set, wherein the deletion method directly deletes the data points where the missing values ​​are located, and the interpolation method and the filling method fill the missing values. By detecting and processing the missing values ​​of the first meteorological data set, the consistency and integrity of each data in the data set are ensured. Finally, the second meteorological data set is deduplicated to remove redundant and repeated data, further streamline the second meteorological data set, and further improve the quality and reliability of the data.

[0012] In a possible implementation, the using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the using the Z-score comparison method to process outliers in the original meteorological data set includes:

[0013] Calculate the mean and standard deviation of the original meteorological data set;

[0014] Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation;

[0015] Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point;

[0016] When the retrieval is completed, the remaining data points are the first meteorological data set.

[0017] In an embodiment of the present invention, the Z-score comparison method is used to process the outliers in the original meteorological data set. The Z-score method is an outlier detection method based on statistical principles. It can calculate the distance between each data point and the mean and standard deviation of the entire data set, and use this distance to measure the degree of abnormality of the data point. When the distance between a first data point and the mean and standard deviation of the original data set exceeds a preset threshold, it means that the value of the data point is an outlier. The existence of the data point will affect the accuracy and reliability of the original data set. Therefore, the first data point is directly deleted. When the retrieval is completed, the remaining undeleted data points are all normal data points that meet the requirements of the data set, which can constitute the first meteorological data set, and the outlier processing of the original meteorological data set is completed.

[0018] In a possible implementation, the using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the using the quantile detection method to process outliers in the original meteorological data set includes:

[0019] Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set;

[0020] Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range;

[0021] Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set;

[0022] When the retrieval is completed, the first meteorological data set is obtained.

[0023] In an embodiment of the present invention, the quantile detection method is used to process the abnormal values ​​in the original meteorological data set. First, the interquartile range of the original meteorological data set is calculated, and then the numerical upper threshold and the numerical lower threshold of each data point in the original meteorological data set are determined according to the interquartile range. Then, by searching and comparing each data point and the threshold in the original meteorological data set, the data points of the abnormal values ​​can be marked. After the retrieval is completed, these abnormal values ​​are replaced with the median. In addition, during the retrieval process, each time an abnormal value is retrieved, the abnormal value is immediately replaced with the median. The quantile detection method is applicable to continuous data, and can quickly process abnormal values ​​in the data set, and will not change the data distribution and the number of data points in the data set. Compared with the Z-score comparison method, the quantile detection method realizes the repair of abnormal values ​​in the data set and also ensures the integrity of the original data set. It should be noted that this method is only applicable to continuous data, and for some data sets with a wider distribution or more extreme values, it may be necessary to adjust the threshold or adopt other processing methods.

[0024] In a possible implementation, the using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the using the autoencoder method to process outliers in the original meteorological data set includes:

[0025] Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model;

[0026] The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

[0027] In an embodiment of the present invention, the autoencoder method is used to process the outliers in the original meteorological data set. The autoencoder model is constructed based on an unsupervised deep learning algorithm, and its purpose is to compress the input data into a low-dimensional vector and restore it to the original data through a decoder. If there are outliers in the input data, the autoencoder will show a high error during reconstruction, thereby realizing the identification of outliers. Since the autoencoder method uses a deep learning algorithm, it is suitable for outlier processing of large-scale meteorological data. Compared with the Z-score comparison method and the quantile detection method, the autoencoder method does not need to calculate certain indicator values ​​of the original meteorological data set as a whole, and can directly detect each data in the data set. When processing large-scale data sets, the running speed is significantly higher than the Z-score comparison method and the quantile detection method, and the autoencoder model can optimize itself during the outlier processing process, further improving the accuracy of outlier identification.

[0028] Further, the detecting missing values ​​in the first meteorological data set, and processing the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set includes:

[0029] Detecting missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score;

[0030] When the missing value ratio is less than a first preset value, a deletion method is used to process the missing values ​​in the first meteorological data set, specifically: directly deleting the data points where the missing values ​​are located;

[0031] When the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, a constant filling method is used to process the missing values ​​in the first meteorological data set, specifically: the missing values ​​are filled with the average, median or mode of the same row or column as the missing value;

[0032] When the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value, an interpolation method is used to process the missing values ​​in the first meteorological data set, specifically: predicting the values ​​of the missing values ​​based on the non-missing value data in the first meteorological data set and filling them accordingly, the interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0033] An embodiment of the present invention provides a method for processing missing values ​​according to a detection result. First, the missing value ratio and the missing value importance score in a first meteorological data set are detected, and a corresponding missing value processing method is determined based on these two indicators. When the missing value ratio is less than the first preset value, it means that the missing part of the data set is small, and the corresponding data points can be directly deleted, which has little impact on the integrity of the data set; when the missing value ratio is greater than the first preset value, and the missing value importance score is less than the second preset value, the missing value ratio is large, which means that the data points cannot be simply deleted at this time, otherwise the number of samples in the data set will be greatly reduced, and the missing value importance score is small, which means that the missing value does not affect the subsequent use of the data set, and the missing value can be directly filled with a constant to ensure the numerical integrity of each data point. The constant can be the mean, median or mode of the same row or column as the missing value; when the missing value ratio is greater than the first preset value, and the missing value importance score is greater than the second preset value, the missing value has affected the subsequent normal use of the data set and cannot be filled at will. It is necessary to analyze the non-missing value data in the first meteorological data set, use the information of the existing data, and use a certain method to infer the value of the missing value, including linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0034] Furthermore, the machine learning interpolation method includes:

[0035] Use historical meteorological data to train machine learning models to obtain meteorological data prediction models;

[0036] The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set.

[0037] The embodiment of the present invention provides a machine learning interpolation method, which obtains a meteorological data prediction model by training a machine learning model with historical meteorological data. The meteorological data prediction model can automatically analyze the target data set, predict the specific values ​​of each missing value in the data set and fill in the missing values ​​accordingly. Therefore, a first meteorological data set is input into the meteorological data prediction model, and the meteorological data prediction model is used to process the missing values ​​in the first meteorological data set. The machine learning interpolation method can reasonably and quickly process the missing values ​​in the data set, improve the efficiency of data cleaning, and further ensure the quality and reliability of the meteorological data set.

[0038] In a second aspect, accordingly, the present invention provides a meteorological data cleaning system based on an electric power environment, comprising an acquisition module, an image generation module, an abnormal value processing module, a missing value processing module, and a de-duplication module;

[0039] Wherein, the acquisition module is used to acquire the original meteorological data set;

[0040] The image generation module is used to generate a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set;

[0041] The outlier processing module is used to determine whether there are outliers in the original meteorological data set according to the data distribution image, and if there are outliers, use a preset outlier processing method to process the outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method or an autoencoder method;

[0042] The missing value processing module is used to detect missing values ​​in the first meteorological data set, and process the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method;

[0043] The deduplication module is used to perform deduplication processing on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

[0044] In a possible implementation, the outlier processing module uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the Z-score comparison method, including:

[0045] Calculate the mean and standard deviation of the original meteorological data set;

[0046] Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation;

[0047] Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point;

[0048] When the retrieval is completed, the remaining data points are the first meteorological data set.

[0049] In a possible implementation, the outlier processing module uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the quantile detection method, including:

[0050] Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set;

[0051] Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range;

[0052] Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set;

[0053] When the retrieval is completed, the first meteorological data set is obtained.

[0054] In a possible implementation, the outlier processing module uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the autoencoder method, including:

[0055] Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model;

[0056] The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

[0057] Furthermore, the missing value processing module includes a detection unit, a deletion unit, a filling unit and an interpolation unit;

[0058] The detection unit is used to detect missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score;

[0059] The deletion unit is used to process the missing values ​​in the first meteorological data set using a deletion method when the missing value ratio is less than a first preset value, specifically: directly deleting the data point where the missing value is located;

[0060] The filling unit is used to process the missing values ​​in the first meteorological data set using a constant filling method when the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, specifically: fill the missing values ​​with the average value, median or mode of the same row or column as the missing value;

[0061] The interpolation unit is used to process the missing values ​​in the first meteorological data set using an interpolation method when the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value. Specifically, the numerical value of the missing value is predicted based on the non-missing value data in the first meteorological data set and filled in accordingly. The interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0062] Furthermore, the machine learning interpolation method includes:

[0063] Use historical meteorological data to train machine learning models to obtain meteorological data prediction models;

[0064] The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 : A flow chart of an embodiment of a meteorological data cleaning method based on power environment provided by the present invention.

[0066] Figure 2 : A schematic diagram of the process of processing missing values ​​in an embodiment of a meteorological data cleaning method based on an electric power environment provided by the present invention.

[0067] Figure 3 : A structural schematic diagram of an embodiment of a meteorological data cleaning system based on power environment provided by the present invention.

[0068] Figure 4 : A structural schematic diagram of a missing value processing module of an embodiment of a meteorological data cleaning system based on an electric power environment provided by the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0071] Embodiment 1:

[0072] like Figure 1 As shown, embodiment 1 provides a meteorological data cleaning method based on power environment, including steps S1-S5:

[0073] Step S1, obtaining an original meteorological data set;

[0074] Step S2, generating a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set;

[0075] Step S3, judging whether there are outliers in the original meteorological data set according to the data distribution image, and if there are outliers, using a preset outlier processing method to process the outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method or an autoencoder method;

[0076] Step S4, detecting missing values ​​in the first meteorological data set, and processing the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method;

[0077] Step S5: De-duplication processing is performed on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

[0078] The embodiment of the present invention provides a meteorological data cleaning method based on the power environment. First, a data distribution image is established for the original meteorological data set, which can more intuitively display the distribution status of the original meteorological data and quickly find abnormal data in the original meteorological data set. Further, when it is found that there are abnormal values ​​in the original meteorological data, the abnormal values ​​are processed using a preset abnormal value processing method to improve the accuracy of the meteorological data and obtain a first meteorological data set. Then, the missing values ​​in the first meteorological data set are detected, and the missing values ​​are processed using a deletion method, an interpolation method, or a constant filling method according to the detection result of the missing values ​​to obtain a second meteorological data set, wherein the deletion method directly deletes the data points where the missing values ​​are located, and the interpolation method and the filling method fill the missing values. By detecting and processing the missing values ​​of the first meteorological data set, the consistency and integrity of each data in the data set are ensured. Finally, the second meteorological data set is deduplicated to remove redundant and repeated data, further streamline the second meteorological data set, and further improve the quality and reliability of the data.

[0079] After obtaining the original meteorological data set, when the original meteorological data set contains outliers, these outliers may affect the statistical results of the data and the accuracy of the machine learning model, so it is necessary to perform outlier detection and outlier processing on the data set, i.e., step S2-step S3. Wherein, in step S2, a box plot or a scatter plot is usually used to represent the data distribution image of the original meteorological data set, and the box plot is a method of detecting outliers by drawing the box and tentacles of the data distribution. In the box plot, the box represents the quartiles of the data (i.e., the upper quartile and the lower quartile), the median is represented by a horizontal line, and the tentacles are used to represent the range of normal values. In the box plot, if a data point is outside the range of the tentacles, it is considered to be an outlier. In step S3, outlier processing is performed by selecting the Z-score comparison method, the quantile detection method, or the autoencoder method, and the specific outlier processing method can be selected by the user according to the characteristics of the data set and the actual effects of various methods. A computer program code for generating a box plot based on the original meteorological data set is given below:

[0080] import pandas as pd

[0081] import matplotlib.pyplot as plt

[0082] #Read the dataset

[0083] df = pd.read_csv('data1.csv')

[0084] # draw box plot

[0085] df.boxplot()

[0086] # Display the graph

[0087] plt.show()

[0088] In a possible implementation, in step S3, the outliers in the original meteorological data set are processed using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the Z-score comparison method, including:

[0089] Calculate the mean and standard deviation of the original meteorological data set;

[0090] Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation;

[0091] Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point;

[0092] When the retrieval is completed, the remaining data points are the first meteorological data set.

[0093] In a preferred embodiment, the computer program code for processing the outliers in the original meteorological data set using the Z-score comparison method is:

[0094] import pandas as pd

[0095] #Read the dataset

[0096] df = pd.read_csv('filled_data-decision-tree.csv')

[0097] #Calculate Z-score

[0098] z = (df - df.mean()) / df.std()

[0099] #Replace the value of Z-score exceeding the threshold with NaN

[0100] threshold=3

[0101] df[z.abs()>threshold]=None

[0102] # Delete rows containing NaN values

[0103] df = df.dropna()

[0104] #Save the dataset

[0105] df.to_csv('filled_data-decision tree-null value-outlier.csv',index=False)

[0106] In an embodiment of the present invention, the Z-score comparison method is used to process the outliers in the original meteorological data set. The Z-score method is an outlier detection method based on statistical principles. It can calculate the distance between each data point and the mean and standard deviation of the entire data set, and use this distance to measure the degree of abnormality of the data point. When the distance between a first data point and the mean and standard deviation of the original data set exceeds a preset threshold, it means that the value of the data point is an outlier. The existence of the data point will affect the accuracy and reliability of the original data set. Therefore, the first data point is directly deleted. When the retrieval is completed, the remaining undeleted data points are all normal data points that meet the requirements of the data set, which can constitute the first meteorological data set, and the outlier processing of the original meteorological data set is completed.

[0107] In a possible implementation, in step S3, the outliers in the original meteorological data set are processed using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the quantile detection method, including:

[0108] Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set;

[0109] Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range;

[0110] Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set;

[0111] When the retrieval is completed, the first meteorological data set is obtained.

[0112] In a preferred embodiment, the computer program code for processing the abnormal values ​​in the original meteorological data set using the quantile detection method is:

[0113] import pandas as pd

[0114] #Read data

[0115] data=pd.read_csv('data.csv')

[0116] #Define threshold

[0117] q1 = data.quantile(0.25)

[0118] q3 = data.quantile(0.75)

[0119] iqr=q3-q1

[0120] lower_bound=q1-1.5*iqr

[0121] upper_bound=q3+1.5*iqr

[0122] #Determine whether it is an outlier

[0123] outliers=(data<lower_bound)|(data> upper_bound)

[0124] #Replace outliers with the median

[0125] median = data.median()

[0126] data[outliers]=median

[0127] The code first calculates the 25% quantile, 75% quantile, and interquartile range (IQR) of the data set, and then calculates the lower and upper thresholds based on the IQR. Next, by comparing each data point with the threshold, a Boolean data frame is obtained to indicate which data points are outliers. Finally, these outliers are replaced with the median.

[0128] In an embodiment of the present invention, the quantile detection method is used to process the abnormal values ​​in the original meteorological data set. First, the interquartile range of the original meteorological data set is calculated, and then the numerical upper threshold and the numerical lower threshold of each data point in the original meteorological data set are determined according to the interquartile range. Then, by searching and comparing each data point and the threshold in the original meteorological data set, the data points of the abnormal values ​​can be marked. After the retrieval is completed, these abnormal values ​​are replaced with the median. In addition, during the retrieval process, each time an abnormal value is retrieved, the abnormal value is immediately replaced with the median. The quantile detection method is applicable to continuous data, and can quickly process abnormal values ​​in the data set, and will not change the data distribution and the number of data points in the data set. Compared with the Z-score comparison method, the quantile detection method realizes the repair of abnormal values ​​in the data set and also ensures the integrity of the original data set. It should be noted that this method is only applicable to continuous data, and for some data sets with a wider distribution or more extreme values, it may be necessary to adjust the threshold or adopt other processing methods.

[0129] In a possible implementation, in step S3, the outliers in the original meteorological data set are processed using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the autoencoder method, including:

[0130] Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model;

[0131] The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

[0132] A common method in deep learning is to use anomaly detection using an autoencoder. An autoencoder is an unsupervised learning algorithm that aims to compress input data into a low-dimensional vector and restore it to the original data through a decoder. If there are outliers in the input data, the autoencoder will show a high error in reconstruction.

[0133] In a preferred embodiment, the computer program code for processing outliers in the original meteorological data set using the autoencoder method is:

[0134] import pandas as pd

[0135] import numpy as np

[0136] import tensorflow as tf

[0137] from sklearn.model_selection import train_test_split

[0138] #Read data

[0139] df = pd.read_csv('data.csv')

[0140] #Mark abnormal values ​​as 1 and normal values ​​as 0

[0141] df['is_anomaly']=np.where(df['column1']>5,1,0)

[0142] # Separate the labels from the data

[0143] y = df.pop('is_anomaly')

[0144] # Divide the data into training set and test set

[0145] X_train,X_test,y_train,y_test=train_test_split(df,y,test_size=0.2,random_state=42)

[0146] #Build the autoencoder model

[0147] input_dim = X_train.shape[1]

[0148] encoding_dim=10

[0149] input_layer=tf.keras.layers.Input(shape=(input_dim,))

[0150] encoder=tf.keras.layers.Dense(encoding_dim,activation='relu')(input_layer)

[0151] decoder=tf.keras.layers.Dense(input_dim,activation='sigmoid')(encoder)

[0152] autoencoder=tf.keras.models.Model(inputs=input_layer, outputs=decoder)

[0153] # Compile the model

[0154] autoencoder.compile(optimizer='adam',loss='mean_squared_error')

[0155] #Train model

[0156] autoencoder.fit(X_train,X_train,epochs=50,batch_size=32,validation_data=(X_test,X_test))

[0157] # Use the trained autoencoder to detect outliers

[0158] reconstructed=autoencoder.predict(df)

[0159] mse=np.mean(np.power(df-reconstructed,2),axis=1)

[0160] threshold=np.percentile(mse,95)

[0161] is_anomaly=np.where(mse>threshold,1,0)

[0162] df['is_anomaly'] = is_anomaly

[0163] In an embodiment of the present invention, the autoencoder method is used to process the outliers in the original meteorological data set. The autoencoder model is constructed based on an unsupervised deep learning algorithm, and its purpose is to compress the input data into a low-dimensional vector and restore it to the original data through a decoder. If there are outliers in the input data, the autoencoder will show a high error during reconstruction, thereby realizing the identification of outliers. Since the autoencoder method uses a deep learning algorithm, it is suitable for outlier processing of large-scale meteorological data. Compared with the Z-score comparison method and the quantile detection method, the autoencoder method does not need to calculate certain indicator values ​​of the original meteorological data set as a whole, and can directly detect each data in the data set. When processing large-scale data sets, the running speed is significantly higher than the Z-score comparison method and the quantile detection method, and the autoencoder model can optimize itself during the outlier processing process, further improving the accuracy of outlier identification.

[0164] Further, in step S4, the missing values ​​in the first meteorological data set are detected, and the missing values ​​in the first meteorological data set are processed using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, such as Figure 2 As shown, steps S401-S401 are included:

[0165] Step S401: Detect missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score;

[0166] Step S402: when the missing value ratio is less than a first preset value, a deletion method is used to process the missing values ​​in the first meteorological data set, specifically: directly deleting the data points where the missing values ​​are located;

[0167] Step S403: when the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, a constant filling method is used to process the missing values ​​in the first meteorological data set, specifically: the missing values ​​are filled with the average, median or mode of the same row or column as the missing value;

[0168] Step S404: When the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value, an interpolation method is used to process the missing values ​​in the first meteorological data set, specifically: predicting the values ​​of the missing values ​​based on the non-missing value data in the first meteorological data set and filling them accordingly, the interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0169] In a preferred embodiment, in step S402, the first meteorological data set is a data set including temperature, humidity and atmospheric pressure, wherein temperature and humidity have missing values, and the specific computer program code for deleting these missing values ​​is:

[0170]

[0171]

[0172] In a preferred embodiment, in step S403, the first meteorological data set is a data set including temperature, humidity and atmospheric pressure, wherein there are missing values ​​for atmospheric pressure, and the specific computer program code for filling these missing values ​​with the mean is:

[0173]

[0174]

[0175] It should be noted that when there are missing values ​​in data of object type, the processing method is different from that of missing values ​​of numeric type. The following methods are usually used to process missing values ​​of object type:

[0176] Fill with mode: For object type features with limited values, missing values ​​can be filled with the value with the highest frequency, which can preserve the original distribution to the greatest extent.

[0177] Fill with a specific value: If you know the actual value corresponding to the missing value, you can directly fill it with that value.

[0178] Fill with special values ​​such as "Unknown": If the missing values ​​account for a high proportion, you can use "Unknown" as a value to fill the missing values.

[0179] Fill with interpolation: For non-text data, you can use interpolation to fill.

[0180] In a preferred embodiment, the specific code for filling the missing values ​​of the object type with the fillna() function in the Pandas library is:

[0181]

[0182] In the above code, select_dtypes() function is first used to find all columns of type object and store them in object_cols. Then, for each column, mode() function is used to calculate the mode of the column, and fillna() function is used to fill the missing values ​​with the mode. It should be noted that the inplace parameter of fillna() function is set to True, which means that the filling operation will be performed on the original data set. If you do not want to change the original data, you can set the inplace parameter of fillna() function to False. In this case, additional space will be needed to store the data set after the missing values ​​are filled.

[0183] An embodiment of the present invention provides a method for processing missing values ​​according to a detection result. First, the missing value ratio and the missing value importance score in a first meteorological data set are detected, and a corresponding missing value processing method is determined based on these two indicators. When the missing value ratio is less than the first preset value, it means that the missing part of the data set is small, and the corresponding data points can be directly deleted, which has little impact on the integrity of the data set; when the missing value ratio is greater than the first preset value, and the missing value importance score is less than the second preset value, the missing value ratio is large, which means that the data points cannot be simply deleted at this time, otherwise the number of samples in the data set will be greatly reduced, and the missing value importance score is small, which means that the missing value does not affect the subsequent use of the data set, and the missing value can be directly filled with a constant to ensure the numerical integrity of each data point. The constant can be the mean, median or mode of the same row or column as the missing value; when the missing value ratio is greater than the first preset value, and the missing value importance score is greater than the second preset value, the missing value has affected the subsequent normal use of the data set and cannot be filled at will. It is necessary to analyze the non-missing value data in the first meteorological data set, use the information of the existing data, and use a certain method to infer the value of the missing value, including linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0184] Furthermore, in step S404, the machine learning interpolation method includes:

[0185] Use historical meteorological data to train machine learning models to obtain meteorological data prediction models;

[0186] The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set.

[0187] In a preferred embodiment, the recommendation system method in machine learning is used to analyze the behavior and preference of the user's historical data, predict the data trend, and recommend the insertion of missing data that meets the actual situation. For example, the timing point B between timing points A and C is missing a point of voltage data. Through machine learning to analyze the historical voltage trend, select and insert the data that meets the requirements according to the data trend.

[0188] The embodiment of the present invention provides a machine learning interpolation method, which obtains a meteorological data prediction model by training a machine learning model with historical meteorological data. The meteorological data prediction model can automatically analyze the target data set, predict the specific values ​​of each missing value in the data set and fill in the missing values ​​accordingly. Therefore, a first meteorological data set is input into the meteorological data prediction model, and the meteorological data prediction model is used to process the missing values ​​in the first meteorological data set. The machine learning interpolation method can reasonably and quickly process the missing values ​​in the data set, improve the efficiency of data cleaning, and further ensure the quality and reliability of the meteorological data set.

[0189] In step S5, data deduplication is an important step in data cleaning, which can help us eliminate duplicate records in the data set and make the data more accurate and reliable. Deduplication is usually divided into two steps. The first step is to detect whether there are duplicate records in the data set, and the second step is to delete the duplicate records. The code example of data deduplication is as follows:

[0190] Example 1: Single column deduplication

[0191]

[0192]

[0193] As you can see, in this example, we first use the duplicated() function to detect duplicate records in the data set and save the result in the duplicated_data variable. Then, we use the drop_duplicates() function to delete duplicate records and save the result in the unique_data variable. As you can see, after deleting duplicate records, the size of the data set is reduced from 6 records to 4 records.

[0194] Example 2: Multiple columns for deduplication

[0195] In some cases, we need to remove duplicates from multiple columns in a data set. In Pandas, we can use the subset parameter of the drop_duplicates() function to specify the columns that need to be removed. Here is an example:

[0196]

[0197] In addition to the above data cleaning process, relevant technical personnel can process the data format according to actual needs, including modifying column names, format conversion, and merging data sets.

[0198] Among them, modifying the column name can be achieved through the rename method in the pandas library. For example, the following code renames the column named "old_column_name" to "new_column_name".

[0199] import pandas as pd

[0200] data=pd.read_csv('data.csv')

[0201] data=data.rename(columns={'old_column_name':'new_column_name'})

[0202] Format conversion needs to be processed according to the actual situation of the data set. For example, if the data type of a column in the data set is a string, but numerical calculations are required, the data type of the column needs to be converted to a numerical type. Data type conversion can be achieved through the astype method in the pandas library.

[0203] data['column_name']=data['column_name'].astype('float')

[0204] When you need to merge datasets in different formats, you need to first understand the format of each dataset and then use the appropriate method to convert them into the same format. For example, when you need to merge two datasets by row, you need to make sure they have the same column names and number of columns. If the column names of the two datasets are different, you can use the rename method in the pandas library to rename them to the same column name. If the number of columns of the two datasets is different, you can use the merge method in the pandas library to merge them according to the same key.

[0205] data1=pd.read_csv('data1.csv')

[0206] data2=pd.read_excel('data2.xlsx')

[0207] # Rename the columns

[0208] data2=data2.rename(columns={'old_column_name':'new_column_name'})

[0209] #Merge by the same key

[0210] merged_data=pd.merge(data1,data2,on='key_column')

[0211] For datasets of the same format, you can use the concat() function of the Pandas library to merge multiple DataFrames by columns or rows. For example, merge two DataFrames by columns:

[0212] import pandas as pd

[0213] df1=pd.DataFrame({'A':[1,2,3],'B':[4,5,6]})

[0214] df2=pd.DataFrame({'C':[7,8,9],'D':[10,11,12]})

[0215] df_combined=pd.concat([df1,df2],axis=1)

[0216] Embodiment 2:

[0217] like Figure 3 As shown, accordingly, embodiment 2 provides a meteorological data cleaning system based on the power environment, including an acquisition module 10, an image generation module 20, an abnormal value processing module 30, a missing value processing module 40 and a de-duplication module 50;

[0218] Wherein, the acquisition module 10 is used to acquire the original meteorological data set;

[0219] The image generation module 20 is used to generate a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set;

[0220] The outlier processing module 30 is used to determine whether there are outliers in the original meteorological data set according to the data distribution image. If there are outliers, a preset outlier processing method is used to process the outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method or an autoencoder method;

[0221] The missing value processing module 40 is used to detect missing values ​​in the first meteorological data set, and process the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method;

[0222] The deduplication module 50 is used to perform deduplication processing on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

[0223] In a possible implementation, the outlier processing module 30 uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the Z-score comparison method, including:

[0224] Calculate the mean and standard deviation of the original meteorological data set;

[0225] Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation;

[0226] Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point;

[0227] When the retrieval is completed, the remaining data points are the first meteorological data set.

[0228] In a possible implementation, the outlier processing module 30 uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the quantile detection method, including:

[0229] Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set;

[0230] Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range;

[0231] Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set;

[0232] When the retrieval is completed, the first meteorological data set is obtained.

[0233] In a possible implementation, the outlier processing module 30 uses a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the autoencoder method, including:

[0234] Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model;

[0235] The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

[0236] Further, such as Figure 4 As shown, the missing value processing module 40 includes a detection unit 401, a deletion unit 402, a filling unit 403 and an interpolation unit 404;

[0237] The detection unit 401 is used to detect missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score;

[0238] The deletion unit 402 is used to process the missing values ​​in the first meteorological data set using a deletion method when the missing value ratio is less than a first preset value, specifically: directly deleting the data point where the missing value is located;

[0239] The filling unit 403 is used to process the missing values ​​in the first meteorological data set using a constant filling method when the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, specifically: fill the missing values ​​with the average value, median or mode of the same row or column as the missing value;

[0240] The interpolation unit 404 is used to process the missing values ​​in the first meteorological data set using an interpolation method when the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value. Specifically, the values ​​of the missing values ​​are predicted based on the non-missing value data in the first meteorological data set and filled in accordingly. The interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

[0241] Furthermore, the machine learning interpolation method includes:

[0242] Use historical meteorological data to train machine learning models to obtain meteorological data prediction models;

[0243] The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set.

[0244] The embodiment of the present invention provides a meteorological data cleaning system based on the power environment. First, a data distribution image is established for the original meteorological data set, which can more intuitively display the distribution status of the original meteorological data and quickly find abnormal data in the original meteorological data set. Further, when it is found that there are abnormal values ​​in the original meteorological data, the abnormal values ​​are processed using a preset abnormal value processing method to improve the accuracy of the meteorological data and obtain a first meteorological data set. Then, the missing values ​​in the first meteorological data set are detected, and the missing values ​​are processed using a deletion method, an interpolation method, or a constant filling method according to the detection result of the missing values ​​to obtain a second meteorological data set, wherein the deletion method directly deletes the data points where the missing values ​​are located, and the interpolation method and the filling method fill the missing values. By detecting and processing the missing values ​​of the first meteorological data set, the consistency and integrity of each data in the data set are ensured. Finally, the second meteorological data set is deduplicated to remove redundant and repeated data, further streamline the second meteorological data set, and further improve the quality and reliability of the data.

[0245] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.

[0246] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A meteorological data cleaning method based on power environment, characterized in that: include: Get the original meteorological dataset; Generate a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set; Determine whether the original meteorological data set has an outlier according to the data distribution image; if an outlier exists, use a preset outlier processing method to process the outlier in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method, or an autoencoder method; Detect missing values ​​in the first meteorological data set, and process the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method; Deduplication processing is performed on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

2. A meteorological data cleaning method based on power environment as claimed in claim 1, characterized in that: The method of using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set includes: Calculate the mean and standard deviation of the original meteorological data set; Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation; Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point; When the retrieval is completed, the remaining data points are the first meteorological data set.

3. A meteorological data cleaning method based on power environment as claimed in claim 1, characterized in that: The method of using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set includes: Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set; Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range; Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set; When the retrieval is completed, the first meteorological data set is obtained.

4. A meteorological data cleaning method based on power environment as claimed in claim 1, characterized in that: The method of using a preset outlier processing method to process outliers in the original meteorological data set to obtain a first meteorological data set includes: Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model; The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

5. A meteorological data cleaning method based on power environment as claimed in claim 1, characterized in that: The detecting missing values ​​in the first meteorological data set, and processing the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set includes: Detecting missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score; When the missing value ratio is less than a first preset value, a deletion method is used to process the missing values ​​in the first meteorological data set, specifically: directly deleting the data points where the missing values ​​are located; When the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, a constant filling method is used to process the missing values ​​in the first meteorological data set, specifically: the missing values ​​are filled with the average, median or mode of the same row or column as the missing value; When the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value, an interpolation method is used to process the missing values ​​in the first meteorological data set, specifically: predicting the values ​​of the missing values ​​based on the non-missing value data in the first meteorological data set and filling them accordingly, the interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

6. A meteorological data cleaning method based on power environment as claimed in claim 5, characterized in that: The machine learning interpolation method comprises: Use historical meteorological data to train machine learning models to obtain meteorological data prediction models; The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set.

7. A meteorological data cleaning system based on power environment, characterized in that: It includes an acquisition module, an image generation module, an outlier processing module, a missing value processing module and a de-duplication module; Wherein, the acquisition module is used to acquire the original meteorological data set; The image generation module is used to generate a corresponding data distribution image according to the distribution status of each data point in the original meteorological data set; The outlier processing module is used to determine whether there are outliers in the original meteorological data set according to the data distribution image, and if there are outliers, use a preset outlier processing method to process the outliers in the original meteorological data set to obtain a first meteorological data set, wherein the outlier processing method is a Z-score comparison method, a quantile detection method or an autoencoder method; The missing value processing module is used to detect missing values ​​in the first meteorological data set, and process the missing values ​​in the first meteorological data set using a corresponding missing value processing method according to the detection result to obtain a second meteorological data set, wherein the missing value processing method includes a deletion method, an interpolation method, and a constant filling method; The deduplication module is used to perform deduplication processing on the second meteorological data set to obtain a meteorological data set that has completed data cleaning.

8. A meteorological data cleaning system based on electric power environment as claimed in claim 7, characterized in that: The outlier processing module processes the outliers in the original meteorological data set using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the Z-score comparison method, including: Calculate the mean and standard deviation of the original meteorological data set; Calculating the distance between each data point in the original meteorological data set and the mean value and standard deviation; Retrieving each data point in the original meteorological data set in turn, and when the distance between a first data point and the mean value and the standard deviation exceeds a preset threshold, deleting the first data point; When the retrieval is completed, the remaining data points are the first meteorological data set.

9. A meteorological data cleaning system based on electric power environment as claimed in claim 7, characterized in that: The outlier processing module processes the outliers in the original meteorological data set using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the quantile detection method, including: Calculate the 25% quantile and the 75% quantile of the original meteorological data set, and then calculate the interquartile range of the original meteorological data set; Determine the upper and lower numerical thresholds of each data point in the original meteorological data set according to the interquartile range; Searching each data point in the original meteorological data set in turn, when the value of a first data point is greater than the upper numerical threshold or less than the lower numerical threshold, replacing the value of the first data point with the median of the original meteorological data set; When the retrieval is completed, the first meteorological data set is obtained.

10. A meteorological data cleaning system based on electric power environment as claimed in claim 7, characterized in that: The outlier processing module processes outliers in the original meteorological data set using a preset outlier processing method to obtain a first meteorological data set, wherein the outliers in the original meteorological data set are processed using the autoencoder method, including: Constructing an initial autoencoder model based on a deep learning algorithm and using historical meteorological data with outlier marks to train the initial autoencoder to obtain a trained autoencoder model; The original meteorological data set is input into the autoencoder model so that the encoder model compresses and encodes the original meteorological data set and then decodes and restores it. During the decoding and restoration process, outliers in the original meteorological data set are detected and deleted.

11. A meteorological data cleaning system based on electric power environment as claimed in claim 7, characterized in that: The missing value processing module includes a detection unit, a deletion unit, a filling unit and an interpolation unit; The detection unit is used to detect missing values ​​in the first meteorological data set to obtain a detection result, wherein the detection result includes a missing value ratio and a missing value importance score; The deletion unit is used to process the missing values ​​in the first meteorological data set using a deletion method when the missing value ratio is less than a first preset value, specifically: directly deleting the data point where the missing value is located; The filling unit is used to process the missing values ​​in the first meteorological data set using a constant filling method when the missing value ratio is greater than a first preset value and the missing value importance score is less than a second preset value, specifically: fill the missing values ​​with the average value, median or mode of the same row or column as the missing value; The interpolation unit is used to process the missing values ​​in the first meteorological data set using an interpolation method when the missing value ratio is greater than a first preset value and the missing value importance score is greater than a second preset value. Specifically, the numerical value of the missing value is predicted based on the non-missing value data in the first meteorological data set and filled in accordingly. The interpolation method includes linear interpolation, polynomial interpolation, spline interpolation or machine learning interpolation.

12. A meteorological data cleaning system based on electric power environment as claimed in claim 11, characterized in that: The machine learning interpolation method comprises: Use historical meteorological data to train machine learning models to obtain meteorological data prediction models; The first meteorological data set is input into the meteorological data prediction model so that the meteorological data prediction model predicts the data change trend of the first meteorological data set, and generates corresponding prediction values ​​for filling according to the positions of each missing value in the first meteorological data set.

Citation Information

Patent Citations

  • Meteorological data knowledge discovery method and system

    CN116860824A

  • Regression analysis method for predicting air traffic flow management delay value based on deep learning

    CN117334087A

  • Meteorological environment data analysis method and system

    CN117493798A