A dimensionality reduction method for high-dimensional weather forecast data based on big data platform
By integrating, preprocessing and dimensionality reduction on the big data platform, the problem of low processing efficiency of high-dimensional weather forecast data is solved, the power grid disaster prevention capabilities and data analysis accuracy are improved, and efficient data dimensionality reduction and dimensionality reduction are achieved. Forecast accuracy.
Patent Information
- Application Number
- CN201910398824.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-05-14
AI Technical Summary
When processing high-dimensional weather forecast data in the prior art, linear dimensionality reduction methods are poor in the nonlinear structural data, resulting in low data processing efficiency and difficult to meet the precise needs of power grid disaster prevention and resistance.
Using a method based on the big data platform, we use multi-source meteorological data fusion, preprocessing, dimension screening and dimensionality reduction to obtain feature vectors for dimensionality reduction, reduce the number of features, and improve data utilization and forecasting accuracy.
It improves the accuracy and availability of numerical weather forecast data, enhances the disaster prevention and mitigation capabilities of the power grid, simplifies the preprocessing and storage of meteorological data, and realizes more efficient data analysis.
Smart Images

Figure CN111948736B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of numerical weather forecasting in the electric power industry, and specifically relates to a high-dimensional weather forecast data dimensionality reduction method based on a big data platform. Background Art
[0002] With the increasing frequency of meteorological disasters, especially high temperatures, floods, ice and other serious hazards to the safety of power grids, numerical weather forecasting technology has gradually been applied to the power industry. A large number of numerical weather forecast data development and research have been carried out in the field of new energy at home and abroad. In the United States, Windlogic, 3Tier, AWStruewind and other companies have developed weather forecast systems based on real-time data and historical data of meteorological observations, coarse and fine grid numerical weather original data. The German Weather Service Center provides regional and global numerical weather forecast data with resolutions of 7km×7km and 60km×60km respectively. The spatial resolution of the regional forecast model of the French Meteorological Center is 9.5km×9.5km. Sweden, Denmark, Norway, Spain, Iceland, Ireland and other countries obtain numerical weather forecast products based on the HIRLAM model with a resolution of 10-50km. The European Center for Medium-Term Forecasts provides 40km×40km numerical weather forecasts. Based on the BJ-RUC rapid cycle update model system, the China Meteorological Administration obtains 15-minute numerical weather forecast data with a horizontal spatial resolution of 9km×9km for the next 72 hours through the multiple nesting technology of the mesoscale forecast model. China Electric Power Research Institute has established a numerical forecast model system for business operation of wind power forecasting, and releases numerical weather forecast results twice a day at 03:00 and 15:00 Beijing time. Each forecast lasts for 72 hours, with a horizontal spatial resolution of 5km×5km and a temporal resolution of 15min. It covers major wind power and photovoltaic power generation development areas such as Northeast China, North China, and Northwest China. The forecast results include various meteorological elements such as wind direction, wind speed, air pressure, temperature, humidity, near-ground long / shortwave radiation flux at different altitudes that directly affect wind power / photovoltaic power generation capacity.
[0003] The current numerical weather forecast data has the characteristics of multiple meteorological factors, high spatiotemporal resolution and long forecast time. The explosive increase of wide-area spatiotemporal big data is prone to cause problems such as "dimensionality disaster" and modeling high-dimensional data. Dimensionality reduction method is a typical data processing technology used to overcome the "dimensionality disaster" and model high-dimensional data, and is one of the effective means to solve this problem. It can explore different styles of intrinsic low-dimensional manifolds embedded in high-dimensional data space by analyzing discrete data sets, and seek the essential regularity of things. Consider such a data processing system, such as the data processing system of numerical weather forecast, which requires real-valued vectors as input. This system can only work effectively when the dimension of each input data is not too high. Then when the vector dimension reaches a certain level, certain measures must be taken to enable the system to effectively process these data and work normally.
[0004] Certain achievements have been made in the dimensionality reduction of high-dimensional weather forecast data at home and abroad. The classification of common dimensionality reduction algorithms is based on different data processing methods. Data dimensionality reduction algorithms can be divided into linear data dimensionality reduction methods and nonlinear data dimensionality reduction methods. Linear data dimensionality reduction algorithms mainly include principal component analysis (PCA), linear discriminant analysis (LDA), multidimensional scaling (MDS), etc., which are widely used by researchers because of their simple methods and efficient execution. Nonlinear dimensionality reduction methods include multidimensional scaling, ISOMAP method, LLE method, and Laplacian Eigenmap method. The nonlinear dimensionality reduction process of a data set is equivalent to two processes that are both interrelated and independent of each other.
[0005] However, although there is a certain research basis for weather forecast data dimensionality reduction, there are still the following problems: linear data dimensionality reduction methods are all based on an assumption, that is, they assume that the data set under study is embedded in a global linear structure, so when faced with a data set with a nonlinear structure, this type of method is not very effective. Summary of the invention
[0006] In order to overcome the technical bottleneck, the present invention proposes a high-dimensional weather forecast data dimensionality reduction method based on a big data platform, which is used to improve the effective utilization of data and realize the establishment and application of numerical weather forecasting in the power industry, including the following steps:
[0007] Step 1: Build a big data platform to obtain multi-source meteorological data;
[0008] Step 2, fusing the multi-source meteorological data through the big data platform to form numerical weather forecast data;
[0009] Step 3, performing data preprocessing on the numerical weather forecast data;
[0010] Step 4, performing dimension screening of the numerical weather forecast data;
[0011] Step 5, reducing the dimension of the high-dimensional weather forecast data.
[0012] The beneficial effects of the present invention include: first, the disaster prevention and disaster resistance capabilities of the power grid, in addition to the adaptability of the power grid itself, are more dependent on accurate meteorological monitoring data and forecast data. With the acceleration of the construction process of numerical weather forecasting applied to new energy stations, power big data has also emerged. As a result, the amount of data continues to increase, the frequency of data anomalies increases, and the data format is changeable. Compared with the existing integrated system or method, the present invention greatly improves the availability and accuracy of data, improves the forecast accuracy of the numerical forecast model, and thus improves the disaster prevention and mitigation capabilities of the power grid. Secondly, the special fusion method for multi-source meteorological data, according to the correlation matching of the multi-source meteorological data itself, analyzes the meteorological data more comprehensively and deeply, and greatly improves the quality of the fused numerical weather forecast data. Thirdly, the multi-source meteorological data can be integrated with the big data platform of the power industry, and the power big data platform can be combined, which greatly reduces the difficulties of meteorological data preprocessing, calculation and storage. Finally, the method for reducing the dimension of high-dimensional weather forecast data and the method for obtaining the covariance matrix therein greatly reduce the number of features and achieve the purpose of multiple dimensionality reduction. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 The present invention is a flowchart of a method for reducing the dimensionality of high-dimensional weather forecast data based on a big data platform in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] In order to better understand the present invention, the method and system of the present invention are further described below with reference to the description of the embodiments in combination with the accompanying drawings.
[0015] In order to fully understand the present invention, many specific details are mentioned in the following detailed description. However, it should be understood by those skilled in the art that the present invention can be implemented without these specific details. In the embodiments, well-known methods, processes, components are not described in detail to avoid making the embodiments unnecessarily cumbersome.
[0016] See also Figure 1 As shown, the present invention provides a high-dimensional weather forecast data dimensionality reduction method based on a big data platform, comprising the following steps:
[0017] Step 1: Build a big data platform to obtain multi-source meteorological data;
[0018] Step 2, fusing the multi-source meteorological data through the big data platform to form numerical weather forecast data;
[0019] Step 3, performing data preprocessing on the numerical weather forecast data;
[0020] Step 4, performing dimension screening of the numerical weather forecast data;
[0021] Step 5, reducing the dimension of the high-dimensional weather forecast data.
[0022] Preferably, the step 2, fusing the multi-source meteorological data through the big data platform to form numerical weather forecast data, specifically includes:
[0023] Step 2-1, assigning initial weights to each type of meteorological data;
[0024] Step 2-2, sampling each type of meteorological data sample based on the distribution of weights;
[0025] Step 2-3, iteratively train weak classifiers in parallel for each type of meteorological data;
[0026] Step 2-4, selecting the weak classifier corresponding to the sample with the lowest training error rate, and calculating the combined weight value associated with the selected weak classifier according to the lowest training error rate of the iteration;
[0027] Step 2-5, assigning updated weights to each type of meteorological data, and the updated weight distribution corresponds to the iterative function with the lowest training error rate;
[0028] Step 2-6, repeating steps 2-2 to 2-5 for a predetermined number of iterations;
[0029] Step 2-7, forming a final classifier, which is the sum of the selected weak classifiers weighted by the updated combination weight value at each iteration.
[0030] Preferably, the step 3, performing data preprocessing on the numerical weather forecast data, specifically includes: converting the numerical weather forecast data into ASCII text data and performing data quality control.
[0031] Preferably, the numerical weather forecast data is converted into ASCII text data and data quality control is performed, specifically including:
[0032] Step 3-1, identify the format of numerical weather forecast data. If the data format is ASCII text data, directly import it into the database for data quality control. Otherwise, convert it.
[0033] Step 3-2, convert the data into ASCII text data;
[0034] Step 3-3, conduct data quality control from the aspects of logic, time continuity, and spatial consistency to achieve automatic storage.
[0035] Preferably, in step 4, dimensional screening of the numerical weather forecast data is performed. The larger the information entropy value of a dimension is, the greater the amount of information of the original data it contains, and it is a feature that should be retained; if the information entropy of a certain dimension is lower, it means that the amount of data information contained in the dimension is less, and it is a feature that should be eliminated.
[0036] The step 4 specifically includes:
[0037] Step 4-1, calculate each dimension a of the high-dimensional weather forecast data i The information entropy value H(a i );
[0038] Step 4-2, preset the information entropy threshold δ,
[0039] Step 4-3, if each dimension a of the high-dimensional weather forecast data i The information entropy value H(a i ) is greater than δ, retaining dimension a i ; Otherwise, remove dimension a i .
[0040] Preferably, the step 5 of reducing the dimension of the high-dimensional weather forecast data specifically includes:
[0041] Step 5-1: Subtract the mean of each dimension of the high-dimensional weather forecast data from the dimension, perform sample matrix centering, and obtain a high-dimensional weather forecast data centering matrix X m×n ;
[0042] Step 5-2, calculate the dimension covariance matrix;
[0043] Step 5-3, calculating the eigenvectors and eigenvalues of the dimensional covariance matrix;
[0044] Step 5-4: Select the k eigenvectors corresponding to the largest k eigenvalues as column vectors to form the eigenvector matrix V n×k ; Among them, the largest k eigenvalues represent the number of principal components, and the number k is determined by the eigenvalue contribution rate:
[0045] Step 5-5, calculating the result Y=V of reducing the dimension of the high-dimensional weather forecast data n×k T ×X m×n ;
[0046] Wherein, the step 5-2, calculating the covariance matrix of the dimension, specifically includes:
[0047] Step 5-2-1, selecting multiple dimensional sample subsets from the high-dimensional weather forecast data;
[0048] Step 5-2-2, extracting a d-dimensional feature vector for each sample in each sample subset, wherein the feature vector includes an index pointing to the corresponding sample and the sample attribute;
[0049] Step 5-2-3, combining the feature vectors of each sample subset into a d×d feature vector covariance matrix as the descriptor of the corresponding sample subset;
[0050] The eigenvector covariance matrix
[0051] Among them, p represents the number of samples in the subset, V represents the feature vector, μ is the feature mean, and T represents the transpose.
[0052] Step 5-2-4, determining the pairwise distance value between each pair of eigenvector covariance matrices to measure the similarity between sample subsets;
[0053] The distance value
[0054] Among them, λ i (C1, C2), i = 1, ... p are the generalized eigenvalues of the two covariance matrices C1 and C2;
[0055] Step 5-2-5, define the distance threshold and screen the pairwise covariance distances that meet the conditions;
[0056] Step 5-2-6, construct a d×d dimensional distance matrix based on the pairwise covariance distance values;
[0057] Step 5-2-7, and use the distance matrix as the high-dimensional weather forecast data dimension covariance matrix.
[0058] Preferably, wherein, the step 5-2, calculating the dimensional covariance matrix, further comprises:
[0059] Step 5-2-8: Configure N mapping functions Mapper;
[0060] Step 5-2-9: Configure a reduction function Reducer, the input of which is the output result of each Mapper;
[0061] Step 5-2-10: Substitute the summary results of the Reducer into the calculation of the covariance matrix of the high-dimensional weather forecast data dimension.
[0062] Compared with the prior art, the present invention has the following significant advantages: first, the disaster prevention and disaster resistance ability of the power grid depends more on accurate meteorological monitoring data and forecast data in addition to the adaptability of the power grid itself. With the acceleration of the construction process of numerical weather forecasting applied to new energy stations, power big data has also emerged. As a result, the amount of data continues to increase, the frequency of data anomalies increases, and the data format is changeable. Compared with the existing integrated system or method, the present invention greatly improves the availability and accuracy of data, improves the forecast accuracy of the numerical forecast model, and thus improves the disaster prevention and mitigation ability of the power grid. Secondly, the special fusion method for multi-source meteorological data, according to the correlation matching of the multi-source meteorological data itself, analyzes the meteorological data more comprehensively and deeply, and greatly improves the quality of the fused numerical weather forecast data. Thirdly, the multi-source meteorological data can be integrated with the big data platform of the power industry, and the power big data platform can be combined, which greatly reduces the difficulties of meteorological data preprocessing, calculation and storage. Finally, the method for reducing the dimension of high-dimensional weather forecast data and the method for obtaining the covariance matrix therein greatly reduce the number of features and achieve the purpose of multiple dimensionality reduction.
[0063] Only preferred embodiments of the present invention are described herein, but are not intended to limit the scope, applicability and configuration of the present invention. On the contrary, the detailed description of the embodiments enables those skilled in the art to implement. It should be understood that some details may be appropriately changed and modified without departing from the spirit and scope of the present invention as defined in the appended claims.
Claims
1. A high-dimensional weather forecast data dimensionality reduction method based on a big data platform, characterized in that: The following steps are involved: Step 1: Build a big data platform to obtain multi-source meteorological data; Step 2, fusing the multi-source meteorological data through the big data platform to form numerical weather forecast data; Step 3, performing data preprocessing on the numerical weather forecast data; Step 4, performing dimension screening of the numerical weather forecast data; Step 5, reducing the dimension of the high-dimensional weather forecast data; The step 5, reducing the dimension of the high-dimensional weather forecast data, specifically includes: Step 5-1: Subtract the mean of each dimension of the high-dimensional weather forecast data from the dimension, perform sample matrix centering, and obtain a high-dimensional weather forecast data centering matrix X m×n ; Step 5-2, calculate the dimension covariance matrix; Step 5-3, calculating the eigenvectors and eigenvalues of the dimensional covariance matrix; Step 5-4: Select the k eigenvectors corresponding to the largest k eigenvalues as column vectors to form the eigenvector matrix V n×k ; Among them, the largest k eigenvalues represent the number of principal components, and the number k is determined by the eigenvalue contribution rate; Step 5-5, calculating the result Y=V of reducing the dimension of the high-dimensional weather forecast data n×k T ×X m×n ; Wherein, the step 5-2, calculating the covariance matrix of the dimension, specifically includes: Step 5-2-1, selecting multiple dimensional sample subsets from the high-dimensional weather forecast data; Step 5-2-2, extracting a d-dimensional feature vector for each sample in each sample subset, wherein the feature vector includes an index pointing to the corresponding sample and the sample attribute; Step 5-2-3, combining the feature vectors of each sample subset into a d×d feature vector covariance matrix as the descriptor of the corresponding sample subset; The eigenvector covariance matrix Where p represents the number of samples in the subset, V represents the feature vector, μ is the feature mean, and T represents the transpose; Step 5-2-4, determining the pairwise distance value between each pair of eigenvector covariance matrices to measure the similarity between sample subsets; The distance value Among them, λ i (C1, C2), i = 1, ... p are the generalized eigenvalues of the two covariance matrices C1 and C2; Step 5-2-5, define the distance threshold and screen the pairwise covariance distances that meet the conditions; Step 5-2-6, construct a d×d dimensional distance matrix based on the pairwise covariance distance values; Step 5-2-7, and use the distance matrix as the high-dimensional weather forecast data dimension covariance matrix.
2. The method according to claim 1, wherein: The step 2, fusing the multi-source meteorological data through the big data platform to form numerical weather forecast data, specifically includes: Step 2-1, assigning initial weights to each type of meteorological data; Step 2-2, sampling each type of meteorological data sample based on the distribution of weights; Step 2-3, iteratively train weak classifiers in parallel for each type of meteorological data; Step 2-4, selecting the weak classifier corresponding to the sample with the lowest training error rate, and calculating the combined weight value associated with the selected weak classifier according to the lowest training error rate of the iteration; Step 2-5, assigning updated weights to each type of meteorological data, and the updated weight distribution corresponds to the iterative function with the lowest training error rate; Step 2-6, repeating steps 2-2 to 2-5 for a predetermined number of iterations; Step 2-7, forming a final classifier, which is the sum of the selected weak classifiers weighted by the updated combination weight value at each iteration.
3. The method according to claim 1, wherein: The step 3 is to perform data preprocessing on the numerical weather forecast data, specifically including: converting the numerical weather forecast data into ASCII text data and performing data quality control.
4. The method according to claim 3, wherein: The numerical weather forecast data is converted into ASCII text data and data quality control is performed, specifically including: Step 3-1, identify the format of numerical weather forecast data. If the data format is ASCII text format data, directly import it into the database for data quality control. Otherwise, convert it; Step 3-2, convert the data into ASCII text data; Step 3-3, conduct data quality control from the aspects of logic, time continuity, and spatial consistency to achieve automatic storage.
5. The method according to claim 1, wherein: The step 4 is to perform dimension screening of the numerical weather forecast data. The larger the information entropy value of a dimension is, the larger the amount of information of the original data it contains is, and the feature should be retained; If the information entropy of a dimension is lower, it means that the dimension contains less data information and is a feature that should be eliminated; The step 4 specifically includes: Step 4-1, calculate each dimension a of the high-dimensional weather forecast data i The information entropy value H(a i ); Step 4-2, presetting the information entropy threshold δ; Step 4-3, if each dimension a of the high-dimensional weather forecast data i The information entropy value H(a i ) is greater than δ, retaining dimension a i ; Otherwise, remove dimension a i .
6. The method according to claim 1, wherein: Among them, The step 5-2, calculating the dimensional covariance matrix, also includes: Step 5-2-8, configure N mapping functions Mapper; Step 5-2-9, configure a reduction function Reducer, the input of the Reducer is the output result of each Mapper; Step 5-2-10, substitute the summary result of the Reducer into the calculation of the covariance matrix of the high-dimensional weather forecast data dimension.
