Data sampling period integration method based on multimode fusion
Through the multi-mode fusion machine learning model and timestamp integration method, data integration difficulties caused by the sampling period of different sensors are solved, and higher data integration accuracy and reflection of the true changes of feature data are achieved.
Patent Information
- Application Number
- CN202311250729.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-03-28
AI Technical Summary
The sampling period differences between different sensors make it difficult to integrate data, and the accuracy of the existing filling methods is low, which cannot reflect the true changes in feature data.
Using a multi-mode fusion-based data sampling period integration method, the relationship between time characteristics and sensor data is mined using the timestamp as the benchmark through multiple machine learning models (such as linear regression, GBDT regression, ARIMA), and then the missing data is filled with upsampling.
It improves the accuracy of data integration, can more accurately reflect the relationship between sensor data and time, and reduces the risk of overfitting during training.
Smart Images
Figure CN119720067A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of power system control, and in particular to a data sampling period integration method based on multi-mode fusion. Background Art
[0002] With the 3060 carbon neutrality goal proposed, the new power system faces the problem of large-scale access to new energy. Among them, wind power and photovoltaics are the main forms of new energy, and energy storage power stations are usually built simultaneously with wind power and photovoltaics as necessary supporting facilities. Since the output of new energy is greatly disturbed by external factors, especially meteorological factors, national standards and specifications require accurate prediction of output. In this case, in order to achieve higher prediction accuracy, it is necessary to collect data from many different sensors. Different sensors and even sensors of different brands often have differences in data sampling cycles. Some use hours as the sampling cycle, while others use 5 minutes as the sampling cycle, which makes it difficult for information systems to integrate data. If the sensor with the smallest sampling cycle is used as the integration reference, a large number of missing values will appear in the data of other sensors with larger sampling cycles. In this case, the sampling timestamp of a single piece of data often becomes the only available and absolutely accurate quantifiable reference feature.
[0003] In the scenario of new energy data prediction, the most commonly used and most studied is power station output prediction, which generally uses machine learning or deep learning algorithms to predict the output of power stations based on input feature data. When designing machine learning and deep learning algorithms, since most of the input features of new energy and energy storage power stations are meteorological features such as wind speed, irradiance, temperature, air pressure, humidity, etc., the values of these features come from different sensors, and the sampling periods may be different. In this case, the only absolutely accurate and quantitative feature is the data acquisition timestamp, which can reflect the changes of other features and the required prediction values over time. Therefore, a variety of machine learning models can be used to learn time features and data from training data sets and test data sets, explore the relationship between time features and sensor data, and use the data timestamp of the sensor with the smallest sampling period as the benchmark to fill in the missing values caused by sampling period differences, so as to facilitate the information system to integrate sensor data with multiple sampling periods.
[0004] The solutions of the prior art include: using the data timestamp of the sensor with the smallest sampling period as a reference to fill the missing data values of other sensors due to sampling period differences with the average value. Alternatively, using the data timestamp of the sensor with the smallest sampling period as a reference to fill the missing data values of other sensors due to sampling period differences with the median value. Alternatively, using the data timestamp of the sensor with the smallest sampling period as a reference to fill the missing data values of other sensors due to sampling period differences with the data of the previous timestamp. Alternatively, using the data timestamp of the sensor with the largest sampling period as a reference to discard the redundant data caused by sampling period differences of other sensors.
[0005] Under existing technical methods, the average value, median or data of the previous time stamp is usually used for filling. However, since various measurement data of new energy power stations vary greatly with weather factors, these filling methods have low accuracy and cannot reflect the actual changes in the corresponding feature data. Summary of the invention
[0006] The purpose of this application is to provide a data sampling period integration method based on multi-mode fusion to solve the problem of integrating sensor data with different sampling periods.
[0007] The present application discloses a data sampling period integration method based on multi-mode fusion, including a data preprocessing part, a model training part, and a missing value filling part; wherein the data preprocessing part includes the following steps:
[0008] (S1) receiving data from a plurality of sensors, wherein the plurality of sensors have different sampling periods;
[0009] (S2) selecting the data of the sensor with the smallest sampling period and integrating the data based on the timestamp;
[0010] (S3) converting the timestamp into a decimal value;
[0011] (S4) filtering out sensor data that needs to be filled with missing values based on the sensor as a unit;
[0012] The model training part includes: a training data set, a cross-validation data set, and a test data set. The data preprocessed by the data is based on a multi-mode fusion method to establish the relationship between the time feature and the sensor data;
[0013] In the missing value filling part, the missing values obtained in (S4) are filled based on the data timestamp of the sensor with the smallest sampling period.
[0014] In a preferred embodiment, the data after the data preprocessing part is transmitted to the model training part, and the total data is divided into three parts:
[0015] The first part is used as a training data set to build a linear regression model through a linear regression algorithm;
[0016] The second part is used as a cross-validation data set to establish a GBDT regression model through the GBDT regression algorithm;
[0017] The third part is used as a test data set to establish an ARIMA model through the ARIMA algorithm.
[0018] In a preferred embodiment, the first part includes 60% of the total data; and / or
[0019] The second portion comprises 20% of the total data; and / or
[0020] The third part comprises 20% of the total data.
[0021] In a preferred embodiment, in the missing value filling part, upsampling and interpolation are performed on the data integrated based on the linear regression model, the GBDT model, and the ARIMA model, and
[0022] The upsampling interpolation results are averaged as the final result to fill in the missing values.
[0023] In a preferred example, the linear regression algorithm and the GBDT regression algorithm use date values and / or time values as independent variables and sensor data as dependent variables.
[0024] In a preferred example, the data from the multiple sensors are selected from any one of the following: wind speed, irradiance, temperature, air pressure, and humidity.
[0025] In a preferred example, the sampling period of the sensor ranges from 1 hour to 10 hours.
[0026] In a preferred example, the method further includes: using the data timestamp of the sensor with the largest sampling period as a reference to discard missing data values of other sensors caused by differences in sampling periods.
[0027] This application has at least the following technical effects:
[0028] A sensor data sampling period integration method based on multi-mode fusion is adopted to solve the problem of integrating sensor data with different sampling periods. Through multiple sets of machine learning models, the relationship between measurement data and time is mined based on timestamps, and then the missing data is filled by upsampling.
[0029] 1. Since most of the input features for output prediction of new energy and energy storage power stations are meteorological features such as wind speed, irradiance, temperature, air pressure, humidity, etc., the values of these features come from different sensors and the sampling periods may be different. This application adopts a sensor data sampling period integration method based on multi-mode fusion to solve the problem of integrating sensor data with different sampling periods.
[0030] 2. Under the existing technical methods, the average value, median or the data of the previous timestamp are usually used for filling. However, since the various measurement data of new energy power stations vary greatly with weather factors, the accuracy of these filling methods is low and cannot reflect the actual changes of the corresponding feature data. This application adopts a sensor data sampling cycle integration method based on multi-mode fusion, through multiple sets of machine learning models based on timestamps, to mine the relationship between measurement data and time, and then fill in the missing data by upsampling.
[0031] 3. This application adopts a fusion method of multiple machine learning models to learn time features and data from training data sets and test data sets, explore the relationship between time features and sensor data, and fill in missing values caused by sampling period differences based on the data timestamp of the sensor with the smallest sampling period. The multi-mode fusion method can reduce overfitting in the training process.
[0032] The specification of the present application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of the present application (i.e., technical solutions) are to be listed, the specification will be too long. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of the present application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions should all be regarded as having been recorded in this specification), unless the combination of such technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed, and features C and D are equivalent technical means that play the same role. Technically, only one can be used, and it is impossible to use them at the same time. Feature E can be combined with feature C technically. Then, the solution of A+B+C+D should not be regarded as having been recorded because it is technically infeasible, and the solution of A+B+C+E should be regarded as having been recorded. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a flowchart of the algorithm of the method according to the present application. DETAILED DESCRIPTION
[0034] After extensive and in-depth research, the inventors of this application provide a data sampling period integration method based on multi-mode fusion. This method adopts a fusion of multiple machine learning models to learn time features and data from training data sets and test data sets, explore the relationship between time features and sensor data, and fill in missing values caused by sampling period differences based on the data timestamp of the sensor with the smallest sampling period, so as to facilitate the information system to integrate sensor data with multiple sampling periods.
[0035] In the following description, many technical details are provided to help readers better understand the present application. However, those skilled in the art can understand that the technical solution claimed in the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.
[0036] Description of some concepts:
[0037] Missing value filling: fill in missing values in the data.
[0038] Upsampling: The opposite process to decimation is called interpolation or upsampling.
[0039] Linear regression: A statistical analysis method that uses regression analysis in mathematical statistics to determine the quantitative relationship of interdependence between two or more variables.
[0040] GBDT regression: Use a decision tree to fit the residual of the current learner at each step to obtain a new weak learner. Combine the decision trees at each step to get a strong learner. When this learner is used for regression, it is called GBDT regression.
[0041] ARIMA: Differential integrated moving average autoregressive model, also known as integrated moving average autoregressive model (moving can also be called sliding), is one of the time series prediction and analysis methods.
[0042] Supervised learning: The machine learning task of inferring a function from a labeled training dataset.
[0043] Training dataset: The portion of the dataset used for training.
[0044] Cross-validation dataset: The portion of the dataset used to tune model parameters.
[0045] Test dataset: The portion of the dataset used to verify the accuracy of the model.
[0046] The following is a brief description of some innovative aspects of the implementation methods of this application:
[0047] The data sampling period integration method based on multi-mode fusion of the present application is as follows Figure 1As shown in the figure, it is used for the perception layer of the energy big data platform. The method includes three parts: data preprocessing, model training, and missing value filling.
[0048] The data preprocessing part includes the following steps:
[0049] (S1) receiving data from multiple sensors with different sampling periods; in an optional embodiment, the sampling period of the sensor ranges from 1 hour to 10 hours. The data from the multiple sensors are selected from any one of the following: wind speed, irradiance, temperature, air pressure, and humidity.
[0050] (S2) selecting the data of the sensor with the smallest sampling period and integrating the data based on the timestamp;
[0051] (S3) converting the timestamp into a decimal value;
[0052] (S4) filtering out sensor data that needs to be filled with missing values based on the sensor as a unit;
[0053] The model training part includes: training data set, cross-validation data set, and test data set. The data after data preprocessing is transferred to the model training part, and the total data is divided into three parts:
[0054] The first part is used as a training data set, and a linear regression model is established through a linear regression algorithm. In an advantageous embodiment, the data of the training data set is 60% of the total data.
[0055] The second part is used as a cross-validation data set, and a GBDT regression model is established through the GBDT regression algorithm. In an advantageous embodiment, the data of the cross-validation data set accounts for 20% of the total data.
[0056] The third part is used as a test data set, and an ARIMA model is established through the ARIMA algorithm. In an advantageous embodiment, the data of the test data set accounts for 20% of the total data.
[0057] The data that has been preprocessed is based on multi-mode fusion to establish the relationship between time features and sensor data.
[0058] In the missing value filling part, the missing values obtained in the data preprocessing part are filled based on the data timestamp of the sensor with the smallest sampling period. In a preferred embodiment, the data integrated based on the linear regression model, the GBDT model, and the ARIMA model are upsampled and interpolated, and the upsampled and interpolated results are averaged as the final result for missing value filling.
[0059] In an optional implementation, the data timestamp of the sensor with the smallest sampling period is used as a reference to perform average filling on the data missing values of other sensors caused by differences in sampling periods.
[0060] In an optional implementation, the data timestamp of the sensor with the smallest sampling period is used as a reference to perform median filling on the missing data values of other sensors caused by differences in sampling periods.
[0061] In an optional implementation, the data timestamp of the sensor with the smallest sampling period is used as a reference to fill in the missing data values of other sensors due to the difference in sampling periods with zero values.
[0062] In an optional implementation, the data timestamp of the sensor with the largest sampling period is used as a reference to fill in the missing data values of other sensors due to differences in sampling periods with values of normal distribution.
[0063] In an optional implementation, the data timestamp of the sensor with the largest sampling period is used as a reference to discard the missing data values of other sensors caused by differences in sampling periods.
[0064] In order to better understand the technical solution of the present application, a specific example is provided below for illustration. The details listed in the example are mainly for ease of understanding and are not intended to limit the scope of protection of the present application.
[0065] Comparative Example
[0066] Take the output data table of a photovoltaic power station as an example, as shown in the following table:
[0067] Table 1
[0068] date time Sensor 1 data Sensor 2 data Sensor 3 data 1990-1-1 0:00 13846838 1990-1-1 1:00 11685907 1990-1-1 2:00 12741907 1990-1-1 3:00 13456907 1990-1-1 4:00 13283414 1990-1-1 5:00 13788923 15628 1990-1-1 6:00 13288907 1990-1-1 7:00 17454651 1990-1-1 8:00 11745838 1990-1-1 9:00 12712795 1990-1-1 10:00 13488970 13756 64258 1990-1-1 11:00 13288907 1990-1-1 12:00 12964331 1990-1-1 13:00 12177563 1990-1-1 14:00 11846291 … … … … …
[0069] Since the sampling periods of the three sensors measuring different data are different, the sampling period of sensor 1 is 60 minutes, the sampling period of sensor 2 is 5 hours, and the sampling period of sensor 3 is 10 hours. Therefore, when the data integration is performed based on the data timestamp of the sensor with the smallest sampling period, namely sensor 1, the data of other sensors will generate missing values. In the existing technical methods, the average value, median or the data of the previous timestamp are usually used for filling. However, since the various measurement data of the new energy power station vary greatly with weather factors, the accuracy of these filling methods is low and cannot reflect the actual changes of the corresponding feature data. The present invention adopts a sensor data sampling period integration method based on multi-mode fusion to solve the problem of integrating sensor data with different sampling periods. Through multiple sets of machine learning models, the relationship between measurement data and time is mined based on timestamps, and then the missing data is filled by upsampling.
[0070] Example
[0071] The technical solution of the present invention is also introduced by taking the output data table of a photovoltaic power station as an example. The original data table is shown in the following table:
[0072] Table 2
[0073] date time Sensor 1 data Sensor 2 data Sensor 3 data 1990-1-1 0:00 13846838 1990-1-1 1:00 11685907 1990-1-1 2:00 12741907 1990-1-1 3:00 13456907 1990-1-1 4:00 13283414 1990-1-1 5:00 13788923 15628 1990-1-1 6:00 13288907 1990-1-1 7:00 17454651 1990-1-1 8:00 11745838 1990-1-1 9:00 12712795 1990-1-1 10:00 13488970 13756 64258 1990-1-1 11:00 13288907 1990-1-1 12:00 12964331 1990-1-1 13:00 12177563 1990-1-1 14:00 11846291 … … … … …
[0074] First, convert the time features (including date and time) into decimal numerical data for machine learning model training. The converted data table is shown in the following table:
[0075] Table 3
[0076]
[0077]
[0078] The data set is split, with 60% of the total data used as the training data set, 20% of the total data used as the cross-validation data set, and 20% of the total data used as the test data set. On the basis of converting the time features (including date and time) into decimal numerical data, the sensor data that needs to be filled with missing values is split based on the sensor. As shown in the following table:
[0079] Table 4
[0080]
[0081] Table 5
[0082]
[0083] Three machine learning algorithms specifically used for prediction are used to conduct machine learning on time features and corresponding sensor data and establish data models. The three machine learning algorithms are linear regression, GBDT regression and ARIMA. Linear regression is a commonly used linear model, GBDT is a mainstream tree model, and ARIMA is a differential integrated moving average autoregressive model. The three algorithms with different principles can ensure the independence between the three models and improve the accuracy of upsampling. The reason for using three machine learning algorithms is to reduce the impact of overfitting in the modeling process through multi-mode fusion. During the training process, the linear regression algorithm and the GBDT regression algorithm use date (value) and time (value) as independent variables and sensor data as dependent variables; the ARIMA algorithm uses date and time as independent variables and sensor data as dependent variables.
[0084] A cross-validation data set was used to verify the accuracy of the output results of the three machine learning algorithms after training, and the parameters of the three machine learning algorithms were adjusted until all three algorithms were able to train regression models that met the requirements.
[0085] Three sets of trained machine learning models are used to upsample and interpolate the missing values of the corresponding sensors. In the interpolation process, the linear regression algorithm and GBDT regression algorithm use date (value) and time (value) as independent variables and sensor data as dependent variables; the ARIMA algorithm uses date and time as independent variables and sensor data as dependent variables. Finally, the upsampled interpolation results of the three sets of models are averaged as the final result to fill the missing values.
[0086] It should be noted that in the application documents of this patent, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the application documents of this patent, if it is mentioned that an action is performed according to an element, it means that the action is performed at least according to the element, which includes two situations: performing the action only according to the element, and performing the action according to the element and other elements. Expressions such as multiple, multiple, and multiple include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.
[0087] This specification includes combinations of the various embodiments described herein. Individual references to "one embodiment" or a particular embodiment, etc., do not necessarily refer to the same embodiment; however, these embodiments are not mutually exclusive unless indicated as mutually exclusive or clear to those skilled in the art. It should be noted that the word "or" is used in this specification in a non-exclusive sense unless the context clearly indicates or requires otherwise.
[0088] All documents mentioned in this application are considered to be included in the disclosure of this application as a whole, so that they can be used as the basis for modification when necessary. In addition, it should be understood that after reading the above disclosure of this application, those skilled in the art can make various changes or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.
Claims
1. A data sampling period integration method based on multi-mode fusion, characterized in that: It includes data preprocessing part, model training part and missing value filling part; wherein, the data preprocessing part includes the following steps: (S1) receiving data from a plurality of sensors, wherein the plurality of sensors have different sampling periods; (S2) selecting the data of the sensor with the smallest sampling period and integrating the data based on the timestamp; (S3) converting the timestamp into a decimal value; (S4) filtering out sensor data that needs to be filled with missing values based on the sensor as a unit; The model training part includes: a training data set, a cross-validation data set, and a test data set. The data preprocessed by the data is based on a multi-mode fusion method to establish the relationship between the time feature and the sensor data; In the missing value filling part, the missing values obtained in (S4) are filled based on the data timestamp of the sensor with the smallest sampling period.
2. The data sampling period integration method according to claim 1, characterized in that: The data after data preprocessing is transmitted to the model training part, and the total data is divided into three parts: The first part is used as a training data set to build a linear regression model through a linear regression algorithm; The second part is used as a cross-validation data set to establish a GBDT regression model through the GBDT regression algorithm; The third part is used as a test data set to establish an ARIMA model through the ARIMA algorithm.
3. The data sampling period integration method according to claim 2, characterized in that: The first part comprises 60% of the total data; and / or The second portion comprises 20% of the total data; and / or The third part comprises 20% of the total data.
4. The data sampling period integration method according to claim 3, characterized in that: In the missing value filling part, upsampling and interpolation are performed on the data based on the linear regression model, GBDT model, and ARIMA model integration, and The upsampling interpolation results are averaged as the final result to fill in the missing values.
5. The data sampling period integration method according to claim 2, characterized in that: The linear regression algorithm and the GBDT regression algorithm use date values and / or time values as independent variables and sensor data as dependent variables.
6. The data sampling period integration method according to claim 1, characterized in that: The data from the multiple sensors are selected from any one of the following: wind speed, irradiance, temperature, air pressure, and humidity.
7. The data sampling period integration method according to claim 1, characterized in that: The sampling period of the sensor ranges from 1 hour to 10 hours.
8. The data sampling period integration method according to claim 1, characterized in that: Also includes: The data timestamp of the sensor with the largest sampling period is used as a benchmark to discard the missing data values of other sensors due to differences in sampling periods.