Polynomial activation function and adaptive period fused time sequence prediction method

Abstract

The invention belongs to the technical field of power load prediction, and particularly relates to a polynomial activation function and adaptive period fused time sequence prediction method. The method comprises the following steps: generating an input power load sample sequence and a prediction target from power load data through a sliding window; constructing a piecewise polynomial activation function; constructing a double-branch feature extraction framework for periodic feature extraction and non-periodic feature extraction, and extracting periodic features and non-periodic features in the input power load sample sequence; performing dynamic weighted fusion on the periodic features and the non-periodic features to form fused features; taking the fused features as input, and combining with a processing layer to construct a power load prediction model; and training the power load prediction model, and generating a final prediction result by using the trained power load prediction model.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric load forecasting, and particularly relates to a time series forecasting method with polynomial activation function and adaptive period fusion. Background Art

[0002] Electric load forecasting refers to predicting the electricity demand at a certain future moment or time period through the analysis and modeling of historical electric load data. This technology is widely applied in fields such as power system scheduling, load balancing, energy management, demand response, etc. Electric load data usually exhibits the characteristics of a time series. As time changes, the load fluctuations are not only affected by factors such as seasonality and holidays, but may also be affected by changes in the external environment such as meteorological changes and social activities. Therefore, the electric load forecasting problem can be classified as a time series forecasting problem.

[0003] A time series is a sequence formed by sampling data at fixed time intervals, which has timeliness and time dependence. In many practical applications, time series data often exhibits periodic characteristics, that is, the data shows repeated patterns within a specific time interval. The TimesNet model is designed based on the periodic characteristics in the time series. It identifies the potential periodicity in the sequence through Fourier transform and converts the original time series into a two-dimensional tensor composed of multiple periods. Then, it uses a convolutional network for feature extraction, achieving good prediction results. However, for time series containing non-periodic components, the prediction effect of this method may be limited. The time series in practical applications often contains complex periodic changes and non-periodic components that are difficult to capture or extract. Simply relying on the method of dealing with periodic characteristics is difficult to fully utilize all information, which may lead to a decrease in prediction accuracy.

[0004] Activation functions play a crucial role in neural networks, determining whether a neuron is activated and to what extent. Common activation functions include Sigmoid, Tanh, ReLU, Leaky ReLU, and GELU, etc. Sigmoid is suitable for binary classification problems but is prone to gradient vanishing; Tanh performs better than Sigmoid and has a mean closer to zero, but still has the problem of gradient vanishing when the input is large; ReLU is computationally simple and has a fast convergence rate, but the negative part being zero may cause neurons to become inactivated; Leaky ReLU alleviates the problem of neuron inactivation by maintaining a small slope in the negative interval, but its effect is unstable in some applications; GELU is based on the Gaussian error function, has good performance and interpretability, but has a high computational complexity. In deep learning, as the scale of the dataset increases and the number of network layers increases, the computational efficiency of activation functions becomes an important consideration. To improve the training speed, sometimes activation functions with simpler calculations are selected, or complex activation functions are simplified through approximation methods. Therefore, according to the specific task requirements, weighing between performance and computational efficiency and choosing the appropriate activation function becomes an important decision in practice. Summary of the Invention

[0005] To overcome the problems in the prior art, the present invention proposes a time series prediction method combining polynomial activation function and adaptive period fusion.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: The present invention provides a time series prediction method combining polynomial activation function and adaptive period fusion, including the following steps: Generate an input power load sample sequence and a prediction target from power load data through a sliding window; Construct a piecewise polynomial activation function; and build a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction to extract the periodic features and aperiodic features in the input power load sample sequence; dynamically weight and fuse the periodic features and aperiodic features to form the fused features; Use the fused features as input and combine with a processing layer to build a power load prediction model; Train the power load prediction model and use the trained power load prediction model to generate the final prediction result.

[0007] Further, the construction of the piecewise polynomial activation function includes: Construct a piecewise polynomial activation function through interpolation fitting with a cubic Hermite function ; Suppose the function value at the point is , at the point The first derivative value at is , then in the interval on, The definition of ; Among them, for each interpolation interval, the cubic Hermite basis functions , , and The expressions of are respectively: ; In the above formula, represents the target position of interpolation; represents the current data point; represents the next data point; represents the interval between two adjacent data points.

[0008] Furthermore, the extraction of the periodic features in the input power load sample sequence includes: Performing a frequency domain transformation on the input power load sample sequence using Fourier transform to calculate the amplitudes of each frequency; Selecting the first most significant frequency components according to the magnitudes of the amplitudes of each frequency to capture the periodic features in the data; Folding the one-dimensional data into a two-dimensional tensor according to the periodic features; Inputting the folded two-dimensional tensor into the constructed deep learning model, and the constructed deep learning model includes a convolutional layer and a piecewise polynomial activation function, which is used to further enhance the representation ability of the periodic features.

[0009] Furthermore, the process of inputting the folded two-dimensional tensor into the constructed deep learning model, and the constructed deep learning model includes a convolutional layer and a piecewise polynomial activation function, which is used to further enhance the representation ability of the periodic features, is as follows: ; Re-merging the subsequence of the feature tensor of the learned periodic features into a sequence of the original length , and then using the method of adaptive aggregation to perform weighted averaging on the results of different periods to obtain ; Adding to the seasonal term through residual connection to obtain , and then performing an anti-stationarization process on to restore it to the original data distribution, and the final projection layer maps the features to the predicted target dimension, and the implementation process is: ; In the above formula, represents the data normalization method; represents the inverse normalization method; represents the folded two-dimensional tensor; represents the periodic feature tensor after anti-stationarization processing; represents the output periodic feature.

[0010] Furthermore, the extraction of the non-periodic features in the input power load sample sequence includes: Performing normalization processing on the input power load sample sequence to ensure that the scales of each feature are consistent; through multiple linear transformations and non-linear activation functions extract the latent features of the data , and use a fully connected layer for further mapping: ; In the above formula, represents the input sequence; represents the calculation result after layer normalization; represents the layer normalization process; represents the calculation result after non-linear transformation and activation function; represents the non-periodic feature component.

[0011] Furthermore, dynamically weighting and fusing the periodic features and non-periodic features to form the fused features includes: ; In the above formula, represents the fused feature; represents the non-periodic feature extraction process, and the calculation result is ; represents the periodic feature extraction process, and the calculation result is ; represents the learnable parameter weight tensor, where represents the dimension of the feature, is adjusted through an optimization method during the training process.

[0012] Compared with the prior art, the present invention has the following technical effects: By using a piecewise polynomial activation function, the present invention can more flexibly adapt to the complex periodic and aperiodic characteristics in time series data such as power load, thereby improving the performance of the model in practical applications. At the same time, in order to better handle the aperiodic components in the time series, the present invention constructs a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction to process the periodic and aperiodic features in the time series. This method can capture the non-linear relationships in the time series simultaneously and enhance the model's comprehensive modeling ability for changing trends, thus significantly improving the prediction effect. In addition, since the importance of the periodic and aperiodic components in different datasets varies in the overall data, an adaptive periodic weight learning mechanism is proposed, aiming to dynamically learn and adjust the weights of the periodic and aperiodic components in the data to achieve more accurate predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0014] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features, and their effects of the technical solutions proposed according to the present invention. The specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the present invention.

[0016] In one embodiment of the present invention, referring to Figure 1 , a time series prediction method with a polynomial activation function and adaptive periodic fusion is provided, including the following steps: Generate an input power load sample sequence and a prediction target from the power load data through a sliding window; Construct a piecewise polynomial activation function; and construct a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction to extract the periodic and aperiodic features in the input power load sample sequence; dynamically weight and fuse the periodic and aperiodic features to form the fused features; Use the fused features as input and combine with the processing layer to construct an electric load forecasting model; Train the electric load forecasting model and use the trained model to generate the final forecasting result.

[0017] The following details each of the above steps: Step 100: Generate an input electric load sample sequence and a forecasting target from the electric load data through a sliding window.

[0018] As an example, this step may include the following steps: Divide the electric load data set into a training set, a validation set, and a test set according to a predefined ratio, and generate continuous input samples and forecasting targets through a sliding window. Respectively extract the target variable and its corresponding time feature encoding; perform normalization processing on the input samples of the training set to improve the stability and generalization ability of model training; organize the generated input samples, forecasting targets, and their corresponding time feature encodings into a format usable by the model to provide basic data support for subsequent model training and forecasting.

[0019] Specifically, divide the electric load data set into a training set (Train), a validation set (Validation), and a test set (Test) according to a ratio: (1); In the above formula, represents the electric load data set, represents the training set, represents the validation set, represents the test set.

[0020] Use the window size to slide and generate continuous input samples and forecasting targets, and respectively extract the target variable and its corresponding time feature encoding: (2); In the above formula, represents the input sample sequence data, represents the forecasting target, represents the sliding window The data point at the next time step after is used as the forecasting target of the model.

[0021] Normalize the training set data to improve stability: (3); Extract time features for the timestamp including month, day, week, and hour. Let represent the th time point, represents Time feature encoding at a point: (4).

[0022] Step 200: Construct a piecewise polynomial activation function to enhance the non - linear modeling ability and improve the feature representation ability in both non - periodic feature and periodic feature extraction.

[0023] Activation function Designed based on piecewise polynomials, it can more flexibly control the function shape, thus adapting to different data features, especially periodic and non - periodic features. The activation function helps the model better extract the potential non - periodic trends in the data by enhancing the non - linear modeling ability, thus effectively adapting to the complex changes in the data; and the activation function helps to enhance the representation ability of periodic features, enabling the model to better identify periodic fluctuations.

[0024] Specifically, construct the piecewise polynomial activation function through cubic Hermite function interpolation fitting , more flexibly control the shape of the interpolation polynomial, thus quickly generating a smooth curve and maintaining the original features of the data points.

[0025] Suppose the function value at point is , and the first - derivative value at point is , where , then on the interval , Definition of (5); Among them, for each interpolation interval, the expressions of the cubic Hermite basis functions , , and are respectively as follows: (6); In the above formula, represents the target position of interpolation; represents the current data point; represents the next data point; represents the interval between two adjacent data points.

[0026] According to the above formula, it can be observed that for each interpolation interval, only four quantities are needed to determine the expression of the cubic Hermite interpolation polynomial, which are , , and .

[0027] Based on the observation of GELU, ten segmented intervals are divided. , and , and cubic Hermite functions are constructed in 10 intervals respectively. . Based on and the similarity with GELU, the function values of the GELU activation function at the interval endpoints are calculated. After calculation, the function value of GELU at is , and the function value at is .

[0028] Based on the idea of approximate approximation, let have a function value at the point of , and a derivative value . At the point , the function value is , and the derivative value is . At this time, the activation function expressions in the intervals and can be determined as: (7); Next, discuss the construction of a piecewise activation function in the interval, that is, construct a Hermite cubic interpolation function in the interval . Based on and the similarity with GELU, perform least squares fitting on and GELU. Construct an objective function through least squares Let have the smallest sum of squared errors between and .

[0029] First, uniformly insert 11 points in each interval, as follows: , as shown below: (8); Then define the objective function as follows: (9); Find the optimal parameter values by minimizing , that is, take the partial derivatives of respectively, so as to obtain . Since , the overall objective function is defined as follows: (10); Then, for the and in Equation (10), the solution equations are as follows: (11); By calculating the above equation, the corresponding function values at point are respectively , At , the first-order derivatives can be respectively taken as . Substituting the calculated and corresponding values into Formula (6) to calculate the interpolation function, the activation function expressions within each interval can be calculated.

[0030] Step 300: Construct a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction, and extract the periodic features and aperiodic features in the input power load sample sequence.

[0031] Construct a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction, that is, construct a periodic feature extraction module and an aperiodic feature extraction module .

[0032] The purpose of the periodic feature extraction module is to extract the periodic features in the input power load sample sequence, especially the repeated patterns or periodic fluctuations that may exist in the input power load sample sequence. To achieve this goal, this module adopts a method based on the fast Fourier transform (FFT) to analyze the frequency distribution of the input power load sample sequence. First, perform a frequency-domain transformation on the input power load sample sequence using the Fourier transform to calculate the amplitudes of each frequency; select the first most significant frequency components according to the magnitudes of the amplitudes of each frequency to capture the periodic features in the data; then, fold the one-dimensional data into a two-dimensional tensor according to the periodic features. In this way, the periodic feature extraction module can accurately identify the periodic fluctuations in the data and provide these periodic features for the subsequent prediction model; finally, input the folded two-dimensional tensor into the constructed deep learning model. The constructed deep learning model includes a series of convolutional layers and piecewise polynomials to further enhance the representation ability of the periodic features and provide more abundant information for the final output of the model.

[0033] The aperiodic feature extraction module The goal is to extract time - independent features from the input power load sample sequence, capturing the non - periodic trends and patterns in the data. In this module, first, the input power load sample sequence is normalized to ensure that the scales of each feature are consistent; then, through multiple linear transformations and non - linear activation functions the potential features in the input power load sample sequence are extracted, and a fully - connected layer is used for further mapping. The core purpose of these operations is to learn the non - periodic trends and features existing in the input power load sample sequence to better adapt to the complex changes in the data.

[0034] As a specific example, extracting the periodic features from the input power load sample sequence may include the following sub - steps: Step 3101: Use the Fourier transform to perform a frequency - domain transformation on the input power load sample sequence, calculate the amplitudes of each frequency; select the first most significant frequency components according to the magnitudes of the amplitudes of each frequency to capture the periodic features in the input power load sample sequence.

[0035] Use the FFT to obtain the spectral distribution, and select the indices of the first most significant frequencies according to the magnitudes of the frequencies, and calculate the corresponding periods and frequency weights based on these indices.

[0036] The specific implementation process is shown in formula (12): (12); In the above formula, represents the frequency weight; represents taking the magnitude after performing the FFT transformation on the seasonal term ; represents the corresponding period; represents the frequency; represents the length of the input sequence; represents the amplitude spectrum of the Topk Fourier transform results.

[0037] Step 3102: Fold the one - dimensional data into a two - dimensional tensor according to the periodic features.

[0038] According to the calculated potential periodic patterns, fold the sequence according to the period to obtain a set of two - dimensional tensors based on multiple periods. The specific implementation method is shown in formula (13), (13); In the above formula, represents the folded two - dimensional tensor; Indicates reshaping the folded two-dimensional tensor back into a one-dimensional tensor; Indicates data padding.

[0039] Step 3103: Input the folded two-dimensional tensor into the constructed deep learning model. The constructed deep learning model includes a series of convolutional layers and piecewise polynomials, which are used to further enhance the representation ability of periodic features and provide richer information for the final output of the model.

[0040] For the folded two-dimensional tensor Hierarchical feature extraction is performed through recurrent multi-scale convolutional blocks fused with piecewise polynomial activation functions, and different receptive fields are obtained through multiple convolutional kernels to capture the periodic patterns of the data at different time scales, enabling the model to more comprehensively understand the periodic structure. The piecewise polynomial activation function introduces non-linear features, enabling the model to better capture the non-linear structure of the sequence. For periodic signals, especially those with multiple periods of different frequencies, this non-linear activation function helps to more accurately model the periodic features. The specific implementation process is shown in Equation (14): (14); For the subsequence of the feature tensor of the learned periodic features re-combine it into a sequence of the original length , and then use the method of adaptive aggregation to perform weighted averaging on the results of different periods to obtain ; Through residual connection, is added to the seasonal term to obtain , and then is de-seasonalized to restore to the original data distribution. The final projection layer maps the features to the predicted target dimension. The specific implementation process is shown in Equation (15): (15); In the above formula, represents the data normalization method; represents the denormalization method; represents the periodic feature tensor after de-seasonalization; represents the output periodic features. As a specific example, extracting the non-periodic features in the input power load sample sequence may include the following sub-steps: Step 3201: Normalize the input power load sample sequence to ensure that the scales of each feature are consistent; (16); In the above formula, represents the input sequence; Represents the calculation result after layer normalization; Represents layer normalization processing.

[0041] Step 3202: Extract the latent features of the data through multiple linear transformations and non-linear activation functions , and use a fully connected layer for further mapping: (17); In the above formula, represents the non-periodic feature tensor after the linear layer and the activation function; represents the non-periodic feature.

[0042] Step 330: Dynamically weight and fuse the periodic feature and the non-periodic feature to form a fused feature.

[0043] In the feature fusion and output stage, the non-periodic feature and the periodic feature are combined to form a more comprehensive input feature set. The periodic feature and the non-periodic feature are fused by means of weighted sum, and the weights are dynamically adjusted according to the importance of the periodic feature and the contribution of the non-periodic feature. The goal of this stage is to enable the model to comprehensively consider the two types of features, thereby improving the prediction ability.

[0044] As a specific implementation, this step may include the following sub-steps: Step 3301: Use an adaptive learnable periodic weight learner to learn and strip out the weight ratios of the periodic component and the non-periodic component in the entire power load sample sequence, and judge the role weights of the periodic feature and the non-periodic feature in the prediction of the entire power load sample sequence.

[0045] The adaptive learnable periodic weight learner adopts a learnable parameter weight tensor , where represents the dimension of the feature, which will be adjusted by optimization methods such as gradient descent during training. This enables the power load prediction model to dynamically learn the relative importance of the periodic and non-periodic features in the sequence according to the characteristics of the input data, enabling the model to achieve weighted fusion of the two types of features, thereby obtaining the final output.

[0046] Step 3302: Based on the role weights, fuse the periodic feature and the non-periodic feature: (18); In the above formula, represents the fused feature; Represents the aperiodic feature extraction process, and the calculation result is ; Represents the periodic feature extraction process, and the calculation result is .

[0047] Step 400: Use the fused features as input, combine with the fully connected layer and activation function to construct a power load prediction model.

[0048] After feature fusion, these features are further optimized through the fully connected layer and activation function, and finally the prediction results are generated. In the output stage, after the transformation of multiple layers of networks, the final prediction results are output, and these results will reflect the comprehensive influence of periodic and aperiodic features in the sequence data.

[0049] Step 500: Train the power load prediction model on the training set, apply the trained power load prediction model to the test set, generate prediction results, and quantitatively analyze the model performance through evaluation metrics such as MSE and MAE.

[0050] Apply the trained model to the test set input , generate prediction results . Quantitatively analyze the model performance through the defined evaluation metrics (such as mean square error MSE and mean absolute error MAE).

[0051] Quantitatively analyze the model performance through the defined evaluation metrics (such as mean square error MSE and mean absolute error MAE): (19); (20); Among them, h is the number of samples, is the predicted value, is the real data.

[0052] Starting from the perspective that the power load time series has both periodicity and aperiodicity, the present invention proposes a dual-branch feature extraction architecture based on periodic feature extraction and aperiodic feature extraction, and uses an adaptive and learnable periodic weight learner to learn and strip out the weight proportions of the periodic component and aperiodic component in the entire sequence, and judge the role weights of periodic features and aperiodic features in the prediction of the entire sequence; and proposes a construction method for a piecewise polynomial activation function, through which different piecewise polynomial activation functions can be constructed to adapt to different model architectures, so that the activation function can play a better role in the model. Through a large number of experiments, it is proved that the present invention improves the prediction effect while increasing the interpretability.

[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A time series prediction method based on polynomial activation function and adaptive period fusion, characterized in that Including the following steps: Generating an input power load sample sequence and a prediction target from the power load data through a sliding window; Constructing a piecewise polynomial activation function; and building a dual-branch feature extraction architecture for periodic feature extraction and aperiodic feature extraction to extract the periodic features and aperiodic features in the input power load sample sequence; Dynamically weighting and fusing the periodic features and aperiodic features to form the fused features; Using the fused features as the input and combining with a processing layer to build a power load prediction model; Training the power load prediction model and using the trained power load prediction model to generate the final prediction result.

2. The polynomial activation function and the time series prediction method with adaptive periodic fusion according to claim 1, characterized in that The constructing of the piecewise polynomial activation function includes: Constructing a piecewise polynomial activation function by interpolation fitting with cubic Hermite functions ; Let the function value at the point be , and the first derivative value at the point be , where . Then on the interval , the definition of is: ； Among them, for each interpolation interval, the expressions of the cubic Hermite basis functions , , and are respectively as follows: ； In the above formula, represents the target position of interpolation; represents the current data point; represents the next data point; represents the interval between two adjacent data points.

3. The polynomial activation function and the time series prediction method with adaptive periodic fusion according to claim 1, characterized in that The extracting of the periodic features in the input power load sample sequence includes: Perform a frequency-domain transformation on the input power load sample sequence using the Fourier transform to calculate the amplitudes of each frequency; select the first most significant frequency components according to the magnitudes of the amplitudes of each frequency to capture the periodic characteristics in the data; Folding the one-dimensional data into a two-dimensional tensor according to the periodic features; Inputting the folded two-dimensional tensor into the constructed deep learning model, where the constructed deep learning model includes a convolutional layer and a piecewise polynomial activation function for further enhancing the representation ability of the periodic features.

4. The polynomial activation function and the time series prediction method with adaptive periodic fusion according to claim 3, characterized in that, The process of inputting the folded two-dimensional tensor into the constructed deep learning model, where the constructed deep learning model includes a convolutional layer and a piecewise polynomial activation function for further enhancing the representation ability of the periodic features is as follows: ； Feature tensor subsequences of the learned periodic features Re - merged into a sequence of the original length , and then the results of different periods are weighted - averaged using the adaptive aggregation method to obtain ; Through residual connection, is added to the seasonal term to obtain , and then is de - detrended to restore to the original data distribution. The final projection layer maps the features to the predicted target dimension, and the implementation process is as follows: ； In the above formula, represents a data normalization method; represents an inverse normalization method; represents the two-dimensional tensor after folding; represents the periodic feature tensor after anti-stationarization processing; represents the output periodic feature.

5. The polynomial activation function and the time series prediction method with adaptive periodic fusion according to claim 3, characterized in that The extracting of the aperiodic features in the input power load sample sequence includes: Normalize the input power load sample sequence to ensure that the scales of all features are consistent; through multiple linear transformations and non-linear activation functions extract the latent features of the data , and use a fully connected layer for further mapping: ； In the above formula, represents the input sequence; represents the calculation result after layer normalization; represents the layer normalization process; represents the calculation result after non-linear transformation and activation function; represents the aperiodic feature component.

6. The polynomial activation function and the time series prediction method with adaptive periodic fusion according to claim 5, wherein The dynamically weighting and fusing of the periodic features and aperiodic features to form the fused features includes: ； In the above formula, represents the fused feature; represents the aperiodic feature extraction process, and the calculation result is ; represents the periodic feature extraction process, and the calculation result is ; represents the learnable parameter weight tensor, where represents the dimension of the feature, which is adjusted by an optimization method during the training process.

Images