Transform-based charging pile long time sequence prediction model and prediction system

Abstract

The invention discloses a long time sequence prediction model of a charging pile based on Transform, so as to better predict the electricity consumption of the charging pile with intermittent characteristics. The model is based on a Transform architecture capable of capturing a long-time sequence dependency relationship, a double-input mode of respectively inputting positions and values is adopted in an input process, and a module with time segmentation and fusion is selected in the model to replace an attention mechanism of a general Transform model. In order to conform to a double-input mechanism of the model, the value and position mixed Splitt-Attention is adopted in the Attention part, and information fusion is performed at the end of the model, so that the fitting capability of an intermittent sequence is improved.

Description

Technical Field

[0001] The present invention relates to a Transformer-based long time series prediction model and a prediction system in the context of predicting the power consumption of charging piles with intermittent characteristics. Background Art

[0002] With the rapid development of new energy vehicles in recent years, the demand for charging stations has increased rapidly, and the management and planning of charging stations has become an important issue. Reasonable planning can help reduce energy waste and improve energy efficiency. With the help of the collected power consumption data, the power consumption forecast of the power station can assist the charging station operators to make reasonable decisions, thereby increasing corporate profits. As an important method for scientific research based on time series data, time series forecasting can reasonably complete this task. Under the influence of the global energy crisis and policy promotion, new energy vehicles need to pay attention to more subsequent development, so the need for long time series forecasting is proposed.

[0003] Since the problem of time series prediction was proposed in the last century, the theory has gradually improved, starting from the traditional regression model. In recent years, the emerging machine learning has improved on the basis of traditional time series prediction, thereby improving and developing a more accurate prediction model. Representative methods include convolutional neural networks as basic neural networks, recurrent neural networks and their variants LSTM, GRU, etc. In addition, network models originally used for other purposes have been improved to obtain the ability to fit sequence data and are also used to predict time series, such as Transformer, Mamba, etc. Measures such as decomposition are also applied to time series to better capture time series features, such as Autoformer.

[0004] Although researchers have proposed a variety of optimization methods and built a variety of models for time series prediction, the following problems still exist: (1) Research is mostly based on complete data, such as public data sets ETT, Whether, etc. When faced with sparse time series or intermittent time series, researchers have not paid much attention. (2) The prediction effect of general time series prediction methods will be greatly reduced when faced with intermittent time series. (3) When faced with time series with a large number of zero values, researchers often use preprocessing to fill and smooth the zero values; however, in actual combat, this preprocessing needs to be considered on a case-by-case basis. The reasons for the occurrence of zero values ​​need to be considered on a case-by-case basis. Some are because the data has not been collected, such as medical data that requires wearing sensors; some represent that the current observation value is 0, such as the charging scenario mentioned in this article. The former can be processed as a general time series through interpolation, while the latter's 0 needs to be retained for model construction and training. Summary of the invention

[0005] Purpose of the invention: Traditional statistical methods often have difficulty processing complex time series data, and their prediction accuracy and generalization capabilities are often limited when faced with highly dynamic, nonlinear time series data such as charging pile power consumption. In addition to how to accurately predict the power consumption of charging piles in the future, we also face an important problem, that is, the charging pile data has intermittent characteristics and a large number of zero values. These zero values ​​may represent the situation where the charging pile has no charging activities in certain periods, so they cannot be interpolated or filled in according to traditional methods; this intermittent feature will challenge the performance of the prediction model.

[0006] In order to predict the time series data with intermittent characteristics in real data, we need to develop a new model to solve this challenge. As a solution, we propose a charging pile power consumption prediction model based on the Transformer architecture to accurately predict the power consumption of the charging pile in the future. The content of the invention can be abstracted as follows: Given a multivariate time series X n ={x1,x2,…,x n}, we need to build a suitable model to predict the value X of the next m time series m =x n+1 ,x n+2 ,…,x n+m Where X n and X m The data are collected at equal intervals.

[0007] Technical solution: By observing the processed charging pile data, it can be seen that although there are many zero values ​​in the data set, their occurrence is mostly concentrated in a certain time period. It can be seen that it is still regular and shows a regular daily seasonal pattern. The deep model with strong fitting ability will not be affected by it, resulting in too much reduction in prediction ability. Although the Transformer-based model has strong fitting ability and can learn the regularity in sparse sequences, its accuracy cannot achieve the same good effect on ordinary time series data, and it also brings certain challenges in efficiency. Therefore, this application first proposes a long-term time series prediction model for charging piles based on Transformer. In terms of model construction and training, the following operations are adopted:

[0008] 1) In order to solve the accuracy problem, this study will use a deep learning-based method to predict the time series of charging pile power consumption. Specifically, we must first select a suitable deep learning model framework to adapt to the time series characteristics of charging pile power consumption data; then, we need to improve the module based on the framework to effectively capture the time series characteristics of charging pile power consumption data, including seasonality, periodicity, and trend. In addition, the parameters of the model also need to be adjusted, such as by adjusting the number of network layers, number of units, learning rate, batch size and other hyperparameters to improve prediction accuracy and generalization ability.

[0009] 2) In order to solve the intermittent problem, it is necessary to analyze the characteristics of zero values. Although the charging pile power data selected in this paper accounts for a large proportion of zero, considering that zero values ​​mostly appear at night to early morning, it shows a regular daily seasonal pattern. Therefore, in the research process, it is necessary to focus on learning a baseline model that has a good learning of the daily seasonal pattern, and make improvements on this basis. In addition, in order to highlight the intermittent characteristics in the results, in addition to the two indicators of MSE and MAE commonly used in time series prediction tasks, three other indicators that are more suitable for intermittent time series are also used: MASE, MAPPE and RMSSE, to better show the advantages of the model when facing intermittent time series data.

[0010] 3) In order to solve the problem of overfitting due to insufficient data volume, it is necessary to give priority to simple models such as MLP and RNN to avoid excessive stacking of models. If it is a Transformer model with more parameters, it needs to be improved, such as using a module composed of MLP to replace the original Attention module with relatively high temporal and spatial complexity, but this improvement needs to be based on the fact that the prediction accuracy will not decrease. In addition, when adjusting the number of model layers, the number of hidden units, etc., when the amount of data is limited, it is usually preferred to choose a relatively small model to prevent overfitting. It is also recommended to use smaller parameters for the learning rate and batch size to reduce the noise of gradient updates and avoid falling into the local minimum or overfitting too early during the training process.

[0011] Specifically, there are the following steps:

[0012] 1) Using the Transformer architecture and using the same Decoder input as Informer can achieve better results in processing LSTF problems.

[0013] 2) In the Embedding layer, the model's value and position are input separately to facilitate the processing of intermittent series. The relationship between the value and position of intermittent time series is more important. If they are fused, it may cause information loss or limit the model's understanding ability. This method of independently processing values ​​and positions is more conducive to capturing information.

[0014] 3) A module with time segmentation and fusion is used to replace the Attention in the general Former model. Considering that the new module is mainly composed of fully connected layers, it reduces the running time, helps to suppress overfitting, and improves prediction efficiency.

[0015] 4) In the Attention part, Split-Attention, which is a mixture of value and position, is used to fuse information at the end of the model, which is more in line with the dual-input mechanism of the model.

[0016] 5) The quantile loss function is selected as the loss function. The quantile loss function can be adjusted to make it more sensitive to zero values. The q = 0.5 quantile loss function will regress to the MAE loss function, which is a commonly used loss function for time series prediction.

[0017] 6) Select evaluation indicators such as RMSSE, MAPPE, and MASE designed for intermittent time series data.

[0018] The present application also discloses a Transformer-based charging pile long time series prediction system, wherein the prediction system comprises at least one processor and a memory connected to the processor, wherein:

[0019] The memory is used to store computer programs or instructions;

[0020] The processor is used to obtain and execute the computer program or instructions in the memory so that the prediction system executes the prediction model constructed by any of the methods described above.

[0021] Beneficial effects: This study explored the problem of long-term prediction of charging pile power consumption. Considering that the power consumption time series presents the characteristics of intermittent time series, we proposed a new deep time series prediction model based on the Transformer model, which adopts the decoder architecture of Informer and designs a model structure with position and value input separately to achieve the purpose of more accurate prediction of intermittent time series. In addition, the factorization hybrid module is used to replace the ordinary attention mechanism, which reduces overfitting and improves the operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a Transformer-based model architecture diagram constructed by the present invention, which is more suitable for time series prediction tasks with intermittent characteristics.

[0023] Figure 2 This is the Split-Attention structure diagram used in the improved model.

[0024] Figure 3This is a diagram of the slice fusion technology used by Mixing Block in the improved model. DETAILED DESCRIPTION

[0025] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0026] The present invention discloses a Transformer-based charging pile long-time series prediction model, which is constructed by the following steps:

[0027] S1: Based on the original data set of historical charging piles, screen and comprehensively organize the experimental data set; in a preferred embodiment, the experimental data set - the charging pile data set is provided by the Xingxing Charging Platform, and the original data is the order data generated by the user, which contains valid information such as: order number, charging station number, charging pile number, start charging time, end charging time, power consumption, power change, etc. The Heyuan Charging Station in Lishui District, Nanjing City, with a power station number of 2408, was selected. The charging pile number was used as the classification standard, and the orders generated every hour were screened according to the start charging time of the order, and the total power consumption of each charging pile was calculated hour by hour. Finally, a data set of 19005 rows and 5 columns was obtained. This new experimental data set records the power consumption of orders generated by each charging pile every hour at the Heyuan Charging Station in Lishui District, Nanjing from January 1, 2021 to April 10, 2023.

[0028] S2: Preprocess the extracted experimental data set to generate training samples and test samples; in a preferred embodiment, the boundaries of the start and end of the data are set according to the required labels to divide the training set, validation set and test set. Then the entire data set is standardized, the time column is converted to date and time format, and time features are generated. Finally, the obtained data is packaged into small batches according to the batch size through DataLoader, a tool for loading data in PyTorch, to facilitate batch processing during model training or testing.

[0029] S3: By observing the collected and collated hourly electricity consumption data of charging piles, we can see that although zero accounts for a large proportion of the data, the occurrence of zero values ​​is mostly concentrated in the late night, and it still shows a regular daily seasonal pattern. Therefore, the deep model with strong fitting ability will not be affected by too many zeros, resulting in a reduction in the ability to predict too much. Therefore, the Transformer with strong fitting ability is selected as the basic framework. Transformer is a milestone innovation in the field of NLP. As a new neural network architecture based on the attention mechanism proposed by Google in 2017, it completely abandons the RNN structure and uses the encoder-decoder architecture instead. This architecture ensures that the input sequence can be converted into a continuous expression through the encoder, and then the decoder can generate an output sequence based on the expression.

[0030] S4: Improve the Transformer-based charging pile long-term prediction model;

[0031] S5: Use training samples to train the Transformer-based charging pile long-term prediction model;

[0032] S6: Use test samples to test the trained Transformer-based charging pile long-term prediction model.

[0033] See also Figure 1 The improved model can receive two types of inputs, value and position, and process the position and value separately in the model until they are fused at the final output. This method allows the model to learn different features independently, enhancing the flexibility of the model so as to capture the complex relationship between position and value; and it can independently process and retain more original information in the early stage, so as to better adapt to the effect of intermittent time series. The dual input mechanism of the model is that the Embedding layer can convert the input sequence into the position sequence Z pos Sum value sequence Z val Two parts: Z pos ,Z val = Embedding(X), where X is the input of the model. The framework of the model is shown in Figure X. The Encoder receives the input of the position and value parts, and is composed of a MixingBlock module and a normalization layer with residual connection. The Decoder receives X = {concat(a pos ,b),concat(a val ,b)}, where b is a zero-filled layer with a length equal to the predicted length. The part that replaces 0 in the final output is the final result. It is internally composed of a MixingBlock module, SplitAttention, and a normalized layer with residual connections.

[0034] See also Figure 2-Figure 3 , the specific modules of model construction and training are:

[0035] (1) Mixing Block

[0036] This module implements a factorized hybrid model in the time dimension, which captures the temporal dependencies in the input time series through the MLP method, and finally merges them into a tensor and outputs them. Figure 3 As shown, the implementation process is:

[0037] a) Split the two types of sequence information of input position and location into multiple subsequences according to the sampling rate

[0038] b) Use MLP to extract and learn the temporal information of each sequence. As a combination of linear layers and nonlinear activation functions, this module maps the input data to a higher dimensional space for feature extraction and finally maps it back to the original dimensional space.

[0039] c) Merge the segments in the original order to obtain the segmented sequence information, which is used as the output of the forward propagation.

[0040] (2) Split-Attention

[0041] The scaled dot product attention mechanism is the attention mechanism used in Transformer. After that, Z will generate three embeddings according to the three projection matrices: Query, Key, and Value. The final attention is defined as:

[0042]

[0043] This self-attention mechanism allows elements to pay attention to all other elements rather than adjacent elements, thereby enhancing the model's ability to perceive the world. However, considering that the Transformer here is used for time series prediction rather than the MLP field, the model cannot perceive the content in the subsequent time period, so it is necessary to introduce a mask matrix M to ensure the rationality of the prediction. That is:

[0044]

[0045] Considering that the input of the model is divided into two parts: position and value X = {concat(a pos ,b),concat(a val ,b)}, the attention mechanism needs to make certain adjustments to the input and output, so we constructed its variant SplitAttention based on the scaled dot product attention. Its innovation lies in the dimension splitting and projection of the input value vector and position vector, and decomposing the calculation of the attention weight into four parts: the dot product of the query value vector and the key value vector, the dot product of the query position vector and the key position vector, the dot product of the query value vector and the key position vector, and the dot product of the query position vector and the key value vector. The steps of input, output, splitting and projection are shown in Figure X. The final calculation formula is as follows:

[0046]

[0047] By introducing SplitAttention into the model, we can capture different relationships in the input in a more fine-grained manner and combine different types of information more flexibly to adapt to the dual-input mechanism of model value and position input separately, while enhancing the robustness of the model to better predict intermittent time series.

[0048] (3) Loss Function

[0049] This model selects two types of loss functions: MseLoss and quantile loss. MseLoss is the most commonly used loss function in machine learning and deep learning regression tasks. Quantile loss is an extension of MaeLoss, which can control the error to different values.

[0050] Taking into account the intermittent characteristics of time series and the fact that their distribution is somewhat unbalanced, the quantile loss function can be used to flexibly select the quantile value according to the characteristics of the distribution, thereby better reflecting the characteristics of data imbalance and making the model more robust.

[0051] The above is an exemplary embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A Transformer-based charging pile long-term prediction model, characterized by It is constructed by the following steps: S1: Based on the original data set of historical charging piles, the experimental data set is screened and comprehensively organized; S2: Preprocess the obtained experimental data set to generate training samples and test samples; S3: Select Transformer as the basic deep learning model framework of the model based on data characteristics; S4: Improve the Transformer-based charging pile long-term prediction model; S5: Use training samples to train the Transformer-based charging pile long-term prediction model; S6: Use test samples to test the trained Transformer-based charging pile long-term prediction model.

2. According to the Transformer-based charging pile long time series prediction model according to claim 1, it is characterized in that: In S1, the experimental data set - charging pile data set is provided by the charging platform, and the original data is the order data generated by the user, which contains the following valid information: order number, charging station number, charging pile number, charging start time, charging end time, power consumption, and power change.

3. According to the Transformer-based charging pile long time series prediction model of claim 1, it is characterized in that: After the experimental data set obtained in S1 is read, the start and end boundaries of the data are set according to the required labels to divide the training set, validation set, and test set; then the entire data set is standardized, the time column is converted into date and time format, and the time feature is generated; Finally, DataLoader, a tool used to load data in PyTorch, packages the obtained data into small batches according to the batch size to facilitate batch processing during model training or testing.

4. According to the Transformer-based charging pile long time series prediction model of claim 1, it is characterized in that: Transformer is selected as the basic framework, using the encoder-decoder architecture. The input sequence is converted into a continuous expression by the encoder, and the decoder generates an output sequence based on the expression.

5. According to the Transformer-based charging pile long time series prediction model of claim 1, it is characterized in that: The specific implementation steps for the improvements made based on the Transformer framework are as follows: (1) Dual input mechanism: The value and position of the obtained data are input separately to facilitate the processing of intermittent sequences; the Embedding layer used can convert the input sequence into a position sequence Z pos Sum value sequence Z val Two parts: Z pos ,Z val = Embedding(X), where X is the input of the model; (2) A Mixing Block with time segmentation and fusion is used to replace the Attention in the general Former model. The implementation process is as follows: a) splitting the two types of sequence information of input position and value into multiple subsequences according to the sampling rate; b) Use the MLP module to extract and learn the temporal information of each sequence; The MLP module composed of linear layers and nonlinear activation functions can map the input data to a higher dimensional space for feature extraction, and finally map it back to the original dimensional space; c) Merge the segmented sequence information in the original order and use it as the output of the forward propagation; (3) In the Attention part, Split-Attention, which is a mixture of value and position, is used to fuse information at the end of the model. A variant of Split-Attention is constructed based on the scaled dot product attention. Split-Attention performs dimension segmentation and projection on the input value vector and position vector, and decomposes the calculation of attention weight into four parts: the dot product of the query value vector and the key value vector, the dot product of the query position vector and the key position vector, the dot product of the query value vector and the key position vector, and the dot product of the query position vector and the key value vector. The final calculation formula is as follows: In the formula, the p and v subscripts represent the projection of the position vector and the value vector respectively, Q represents the query vector, K represents the key vector, V represents the value vector, M represents the mask matrix, and d k Represents the dimension of the word vector.

6. According to the Transformer-based charging pile long time series prediction model of claim 1, it is characterized in that: In S5, the program traverses the predetermined training cycles and gradually extracts batches of data using the training loader in each cycle; In each iteration, the optimizer gradient is cleared first, and then the batch data is transferred to the GPU; then, the previously packaged dataloader is used as the input of the model, and the model is used to perform forward propagation to calculate the output and its corresponding loss value; after the loss calculation is completed, backpropagation and parameter update are performed; at the end of each cycle, the code calculates and records the training loss and the validation and test losses; at the same time, it decides whether to trigger the early stopping mechanism based on the validation loss; if the early stopping mechanism is triggered, the training process will end; After training is complete, the code loads the model weights that performed best during training and returns the final trained model.

7. A Transformer-based charging pile long time series prediction model according to claim 6, characterized in that: The training cycle epoch is set to 20, and the loss function uses the quantile loss function; as an extension of the mean absolute error MAELoss, the quantile loss function can control the quantile value so that the error has different values.

8. The Transformer-based charging pile long-term prediction model according to claim 1 is characterized in that: In S6, during the test process, the model weights saved in the previous training are first loaded to ensure that the best model is used during testing. Then the model is set to evaluation mode, gradient calculation is disabled to increase the inference speed, and the batch data in the test data loader is traversed, and forward propagation is performed on each batch to obtain the output of the model. In addition to the mean square error (MSE) and mean absolute error (MAE) commonly used in time series forecasting tasks, three additional evaluation indicators are added to provide a basis for subsequent data analysis and model comparison. The newly added evaluation indicators are: mean absolute scaled error MASE, mean arctangent absolute percentage error MAPPE and root mean square scaled error RMSSE, and the formulas are as follows: In the formula, n represents the number of samples, y i Represented as the actual value at time i, It is represented as the predicted value at time i, and h is represented as the length of the prediction range.

9. A Transformer-based charging pile long-term prediction system, the prediction system comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs or instructions; The processor is used to obtain and execute the computer program or instructions in the memory so that the prediction system executes the prediction model constructed by any method according to claims 1-8.

Images