Load prediction method and system based on attention mechanism

Abstract

The invention relates to a load prediction method and system based on an attention mechanism, and the system comprises the steps: 1, carrying out the deep analysis of power utilization data, calculating the importance of each feature, selecting a key feature according to the importance, learning the dependency relationship of long sequence input based on a multi-head probability sparse self-attention model, and carrying out the calculation of the importance of each feature; a relative position coding method is adopted to enhance the internal relation between the data, and a data feature set is obtained; and step 2, training a multi-scale time sequence prediction model based on the data feature set obtained in the step 1 as an input feature set, and predicting a load sequence and peak valleys. According to the method, different parts in the time sequence are dynamically focused through the attention mechanism, the long-time dependency relationship in the time sequence is captured, important features can be extracted more effectively, complex modes in data can be captured more effectively, the accuracy of power load prediction is improved, and decision support is provided for defining the adjustable range of the load and optimizing energy distribution.

Description

Technical Field

[0001] The present invention belongs to the field of electric power technology, and in particular relates to a load forecasting method and system based on an attention mechanism. Background Art

[0002] With the advancement of power market reform and the construction of new power systems, the challenges faced by the power grid are increasing. With the access of new loads such as distributed energy and electric vehicles, the volatility and randomness of power grid load have increased significantly, which puts higher requirements on maintaining the stability of the power grid. Accurate prediction of power load plays a vital role in power grid dispatching and management. It is not only the basis for power system operation management and power generation planning, but also the key to ensure that metering equipment operates in the best state and achieves accurate metering, regulation and management.

[0003] At present, the methods of power load forecasting are mainly divided into three categories: statistical methods such as autoregressive models, multivariate linear regression and Kalman filtering, machine learning such as random forests and support vector machines, time series analysis methods and simple neural network methods. Although the time series analysis method has a simple model and less data requirements, it is mainly suitable for stable time series and has limitations in processing complex nonlinear relationships and unstructured information; machine learning methods have strong data modeling capabilities, but require a lot of feature engineering, and fail to fully capture the long-term and short-term characteristics of the load, and the prediction accuracy is limited; simple neural network methods, such as back propagation, deep learning, and long short-term memory networks, have strong model prediction capabilities, but have high requirements for data volume. In practical applications, insufficient data will greatly reduce the learning and prediction capabilities of the model, making it difficult to meet the needs of complex scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a load forecasting method and system based on an attention mechanism, taking into account the diverse access resources in energy consumption scenarios, the difficulty in accurately perceiving the adjustable range of the load, and the difficulty of existing power load forecasting methods in meeting the needs in current complex scenarios. The attention mechanism is used to dynamically focus on different parts of a time series and capture long-term dependencies in the time series, thereby more effectively extracting important features, capturing complex patterns in the data, and improving the accuracy of power load forecasting, providing decision support for clarifying the adjustable range of the load and optimizing energy distribution.

[0005] The present invention provides a load forecasting method based on an attention mechanism, comprising the following steps:

[0006] Step 1: Conduct in-depth analysis of electricity consumption data, calculate the importance of each feature, and select key features based on the importance. Based on the multi-head probabilistic sparse self-attention model, learn the dependency relationship of long sequence inputs, and use the relative position encoding method to enhance the intrinsic connection between data to obtain the data feature set;

[0007] Step 2: Based on the data feature set obtained in step 1 as the input feature set, the multi-scale time series prediction model is trained, and the load time series and peaks and valleys are predicted; the multi-scale time series prediction model framework includes N layers, each layer includes multiple branches, which are used to simultaneously learn the time series representations of different scales; wherein each branch includes a marker for converting the input representation into several fragments, and the fragments are processed into tags through a self-attention mechanism and relative position encoding; the output representations of each branch in each layer of the framework are finally aggregated to form a separate embedding body and passed to the next layer.

[0008] Furthermore, the step 1 comprises:

[0009] Step 1.1, collect user-side multi-time scale load data and user-side multi-factor load data; the multi-time scale load data includes load data in units of hours, days, weeks, months and years; the collected user-side multi-factor load data includes historical load data, electricity, heat, cooling, gas data, user behavior data and electricity price data;

[0010] Step 1.2, multi-load coupling feature mining:

[0011] The Pearson correlation coefficient is used to analyze the internal coupling relationship of multi-factor loads and the correlation between multi-factor loads and meteorological factors.

[0012] Step 1.3, calculate feature importance and select key features:

[0013] For the original load data, the original data matrix is ​​expanded by taking the load ratio, load high power and load logarithm to construct the feature engineering of the original data:

[0014] Step,1.4, uses the multi-head probabilistic sparse self-attention mechanism for feature extraction,capturing long- and short-term features.

[0015] Furthermore, the calculation formula of the Pearson correlation coefficient in step 1.2 is:

[0016]

[0017] Where: is the Pearson correlation coefficient of vectors Y1 and Y2; cov(Y1,Y2) is the covariance of feature vectors Y1 and Y2; are the standard deviations of Y1 and Y2 respectively; the correlation coefficient ranges from [-1,1], the larger the absolute value of the correlation coefficient, the stronger the linear correlation; a correlation coefficient of 0.8-1.0 indicates an extremely strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.

[0018] Furthermore, the step 1.3 includes:

[0019] Based on the cold, hot and electric data, the expanded matrix of the original data is calculated as follows:

[0020] Z=[Z1Z2Z3]

[0021]

[0022] Where: Z1Z2Z3 represent the matrix after the original data of cooling, heating and electricity loads are expanded according to ratio, logarithm and high power respectively; each row in the matrix represents the data of a sample, and in the i-th row, i=1,2,3,…,n; z EC,i 、z EH,i 、z CH,i represent the electric cooling ratio, electric heating ratio and cooling-heating ratio of the i-th sample respectively; z logE,i 、z logC,i 、z logH,i They represent the logarithm of the electricity, cooling and heating loads of the ith sample respectively; They represent the kth power of the electric, cooling and heating loads of the ith sample respectively: L represents the highest power series;

[0023] For the Z1 and Z2 matrices, the Pearson correlation coefficient is directly used to analyze the correlation between the original electric, cold and hot load data and the load ratio and logarithmic expansion matrix. For the Z3 matrix, the coupling matrix within the load and between the loads is calculated respectively; for the electric load:;

[0024]

[0025] Where: They are the coupling matrix between electric loads and the coupling matrix between electric loads and cold and hot; The element is in the i-th row and j-th column, i,j=1,2,3,…,L; The correlation coefficient between the column vectors representing the i-th and j-th powers of the electrical load; the matrix middle Correlation coefficient between column vectors representing the i-th power of electric load and the j-th power of cooling load, i,j=1,2,3,…,L; Correlation coefficient between column vectors representing the i-th power of electrical load and the j-th power of thermal load, i,j=1,2,3,…,L;

[0026] Calculate the coupling characteristics of the matrix Z3, the calculation formula is:

[0027]

[0028] Where: Zb Represents the original data matrix of electric load, L = 1; Z c represents the electric load expansion data matrix, L≠1; k=[(1 / 1!),(1 / 2!),…(1 / L!)] is the coefficient matrix required to establish the Taylor-like expansion; ⊙ represents the Hadamard product; F(E) is the coupling characteristic of the matrix Z3; is the coupling matrix between the above-mentioned electric loads and the coupling matrix between the electric loads and the cold and hot parts; where L is 3; Z i To represent the expanded data matrix of electric load, it covers the data expanded to different powers and is used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k;

[0029] The obtained coupling characteristic variable matrix is:

[0030]

[0031] Where: F(C) and F(H) are the coupling characteristics of the cooling and heating loads obtained after calculation.

[0032] Furthermore, the step 1.4 comprises:

[0033] The calculation formula of the self-attention model is:

[0034]

[0035] Where: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q , W k , W v The trainable linear transformation matrices corresponding to Q, K, and V vectors respectively; d is the input vector dimension; r is the scaling factor, which is used to adjust the scale of the dot product result between the key vector K and the query vector Q in the self-attention mechanism. ;

[0036] The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The metric value calculation formula is:

[0037]

[0038] Where: M(q i ,K) is the sparsity measure: q i is the i-th element in Q; k j is the jth element in K; L k is the length of K;

[0039] Based on the sparsity measurement method, the formula of the probabilistic sparse self-attention mechanism is obtained, namely:

[0040]

[0041] Where: is the probabilistic sparse self-attention score matrix; is the first u largest M(q i ,K) corresponding to the query element set, where u = clnL Q , c is the sampling factor, in practical applications L Q = L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from the original O(L 2 ) is reduced to O(Ll nL);

[0042] The multi-head self-attention mechanism divides the model's Q, K, and V into multiple branches to form multiple subspaces. After combining with the sparse self-attention model, each subspace generates different sparse Query-Key pairs, allowing the model to focus on information from subspaces at different positions to avoid information loss. The calculation formula is:

[0043] A MH (Q,K,V)=Concat(h1,h2,…,h n )W α

[0044]

[0045] Where: A MH (Q, K, V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α is a trainable linear transformation matrix; h i is the attention score matrix of each head after calculation by the sparse self-attention mechanism; is the self-attention score matrix;

[0046] In the multi-head attention mechanism, the linear transformation matrix of each head is a trainable parameter, and the model adaptively learns the sequence context that each head pays attention to.

[0047] Furthermore, the step 2 comprises:

[0048] Step 2.1, branch-specific tokenization:

[0049] Each branch in each layer of the model processes the input payload data using a fragment-based labeling method, converting the input dataset Divide into different branch fragment sets according to different granularities b represents the ordinal number of the branch; each branch has segments of the same length, and each segment consists of multiple consecutive timestamps, representing a time step window; the size of the segment controls the granularity of the temporal features that the branch focuses on. Branches with shorter segments are used to capture high-frequency patterns, while branches with longer segments are used to identify lower-frequency seasonal trends;

[0050] Step 2.2, self-attention mechanism and relative position encoding:

[0051] In each branch, the self-attention mechanism in step 1.4 is applied to the tokenized segments to identify the dependencies between different time intervals; in each branch cut in each layer, the self-attention mechanism of Transformer is applied to the fragmented data Get the output matrix Considering the periodicity of load data and the distance between time steps, relative position encoding is applied in the model, so that the model can understand and predict recurring trends;

[0052] Step 2.3, characterization aggregation and result prediction:

[0053] In each layer, the representations from all branches are combined into a single, unified result to be passed to the next layer; the output matrix of each branch is is flattened into a row vector B is the total number of branches; the row vectors of all branches are then concatenated into a total row vector Finally, it is input into the linear layer to obtain the final result of this layer. Through layer by layer, the last layer of the model outputs the predicted load value of the future time step.

[0054] Furthermore, the step 2.1 also includes:

[0055] For each branch, the time series data is split into segments with partial overlaps to ensure that every part of the load data is considered in the prediction process.

[0056] The present invention also provides a load forecasting system based on an attention mechanism, comprising a load forecasting module, wherein the load control module executes the load forecasting method based on the attention mechanism.

[0057] The present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the load forecasting method based on the attention mechanism is implemented.

[0058] The present invention also provides an electronic device, comprising:

[0059] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the load forecasting method based on the attention mechanism by executing the computer instructions.

[0060] Through the above scheme, the load forecasting method and system based on the attention mechanism have the following technical effects:

[0061] 1) The present invention adopts a multi-resolution attention mechanism, which processes load data of different time scales through multiple branches, and can accurately capture short-term load fluctuations (such as daily peak and valley load changes) and identify long-term trends (such as seasonal changes). This technical advantage ensures the accuracy and timeliness of load forecasting.

[0062] 2) Compared with traditional load forecasting methods, this invention greatly optimizes the computational complexity through multi-time scale Transformer, which can effectively reduce the computational cost of large-scale load data while maintaining high prediction accuracy. This advantage makes this method suitable for real-time load forecasting in large-scale power grid environments.

[0063] 3) The present invention adopts relative position encoding technology, and the model can automatically perceive the distance between time steps, so as to better capture the periodic characteristics in the load data. This technical advantage enables the model to learn adaptively in different load scenarios and adapt to a variety of complex load modes.

[0064] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of a load prediction method of the attention mechanism of the present invention;

[0066] Figure 2 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0067] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0068] Ginseng Figure 1 As shown, this embodiment provides a load forecasting method based on an attention mechanism, comprising the following steps:

[0069] Step S1, deeply analyze the electricity consumption data, calculate the importance of each feature, and select key features according to the importance, learn the dependency relationship of long sequence input based on the multi-head probabilistic sparse self-attention model, and use the relative position encoding method to enhance the intrinsic connection between data to obtain the data feature set;

[0070] Step S2, based on the data feature set obtained in step S1 as the input feature set, the multi-scale time series prediction model is trained, and the load time series and peaks and valleys are predicted; the multi-scale time series prediction model framework includes N layers, each layer includes multiple branches, which are used to simultaneously learn the time series representations of different scales; wherein each branch includes a marker for converting the input representation into several fragments, and the fragments are processed into tags through a self-attention mechanism and relative position encoding; the output representations of each branch in each layer in the framework are finally aggregated to form a separate embedding body and passed to the next layer.

[0071] The present invention is described in further detail below.

[0072] The load forecasting method based on attention mechanism mainly includes two aspects: perception and extraction of key features of multiple loads, and load timing and peak-valley prediction.

[0073] a) Multi-dimensional key feature perception and extraction

[0074] Through the load data feature selection algorithm based on the improved feature extraction technology of Transformer and the user key feature description algorithm, the user's electricity consumption data is deeply analyzed and feature extracted. The algorithm is used to calculate the importance of each feature, select key features based on these importances, and learn the dependency relationship of long sequence inputs through a multi-head probabilistic sparse self-attention model. The relative position encoding method is used to enhance the intrinsic connection between data and improve the prediction performance of the model, thereby optimizing the input feature set of the load forecasting model and improving the prediction accuracy and robustness of the model.

[0075] The specific operations and steps of this method are as follows:

[0076] Step 1.1 Collect multi-time scale load data on the user side. Multi-time scale load data includes load data in units of hours, days, weeks, months and years; collect multi-dimensional load data on the user side. Multi-dimensional load data includes historical load data, electricity, heat, cooling, gas data, user behavior data and electricity price data, etc.

[0077] Step 1.2 Mining the coupling characteristics of multiple loads. The Pearson correlation coefficient (PCC) is used to analyze the internal coupling relationship of multiple loads and the correlation between multiple loads and meteorological factors. The calculation formula of the Pearson correlation coefficient is:

[0078]

[0079] Where: is the Pearson correlation coefficient of vectors Y1 and Y2; cov(Y1,Y2) is the covariance of feature vectors Y1 and Y2; are the standard deviations of Y1 and Y2 respectively. The range of the correlation coefficient is [-1,1]. The larger the absolute value of the correlation coefficient, the stronger the linear correlation. It is generally believed that a correlation coefficient of 0.8-1.0 indicates an extremely strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.

[0080] Step 1.3 calculates the importance of features and selects key features. In Step 1.2, the Pearson coefficient can only describe the linear correlation between loads. In order to further explore the nonlinear coupling relationship between cold, hot and electric loads, the original data matrix is ​​expanded by taking the ratio between loads, the high power of loads and the logarithm of loads for the original load data to construct the feature engineering of the original data. Taking cold, hot and electric data as an example, the expansion matrix of the original data is first calculated as follows:

[0081] Z=[Z1Z2Z3]

[0082]

[0083] Where: Z1Z2Z3 represent the matrix after the original data of cooling, heating and electricity loads are expanded according to ratio, logarithm and high power; EC,i 、z EH,i 、z CH,i represent the electric cooling ratio, electric heating ratio and cooling-heating ratio of the i-th sample respectively; z logE,i 、z logC,i 、z logH,i They represent the logarithm of the electricity, cooling and heating loads of the ith sample respectively; They represent the kth power of the electricity, cooling and heating loads of the ith sample respectively: L represents the highest power series.

[0084] For the Z1 and Z2 matrices, the Pearson correlation coefficient is directly used to analyze the correlation between the original electricity, cooling and heating load data and the load ratio and logarithmic expansion matrix. For the Z3 matrix, the coupling matrices within the load and between the loads are calculated respectively (taking the electricity load as an example).

[0085]

[0086] Where: They are the coupling matrix between electric loads and the coupling matrix between electric loads and cold and hot; The element in row i and column j (i, j = 1, 2, 3, ..., L) The correlation coefficient between the column vectors representing the i-th and j-th powers of the electrical load; the matrix middle The correlation coefficient between the column vectors representing the ith power of the electric load and the jth power of the cooling load; The correlation coefficient between the column vectors representing the ith power of the electrical load and the jth power of the thermal load;

[0087] Next, the coupling characteristics of the matrix Z3 are calculated using the following formula (taking the electric load as an example):

[0088]

[0089] Where: Z b Represents the original data matrix of electric load (L=1; Z c represents the electric load expansion data matrix (L≠1); k=[(1 / 1!),(1 / 2!),…,(1 / L!)] is the coefficient matrix required to establish the Taylor-like expansion; ⊙ represents the Hadamard product. In order to avoid excessive calculation, L is taken as 3. F(E) is the coupling characteristic of the matrix Z3; is the coupling matrix between the above-mentioned electric loads and the coupling matrix between the electric loads and the cold and hot parts; Z i To represent the expanded data matrix of electric load, it covers the data expanded to different powers and is used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k.

[0090] Finally, the coupled characteristic variable matrix is ​​obtained as:

[0091]

[0092] Where: F(C) and F(H) are the coupling characteristics of the cold and hot loads obtained after calculation. The coupling feature mining algorithm used here can mine the nonlinear coupling relationship between loads. In the subsequent modeling, it can be used as the input variable of the prediction model to assist in improving the feature extraction effect of the model.

[0093] Step 1.4 uses the multi-head probabilistic sparse self-attention mechanism to extract features and capture long-term and short-term features. The self-attention model is the most widely used attention model at present, and its standard calculation formula is:

[0094]

[0095] Where: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q , Wk , W v The trainable linear transformation matrices corresponding to the Q, K, and V vectors respectively; d is the input vector dimension; r is the scaling factor, which is used to adjust the scale of the dot product result between the key vector K and the query vector Q in the self-attention mechanism.

[0096] The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The metric value calculation formula is:

[0097]

[0098] Where: M(q i ,K) is the sparsity measure: q i is the i-th element in Q; k j is the jth element in K; L k is the length of K;

[0099] Based on the sparsity measurement method, the formula of the probabilistic sparse self-attention mechanism is obtained, namely:

[0100]

[0101] Where: is the probabilistic sparse self-attention score matrix; is the first u largest M(q i ,K) corresponding to the query element set, where u = clnL Q , c is the sampling factor, in practical applications L Q = L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from the original O(L 2 ) is reduced to O(Ll nL).

[0102] The multi-head self-attention mechanism divides the model's Q, K, and V into multiple branches to form multiple subspaces. After combining with the sparse self-attention model, each subspace generates different sparse Query-Key pairs, so that the model can focus on information from subspaces at different positions, effectively avoiding information loss. The calculation formula is:

[0103] A MH (Q,K,V)=Concat(h1,h2,…,h n )W α

[0104]

[0105] A MH(Q, K, V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α is a trainable linear transformation matrix; h i is the attention score matrix of each head after calculation by the sparse self-attention mechanism; is the self-attention score matrix;

[0106] b) Load timing and peak-valley prediction

[0107] The multi-scale time series prediction model is trained and inferred using the data feature set extracted in a). Model Overview: The load time series and peak-valley multi-time scale prediction model framework contains N layers, each layer contains multiple branches. Such a multi-branch structure allows the model to learn time series representations of different scales at the same time. Each branch contains a marker to convert the input representation into some fragments (patches). These fragments will be processed into tokens through self-attention mechanism and relative position encoding. The output representations of each branch in each layer of the framework will finally aggregate to form a separate embedding and pass it to the next layer. The following is a detailed introduction to the model framework process.

[0108] Step 2.1 Branch-specific tokenization. Each branch in each layer of the model uses a fragment-based tokenization method to process the input payload data. Divide into different branch fragment sets according to different granularities b represents the ordinal number of the branch. Each branch has segments of the same length, and each segment consists of multiple consecutive timestamps, representing a time step window. The size of the segment controls the granularity of the temporal features that the branch focuses on. Branches with shorter segments are used to capture high-frequency patterns (such as daily load fluctuations), while branches with longer segments help identify lower-frequency seasonal trends. For each branch, the time series data is split in a partially overlapping manner during the segment splitting process to ensure that every part of the load data is considered in the forecasting process.

[0109] Step 2.2 Self-attention mechanism and relative position encoding. In each branch, the self-attention mechanism in step 1.4 will be applied to the tokenized segments to identify the dependencies between different time intervals. In each layer, in each branch cut as above, the Transformer's self-attention mechanism will be applied to these fragmented data. So as to obtain the output matrix This includes conventional techniques in the model such as multi-head self-attention, residual connections, normalization, and feed-forward networks. This mechanism allows the model to focus on the key time steps that have the greatest impact on future load forecasts. The self-attention mechanism is designed to operate on segments, making it ideal in capturing temporal dependencies within and between segments. In addition, given the periodicity of the load data, relative position encoding is applied in the model. This encoding takes into account the distance between time steps, allowing the model to better understand and predict recurring trends such as daily peaks or seasonal changes in load demand.

[0110] Step 2.3 Representation aggregation and result prediction. In each layer, the representations from all branches are combined into a single, unified result to be passed to the next layer. The output matrix of each branch is (b=1,2,…,B; B is the total number of branches) is flattened into a row vector The row vectors of all branches are then concatenated into a total row vector Finally, it is fed into the linear layer to get the final result of this layer. This fusion process integrates short-term and long-term load patterns, allowing the model to make accurate load forecasts while considering both immediate fluctuations and broader trends. In this framework, the last layer of the model outputs the predicted load value for future time steps. By utilizing a multi-time scale attention mechanism, the model is able to predict short-term peaks and troughs as well as long-term seasonal trends in load demand.

[0111] The load forecasting method based on the attention mechanism has the following technical effects:

[0112] 1) The multi-resolution attention mechanism is used to process load data of different time scales through multiple branches, which can accurately capture short-term load fluctuations (such as daily peak and valley load changes) and identify long-term trends (such as seasonal changes). This technical advantage ensures the accuracy and timeliness of load forecasting.

[0113] 2) Compared with traditional load forecasting methods, the computational complexity is greatly optimized through multi-time scale Transformer, which can effectively reduce the computational cost of large-scale load data while maintaining high prediction accuracy. This advantage makes this method suitable for real-time load forecasting in large-scale power grid environments.

[0114] 3) By using relative position encoding technology, the model can automatically perceive the distance between time steps, thereby better capturing the periodic characteristics in the load data. This technical advantage enables the model to adaptively learn in different load scenarios and adapt to a variety of complex load patterns.

[0115] This embodiment also provides a load prediction system based on an attention mechanism, including a load prediction module, which executes the load prediction method based on the attention mechanism.

[0116] This embodiment also provides a non-transitory computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the load prediction method based on the attention mechanism is implemented.

[0117] Ginseng Figure 2 As shown, this embodiment also provides an electronic device, including:

[0118] A memory 201 and a processor 202, wherein the memory 201 and the processor 202 are communicatively connected to each other, the memory 201 stores computer instructions, and the processor 202 executes the load forecasting method based on the attention mechanism by executing the computer instructions.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the technical principles of the present invention, and these improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A load forecasting method based on attention mechanism, characterized in that: The steps include: Step 1: Conduct in-depth analysis of electricity consumption data, calculate the importance of each feature, and select key features based on the importance. Based on the multi-head probabilistic sparse self-attention model, learn the dependency relationship of long sequence inputs, and use the relative position encoding method to enhance the intrinsic connection between data to obtain the data feature set; Step 2: Based on the data feature set obtained in step 1 as the input feature set, the multi-scale time series prediction model is trained, and the load time series and peaks and valleys are predicted; the multi-scale time series prediction model framework includes N layers, each layer includes multiple branches, which are used to simultaneously learn the time series representations of different scales; wherein each branch includes a marker for converting the input representation into several fragments, and the fragments are processed into tags through a self-attention mechanism and relative position encoding; the output representations of each branch in each layer of the framework are finally aggregated to form a separate embedding body and passed to the next layer.

2. The load forecasting method based on the attention mechanism according to claim 1 is characterized in that: The step 1 comprises: Step 1.1, collect user-side multi-time scale load data and user-side multi-factor load data; the multi-time scale load data includes load data in units of hours, days, weeks, months and years; the collected user-side multi-factor load data includes historical load data, electricity, heat, cooling, gas data, user behavior data and electricity price data; Step 1.2, multi-load coupling feature mining: The Pearson correlation coefficient is used to analyze the internal coupling relationship of multi-factor loads and the correlation between multi-factor loads and meteorological factors. Step 1.3, calculate feature importance and select key features: For the original load data, the original data matrix is ​​expanded by taking the load ratio, load high power and load logarithm to construct the feature engineering of the original data: Step,1.4, uses the multi-head probabilistic sparse self-attention mechanism for feature extraction,capturing long- and short-term features.

3. The load forecasting method based on the attention mechanism according to claim 2 is characterized in that: The calculation formula of the Pearson correlation coefficient described in step 1.2 is: Where: is the Pearson correlation coefficient of vectors Y1 and Y2; cov(Y1,Y2) is the covariance of feature vectors Y1 and Y2; are the standard deviations of Y1 and Y2 respectively; the correlation coefficient ranges from [-1,1], the larger the absolute value of the correlation coefficient, the stronger the linear correlation; a correlation coefficient of 0.8-1.0 indicates an extremely strong correlation, 0.6-0.8 indicates a strong correlation, 0.2-0.6 indicates a moderate correlation, and 0-0.2 indicates a weak correlation or no correlation.

4. The load forecasting method based on the attention mechanism according to claim 3 is characterized in that: The step 1.3 comprises: Based on the cold, hot and electric data, the expanded matrix of the original data is calculated as follows: Z=[Z1Z2Z3] Where: Z1Z2Z3 represent the matrix after the original data of cooling, heating and electricity loads are expanded according to ratio, logarithm and high power respectively; each row in the matrix represents the data of a sample, and in the i-th row, i=1,2,3,…,n; z EC,i 、z EH,i 、z CH,i represent the electric cooling ratio, electric heating ratio and cooling-heating ratio of the i-th sample respectively; z logE,i 、z logC,i 、z logH,i They represent the logarithm of the electricity, cooling and heating loads of the ith sample respectively; They represent the kth power of the electric, cooling and heating loads of the ith sample respectively: L represents the highest power series; For the Z1 and Z2 matrices, the Pearson correlation coefficient is directly used to analyze the correlation between the original electric, cold and hot load data and the load ratio and logarithmic expansion matrix. For the Z3 matrix, the coupling matrix within the load and between the loads is calculated respectively; for the electric load:; Where: They are the coupling matrix between electric loads and the coupling matrix between electric loads and cold and hot; The element is in the i-th row and j-th column, i,j=1,2,3,…,L; The correlation coefficient between the column vectors representing the i-th and j-th powers of the electrical load; the matrix middle Correlation coefficient between column vectors representing the i-th power of electric load and the j-th power of cooling load, i,j=1,2,3,…,L; Correlation coefficient between column vectors representing the i-th power of electrical load and the j-th power of thermal load, i,j=1,2,3,…,L; Calculate the coupling characteristics of the matrix Z3, the calculation formula is: Where: Z b Represents the original data matrix of electric load, L = 1; Z c represents the electric load expansion data matrix, L≠1; k=[(1 / 1!),(1 / 2!),…(1 / L!)] is the coefficient matrix required to establish the Taylor-like expansion; ⊙ represents the Hadamard product; F(E) is the coupling characteristic of the matrix Z3; is the coupling matrix between the above-mentioned electric loads and the coupling matrix between the electric loads and the cold and hot parts; where L is 3; Z i To represent the expanded data matrix of electric load, it covers the data expanded to different powers and is used to calculate the nonlinear coupling relationship between loads; [kk] is the concatenation of two coefficient matrices k; The coupled characteristic variable matrix obtained is: Where: F(C) and F(H) are the coupling characteristics of the cooling and heating loads obtained after calculation.

5. The load forecasting method based on the attention mechanism according to claim 4 is characterized in that: The step 1.4 comprises: The calculation formula of the self-attention model is: Where: A(Q,K,V) is the self-attention score matrix; Q is the query vector; K is the key vector; V is the value vector; X is the input vector; W q , W k , W v The trainable linear transformation matrices corresponding to Q, K, and V vectors respectively; d is the input vector dimension; r is the scaling factor, which is used to adjust the scale of the dot product result between the key vector K and the query vector Q in the self-attention mechanism. ; The sparse self-attention model uses Kullback Leibler divergence to measure sparsity. The calculated metric value reflects the importance of the corresponding query element. The metric value calculation formula is: Where: M(q i ,K) is the sparsity measure: q i is the i-th element in Q; k j is the jth element in K; L k is the length of K; Based on the sparsity measurement method, the formula of the probabilistic sparse self-attention mechanism is obtained, namely: Where: is the probabilistic sparse self-attention score matrix; is the first u largest M(q i ,K) corresponding to the query element set, where u = clnL Q , c is the sampling factor, in practical applications L Q = L, where L is the length of the input sequence. After selection, the computational complexity of attention is reduced from the original O(L 2 ) is reduced to O(Ll nL); The multi-head self-attention mechanism divides the model's Q, K, and V into multiple branches to form multiple subspaces. After combining with the sparse self-attention model, each subspace generates different sparse Query-Key pairs, allowing the model to focus on information from subspaces at different positions to avoid information loss. The calculation formula is: A MH (Q,K,V)=Concat(h1,h2,…,h n )W α Where: A MH (Q, K, V) is the attention score matrix after multi-head self-attention calculation; n is the number of multi-heads, indicating that the probabilistic sparse self-attention is calculated n times; Concat(·) is the concatenation operation; W α is a trainable linear transformation matrix; h i is the attention score matrix of each head after calculation by the sparse self-attention mechanism; is the self-attention score matrix; In the multi-head attention mechanism, the linear transformation matrix of each head is a trainable parameter, and the model adaptively learns the sequence context that each head pays attention to.

6. The load forecasting method based on the attention mechanism according to claim 5 is characterized in that: The step 2 comprises: Step 2.1, branch-specific tokenization: Each branch in each layer of the model processes the input payload data using a fragment-based labeling method, converting the input dataset Divide into different branch fragment sets according to different granularities b represents the ordinal number of the branch; each branch has segments of the same length, and each segment consists of multiple consecutive timestamps, representing a time step window; the size of the segment controls the granularity of the temporal features that the branch focuses on. Branches with shorter segments are used to capture high-frequency patterns, while branches with longer segments are used to identify lower-frequency seasonal trends; Step 2.2, self-attention mechanism and relative position encoding: In each branch, the self-attention mechanism in step 1.4 is applied to the tokenized segments to identify the dependencies between different time intervals; in each branch cut in each layer, the self-attention mechanism of Transformer is applied to the fragmented data Get the output matrix Considering the periodicity of load data and the distance between time steps, relative position encoding is applied in the model, so that the model can understand and predict recurring trends; Step 2.3, characterization aggregation and result prediction: In each layer, the representations from all branches are combined into a single, unified result to be passed to the next layer; the output matrix of each branch is is flattened into a row vector b = 1, 2, ..., B; B is the total number of branches; the row vectors of all branches are then concatenated into a total row vector Finally, it is input into the linear layer to obtain the final result of this layer. Through layer by layer, the last layer of the model outputs the predicted load value of the future time step.

7. The load forecasting method based on the attention mechanism according to claim 6 is characterized in that: The step 2.1 also includes: For each branch, the time series data is split into segments with partial overlaps to ensure that every part of the load data is considered in the prediction process.

8. A load forecasting system based on attention mechanism, characterized in that: It includes a load prediction module, and the load control module executes a load prediction method based on an attention mechanism as described in any one of claims 1-7.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a processor, implement a load forecasting method based on an attention mechanism as described in any one of claims 1-7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the load forecasting method based on the attention mechanism as described in any one of claims 1 to 7 by executing the computer instructions.

Images