Power load prediction method based on load data intrinsic characteristic mining and DCAformer

Abstract

The invention discloses a power load prediction method based on load data intrinsic characteristic mining and DCAformer, and the method comprises the steps: 1, collecting all factors which may affect power load data, and adding weekend data; 2, decomposing the original load data into a plurality of components by using a VMD method; and 3, performing optimization by using an LEA lotus effect optimization algorithm, determining an optimal decomposition number K and a penalty factor alpha, and decomposing the original load data by using optimized parameters. 4, trend data, periodic data, weekend data and meteorological data are extracted from the decomposition mode to be integrated, and an extended data set is constructed. And 5, dividing the pre-processed extended data set into a training set, a verification set and a test set, and inputting data of the test set into the constructed DCAformer network model for training. And step 6, sending the extended data set into the trained DCAformer network to obtain a load prediction result.

Description

Technical Field

[0001] The present invention relates to the field of power system technology, and in particular to a power load forecasting technology that effectively integrates the intrinsic characteristics of load sequences and external influencing factors, and more particularly to a DCAformer power load forecasting method based on the coordination of intrinsic characteristics of load data and meteorological factors. Background Art

[0002] Electricity is a vital source of energy driving stable industrial development. Improving the utilization of electricity resources has become a crucial approach to addressing crises such as fossil fuel depletion, climate change, and environmental pollution. The application of the power Internet of Things (IoT) has led to an explosive growth in power data. Leveraging big data technologies to unlock the value of this data will drive intelligent power grids. Power load forecasting also plays a crucial role in modern power supply systems, serving as the foundation for efficient power dispatch.

[0003] Power load fluctuations are influenced by multiple factors and often exhibit nonlinear and high volatility. Relying solely on information from the load series itself is insufficient to achieve accurate forecasts. Therefore, incorporating multidimensional characteristic variables into load forecasting systems has become a research hotspot. In addition to traditional exogenous variables such as meteorological factors and holiday effects, it is also particularly important to delve into the inherent characteristics of load series, such as periodicity and trends, to assist in forecasting.

[0004] Vaswani et al. first proposed the Transformer network architecture in "Attention Is All You Need." This architecture completely breaks away from traditional recurrent neural network structures and relies on a self-attention mechanism to efficiently model global dependencies. Zhou et al. proposed the Informer model in "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting," replacing the original self-attention mechanism with a new sparse attention mechanism to reduce time and space complexity. Nie et al. proposed the PatchTST model in "A Time Series is Worth 64 Words: Long-Term Forecasting with Transformers" to extract local information from time series by segmenting them into subsequences. However, these models fail to fully exploit the inherent characteristics of the time series when faced with the complex, nonlinear, and highly volatile nature of modern power load fluctuations, resulting in forecasting accuracy and generalization capabilities that often fall short of practical requirements. Furthermore, a single model architecture often struggles to balance local feature extraction with global modeling capabilities.

[0005] Based on this background, this patent proposes a new power load forecasting method based on load sequence feature mining and multi-source information fusion. This method effectively integrates the intrinsic characteristics of the load sequence and external influencing factors, and constructs a high-precision power load forecasting model. This research not only helps optimize power system scheduling strategies and improve energy efficiency, but also has important theoretical value and practical significance for promoting energy structure transformation and promoting green and low-carbon development. Summary of the Invention

[0006] The present invention aims to address the problem that existing forecasting methods fail to fully exploit the inherent characteristics of load data, resulting in low prediction accuracy in subsequent forecasting models when processing complex data. Furthermore, most existing forecasting models rely on a single model, which often fails to fully capture the characteristics of time series data and presents limitations. Therefore, to address these issues, the present invention proposes a new load forecasting method that improves prediction accuracy by mining the inherent characteristics of load data and establishing a hybrid forecasting model.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] The power load forecasting method based on load data intrinsic feature mining and DCAformer includes the following steps:

[0009] Step 1: Collect all factors that may affect the power load data; and add weekend data. That is, for data points that fall on Saturday or Sunday, set the value of the Week variable to 1; otherwise, its value is 0;

[0010] Step 2: Decompose the original load data into multiple components using the VMD method;

[0011] Step 3: Use the LEA lotus effect optimization algorithm to find the optimal number of decompositions K and penalty factor α, and use the optimized parameters to decompose the original load data;

[0012] Step 4: Select the mode with the lowest frequency from the multiple modes obtained by decomposition as the trend feature; calculate the sample entropy value of each mode separately, and select the one with the lowest sample entropy value in the medium-frequency mode as the period feature. The extracted trend feature and period feature are integrated with the weekend-weekday one-hot encoded data and meteorological factor data (including temperature, relative humidity and wind direction) obtained in step 1 to construct an extended data set as the input of the prediction model;

[0013] Step 5: Divide the preprocessed extended dataset into a training set, a validation set, and a test set. Input the test set data into the constructed DCAformer network model for training to obtain a trained network.

[0014] Step 6: Send the expanded dataset into the trained DCAformer network to obtain the load forecast results.

[0015] In step 2, the specific steps of using VMD decomposition are:

[0016] Step 2-1) VMD ensures that the decomposed sequence is a modal component with finite bandwidth at the center frequency while minimizing the sum of the estimated bandwidths of all modes, with the constraint that the sum of all modes is equal to the original signal. The resulting constrained optimization problem is as follows:

[0017]

[0018] Among them, u k represents the modal component, ω k represents the center frequency of the kth modal component, K represents the number of decomposed modes, represents the partial derivative operator, δ(t) represents the Dirac function, u k (t) represents the kth modal component decomposed, f(t) represents the original signal, which is the sum of all modal components, and t represents the time variable. Represents the modal u k Perform the Hilbert transform and take the derivative, represents the square of the two-norm;

[0019] Step 2-2) In order to solve the above constrained optimization problem, the constrained variational problem is transformed into an unconstrained variational problem, and the augmented Lagrangian function is introduced using the quadratic penalty term and the Lagrange multiplier method:

[0020]

[0021] Among them, {u k} and {w k} is the set of all modes and their center frequencies, f(t) is the original signal, δ(t) and are the Dirac function and the partial derivative with respect to time t, K represents the number of decomposed modes, represents the imaginary part of the Hilbert transform kernel, represents the partial derivative operator, δ(t) represents the Dirac function, λ(t) represents the Lagrange multiplier, u k (t) represents the kth modal component decomposed, represents the square of the two-norm;

[0022] Step 2-3) The alternating direction multiplier method is used to perform alternating optimization iterations to obtain the final optimization result. The alternating optimization formula is as follows:

[0023]

[0024] Where ω is the frequency variable, f(ω) represents the spectrum of the original signal f(t), and K represents the number of decomposed modes. Represents the nth iteration value of the i-th mode in the frequency domain, It represents the n+1th iteration value of the ith mode in the frequency domain, represents the n+1th iteration value of the kth mode in the frequency domain, represents the nth iteration value of the Lagrange multiplier in the frequency domain, α represents the penalty factor, represents the center frequency of the kth mode in the nth iteration, represents the center frequency of the kth mode at the n+1th iteration.

[0025] In step 3, the specific steps of using the LEA lotus optimization algorithm to optimize the VMD decomposition parameters are as follows:

[0026] Step 3-1) Initialization phase:

[0027] Initialize a dragonfly swarm. The swarm size needs to be pre-set. Each dragonfly represents a candidate solution. The parameters of each candidate solution are the decomposition mode number K and the penalty factor α in VMD. To ensure the effectiveness and rationality of the algorithm, the value range of the mode number K and the penalty factor α needs to be pre-set. In addition, parameters such as the maximum number of iterations, noise tolerance, and frequency initialization method need to be defined.

[0028] Step 3-2) Evaluate the fitness of each dragonfly:

[0029] For each dragonfly individual, i.e., each candidate solution, we extract its corresponding decomposition parameter K and penalty factor α. We use these parameters to perform VMD decomposition on the input signal to obtain K modal components. Then, we calculate the envelope entropy value for each modal component. Envelope entropy is an indicator to measure signal complexity and is defined as:

[0030]

[0031] Among them, x i is the amplitude of each decomposed mode, H represents the envelope entropy value;

[0032] Step 3-3) In order to evaluate the quality of each candidate solution, the fitness function is defined as the minimum envelope entropy value of all modal components. The smaller the minimum envelope entropy value, the lower the complexity of the signal and the better the decomposition effect. Therefore, the fitness function can be expressed as:

[0033] F=min(E1,E2,...,E k ) (7)

[0034] Among them, E irepresents the envelope entropy value of the i-th mode, and F represents the fitness value of the overall decomposition;

[0035] Steps 3-4) Update the position of each dragonfly based on its fitness value, that is, update the decomposition modal number K and penalty factor α. Update the position by simulating the group behavior of dragonflies, which specifically includes the following five behavioral mechanisms:

[0036] Separation: Avoid collisions between individuals and adjacent individuals. The calculation formula is as follows:

[0037]

[0038] Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the i-th individual, N is the number of adjacent individuals, S i represents the separation vector of the i-th individual;

[0039] Alignment: The speed of an individual is aligned with the speed of its neighbors, calculated as follows:

[0040]

[0041] Among them, V j represents the speed of the jth neighboring individual, N is the number of neighboring individuals, A i Represents the alignment vector of the i-th individual; cohesion: the individual moves towards the center position of the adjacent individual, and the calculation formula is as follows:

[0042]

[0043] Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the ith individual, N is the number of adjacent individuals, C i represents the cohesion vector of the i-th individual;

[0044] Food attraction: Individuals move towards food sources, calculated as follows:

[0045] F i =X food -X i (11)

[0046] Among them, X i Indicates the current individual's position, X food Indicates the location of the food source, F i Represents the target attraction vector of the i-th individual; enemy avoidance: the individual moves away from the enemy, and the calculation formula is as follows:

[0047] E i =Xenemy -X i (12)

[0048] Among them, X i Indicates the current individual's position, X enemy Indicates the enemy's position, E i represents the cohesion vector of the i-th individual away from the enemy;

[0049] In LEA, these mechanisms are used to simulate the behavior of dragonflies in global search to achieve global optimization of the problem;

[0050] Steps 3-5) Local search phase: To further enhance the algorithm’s local search capability, LEA introduces a local search mechanism based on water droplets moving on lotus leaves. The specific steps are as follows:

[0051] Local search step size update: Perform local search near the current optimal solution. The search step size will gradually decrease with the increase in the number of iterations to achieve a refined search for the optimal solution. The calculation formula is as follows:

[0052]

[0053] in, represents the current solution, g * represents the current optimal solution, Represents the updated solution, R is the step attenuation coefficient, and the calculation formula is as follows:

[0054]

[0055] Where t is the current number of iterations and L is the maximum number of iterations;

[0056] Water droplet movement mechanism: When too many water droplets accumulate in a pit (local optimal solution), they will overflow and flow to other pits. The overflowing water droplets will select the next target pit based on the pit capacity (fitness value). The formula for updating the movement speed and position of the water droplets is as follows:

[0057] V i t+1 =qV i t (15)

[0058]

[0059] Among them, V i t represents the current velocity of the i-th water droplet at the t-th iteration, q is the velocity attenuation coefficient, V i t+1 represents the updated velocity of the i-th water drop after t+1 iterations; represents the current position of the i-th water drop at the t-th iteration, represents the updated position of the i-th water drop after t+1 iterations;

[0060] Water droplet overflow mechanism: When too many water droplets accumulate in a pit (local optimal solution), they will overflow and flow to other pits. The overflowing water droplets will select the next target pit based on the pit capacity (fitness value). The calculation formula is as follows:

[0061]

[0062] Among them, f i represents the fitness value of the current position of the i-th water drop, f max and f min Represents the maximum and minimum fitness values respectively, const is a constant, c i represents the probability factor of the i-th water droplet overflowing and jumping to a new pit. Through the movement and overflow mechanism of water droplets, LEA can more effectively find the optimal solution in local search.

[0063] Step 3-6) Termination condition: After each iteration, check whether the termination condition is met: the maximum number of iterations is reached or the fitness value reaches the preset convergence threshold; if the termination condition is met, return the current optimal mode number K and penalty factor α as the optimal parameters of VMD decomposition; if the condition is not met, continue iteration.

[0064] In step five, the constructed DCAformer network model specifically includes a parallel convolution module, a multi-head differential attention mechanism, and an encoder module that introduces a destabilizing attention mechanism; the input data first passes through the parallel convolution module, which is composed of ICB convolution and PDP convolution, and the features extracted by ICB convolution and PDP convolution are weightedly fused as the final extracted local features; the output of this module is embedded and connected to the input of the multi-head differential attention mechanism, and the output of the multi-head differential attention mechanism is connected to the input of the encoder module, and finally mapped to the final prediction result through a linear layer.

[0065] The constructed DCAformer network model is specifically as follows:

[0066] Step 5-1) Normalize the preprocessed data Z in the time dimension to obtain the normalized data x:

[0067] Calculate the mean:

[0068] Calculate the variance:

[0069] where μ Z ,σ Z ∈R C×1, represents the mean and variance, Z i Represents the data of the i-th time step, ⊙ represents the element-by-element product;

[0070] Step 5-2) Input the normalized data x into the interactive convolution module and the PDP convolution module respectively to extract local features, and then perform weighted fusion of the features extracted from each module:

[0071] X=ω1·x1+ω2·x2 (21)

[0072] Where ω1 and ω2 are learnable weight parameters, corresponding to the weights of x1 and x2 respectively, and ω1+ω2=1;

[0073] Step 5-3) The entire time series X∈{X1,X2,...,X m}Embedded as independent vectors:

[0074] h n =Embedding(X :,n ) (twenty two)

[0075] where X :,n represents the entire time series of each variable indexed by n;

[0076] Step 5-4) Send the embedded variables to the differential attention mechanism to extract key features, and add residual connections to retain some features of the original input data:

[0077] Y=h n +MultiHead(X) (23)

[0078] Step 5-5) After introducing the encoder module with the non-stationary attention mechanism, a residual connection is added based on the de-stationary attention output:

[0079] output=Y+Attention(Q',K',V',τ,Δ) (24)

[0080] Where Q', K', V' are query, key and value matrices respectively, τ is the scaling factor, Δ is the offset factor,

[0081] Pass the output through the forward propagation network:

[0082] LN2=LN(FeedForward(LN(output))) (25)

[0083] Among them, LN(·) represents normalization, and FeedForward represents forward propagation;

[0084] Finally, the final prediction result is obtained through linear layer mapping:

[0085] Output=Projection(LN2) (26).

[0086] In step 5-1, the interactive convolution module uses a double-layer structure, such as Figure 2 (b) shows that its internal process is as follows:

[0087] A1=φ(Conv1(x))⊙Conv2(x) (27)

[0088] A2=φ(Conv2(x))⊙Conv1(x) (28)

[0089] Where Conv1(.) and Conv2(.) are two convolutional layers, and φ represents the GELU activation function;

[0090] Send the activated features into the third convolutional layer:

[0091] x1=Conv3(A1+A2) (29)

[0092] Where Conv3 represents the third convolutional layer, and x1 represents the feature matrix extracted by the interactive convolution module;

[0093] In step 5-1, the PDP convolution module consists of point-by-point convolution, depth-wise convolution, and point-by-point convolution, and is finally output through the GELU activation function. The point convolution size is 1×1, as shown in the following example: Figure 2 As shown in (a), the processing flow is as follows:

[0094] Point-wise convolution:

[0095] in, is the output feature after the first point convolution, t is the time step, c' is the output channel index, C is the set of input channels, representing all input feature channels, is the weight of the point convolution, which represents the relationship between different input channels c and output channels c, x t,c is the value of the input feature at time step t and channel c, is the bias term;

[0096] Depthwise Convolution:

[0097] Among them, m represents the spatial position of the convolution kernel, K represents the kernel size, t is the time step, C represents the set of input channels, ω m and b c are the weights and bias terms of the depthwise convolution, Represents the output of the first point convolution;

[0098] Point-wise convolution:

[0099] in Represents the output feature after the second point convolution, t is the time step, c' is the output channel index, C is the set of input channels, representing all input feature channels, is the weight of the second point convolution, which represents the relationship between different input channels c and output channels c', yt ,c is the output feature after depth convolution, is the bias term;

[0100] Then x p2 The output is obtained after the activation function:

[0101] x2=GELU(x p2 ) (33)

[0102] Where x2 is the output feature matrix of the PDP convolution module.

[0103] In step 5-4, the multi-head differential attention mechanism is specifically as follows:

[0104] First, the input embedding vector is projected into the query, key, and value space:

[0105] Query vector: Q 1, Q2=h n W Q (34)

[0106] Key vector: K1, K2 = h n W K (35)

[0107] Value vector: V = h n W V (36)

[0108] Among them, W Q ,W K ,W V is a learnable weight matrix;

[0109] The differential attention mechanism is calculated by the following formula:

[0110]

[0111] The first softmax term is the attention score calculated based on the query Q1 and the key K1, and the λ softmax term is the attention score calculated based on the query Q2 and the key K2, and is scaled by a learnable scalar λ; in order to synchronize the learning dynamics, the scalar λ is reparameterized as follows:

[0112] λ=exp(α1·λ1)-exp(α2·λ2)+λ init (38)

[0113] Among them, λ1,λ2 are learnable vectors, λ init is a constant scalar used to initialize λ, and α1, α2 are learnable hyperparameters;

[0114] To further improve the expressiveness of the model, a multi-head attention mechanism is used; that is, for each attention head, a differential attention operation is calculated separately, and the results of all heads are spliced together; the specific steps are as follows:

[0115] For the i-th head, use different weight matrices W i Q , W i K ,W i V To calculate:

[0116] head i =DiffAttn(X;W i Q ,W i K ,W i V ,λ)(39)

[0117] After computing the head output, normalize each head:

[0118]

[0119] where λ init is a constant scalar, LN(·) indicates the normalization operation is applied to each head, represents the output of the i-th attention head;

[0120] Finally, the outputs of all heads are concatenated and processed through a linear transformation W O Get the final output:

[0121]

[0122] Concat(·) means concatenating the outputs of all h heads by columns, W O is a learnable linear projection matrix, and MultiHead(X) represents the output of the final multi-head differential attention.

[0123] The specific steps of the destabilizing attention mechanism in step 5-5 are:

[0124] First, project Y into the query space, key space, and value space:

[0125] Q'=YW Q (42)

[0126] K'=YW K (43)

[0127] V'=YW V (44)

[0128] Among them, W Q ,W K ,W V is a learnable weight matrix, Q', K', V' are query, key and value matrices respectively;

[0129] The scaling factor τ is converted from the unnormalized statistic μ via a multilayer perceptron z ,σ z Learning:

[0130] logτ=MLP(σ Z ,Z) (45)

[0131] Δ=MLP(μ Z ,Z) (46)

[0132] τ is the scaling factor, Δ is the offset factor, and σ z represents the variance, μ z represents the mean, the variance and mean come from the normalization operation in the previous step 5-1, and MLP(·) represents a multi-layer perceptron, which is used to extract τ and Δ from statistical information;

[0133] Combined with the learned scaling factor τ and offset factor Δ, the adjusted attention output matrix is calculated:

[0134]

[0135] Among them, d k is the dimension of the key vector, τ is the scaling factor, Δ is the offset factor, and 1 represents an all-one vector.

[0136] Compared with the prior art, the present invention has the following technical effects:

[0137] 1) This paper proposes a method for enhancing power load data: trend data and periodic data are obtained using VMD decomposition, and a lotus effect optimization algorithm is introduced to find the optimal modal number K for decomposition. The impact of holidays on load is also considered, and weekend-holiday hot encoding is used as weekend data. Meteorological factors with a strong correlation with load values are screened out by calculating the maximum information coefficient (MIC) between load values and meteorological data. Finally, the trend features, periodic features obtained through VMD decomposition, weekend features obtained through hot encoding, and screened meteorological features are combined with the original load data to form an extended data set, providing a richer feature representation for subsequent prediction models.

[0138] 2) The hybrid network model DCAformer constructed in this paper improves prediction accuracy: the model uses parallel convolution to extract local features, a multi-head differential attention mechanism to extract key information, and a non-stationary attention mechanism to model the global characteristics of the load series and its time dependence;

[0139] 3) The present invention applies the proposed data enhancement, feature selection and DCAformer network model to the real power load data of a certain region and obtains accurate power load forecast results. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] The present invention will be further described below with reference to the accompanying drawings and examples:

[0141] Figure 1 is a flow chart of the method of the present invention;

[0142] Figure 2 Schematic diagram of the DCAformer network structure in the present invention;

[0143] Figure 3 Schematic diagram of comparison between predicted value and true value in the embodiment;

[0144] Figure 4 Schematic diagram comparing the predicted values in the embodiment with the predicted values of the Informer model;

[0145] Figure 5 This is a MAE comparison chart for multi-step prediction in the embodiment. DETAILED DESCRIPTION

[0146] like Figure 1 As shown, a power load forecasting method based on load data intrinsic characteristic mining and DCAformer includes the following steps:

[0147] Step 1: Collect all factors that may affect power load data;

[0148] In step 1, these factors / data include historical power load factors and meteorological power load factors. Historical power load factors refer to historical power load values; meteorological factors include temperature, surface albedo, wind speed, precipitation, wind direction, relative humidity, air pressure, cloud type, and dew point in the forecast area. The maximum information coefficient (MIC) value is calculated for the power load data and various meteorological data points, and meteorological factors with strong correlations with the load data are selected: temperature, relative humidity, and wind direction. To capture the characteristics and trends that load data may exhibit on weekends, the value of the Week variable is set to 1 for data points that fall on Saturdays or Sundays; otherwise, its value is 0.

[0149] Step 2: Use VMD variational mode decomposition method to decompose the original load data into multiple components;

[0150] Step 2-1) VMD ensures that the decomposed sequence is a modal component with finite bandwidth at the center frequency while minimizing the sum of the estimated bandwidths of all modes, with the constraint that the sum of all modes is equal to the original signal. The resulting constrained optimization problem is as follows:

[0151]

[0152] Among them, u k represents the modal component, ω k represents the center frequency of the kth modal component, K represents the number of decomposed modes, represents the partial derivative operator, δ(t) represents the Dirac function, u k (t) represents the kth modal component decomposed, f(t) represents the original signal, which is the sum of all modal components, and t represents the time variable. Represents the modal u k Perform the Hilbert transform and take the derivative, represents the square of the 2-norm.

[0153] Step 2-2) In order to solve the above constrained optimization problem, the constrained variational problem is transformed into an unconstrained variational problem, and the augmented Lagrangian function is introduced using the quadratic penalty term and the Lagrange multiplier method:

[0154]

[0155] Among them, {u k} and {w k} is the set of all modes and their center frequencies, f(t) is the original signal, δ(t) and are the Dirac function and the partial derivative with respect to time t, K represents the number of decomposed modes, represents the imaginary part of the Hilbert transform kernel, represents the partial derivative operator, δ(t) represents the Dirac function, λ(t) represents the Lagrange multiplier, u k (t) represents the kth modal component decomposed, represents the square of the 2-norm.

[0156] Step 2-3) The alternating direction multiplier method is used to perform alternating optimization iterations to obtain the final optimization result. The alternating optimization formula is as follows:

[0157]

[0158] Where ω is the frequency variable, f(ω) represents the spectrum of the original signal f(t), and K represents the number of decomposed modes. Represents the nth iteration value of the i-th mode in the frequency domain, It represents the n+1th iteration value of the ith mode in the frequency domain, represents the n+1th iteration value of the kth mode in the frequency domain, represents the nth iteration value of the Lagrange multiplier in the frequency domain, α represents the penalty factor, represents the center frequency of the kth mode in the nth iteration, represents the center frequency of the kth mode at the n+1th iteration.

[0159] Step 3: Use the LEA lotus effect optimization algorithm to find the optimal number of decompositions K and penalty factor α, and use the optimized parameters to decompose the original load data. The specific steps are as follows:

[0160] Step 3-1) Initialization phase:

[0161] Initialize a dragonfly swarm. The swarm size must be predefined, with each dragonfly representing a candidate solution. The parameters for each candidate solution are the number of decomposition modes K and the penalty factor α in VMD. To ensure the effectiveness and rationality of the algorithm, the range of values for the number of modes K and the penalty factor α must be predefined. Additionally, parameters such as the maximum number of iterations, noise tolerance, and frequency initialization method must be defined.

[0162] Step 3-2) Evaluate the fitness of each dragonfly:

[0163] For each individual dragonfly, or each candidate solution, we extract its corresponding decomposition parameter K and penalty factor α. Using these parameters, we perform VMD decomposition on the input signal, obtaining K modal components. Subsequently, we calculate the envelope entropy for each modal component. Envelope entropy is a measure of signal complexity and is defined as:

[0164]

[0165] Among them, x iis the amplitude of each decomposed mode, and H represents the envelope entropy value.

[0166] Step 3-3) To evaluate the quality of each candidate solution, define the fitness function as the minimum envelope entropy value of all modal components. The smaller the minimum envelope entropy value, the lower the signal complexity and the better the decomposition effect. Therefore, the fitness function can be expressed as:

[0167] F=min(E1,E2,...,E k ) (7)

[0168] Among them, E i represents the envelope entropy value of the i-th mode, and F represents the fitness value of the overall decomposition.

[0169] Steps 3-4) Update the position of each dragonfly based on its fitness, i.e., update the decomposition modal number K and penalty factor α. This is done by simulating the swarm behavior of dragonflies, specifically including the following five behavioral mechanisms:

[0170] Separation: Avoid collisions between individuals and adjacent individuals. The calculation formula is as follows:

[0171]

[0172] Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the i-th individual, N is the number of adjacent individuals, S i Represents the separation vector of the i-th individual.

[0173] Alignment: The speed of an individual is aligned with the speed of its neighbors, calculated as follows:

[0174]

[0175] Among them, V j represents the speed of the jth neighboring individual, N is the number of neighboring individuals, A i Represents the alignment vector of the i-th individual. Cohesion: Individuals move toward the center of adjacent individuals. The calculation formula is as follows:

[0176]

[0177] Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the ith individual, N is the number of adjacent individuals, C i represents the cohesion vector of the i-th individual.

[0178] Food attraction: Individuals move towards food sources, calculated as follows:

[0179] F i =X food -X i (11)

[0180] Among them, X i Indicates the current individual's position, X food Indicates the location of the food source, F i Represents the target attraction vector of the i-th individual. Enemy avoidance: The individual moves away from the enemy. The calculation formula is as follows:

[0181] E i =X enemy -X i (12)

[0182] Among them, X i Indicates the current individual's position, X enemy Indicates the enemy's position, E i Represents the cohesion vector of the i-th individual away from the enemy.

[0183] In LEA, these mechanisms are used to simulate the behavior of dragonflies in global search to achieve global optimization of the problem.

[0184] Steps 3-5) Local search phase: To further enhance the algorithm’s local search capability, LEA introduces a local search mechanism based on water droplets moving on lotus leaves. The specific steps are as follows:

[0185] Local search step size update: Perform local search near the current optimal solution. The search step size will gradually decrease with the increase in the number of iterations to achieve a refined search for the optimal solution. The calculation formula is as follows:

[0186]

[0187] in, represents the current solution, g * represents the current optimal solution, Represents the updated solution, R is the step attenuation coefficient, and the calculation formula is as follows:

[0188]

[0189] Where t is the current iteration number and L is the maximum iteration number.

[0190] Water droplet movement mechanism: When too many water droplets accumulate in a pit (local optimal solution), they will overflow and flow to other pits. The overflowing water droplets will select the next target pit based on the pit capacity (fitness value). The formula for updating the movement speed and position of the water droplet is as follows:

[0191] V i t+1=qV i t (15)

[0192]

[0193] Among them, V i t represents the current velocity of the i-th water droplet at the t-th iteration, q is the velocity attenuation coefficient, V i t+1 Represents the updated velocity of the i-th water droplet after t+1 iterations. represents the current position of the i-th water drop at the t-th iteration, represents the updated position of the i-th water drop after t+1 iterations.

[0194] Water droplet overflow mechanism: When water droplets accumulate too much in a pit (local optimal solution), they will overflow and flow to other pits. The overflowing water droplets will select the next target pit based on the pit capacity (fitness value). The calculation formula is as follows:

[0195]

[0196] Among them, f i represents the fitness value of the current position of the i-th water drop, f max and f min Represents the maximum and minimum fitness values respectively, const is a constant, c i represents the probability factor of the i-th water droplet overflowing and jumping to a new pit. Through the movement and overflow mechanism of water droplets, LEA can find the optimal solution more effectively in local search.

[0197] Steps 3-6) Termination Condition: After each iteration, the termination condition is checked: reaching the maximum number of iterations or the fitness value reaching the preset convergence threshold. If the termination condition is met, the optimal mode number K and penalty factor α are returned as the optimal parameters for the VMD decomposition. If the condition is not met, the iteration continues.

[0198] Step 4: Use the Lotus Effect optimization algorithm to find the optimal decomposition number K to perform VMD decomposition on the power load data. From the eight decomposed modes, the mode with the lowest frequency is selected as the trend feature. The sample entropy value of each mode is calculated separately, and the sample with the lowest entropy value among the medium-frequency modes is selected as the periodic feature. The extracted trend and periodic features are integrated with the weekend-weekday one-hot encoded data obtained in Step 1 and meteorological factor data (including temperature, relative humidity, and wind direction) to construct an extended dataset as the input of the prediction model.

[0199] Step 4: From the eight decomposed modes, select the mode with the lowest frequency as the trend feature. Calculate the sample entropy value for each mode and select the one with the lowest sample entropy value among the intermediate-frequency modes as the periodic feature. Combine the extracted trend and periodic features with the weekend-weekday one-hot encoded data and meteorological data (including temperature, relative humidity, and wind direction) obtained in Step 1 to construct an extended dataset as input for the prediction model.

[0200] Step 5: Divide the preprocessed extended dataset into a training set, a validation set, and a test set. Input the test set data into the constructed DCAformer network model for training to obtain a trained network. The DCAformer network model processing process is as follows:

[0201] Step 5-1) Normalize the preprocessed data Z in the time dimension to obtain the normalized data x:

[0202] Calculate the mean:

[0203] Calculate the variance:

[0204] where μ Z ,σ Z ∈R C×1 , represents the mean and variance, Z i Represents the data of the i-th time step, and ⊙ represents the element-by-element product.

[0205] Step 5-2) Input the normalized data x into the interactive convolution module and the PDP convolution module respectively to extract local features, and then perform weighted fusion of the features extracted from each module:

[0206] X=ω1·x1+ω2·x2 (21)

[0207] Where ω1 and ω2 are learnable weight parameters, corresponding to the weights of x1 and x2 respectively, and ω1+ω2=1.

[0208] Step 5-3) The entire time series X∈{X1,X2,...,X m}Embedded as independent vectors:

[0209] h n =Embedding(X :,n ) (twenty two)

[0210] where X :,n represents the entire time series for each variable indexed by n.

[0211] Step 5-4) Send the embedded variables to the differential attention mechanism to extract key features, and add residual connections to retain some features of the original input data:

[0212] Y=h n +MultiHead(X) (23)

[0213] Step 5-5) After introducing the encoder module with the non-stationary attention mechanism, a residual connection is added based on the de-stationary attention output:

[0214] output=Y+Attention(Q',K',V',τ,Δ) (24)

[0215] where Q', K', V' are the query, key and value matrices respectively, τ is the scaling factor and Δ is the offset factor.

[0216] Pass the output through the forward propagation network:

[0217] LN2=LN(FeedForward(LN(output))) (25)

[0218] Among them, LN(·) represents normalization and FeedForward represents forward propagation.

[0219] Finally, the final prediction result is obtained through linear layer mapping:

[0220] Output=Projection(LN2) (26)

[0221] In step 5-1, the interactive convolution module uses a double-layer structure, such as Figure 2 (b) shows that its internal process is as follows:

[0222] A1=φ(Conv1(x))⊙Conv2(x) (27)

[0223] A2=φ(Conv2(x))⊙Conv1(x) (28)

[0224] Where Conv1(.) and Conv2(.) are two convolutional layers, and φ represents the GELU activation function.

[0225] Send the activated features into the third convolutional layer:

[0226] x1=Conv3(A1+A2) (29)

[0227] Where Conv3 represents the third convolutional layer, and x1 represents the feature matrix extracted by the interactive convolution module.

[0228] In step 5-1, the PDP convolution module consists of point-by-point convolution, depth-wise convolution, and point-by-point convolution, and finally outputs the GELU activation function. The point convolution size is 1×1, as shown in Figure 2 As shown in (a), the processing flow is as follows:

[0229] Point-wise convolution:

[0230] in, is the output feature after the first point convolution, t is the time step, c' is the output channel index, C is the set of input channels, representing all input feature channels, is the weight of the point convolution, which represents the relationship between different input channels c and output channels c', x t,c is the value of the input feature at time step t and channel c, is the bias term.

[0231] Depthwise Convolution:

[0232] Among them, m represents the spatial position of the convolution kernel, K represents the kernel size, t is the time step, C represents the set of input channels, ω m and b c are the weights and bias terms of the depthwise convolution, Represents the output of the first point convolution.

[0233] Point-wise convolution:

[0234] in Represents the output feature after the second point convolution, t is the time step, c' is the output channel index, C is the set of input channels, representing all input feature channels, is the weight of the second point convolution, which represents the relationship between different input channels c and output channels c', y t,c is the output feature after depth convolution, is the bias term.

[0235] Then x p2 The output is obtained after the activation function:

[0236] x2=GELU(x p2 ) (33)

[0237] Where x2 is the output feature matrix of the PDP convolution module.

[0238] In step 5-4, the specific steps of the multi-head differential attention mechanism are:

[0239] First, the input embedding vector is projected into the query, key, and value space:

[0240] Query vector: Q 1, Q2=h n W Q (34)

[0241] Key vector: K1, K2 = h n W K (35)

[0242] Value vector: V = h n W V (36)

[0243] Among them, W Q ,W K ,W V is a learnable weight matrix.

[0244] The differential attention mechanism is calculated by the following formula:

[0245]

[0246] The first softmax term is the attention score calculated based on the query Q1 and key K1, and the λsoftmax term is the attention score calculated based on the query Q2 and key K2, and is scaled by a learnable scalar λ. In order to synchronize the learning dynamics, the scalar λ is reparameterized as:

[0247] λ=exp(α1·λ1)-exp(α2·λ2)+λ init (38)

[0248] Among them, λ1,λ2 are learnable vectors, λ init is a constant scalar used to initialize λ, and α1, α2 are learnable hyperparameters.

[0249] To further improve the expressiveness of the model, a multi-head attention mechanism is used. That is, for each attention head, a differential attention operation is calculated separately, and the results of all heads are spliced together. The specific steps are as follows:

[0250] For the i-th head, use different weight matrices W i Q , W i K ,W i V To calculate:

[0251] head i =DiffAttn(X;W i Q ,W i K ,W i V ,λ) (39)

[0252] After computing the head output, normalize each head:

[0253]

[0254] where λ init is a constant scalar, LN(·) indicates the normalization operation is applied to each head, represents the output of the i-th attention head.

[0255] Finally, the outputs of all heads are concatenated and processed through a linear transformation W O Get the final output:

[0256]

[0257] Concat(·) means concatenating the outputs of all h heads by columns, W O is a learnable linear projection matrix, and MultiHead(X) represents the output of the final multi-head differential attention.

[0258] The specific steps of the destabilizing attention mechanism in step 5-5 are:

[0259] First, project Y into the query space, key space, and value space:

[0260] Q'=YW Q (42)

[0261] K'=YW K (43)

[0262] V'=YW V (44)

[0263] Among them, W Q ,W K ,W V is a learnable weight matrix, Q', K', V' are query, key and value matrices respectively.

[0264] The scaling factor τ is converted from the unnormalized statistic μ via a multilayer perceptron z ,σ z Learning:

[0265] logτ=MLP(σ Z ,Z) (45)

[0266] Δ=MLP(μ Z ,Z) (46)

[0267] τ is the scaling factor, Δ is the offset factor, and σ z represents the variance, μ zrepresents the mean, the variance and mean come from the normalization operation in the previous step 5-1, and MLP(·) represents a multi-layer perceptron, which is used to extract τ and Δ from the statistical information.

[0268] Combined with the learned scaling factor τ and offset factor Δ, the adjusted attention output matrix is calculated:

[0269]

[0270] Among them, d k is the dimension of the key vector, τ is the scaling factor, Δ is the offset factor, and 1 represents an all-one vector.

[0271] In summary, the present invention, on the one hand, takes into account the various factors that affect the power load, and uses correlation analysis to select the influencing factors that are more closely related to the load value, effectively reducing the feature dimension. Starting from the load data itself, the VMD decomposition method optimized by the Lotus Optimization Algorithm is used to decompose the load data, explore its existing trend characteristics and periodic characteristics, and use these characteristics for auxiliary prediction. On the other hand, a high-precision power load prediction model with dual convolution parallel plus attention mechanism is constructed. This model not only extracts the local features of the data, but also models the global correlation of time series data. Combining these two aspects, high-precision prediction of regional load data is achieved.

[0272] pass Figure 3 and Figure 4 It can be seen that the method proposed in this patent can well complete the prediction task and obtain relatively accurate power forecast data when facing huge and highly fluctuating power loads. Compared with the Informer model, the method proposed in this patent has good performance. When the number of prediction steps is 48, the MAE value is reduced by 52%. Figure 5 It can be seen that the method proposed in the patent of this invention has a small error increase when performing multi-step power load forecasting and has strong stability.

[0273] DCAformer PatchTST TimesNet Informer Transformer 16 84.6040 131.1803 178.0183 187.7460 198.5590 32 90.7787 160.4475 196.6221 208.4414 223.5146 48 103.8312 180.2794 205.7886 236.2747 260.8392 64 121.1811 131.1803 223.4050 276.6770 292.5801

[0274] Table 1 Comparison of MAE values of extended data plus network prediction results

[0275] DCAformer PatchTST TimesNet Informer Transformer 16 122.2882 166.1913 195.3594 254.5637 220.1419 32 168.0903 201.7237 230.1148 264.4472 248.6553 48 188.0621 228.0763 253.6577 276.9057 281.0298 64 202.4211 266.9533 279.7856 304.7340 301.3486

[0276] Table 2 Comparison of MAE values of univariate prediction results

[0277] Table 1 compares the MAE values of the prediction results using the extended dataset and different models, while Table 2 compares the MAE values of the single-variable prediction results. As can be seen from the table, DCAformer has lower MAE values for the 16th, 32th, 48th, and 64th steps, outperforming the other models. Furthermore, its performance remains relatively stable as the number of prediction steps increases. Comparing the results in Tables 1 and 2 also shows that the proposed extended dataset method improves the prediction performance of each model to varying degrees.

[0278] In terms of feature selection, this invention first screens for meteorological factors with a strong correlation with load values and incorporates weekend data. Secondly, the Lotus Optimization Algorithm is used to automatically optimize the parameters for VMD decomposition. The load data is then decomposed using the optimal parameters. Data that reflects the trend and cyclical characteristics of the load data is selected from the decomposed modes and used to assist in forecasting. Regarding the load forecasting model, a hybrid model is constructed that takes into account both local and global characteristics.

[0279] This paper uses the Lotus Optimization Algorithm to automatically optimize the parameters of VMD decomposition. Trend and periodic features are extracted from the decomposed modalities, and meteorological data and weekend-weekday one-hot encoded data are added to form an extended dataset. A DCAformer hybrid network model with dual convolution and attention is constructed for load forecasting, capturing both local and global features and enhancing the extraction of key features.

[0280] The dataset used in this patent application comes from load data for a specific region in 2013, provided by a power grid company in southern China. Data collection was conducted at 15-minute intervals, covering 96 time points daily, for a total of 35,040 data points over four months. All experiments were conducted on a Linux server equipped with four CPUs, an NVIDIA GeForce RTX 3090 GPU, and 64GB of memory. The DCAformer network was implemented in Python using the PyTorch framework.

Claims

1. A power load forecasting method based on load data intrinsic characteristics mining and DCAformer, characterized by: The following steps are involved: Step 1: Collect all factors that may affect power load data; And add weekend data, that is, for data points that fall on Saturday or Sunday, set the value of the Week variable to 1; Otherwise, its value is 0; Step 2: Decompose the original load data into multiple components using the VMD method; Step 3: Use the LEA lotus effect optimization algorithm to find the optimal number of decompositions K and penalty factor α, and use the optimized parameters to decompose the original load data; Step 4: Take the mode with the lowest frequency from the multiple modes obtained by decomposition as the trend feature; Calculate the sample entropy value of each mode separately, take the sample with the lowest entropy value in the intermediate frequency mode as the period feature, integrate the extracted trend feature and period feature with the weekend-weekday one-hot encoded data and meteorological factor data obtained in step 1, and construct an extended data set as the input of the prediction model; Step 5: Divide the preprocessed extended dataset into a training set, a validation set, and a test set. Input the test set data into the constructed DCAformer network model for training to obtain a trained network. Step 6: Send the expanded dataset into the trained DCAformer network to obtain the load forecast results.

2. The method according to claim 1, characterized in that In step 2, the specific steps of using VMD decomposition include Step 2-1) Obtain a constrained optimization problem; Step 2-2) In order to solve the above constrained optimization problem, the constrained variational problem is transformed into an unconstrained variational problem, and the augmented Lagrangian function is introduced using the quadratic penalty term and the Lagrange multiplier method; Step 2-3) The alternating direction multiplier method is used to perform alternating optimization iterations to obtain the final optimization result.

3. The method according to claim 2, characterized in that In step 2-1), when obtaining the constrained optimization problem, the constraint condition is that the sum of all modes is equal to the original signal. The resulting constrained optimization problem is as follows; Among them, u k represents the modal component, ω k represents the center frequency of the kth modal component, K represents the number of decomposed modes, represents the partial derivative operator, δ(t) represents the Dirac function, u k (t) represents the kth modal component decomposed, f(t) represents the original signal, which is the sum of all modal components, and t represents the time variable. Represents the modal u k Do the Hilbert transform and take the derivative, represents the square of the two-norm; In step 2-2), the augmented Lagrangian function is introduced using the quadratic penalty term and the Lagrangian multiplier method: Among them, {u k } and {w k } is the set of all modes and their center frequencies, f(t) is the original signal, δ(t) and are the Dirac function and the partial derivative with respect to time t, K represents the number of decomposed modes, represents the imaginary part of the Hilbert transform kernel, represents the partial derivative operator, δ(t) represents the Dirac function, λ(t) represents the Lagrange multiplier, u k (t) represents the kth modal component decomposed, represents the square of the 2-norm.

4. The method according to claim 2 or 3, characterized in that In steps 2-3), the alternating optimization formula is as follows: Where ω is the frequency variable, f(ω) represents the spectrum of the original signal f(t), and K represents the number of decomposed modes. Represents the nth iteration value of the i-th mode in the frequency domain, It represents the n+1th iteration value of the ith mode in the frequency domain, represents the n+1th iteration value of the kth mode in the frequency domain, represents the nth iteration value of the Lagrange multiplier in the frequency domain, α represents the penalty factor, represents the center frequency of the kth mode in the nth iteration, represents the center frequency of the kth mode at the n+1th iteration.

5. The method according to any one of claims 1 to 3, characterized in that In step 3, the specific steps of using the LEA lotus optimization algorithm to optimize the VMD decomposition parameters are as follows: Step 3-1) Initialization phase: Initialize a dragonfly swarm. The swarm size needs to be pre-set. Each dragonfly represents a candidate solution. The parameters of each candidate solution are the decomposition mode number K and the penalty factor α in VMD. To ensure the effectiveness and rationality of the algorithm, the value range of the mode number K and the penalty factor α needs to be pre-set. In addition, parameters such as the maximum number of iterations, noise tolerance, and frequency initialization method need to be defined. Step 3-2) Evaluate the fitness of each dragonfly: For each dragonfly individual, i.e., each candidate solution, we extract its corresponding decomposition parameter K and penalty factor α. We use these parameters to perform VMD decomposition on the input signal to obtain K modal components. Then, we calculate the envelope entropy value for each modal component. Envelope entropy is an indicator to measure signal complexity and is defined as: Among them, x i is the amplitude of each decomposed mode, H represents the envelope entropy value; Step 3-3) In order to evaluate the quality of each candidate solution, the fitness function is defined as the minimum envelope entropy value of all modal components. The smaller the minimum envelope entropy value, the lower the complexity of the signal and the better the decomposition effect. Therefore, the fitness function can be expressed as: F=min(E1,E2,...,E k ) (7) Among them, E i represents the envelope entropy value of the i-th mode, and F represents the fitness value of the overall decomposition; Steps 3-4) Update the position of each dragonfly based on its fitness value, that is, update the decomposition modal number K and penalty factor α. Update the position by simulating the group behavior of dragonflies, which specifically includes the following five behavioral mechanisms: Separation: Avoid collisions between individuals and adjacent individuals. The calculation formula is as follows: Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the i-th individual, N is the number of adjacent individuals, S i represents the separation vector of the i-th individual; Alignment: The speed of an individual is aligned with the speed of its neighbors, calculated as follows: Among them, V j represents the speed of the jth neighboring individual, N is the number of neighboring individuals, A i Represents the alignment vector of the i-th individual; cohesion: the individual moves towards the center position of the adjacent individual, and the calculation formula is as follows: Among them, X i Indicates the current individual's position, X j represents the position of the jth individual adjacent to the ith individual, N is the number of adjacent individuals, C i represents the cohesion vector of the i-th individual; Food attraction: Individuals move towards food sources, calculated as follows: F i =X food -X i (11) Among them, X i Indicates the current individual's position, X food Indicates the location of the food source, F i Represents the target attraction vector of the i-th individual; enemy avoidance: the individual moves away from the enemy, and the calculation formula is as follows: E i =X enemy -X i (12) Among them, X i Indicates the current individual's position, X enemy Indicates the enemy's position, E i represents the cohesion vector of the i-th individual away from the enemy; In LEA, these mechanisms are used to simulate the behavior of dragonflies in global search to achieve global optimization of the problem; Steps 3-5) Local search phase: To further enhance the algorithm’s local search capability, LEA introduces a local search mechanism based on the movement of water droplets on a lotus leaf. The specific steps are as follows: Local search step size update: Perform local search near the current optimal solution. The search step size will gradually decrease with the increase in the number of iterations to achieve a refined search for the optimal solution. The calculation formula is as follows: in, represents the current solution, g * represents the current optimal solution, Represents the updated solution, R is the step attenuation coefficient, and the calculation formula is as follows: Where t is the current number of iterations and L is the maximum number of iterations; Water droplet movement mechanism: When too many water droplets accumulate in a pit, they overflow and flow to other pits. The overflowing water droplets select the next target pit based on the pit capacity. The formula for updating the water droplet's movement speed and position is as follows: V i t+1 =qV i t (15) Among them, V i t represents the current velocity of the i-th water droplet at the t-th iteration, q is the velocity attenuation coefficient, V i t+1 represents the updated velocity of the i-th water drop after t+1 iterations; represents the current position of the i-th water droplet at the t-th iteration, represents the updated position of the i-th water drop after t+1 iterations; Water droplet overflow mechanism: When too many water droplets accumulate in a pit, they overflow and flow to other pits. The overflowing water droplets will select the next target pit based on the pit capacity. The calculation formula is as follows: Among them, f i represents the fitness value of the current position of the i-th water drop, f max and f min Represents the maximum and minimum fitness values respectively, const is a constant, c i represents the probability factor of the i-th water droplet overflowing and jumping to a new pit. Through the movement and overflow mechanism of water droplets, LEA can more effectively find the optimal solution in local search. Step 3-6) Termination condition: After each iteration, check whether the termination condition is met: the maximum number of iterations is reached or the fitness value reaches the preset convergence threshold; if the termination condition is met, return the current optimal mode number K and penalty factor α as the optimal parameters of VMD decomposition; if the condition is not met, continue iteration.

6. The method according to claim 1 or 2 or 3 or 5, characterized in that The constructed DCAformer network model is specifically as follows: Step 5-1) Normalize the preprocessed data Z in the time dimension to obtain the normalized data x: Calculate the mean: Calculate the variance: where μ Z ,σ Z ∈R C×1 , represents the mean and variance, Z i Represents the data of the i-th time step, ⊙ represents the element-by-element product; Step 5-2) Input the normalized data x into the interactive convolution module and the PDP convolution module respectively to extract local features, and then perform weighted fusion of the features extracted from each module: X=ω1·x1+ω2·x2 (21) Where ω1 and ω2 are learnable weight parameters, corresponding to the weights of x1 and x2 respectively, and ω1+ω2=1; Step 5-3) The entire time series X∈{X1,X2,...,X m }Embedded as independent vectors: h n =Embedding(X :,n ) (22) where X :,n represents the entire time series of each variable indexed by n; Step 5-4) Feed the embedded variables into the differential attention mechanism to extract key features, while adding residual connections to retain some features of the original input data: Y=h n +MultiHead(X) (23) Step 5-5) After introducing the encoder module with the non-stationary attention mechanism, a residual connection is added based on the de-stationary attention output: output=Y+Attention(Q',K',V',τ,Δ) (24) Where Q', K', V' are query, key and value matrices respectively, τ is the scaling factor, Δ is the offset factor, Pass the output through the forward propagation network: LN2=LN(FeedForward(LN(output))) (25) Among them, LN(·) represents normalization, and FeedForward represents forward propagation; Finally, the final prediction result is obtained through linear layer mapping: Output=Projection(LN2) (26).

7. The method according to claim 6, characterized in that In step 5-1, the interactive convolution module workflow is: A1=φ(Conv1(x))⊙Conv2(x) (27) A2=φ(Conv2(x))⊙Conv1(x) (28) Where Conv1(.) and Conv2(.) are two convolutional layers, and φ represents the GELU activation function; Send the activated features into the third convolutional layer: x1=Conv3(A1+A2) (29) Where Conv3 represents the third convolutional layer, and x1 represents the feature matrix extracted by the interactive convolution module.

8. The method according to claim 6, characterized in that In step 5-1, the PDP convolution module workflow is as follows: Point-by-point convolution: in, is the output feature after the first point convolution, t is the time step, c' is the output channel index, C is the set of input channels, representing all input feature channels, is the weight of the point convolution, which represents the relationship between different input channels c and output channels c', x t,c is the value of the input feature at time step t and channel c, is the bias term; Depthwise Convolution: Among them, m represents the spatial position of the convolution kernel, K represents the kernel size, t is the time step, ω m and b c are the weights and bias terms of the depthwise convolution, Represents the output of the first point convolution; Point-wise convolution: in Represents the output feature after the second point convolution, t is the time step, c' is the output channel index, representing all input feature channels, is the weight of the second point convolution, which represents the relationship between different input channels c and output channels c', y t,c is the output feature after depth convolution, is the bias term; Then x p2 The output is obtained after the activation function: x2=GELU(x p2 ) (33) Where x2 is the output feature matrix of the PDP convolution module.

9. The method according to claim 6, characterized in that In step 5-4, the multi-head differential attention mechanism is specifically: First, the input embedding vector is projected into the query, key, and value space: Query vector: Q 1, Q2=h n W Q (34) Key vector: K1, K2 = h n W K (35) Value vector: V = h n W V (36) Among them, W Q ,W K ,W V is a learnable weight matrix; The differential attention mechanism is calculated by the following formula: The first softmax term is the attention score calculated based on the query Q1 and the key K1, and the λ softmax term is the attention score calculated based on the query Q2 and the key K2, and is scaled by a learnable scalar λ; in order to synchronize the learning dynamics, the scalar λ is reparameterized as follows: λ=exp(α1·λ1)-exp(α2·λ2)+λ init (38) Among them, λ1,λ2 are learnable vectors, λ init is a constant scalar used to initialize λ, and α1, α2 are learnable hyperparameters; To further improve the expressiveness of the model, a multi-head attention mechanism is used; that is, for each attention head, a differential attention operation is calculated separately, and the results of all heads are spliced together; the specific steps are as follows: For the i-th head, use different weight matrices W i Q , W i K ,W i V To calculate: head i =DiffAttn(X;W i Q ,W i K ,W i V ,λ) (39) After computing the head output, normalize each head: where λ init is a constant scalar, LN(·) indicates the normalization operation is applied to each head, represents the output of the i-th attention head; Finally, the outputs of all heads are concatenated and processed through a linear transformation W O Get the final output: Concat(·) means concatenating the outputs of all h heads by columns, W O is a learnable linear projection matrix, and MultiHead(X) represents the output of the final multi-head differential attention.

10. The method according to claim 6, characterized in that The specific steps of the destabilizing attention mechanism in step 5-5 are: First, project Y into the query space, key space, and value space: Q'=IS Q (42) K'=IS K (43) V'=IS V (44) Among them, W Q ,W K ,W V is a learnable weight matrix, Q', K', V' are query, key and value matrices respectively; The scaling factor τ is converted from the unnormalized statistic μ via a multilayer perceptron z ,σ z Learning: logτ=MLP(σ Z ,Z) (45) Δ=MLP(μ Z ,Z) (46) τ is the scaling factor, Δ is the offset factor, and σ z represents the variance, μ z represents the mean, the variance and mean come from the normalization operation in the previous step 5-1, and MLP(·) represents a multi-layer perceptron, which is used to extract τ and Δ from statistical information; Combined with the learned scaling factor τ and offset factor Δ, the adjusted attention output matrix is calculated: Among them, d k is the dimension of the key vector, τ is the scaling factor, Δ is the offset factor, and 1 represents an all-one vector.

Images