Non-intrusive load decomposition method and system based on xLSTM-Transform model

Abstract

The invention discloses a non-intrusive load decomposition method and system based on an xLSTM-Transform model, and is applied to the technical field of non-intrusive load decomposition. XLSTM is constructed by combining sLSTM (scalar LSTM) and mLSTM (matrix LSTM), so that the processing capability of the model on time sequence data is enhanced; a causal convolution module is added behind an input layer, a sliding window in a fixed direction of the causal convolution module can more accurately extract a local load mode of an electric appliance while ensuring causality, and future information leakage is avoided; a self-attention mechanism in a Transform architecture is introduced to establish an xLSTM-Transform combination model, historical data is adopted to train the model, and finally, the model is used to perform electric appliance load decomposition. According to the method, a multi-scale electrical load mode can be effectively processed, and the problem of long-term dependence faced by a traditional time sequence model is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of non-intrusive load decomposition, and more particularly to a non-intrusive load decomposition method and system based on an xLSTM-Transformer model. Background Art

[0002] Against the backdrop of global decarbonization, improving energy efficiency and optimizing building energy consumption have become a consensus. In recent years, electricity demand has grown at an average annual rate of approximately 3.4%. Research shows that providing accurate load operation information to users can reduce energy consumption by up to 20%. Feedback from power equipment can help utility companies with load decomposition and demand-side management. It also has applications in fault diagnosis and non-technical loss detection.

[0003] Load decomposition is an important means of monitoring power consumption. There are two common approaches to device load monitoring: intrusive load monitoring (ILM) and non-intrusive load monitoring (NILM). ILM involves installing special sensors on each appliance to monitor its energy consumption and operating status. While providing accurate load readings, ILM is expensive due to the large number of sensors required to obtain load data for each appliance. Furthermore, ILM requires a complex data storage and transmission network. In contrast, NILM is a technology that detects and monitors appliance operation from aggregated signals. It offers the advantages of low hardware cost, easy access to historical data, and user privacy protection. NILM technology not only provides information about the operating patterns and power consumption of electrical devices but also reduces overall energy consumption. Therefore, developing efficient and accurate NILM technology is cost-effective and critical for optimizing demand-side management in smart grids. A typical NILM framework can be implemented through a three-stage process: (a) Data collection: This process collects aggregate demand from smart meters. Typically, data from smart meters is sampled at either a high frequency or a low sampling rate. A low sampling rate helps extract profiles for appliances with higher power ratings, while a high sampling rate helps extract profiles for appliances with lower power ratings. (b) Feature extraction: Load characteristics / signatures are considered as measurable parameters of the total load, which provide information about the operating status and nature of the working equipment. (c) Load identification: This is the last stage, in which the characteristics of the equipment are extracted using algorithms that identify the load.

[0004] Existing NILM algorithms are generally divided into two categories: the first category is called supervised learning methods, which require data from sub-metered appliances or external labeled data from the target building; the second category is called unsupervised learning methods, in which the NILM task does not include any pre-training step and therefore there is no final labeling stage, which is responsible for assigning appropriate device names to the decomposed profiles. Machine learning (ML) and deep learning (DL) are becoming increasingly popular in solving these two decomposition tasks. For example, decision trees, support vector machines, Bayesian, deep neural networks, bidirectional LSTM, and convolutional neural networks (CNNs) are commonly used to supervise NILM. On the other hand, unsupervised methods such as fuzzy clustering, DBSCAN, K-means, deep sparse coding, and factorial hidden Markov have also done excellent work.

[0005] While existing NILM algorithms perform well in certain scenarios, they still have significant limitations when dealing with diverse appliance load characteristics. First, traditional deep learning models (such as RNNs) often suffer from information forgetting when processing long time series and are unable to effectively capture key patterns over long time spans. Second, many models fail to fully consider the multi-scale characteristics of appliance load signals, such as the stable periodicity and short-term high-frequency fluctuations exhibited by air conditioners and dishwashers at different time scales. This results in the model being unable to effectively identify the load characteristics of appliances with complex load patterns. Finally, existing NILM models have not yet fully met the requirements of real-time and causality. In particular, when ensuring that current load forecasts are not affected by future information, many methods still risk leaking future information. Therefore, how to provide a non-invasive load decomposition method and system based on the xLSTM-Transformer model and causal convolution is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a non-intrusive load decomposition method and system based on the xLSTM-Transformer model. By combining sLSTM and mLSTM to construct xLSTM, the model's processing capability for time series data is enhanced. A causal convolution module is added after the input layer to ensure causality while avoiding future information leakage. The self-attention mechanism in the Transformer architecture is introduced to establish an xLSTM-Transformer combined model to effectively process multi-scale electrical load patterns.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A non-intrusive load decomposition method based on the xLSTM-Transformer model includes the following steps:

[0009] S1. Obtain historical load data of electrical appliances and preprocess the historical load data;

[0010] S2. Build an xLSTM-Transformer model with causal convolution. First, normalize and split the multi-channel load sequence into sliding windows. Then, use causal convolution to capture local temporal information. Then, use xLSTM modules, which alternately stack sLSTM and mLSTM, to further extract high-dimensional temporal features. This is then concatenated with the original input and mapped to the Transformer model dimension with positional encoding. The Transformer encoder, consisting of a multi-head self-attention and feedforward network, is responsible for capturing global long-range dependencies.

[0011] S3. Use historical household electricity consumption dataset to train the load decomposition model;

[0012] S4. Use the trained load decomposition model to identify and decompose the historical total household electricity load to obtain the load of different electrical appliances.

[0013] Optionally, the causal convolution in S2 is specifically:

[0014] The causal convolution layer is placed between the original input and the xLSTM module as the first-stage feature extractor. The causal convolution adopts a hierarchical structure of input layer, double hidden layer and output layer. In the one-dimensional convolution operation, only the current and previous historical data are used to meet the causal constraint. The mathematical form is:

[0015]

[0016] Where w k ∈R is the learnable convolution kernel weight, K is the convolution kernel size, x t-k Represents the input signal at a historical moment, y t Represents the output signal at the current moment; an element in the output sequence can only depend on the elements before it in the input sequence. To ensure that the output tensor has the same length as the output tensor, the input tensor is padded with zeros on the left to ensure that future information is not leaked; in view of the multi-scale characteristics of electrical loads, a serial processing strategy is adopted, first using a short kernel K = 3 to extract local details, and then using a long kernel K = 5 to model long-term patterns.

[0017] Optionally, the xLSTM module based on the msm structure combination in S2 is specifically:

[0018] The combined structure of mLSTM-sLSTM-mLSTM is adopted. The first mLSTM layer quickly captures global context information at the beginning of sequence input through the matrix memory mechanism, and efficiently extracts high-dimensional correlation features in the time series. The middle layer sLSTM performs further processing and enhances the stability of model training and the ability to model local dynamic changes through the exponential gating mechanism and state normalization mechanism. The last layer mLSTM further refines the feature expression, aggregates context information, and provides sequence representation for the subsequent Transformer module.

[0019] Optionally, mLSTM is:

[0020] mLSTM introduces matrix memory units and a covariance update mechanism for key-value pair storage, increasing the model's memory capacity. The gating mechanism works in conjunction with the covariance update rule to effectively manage memory updates. By removing connections between hidden layers, mLSTM performs parallel operations, accelerating training and inference. The mLSTM operation process is as follows:

[0021] C t =f t C t-1 +i t v t k t

[0022] Where c t is the cell state at time step t, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, v t is the value matrix, k t is the key matrix; the normalization process of mLSTM is:

[0023] n t =f t n t-1 +i t k t

[0024]

[0025] Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, o t is the output gate, h t is the hidden state at time step t, q t is the query matrix;

[0026] The query matrix q of mLSTM t , key matrix kt Sum value matrix v t is defined as follows:

[0027] q t =W q x t +b q

[0028]

[0029] v t =W v x t +b v

[0030]

[0031] Where W q is the weight matrix of query vector mapping, W k is the weight matrix of key vector mapping, W v is the weight matrix of the value vector mapping, b q is the bias term for query vector mapping, b k is the bias of the key vector mapping, b v is the bias term for the value vector mapping;

[0032] x t is the input vector; i t is the input gate, which is used to control the extent to which new information is introduced into memory, w i is the weight matrix of the input gate, b i is the bias term of the input gate; the forget gate determines the extent to which the memory of the previous time step should be retained and is defined as:

[0033]

[0034] Where w f is the weight matrix of the forget gate, b f is the bias term of the forget gate, σ is the activation function; the output gate controls the content and amount passed from the retrieved memory to the hidden state, and is defined as:

[0035]

[0036] Where w o is the weight matrix of the output gate, b o is the bias term of the output gate.

[0037] Optionally, sLSTM is:

[0038] sLSTM combines exponential gating, memory mixing, and stabilization mechanisms to enhance the model's ability to make effective storage decisions, handle rare tag predictions in natural language processing, capture complex dependencies, and maintain robustness during training and inference. For sLSTM, the cyclic relationship between input and state is described as:

[0039] c t =f t c t-1 +i t z t

[0040] Where c t is the cell state at time step t, which preserves the long-term memory of the network, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, z t Used to control the amount of input added to the cell state and the previous hidden state h t-1 , specifically defined as:

[0041]

[0042] Where x t is the input vector, is the activation function, is the weight matrix, r z is the recursive weight matrix, b z is the bias term; sLSTM uses the normalized state as follows:

[0043] n t =f t n t-1 +i t

[0044] Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, which normalizes the update of the unit state; the hidden state is used for recurrent connection, and the formula is:

[0045]

[0046] In the formula, o t is the output gate; input gate i t Controls the extent to which new information is added to the cell state and is defined as:

[0047]

[0048] Where r i is the input gate i tThe recursive weight matrix of i is the input gate i t Bias term; forget gate f t The cell state c at the previous time step t-1 The degree of preservation is expressed as follows:

[0049]

[0050] Where, It is the forget gate f t The weight matrix, r f It is the forget gate f t The recursive weight matrix of f It is the forget gate f t Bias term of output gate o t Controls the flow of information from the cell state to the hidden state:

[0051]

[0052] Where, is the output gate o t The weight matrix applied to the current input, r o is used for the previous hidden state h t-1 The output gate o t The recursive weight matrix of o is the output gate o t To provide numerical stability of the exponential gate, the forget gate f t and input gate i t Combined into another state m t , defined as:

[0053] m t =max(log(f t )+m t-1 ,log(i t ))

[0054]

[0055] f t ′=exp(log(f t )+m t-1 -m t )

[0056] Where i t ′ is the stabilized input gate, which is a rescaled version of the original input gate; f t ′ is the stabilized forget gate, which is a rescaled version of the original forget gate.

[0057] Optionally, the position code in S2 is:

[0058]

[0059] Where p represents the position index, d model Indicates the dimension of the input feature, j indicates the dimensional encoding of the input feature, and its value range is [0,d model / 2].

[0060] Optionally, the Transformer module in S4 is:

[0061] The Transformer module performs three different linear mappings on the input variable X through a single-head self-attention mechanism to generate a query matrix, a key matrix, and a value matrix. The dimensions of these three matrices are d k ; The attention score is calculated by normalized dot product operation, and its formula is:

[0062]

[0063] Where, d k is the dimension of the key matrix, which is used to scale the dot product results to avoid the gradient vanishing problem caused by excessive values; the softmax function normalizes the dot product results into a probability distribution, representing the attention weight of each position; the final output is a weighted value matrix based on the input, and the weight is determined by the attention distribution; the input matrix generates a set of multiple query matrices, key matrices, and value matrices through a variety of different linear mappings, and the attention results of these outputs are spliced ​​together to enhance the performance of the model. The expression of the multi-head attention mechanism is as follows:

[0064] multihead=concat(head1,...,head h )W

[0065] Where head1 represents the attention result of the first head, h is the number of multi-head attention, concat is the operation of splicing the results of each head into a whole, and W is the weight matrix used to perform a linear transformation on the splicing result, which is used to integrate the information of each head to generate the final output.

[0066] Optionally, a sequence-to-point learning data partitioning mechanism is used to implement the training task of the xLSTM-Transformer model. The aggregated power data is partitioned using a window size of a preset value and a step size of 1, and multiple data windows are obtained by overlapping.

[0067] A non-intrusive load decomposition system based on an xLSTM-Transformer model implements the aforementioned non-intrusive load decomposition method based on an xLSTM-Transformer model and causal convolution, including:

[0068] Load data collection module, used to collect historical load data of electrical appliances;

[0069] Data preprocessing module, used to preprocess the historical load data of electrical appliances;

[0070] The xLSTM-Transformer prediction model performs load decomposition based on the load data of electrical appliances;

[0071] Among them, the xLSTM-Transformer prediction model includes a causal convolution module, an xLSTM module, a position encoding module, a Transformer module and a linear layer connected in sequence.

[0072] Optionally, the xLSTM module includes a first mLSTM unit, an sLSTM unit, and a second mLSTM unit connected in sequence.

[0073] It can be seen from the above technical solution that, compared with the prior art, the present invention provides a non-intrusive load decomposition method and system based on the xLSTM-Transformer model, which has the following beneficial effects:

[0074] 1. This invention leverages the fixed-direction sliding window feature of causal convolution to naturally meet causal requirements in time series, avoiding the leakage of future information and ensuring the accuracy of model predictions. Furthermore, causal convolution can efficiently extract local patterns, offering significant advantages for decomposing electrical loads with short-term high power peaks or stable fluctuating load patterns.

[0075] 2. This paper introduces xLSTM (extended long short-term memory) networks. By combining two different types of LSTMs, sLSTM (scalar LSTM) and mLSTM (matrix LSTM), the model's ability to process time series data is enhanced. The sLSTM block utilizes exponential gating and memory hybrid techniques, while the mLSTM block employs matrix memory and covariance update rules. The combination of the two provides stronger sequence modeling capabilities and higher storage capacity, enabling more efficient processing of high-dimensional time series data.

[0076] 3. This paper introduces the Transformer architecture based on xLSTM and uses the self-attention mechanism to process features at different time scales, thereby enhancing the model's adaptability to different energy consumption patterns of electrical appliances. This combined architecture enables the model to simultaneously utilize the advantages of xLSTM in sequence modeling and the Transformer's ability to process long-distance dependencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0078] Figure 1 is a flow chart of the non-invasive load decomposition method of the present invention;

[0079] Figure 2 A comparison diagram of intrusive load decomposition and non-intrusive load decomposition in the prior art;

[0080] Figure 3 This is a flow chart of non-intrusive load decomposition in the prior art;

[0081] Figure 4 Schematic diagram of the xLSTM-Transformer framework of the present invention;

[0082] Figure 5 This is the causal convolution hierarchical structure diagram of the present invention;

[0083] Figure 6 This is the sLSTM architecture diagram of the present invention;

[0084] Figure 7 This is the mLSTM architecture diagram of the present invention;

[0085] Figure 8 This is the causal convolution and xLSTM module architecture diagram of the present invention;

[0086] Figure 9 This is the Transformer module architecture diagram of the present invention;

[0087] Figure 10 Schematic diagram of the sequence-to-point learning framework of the present invention;

[0088] Figure 11 Schematic diagram of the power timing characteristics and total power after superposition of typical household appliances in an embodiment of the present invention;

[0089] Figure 12 Schematic diagram of the load decomposition result of refrigerator power in an embodiment of the present invention;

[0090] Figure 13 This is a schematic diagram of the load decomposition effect of the xLSTM-Transformer model on a television set according to an embodiment of the present invention;

[0091] Figure 14This is a schematic diagram of the load decomposition effect of the xLSTM-Transformer model on a dishwasher in an embodiment of the present invention;

[0092] Figure 15 Schematic diagram of the training loss and validation loss of the xLSTM-Transformer model in the refrigerator scenario in an embodiment of the present invention. DETAILED DESCRIPTION

[0093] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0094] The embodiment of the present invention discloses a non-intrusive load decomposition method based on the xLSTM-Transformer model. Figure 1 As shown, the following steps are included:

[0095] S1. Obtain historical load data of electrical appliances and preprocess the historical load data;

[0096] S2. Build an xLSTM-Transformer model with causal convolution. First, normalize and split the multi-channel load sequence into sliding windows. Then, use causal convolution to capture local temporal information. Then, use xLSTM modules, which alternately stack sLSTM and mLSTM, to further extract high-dimensional temporal features. This is then concatenated with the original input and mapped to the Transformer model dimension with positional encoding. The Transformer encoder, consisting of a multi-head self-attention and feedforward network, is responsible for capturing global long-range dependencies.

[0097] S3. Use historical household electricity consumption dataset to train the load decomposition model;

[0098] S4. Use the trained load decomposition model to identify and decompose the historical total household electricity load to obtain the load of different electrical appliances.

[0099] In the prior art, the difference between intrusive load monitoring (ILM) and non-intrusive load monitoring (NILM) is as follows: Figure 2 As shown in the figure, the process of conventional non-intrusive load monitoring (NILM) is as follows Figure 3 As shown, the xLSTM-Transformer model framework proposed in the embodiment of the present invention is as follows Figure 4 shown.

[0100] Further, such as Figure 5As shown, the causal convolution in S2 is specifically:

[0101] The causal convolution layer is placed between the original input and the xLSTM module as the first-stage feature extractor. The causal convolution adopts a hierarchical structure of input layer, double hidden layer and output layer. In the one-dimensional convolution operation, only the current and previous historical data are used to meet the causal constraint. The mathematical form is:

[0102]

[0103] Where w k ∈R is the learnable convolution kernel weight, K is the convolution kernel size, x t-k Represents the input signal at a historical moment, y t Represents the output signal at the current moment; an element in the output sequence can only depend on the elements before it in the input sequence. To ensure that the output tensor has the same length as the output tensor, the input tensor is padded with zeros on the left to ensure that future information is not leaked; in view of the multi-scale characteristics of electrical loads, a serial processing strategy is adopted, first using a short kernel K = 3 to extract local details (time scale of 0.1-1 second), and then using a long kernel K = 5 to model long-period patterns (time scale of 10-60 seconds).

[0104] In non-intrusive load decomposition scenarios, the model only has access to current and historical load values. Leaking future information would lead to prediction bias and undermine causality. Therefore, this embodiment of the present invention introduces causal convolution. The output of causal convolution not only ensures causal consistency but also provides a multi-scale, locally focused input representation for the subsequent xLSTM and Transformer layers.

[0105] Further, such as Figure 8 As shown, the xLSTM module based on the msm structure combination in S2 is specifically:

[0106] The combined structure of mLSTM-sLSTM-mLSTM is adopted. The first mLSTM layer quickly captures global context information at the beginning of sequence input through the matrix memory mechanism, and efficiently extracts high-dimensional correlation features in the time series. The middle layer sLSTM performs further processing and enhances the stability of model training and the ability to model local dynamic changes through the exponential gating mechanism and state normalization mechanism. The last layer mLSTM further refines the feature expression, aggregates context information, and provides sequence representation for the subsequent Transformer module.

[0107] Further, such as Figure 7 As shown, mLSTM is specifically:

[0108] mLSTM (Matrix Long Short-Term Memory, mLSTM) introduces matrix memory units and a covariance update mechanism for key-value pair storage, increasing the model's memory capacity. The gating mechanism works in conjunction with the covariance update rule to effectively manage memory updates. By removing connections between hidden layers, mLSTM performs parallel operations, speeding up training and inference processes. These improvements enable mLSTM to efficiently store and retrieve information, making it ideal for sequence modeling tasks that require large memory capacity, such as language modeling, speech recognition, and time series prediction. mLSTM represents an important advancement in recurrent neural networks and effectively addresses the challenges of complex sequence modeling. The operating process of mLSTM is as follows:

[0109] C t =f t C t-1 +i t v t k t

[0110] Where c t is the cell state at time step t, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, v t is the value matrix, k t is the key matrix; the normalization process of mLSTM is:

[0111] n t =f t n t-1 +i t k t

[0112]

[0113] Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, o t is the output gate, h t is the hidden state at time step t, q t is the query matrix;

[0114] The query matrix q of mLSTM t , key matrix k t Sum value matrix v t is defined as follows:

[0115] q t =W q x t +b q

[0116]

[0117] v t =W v x t +b v

[0118]

[0119] Where W q is the weight matrix of query vector mapping, W k is the weight matrix of key vector mapping, W v is the weight matrix of the value vector mapping, b q is the bias term for query vector mapping, b k is the bias of the key vector mapping, b v is the bias term for the value vector mapping;

[0120] x t is the input vector; i t is the input gate, which is used to control the extent to which new information is introduced into memory, w i is the weight matrix of the input gate, b i is the bias term of the input gate; the forget gate determines the extent to which the memory of the previous time step should be retained and is defined as:

[0121]

[0122] Where w f is the weight matrix of the forget gate, b f is the bias term of the forget gate, σ is the activation function; the output gate controls the content and amount passed from the retrieved memory to the hidden state, and is defined as:

[0123]

[0124] Where w o is the weight matrix of the output gate, b o is the bias term of the output gate.

[0125] Further, such as Figure 6 As shown, sLSTM is specifically:

[0126] sLSTM (Stabilized Long Short-Term Memory, sLSTM) is an advanced variant of the traditional LSTM architecture. It combines exponential gating, memory mixing, and stabilization mechanisms to enhance the model's ability to make effective storage decisions, handle rare tag predictions in natural language processing, capture complex dependencies, and maintain robustness during training and inference. For sLSTM, the recurrent relationship between input and state is described as:

[0127] c t =f t c t-1 +i t z t

[0128] Where c t is the cell state at time step t, which preserves the long-term memory of the network, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, z t Used to control the amount of input added to the cell state and the previous hidden state h t-1 , specifically defined as:

[0129]

[0130] Where x t is the input vector, is the activation function, is the weight matrix, r z is the recursive weight matrix, b z is the bias term; sLSTM uses the normalized state as follows:

[0131] n t =f t n t-1 +i t

[0132] Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, which normalizes the update of the unit state; the hidden state is used for recurrent connection, and the formula is:

[0133]

[0134] In the formula, o t is the output gate; input gate i t Controls the extent to which new information is added to the cell state and is defined as:

[0135]

[0136] Where r i is the input gate i t The recursive weight matrix of i is the input gate i t Bias term; forget gate f t The cell state c at the previous time step t-1 The degree of preservation is expressed as follows:

[0137]

[0138] Where, It is the forget gate f t The weight matrix, r f It is the forget gate f t The recursive weight matrix of f It is the forget gate f t Bias term of output gate o t Controls the flow of information from the cell state to the hidden state:

[0139]

[0140] Where, is the output gate o t The weight matrix applied to the current input, r o is used for the previous hidden state h t-1 The output gate o t The recursive weight matrix of o is the output gate o t To provide numerical stability of the exponential gate, the forget gate f t and input gate i t Combined into another state m t , defined as:

[0141] m t =max(log(f t )+m t-1 ,log(i t ))

[0142]

[0143] f t ′=exp(log(f t )+m t-1 -m t )

[0144] Where i t ′ is the stabilized input gate, which is a rescaled version of the original input gate; f t′ is the stabilized forget gate, which is a rescaled version of the original forget gate.

[0145] Furthermore, the position code in S2 is specifically:

[0146]

[0147] Where p represents the position index, d model Indicates the dimension of the input feature, j indicates the dimensional encoding of the input feature, and its value range is [0,d model / 2]. In the embodiment of the present invention, the mechanism of the Transformer model is different from that of the LSTM model. It does not process data sequentially, but processes the entire sequence simultaneously in parallel. This parallel processing method requires providing position information for the input data. Therefore, this embodiment uses cosine and sine functions to perform positional encoding (PE).

[0148] Further, such as Figure 9 As shown, the Transformer module in S2 is specifically:

[0149] The Transformer module performs three different linear mappings on the input variable X through a single-head self-attention mechanism to generate a query matrix, a key matrix, and a value matrix. The dimensions of these three matrices are d k ; The attention score is calculated by normalized dot product operation, and its formula is:

[0150]

[0151] Where, d k is the dimension of the key matrix, which is used to scale the dot product results to avoid the gradient vanishing problem caused by excessive values; the softmax function normalizes the dot product results into a probability distribution, representing the attention weight of each position; the final output is a weighted value matrix based on the input, and the weight is determined by the attention distribution; the input matrix generates a set of multiple query matrices, key matrices, and value matrices through a variety of different linear mappings, and the attention results of these outputs are spliced ​​together to enhance the performance of the model. The expression of the multi-head attention mechanism is as follows:

[0152] multihead=concat(head1,...,head h )W

[0153] In the formula, head1 represents the attention result of the first head, h is the number of multi-head attention, concat is the operation of splicing the results of each head into a whole, and W is the weight matrix used to perform linear transformation on the splicing result, which is used to integrate the information of each head to generate the final output. In this embodiment of the present invention, when the input variable is X = {x1, x2, ..., x n}, the formula for calculating attention distribution is as follows:

[0154] α i =softmax(h(x i ,p))

[0155] Among them, α i Represents the i-th variable x i The attention distribution weight is also a probability distribution, which means α1+α2+…+α n = 1. Here is x i The attention score of the i-th input variable is determined by a predefined vector p. There are two methods for calculating the attention score:

[0156]

[0157] Among them, s(x i ,p) is the attention score, W is the hidden matrix between layers;

[0158] The input variable x i Multiplying by the obtained attention distribution and then summing, we can get the following formula:

[0159]

[0160] Among them, Z represents the new representation of the input variable after the attention mechanism is processed, and att(X,p) is the input variable x i and the matrix product of weight p.

[0161] In this embodiment of the present invention, the Transformer module performs three different linear mappings on the input variable X through a single-head self-attention mechanism to generate the query matrix, key matrix, and value matrix as follows:

[0162] Q=Linear(X)=XW Q

[0163] K=Linear(X)=XW K

[0164] V=Linear(X)=XW V

[0165] In the formula, Linear represents the linear transformation layer (fully connected layer), WQ is the query transformation weight matrix, W K is the key transformation weight matrix, W V is the value transformation weight matrix.

[0166] Further, such as Figure 10 As shown in the figure, the sequence-to-point learning data partitioning mechanism is used to implement the training task of the xLSTM-Transformer model. The aggregated power data is partitioned by using a preset window size and a step size of 1, and multiple data windows are obtained by overlapping.

[0167] Compared with the traditional sequence-to-sequence (S2S) model, sequence-to-point learning can effectively reduce the error accumulation problem caused by long-term dependencies by focusing on the correlation of local temporal features, while improving the modeling accuracy of transient power events (such as the start and stop of electric kettles) and steady-state operating modes (such as periodic fluctuations of refrigerators).

[0168] In one embodiment of the present invention, the effectiveness of the xLSTM-Transformer model is verified. Specifically:

[0169] The dataset uses the UK-DALE dataset, an open-access dataset that records appliance-level electricity consumption data for UK households. This dataset has become a benchmark for non-intrusive load decomposition research. The UK-DALE dataset records total active power consumption data for five UK households from November 2012 to January 2015, along with active power consumption data for over ten appliance types. The dataset is sampled at 16 kHz (for the entire household) and 1 / 6 Hz (for individual appliances), making it the first open-access UK dataset with this temporal resolution.

[0170] In this example, five appliances were selected for research: refrigerators, electric kettles, televisions, dishwashers, and washing machines. The refrigerator is a periodic appliance with simple switching states and low power consumption. The electric kettle is an appliance with high power consumption, short operating time, and simple switching states. The television, as a frequently used household appliance, exhibits a significant alternation between bursts of high power consumption (such as video playback) and low-power standby mode, effectively validating the model's ability to capture complex time series dynamics. Dishwashers and washing machines are more complex appliances with multiple switching states, high power consumption, and long operating times. This selection of appliances not only captures the diversity of common household electrical appliances but also fully demonstrates the challenges posed by different energy consumption patterns (such as continuous, intermittent, and transient) to load decomposition algorithms. This provides a scientific basis for comprehensively evaluating the generalization performance of the xLSTM-Transformer model and aligns with the device selection criteria used in existing research.

[0171] Among the five studied appliances, refrigerators and kettles have only two states: on / off. Washing machines, dishwashers, and microwave ovens have multiple operating states and varying operating times. Therefore, the on / off state of each appliance is determined by a simple comparison between the current power data and the on-state power threshold. An appliance is considered on if the current power exceeds the on-state power threshold, and a state change is considered valid only if the duration exceeds the minimum on and off durations. Table 1 specifies the maximum power, on threshold, and minimum on and off times for each appliance in the experiment:

[0172] Table 1 Maximum power, on threshold, minimum on and off time of each appliance

[0173]

[0174] like Figure 11As shown in the figure, based on the UK-DALE dataset, the power time series characteristics of typical household appliances and their total power after superposition are revealed, and the heterogeneity of power patterns of different devices is quantified through comparative analysis. The visualization results clearly show the operating characteristics of five typical types of devices: refrigerators exhibit quasi-periodic low power consumption fluctuations (100-200W), and their operating state causes the power curve to exhibit regular oscillations; electric kettles produce instantaneous high-power pulses (2000-3000W), the duration of which strictly corresponds to the rapid boiling process of 2-5 minutes; dishwashers and washing machines exhibit multi-stage power evolution (200-2500W), which fully maps the discrete operating stages within the 1-2 hour operating cycle, including key processes such as water filling, heating, and drainage; and microwave ovens' intermittent power pulses (800-1500W) are significantly correlated with single operating durations of generally less than 10 minutes. The superimposed total power curve effectively captures the complex coupling effects of multiple devices operating concurrently. The 1 / 6Hz sampling strategy employed by the UK-DALE dataset successfully preserves both transient power events (such as the steep power transitions during the start-up and shutdown of electric kettles) and steady-state operating modes (such as the smooth periodic oscillations of refrigerators), providing a reliable data foundation for building a high-fidelity, non-intrusive load decomposition model. The results validate the device selection criteria across three dimensions: power magnitude, timing characteristics, and operating cycle. The selected devices systematically capture the typical dynamic characteristics of residential electricity usage scenarios.

[0175] In this example, the hardware environment is a server with two NVIDIA GeForce RTX 3090 GPUs, and the software platform is the Windows 10 Professional operating system, Python 3.8.0 (64-bit), and the TensorFlow 2.11.0 deep learning framework. The model disclosed in this example is compared with the LSTM, XLSTM, and Transformer models in the prior art. Specifically:

[0176] To improve the model's ability to extract electrical load characteristics and model temporal relationships, the present invention uses a deep neural network model that integrates a hybrid long short-term memory network (xLSTM) with a Transformer structure. The hyperparameters are set as follows:

[0177] Input dimension (input_size=1): indicates that the number of input features for each time step is 1, which is suitable for prediction tasks of univariate time series data (such as refrigerator temperature series).

[0178] Block header dimension and number of heads (head_size = 8, num_heads = 4): The xLSTM module uses a block diagonal structure, dividing the hidden layer vector into num_head heads, each with a dimension of head_size. The total hidden layer dimension is hidden_size = head_size * num_heads = 8 × 4 = 32. Multiple heads (i.e., parallel subspaces) can process information in parallel, helping the model learn features from different subspaces. Here, four heads are used, each with a dimension of 8, to keep the hidden layer dimension moderate, balancing model capacity and computational efficiency.

[0179] xLSTM module layer type (layers = 'msm'): Indicates that the xLSTM consists of three blocks in sequence: mLSTM → sLSTM → mLSTM. mLST has enhanced matrix memory capabilities, while sLST has traditional state tracking capabilities. This embodiment uses alternating combinations of m, s, and m to leverage the large-capacity parallelism of mLSTM while retaining the memory mixing capabilities of sLSTM.

[0180] Projection factor (proj_factor_slstm=4 / 3, proj_factor_mlstm=2): The sLSTM block in the xLSTM upsamples the hidden state by a ratio of 4 / 3, and the mLSTM block upsamples the hidden state by 2. That is, the intermediate dimension of the sLSTM block is expanded to 1.33 times the original dimension, and the mLSTM is expanded to 2 times. This can effectively increase the intermediate feature capacity.

[0181] Transformer input dimension (d_model = 128): The feature dimension inside the Transformer module is 128, which controls the parameter scale while ensuring expressiveness. 128 is divisible by the number of heads (nhead = 2), which meets the requirements of multi-head attention.

[0182] Number of multi-head attention heads (nhead = 2): Transformer uses 2 attention heads per layer, corresponding to a dimension of 128 / 2 = 64 per head; multi-head attention allows the model to extract information from different subspaces; in this embodiment, 2 heads are selected to keep the model lightweight while keeping the dimension of each head moderate.

[0183] Number of Transformer layers (num_layers = 2): A two-layer Transformer encoder is used to extract features from the sequence. The multi-layer structure helps capture deep temporal relationships. At the same time, the number of layers should not be too large to avoid overfitting and computational overhead.

[0184] Feedforward network dimension (dim_feedforward=256): The hidden layer dimension of the feedforward network (Feedforward) in the Transformer sublayer is 256, balancing complexity and performance.

[0185] Output dimension (output_dim=1): The model ultimately outputs a scalar, corresponding to the single-step prediction target (such as the value at the next time point); the output linear layer is defined in the code as nn.Linear(d_model, output_dim).

[0186] Dropout (dropout = 0.1): A dropout rate of 0.1 is used for regularization in Transformer.

[0187] The input and output data dimensions of each module of the model are shown in Table 2:

[0188] Table 2 Data dimensions and hyperparameters of each module

[0189]

[0190]

[0191] In order to evaluate the performance of load decomposition, the embodiment of the present invention uses the normalized mean absolute error (NMAE), mean square error (MSE), and root mean square error (RMSE) for evaluation, which are:

[0192]

[0193] Where x t Represents the true value of the device at time t; represents the predicted value of the device at time t; r max and r min are the maximum and minimum values ​​of the sampling points respectively.

[0194] Based on the UK-DALE dataset, the embodiment of the present invention compares the performance of LSTM, Transformer, xLSTM and its fusion model xLSTM-Transformer in the non-intrusive load decomposition task. The comparison results are shown in Table 3 through the three indicators of mean square error (MSE), root mean square error (RMSE) and normalized mean absolute error (NMAE) (as shown in Table 1). The load decomposition curve of the model is shown in Table 3. Figure 11-14 shown.

[0195] Table 3 Performance evaluation results of each model

[0196]

[0197]

[0198] As can be seen, the xLSTM-Transformer model achieved optimal performance across all appliance scenarios. For the refrigerator, for example, its MSE (0.0291) was 49.6% lower than the traditional LSTM (0.0577), while its RMSE (0.1704) and NMAE (0.0785) were improved by 29.1% and 49.0%, respectively. Similarly, in the television power decomposition task, the xLSTM-Transformer achieved a significantly lower RMSE (9.1237) than the other models, demonstrating its robustness in high power fluctuation scenarios. Notably, in the microwave oven decomposition task, the performance differences among the models were relatively small (MSE range: 0.0011–0.0012), presumably due to its relatively simple power model and less noise interference.

[0199] Figure 12 The load decomposition results of refrigerator power are shown. In the range of 700-900 sampling points, the load decomposition curve of xLSTM-Transformer is most consistent with the true value (True), especially during the peak period (such as sampling points 820-860), where the error is significantly lower than that of LSTM and Transformer. Similarly, Figure 13 and Figure 14 The results show that the xLSTM-Transformer model can more accurately capture short-term high-power pulses (such as the moment a TV is turned on) and long-term low-power steady states (such as during the dishwasher cleaning phase). This advantage is attributed to the model's fusion of the local feature extraction capabilities of the xLSTM and the global attention mechanism of the Transformer, thereby taking into account both the long-term and short-term characteristics of temporal dependencies. The difficulty of decomposition varies significantly between different appliances. The dishwasher has the lowest RMSE (0.0482) and NMAE (0.0073), indicating that its power pattern is easier to model. However, the performance of the refrigerator model is relatively limited due to its frequent start-stop and large power fluctuations. Despite this, the xLSTM-Transformer model demonstrates stability in various scenarios, validating its generalization capabilities.

[0200] The trends of the training loss and validation loss of the xLSTM-Transformer model in the refrigerator scenario as the training rounds change are as follows: Figure 15As shown in the figure, the training loss and validation loss decrease rapidly in the initial stage, and then the rate of decrease gradually slows down and eventually stabilizes at a low level. The training loss converges to approximately 0.0249 in the final round, and the validation loss converges to approximately 0.0251. The two values ​​are very close and always fluctuate at the same order of magnitude, indicating that the model does not have obvious overfitting. It is worth noting that in the middle and late stages of training (around the 110th round), the validation loss shows a slight upward fluctuation, but the fluctuation is small and quickly recovers to decline, and the overall trend remains stable. These changes indicate that the loss function of the model in the refrigerator scenario converges well, and the performance on the training set and validation set is basically the same.

[0201] and Figure 1 Corresponding to the method shown, an embodiment of the present invention further discloses a non-intrusive load decomposition system based on the xLSTM-Transformer model, which executes the above-mentioned non-intrusive load decomposition method, including:

[0202] Load data acquisition module, used to collect load data of electrical appliances;

[0203] Data preprocessing module, used for preprocessing the load data of electrical appliances;

[0204] The xLSTM-Transformer prediction model performs load decomposition based on the load data of electrical appliances;

[0205] Among them, the xLSTM-Transformer prediction model includes a causal convolution module, an xLSTM module, a position encoding module, a Transformer module and a linear layer connected in sequence.

[0206] Furthermore, the xLSTM module includes a first mLSTM unit, an sLSTM unit, and a second mLSTM unit connected in sequence.

[0207] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0208] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A non-intrusive load decomposition method based on the xLSTM-Transformer model, characterized in that: The following steps are involved: S1. Obtain historical load data of electrical appliances and preprocess the historical load data; S2. Build an xLSTM-Transformer model with causal convolution. First, normalize and split the multi-channel load sequence into sliding windows. Then, use causal convolution to capture local temporal information. Then, use xLSTM modules, which alternately stack sLSTM and mLSTM, to further extract high-dimensional temporal features. This is then concatenated with the original input and mapped to the Transformer model dimension with positional encoding. The Transformer encoder, consisting of a multi-head self-attention and feedforward network, is responsible for capturing global long-range dependencies. S3. Use historical household electricity consumption dataset to train the load decomposition model; S4. Use the trained load decomposition model to identify and decompose the historical total household electricity load to obtain the load of different electrical appliances.

2. A non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 1, characterized in that: The causal convolution in S2 is specifically: The causal convolution layer is placed between the original input and the xLSTM module. As the first-stage feature extractor, the causal convolution adopts a hierarchical structure of input layer, double hidden layer, and output layer. In the one-dimensional convolution operation, only the current and previous historical data are used to meet the causal constraint. The mathematical form is: Where w k ∈R is the learnable convolution kernel weight, K is the convolution kernel size, x t-k Represents the input signal at a historical moment, y t Represents the output signal at the current moment; an element in the output sequence can only depend on the elements before it in the input sequence. To ensure that the output tensor has the same length as the output tensor, the input tensor is padded with zeros on the left to ensure that future information is not leaked; in view of the multi-scale characteristics of electrical loads, a serial processing strategy is adopted, first using a short kernel K = 3 to extract local details, and then using a long kernel K = 5 to model long-term patterns.

3. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 1 is characterized in that: The xLSTM module based on the msm structure combination in S2 is specifically: The combined structure of mLSTM-sLSTM-mLSTM is adopted. The first mLSTM layer uses a matrix memory mechanism to quickly capture global context information at the beginning of sequence input and efficiently extract high-dimensional correlation features in time series. The middle layer sLSTM performs further processing, enhancing the stability of model training and improving the ability to model local dynamic changes through exponential gating and state normalization mechanisms; the last layer mLSTM further refines feature expression, aggregates contextual information, and provides sequence representation for subsequent Transformer modules.

4. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 3 is characterized in that: mLSTM is specifically: mLSTM introduces matrix memory units and a covariance update mechanism for key-value pair storage, increasing the model's memory capacity. The gating mechanism works in conjunction with the covariance update rule to effectively manage memory updates. By removing connections between hidden layers, mLSTM performs parallel operations, accelerating training and inference. The mLSTM operation process is as follows: C t =f t C t-1 +i t v t k t Where c t is the cell state at time step t, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, v t is the value matrix, k t is the key matrix; the normalization process of mLSTM is: n t =f t n t-1 +i t k t Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, o t is the output gate, h t is the hidden state at time step t, q t is the query matrix; The query matrix q of mLSTM t , key matrix k t Sum value matrix v t is defined as follows: q t =W q x t +b q v t =W v x t +b v Where W q is the weight matrix of query vector mapping, W k is the weight matrix of key vector mapping, W v is the weight matrix of the value vector mapping, b q is the bias term of query vector mapping, b k is the bias of the key vector mapping, b v is the bias term for the value vector mapping; x t is the input vector; i t is the input gate, which controls the extent to which new information is introduced into memory, w i is the weight matrix of the input gate, b i is the bias term of the input gate; the forget gate determines the extent to which the memory of the previous time step should be retained and is defined as: Where w f is the weight matrix of the forget gate, b f is the bias term of the forget gate, σ is the activation function; the output gate controls the content and amount passed from the retrieved memory to the hidden state, and is defined as: Where w o is the weight matrix of the output gate, b o is the bias term of the output gate.

5. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 3 is characterized in that: sLSTM is specifically: sLSTM combines exponential gating, memory mixing, and stabilization mechanisms to enhance the model's ability to make effective storage decisions, handle rare tag predictions in natural language processing, capture complex dependencies, and maintain robustness during training and inference. For sLSTM, the cyclic relationship between input and state is described as: c t =f t c t-1 +i t z t Where c t is the cell state at time step t, which preserves the long-term memory of the network, c t-1 is the cell state at time step t-1, f t For the forget gate, i t is the input gate, z t Used to control the amount of input added to the cell state and the previous hidden state h t-1 , specifically defined as: Where x t is the input vector, is the activation function, is the weight matrix, r z is the recursive weight matrix, b z is the bias term; sLSTM uses the normalized state as follows: n t =f t n t-1 +i t Where n t is the normalized state at time step t, n t-1 is the normalized state at time step t-1, which normalizes the update of the unit state; the hidden state is used for recurrent connection, and the formula is: In the formula, o t is the output gate; input gate i t Controls the extent to which new information is added to the cell state and is defined as: Where r i is the input gate i t The recursive weight matrix of i is the input gate i t Bias term; forget gate f t The cell state c at the previous time step t-1 The degree of preservation is expressed as follows: Where, It is the forget gate f t The weight matrix, r f It is the forget gate f t The recursive weight matrix of f It is the forget gate f t Bias term of output gate o t Controls the flow of information from the cell state to the hidden state: Where, is the output gate o t The weight matrix applied to the current input, r o is used for the previous hidden state h t-1 The output gate o t The recursive weight matrix of o is the output gate o t To provide numerical stability of the exponential gate, the forget gate f t and input gate i t Combined into another state m t , defined as: m t =max(log(f t )+m t-1 ,log(i t )) f t ′=exp(log(f t )+m t-1 -m t ) Where i t ′ is the stabilized input gate, which is a rescaled version of the original input gate; f t ′ is the stabilized forget gate, which is a rescaled version of the original forget gate.

6. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 1 is characterized in that: The position encoding in S2 is specifically: Where p represents the position index, d model Indicates the dimension of the input feature, j indicates the dimensional encoding of the input feature, and its value range is [0,d model / 2].

7. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 1 is characterized in that: The Transformer module in S4 is specifically: The Transformer module performs three different linear mappings on the input variable X through a single-head self-attention mechanism to generate a query matrix, a key matrix, and a value matrix. The dimensions of these three matrices are d k ; The attention score is calculated by normalized dot product operation, and its formula is: Where, d k is the dimension of the key matrix, which is used to scale the dot product result to avoid the gradient vanishing problem caused by excessive values; the softmax function normalizes the dot product result into a probability distribution, which represents the attention weight of each position; The final output is a weighted value matrix based on the input, with the weights determined by the attention distribution. The input matrix generates a set of multiple query matrices, key matrices, and value matrices through a variety of different linear mappings. The attention results of these outputs are spliced ​​together to enhance the performance of the model. The expression of the multi-head attention mechanism is as follows: multihead=concat(head1,...,head h )W Where head1 represents the attention result of the first head, h is the number of multi-head attention, concat is the operation of splicing the results of each head into a whole, and W is the weight matrix used to perform a linear transformation on the splicing result, which is used to integrate the information of each head to generate the final output.

8. The non-intrusive load decomposition method based on the xLSTM-Transformer model according to claim 1 is characterized in that: The sequence-to-point learning data partitioning mechanism is used to implement the training task of the xLSTM-Transformer model. The aggregated power data is partitioned by using a preset window size and a step size of 1, and multiple data windows are obtained by overlapping.

9. A non-intrusive load decomposition system based on the xLSTM-Transformer model, characterized in that: Executing the non-intrusive load decomposition method based on the xLSTM-Transformer model and causal convolution according to any one of claims 1 to 8, comprising: Load data collection module, used to collect historical load data of electrical appliances; Data preprocessing module, used to preprocess the historical load data of electrical appliances; The xLSTM-Transformer prediction model performs load decomposition based on the load data of electrical appliances; Among them, the xLSTM-Transformer prediction model includes a causal convolution module, an xLSTM module, a position encoding module, a Transformer module and a linear layer connected in sequence.

10. The non-intrusive load decomposition system based on the xLSTM-Transformer model according to claim 9, characterized in that: The xLSTM module includes a first mLSTM unit, an sLSTM unit, and a second mLSTM unit connected in sequence.

Images