Edge load prediction method based on multivariable space-time inversion Transform
Through technologies such as multivariate space-time inversion Transformer and static content perception layer, the prediction performance degradation of multivariate time-series data in edge load prediction is solved, and efficient load prediction is achieved, especially in the use of node position information and high-dimensional features. Excellent performance.
Patent Information
- Application Number
- CN202411067400.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing edge load prediction methods are difficult to fully utilize node position information and high-dimensional cross-domain features when processing multivariate timing data, resulting in a degradation in prediction performance, and traditional methods are inefficient in long-term dependency learning.
Using multivariate space-time inversion Transformer, maintain variable independence through sequence inversion, introduce static content perception layer and learnable position embedding, use Transformer's self-attention mechanism to capture multivariate correlation and space-time dependencies, and design static feature fusion Methods to improve model training speed and prediction accuracy.
It significantly improves the accuracy and efficiency of edge load prediction, can effectively utilize static features and spatial dependencies in complex edge environments, improves the performance of the model in multivariable dimensions, and performs better than other methods.
Smart Images

Figure CN119071235A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of edge load prediction, and in particular relates to an edge load prediction method based on a multivariate space-time inversion Transformer. Background Art
[0002] With the rapid development of the Internet of Things (IoT), the number of devices has increased dramatically. According to statistics, by 2025, the total number of IoT devices worldwide will reach 30.9 billion. At the same time, the rapid development of the Internet of Things has also promoted the emergence of new applications, such as smart transportation, smart healthcare, and maritime IoT. These rapidly growing devices and latency-sensitive applications make it difficult for existing IoT devices to simultaneously guarantee user experience (Quality of Experience, QoE), service level agreement (Service Level Agreement, SLA), and quality of service (Quality of Service, QoS). The traditional cloud computing model is suitable for scenarios of large-scale data storage and processing, but with the explosive growth of IoT devices, the cloud computing model has bottlenecks such as transmission delay, device energy consumption, and privacy security. The emergence of Mobile Edge Computing (MEC) effectively alleviates these problems. MEC reduces the dependence of terminal devices on data centers by distributing computing tasks to multiple edge servers, and achieves faster response time and higher reliability. At the same time, this computing method can place data processing and analysis closer to the data source, thereby responding to user requests in a timely manner. Therefore, MEC is regarded as an emerging paradigm that can effectively solve problems such as response delay and shortage of computing resources caused by the rapid development of IoT.
[0003] MEC deploys computing resources at the edge close to end users to reduce service latency and network bandwidth pressure. Therefore, the resource demand of each edge data center (EDC) depends largely on the geographical distribution of the infrastructure. This geographical distribution and the heterogeneity of the EDC infrastructure together form the resource management challenges faced by edge service providers. Virtualization technology virtualizes multiple physical devices into virtual machine instances, enabling service providers to effectively manage and allocate resources and improve the flexibility and efficiency of edge computing. However, the most advanced resource virtualization technology currently also requires a certain amount of time to start resources to provide to end users. For example, it takes an average of several minutes to start a virtual machine (VM), and some containers (such as Docker, Kubernetes, etc.) also require a certain amount of time to allocate resources. Therefore, it is particularly important to introduce a load prediction mechanism, which aims to capture the characteristics of load changes and pre-estimate the load changes of the MEC system, so as to help service operators proactively make corresponding key management decisions and configure resources in advance, thereby achieving more efficient and reasonable resource supply.
[0004] Most of the existing load forecasting work is based on cloud data centers, and mainly includes linear regression, heuristic algorithms, and traditional neural networks. As a classic neural network method, recurrent neural network (RNN) has been widely used in load forecasting because of its ability to model time series changes. However, due to the problem of gradient vanishing, RNN is inefficient in learning long-term dependencies. In order to alleviate the above problems, some improved RNNs have been proposed, including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), which have shown strong capabilities in learning long-term dependencies of time series.
[0005] Although improved RNNs such as GRU and LSTM have alleviated the gradient vanishing problem of classic RNNs to a certain extent, they still cannot completely solve this problem and have limited effect on processing long sequence data. In contrast, the Transformer model avoids the gradient vanishing problem by introducing a self-attention mechanism to achieve global context modeling and parallel computing, and is more suitable for processing long sequence data and capturing long-term dependencies. In view of this, Transformer has been widely used in natural language processing and computer vision. However, in the field of time series prediction, the effectiveness of Transformer-based multivariate time series prediction methods still needs further study. Typically, existing methods embed multiple variables of the same timestamp into indistinguishable channels and use attention to capture temporal dependencies. However, this approach obscures the important associations between variables, and in multivariate scenarios, the self-attention mechanism will learn the wrong relationship between time series, which will lead to a decrease in prediction performance. Some studies have shown that simple linear layers outperform the more complex Transformer in both performance and efficiency. At the same time, ensuring the independence of variables and effectively learning the correlation of multiple variables will help the accuracy of prediction. Inspired by this, the present invention considers reversing the time series so that the complete time series of each variable is embedded in the input channel independently. Therefore, the local perception ability of the model is expanded, which enables the Transformer model to fully utilize the self-attention mechanism to associate the potential features between multiple variables, which will also be more suitable for time series prediction problems facing multi-dimensional variable features. In addition, there are still two key issues in edge load prediction that have not been well considered and solved, as follows:
[0006] (1) Most existing works consider load prediction on a single edge node, and do not fully consider the potential value of node location information for edge load prediction. However, in real scenarios, different edge nodes may have similar load change patterns. Therefore, the location information of edge nodes will play an effective reference and assisting role in predicting future edge load conditions.
[0007] (2) In the MEC environment, edge nodes usually have high-dimensional cross-domain features (for example, static data such as infrastructure configuration and edge node location). Existing methods usually use static feature labeling input, but it is difficult to fully understand these multi-dimensional cross-domain data through such a simple encoding method. Summary of the invention
[0008] The purpose of the present invention is to solve the problems existing in the background technology, and to provide an edge load prediction method based on a multivariate spatiotemporal inversion Transformer. Unlike the traditional method that uses the same timestamp to divide the load data, the proposed method reverses the sequence and divides the data along the variable dimension. At the same time, the model learns the spatial features of different nodes through the position information of the edge nodes, and can quickly learn complex spatial dependencies without introducing additional neural networks (such as Graph Neural Network, GNN). In addition, the present invention uses the self-attention mechanism of Transformer to capture the correlation between multiple variables, and uses a feedforward neural network to learn the global feature representation of the time series. In addition, a new static feature fusion method is designed, which allows the prediction model to select the most suitable static features for fusion. Therefore, the model can reasonably utilize cross-domain static features without being interfered by redundant static features, and significantly improves the training speed of the model in multivariate dimensions.
[0009] To achieve the above objectives, the technical solution of the present invention is: an edge load prediction method based on a multivariate spatiotemporal inversion Transformer, which reorders the load sequence by sequence inversion to maintain the independence of the multivariate input sequence; in addition, a static content perception layer is introduced to extract static features closely related to the current load through a cross-attention mechanism; at the same time, a learnable position embedding is designed, which can quickly learn the global spatial dependency between edge nodes without the need for additional neural networks, and finally realize edge load prediction in combination with a Transformer-based encoding layer.
[0010] In one embodiment of the present invention, the method includes:
[0011] S1. Build an edge load prediction model and define the problem;
[0012] S2. Predict future edge load changes.
[0013] In one embodiment of the present invention, step S1 is specifically implemented as follows:
[0014] Based on the historical load data in the real edge system, an edge load prediction model is proposed to predict future load changes, which is represented by the usage of bandwidth and CPU in a fixed interval. For the high-variance and high-dimensional edge load data pattern, the edge load prediction model takes minimizing the error between the predicted load and the actual load as the optimization goal to improve the resource utilization and service quality of the edge system. Considering the dynamic and static characteristics of the edge load including date, node location, and disk capacity, edge load prediction is defined as a multi-variable and multi-sequence prediction problem. Under a fixed-size prediction window, multiple load sequences on the same infrastructure are integrated as historical load inputs, defined as
[0015]
[0016] Among them, x t is the load data at time t, T is the input sequence length, Γ is the last time index observable in the historical load data, R is a set of real numbers, d t is the dynamic feature dimension, d t >1; static characteristics are expressed as where d s represents the static feature dimension, d s >1;
[0017] For a set of load data input, the prediction process is expressed as
[0018]
[0019] in, is the predicted load sequence, are the predicted loads from time t to time t+L-1, respectively, the prediction sequence L is the prediction length, and f is the edge load prediction model;
[0020] In one embodiment of the present invention, in order to evaluate the accuracy of edge load prediction, MSE and MAE are introduced as performance indicators, which are defined as
[0021]
[0022] Among them, y i represents the actual load of the edge server in an interval i, and N is the number of historical load sequences.
[0023] In one embodiment of the present invention, step S2 is specifically implemented as follows:
[0024] S21, extracting static features and spatial features of the load data by designing the SA layer and the learnable position embedding, and performing sequence reversal on the load data;
[0025] S22. Use the Transformer-based encoding layer to extract and fuse complex spatiotemporal correlations and high-dimensional static features to ultimately achieve edge load prediction.
[0026] In one embodiment of the present invention, step S21 is specifically implemented as follows:
[0027] S211. Use the self-attention mechanism to extract the association between different features and reverse the sequence of the load data, which is expressed as:
[0028]
[0029] Among them, permute(X t ,[1,3,2]) means to convert the original tensor X t The second and third dimensions exchange positions to achieve the interchange of time and node dimensions;
[0030] S212, extract the load data from the inverted load sequence and connect it with the uniformly processed timestamp as the time series dynamic feature, which is defined as
[0031]
[0032] in, Represents the temporal dynamic characteristics, Indicates that the payload data (usually in the first dimension) is taken out from the reversed payload sequence, Time indicates the timestamp vector, and Concat indicates the concatenation operation. Next, the SA layer uses cross attention to select the most effective static content for integration based on the current sequence. The detailed information contained in the SA layer is represented as
[0033]
[0034] Among them, softmax -1 Express Q t The last dimension of is softmaxed, that is, the weight is calculated for each static feature of each sequence sample, Linear 1 、Linear 2 represents the linear layer, Norm represents the normalization operation, and Dropout(·) represents randomly selecting some neurons in the linear layer to be discarded. It indicates the sequence of static features after being processed by the SA layer;
[0035] S213. Introducing a learnable position embedding The position of each node is embedded as M represents the number of edge nodes, d modelRepresents the dimension of the feature vector; through this design, position embedding is adaptively performed during the training process to learn the positional relationship between different nodes;
[0036] S214. Combine the static feature embedding processed by the SA layer with the learnable position embedding as the input vector of the encoding layer, which is defined as
[0037]
[0038] In one embodiment of the present invention, in step S213, based on the gradient descent algorithm, the parameters of the position embedding matrix are updated using the gradient of the loss function. The updating process is defined as
[0039]
[0040] Among them, η represents the learning rate, L represents the loss function, and θ LP represents the parameters of the position embedding matrix, represents the gradient of the loss function with respect to the position embedding matrix parameters.
[0041] In one embodiment of the present invention, in step S22, the Transformer-based encoding layer includes layer normalization, a feedforward neural network, and a self-attention layer; wherein,
[0042] 1) Layer Normalization
[0043] Let the feature vector input of the lth dimension be h (l) , whose mean and variance are
[0044]
[0045] Among them, n (l) is the number of neurons in the lth dimension;
[0046] Therefore, layer normalization is defined as
[0047]
[0048] in, and β represent the parameter vectors of scaling and translation, respectively, and their dimensions are the same as h (l) Similarly, ε is a very small constant, the purpose of which is to make the denominator non-zero;
[0049] 2) Feedforward Neural Network
[0050] A feedforward neural network is composed of a set of neural network layers stacked in sequence. Data propagates forward from the input layer to the output layer without loops. A feedforward neural network is used to capture the relationship between time points, which is expressed as
[0051]
[0052] in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix of the corresponding hidden layer, b (l) is the bias vector, gelu(·) is the activation function;
[0053] 3) Self-Attention Layer
[0054] The self-attention layer is used to extract the correlation of multiple variables; the self-attention layer fuses the high-dimensional static features extracted by the SA layer and the spatial information extracted by the learnable position embedding to achieve an effective combination of temporal and spatial features; specifically, the self-attention layer converts each element in the input sequence into a query Q, key K, and value V vector, and calculates the dot product of the query with all keys. The whole process is defined as
[0055]
[0056] The multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions; specifically, multiple parallel attention layers are used and embedded in the Transformer-based encoding layer, as follows
[0057] H k =MultiHead(Q,K,V)=Concat(head 1 ,...,head n )W O , (13)
[0058] Among them, MultiHead(·) represents the multi-head attention mechanism, k = {0,…,L} represents the encoding layer index based on Transformer, and head n represents the nth parallel attention layer, is the parameter matrix.
[0059] The present invention also provides an edge load prediction system based on a multivariable space-time inversion Transformer, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0060] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0061] Compared with the prior art, the present invention has the following beneficial effects: The edge load prediction method based on multivariable spatiotemporal inversion Transformer proposed in the present invention focuses on edge load prediction. First, the method follows the core concept of the Transformer self-attention mechanism, and reorders the load sequence by sequence inversion, thereby maintaining the independence of the multivariables of the input sequence and avoiding dimensional confusion. This inversion allows the responsibilities of the self-attention mechanism and the feedforward neural network to be interchanged in the Transformer encoding layer. The self-attention mechanism is used to capture the correlation between multiple variables, while the feedforward neural network is responsible for capturing the long-term and short-term dependencies of the time series. This sequence inversion method helps to give full play to the advantages of the self-attention mechanism in time series prediction, making the Transformer more applicable in the multivariable edge load prediction environment. In addition, MSTIT introduces a static content perception layer, which extracts static features closely related to the current load through a cross-attention mechanism. This layer can effectively utilize static features and reduce the impact of redundant features on edge load prediction. At the same time, a learnable position embedding is designed, which can quickly learn the global spatial dependencies between edge nodes without the need for additional neural networks. Therefore, MSTIT can cope with complex edge environments and achieve efficient load prediction. Through experimental verification based on real-world datasets, MSTIT demonstrates excellent performance. Compared with other methods, it performs better in terms of MSE and RMSE indicators. It is particularly worth mentioning that MSTIT can ensure feature independence, effectively utilize the self-attention mechanism, and take into account the spatial dependencies between edge servers, further proving its practicality in complex edge computing environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 This is the edge load prediction model of the present invention.
[0063] Figure 2 The following is an overview of the method proposed in the present invention.
[0064] Figure 3 Reverse the sequence.
[0065] Figure 4 It is a static content-aware layer.
[0066] Figure 5 is an example of LP attention coefficient.
[0067] Figure 6 It is based on the Transformer encoding layer.
[0068] Figure 7 The prediction performance under different prediction lengths. DETAILED DESCRIPTION
[0069] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0070] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0071] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0072] The present invention provides an edge load prediction method based on a multivariate spatiotemporal inversion Transformer, which reorders the load sequence by sequence inversion to maintain the independence of the multivariate input sequence. In addition, a static content perception layer is introduced to extract static features closely related to the current load through a cross-attention mechanism. At the same time, a learnable position embedding is designed to quickly learn the global spatial dependency between edge nodes without the need for an additional neural network, and finally edge load prediction is achieved in combination with a Transformer-based encoding layer.
[0073] The following is the specific implementation process of the present invention.
[0074] 1. System model and problem definition
[0075] In edge systems, the mismatch between load conditions and resource supply can lead to resource waste or degraded service quality. For example, when the system load is low, configuring too many resources will lead to resource waste; when the system load is high, insufficient resource supply will lead to degraded service quality. Therefore, in order to improve resource utilization and ensure service quality, it is necessary to accurately predict future load changes to support service providers to allocate resources more efficiently and reasonably. However, edge loads have the characteristics of high variance and high dimensionality, which poses a huge challenge to load prediction. Specifically, high variance means that the load may change dramatically in a short period of time; high dimensionality means that the load data covers a large number of static features from servers in different regions and with different specifications, which also poses a challenge to extracting valuable information. For edge systems, the present invention proposes a load prediction model, such as Figure 1 shown.
[0076] Based on the historical load data in the real edge system, the proposed edge load prediction model aims to predict future load changes, which can be expressed as the usage of bandwidth, CPU, etc. at a fixed interval. For the high-variance and high-dimensional edge load data pattern, the proposed edge load prediction model takes minimizing the error between the predicted load and the actual load as the optimization goal to improve the resource utilization and service quality of the edge system. Considering the dynamic and static characteristics of the edge load, such as date, node location, disk capacity, etc., edge load prediction can be defined as a multi-variable and multi-sequence prediction problem. Under a fixed-size prediction window, multiple load sequences on the same infrastructure can be integrated as historical load input, defined as
[0077]
[0078] Among them, x t is the load data at time t, T is the input sequence length, Γ is the last time index observable in the historical load data, R is a set of real numbers, d t is the dynamic feature dimension, d t >1; static characteristics are expressed as where d s represents the static feature dimension, d s >1;
[0079] For a set of load data input, the prediction process is expressed as
[0080]
[0081] in, is the predicted load sequence, are the predicted loads from time t to time t+L-1, respectively, the prediction sequence L is the prediction length, and f is the edge load prediction model;
[0082] The results of edge load prediction will assist edge service providers in formulating effective resource allocation strategies to achieve load balancing of edge systems. In order to accurately and efficiently predict future edge load changes, this paper proposes a new MSTIT method. In addition, in order to evaluate the accuracy of edge load prediction, MSE and MAE are introduced as performance indicators, which are defined as
[0083]
[0084] Among them, y i represents the actual load of the edge server in a certain interval, and N is the number of historical load sequences.
[0085] 2. Overview of the proposed MSTIT technical solution
[0086] In order to solve the multivariate load prediction problem in edge environments and utilize the edge server node location information, this paper proposes MSTIT, a novel edge load prediction method based on multivariate spatiotemporal inversion transformer. The key steps are shown in Algorithm 1.
[0087]
[0088]
[0089] like Figure 2 As shown in the figure, the proposed MSTIT method mainly consists of three parts, including an embedding layer, an encoder layer, and an output layer, where the output layer is a linear layer. First, the static features and spatial features of the load data are extracted through the designed SA layer and the learnable position embedding. Secondly, the load data is sequence-reversed and combined with the Transformer encoding layer to capture long-term and short-term dependencies, thereby achieving efficient and accurate load prediction. The design of the embedding layer and the encoding layer will be introduced in detail below.
[0090] 2.1 Embedding Layer
[0091] like Figure 2 As shown in the figure, in the MSTIT method, the embedding layer (Input Embedding) consists of three parts, including sequence inverting (Sequence Invertion), static context aware layer (Static Context Aware Layer, SALayer), and learnable position embedding (Learnable Position Embedding, LP).
[0092] 1) Sequence reversal
[0093] In the classic Transformer, the self-attention mechanism can learn the long-term and short-term dependencies in the input sequence by calculating the correlation between each vector in the input sequence, but it usually divides the time series along the time dimension, such as Figure 3(left). In this way, the input vector of each time step contains multiple variable information that may have different meanings and importance and embeds them into the same input channel, which will lead to dimensional confusion. Therefore, the attention mechanism may not be able to accurately distinguish the relationship between each variable when learning variable dependencies. At the same time, a large number of devices cause the time vector dimension to be too large, so the vector dimension needs to be compressed, resulting in the loss of feature information. Therefore, the Transformer is weakened in capturing time series representation and describing multivariate correlations, limiting its capacity and generalization ability on different time series data. For the above unreasonable design, the present invention uses sequence reversal to ensure the independence between variables so as to extract the dependency between variables using the self-attention mechanism. Specifically, the MSTIT method considers dividing the time series along the node dimension rather than the time dimension to avoid the high complexity caused by processing vectors with too large dimensions, such as Figure 3 (right). The process can be expressed as
[0094]
[0095] Among them, permute(X t ,[1,3,2]) means to convert the original tensor X t The second and third dimensions exchange positions to achieve the interchange of time and node dimensions;
[0096] The proposed sequence reversal combines the SA layer with the self-attention mechanism to extract the association between different features, thereby integrating multi-dimensional features into the time series dynamic features. Through this design, the MSTIT method can use a feedforward neural network to efficiently extract the temporal features in the time series so that they can be subsequently combined with the spatial features to form spatiotemporal information to support accurate load forecasting.
[0097] 2) Static content perception layer
[0098] Since the complex MEC environment contains various heterogeneous devices, load data usually has high-dimensional features. Existing methods directly use static features as input without any processing of the static features, but it is difficult to fully understand high-dimensional data through simple encoding methods, so the improvement effect is limited. In order to make full use of these high-dimensional data, the MSTIT method uses the SA layer and self-attention mechanism to extract potential correlations between different features. The SA layer is used to combine static context with dynamic load sequence data. The present invention considers high-dimensional cross-domain data recorded in the edge system (such as maximum infrastructure bandwidth, memory, hard disk, geographic location, etc.) and uses it as additional static content. Figure 4 As shown, the SA layer uses cross attention to select the most effective static content for integration according to the current sequence. Its input includes the load sequence and static feature data P tSpecifically, the inverted load sequence is first connected with the uniformly processed timestamp as the timing dynamic feature, which is defined as
[0099]
[0100] in, Represents the temporal dynamic characteristics, Represents the payload data in the reversed payload sequence (usually in the first dimension), Time represents the timestamp vector, and Concat represents the concatenation operation.
[0101] Next, the detailed information contained in the SA layer is expressed as
[0102]
[0103] Among them, softmax -1 Express Q t The last dimension of is softmaxed, that is, the weight is calculated for each static feature of each sequence sample, Linear 1 、Linear 2 represents the linear layer, Norm represents the normalization operation, and Dropout(·) represents randomly selecting some neurons in the linear layer to be discarded. It represents a sequence of static features after being processed by the SA layer; unlike the traditional method of connecting static data as features to the input payload data, the SA layer acts as a filter to learn effective static content. Therefore, the SA layer can help learn the inherent properties of the infrastructure, which can effectively help solve the problems of redundant information and training overhead caused by high-dimensional data in edge environments.
[0104] 3) Learnable Position Embedding
[0105] Edge servers are usually distributed in different geographical locations, so their load conditions may be affected by spatial factors. Based on the above situation, the load data input in MSTIT are not strictly ordered, and they correspond to the corresponding edge servers. Therefore, the present invention does not use the fixed sinusoidal position encoding in the traditional Transformer architecture. By considering the spatial information of different devices and using it as the feature input of the load prediction model, the changing law of the load can be better understood and captured, thereby better optimizing resource allocation and improving the load balancing of the edge system. Typically, neighboring edge servers may have similar load patterns. Most existing spatiotemporal prediction methods use GNN to extract spatiotemporal information, which can learn the spatial patterns of neighbors, but ignore nodes that are farther away but have similar load patterns. To address this problem, the MSTIT method introduces a learnable position embedding The position of each node is embedded as Through this design, position embedding can be adaptively performed during the training process, thereby learning the position relationship between different nodes. Based on the gradient descent algorithm, the gradient of the loss function is used to update the parameters of the position embedding matrix. The update process is defined as
[0106]
[0107] Among them, η represents the learning rate, L represents the loss function, and θ LP represents the parameters of the position embedding matrix, represents the gradient of the loss function with respect to the parameters of the position embedding matrix. Therefore, by using learnable position embeddings, the temporal similarities between different edge nodes can be better considered globally.
[0108] like Figure 5 As shown in the figure, a region is randomly selected and the attention coefficient between regions is analyzed, where the horizontal and vertical axes represent the edge node IDs. It can be observed that nodes that are farther away may also have a high degree of temporal pattern similarity. For example, the correlation between node 2 and node 178 is 0.75. The results show the effectiveness of the proposed learnable position embedding in extracting spatial information.
[0109] Finally, the static feature embedding processed by the SA layer is combined with the learnable position embedding as the input vector of the encoding layer, which is defined as
[0110]
[0111] 2.2 Coding Layer
[0112] The MSTIT method uses a Transformer-based encoding layer to extract and fuse complex spatiotemporal correlations and high-dimensional static features, including layer normalization, feedforward neural networks, and self-attention layers, such as Figure 6 shown.
[0113] 1) Layer Normalization
[0114] Layer normalization is a regularization technique that aims to improve the convergence and generalization of neural networks. Layer normalization normalizes the output of the intermediate layers of the neural network by normalizing the input. It only considers the normalization of the input within each layer and does not rely on batch data. Specifically, for the output of each layer of the neural network, layer normalization calculates the mean and standard deviation of its feature dimension, and normalizes the output by subtracting the mean and dividing by the standard deviation. Let the feature vector input of the lth dimension be h (l) , whose mean and variance are
[0115]
[0116] Among them, n (l)is the number of neurons in the lth dimension.
[0117] Therefore, layer normalization can be defined as
[0118]
[0119] in, and β represent the parameter vectors of scaling and translation, respectively, and their dimensions are the same as h (l) ε is a very small constant, the purpose of which is to make the denominator non-zero.
[0120] The advantage of layer normalization is that it does not require statistics for the entire batch of data, and can also work for small batches of data or when processing a single sample during the inference phase. In addition, layer normalization can effectively reduce the model's sensitivity to learning rate and initial parameters, making it easier to train.
[0121] 2) Feedforward Neural Network
[0122] A feedforward neural network is a set of neural network layers stacked in sequence, where data propagates forward from the input layer to the output layer without loops. Based on the universal approximation theorem, feedforward neural networks can describe time series by extracting complex representations. Recent work on multilayer perceptrons (MLPs) has shown that linear operations on time series can help predict time series. Therefore, the MSTIT method uses a feedforward neural network to capture the relationship between time points, which is expressed as
[0123]
[0124] in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix of this layer, b (l) is the bias vector and gelu(·) is the activation function.
[0125] 3) Self-Attention Layer
[0126] The MSTIT method uses a self-attention layer to extract multivariate correlations. The self-attention layer fuses the high-dimensional static features extracted by the SA layer and the spatial information extracted by the learnable position embedding, thereby achieving an effective combination of temporal and spatial features. Specifically, the self-attention layer converts each element in the input sequence into a query (Q), key (K), and value (V) vector, and calculates the dot product of the query with all keys. The process is defined as
[0127]
[0128] In particular, the multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions. Specifically, the MSTIT method uses multiple parallel attention layers and embeds the Transformer encoder as follows
[0129] H k =MultiHead(Q,K,V)=Concat(head 1 ,...,head n )W O , (13)
[0130] Among them, MultiHead(·) represents the multi-head attention mechanism, k = {0,…,L} represents the encoding layer index, head n represents the nth parallel attention layer, is the parameter matrix.
[0131] 3. Method evaluation
[0132] This section first introduces the dataset and experimental settings. Then, a large number of comparative experiments are conducted between the proposed MSTIT method and other baseline methods.
[0133] Dataset and Experimental Settings
[0134] In order to verify the performance of the proposed MSTIT method in different edge environments, three real edge load datasets were used in the experiment. The specific description is as follows:
[0135] (1) ECW dataset. This dataset contains the upload bandwidth changes of 797 edge servers within 720 hours in August 2022, and also contains 12-dimensional static features such as location and number of node CPUs. For this dataset, the upload bandwidth is regarded as a load indicator.
[0136] (2) Alibaba dataset. This dataset records the operating status of 4,000 servers over 8 days, including machine ID, timestamp, CPU utilization, memory utilization and other operating data. For this dataset, CPU utilization is regarded as a load indicator.
[0137] (3) NEP dataset. This dataset samples the running information of VMs on NEP from June 1 to September 1, 2020 at a frequency of 1 minute. The experiment uses the load data of 6,870 edge servers in this dataset during June and resamples it at a frequency of 1 hour. Specifically, the CPU usage is regarded as a load indicator, and the location of the edge server, the user ID to which it belongs, and other information are taken as components of the dataset.
[0138] Next, each dataset is divided into training set, validation set and test set in a ratio of 7:1:2. The training set is used for model training (calculating the weights of the neural network), the validation set is used for model parameter selection (selecting hyperparameters and preventing overfitting), and the test set is used to evaluate model performance. In each dataset, multiple load instances are generated according to the load input length and load prediction length. The default setting is 96 for the input sequence length and 96 for the prediction length. At the same time, the static features are content extracted, the position information is learnably embedded, and used as the input of the prediction model together with the load data. In addition, the total number of training rounds is set to 10, the learning rate is 0.0001, the number of Transformer encoding layers is 2, and the batch size is 32. Different time granularities are used for different datasets. The time granularity of the ECW and NEP datasets is 1 hour, and the time granularity of the Alibaba dataset is 1 minute.
[0139] Based on the above datasets and experimental settings, the proposed MSTIT method was extensively compared with the following benchmark methods:
[0140] (1) Crossformer: uses a two-stage attention mechanism and a hierarchical encoder-decoder architecture to capture the correlations between multivariate variables.
[0141] (2) Informer: It adopts the ProbSpar self-attention mechanism and distills it to reduce the time complexity of the model.
[0142] (3) Flashformer: Flash attention mechanism is used to improve the scope and speed of the attention mechanism to perceive context.
[0143] (4) Flowformer: Based on the network flow conservation theorem, the Flow attention mechanism is designed to reduce its time complexity.
[0144] (5) Reformer: Uses the locality-sensitive hashing method to replace the dot-product attention mechanism and uses the reversible residual layer instead of the standard residual layer to reduce the computational cost required for training.
[0145] (6) Autoformer: It uses a deep decomposition architecture to decompose useful information from temporal patterns and exploits the autocorrelation mechanism to reduce the temporal complexity of the traditional self-attention mechanism.
[0146] (7) Transformer: combines the self-attention mechanism and feedforward neural network, and does not depend on the order of the time series.
[0147] (8) DLinear: Decomposes the time series into trend and residual series and models them using two single-layer linear networks.
[0148] (9) GRU+FiLM: Use Legendre projection to preprocess sequence information, filter information through Fourier transform, and use GRU for load prediction.
[0149] Experimental results and analysis
[0150] First, the prediction performance of various methods under different datasets was evaluated. As shown in Table 1, the prediction accuracy of the proposed MSTIT method under different datasets is significantly better than that of the other eight benchmark comparison methods, demonstrating its superior load prediction performance in different edge environments. Generally, the loads in different edge environments present highly different characteristics, and the MSTIT method can effectively cope with complex edge environments and achieve accurate load prediction. In particular, the load prediction performance of the MSTIT method on multiple datasets far exceeds that of the Transformer-based methods. This is because the MSTIT method can efficiently extract the static features of the load by combining sequence reversal and SA layers. Through the above design, the MSTIT method improves the self-attention mechanism in the classic Transformer, making it suitable for load prediction tasks in complex edge environments. In addition, the MSTIT method uses a feedforward neural network to assist in capturing the long-term and short-term dependencies of the time series, and uses LP embedding to extract the spatial feature associations of different edge nodes to efficiently utilize similar time series patterns. Experimental results show that this design is effective in improving the accuracy of multivariate time series prediction tasks. Although other Transformer-based methods have also made some improvements to the attention mechanism, they usually embed different measurement records at the same time step into tags that eliminate multivariate correlations. This approach causes the model to focus too much on local perception and ignore the potential correlation between time series features. In addition, these methods treat node positions as ordinary static features without fully considering the spatial information of different nodes. Therefore, other Transformer-based methods perform poorly on multivariate time series prediction tasks such as edge load prediction. In particular, Crossformer has a strong ability to capture multivariate correlations, but its performance is still lower than MSTIT, which shows that the dimensional confusion caused by dividing time series data along the time dimension in the Transformer architecture will bring unnecessary noise to the prediction; Informer's self-attention distillation mechanism can effectively reduce the dimension of feature maps and thus reduce memory consumption, but its use of convolutional layers and maximum pooling to distill feature maps leads to the loss of feature map information and thus a decrease in accuracy; FlashAttention divides the input into blocks and performs an attention mechanism on each block separately. This approach limits the model's ability to capture long-range dependencies and contextual information in the entire time series; the Reformer model uses a reversible residual layer to reduce the number of storage times, thereby improving memory efficiency. However, in multivariate and high-dimensional edge environments, the model may require more parameters to capture complex relationships and patterns, while the Reformer model has a limited number of parameters; compared with the DLinear and GRU+FiLM methods, the Transformer-based method performs better.This is because the self-attention mechanism introduced by Transformer can better capture the long-distance dependencies between different positions in the sequence compared to completely linear-based methods and traditional RNN-based methods, thereby improving the learning ability of the prediction model.
[0151] Table 1. Comparison of various methods on different datasets
[0152]
[0153] Next, the performance of several representative methods in different prediction length scenarios is discussed in detail. Based on the experimental results on the ECW dataset in Table 1, the load prediction effects of the proposed MSTIT method and the other two best performing Transformer-based methods (i.e., Flashformer and Flowformer) in different input sequence lengths and prediction lengths are tested. Specifically, Figure 7 (a) is the prediction scenario with an input sequence length of 48 and a prediction length of 24. Figure 7 (b) is a prediction scenario with an input sequence length of 96 and a prediction length of 96. The experimental results show that the proposed MSTIT method can simulate the load change trend well in both long-term and short-term prediction scenarios. When facing actual load prediction needs, the input sequence length and prediction length may change frequently. The proposed MSTIT method can still effectively cope with such scenario changes and demonstrates strong adaptive capabilities. Since the growing input sequence will distract the attention of the self-attention mechanism of the traditional Transformer-based method, methods such as Flowformer and Flashformer cannot accurately characterize load changes. Unlike existing methods, the MSTIT method uses the self-attention mechanism to extract the influence between variables and uses a feedforward neural network to capture temporal dependencies. Since the sequential arrangement of neurons in the feedforward neural network corresponds to the temporal order of the load data, the MSTIT method can review and utilize historical observation data well, not only performing well in short-term prediction scenarios, but also benefiting from scenarios with increased review length and prediction length.
[0154] Finally, ablation experiments are conducted to test the effectiveness of different components in the proposed MSTIT method. Among them, MSTIT(-SA), MSTIT(-LP), and MSTIT(0) represent the removal of the SA layer, LP embedding, and SA layer and LP embedding on the basis of the MSTIT method, respectively. As shown in Table 2, each component in the MSTIT method can affect the performance of its edge load prediction to varying degrees. Among them, the SA layer uses cross attention to calculate the attention score of static features (including position, node memory, etc.) and the current sequence, so the SA layer can select the static features most related to the current sequence in different sequences for integration. Through this design, the MSTIT method can effectively deal with the impact of redundant information contained in high-dimensional time series data in edge systems, thereby improving the accuracy of the prediction model. In addition, LP embedding uses a learnable parameter matrix to continuously adjust the parameter weights between different node positions during the training process to learn the spatial information of each node. Therefore, through LP embedding, the MSTIT method can fully consider the similar time series patterns between edge nodes, and then better utilize the correlation between different nodes to improve the accuracy of load prediction.
[0155] Table 2. Ablation experiments of the proposed MSTIT method
[0156]
[0157] The present invention also provides an edge load prediction system based on a multivariable space-time inversion Transformer, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0158] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.
[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0160] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0163] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. An edge load prediction method based on multivariate spatiotemporal inversion Transformer, characterized in that: The load sequence is reordered by sequence reversal to maintain the independence of multiple variables of the input sequence. In addition, a static content-aware layer is introduced to extract static features closely related to the current load through a cross-attention mechanism. At the same time, a learnable position embedding is designed to quickly learn the global spatial dependencies between edge nodes without the need for additional neural networks. Finally, the Transformer-based encoding layer is combined to achieve edge load prediction.
2. The edge load prediction method based on multivariate space-time inversion Transformer according to claim 1 is characterized in that: The method comprises: S1. Build an edge load prediction model and define the problem; S2. Predict future edge load changes.
3. The edge load prediction method based on multivariate time-space inversion Transformer according to claim 1 is characterized in that: Step S1 is specifically implemented as follows: Based on the historical load data in the real edge system, an edge load prediction model is proposed to predict future load changes, which is represented by the usage of bandwidth and CPU in a fixed interval. For the high-variance and high-dimensional edge load data pattern, the edge load prediction model takes minimizing the error between the predicted load and the actual load as the optimization goal to improve the resource utilization and service quality of the edge system. Considering the dynamic and static characteristics of the edge load including date, node location, and disk capacity, edge load prediction is defined as a multi-variable and multi-sequence prediction problem. Under a fixed-size prediction window, multiple load sequences on the same infrastructure are integrated as historical load inputs, defined as Among them, x t is the load data at time t, T is the input sequence length, Γ is the last time index observable in the historical load data, R represents a real number set, d t is the dynamic feature dimension, d t >1; static characteristics are expressed as where d s represents the static feature dimension, d s >1; For a set of load data input, the prediction process is expressed as in, is the predicted load sequence, are the predicted loads from time t to time t+L-1, the prediction sequence L is the prediction length, and f is the edge load prediction model; The results of edge load prediction will assist edge service providers in formulating effective resource allocation strategies to achieve load balancing of edge systems.
4. The edge load prediction method based on multivariate time-space inversion Transformer according to claim 3 is characterized in that: In order to evaluate the accuracy of edge load prediction, MSE and MAE are introduced as performance indicators, which are defined as Among them, y i represents the actual load of the edge server in an interval i, and N is the number of historical load sequences.
5. The edge load prediction method based on multivariate time-space inversion Transformer according to claim 3 is characterized in that: Step S2 is specifically implemented as follows: S21, extracting static features and spatial features of the load data by designing the SA layer and the learnable position embedding, and performing sequence reversal on the load data; S22. Use the Transformer-based encoding layer to extract and fuse complex spatiotemporal correlations and high-dimensional static features to ultimately achieve edge load prediction.
6. The edge load prediction method based on multivariate space-time inversion Transformer according to claim 5 is characterized in that: Step S21 is specifically implemented as follows: S211. Use the self-attention mechanism to extract the association between different features and reverse the sequence of the load data, which is expressed as: Among them, permute(X t ,[1,3,2]) means to convert the original tensor X t The second and third dimensions exchange positions to achieve the interchange of time and node dimensions; S212, extract the load data from the inverted load sequence and connect it with the uniformly processed timestamp as the time series dynamic feature, which is defined as in, Represents the temporal dynamic characteristics, represents the payload data taken from the reversed payload sequence, Time represents the timestamp vector, and Concat represents the concatenation operation; then, the SA layer uses cross attention to select the most effective static content for integration according to the current sequence. The detailed information contained in the SA layer is represented as α t =softmax -1 (Q t ), V=Linear2(P t ), Among them, softmax -1 Express Q t The last dimension of is softmaxed, that is, the weights of each static feature of each sequence sample are calculated. Linear1 and Linear2 represent linear layers, Norm represents normalization operation, and Dropout(·) represents randomly selecting some neurons in the linear layer to be discarded. It represents the sequence of fused static features after being processed by the SA layer; S213. Introducing a learnable position embedding The position of each node is embedded as M represents the number of edge nodes, d model Represents the dimension of the feature vector; through this design, position embedding is adaptively performed during the training process to learn the positional relationship between different nodes; S214. Combine the static feature embedding processed by the SA layer with the learnable position embedding as the input vector of the encoding layer, which is defined as 7. The edge load prediction method based on multivariate time-space inversion Transformer according to claim 6 is characterized in that: In step S213, based on the gradient descent algorithm, the parameters of the position embedding matrix are updated using the gradient of the loss function. The update process is defined as Among them, η represents the learning rate, L represents the loss function, and θ LP represents the parameters of the position embedding matrix, represents the gradient of the loss function with respect to the position embedding matrix parameters.
8. The edge load prediction method based on multivariate time-space inversion Transformer according to claim 7 is characterized in that: In step S22, the Transformer-based encoding layer includes layer normalization, a feedforward neural network, and a self-attention layer; wherein, 1) layer normalization Let the feature vector input of the lth dimension be h (l) , whose mean and variance are Among them, n (l) is the number of neurons in the lth dimension; Therefore, layer normalization is defined as in, and β represent the parameter vectors of scaling and translation, respectively, and their dimensions are the same as h (l) Similarly, ε is a very small constant, the purpose of which is to make the denominator non-zero; 2) Feedforward Neural Network A feedforward neural network is composed of a set of neural network layers stacked in sequence. Data propagates forward from the input layer to the output layer without loops. A feedforward neural network is used to capture the relationship between time points, which is expressed as in, represents the state of the l-1th hidden layer, W (l) represents the weight matrix of the corresponding hidden layer, b (l) is the bias vector, gelu(·) is the activation function; 3) Self-Attention Layer The self-attention layer is used to extract the correlation of multiple variables; the self-attention layer fuses the high-dimensional static features extracted by the SA layer and the spatial information extracted by the learnable position embedding to achieve an effective combination of temporal and spatial features; specifically, the self-attention layer converts each element in the input sequence into a query Q, key K, and value V vector, and calculates the dot product of the query with all keys. The whole process is defined as The multi-head attention mechanism consists of multiple self-attention layers, focusing on the representation subspace information from different positions; specifically, multiple parallel attention layers are used and embedded in the Transformer-based encoding layer, as follows H k =MultiHead(Q,K,V)=Concat(head1,...,head n )W O , (13) Among them, MultiHead(·) represents the multi-head attention mechanism, k = {0,…,L} represents the encoding layer index based on Transformer, and head n represents the nth parallel attention layer, is the parameter matrix.
9. An edge load prediction system based on multivariable space-time inversion Transformer, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.
10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.
Citation Information
Patent Citations
Depth model load distribution prediction method based on edge environment
CN117290112A
Method for temporal knowledge graph reasoning based on distributed attention
US20230401466A1
Cited By
Personalized federal map learning Transform-based space-time load prediction method in multi-edge system
CN119892661A
Cloud computing resource elastic scaling method and system based on Kubernetes cluster
CN120849089A
