Server load prediction model, training method thereof and server load prediction method
By adopting subtraction information aggregation in the Transformer model, the problem of overfitting in server load prediction is solved, the accuracy and generalization ability of the model are improved, and more accurate load prediction is achieved.
Patent Information
- Application Number
- CN202411200217.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-24
AI Technical Summary
The existing Transformer model is prone to overfitting in server load prediction and insufficient generalization capabilities, resulting in inaccurate prediction.
In Transformer's information aggregation, subtraction method is adopted instead of the traditional addition method. By subtraction operations between the decoder block and the sublayer, information redundancy is reduced, and the model's deredundancy ability and learning efficiency are improved.
Through subtraction information aggregation, the risk of overfitting is reduced, the accuracy and generalization ability of the model are improved, and more accurate load prediction results are generated.
Smart Images
Figure CN119179630A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of server management, and in particular to a server load prediction model and a training method thereof, and a server load prediction method. Background Art
[0002] Server load prediction is an important research direction that aims to predict future server load by analyzing historical data. With the development of cloud computing and big data technology, the frequency and complexity of server usage have increased significantly. Effective load prediction can help improve resource utilization, optimize performance, reduce costs, and enhance user experience. By accurately predicting server load, managers can better allocate, expand, and maintain resources.
[0003] Therefore, the main goals of server load forecasting include: predicting future load peaks so that resources can be expanded or scheduled in advance; dynamically adjusting resource allocation based on forecast results to avoid resource waste or shortage; reducing unnecessary energy consumption through reasonable load management; and reducing the risk of system crashes or performance degradation by predicting and managing load fluctuations.
[0004] Based on neural networks, by learning historical data (such as recording the usage of CPU, memory, network and other resources of the server in different time periods), it is possible to predict the future load of the server.
[0005] In the field of server load prediction, the application principle of the Transformer model has unique advantages over traditional methods, but it also has some disadvantages.
[0006] The core of the Transformer model is to solve the sequence-to-sequence task in the field of time series information processing. The core idea is to capture the dependencies between elements in the sequence through the self-attention mechanism without the sequential processing of the traditional model structure. The self-attention mechanism calculates the correlation between each element in the input sequence and other elements, generates a series of weights, and these weights are used to weighted average the elements in the input sequence to capture global dependencies. Specifically, the self-attention mechanism contains three main steps: generating query, key, and value vectors; calculating the dot product of the query and the key to obtain the attention scores; performing softmax processing on the attention scores to obtain the attention weights and weighted average vectors. In addition, in order to enhance the expressiveness of the model, the Transformer introduces a multi-head attention mechanism, that is, performing multiple self-attention calculations in parallel in different subspaces and then concatenating the results. The multi-head attention mechanism can capture dependencies of different scales and different features. However, since the self-attention mechanism itself does not contain position information, the Transformer retains the position information of the elements in the sequence by adding position encoding. The positional encoding is usually generated by sine and cosine functions and added to the input vector. Typically, the Transformer adopts an encoder-decoder structure, where the encoder is used to map the input sequence to a latent space representation, and the decoder converts the latent space representation into a target sequence. Both the encoder and the decoder consist of multiple stacked self-attention layers and feedforward neural networks.
[0007] The Transformer model is widely used in processing sequence data with complex time dependencies due to its ability to efficiently capture long-range dependencies. In the field of server load prediction, the Transformer can model the sequence information of historical data of server load and thus predict the server load.
[0008] Therefore, although the application potential of the Transformer model in the field of sequence data processing is huge, in the face of massive server load data, most Transformer models will use a huge number of model parameters, such as the heads in the multi-head attention mechanism, the hidden layer nodes in the feedforward neural network, etc. This will make the model perform well on the training data, but the generalization ability on the test data is insufficient. This is because the Transformer model often lacks effective means of de-redundancy and de-fitting in actual use. Therefore, when faced with tasks such as server load prediction, due to its greatly fluctuating behavioral data, the Transformer model with super-large parameters is easy to overfit to the training data. Therefore, an accurate server load prediction method is needed. Summary of the invention
[0009] In view of the deficiencies in the prior art, the present invention provides a server load prediction model and a training method thereof and a server load prediction method, which solve the problem of inaccurate server load prediction methods in the prior art.
[0010] In a first aspect, according to an embodiment of the present invention, there is provided a method for training a server load prediction model, wherein the server load prediction model is used to predict the load of the server based on the number of tasks currently processed by the server, the server load prediction model comprises a transformer network Transformer, and the training method comprises the following steps: S1, constructing a training data set, wherein the training data set comprises a sequence of task quantities and corresponding loads collected at multiple moments, adjacent moments in the multiple moments are separated by a fixed time interval, and the load at the current moment is used as a label of the previous moment; S2, the task quantity sequence is processed by the Transformer to obtain a predicted load sequence, wherein information aggregation in the Transformer adopts a subtraction method; S3, based on the obtained predicted load sequence, a preset loss function is used to calculate the loss, the model parameters are updated, and iterative training is performed until the model converges.
[0011] According to an embodiment of the present invention, the information aggregation in the Transformer includes: S21, performing subtraction operations on the input and output of each decoder block respectively to obtain a decoder block residual value as the input of the next decoder block; S22, subtracting the output residual value of the previous decoder block from the output value of each decoder block to obtain the output residual value of the current decoder block; wherein the output residual value of the first decoder block is calculated based on the initial output residual value of 0, and the output residual value of the last decoder block is used as the output value of the decoder.
[0012] According to an embodiment of the present invention, the information aggregation in the Transformer includes: S27, performing subtraction operations on the input and output of the sub-layer in each decoder block respectively to obtain a sub-layer residual value as the input of the next sub-layer; S28, performing a link operation on the output value of each sub-layer, and using the link result as the output of the decoder block.
[0013] According to an embodiment of the present invention, the sub-layers in the decoder block include a feed-forward sub-layer and an attention sub-layer.
[0014] According to an embodiment of the present invention, the feedforward sublayer includes two linear transformation units and one activation function unit.
[0015] According to an embodiment of the present invention, the attention sublayer adopts a multi-head self-attention mechanism.
[0016] According to an embodiment of the present invention, the preset loss function is a cross entropy loss function.
[0017] In a second aspect, according to another embodiment of the present invention, a server load prediction model is provided, which is trained using a server load prediction model training method as described in the first aspect, and includes: a feature extraction module, used to extract features of a task quantity sequence currently processed by an input server to obtain a feature vector of the task quantity sequence; a transformer network Transformer, used to process the feature vector of the task quantity sequence to obtain a predicted load sequence, wherein information aggregation in the Transformer adopts a subtraction method.
[0018] According to another embodiment of the present invention, the feature extraction module includes a position encoding unit.
[0019] In a third aspect, according to another embodiment of the present invention, a server load prediction method is provided, comprising the following steps: P1, obtaining a task quantity sequence; P2, using a server load prediction model as described in the second aspect to perform load prediction on the obtained task quantity sequence.
[0020] The technical principle of the present invention is: in the information aggregation of Transformer, the addition aggregation method in the prior art is changed to subtraction, and the aggregation involves aggregation between decoder blocks and between sub-layers in the decoder blocks.
[0021] Compared with the prior art, the present invention has the following beneficial effects: by adopting subtraction-based information aggregation, it solves the technical problem of redundancy of load prediction models and produces a technical effect of more accurate load prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1FIG. 4 is a load fluctuation diagram of the prior art of an embodiment of the present invention.
[0023] Figure 2 The present invention is a flowchart of a method for training a server load prediction model according to an embodiment of the present invention.
[0024] Figure 3 Schematic diagram of decoder block information aggregation according to an embodiment of the present invention.
[0025] Figure 4 A schematic diagram of sub-layer information aggregation in a decoder block according to an embodiment of the present invention.
[0026] Figure 5 A schematic diagram of a decoder including decoder block information aggregation and sub-layer information aggregation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0028] As described in the background, due to the large fluctuations in the behavior data of tasks such as server load prediction, Figure 1 As shown in the figure, corresponding to the load recorded in 2000 hours, the number of tasks received by the server fluctuates greatly in the range of 0-1 million, making the Transformer model based on super-large parameters easy to overfit to the training data. Figure 2 As shown, according to an embodiment of the present invention, a training method for a server load prediction model is provided, wherein the server load prediction model is used to predict the load of the server based on the number of tasks currently processed by the server, and the server load prediction model includes a transformer network Transformer, and the training method includes the following steps: S1, constructing a training data set, wherein the training data set includes a sequence of task quantities and corresponding loads collected at multiple moments, wherein adjacent moments in the multiple moments are separated by a fixed time interval, and the load at the current moment is used as a label of the previous moment; S2, the task quantity sequence is processed by the Transformer to obtain a predicted load sequence, wherein the information aggregation in the Transformer adopts a subtraction method; S3, based on the obtained predicted load sequence, a preset loss function is used to calculate the loss, the model parameters are updated, and iterative training is performed until the model converges. By changing the addition aggregation method in the prior art to subtraction in the information aggregation of the Transformer, the problem of inaccurate server load prediction model in the prior art is solved.
[0029] According to an embodiment of the present invention, Figure 3As shown, the information aggregation in the Transformer includes: S21, respectively subtracting the input and output of each decoder block to obtain a decoder block residual value as the input of the next decoder block; S22, subtracting the output value O of each decoder block i Subtract the output residual value of the previous decoder block To get the output residual value of the current decoder block The output residual value of the first decoder block is calculated based on the initial output residual value of 0, and the output residual value of the last decoder block is used as the output value of the decoder. For the output stream: the original input is empty, and the output O obtained in each decoder block i , and then use the information aggregation mechanism to subtract the knowledge learned in the previous layer, that is, In addition, in addition to the output value O i In addition, the last decoder block also outputs a residual value R. The input of the decoder is the feature vector X of the task quantity sequence.
[0030] According to an embodiment of the present invention, Figure 4 As shown, the information aggregation in the Transformer includes: S27, respectively subtracting the input and output of the sub-layer in each decoder block to obtain the sub-layer residual value as the input of the next sub-layer; S28, performing a link operation on the output value of each sub-layer, and using the link result as the output of the decoder block. In the traditional Transformer model, the aggregation of the input and output of the sub-layer is achieved through addition. This model replaces addition with subtraction to achieve a new information aggregation method. This change is intended to reduce information redundancy so that the model can focus more on important features in the data. In practical applications, the subtraction operation helps to highlight the difference between the input information and the output of the encoder block, thereby improving the learning efficiency and accuracy of the model.
[0031] In detail, the sublayers in the decoder block include a feedforward sublayer and an attention sublayer. In the attention sublayer, the input sequence is processed by a multi-head self-attention mechanism to learn the relationship between sequence data, calculate the correlation between each data and other data, generate attention weights, and use Y i Indicates. i Represents the correlation information between the sequence data captured under the current decoder block, Y i =Attention i (Y i-1 ), where Y i-1 is the input of the current attention sublayer. i, where the output after information aggregation is Y i ’ =Y i-1 -Y i , where Y i ’ represents the information obtained after information aggregation of the attention sublayer in the i-th decoder block. In the attention module, the correlation of the sequence data is modeled, and the information aggregation using subtraction is used to make Y i ’ It can decompose the timing signal without carrying the information to the next level of structure.
[0032] The structure of the feedforward sublayer is similar to that of the traditional Transformer, including two linear transformations and an activation function. i ’ After the feedforward neural network, we get Z i , that is, Z i =MLP(Y i ’ ). The difference is that the information aggregation method also uses subtraction operation, that is, Z i ’ =Z i -Y i ’ .
[0033] Reference again Figure 3 , after the attention sublayer, the output value Y is retained i For subsequent calculations. Similarly, after the feedforward sublayer ends, the output value Z is also retained i In this part of the information aggregation, Y i Will and Z i Perform link operation, that is, P i =concat(Z i -Y i The result is passed through a linear layer to obtain the output of the decoder block, i.e., i = linear(P i ).
[0034] In addition, if Figure 5 As shown, a schematic diagram of information subtraction aggregation including the decoder level and the sub-layer within the decoder is shown. Through the improvement of these two levels, the model has a stronger ability to remove redundancy.
[0035] According to an embodiment of the present invention, the preset loss function is a cross entropy loss function.
[0036] According to another embodiment of the present invention, there is provided a server load prediction model, which is trained using a server load prediction model training method as described above, and includes: a feature extraction module, used to extract features from an input sequence of task quantities currently processed by a server to obtain a feature vector of the task quantity sequence; a transformer network Transformer, used to process the feature vector of the task quantity sequence to obtain a predicted load sequence, wherein information aggregation in the Transformer adopts a subtraction method.
[0037] According to another embodiment of the present invention, the feature extraction module includes a position encoding unit. Since the self-attention mechanism itself does not contain position information, the Transformer retains the position information of the elements in the sequence by adding position encoding. The position encoding is usually generated by sine and cosine functions and added to the input vector. The input sequence data is first transformed into a high-dimensional vector representation through an embedding layer, and the dimension can be 4096.
[0038] According to another embodiment of the present invention, a server load prediction method is provided, comprising the following steps: P1, obtaining a task quantity sequence; P2, using a server load prediction model as described above to perform load prediction on the obtained task quantity sequence.
[0039] The model adopts a special information aggregation mechanism to decompose the time series signal layer by layer through multiple residual blocks. When processing time series signals, the traditional Transformer model often processes the entire sequence at one time, which may lead to information redundancy and low learning efficiency. The strategy of step-by-step decomposition is different. It divides the processing process of time series signals into multiple steps, and each step refines and decomposes the signal through residual blocks. By capturing the residual stream and output stream at each layer, the model extracts the features that can be captured by the current layer and passes the remaining uncaptured features to the next layer. This method can not only better retain and extract important features in the time series signal, but also reduce the interference of redundant information, so that the model can focus more on the important information that is refined layer by layer during the learning process, and reduce the risk of overfitting. The de-redundancy method and the idea of gradually learning residuals proposed in the present invention provide a new Transformer structure design for solving the overfitting problem in sequence data prediction. The model improves the generalization ability of the model by gradually refining the prediction. Due to the robustness of the model proposed in the present invention to overfitting, the Transformer structure can be designed to be deeper without reducing performance, which improves the theoretical upper limit of the Transformer model.
[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
Claims
1. A method for training a server load prediction model, wherein the server load prediction model is used to predict the load of a server based on the number of tasks currently processed by the server, and the server load prediction model includes a transformer network Transformer, characterized in that: The training method comprises the following steps: S1. Construct a training data set, wherein the training data set includes a sequence of task quantities and corresponding loads collected at multiple moments, wherein adjacent moments in the multiple moments are separated by a fixed time interval, and the load at the current moment is used as a label of the previous moment; S2, the Transformer processes the task quantity sequence to obtain a predicted load sequence, wherein information aggregation in the Transformer adopts a subtraction method; S3. Based on the obtained predicted load sequence, a preset loss function is used to calculate the loss, update the model parameters, and perform iterative training until the model converges.
2. A server load prediction model training method as claimed in claim 1, characterized in that: The information aggregation in the Transformer includes: S21, performing a subtraction operation on the input and output of each decoder block respectively to obtain a decoder block residual value as an input of a next decoder block; S22, subtracting the output residual value of the previous decoder block from the output value of each decoder block to obtain the output residual value of the current decoder block; The output residual value of the first decoder block is calculated based on the initial output residual value of 0, and the output residual value of the last decoder block is used as the output value of the decoder.
3. A server load prediction model training method as claimed in claim 1, characterized in that: The information aggregation in the Transformer includes: S27, performing a subtraction operation on the input and output of each sub-layer in each decoder block respectively to obtain a sub-layer residual value as an input of the next sub-layer; S28. Perform a concatenation operation on the output value of each sub-layer, and use the concatenation result as the output of the decoder block.
4. A server load prediction model training method as claimed in claim 3, characterized in that: The sublayers in the decoder block include a feedforward sublayer and an attention sublayer.
5. A server load prediction model training method as claimed in claim 3, characterized in that: The feedforward sublayer includes two linear transformation units and one activation function unit.
6. A server load prediction model training method as claimed in claim 3, characterized in that: The attention sublayer adopts a multi-head self-attention mechanism.
7. A server load prediction model training method as claimed in claim 1, characterized in that: The preset loss function is a cross entropy loss function.
8. A server load prediction model, obtained by training using a server load prediction model training method according to any one of claims 1 to 7, characterized in that: include: A feature extraction module is used to extract features from the input task quantity sequence currently processed by the server to obtain a feature vector of the task quantity sequence; The transformer network Transformer is used to process the feature vector of the task quantity sequence to obtain a predicted load sequence, wherein information aggregation in the transformer adopts a subtraction method.
9. A server load prediction model training method as claimed in claim 8, characterized in that: The feature extraction module includes a position encoding unit.
10. A server load prediction method, characterized in that: The following steps are involved: P1. Get the task quantity sequence; P2. Use a server load prediction model as described in claim 9 to perform load prediction on the acquired task quantity sequence.