Server memory capacity prediction method and device

[0020]In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following further describes the embodiments of the present invention in detail with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and the description thereof are used to explain the present invention, but are not intended to limit the present invention.

[0021]First, introduce the terms involved in the embodiments of this application:

[0022]LSTM: Long Short Term Memory Network, the full name is Long Short Term Memory. It is a time cyclic neural network, which is specially designed to solve the long-term dependence problem of general RNN (circular neural network). It is an improved version of RNN neural network. The difference between it and RNN is that it introduces the concept of cell state. Unlike RNN which only considers the most recent state, the cell state of LSTM determines which states are left and which states are forgotten. In this way, even the state long ago will have an impact on the next prediction, which largely compensates for the shortcomings of RNN.

[0023]As mentioned above, the banking industry produces a large amount of data every day, such as customer information, financial information, various account information, etc., storing this information requires a large amount of memory, and if the current capacity is not enough to store, it is likely to cause capacity Overflow, there will be a risk of information loss. Therefore, it is necessary to purchase enough memory at the end of each month to store the next month's big data, but excessive memory will cause economic waste. Therefore, the algorithm proposed by the present invention forms a time series based on the memory usage of the previous month, uses the LSTM long short-term memory neural network to predict the expected memory usage of the next month, and solves the problems of insufficient memory purchase and excess purchase. In the prior art, a differential autoregressive moving average model (ARIMA) method is used to predict server memory capacity. However, the ARIMA model also requires the stability of time series data and can only process stable data. Therefore, this method has great limitations, and the ARIMA model is complex and expensive to maintain. Specifically, the ARIMA model requires time series data to be stable, or stable after differentiation. In short, the final data used for prediction must be stable, but the operation and maintenance memory data may not meet the stability requirements, so it may not be applicable to the ARIMA model. The data processing program is too complicated. In addition to common data extraction and data cleaning, data stability processing is also required, and the model is not concise enough. In addition, different models need to be established according to different types of partial autocorrelation functions. Increase the complexity of the model, which is not concise and convenient when used, and increases the maintenance cost of the model.

[0024]In order to predict the memory capacity of the server, expand the application scope of data, and reduce maintenance costs, an embodiment of the present invention provides a method for predicting the memory capacity of a server, such asfigure 1 As shown, the method can include:

[0025]Step 101: Obtain memory usage data and corresponding time point data;

[0026]Step 102. Input the memory usage data and the corresponding time point data into the trained LSTM neural network model to predict the server memory capacity. The LSTM neural network model performs processing based on the memory historical usage data and the corresponding historical time point data. training.

[0027]byfigure 1 As shown, it can be seen that the embodiment of the present invention obtains memory usage data and corresponding time point data; inputs the memory usage data and corresponding time point data into the trained LSTM neural network model to predict the server memory capacity, The LSTM neural network model is trained according to historical memory usage data and corresponding historical time point data. The embodiment of the present invention obtains the memory usage data and the corresponding time point data, and then uses the trained LSTM neural network model to predict the memory capacity of the server. The LSTM neural network model adopted has low requirements for data stability, and ensures the accuracy of prediction. Basically, it can process more types of data, and the model is simple to maintain and cost less.

[0028]During specific implementation, the memory usage data and the corresponding time point data are obtained.

[0029]In an embodiment, after obtaining the memory usage data and the corresponding time point data, performing data preprocessing on the memory usage data and the corresponding time point data includes: performing data cleaning on the memory usage data; The memory usage data and the corresponding time point data are interpolated to obtain the memory usage data of the set time interval and the corresponding time point data.

[0030]In this embodiment, the data collected from the operation and maintenance platform is always excessively redundant, there will be missing, disorderly format, or some unwanted data attributes, so a data cleaning step is required to sort out the required clean Data, easy to run. Although the data obtained after cleaning is relatively clean and pure, the time intervals of the data test points are inconsistent. This inconsistency may cause the model to fail to obtain the expected results. Therefore, the data needs to be linearly interpolated with a time interval of 1 hour to ensure the consistency of the time step in the model. Specifically, decompress the received compressed package, use python to traverse the operation and maintenance data files in it, and process them, delete redundant data and invalid attributes, and fill in empty data. Put the sorted data into the database. Recall the data from the database, perform linear interpolation on it, and store the processed results in the database again.

[0031]In an embodiment, the memory usage data may be the data of the most recent month, because too long collection time will affect the prediction accuracy. Memory usage data is the source data passed from operation and maintenance. Organize these data in a folder, package them for subsequent delivery.

[0032]In specific implementation, the memory usage data and the corresponding time point data are input into the trained LSTM neural network model to predict the server memory capacity. The LSTM neural network model is based on the memory historical usage data and the corresponding historical time point data Conduct training.

[0033]In the embodiment, the LSTM neural network model is trained as follows: obtain historical memory usage data and corresponding historical time point data at a preset time; according to the historical memory usage data, corresponding historical time point data, and LSTM neural The output data of the network model at the previous moment is used to obtain the input data of the LSTM neural network model at the preset time; the input data is used to train the LSTM neural network model.

[0034]In an embodiment, using the input data to train the LSTM neural network model includes: using the input data to train the weight parameters in the LSTM neural network model, and using the backpropagation algorithm to adjust the weight parameters during the training process, Until the gradient obtained according to the loss function of the LSTM neural network model meets the preset convergence conditions, the trained neural network model is determined.

[0035]The long-term short-term memory network LSTM used in the embodiment of the present invention adds a long-term state c to remember data long ago on the basis of the cyclic network RNN, and a short-term state h is used to remember recent events and their states. The one-month time dimension is not considered the range of short-term memory for neural networks, so LSTM is needed to solve this problem. The core of LSTM is the cell state c, and the long-term state c can record the past state mainly depends on the three gates of LSTM to delete or add information to the cell state. After linearly interpolating the data, we set the time point x1And memory usage x2Combine the input vector X and do batch input with 1 hour as the step length.

[0036]The structure of the long short-term memory network LSTM used in the embodiment of the present invention is described below. Such asfigure 2 As shown, the input vector will undergo the control of three gates. The structure of the gate is a combination of a sigmoid layer and a point multiplication operation. Because the output of the sigmiod layer is a value between 0-1, this represents how much information can flow through the sigmoid layer. 0 means none pass, 1 means all pass. LSTM has three gates to control the cell state. These three gates are called forget gate, input gate and output gate.

[0037]image 3 It is the forget gate in the long short-term memory network LSTM used in the embodiment of the present invention. The first step of LSTM is to decide what information the cell state needs to discard. This part of the operation is handled by a sigmoid unit called the forget gate. It looks at ht-1And xtInformation to output a vector between 0-1, the value 0-1 in the vector represents the cell state Ct-1Which information is retained or how much is discarded. 0 means not reserved, 1 means both reserved.image 3 Medium, WfIs the weight, [ht-1,xt] Is the input data, bfIs the bias term.

[0038]Figure 4 It is the input gate in the long short-term memory network LSTM used in the embodiment of the present invention. This step is to decide what new information to add to the cell state. Specifically, it is divided into two steps. First, use ht-1And xtAn operation called an input gate determines which information to update. Then use ht-1And xtGet new candidate cell information through a tanh layer This information may be updated to the cell information.Figure 4 Medium, WiAnd WcIs the weight, [ht-1,xt] is the input data, biAnd bcIs the bias term.

[0039]Figure 5 For the cell state update process, the old cell information Ct-1Change to new cell information Ct. The updated rule is to choose to forget part of the old cell information through the forget gate, and to add candidate cell information through the input gate. Part of get new cell information Ct.

[0040]Figure 6 It is the output gate in the long short-term memory network LSTM used in the embodiment of the present invention. After updating the cell state, it needs to be based on the input information ht-1And xtTo determine which state characteristics of the output cell, it is necessary to pass the input through a sigmoid layer called the output gate to obtain the judgment condition, and then pass the cell state through the tanh layer to obtain a vector with a value between -1 and 1, which is the same as the output gate The obtained judgment conditions are multiplied to obtain the final output of the RNN unit.Figure 6 Medium, WoIs the weight, (ht-1,xt] Is the input data, boIs the bias term.

[0041]In the embodiment, first the time point x1And memory usage x2Combine the input vector X, the input vector x at time ttAnd the output h of the cell at the previous momentt-1Merge into a vector [ht-1,xt], enter the forget gate, the forget gate will determine what information the cell state needs to discard; then the vector [ht-1,xt] Enter the input gate. The input gate decides which new information to add to the cell state. This step is divided into two steps. First, use and pass an operation called the input gate to determine which information to update, and then use and pass a tanh The layer obtains the new candidate cell information; then updates the cell state, updating the old cell information, and turning it into new cell information; after the cell state is finally updated, it is necessary to determine which state characteristics of the output cell are based on the input information. So how do these gates know which information to discard and input which information? This judgment process mainly depends on the weight, namely Wf, Wi, WcAnd Wo, These weights are all parameters obtained through neural network training, and the training method is forward feedback and back propagation. The core idea of ​​backpropagation is to update the gradient according to the error. Since the output result of the neural network has an error with the actual result, the error between the estimated value and the actual value is calculated, and the error is propagated back from the output layer to the hidden layer until it propagates to the input layer; in the process of back propagation In, the values ​​of various parameters (weights of connected neurons) are adjusted according to the error, so that the total loss function is reduced. All the weights are initialized first, and then forward propagation is performed to obtain the output, and then the error between the output of the neural network and the actual output is calculated. Use the error to calculate the loss function, use the loss function to Wf, Wi, WcAnd WoThe partial derivative is obtained separately, and the method of obtaining the partial derivative is the chain rule to obtain the gradient. Finally, the weight is updated by the gradient descent method, and iteratively updated until the gradient no longer changes significantly, then the iteration ends and the weight update is completed.

[0042]The model used in the embodiment of the present invention does not require data stability, can process unstable data, and reduces the limitations of the model. In the data processing stage, only data cleaning is required. Since there is no requirement for data stability, the data The process is concise. And there is only one algorithm model of neural network LSTM, which is convenient and effective without repeated construction. Manpower to predict the memory capacity is a time-consuming and laborious task, and requires very experienced operation and maintenance staff, and even so, the predicted results may not be very accurate. The embodiment of the present invention can save manpower and material resources, and the algorithm is used to replace manpower to do such things, which not only saves manpower and material resources, but the algorithm also improves the accuracy of prediction and saves a lot of resources. At the same time, it has high efficiency and practicality, which saves the time used by operation and maintenance personnel for manpower forecasts such as evaluation and review, and can greatly improve work efficiency. The programmers regularly maintain the algorithm behind and maintain the model structure. Programmers and salesmen do not need to do a lot of work, so work efficiency has risen sharply. At the same time, it is very simple and practical. The application in the banking system can effectively help purchasers determine the memory capacity, save purchase costs and funds, and reduce the probability of insufficient memory or memory overflow. The safety of the system is guaranteed, and the loss of banks is greatly reduced in Chengdu, thereby bringing potential intangible benefits.

[0043]Based on the same inventive concept, the embodiment of the present invention also provides a device based on server memory capacity prediction, as described in the following embodiments. Since these problem-solving principles are similar to the method based on server memory capacity prediction, the implementation of the device can refer to the implementation of the method, and the repetition will not be repeated.

[0044]Figure 7 It is a structural diagram of an apparatus for predicting memory capacity based on a server in an embodiment of the present invention, such asFigure 7 As shown, the device includes:

[0045]The data obtaining module 701 is used to obtain memory usage data and corresponding time point data;

[0046]The capacity prediction module 702 is configured to input the memory usage data and the corresponding time point data into the trained LSTM neural network model to predict the server memory capacity. The LSTM neural network model is based on the memory historical usage data and the corresponding history Time point data for training.

[0047]In one embodiment, such asFigure 8 As shown,Figure 7 The server memory capacity prediction device also includes:

[0048]The preprocessing module 703 is configured to perform data preprocessing on the memory usage data and the corresponding time point data after obtaining the memory usage data and the corresponding time point data, wherein the data preprocessing is performed as follows:

[0049]Perform data cleaning on memory usage data;

[0050]Interpolate the cleaned memory usage data and the corresponding time point data to obtain the memory usage data at the set time interval and the corresponding time point data.

[0051]In an embodiment, the capacity prediction module is further used to train the LSTM neural network model in the following manner:

[0052]Obtain historical memory usage data and corresponding historical time point data at a preset time;

[0053]According to the historical memory usage data, the corresponding historical time point data, and the output data of the LSTM neural network model at the previous time, the input data of the LSTM neural network model at the preset time is obtained;

[0054]The LSTM neural network model is trained using the input data.

[0055]In one embodiment, using the input data to train an LSTM neural network model includes:

[0056]Use the input data to train the weight parameters in the LSTM neural network model, and use the backpropagation algorithm to adjust the weight parameters during the training process until the gradient obtained according to the loss function of the LSTM neural network model meets the preset convergence conditions. Neural network model after training.

[0057]In summary, the embodiment of the present invention obtains memory usage data and corresponding time point data; inputs the memory usage data and corresponding time point data into the trained LSTM neural network model to predict the server memory capacity. The LSTM neural network model is trained based on historical memory usage data and corresponding historical time point data. The embodiment of the present invention obtains the memory usage data and the corresponding time point data, and then uses the trained LSTM neural network model to predict the memory capacity of the server. The LSTM neural network model adopted has low requirements for data stability, and ensures the accuracy of prediction. On the basis, more types of data can be processed, and the model is simple and low in maintenance cost.

[0058]Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may be in the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0059]The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated In the processFigure oneProcess or multiple processes and/or boxesFigure oneA device with functions specified in a block or multiple blocks.

[0060]These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device. The device is implemented in the processFigure oneProcess or multiple processes and/or boxesFigure oneFunction specified in a box or multiple boxes.

[0061]These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment. Instructions are provided to implement the processFigure oneProcess or multiple processes and/or boxesFigure oneSteps for functions specified in a box or multiple boxes.

[0062]The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in further detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. The protection scope, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.