Memory utilization rate prediction method and device, equipment, storage medium and program product

Abstract

The invention relates to a memory utilization rate prediction method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring current memory utilization rate data of a target object; inputting the current memory utilization rate data of the target object into the memory utilization rate student model, and obtaining change trend prediction information of the memory utilization rate of the target object output by the memory utilization rate student model; wherein the memory utilization rate student model is a model generated after knowledge distillation training is carried out by taking output of the memory utilization rate teacher model as a soft label, and model parameters of the memory utilization rate student model are less than model parameters of the memory utilization rate teacher model. By adopting the method, the computing resource consumption can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to a method, device, equipment, storage medium and program product for predicting memory utilization rate. Background Art

[0002] In systems of data centers and cloud computing, memory is one of the key resources, and fluctuations in memory utilization rate directly affect the overall performance and reliability of the system. Effectively predicting the memory utilization rate can help system administrators identify potential memory bottlenecks in advance, optimize resource allocation, and thus avoid system performance degradation or system crashes.

[0003] In related technologies, machine learning techniques can be used to predict the memory utilization rate, so as to learn complex memory change patterns and trends from a large amount of historical data. However, using machine learning techniques to predict the memory utilization rate is data-driven and often requires the use of machine learning models with complex structures. These structures usually have a large number of parameters and complex computational graphs, and are prone to problems of large computational resource consumption when dealing with large-scale data. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, equipment, storage medium and program product for predicting memory utilization rate that can reduce computational resource consumption in view of the above technical problems.

[0005] In a first aspect, the present application provides a method for predicting memory utilization rate, including:

[0006] Obtain the current memory utilization rate data of the target object;

[0007] Input the current memory utilization rate data of the target object into the memory utilization rate student model, and obtain the prediction information on the change trend of the memory utilization rate of the target object output by the memory utilization rate student model;

[0008] Wherein, the memory utilization rate student model is a model generated after knowledge distillation training using the output of the memory utilization rate teacher model as a soft label, and the model parameters of the memory utilization rate student model are fewer than those of the memory utilization rate teacher model.

[0009] In one of the embodiments, the method further includes:

[0010] Obtain the historical memory utilization rate data of the target object at different time periods, and perform a cleaning operation on the historical memory utilization rate data;

[0011] Construct a training data set and a test data set according to the cleaned historical memory utilization rate data;

[0012] Use the training dataset and the test dataset to iteratively train the initial teacher model to obtain a memory usage teacher model;

[0013] Use the memory usage teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain the memory usage student model;

[0014] Use the test dataset to evaluate the model performance of the memory usage student model.

[0015] In one embodiment, the using the memory usage teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain the memory usage student model includes:

[0016] Input the sample data in the training dataset into the initial student model to obtain first prediction data output by the initial student model;

[0017] According to the first prediction data and the hard label data in the training dataset, obtain the hard label loss of the knowledge distillation training;

[0018] Input the sample data in the training dataset into the initial student model to obtain second prediction data output by the memory usage teacher model;

[0019] Perform temperature adjustment on the second prediction data to determine the soft label loss of the knowledge distillation training;

[0020] According to the hard label loss and the soft label loss, obtain the knowledge distillation loss;

[0021] Optimize the model parameters in the initial student model according to the knowledge distillation loss to obtain the memory usage student model.

[0022] In one embodiment, the obtaining the knowledge distillation loss according to the hard label loss and the soft label loss includes:

[0023] Perform weighted calculation on the hard label loss and the soft label loss according to a weighting parameter to obtain the knowledge distillation loss; wherein, the value of the weighting parameter is different in different knowledge distillation training stages.

[0024] In one embodiment, the using the test dataset to evaluate the model performance of the memory usage student model includes:

[0025] Load the memory usage student model and the memory usage teacher model;

[0026] Use the memory utilization student model to predict the test data in the test dataset to obtain the third prediction data, and use the memory utilization teacher model to predict the test data in the test dataset to obtain the fourth prediction data;

[0027] Calculate the loss function difference according to the third prediction data and the fourth prediction data;

[0028] Evaluate the model performance of the memory utilization student model according to the loss function difference.

[0029] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling missing data.

[0030] In a second aspect, the present application further provides a memory utilization prediction device, including:

[0031] An acquisition module, configured to acquire the current memory utilization data of the target object;

[0032] A prediction module, configured to input the current memory utilization data of the target object into the memory utilization student model, and obtain the prediction information on the change trend of the memory utilization of the target object output by the memory utilization student model;

[0033] Wherein, the memory utilization student model is a model generated after knowledge distillation training using the output of the memory utilization teacher model as soft labels, and the model parameters of the memory utilization student model are fewer than those of the memory utilization teacher model.

[0034] In one embodiment, the acquisition module is further configured to acquire the historical memory utilization data of the target object at different time periods, and perform a cleaning operation on the historical memory utilization data; construct a training dataset and a test dataset according to the cleaned historical memory utilization data;

[0035] The memory utilization prediction device further includes: a training module, configured to perform iterative training on the initial teacher model using the training dataset and the test dataset to obtain the memory utilization teacher model; perform knowledge distillation training on the initial student model using the memory utilization teacher model and the training dataset to obtain the memory utilization student model;

[0036] A test module, configured to evaluate the model performance of the memory utilization student model using the test dataset.

[0037] In one embodiment, the training module is further configured to input the sample data in the training dataset into the initial student model to obtain first prediction data output by the initial student model; obtain the hard-label loss of the knowledge distillation training according to the first prediction data and the hard-label data in the training dataset; input the sample data in the training dataset into the initial student model to obtain second prediction data output by the memory utilization teacher model; perform temperature adjustment on the second prediction data to determine the soft-label loss of the knowledge distillation training; obtain the knowledge distillation loss according to the hard-label loss and the soft-label loss; and optimize the model parameters in the initial student model according to the knowledge distillation loss to obtain the memory utilization student model.

[0038] In one embodiment, the training module is further configured to perform weighted calculation on the hard-label loss and the soft-label loss according to a weighting parameter to obtain the knowledge distillation loss; wherein the value of the weighting parameter is different in different knowledge distillation training stages.

[0039] In one embodiment, the testing module is further configured to load the memory utilization student model and the memory utilization teacher model; use the memory utilization student model to predict the test data in the test dataset to obtain third prediction data, and use the memory utilization teacher model to predict the test data in the test dataset to obtain fourth prediction data; calculate the loss function difference according to the third prediction data and the fourth prediction data; and evaluate the model performance of the memory utilization student model according to the loss function difference.

[0040] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling missing data.

[0041] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the memory utilization prediction method in the first aspect is implemented.

[0042] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the memory utilization prediction method in the first aspect is implemented.

[0043] In a fifth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the memory utilization prediction method in the first aspect is implemented.

[0044] The above memory usage prediction method, device, equipment, storage medium and program product obtain the current memory usage data of the target object; input the current memory usage data of the target object into the memory usage student model, and obtain the predicted information on the change trend of the memory usage of the target object output by the memory usage student model; wherein, the memory usage student model is a model generated after knowledge distillation training using the output of the memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the memory usage teacher model. Since the memory usage teacher model is used for knowledge distillation to train the memory usage student model, the smaller memory usage student model has a faster inference speed, so that while maintaining high prediction performance, the consumption of computing resources can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for describing the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of a memory usage prediction method provided by an embodiment of the present application;

[0047] Figure 2 It is a schematic diagram of a memory usage teacher model and a memory usage student model provided by an embodiment of the present application;

[0048] Figure 3 It is a schematic flowchart of another memory usage prediction method provided by an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the test and training principle of a memory usage student model provided by an embodiment of the present application;

[0050] Figure 5 It is a schematic structural diagram of a memory usage prediction device provided by an embodiment of the present application;

[0051] Figure 6 It is an internal structure diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0053] First, the related technologies will be described below.

[0054] In the systems of data centers and cloud computing, memory is one of the key resources. The fluctuation of memory utilization directly affects the overall performance and reliability of the system. Effectively predicting the memory utilization can help system administrators identify potential memory bottlenecks in advance, optimize resource allocation, and thus avoid system performance degradation or system crashes.

[0055] Traditional memory monitoring methods can only provide real-time data and simple statistical analysis, and cannot effectively adapt to dynamically changing loads and long-term trend changes. In related technologies, machine learning techniques can be used for memory utilization prediction, so that complex memory change patterns and trends can be learned from a large amount of historical data. However, using machine learning techniques for memory utilization prediction is data-driven and often requires the use of machine learning models with complex structures. These structures usually have a large number of parameters and complex computational graphs, and are prone to the problem of large computational resource consumption when dealing with large-scale data. Moreover, memory usage behavior is affected by factors such as the running state of applications, system load changes, and user behavior, and it is necessary to handle changes at different time scales, such as short-term memory fluctuations and long-term usage trends. This not only increases the difficulty of time series feature extraction, but also easily leads to the problems of gradient disappearance or gradient explosion of machine learning models. Finally, traditional memory prediction methods often fit the training data through complex decision boundaries in high-dimensional space. When the training data is insufficient or there is feature redundancy, the machine learning model is prone to capturing useless patterns, resulting in overfitting.

[0056] To solve the above problems, the embodiments of the present application provide a memory utilization prediction method, device, device, storage medium, and program product. By using a memory utilization teacher model for knowledge distillation to train a memory utilization student model, the smaller memory utilization student model has a faster inference speed, and thus can reduce computational resource consumption while maintaining high prediction performance.

[0057] The memory usage prediction method provided by the embodiments of this application can be applied to a memory usage prediction device, which may include a terminal, a server, a system including a terminal and a server, etc. Among them, the terminal can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, projection devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In one embodiment, as Figure 1 shown, a memory usage prediction method is provided. In this embodiment, an example is given where this method is applied to a server. The memory usage prediction method includes S201 - S202:

[0059] S201. Obtain the current memory usage data of the target object.

[0060] In this application, when it is necessary to predict the memory usage of the target object, the server can first obtain the current memory usage data of the target object.

[0061] Among them, the above-mentioned target object can include any device or system with memory, such as a terminal, a server, a data center system, and a cloud computing system.

[0062] It should be understood that the above-mentioned current memory usage data can be the memory usage data of the target object at the current moment and / or before the current moment. Exemplarily, the memory usage data can include used memory, available memory, cached memory, buffered memory, memory usage percentage, etc.

[0063] S202. Input the current memory usage data of the target object into the memory usage student model, and obtain the prediction information on the change trend of the memory usage of the target object output by the memory usage student model.

[0064] In this step, after the server obtains the current memory usage data of the target object, it can input the current memory usage data of the target object into the memory usage student model, and obtain the prediction information on the change trend of the memory usage of the target object output by the memory usage student model.

[0065] Among them, the above-mentioned change trend prediction information is the predicted change of the memory usage rate of the target object in a future period of time.

[0066] Among them, the memory usage rate student model is a model generated after knowledge distillation training using the output of the memory usage rate teacher model as soft labels. The model parameters of the memory usage rate student model are fewer than those of the memory usage rate teacher model.

[0067] In some embodiments, the above-mentioned memory usage rate teacher model and memory usage rate student model can be the same model or different models. A suitable combination of the memory usage rate teacher model and memory usage rate student model can be dynamically selected according to the data scale of the memory usage rate data and the requirements of the prediction task.

[0068] Exemplarily, for the prediction of complex memory usage patterns, a larger-scale and more powerful memory usage rate teacher model based on the Transformer can be constructed; for resource-constrained environments or scenarios with high real-time requirements, a more lightweight memory usage rate student model can be constructed. For example, the memory usage rate student model can be based on the Transformer model or a more lightweight Long Short-Term Memory (LSTM) model, etc.

[0069] In this application, the combined use of knowledge distillation and the Transformer model reduces the number of model parameters and computational complexity, enabling the smaller memory usage rate student model to have a faster inference speed, thereby accelerating the response speed of memory usage rate prediction while maintaining high prediction performance.

[0070] Since the Transformer model has powerful parallel computing capabilities, compared with traditional Recurrent Neural Network (RNN) or LSTM models, the parallelism of the Transformer model significantly improves the training speed and processing efficiency of memory usage rate data, and can provide accurate prediction results in a short time to help the system resource management administrator make decisions in a timely manner. Due to the existence of long-term trends and periodicity in memory usage rate data, compared with traditional time series prediction methods, the model based on the Transformer as the memory usage rate student model and memory usage rate teacher model does not rely on the mechanism of processing sequences step by step, which makes it more accurate and efficient in dealing with long-term dependencies.

[0071] Exemplarily, Figure 2 is a schematic diagram of a memory usage rate teacher model and a memory usage rate student model provided by an embodiment of this application, as Figure 2As shown, both the memory usage teacher model and the memory usage student model are Transformer models. The memory usage teacher model can consist of 6 encoder layers and 6 decoder layers.

[0072] Among them, each encoder layer contains four units: a multi-head attention mechanism, a residual connection, normalization, and a feed-forward neural network. Correspondingly, the decoder layer has the same structure as the encoder layer, but in the multi-head attention mechanism of the decoder layer, a mask is used to adjust the attention scores, and an attention mechanism for the encoder output is added.

[0073] Continue to refer to Figure 2 , compared with the memory usage teacher model, the structure of the memory usage student model is relatively simple and has fewer model parameters. The memory usage student model can be composed of 3 encoder layers and 3 decoder layers, and the structures of the encoder layer and the decoder layer in the memory usage student model are the same as those of the memory usage teacher model.

[0074] Next, the structure of the encoder layer in the memory usage teacher model will be described.

[0075] Exemplarily, the multi-head attention mechanism unit in the encoder layer allows the memory usage teacher model to simultaneously focus on information from different representation subspaces at different positions. After receiving the input sequence of the current memory usage data, the memory usage teacher model can pass the input sequence X to the h-th self-attention mechanism. Through three different linear transformation operations, the query matrix Q h , key matrix K h and value matrix V h of the h-th self-attention mechanism can be obtained respectively. The calculation of the query matrix Q h is shown in formula (1), the calculation of the key matrix K h is shown in formula (2), and the calculation of the value matrix V h is shown in formula (3).

[0076]

[0077]

[0078] Among them, X is the input sequence, X = [x1, x2,... x i ,... x n , and x i is the i-th element in the input sequence. is the learnable weight matrix in the multi-head self-attention mechanism.

[0079] Exemplarily, the self-attention mechanism unit multiplies the query matrix Q h with the key matrix K hPerforming a dot product operation can obtain attention scores. To avoid the dot product result from being too large, it can be divided by a scaling factor, i.e., the square root of the dimension of the key vector. Subsequently, the self-attention mechanism unit uses the activation function (softmax) to normalize the attention scores to obtain attention weights. Finally, the self-attention mechanism unit multiplies the attention weights with the value matrix V. h Performs a weighted sum to obtain the output Z of the h-th self-attention mechanism. h The output Z of the self-attention mechanism h is calculated as shown in Equation (4).

[0080]

[0081] Correspondingly, the outputs of multiple single-head self-attention mechanism units are concatenated and passed through a linear transformation to obtain the final output of the multi-head self-attention mechanism, as shown in Equation (5);

[0082] MultiHead_Z = Concat(Z1, Z2…, Zh)W Z (5)

[0083] where Z h is the output of the h-th self-attention mechanism unit, and W Z is the weight matrix of the output of the multi-head self-attention mechanism.

[0084] Exemplarily, the residual connection unit in the encoder layer adds the input sequence X to the output of the multi-head self-attention mechanism to form the residual connection result ResidualOutput. The residual connection helps to alleviate the problem of gradient disappearance in deep neural networks, enabling information to flow more effectively in the network. The residual connection is shown in Equation (6);

[0085] ResidualOutput = MultiHead Z + X (6)

[0086] Exemplarily, the normalization unit in the encoder layer receives the residual connection result ResidualOutput output by the residual connection unit. The normalization unit is a method of normalizing each input feature, which can accelerate the training of the model and improve the stability of the model. The processing of the input features by the normalization unit can be as shown in Equation (7);

[0087] LayerNormOutput = LayerNorm(ResidualOutput) (7)

[0088] Among them, LayerNorm is used to represent the normalization process, LayerNormOutput is the result of the normalization process, and ResidualOutput is the result of the residual connection.

[0089] Exemplarily, the feed-forward neural network unit in the encoder layer includes two linear sub-units and one activation function unit. The feed-forward neural network unit can further process and transform the features extracted by the multi-head self-attention mechanism unit to enhance the expression ability of the model. The feed-forward neural network unit sends the output of the normalization unit to the first linear sub-unit to expand the feature dimension of the input. Subsequently, the feed-forward neural network unit passes through the activation function (ReLU) in the activation function sub-unit to achieve the non-linear transformation of the input features. Finally, the feed-forward neural network unit passes through the second linear layer sub-unit to restore the original feature dimension. The processing of the features by the feed-forward neural network unit can be shown as in formula (8);

[0090] FFNOutput = max(0, LayerNormOutputW e1 +b e1 )W e2 +b e2 (8)

[0091] Among them, FFNOutput is the output of the feed-forward neural network unit, LayerNormOutput is the result of the normalization process, and W e1 and b e1 are the learnable weight and bias parameters of the first linear layer sub-unit in the feed-forward neural network unit of the encoder layer, and W e2 and b e2 are the learnable weight and bias parameters of the second linear layer sub-unit in the feed-forward neural network unit of the encoder layer.

[0092] Exemplarily, the residual connection unit in the encoder layer can also perform a residual connection between the output of the feed-forward neural network unit and the output of the normalization layer unit, and then send the connection result to the normalization layer unit to obtain the final output EncoderLayerOutput of the encoder layer. The final output of the encoder layer is shown as in formula (9):

[0093] EncoderLayerOutput = LayerNorm(FFNOutput + LayerNormOutput) (9)

[0094] Exemplarily, the memory utilization teacher model stacks 6 encoder layers in sequence. Each encoder layer receives the output of the previous layer as input, can extract rich feature representations from the input sequence, and finally obtains the output of the encoder layer of the memory utilization teacher model.

[0095] The structure of the decoder layer in the memory usage teacher model will be described below.

[0096] Exemplarily, the Mask Multi-Head Self-Attention unit in the decoder layer passes the target sequence Y output by the encoder layer t to the h-th different self-attention mechanism. Through three different linear transformation operations, the query matrix Q of the h-th self-attention mechanism in the decoder layer can be obtained dh , the key matrix K dh and the value matrix V dh . The calculation of the query matrix Q dh is shown in formula (10), the calculation of the key matrix K dh is shown in formula (11), and the calculation of the value matrix V dh is shown in formula (12).

[0097]

[0098]

[0099] Among them, Y t is the target sequence output by the encoder layer, Y t = [y1, y2, … y i , … y n , and y i is the i-th element in the target sequence. and are the learnable weight matrices in the Mask Multi-Head Self-Attention unit.

[0100] It should be understood that since the Mask Multi-Head Self-Attention unit in the decoder layer needs to prevent the current position from attending to future positions, a mask is required to adjust the attention scores. The scores of future positions are set to a very small value (such as negative infinity), so that the attention weights of future positions can become 0 after passing through the activation function (softmax). Based on this, the output of the single-head masked self-attention mechanism can be shown in formula (13). Finally, the outputs of each head are concatenated and passed through a linear transformation to obtain the output of the final Mask Multi-Head Self-Attention unit, as shown in formula (14):

[0101]

[0102] Among them, Z dh is the output of the h-th masked self-attention mechanism in the decoder layer, is the learnable weight matrix.

[0103] Exemplarily, the multi-head self-attention mechanism unit in the decoder layer can obtain the query matrix Q of the new h-th self-attention mechanism through three different linear transformation operations by passing the output of the masked multi-head self-attention mechanism unit into the hh-th different self-attention mechanism. dh2 , key matrix K dh2 and value matrix V dh2 . The calculation of the query matrix Q dh2 is shown in formula (15), the calculation of the key matrix K dh2 is shown in formula (16), and the calculation of the value matrix V dh2 is shown in formula (17).

[0104]

[0105] Among them, and are learnable weight matrices in the multi-head self-attention mechanism module of the decoder layer.

[0106] Exemplarily, the multi-head attention mechanism of the decoder layer uses the input of the decoder as the query and the output of the encoder as the key and value, so as to capture the interaction between the encoder and the decoder. Therefore, the output of the single-head self-attention mechanism is shown in formula (18). Finally, the outputs of each head are concatenated and then linearly transformed to obtain the output of the final multi-head self-attention unit, as shown in formula (19):

[0107]

[0108] Among them, Z dh2 is the output of the h-th self-attention mechanism of the decoder layer, is a learnable weight matrix.

[0109] Exemplarily, by adding the output of the masked multi-head self-attention mechanism module to the output of the multi-head self-attention module in the decoder layer to form a residual connection, it helps to alleviate the problem of gradient disappearance in the deep neural network, enabling information to flow more effectively in the network. Its calculation is shown in formula (20):

[0110]

[0111] Among them, ResidualOutput1 is the residual connection, is the output of the concatenated multi-head self-attention unit, MultiHead_Z dh is the output of the concatenated masked multi-head self-attention mechanism unit.

[0112] Exemplarily, after forming the residual connection in the decoder layer, the result of the residual connection can be sent to the normalization unit, and its calculation is shown in formula (21):

[0113] LayerNormOutput1 = LayerNorm(ResidualOutput1) (21)

[0114] Among them, LayerNormOutput1 is the output of the normalization unit of the decoder layer, LayerNorm is the normalization process, and ResidualOutput1 is the residual connection.

[0115] Exemplarily, the structure and calculation method of the feed-forward neural network of the decoder layer are the same as those of the encoder layer, and its calculation is shown in formula (22). By performing a residual connection between the output of the feed-forward neural network unit and the output of the normalization unit, and then sending the connection result into the normalization unit, the final output of the decoder layer can be obtained, and its calculation is shown in formula (23):

[0116] Decoder_FFNOutput = max(0, LayerNormOutput1W d1 + b d1 )W d2 + b d2 (22)

[0117] DecoderLayerOutput = LayerNorm(Decoder_FFNOutput + LayerNormOutput1) (23)

[0118] Among them, Decoder_FFNOutput is the output of the feed-forward neural network unit, LayerNormOutput1 is the result of the normalization process, W d1 and b d1 are the learnable weight and bias parameters of the first linear layer subunit in the feed-forward neural network unit of the decoder layer, and W d2 and b d2 are the learnable weight and bias parameters of the second linear layer subunit in the feed-forward neural network unit of the encoder layer.

[0119] Exemplarily, the memory usage teacher model gradually generates the output of the decoder layer of the memory usage teacher model by successively stacking 6 decoder layers, where each decoder layer receives the output of the previous layer, the encoder output, and the mask as inputs, and finally converts the output of the decoder into the probability distribution of the target sequence through a linear transformation and an activation function (softmax).

[0120] It should be noted that in the embodiments of the present application, the memory utilization teacher model and the memory utilization student model can be adjusted and extended according to specific tasks and requirements. For example, the number of layers can be increased or decreased, the number of heads can be adjusted, the size of the hidden layer can be changed, etc., to adapt to memory utilization prediction tasks of different scales and complexities.

[0121] In the present application, the memory utilization teacher model and the memory utilization student model process data through multi-level encoder layers and utilize the multi-head self-attention mechanism, thereby enabling the fusion of information and context at different levels. This multi-level information fusion ability enables the model to more comprehensively understand the complex patterns of memory usage data and thus provide more accurate memory utilization prediction results.

[0122] The following describes how to train the memory utilization student model and the memory utilization teacher model.

[0123] In some embodiments, the server can obtain the historical memory utilization data of the target object at different time periods and perform a cleaning operation on the historical memory utilization data. Secondly, the server can construct a training data set and a test data set based on the cleaned historical memory utilization data. Thirdly, the server can use the training data set and the test data set to iteratively train the initial teacher model to obtain the memory utilization teacher model, and use the memory utilization teacher model and the training data set to perform knowledge distillation training on the initial student model to obtain the memory utilization student model. Finally, the server can use the test data set to evaluate the model performance of the memory utilization student model.

[0124] It should be understood that the embodiments of the present application do not limit how to obtain the historical memory utilization data of the target object at different time periods. In some embodiments, the collection of historical memory utilization data can be implemented through a script (such as a shell script) automatically running in the server.

[0125] Exemplarily, a shell script can be written to collect historical memory utilization data. The shell script collects historical memory utilization data at different time periods at an interval of time, and the historical memory utilization data includes data such as used memory, available memory, cached memory, buffered memory, and memory utilization percentage, which are variable parameters and the unit is seconds.

[0126] In some embodiments, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling missing data.

[0127] Exemplarily, after obtaining the historical memory utilization rate data, the server can clean the historical data. During the cleaning process, at least one cleaning operation of clearing measurement error data, clearing abnormal fluctuation data, and filling missing data can be performed. Missing data can be filled by checking for blank fields and specific markers (such as "N / A") in the dataset and filling the missing data with 0. Abnormal fluctuation data can be cleared by identifying whether there are abnormalities in the data fluctuations of each data point. Measurement error data can be determined by identifying the historical memory utilization rate data of each data point that exceeds the reasonable range, and thus the measurement error data can be cleared.

[0128] In this application, through various cleaning operations, the noise and outliers in the data can be identified and processed, improving the availability and representativeness of the historical memory utilization rate data. Compared with the related technology, it enables the model not to rely on complex feature engineering when processing data.

[0129] In some embodiments, after completing the cleaning of the historical memory utilization rate data, the historical memory utilization rate data can be divided according to a preset ratio to generate a training dataset and a test dataset. Among them, the hard label data of each historical memory utilization rate data in the training dataset and the test dataset can be considered labeled.

[0130] Exemplarily, the server can use a random number generator to randomly shuffle the order of the cleaned historical memory utilization rate data to avoid systematic deviation in dividing the training dataset and the test dataset. After randomly shuffling the data, the data is split into a training set and a test set according to a preset ratio (for example, 7:3). For example, if there are 1000 historical memory utilization rate data, according to the 7:3 ratio, the first 700 historical memory utilization rate data are assigned to the training dataset, and the last 300 historical memory utilization rate data are assigned to the test dataset.

[0131] In some embodiments, the server can load the training dataset and the test dataset into the initial teacher model in the form of arrays to perform iterative training of the initial teacher model to obtain a memory utilization rate teacher model.

[0132] Exemplarily, the number of training rounds of the initial teacher model can be 20, the number of training times per round can be 20000, and the training data passed in each time can be 128. By solving the minimization of the mean square error loss function, the iterative training of the initial teacher model is completed to obtain a memory utilization rate teacher model.

[0133] It should be understood that the embodiments of the present application do not limit how to perform knowledge distillation training on the initial student model. In some embodiments, the server can first input the sample data in the training dataset into the initial student model to obtain the first prediction data output by the initial student model, and obtain the hard label loss for knowledge distillation training based on the first prediction data and the hard label data in the training dataset. Secondly, the server can input the sample data in the training dataset into the initial student model to obtain the second prediction data output by the memory usage teacher model, and perform temperature adjustment on the second prediction data to determine the soft label loss for knowledge distillation training. Thirdly, the server obtains the knowledge distillation loss based on the hard label loss and the soft label loss. Finally, the server can optimize the model parameters in the initial student model according to the knowledge distillation loss to obtain the memory usage student model.

[0134] Among them, the above first prediction data can be the prediction data output by the initial student model for the sample data in the training dataset. The above second prediction data can be the prediction data output by the memory usage teacher model for the sample data in the training dataset. The first prediction data and the second prediction data correspond to the same sample data.

[0135] It should be understood that the hard label loss can be a traditional cross-entropy loss and can be calculated using the hard labels in the training dataset. Exemplarily, for a sample data, assume that lable_ac is the hard label (actual class label) corresponding to this sample data marked in the training sample set, and p ij is the first prediction data, that is, the prediction probability distribution of the initial student model for the i-th sample data belonging to the j-th class. Then the calculation of the hard label loss can be shown as formula (24):

[0136]

[0137] Among them, m j is an indicator function. When label ac = j, m j is true and its value is 1; otherwise m j is false and its value is 0. m j is used to ensure that the initial student model can still learn the correct hard labels during training. L hard is the hard label loss.

[0138] It should be understood that the soft label loss is calculated using the soft labels of the memory usage teacher model. The soft label is the second prediction data output by the memory usage teacher model for the sample data. The calculation of the soft label loss can be shown as formula (25):

[0139]

[0140] Among them, is the soft label of the temperature-adjusted memory usage teacher model, is the j-th element in, representing the probability that the i-th sample belongs to the j-th class. L soft is the soft label loss.

[0141] In some embodiments, the server can calculate the weighted sum of the hard label loss and the soft label loss according to the weighting parameter to obtain the knowledge distillation loss.

[0142] Exemplarily, according to the weighting parameter, the weighted sum of the soft label loss and the hard label loss can be calculated to obtain the knowledge distillation loss L KD , the knowledge distillation loss L KD can be calculated as shown in formula (26):

[0143] L KD =(1-β)L hard +βL soft (26)

[0144] where β is the weighting parameter, a hyperparameter used to balance the hard label loss and the soft label loss.

[0145] It should be understood that the value of the weighting parameter is different in different knowledge distillation training stages, so as to dynamically adjust the distillation intensity according to the progress of training and the performance of the model. For example, at the beginning of training, a relatively strong distillation intensity can be set using the weighting parameter to enable the initial student model to learn the knowledge of the memory usage teacher model faster; while at the end of training, a gradually decreasing distillation intensity can be set using the weighting parameter to encourage the initial student model to develop its own feature representation to improve the generalization ability.

[0146] In this application, the memory usage teacher model provides initial parameters for the initial student model by learning the changing trends of historical memory usage data in different time periods and different application scenarios, and at the same time guides the optimization direction of the initial student model to accelerate the training process of the initial learning model. The initial student model can quickly improve its prediction ability by inheriting knowledge from the memory usage teacher model through knowledge distillation, without the need for long-term independent training, and quickly obtain the memory usage student model. In addition, the memory usage student model has a relatively simple structure and is easier to modify and expand to meet the memory usage prediction requirements in different scenarios.

[0147] In some embodiments, the server may first load the memory utilization student model and the memory utilization teacher model. Secondly, the server uses the memory utilization student model to predict the test data in the test dataset to obtain the third prediction data, and uses the memory utilization teacher model to predict the test data in the test dataset to obtain the fourth prediction data. Thirdly, the server calculates the difference in the loss function based on the third prediction data and the fourth prediction data. Finally, the server evaluates the model performance of the memory utilization student model based on the difference in the loss function.

[0148] Among them, the above-mentioned third prediction data is the memory utilization data predicted by the memory utilization student model, and the above-mentioned fourth prediction data is the memory utilization data predicted by the memory utilization teacher model.

[0149] Exemplarily, compare the performance of the memory utilization teacher model and the memory utilization student model on the test dataset, and use the third prediction data and the fourth prediction data to analyze the gap between the performance of the student model and the teacher model. If the error value of the loss function corresponding to the third prediction data and the fourth prediction data is less than 0.01, it can be considered that the student model has met the expected performance requirements.

[0150] In some embodiments, the prediction performance of the memory utilization student model can also be visually displayed by drawing a confusion matrix of the predicted values and the true values. At the same time, accuracy, prediction precision, recall rate, and F1 score can be used as evaluation indicators to evaluate the performance of the memory utilization student model.

[0151] In this application, the memory utilization student model has customizability and portability, and can be optimized for different hardware platforms, so that it can run efficiently on various devices (such as mobile devices, edge devices, etc.), improving the practicality of memory utilization prediction.

[0152] The memory utilization prediction method provided by the embodiments of this application obtains the current memory utilization data of the target object; inputs the current memory utilization data of the target object into the memory utilization student model, and obtains the prediction information on the change trend of the memory utilization of the target object output by the memory utilization student model; among them, the memory utilization student model is a model generated after knowledge distillation training using the output of the memory utilization teacher model as soft labels, and the model parameters of the memory utilization student model are fewer than those of the memory utilization teacher model. Since the memory utilization teacher model is used for knowledge distillation to train the memory utilization student model, the smaller memory utilization student model has a faster inference speed, so that while maintaining high prediction performance, the consumption of computing resources can be reduced.

[0153] The following will explain how to train and test the memory utilization student model. Figure 3Schematic flowchart of another method for predicting memory usage provided by an embodiment of this application Figure 4 Schematic diagram for testing and training a memory usage student model provided by an embodiment of this application, as Figure 3 and 4 shown. The method for predicting memory usage includes S301 - S307:

[0154] S301. Obtain historical memory usage data of a target object at different time periods, and perform cleaning operations on the historical memory usage data.

[0155] In some embodiments, the cleaning operations include at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0156] S302. Construct a training data set and a test data set according to the cleaned historical memory usage data.

[0157] S303. Use the training data set and the test data set to perform iterative training on an initial teacher model to obtain a memory usage teacher model.

[0158] S304. Use the memory usage teacher model and the training data set to perform knowledge distillation training on an initial student model to obtain a memory usage student model.

[0159] In some embodiments, the server first inputs the sample data in the training data set into the initial student model to obtain first prediction data output by the initial student model, and obtains a hard label loss for knowledge distillation training based on the first prediction data and the hard label data in the training data set. Subsequently, the server inputs the sample data in the training data set into the initial student model to obtain second prediction data output by the memory usage teacher model, and performs temperature adjustment on the second prediction data to determine a soft label loss for knowledge distillation training. Again, the server obtains a knowledge distillation loss based on the hard label loss and the soft label loss. Finally, the server optimizes the model parameters in the initial student model according to the knowledge distillation loss to obtain a memory usage student model.

[0160] Exemplarily, the server may perform weighted calculation on the hard label loss and the soft label loss according to a weighting parameter to obtain a knowledge distillation loss; wherein, the value of the weighting parameter is different in different knowledge distillation training stages.

[0161] S305. Use the test data set to evaluate the model performance of the memory usage student model.

[0162] In some embodiments, the server may first load the memory utilization student model and the memory utilization teacher model. Secondly, the server uses the memory utilization student model to predict the test data in the test data set to obtain the third prediction data, and uses the memory utilization teacher model to predict the test data in the test data set to obtain the fourth prediction data. Thirdly, the server calculates the loss function difference according to the third prediction data and the fourth prediction data. Finally, the server evaluates the model performance of the memory utilization student model according to the loss function difference.

[0163] S306. Obtain the current memory utilization data of the target object.

[0164] S307. Input the current memory utilization data of the target object into the memory utilization student model, and obtain the prediction information on the change trend of the memory utilization of the target object output by the memory utilization student model.

[0165] Among them, the memory utilization student model is a model generated after knowledge distillation training using the output of the memory utilization teacher model as soft labels, and the model parameters of the memory utilization student model are fewer than those of the memory utilization teacher model.

[0166] In this application, based on knowledge distillation, by using the soft labels generated by the memory utilization teacher model to guide the training of the memory utilization student model, the training time and calculation cost of the memory utilization student model can be effectively simplified. The complexity of model tuning is effectively reduced, thereby shortening the development cycle.

[0167] Moreover, by transferring the rich knowledge of the memory utilization teacher model to the memory utilization generation model, the generalization ability of the memory utilization student model in different memory usage scenarios can be improved. Compared with traditional models, it can perform more accurately when facing new or unseen memory usage patterns.

[0168] The memory utilization prediction method provided by the embodiments of this application obtains the current memory utilization data of the target object; inputs the current memory utilization data of the target object into the memory utilization student model, and obtains the prediction information on the change trend of the memory utilization of the target object output by the memory utilization student model; among them, the memory utilization student model is a model generated after knowledge distillation training using the output of the memory utilization teacher model as soft labels, and the model parameters of the memory utilization student model are fewer than those of the memory utilization teacher model. Since the memory utilization student model is trained by using knowledge distillation with the memory utilization teacher model, the smaller memory utilization student model has a faster inference speed, so that while maintaining high prediction performance, the consumption of computing resources can be reduced.

[0169] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless specifically stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0170] Based on the same inventive concept, an embodiment of the present application further provides a memory usage prediction device for implementing the memory usage prediction method described above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the memory usage prediction device provided below can refer to the limitations on the memory usage prediction method in the above text, and will not be repeated here.

[0171] In an exemplary embodiment, as Figure 5 shown, a memory usage prediction device 400 is provided, including: an acquisition module 401 and a prediction module 402, where:

[0172] The acquisition module 401 is configured to acquire the current memory usage data of the target object.

[0173] The prediction module 402 is configured to input the current memory usage data of the target object into the memory usage student model, and obtain the prediction information on the change trend of the memory usage of the target object output by the memory usage student model.

[0174] Wherein, the memory usage student model is a model generated after knowledge distillation training using the output of the memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the memory usage teacher model.

[0175] In one of the embodiments, the acquisition module 401 is further configured to acquire the historical memory usage data of the target object in different time periods, and perform a cleaning operation on the historical memory usage data; construct a training data set and a test data set according to the cleaned historical memory usage data;

[0176] The memory usage prediction device 400 further includes: a training module 403, configured to perform iterative training on an initial teacher model using a training data set and a test data set to obtain a memory usage teacher model; and perform knowledge distillation training on an initial student model using the memory usage teacher model and the training data set to obtain a memory usage student model.

[0177] A test module 404, configured to evaluate the model performance of the memory usage student model using the test data set.

[0178] In one embodiment, the training module 403 is further configured to input sample data in the training data set into the initial student model to obtain first prediction data output by the initial student model; obtain a hard label loss for knowledge distillation training based on the first prediction data and the hard label data in the training data set; input the sample data in the training data set into the initial student model to obtain second prediction data output by the memory usage teacher model; perform temperature adjustment on the second prediction data to determine a soft label loss for knowledge distillation training; obtain a knowledge distillation loss based on the hard label loss and the soft label loss; and optimize the model parameters in the initial student model according to the knowledge distillation loss to obtain the memory usage student model.

[0179] In one embodiment, the training module 403 is further configured to perform weighted calculation on the hard label loss and the soft label loss according to a weighting parameter to obtain a knowledge distillation loss; wherein the value of the weighting parameter is different in different knowledge distillation training stages.

[0180] In one embodiment, the test module 404 is further configured to load the memory usage student model and the memory usage teacher model; use the memory usage student model to predict test data in the test data set to obtain third prediction data, and use the memory usage teacher model to predict test data in the test data set to obtain fourth prediction data; calculate a loss function difference according to the third prediction data and the fourth prediction data; and evaluate the model performance of the memory usage student model according to the loss function difference.

[0181] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling missing data.

[0182] Each module in the above memory usage prediction device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0183] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in Figure 6 . The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a method for predicting memory usage.

[0184] Those skilled in the art can understand that Figure 6 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the above-mentioned method for predicting memory usage.

[0186] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the above-mentioned method for predicting memory usage.

[0187] In an embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the above-mentioned method for predicting memory usage.

[0188] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0189] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0190] The above embodiments only illustrate several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for predicting memory usage rate, characterized in that The method includes: Obtaining the current memory utilization rate data of the target object; Inputting the current memory utilization rate data of the target object into the memory utilization student model, and obtaining the predicted information on the change trend of the memory utilization rate of the target object output by the memory utilization student model; Wherein, the memory utilization student model is a model generated after knowledge distillation training using the output of the memory utilization teacher model as soft labels, and the model parameters of the memory utilization student model are fewer than those of the memory utilization teacher model.

2. The method according to claim 1, wherein The method further includes: Obtaining the historical memory utilization rate data of the target object at different time periods, and performing a cleaning operation on the historical memory utilization rate data; Constructing a training data set and a test data set according to the cleaned historical memory utilization rate data; Using the training data set and the test data set to perform iterative training on the initial teacher model to obtain the memory utilization teacher model; Using the memory utilization teacher model and the training data set to perform knowledge distillation training on the initial student model to obtain the memory utilization student model; Using the test data set to evaluate the model performance of the memory utilization student model.

3. The method according to claim 2, wherein The step of using the memory utilization teacher model and the training data set to perform knowledge distillation training on the initial student model to obtain the memory utilization student model includes: Inputting the sample data in the training data set into the initial student model to obtain the first predicted data output by the initial student model; Obtaining the hard label loss of the knowledge distillation training according to the first predicted data and the hard label data in the training data set; Inputting the sample data in the training data set into the initial student model to obtain the second predicted data output by the memory utilization teacher model; Performing temperature adjustment on the second predicted data to determine the soft label loss of the knowledge distillation training; Obtaining the knowledge distillation loss according to the hard label loss and the soft label loss; Optimizing the model parameters in the initial student model according to the knowledge distillation loss to obtain the memory utilization student model.

4. The method according to claim 3, characterized in that, The step of obtaining the knowledge distillation loss according to the hard label loss and the soft label loss includes: Performing weighted calculation on the hard label loss and the soft label loss according to the weighting parameter to obtain the knowledge distillation loss; wherein, the value of the weighting parameter is different in different knowledge distillation training stages.

5. The method according to claim 2, wherein The step of using the test data set to evaluate the model performance of the memory utilization student model includes: Loading the memory utilization student model and the memory utilization teacher model; Using the memory utilization student model to predict the test data in the test data set to obtain the third predicted data, and using the memory utilization teacher model to predict the test data in the test data set to obtain the fourth predicted data; Calculating the difference of the loss function according to the third predicted data and the fourth predicted data; Evaluating the model performance of the memory utilization student model according to the difference of the loss function.

6. The method according to claim 2, characterized in that, The cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

7. A memory usage prediction device, characterized in that, The device includes: an acquisition module, configured to acquire the current memory usage rate data of a target object; a prediction module, configured to input the current memory usage rate data of the target object into a memory usage student model, and acquire the predicted information on the change trend of the memory usage rate of the target object output by the memory usage student model; wherein, the memory usage student model is a model generated after knowledge distillation training using the output of a memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the memory usage teacher model.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images