Memory usage rate prediction method, device, equipment, storage medium and program product

Abstract

The application relates to a memory usage rate prediction method, device, equipment, storage medium and program product. The method comprises the following steps: obtaining current memory usage rate data of a target object; inputting the current memory usage rate data of the target object into a memory usage rate student model, and obtaining change trend prediction information of the memory usage rate of the target object output by the memory usage rate student model; wherein the memory usage rate student model is a model generated by using the output of a memory usage rate teacher model as a soft label for knowledge distillation training, and the model parameters of the memory usage rate student model are less than the model parameters of the memory usage rate teacher model. The method can reduce the consumption of computing resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for predicting memory usage. Background Technology

[0002] In data center and cloud computing systems, memory is one of the critical resources, and fluctuations in memory usage directly affect the overall performance and reliability of the system. Effectively predicting memory usage helps system administrators identify potential memory bottlenecks in advance, optimize resource allocation, and thus avoid system performance degradation or system crashes.

[0003] In related technologies, machine learning techniques can be used to predict memory usage, thereby learning complex memory change patterns and trends from large amounts of historical data. However, memory usage prediction using machine learning techniques is data-driven and often requires the use of complex machine learning models. These models typically have a large number of parameters and complex computational graphs, which can easily lead to high computational resource consumption when processing large-scale data. Summary of the Invention

[0004] Therefore, it is necessary to provide a memory usage prediction method, apparatus, device, storage medium, and program product that can reduce the consumption of computing resources, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for predicting memory usage, including:

[0006] Get the current memory usage data of the target object;

[0007] Input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model;

[0008] The memory usage student model is a model generated by knowledge distillation training using the output of the memory usage teacher model as soft labels. The model parameters of the memory usage student model are fewer than those of the model parameters of the memory usage teacher model.

[0009] In one embodiment, the method further includes:

[0010] Obtain historical memory usage data of the target object at different time periods, and perform cleaning operations on the historical memory usage data;

[0011] Based on the cleaned historical memory usage data, a training dataset and a test dataset are constructed.

[0012] Using the training and test datasets, the initial teacher model is iteratively trained to obtain the memory utilization teacher model.

[0013] Using the memory utilization rate teacher model and the training dataset, the initial student model is trained by knowledge distillation to obtain the memory utilization rate student model.

[0014] The performance of the memory usage student model was evaluated using the test dataset.

[0015] In one embodiment, the step of using the memory usage teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain the memory usage student model includes:

[0016] Input the sample data from the training dataset into the initial student model to obtain the first predicted data output by the initial student model;

[0017] Based on the first predicted data and the hard-label data in the training dataset, the hard-label loss for the knowledge distillation training is obtained;

[0018] Input the sample data from the training dataset into the initial student model to obtain the second prediction data output by the memory usage teacher model;

[0019] Temperature adjustment is performed on the second prediction data to determine the soft label loss of the knowledge distillation training;

[0020] Based on the hard label loss and soft label loss, the knowledge distillation loss is obtained;

[0021] The model parameters in the initial student model are optimized based on the knowledge distillation loss to obtain the memory usage student model.

[0022] In one embodiment, obtaining the knowledge distillation loss based on the hard label loss and the soft label loss includes:

[0023] The hard label loss and soft label loss are weighted and calculated according to the weighting parameters to obtain the knowledge distillation loss; wherein the weighting parameters have different values ​​in different knowledge distillation training stages.

[0024] In one embodiment, evaluating the model performance of the memory usage student model using the test dataset includes:

[0025] Load the memory usage student model and memory usage teacher model;

[0026] The memory usage student model is used to predict the test data in the test dataset to obtain the third predicted data, and the memory usage teacher model is used to predict the test data in the test dataset to obtain the fourth predicted data.

[0027] Calculate the difference in the loss function based on the third and fourth prediction data;

[0028] The performance of the memory usage student model is evaluated based on the difference in the loss function.

[0029] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0030] Secondly, this application also provides a memory usage prediction device, comprising:

[0031] The acquisition module is used to obtain the current memory usage data of the target object;

[0032] The prediction module is used to input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model;

[0033] The memory usage student model is a model generated by knowledge distillation training using the output of the memory usage teacher model as soft labels. The model parameters of the memory usage student model are fewer than those of the model parameters of the memory usage teacher model.

[0034] In one embodiment, the acquisition module is further configured to acquire historical memory usage data of the target object at different time periods, and perform cleaning operations on the historical memory usage data; and construct a training dataset and a test dataset based on the cleaned historical memory usage data.

[0035] The memory usage prediction device further includes: a training module, used to iteratively train an initial teacher model using the training dataset and the test dataset to obtain a memory usage teacher model; and to use the memory usage teacher model and the training dataset to perform knowledge distillation training on an initial student model to obtain the memory usage student model;

[0036] The testing module is used to evaluate the model performance of the memory usage student model using the test dataset.

[0037] In one embodiment, the training module is further configured to input sample data from the training dataset into the initial student model to obtain first predicted data output by the initial student model; obtain the hard label loss for knowledge distillation training based on the first predicted data and the hard label data in the training dataset; input sample data from the training dataset into the initial student model to obtain second predicted data output by the memory usage teacher model; perform temperature adjustment on the second predicted data to determine the soft label loss for knowledge distillation training; obtain the knowledge distillation loss based on the hard label loss and the soft label loss; and optimize the model parameters in the initial student model based on the knowledge distillation loss to obtain the memory usage student model.

[0038] In one embodiment, the training module is further configured to perform weighted calculation on the hard label loss and the soft label loss according to the weighting parameters to obtain the knowledge distillation loss; wherein the weighting parameters have different values ​​in different knowledge distillation training stages.

[0039] In one embodiment, the testing module is further configured to load the memory usage student model and the memory usage teacher model; use the memory usage student model to predict the test data in the test dataset to obtain third predicted data, and use the memory usage teacher model to predict the test data in the test dataset to obtain fourth predicted data; calculate the difference in loss functions based on the third predicted data and the fourth predicted data; and evaluate the model performance of the memory usage student model based on the difference in loss functions.

[0040] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0041] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the memory usage prediction method of the first aspect described above.

[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the memory usage prediction method of the first aspect described above.

[0043] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the memory usage prediction method of the first aspect described above.

[0044] The aforementioned memory usage prediction method, apparatus, device, storage medium, and program products acquire the current memory usage data of a target object; input the current memory usage data of the target object into a memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model; wherein, the memory usage student model is a model generated after knowledge distillation training using the output of the memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the model using the memory usage teacher model. Because the memory usage student model is trained using knowledge distillation with the memory usage teacher model, the smaller memory usage student model has a faster inference speed, thereby reducing computational resource consumption while maintaining high prediction performance. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a memory usage prediction method provided in an embodiment of this application;

[0047] Figure 2 A schematic diagram illustrating a teacher model and a student model based on memory usage, provided for embodiments of this application;

[0048] Figure 3 A flowchart illustrating another memory usage prediction method provided in this application embodiment;

[0049] Figure 4 A schematic diagram illustrating the testing and training of a student model based on memory usage, provided for an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of a memory usage prediction device provided in an embodiment of this application;

[0051] Figure 6 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] The relevant technologies will be explained below.

[0054] In data center and cloud computing systems, memory is one of the critical resources, and fluctuations in memory usage directly affect the overall performance and reliability of the system. Effectively predicting memory usage helps system administrators identify potential memory bottlenecks in advance, optimize resource allocation, and thus avoid system performance degradation or system crashes.

[0055] Traditional memory monitoring methods can only provide real-time data and simple statistical analysis, failing to effectively adapt to dynamically changing loads and long-term trend variations. Machine learning techniques can be used for memory usage prediction, learning complex memory change patterns and trends from large amounts of historical data. However, memory usage prediction using machine learning is data-driven and often requires complex machine learning models with numerous parameters and intricate computational graphs, leading to high computational resource consumption when handling large datasets. Furthermore, memory usage is influenced by application runtime, system load changes, and user behavior, requiring the handling of variations across different time scales, such as short-term memory fluctuations and long-term usage trends. This not only increases the difficulty of temporal feature extraction but also easily leads to gradient vanishing or exploding problems in machine learning models. Finally, traditional memory prediction methods often fit training data to complex decision boundaries in high-dimensional space. When training data is insufficient or feature redundancy exists, machine learning models are prone to capturing useless patterns, resulting in overfitting.

[0056] To address the aforementioned issues, embodiments of this application provide a memory usage prediction method, apparatus, device, storage medium, and program product. By using a memory usage teacher model for knowledge distillation to train a memory usage student model, a smaller memory usage student model can achieve faster inference speed, thereby reducing computational resource consumption while maintaining high prediction performance.

[0057] The memory usage prediction method provided in this application can be applied to a memory usage prediction device, which may include a terminal, a server, or a system including the terminal and the server. The terminal may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection devices. Portable wearable devices may include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays may be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In one embodiment, such as Figure 1 As shown, a memory usage prediction method is provided. This embodiment uses the application of this method to a server as an example for illustration. The memory usage prediction method includes S201-S202:

[0059] S201. Obtain the current memory usage data of the target object.

[0060] In this application, when it is necessary to predict the memory usage of a target object, the server can first obtain the current memory usage data of the target object.

[0061] The aforementioned target objects may include any device or system with memory, such as terminals, servers, data center systems, and cloud computing systems.

[0062] It should be understood that the aforementioned current memory usage data can be the memory usage data of the target object at the current moment and / or before the current moment. For example, memory usage data may include used memory, available memory, cached memory, buffered memory, memory usage percentage, etc.

[0063] S202. Input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model.

[0064] In this step, after the server obtains the current memory usage data of the target object, it can input the current memory usage data of the target object into the memory usage student model and obtain the memory usage trend prediction information of the target object output by the memory usage student model.

[0065] The aforementioned trend prediction information refers to the predicted changes in the memory usage of the target object over a future period.

[0066] Among them, the memory utilization student model is a model generated by knowledge distillation training using the output of the memory utilization teacher model as soft labels. The memory utilization student model has fewer model parameters than the model using the memory utilization teacher model.

[0067] In some embodiments, the memory usage teacher model and the memory usage student model described above can be the same model or different models. The appropriate combination of the memory usage teacher model and the memory usage student model can be dynamically selected according to the data scale of the memory usage data and the requirements of the prediction task.

[0068] For example, for predicting complex memory usage patterns, a larger-scale, more powerful Transformer-based teacher model of memory usage can be built; while for resource-constrained environments or scenarios with high real-time requirements, a lighter-weight student model of memory usage can be built. For example, the student model of memory usage can be based on a Transformer model or a lighter-weight Long Short-Term Memory (LSTM) network model.

[0069] In this application, the fusion of knowledge distillation and the Transformer model reduces the number of model parameters and computational complexity, enabling student models with lower memory usage to have faster inference speeds, thereby accelerating the response speed of memory usage prediction while maintaining high prediction performance.

[0070] Because of its powerful parallel computing capabilities, the Transformer model significantly improves the training speed and processing efficiency of memory usage data compared to traditional Recurrent Neural Networks (RNNs) or LSTMs. It can provide accurate predictions in a short time, helping system resource administrators make timely decisions. Furthermore, since memory usage data exhibits long-term trends and periodicity, compared to traditional time series forecasting methods, the Transformer model, used as both the student and teacher models for memory usage, does not rely on a stepwise processing mechanism. This makes it more accurate and efficient in handling long-term dependencies.

[0071] For example, Figure 2 A schematic diagram of a teacher model and a student model based on memory usage is provided in this application embodiment, as shown below. Figure 2As shown, both the memory usage teacher model and the memory usage student model are Transformer models. The memory usage teacher model can be composed of 6 encoder layers and 6 decoder layers.

[0072] Each encoder layer contains four units: a multi-head attention mechanism, residual connections, normalization, and a feedforward neural network. Correspondingly, the decoder layer has the same structure as the encoder layer, but its multi-head attention mechanism uses a mask to adjust the attention score and adds an attention mechanism to the encoder output.

[0073] Continue to refer to Figure 2 Compared to the memory utilization teacher model, the memory utilization student model has a simpler structure and fewer parameters. The memory utilization student model consists of three encoder layers and three decoder layers, with the encoder and decoder layers having the same structure as those in the memory utilization teacher model.

[0074] The structure of the encoder layer in the memory utilization teacher model is explained below.

[0075] For example, the multi-head attention mechanism unit in the encoder layer allows the memory utilization teacher model to simultaneously pay attention to information from different representation subspaces at different locations. After receiving the input sequence of the current memory utilization data, the memory utilization teacher model can pass the input sequence X to the h-th self-attention mechanism. Through three different linear transformation operations, it can obtain the query matrix Q of the h-th self-attention mechanism. h Key matrix K h Sum matrix V h Query matrix Q h The calculation is shown in formula (1), and the key matrix K h The calculation is shown in formula (2), and the value matrix V h The calculation is shown in formula (3).

[0076]

[0077]

[0078] Where X is the input sequence, X = [x1, x2, ... x i ,…x n ], x i Let i be the i-th element in the input sequence. This is a learnable weight matrix in a multi-head self-attention mechanism.

[0079] For example, the self-attention mechanism unit uses the query matrix Q h With bond matrix K hPerforming a dot product operation yields the attention score. To avoid the dot product result becoming too large, it can be divided by a scaling factor, which is the square root of the key vector dimension. Subsequently, the self-attention mechanism unit normalizes the attention score using an activation function (softmax) to obtain the attention weights. Finally, the self-attention mechanism unit combines the attention weights with the value matrix V. h We perform a weighted summation to obtain the output Z of the h-th self-attention mechanism. h The output Z of the self-attention mechanism h The calculation is shown in formula (4).

[0080]

[0081] Correspondingly, the outputs of multiple single-head self-attention mechanism units are concatenated and a linear transformation is performed to obtain the final multi-head self-attention mechanism output, as shown in Equation (5).

[0082] MultiHead_Z=Concat(Z1,Z2…,Zh)W Z (5)

[0083] Among them, Z h W is the output of the h-th self-attention mechanism unit. Z This is the weight matrix output by the multi-head self-attention mechanism.

[0084] For example, the residual connection unit in the encoder layer adds the input sequence X to the output of the multi-head self-attention mechanism to form the residual connection result ResidualOutput. The residual connection helps to alleviate the gradient vanishing problem in deep neural networks, so that information can flow more effectively in the network. The residual connection is shown in Equation (6).

[0085] ResidualOutput = MultiHead Z +X (6)

[0086] For example, the normalization unit in the encoder layer receives the residual connection result ResidualOutput output by the residual connection unit. The normalization unit is a method for normalizing the features of each input, which can accelerate model training and improve model stability. The normalization unit can process the input features as shown in Equation (7);

[0087] LayerNormOutput=LayerNorm(ResidualOutput) (7)

[0088] Among them, LayerNorm is used to characterize the normalization process, LayerNormOutput is the result of the normalization process, and ResidualOutput is the result of the residual connection.

[0089] For example, the feedforward neural network unit in the encoder layer includes two linear sub-units and one activation function unit. The feedforward neural network unit can further process and transform the features extracted by the multi-head self-attention mechanism unit, enhancing the expressive power of the model. The feedforward neural network unit feeds the output of the normalization unit into the first linear sub-unit to expand the feature dimension of the input. Subsequently, the feedforward neural network unit passes through the activation function (ReLU) in the activation function sub-unit to achieve nonlinear transformation of the input features. Finally, the feedforward neural network unit passes through the second linear layer sub-unit to restore the original feature dimension. The processing of features by the feedforward neural network unit can be shown in Equation (8);

[0090] FFNOutput=max(0,LayerNormOutputW e1 +b e1 W e2 +b e2 (8)

[0091] Where FFNOutput is the output of the feedforward neural network unit, LayerNormOutput is the normalization result, and W... e1 and b e1 W represents the learnable weights and bias parameters of the first linear subunit in the feedforward neural network unit of the encoder layer. e2 and b e2 These are the learnable weights and bias parameters of the second linear subunit in the feedforward neural network unit of the encoder layer.

[0092] For example, the residual connection unit in the encoder layer can also perform a residual connection between the output of the feedforward neural network unit and the output of the normalization layer unit, and then send the connection result to the normalization layer unit to obtain the final encoder layer output EncoderLayerOutput. The final encoder layer output is shown in formula (9):

[0093] EncoderLayerOutput=LayerNorm(FFNOutput+LayerNormOutput) (9)

[0094] For example, the memory utilization teacher model extracts rich feature representations from the input sequence by stacking six encoder layers in sequence. Each encoder layer receives the output of the previous layer as input, and finally obtains the output of the encoder layer of the memory utilization teacher model.

[0095] The following explains the structure of the decoder layer in the memory utilization teacher model.

[0096] For example, the mask multi-head self-attention unit in the decoder layer uses the target sequence Y output from the encoder layer to achieve this. t The query matrix Q of the h-th self-attention mechanism can be obtained by passing it to the h-th different self-attention mechanism through three different linear transformation operations. dh Key matrix K dh Sum matrix V dh Query matrix Q dh The calculation is shown in formula (10), and the key matrix K dh The calculation is shown in formula (11), value matrix V dh The calculation is shown in formula (12).

[0097]

[0098]

[0099] Among them, Y t Y is the target sequence output by the encoder layer. t =[y1,y2,…y i ,…y n ], y i Let i be the i-th element in the target sequence. and This is the learnable weight matrix in the masked multi-head self-attention mechanism unit.

[0100] It should be understood that since the masked multi-head self-attention mechanism unit in the decoder layer needs to prevent the current position from paying attention to the future position, it is necessary to use a mask to adjust the attention score, setting the score of the future position to a minimum value (such as negative infinity), so that the attention weight of the future position can become 0 after passing through the activation function (softmax). Based on this, the output of the single-head masked self-attention mechanism can be as shown in Equation (13). Finally, the outputs of each head are concatenated, and a linear transformation is used to obtain the final output of the masked multi-head self-attention mechanism unit, as shown in Equation (14):

[0101]

[0102] Among them, Z dh This is the output of the h-th mask self-attention mechanism in the decoder layer. This is the weight matrix for learning.

[0103] For example, the multi-head self-attention mechanism unit in the decoder layer can obtain the query matrix Q of the new h-th self-attention mechanism by passing the output of the mask multi-head self-attention mechanism unit to the h-th different self-attention mechanism through three different linear transformation operations. dh2 Key matrix K dh2 Sum matrix V dh2 Query matrix Q dh2 The calculation is shown in formula (15), and the key matrix K dh2 The calculation is shown in formula (16), and the value matrix V dh2 The calculation is shown in formula (17).

[0104]

[0105] in, and This is the learnable weight matrix in the multi-head self-attention mechanism module of the decoder layer.

[0106] For example, the multi-head attention mechanism in the decoder layer uses the decoder input as a query and the encoder output as a key and value, thereby capturing the interaction between the encoder and decoder. Therefore, the output of the single-head self-attention mechanism is shown in Equation (18). Finally, the outputs of each head are concatenated and then linearly transformed to obtain the final output of the multi-head self-attention unit, as shown in Equation (19).

[0107]

[0108] Among them, Z dh2 It is the output of the h-th self-attention mechanism in the decoder layer. It is a learnable weight matrix.

[0109] For example, by adding the output of the masked multi-head self-attention mechanism module to the output of the multi-head self-attention module in the decoder layer, a residual connection is formed, which helps to alleviate the gradient vanishing problem in deep neural networks and allows information to flow more effectively in the network. Its calculation is shown in Equation (20):

[0110]

[0111] ResidualOutput1 is the residual connection. MultiHead_Z is the output of the concatenated multi-head self-attention unit. dh This is the output of the multi-head self-attention mechanism unit of the concatenated mask.

[0112] For example, after the residual connections are formed in the decoder layer, the results of the residual connections can be sent to the normalization unit, and the calculation is shown in formula (21):

[0113] LayerNormOutput1=LayerNorm(ResidualOutput1) (21)

[0114] Here, LayerNormOutput1 is the output of the normalization unit of the decoder layer, LayerNorm is the normalization process, and ResidualOutput1 is the residual connection.

[0115] For example, the structure and calculation method of the feedforward neural network in the decoder layer are the same as those in the encoder layer, and its calculation is shown in formula (22). By performing a residual connection between the output of the feedforward neural network unit and the output of the normalization unit, and then feeding the connection result into the normalization unit, the output of the final decoder layer can be obtained, and its calculation is shown in formula (23):

[0116] Decoder_FFNOutput=max(0,LayerNormOutput1W d1 +b d1 W d2 +b d2 (twenty two)

[0117] DecoderLayerOutput=LayerNorm(Decoder_FFNOutput+LayerNormOutput1)(23)

[0118] Where Decoder_FFNOutput is the output of the feedforward neural network unit, LayerNormOutput1 is the normalization result, and W d1 and b d1 W represents the learnable weights and bias parameters of the first linear subunit in the feedforward neural network unit of the decoder layer. d2 and b d2 These are the learnable weights and bias parameters of the second linear subunit in the feedforward neural network unit of the encoder layer.

[0119] For example, the memory utilization teacher model generates the output of the memory utilization teacher model decoder layer step by step by stacking six decoder layers in sequence. Each decoder layer receives the output of the previous layer, the encoder output and the mask as input, and finally transforms the decoder output into the probability distribution of the target sequence through linear transformation and activation function (softmax).

[0120] It should be noted that the memory usage teacher model and memory usage student model in the embodiments of this application can be adjusted and extended according to specific tasks and needs. For example, the number of layers can be increased or decreased, the number of heads can be adjusted, and the size of the hidden layers can be changed to adapt to memory usage prediction tasks of different scales and complexities.

[0121] In this application, the memory usage teacher model and the memory usage student model process data through multi-layered encoder layers and leverage a multi-head self-attention mechanism to fuse information and context from different levels. This multi-layered information fusion capability enables the model to more comprehensively understand the complex patterns of memory usage data, thereby providing more accurate memory usage prediction results.

[0122] The following explains how to train the student model and the teacher model for memory usage.

[0123] In some embodiments, the server can obtain historical memory usage data of the target object over different time periods and clean the historical memory usage data. Next, the server can construct training and testing datasets based on the cleaned historical memory usage data. Then, the server can use the training and testing datasets to iteratively train the initial teacher model to obtain a memory usage teacher model, and use the memory usage teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain a memory usage student model. Finally, the server can use the testing dataset to evaluate the performance of the memory usage student model.

[0124] It should be understood that the embodiments of this application do not limit how to obtain historical memory usage data of the target object at different time periods. In some embodiments, historical memory usage data can be collected through a script (e.g., a shell script) that runs automatically on a server.

[0125] For example, a shell script can be written to collect historical memory usage data. The shell script collects historical memory usage data for different time periods at time intervals. The historical memory usage data includes data such as used memory, available memory, cached memory, buffered memory, and memory usage percentage. It is a variable parameter and the unit is seconds.

[0126] In some embodiments, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0127] For example, after obtaining historical memory usage data, the server can clean the historical data. During the cleaning process, at least one of the following cleaning operations can be performed: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data. Missing values ​​can be identified by checking blank fields and specific markers (such as "N / A") in the dataset and padded with zeros. Abnormal fluctuation data can be identified and cleared by recognizing whether there are anomalies in the data fluctuations of each data point. Measurement error data can be identified and cleared by recognizing historical memory usage data exceeding reasonable ranges for each data point.

[0128] In this application, various cleaning operations are used to identify and process noise and outliers in the data, improving the usability and representativeness of historical memory usage data. Compared to related technologies, this allows the model to process data without relying on complex feature engineering.

[0129] In some embodiments, after cleaning the historical memory usage data, the historical memory usage data can be divided according to a preset ratio to generate a training dataset and a test dataset. The hard label data of each historical memory usage data in the training dataset and the test dataset can be considered as labeled.

[0130] For example, the server can use a random number generator to randomly shuffle the cleaned historical memory usage data to avoid systematic bias in dividing the training and test datasets. After randomly shuffling the data, it is split into training and test sets according to a preset ratio (e.g., 7:3). For instance, if there are 1000 historical memory usage data points, the first 700 would be allocated to the training dataset, and the last 300 to the test dataset, according to a 7:3 ratio.

[0131] In some embodiments, the server can load the training dataset and the test dataset into the initial teacher model as an array to perform iterative training of the initial teacher model and obtain the memory utilization teacher model.

[0132] For example, the initial teacher model can have 20 training rounds, with 20,000 training iterations per round, and 128 training data points input each time. Iterative training of the initial teacher model is completed by minimizing the mean squared error loss function, resulting in a memory utilization-based teacher model.

[0133] It should be understood that the embodiments of this application do not limit how the initial student model is trained using knowledge distillation. In some embodiments, the server can first input sample data from the training dataset into the initial student model to obtain the first predicted data output by the initial student model, and then obtain the hard label loss for knowledge distillation training based on the first predicted data and the hard label data in the training dataset. Secondly, the server can input sample data from the training dataset into the initial student model to obtain the second predicted data output by the memory usage teacher model, and then perform temperature adjustment on the second predicted data to determine the soft label loss for knowledge distillation training. Thirdly, the server obtains the knowledge distillation loss based on the hard label loss and the soft label loss. Finally, the server can optimize the model parameters in the initial student model based on the knowledge distillation loss to obtain the memory usage student model.

[0134] The first predicted data can be the predicted data output by the initial student model for the sample data in the training dataset. The second predicted data can be the predicted data output by the memory usage teacher model for the sample data in the training dataset. The first and second predicted data correspond to the same sample data.

[0135] It should be understood that the hard label loss can be the traditional cross-entropy loss, which can be calculated using the hard labels in the training dataset. For example, for a sample dataset, suppose label_ac is the hard label (actual class label) corresponding to that sample dataset labeled in the training sample set, and p... ij Let the first prediction data be the probability distribution of the initial student model for the i-th sample data belonging to the j-th category. Then the hard label loss can be calculated as shown in formula (24):

[0136]

[0137] Where, m j For indicator functions, when label ac When =j, m j If true, its value is 1; otherwise, m j The result is false, and its value is 0. m j This is used to ensure that the initial student model can still learn the correct hard labels during training. hard This is the hard label loss.

[0138] It should be understood that the soft-label loss is calculated using the soft label of the memory utilization teacher model, which is the second predicted data output by the memory utilization teacher model for the sample data. The soft-label loss can be calculated as shown in formula (25):

[0139]

[0140] in, Soft labels for the teacher model of temperature-adjusted memory usage. for The j-th element in L represents the probability that the i-th sample belongs to the j-th class. soft This is the soft label loss.

[0141] In some embodiments, the server may perform weighted calculations on the hard label loss and soft label loss according to weighting parameters to obtain the knowledge distillation loss.

[0142] For example, based on the weighting parameters, the soft label loss and hard label loss can be weighted and calculated to obtain the knowledge distillation loss L. KD Knowledge distillation loss L KD The calculation method can be shown in formula (26):

[0143] L KD =(1-β)L hard +βL soft (26)

[0144] Here, β is a weighting parameter, a hyperparameter used to balance the hard label loss and the soft label loss.

[0145] It should be understood that the weighting parameters take different values ​​at different stages of knowledge distillation training, thus allowing the distillation intensity to be dynamically adjusted based on training progress and model performance. For example, in the early stages of training, a stronger distillation intensity can be set using the weighting parameters, enabling the initial student model to learn the knowledge of the memory usage teacher model more quickly; while in the later stages of training, a gradually decreasing distillation intensity can be set using the weighting parameters, thereby encouraging the initial student model to develop its own feature representations to improve generalization ability.

[0146] In this application, the memory usage teacher model learns the changing trends of historical memory usage data across different time periods and application scenarios to provide initial parameters for the initial student model, while also guiding the optimization direction of the initial student model, thereby accelerating the training process of the initial learning model. The initial student model inherits knowledge from the memory usage teacher model through knowledge distillation, rapidly improving its predictive ability without requiring lengthy independent training, thus quickly obtaining the memory usage student model. Furthermore, the memory usage student model has a relatively simple structure, making it easier to modify and extend to meet the memory usage prediction requirements in different scenarios.

[0147] In some embodiments, the server may first load the memory usage student model and the memory usage teacher model. Next, the server uses the memory usage student model to predict test data in the test dataset to obtain third predicted data, and uses the memory usage teacher model to predict test data in the test dataset to obtain fourth predicted data. Then, the server calculates the difference in the loss function based on the third and fourth predicted data. Finally, the server evaluates the model performance of the memory usage student model based on the difference in the loss function.

[0148] The third prediction data is the memory usage data predicted using the student model of memory usage, and the fourth prediction data is the memory usage data predicted using the teacher model of memory usage.

[0149] For example, the performance of the memory usage teacher model and the memory usage student model is compared on the test dataset. Using the third and fourth prediction data, the performance gap between the student model and the teacher model is analyzed. If the error value of the loss function corresponding to the third and fourth prediction data is less than 0.01, the student model can be considered to have met the expected performance requirements.

[0150] In some embodiments, the predictive performance of the memory usage student model can be visually displayed by plotting a confusion matrix between the predicted and true values. Accuracy, prediction precision, recall, and F1 score can be used as evaluation metrics to assess the performance of the memory usage student model.

[0151] In this application, the memory usage student model is customizable and portable, and can be optimized for different hardware platforms, enabling it to run efficiently on various devices (such as mobile devices, edge devices, etc.) and improving the practicality of memory usage prediction.

[0152] The memory usage prediction method provided in this application obtains the current memory usage data of a target object; inputs the current memory usage data of the target object into a memory usage student model, and obtains the memory usage trend prediction information of the target object output by the memory usage student model; wherein, the memory usage student model is a model generated after knowledge distillation training using the output of the memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the model using the memory usage teacher model. Because the memory usage student model is trained using knowledge distillation with the memory usage teacher model, the smaller memory usage student model has a faster inference speed, thereby reducing computational resource consumption while maintaining high prediction performance.

[0153] The following explains how to train and test the student model for memory usage. Figure 3This is a flowchart illustrating another memory usage prediction method provided in an embodiment of this application. Figure 4 A schematic diagram illustrating the testing and training of a memory usage student model is provided in this application embodiment, as shown below. Figure 3 and 4 As shown, this memory usage prediction method includes S301-S307:

[0154] S301. Obtain historical memory usage data of the target object at different time periods, and perform cleaning operations on the historical memory usage data.

[0155] In some embodiments, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0156] S302. Based on the cleaned historical memory usage data, construct the training dataset and the test dataset.

[0157] S303. Using the training dataset and the test dataset, iteratively train the initial teacher model to obtain the memory utilization teacher model.

[0158] S304. Using the memory utilization rate teacher model and training dataset, perform knowledge distillation training on the initial student model to obtain the memory utilization rate student model.

[0159] In some embodiments, the server first inputs sample data from the training dataset into the initial student model to obtain the first predicted data output by the initial student model. Based on the first predicted data and the hard-labeled data in the training dataset, the server obtains the hard-label loss for knowledge distillation training. Subsequently, the server inputs sample data from the training dataset into the initial student model to obtain the second predicted data output by the memory utilization teacher model. The server then performs temperature adjustment on the second predicted data to determine the soft-label loss for knowledge distillation training. Next, the server obtains the knowledge distillation loss based on the hard-label loss and the soft-label loss. Finally, the server optimizes the model parameters in the initial student model based on the knowledge distillation loss to obtain the memory utilization student model.

[0160] For example, the server can calculate the knowledge distillation loss by weighting the hard label loss and the soft label loss according to the weighting parameters; wherein the weighting parameters take different values ​​at different knowledge distillation training stages.

[0161] S305. Using the test dataset, evaluate the performance of the memory usage student model.

[0162] In some embodiments, the server may first load the memory usage student model and the memory usage teacher model. Next, the server uses the memory usage student model to predict test data in the test dataset to obtain third predicted data, and uses the memory usage teacher model to predict test data in the test dataset to obtain fourth predicted data. Then, the server calculates the difference in the loss function based on the third and fourth predicted data. Finally, the server evaluates the model performance of the memory usage student model based on the difference in the loss function.

[0163] S306. Obtain the current memory usage data of the target object.

[0164] S307. Input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model.

[0165] Among them, the memory utilization student model is a model generated by knowledge distillation training using the output of the memory utilization teacher model as soft labels. The memory utilization student model has fewer model parameters than the model using the memory utilization teacher model.

[0166] In this application, knowledge distillation-based training of the memory utilization student model is guided by soft labels generated by the memory utilization teacher model, which effectively simplifies the training time and computational cost of the memory utilization student model. This significantly reduces the complexity of model tuning, thereby shortening the development cycle.

[0167] Furthermore, by transferring the rich knowledge of the memory usage teacher model to the memory usage generator model, the generalization ability of the memory usage student model under different memory usage scenarios can be improved. Compared with traditional models, it can perform more accurately when faced with new or unseen memory usage patterns.

[0168] The memory usage prediction method provided in this application obtains the current memory usage data of a target object; inputs the current memory usage data of the target object into a memory usage student model, and obtains the memory usage trend prediction information of the target object output by the memory usage student model; wherein, the memory usage student model is a model generated after knowledge distillation training using the output of the memory usage teacher model as soft labels, and the model parameters of the memory usage student model are fewer than those of the model using the memory usage teacher model. Because the memory usage student model is trained using knowledge distillation with the memory usage teacher model, the smaller memory usage student model has a faster inference speed, thereby reducing computational resource consumption while maintaining high prediction performance.

[0169] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0170] Based on the same inventive concept, this application also provides a memory usage prediction device for implementing the memory usage prediction method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more memory usage prediction device embodiments provided below can be found in the limitations of the memory usage prediction method described above, and will not be repeated here.

[0171] In one exemplary embodiment, such as Figure 5 As shown, a memory usage prediction device 400 is provided, including: an acquisition module 401 and a prediction module 402, wherein:

[0172] The acquisition module 401 is used to acquire the current memory usage data of the target object.

[0173] The prediction module 402 is used to input the current memory usage data of the target object into the memory usage student model, and to obtain the memory usage trend prediction information of the target object output by the memory usage student model.

[0174] Among them, the memory utilization student model is a model generated by knowledge distillation training using the output of the memory utilization teacher model as soft labels. The memory utilization student model has fewer model parameters than the model using the memory utilization teacher model.

[0175] In one embodiment, the acquisition module 401 is further configured to acquire historical memory usage data of the target object at different time periods, and perform cleaning operations on the historical memory usage data; and construct a training dataset and a test dataset based on the cleaned historical memory usage data.

[0176] The memory usage prediction device 400 further includes: a training module 403, used to iteratively train an initial teacher model using a training dataset and a test dataset to obtain a memory usage teacher model; and to use the memory usage teacher model and the training dataset to perform knowledge distillation training on an initial student model to obtain a memory usage student model.

[0177] Test module 404 is used to evaluate the performance of the student model in terms of memory usage using the test dataset.

[0178] In one embodiment, the training module 403 is further configured to input sample data from the training dataset into the initial student model to obtain first predicted data output by the initial student model; obtain the hard label loss for knowledge distillation training based on the first predicted data and the hard label data in the training dataset; input sample data from the training dataset into the initial student model to obtain second predicted data output by the memory utilization teacher model; perform temperature adjustment on the second predicted data to determine the soft label loss for knowledge distillation training; obtain the knowledge distillation loss based on the hard label loss and the soft label loss; and optimize the model parameters in the initial student model based on the knowledge distillation loss to obtain the memory utilization student model.

[0179] In one embodiment, the training module 403 is further configured to perform weighted calculation on the hard label loss and the soft label loss according to the weighting parameters to obtain the knowledge distillation loss; wherein the weighting parameters have different values ​​in different knowledge distillation training stages.

[0180] In one embodiment, the testing module 404 is further configured to load a student model and a teacher model for memory usage; use the student model for memory usage to predict test data in the test dataset to obtain third predicted data, and use the teacher model for memory usage to predict test data in the test dataset to obtain fourth predicted data; calculate the difference in loss functions based on the third and fourth predicted data; and evaluate the model performance of the student model for memory usage based on the difference in loss functions.

[0181] In one embodiment, the cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

[0182] The modules in the aforementioned memory usage prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0183] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a memory usage prediction method.

[0184] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the memory usage prediction method described above.

[0186] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the memory usage prediction method described above.

[0187] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the memory usage prediction method described above.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0190] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for predicting memory usage, characterized in that, The method includes: Obtain historical memory usage data of the target object at different time periods, and perform cleaning operations on the historical memory usage data; Based on the cleaned historical memory usage data, a training dataset and a test dataset are constructed. Using the training and test datasets, the initial teacher model is iteratively trained to obtain the memory utilization teacher model. Using the memory utilization teacher model and the training dataset, the initial student model is trained by knowledge distillation to obtain the memory utilization student model. Using the test dataset, evaluate the model performance of the memory usage student model; Get the current memory usage data of the target object; Input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model; The memory usage student model is a model generated by knowledge distillation training using the output of the memory usage teacher model as soft labels. The model parameters of the memory usage student model are fewer than those of the model parameters of the memory usage teacher model.

2. The method according to claim 1, characterized in that, The step of using the memory utilization teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain the memory utilization student model includes: Input the sample data from the training dataset into the initial student model to obtain the first predicted data output by the initial student model; Based on the first predicted data and the hard-label data in the training dataset, the hard-label loss for the knowledge distillation training is obtained; Input the sample data from the training dataset into the initial student model to obtain the second prediction data output by the memory usage teacher model; Temperature adjustment is performed on the second prediction data to determine the soft label loss of the knowledge distillation training; Based on the hard label loss and soft label loss, the knowledge distillation loss is obtained; The model parameters in the initial student model are optimized based on the knowledge distillation loss to obtain the memory usage student model.

3. The method according to claim 2, characterized in that, The process of obtaining the knowledge distillation loss based on the hard label loss and the soft label loss includes: The hard label loss and soft label loss are weighted and calculated according to the weighting parameters to obtain the knowledge distillation loss; wherein the weighting parameters have different values ​​in different knowledge distillation training stages.

4. The method according to claim 1, characterized in that, The evaluation of the memory usage student model's performance using the test dataset includes: Load the memory usage student model and memory usage teacher model; The memory usage student model is used to predict the test data in the test dataset to obtain the third predicted data, and the memory usage teacher model is used to predict the test data in the test dataset to obtain the fourth predicted data. Calculate the difference in the loss function based on the third and fourth prediction data; The performance of the memory usage student model is evaluated based on the difference in the loss function.

5. The method according to claim 1, characterized in that, The cleaning operation includes at least one of the following: clearing measurement error data, clearing abnormal fluctuation data, and filling in missing data.

6. A memory usage prediction device, characterized in that, The device includes: The acquisition module is used to acquire historical memory usage data of the target object at different time periods and clean the historical memory usage data; based on the cleaned historical memory usage data, training datasets and test datasets are constructed. The training module is used to iteratively train the initial teacher model using the training dataset and the test dataset to obtain the memory utilization teacher model; and to use the memory utilization teacher model and the training dataset to perform knowledge distillation training on the initial student model to obtain the memory utilization student model. The testing module is used to evaluate the performance of the student model in terms of memory usage using a test dataset. The acquisition module is used to obtain the current memory usage data of the target object; The prediction module is used to input the current memory usage data of the target object into the memory usage student model, and obtain the memory usage trend prediction information of the target object output by the memory usage student model; The memory usage student model is a model generated by knowledge distillation training using the output of the memory usage teacher model as soft labels. The model parameters of the memory usage student model are fewer than those of the model parameters of the memory usage teacher model.

7. The apparatus according to claim 6, characterized in that, The training module is further configured to input sample data from the training dataset into the initial student model to obtain the first predicted data output by the initial student model; obtain the hard label loss for knowledge distillation training based on the first predicted data and the hard label data in the training dataset; input sample data from the training dataset into the initial student model to obtain the second predicted data output by the memory utilization teacher model; perform temperature adjustment on the second predicted data to determine the soft label loss for knowledge distillation training; obtain the knowledge distillation loss based on the hard label loss and the soft label loss; and optimize the model parameters in the initial student model based on the knowledge distillation loss to obtain the memory utilization student model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images