Memory allocation method and system based on deep learning, electronic equipment and storage medium

Abstract

The invention relates to the technical field of memory management, in particular to a memory allocation method and system based on deep learning, electronic equipment and a storage medium, and the method comprises the following steps: obtaining a historical memory task data set and preprocessing the historical memory task data set; extracting memory related features according to the preprocessed historical memory task data set; constructing a memory evaluation model based on deep learning, and training the memory evaluation model by using the memory related features; and acquiring real-time task data, and outputting an optimal memory allocation result by utilizing the trained memory evaluation model. The invention aims to realize differentiated dynamic memory allocation and improve the utilization rate of memory resources.

Description

Technical Field

[0001] The present invention relates to the field of memory management technology, and in particular to a memory allocation method, system, electronic device and storage medium based on deep learning. Background Art

[0002] In the big data analysis task scenario, memory resource allocation plays a vital role. Existing technologies usually use big data computing frameworks such as Hive SQL based on the MapReduce engine to perform tasks. In the memory allocation link, a relatively rough estimation method is often used. Taking the MapReduce computing task as an example, when faced with 300M data, the computing engine generally splits it into (128M+128M+44M), enables 3 task nodes for processing, and allocates 1G memory resources by default. If the task is executed smoothly, it is considered that this 1G memory allocation is reasonable; if the task fails, the memory is continuously increased until the task runs successfully.

[0003] However, this traditional memory allocation method has certain limitations. First, the memory required for data analysis tasks is affected by many complex factors, such as the amount of data to be processed, the number of screening conditions, the screening method, the type of calculation operator, and the number of data columns to be processed. Simply determining the task memory requirements based on a formula such as (data size / 128M+1G) is difficult to accurately adapt to the actual situation. When processing simple tasks, it is easy to cause idle and wasteful memory resources; and when facing complex tasks, it is very easy to cause task failure due to insufficient memory. Secondly, in daily data analysis tasks, key factors such as data size are in a dynamic state of change, and the corresponding memory requirements also fluctuate accordingly. The existing memory allocation strategy lacks dynamic adaptability. When executing periodic tasks, it is impossible to flexibly adjust the memory allocation according to the real-time changes in task characteristics, resulting in a serious disconnect between the allocated resources and actual needs. Finally, in the execution of multi-node tasks, since the minimum memory required for the task depends on the maximum memory of a subtask, for example, a task that requires 3 nodes to process, the minimum memory is 1G, and the maximum memory is 3G. In order to ensure the smooth execution of the task, the existing technology needs to configure 3G memory resources for each node, which aggravates the waste of memory resources. Summary of the invention

[0004] In order to achieve differentiated dynamic memory allocation and improve memory resource utilization, the present invention provides a memory allocation method, system, electronic device and storage medium based on deep learning. The technical solutions adopted are as follows:

[0005] The technical solution of the first aspect of the present invention provides a memory allocation method based on deep learning, the method comprising:

[0006] Acquire a historical memory task data set and preprocess the historical memory task data set;

[0007] Extract memory-related features based on the preprocessed historical memory task dataset;

[0008] Build a memory evaluation model based on deep learning and use memory-related features to train the memory evaluation model;

[0009] Obtain real-time task data and use the trained memory evaluation model to output the optimal memory allocation result.

[0010] Furthermore, obtaining the historical task data set and preprocessing the historical task data set includes:

[0011] Obtain relevant feature vectors and memory spaces of different historical memory tasks to obtain a historical memory task dataset;

[0012] One-hot encoding is used to handle categorical variables in the historical memory task dataset.

[0013] Furthermore, the relevant feature vectors include data size, computing operator, data format, task type, data column set, data screening column set, data output column set, compression ratio and storage type.

[0014] Furthermore, memory-related features are extracted based on the preprocessed historical memory task dataset, including:

[0015] Extract memory-related feature combinations based on the target task type;

[0016] The embedding layer is used to process the memory-related feature combination and extract the embedded representation of the memory-related feature combination.

[0017] Furthermore, a memory evaluation model is constructed based on deep learning, and the memory evaluation model is trained using memory-related features, including:

[0018] A memory evaluation model is built based on a multi-layer perceptron. Each layer defines a weight matrix and a bias vector and uses ReLU as a nonlinear activation function.

[0019] The memory evaluation model is used to transform memory-related features, and the output layer outputs the predicted memory space prediction results.

[0020] Furthermore, building a memory evaluation model based on deep learning and training the memory evaluation model using memory-related features also include:

[0021] The mean square error is used as the regression loss function to calculate the error between the predicted value and the actual value;

[0022] The parameters of the memory evaluation model are updated using gradient descent to minimize the regression loss function.

[0023] Furthermore, the regression loss function is expressed as:

[0024]

[0025] Where L(θ) represents the regression loss function; represents the predicted memory value of the i-th historical memory task; y i represents the actual memory value of the i-th historical memory task; n represents the number of samples of the historical memory task.

[0026] The technical solution of the second aspect of the present invention provides a memory evaluation system based on deep learning, which adopts the memory allocation method based on deep learning described in the technical solution of the first aspect of the present invention, and the system includes:

[0027] A data processing module, configured to obtain a historical memory task data set and preprocess the historical memory task data set;

[0028] a feature extraction module configured to extract memory-related features based on the preprocessed historical memory task dataset;

[0029] A memory evaluation module, configured to build a memory evaluation model based on deep learning and train the memory evaluation model using memory-related features;

[0030] The memory allocation module is configured to obtain real-time task data and output the optimal memory allocation result using the trained memory evaluation model.

[0031] The technical solution of the third aspect of the present invention provides an electronic device, comprising: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the deep learning-based memory allocation method described in the technical solution of the first aspect of the present invention.

[0032] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing a memory allocation method based on deep learning. The program for implementing a memory allocation method based on deep learning is executed by a processor to implement the steps of the memory allocation method based on deep learning described in the technical solution of the first aspect of the present invention.

[0033] The present invention has the following beneficial effects:

[0034] The memory allocation method based on deep learning provided by the present invention has the ability to dynamically allocate memory, can dynamically adapt to the needs according to different scenarios, and does not require manual intervention and adjustment. It can effectively avoid the waste of memory resources and unreasonable allocation in daily periodic tasks, and effectively improve the accuracy and efficiency of memory resource allocation. Through the study of a large amount of historical memory task data, the memory evaluation model can deeply explore the complex relationship between memory-related features and memory usage, so that when facing new real-time tasks, it can accurately predict its memory requirements, avoid the limitations of traditional memory allocation methods based on experience or fixed rules to allocate memory, and reduce the problems of low task execution efficiency and waste of system resources caused by unreasonable memory allocation; especially in complex memory management scenarios such as large-scale data processing and multi-task parallel execution, it has significant advantages and provides an effective allocation method for the refined management of memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A method flow chart of a deep learning-based memory allocation method provided by one embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a deep learning-based memory allocation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the memory allocation method, system, electronic device and storage medium based on deep learning proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0039] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0040] The specific scheme of the deep learning-based memory allocation method, system, electronic device and storage medium provided by the present invention is described in detail below with reference to the accompanying drawings.

[0041] See also Figure 1 , which shows a method flow chart of a memory allocation method based on deep learning provided by an embodiment of the present invention, the method comprising:

[0042] Step S100: Acquire a historical memory task data set and preprocess the historical memory task data set;

[0043] Step S100 specifically includes:

[0044] Step S110: Obtain relevant feature vectors and memory spaces of different historical memory tasks to obtain a historical memory task data set; specifically, extract the task execution log generated by the operating system or application, which records the task start time, end time, various events during the execution process, and related resource usage; for example, in a database management system, the log records in detail the data read volume, write volume, table structure information involved, etc. of each query task, from which features such as data size can be extracted; on the other hand, monitor the memory usage dynamics during the execution of historical memory tasks in real time to obtain the actual memory usage size of each task; in this embodiment, the historical memory task data set can be expressed as where x i Represents the feature vector of the i-th task; y i Indicates the actual memory usage size of the i-th task; the feature vector includes but is not limited to data size, calculation operator, data format, task type, data column set, data filter column set, data output column set, compression ratio and storage type;

[0045] Step S120: Use one-hot encoding to process categorical variables in the historical memory task data set; specifically, extract categorical variables from the constructed historical memory task data set, such as calculation operators, data formats, task types, data column sets, data filter column sets, data output column sets and storage types are all categorical variables, and then use one-hot encoding to represent the category information of the categorical variables in a numerical form that is more suitable for machine learning model processing, avoiding problems such as category order errors that may be caused by traditional numerical encoding methods, so that the model can more accurately learn the relationship between categorical variables and memory requirements, and avoid model learning bias caused by improper encoding.

[0046] Step S200: extracting memory-related features according to the preprocessed historical memory task data set;

[0047] Step S200 specifically includes:

[0048] Step S210: extract memory-related feature combinations according to the target task type; specifically, classify according to the target task type; for example, the target task type can be divided into data processing tasks (such as data cleaning, data conversion), data analysis tasks (such as data statistical analysis, data mining), data storage tasks (such as data backup, data archiving) and other categories; for target tasks of different categories, select matching memory features; for data processing tasks, pay attention to feature factors such as data format, data volume and calculation operators, which directly affect the memory usage and read and write operations of data during processing; for data analysis tasks, in addition to the above features, it is also necessary to consider features such as aggregation operators, number of aggregation columns and number of output columns, which are related to Data aggregation and result output in the data analysis process are related and have a significant impact on memory requirements. For data storage tasks, features such as storage type and compression ratio become key factors because they determine the memory usage and storage efficiency of data when it is stored. This embodiment constructs feature combinations based on the mapping relationship between task types and memory features. For data processing tasks, key features are selected as data format, data volume, and calculation operators, and these features are combined to form a feature combination vector. For data analysis tasks, features such as aggregation operators, number of aggregation columns, and number of output columns are combined. Such feature combinations can specifically reflect the memory requirements of different types of tasks and improve the accuracy of subsequent models in predicting memory requirements.

[0049] Step S220: Process the memory-related feature combination using the embedding layer to extract the embedding representation of the memory-related feature combination; specifically, for each selected feature combination in step S210, the corresponding embedding matrix needs to be initialized; for example, based on the storage-related feature combination: data format, compression ratio, storage type; initialize the data format embedding matrix, compression ratio embedding matrix and storage type embedding matrix respectively; the dimension of the embedding matrix is ​​determined according to the number of feature categories and the desired embedding space dimension; for the selected feature combination, perform embedding representation calculation, first perform one-hot encoding on the data format, compression ratio and storage type; then, multiply the one-hot encoded vectors with the corresponding embedding matrices respectively, and add the results to obtain the embedding representation of the combined feature; this embodiment also provides based on the calculation-related feature combination: aggregation operator, number of aggregation columns, number of output columns; for this feature combination, it can be processed in the same way, and the originally discrete classification features or numerical features are mapped to a low-dimensional continuous vector space through the embedding layer, so that the relationship between the features can be better captured by the model, and these features can be more effectively used in subsequent model calculations to predict memory requirements. Step S220 uses an embedding layer to process feature combinations and maps features to a low-dimensional continuous vector space. On the one hand, this effectively reduces feature dimensions, reduces model calculation complexity, and improves the efficiency of model training and prediction. On the other hand, the embedding layer can mine potential semantic relationships between features, so that the model can better understand and use these features to predict memory requirements, enhance the model's adaptability and accuracy to changes in memory requirements for complex tasks, and reduce problems such as task execution delays or system failures caused by improper memory allocation.

[0050] Step S300: Building a memory evaluation model based on deep learning, and training the memory evaluation model using memory-related features; the memory evaluation model is based on the execution of historical data tasks, extracting feature combinations from execution logs, and training a memory evaluation model, the function of which is to recommend the optimal memory size for the task;

[0051] Step S300 specifically includes:

[0052] Step S310: Build a memory evaluation model based on a multi-layer perceptron, define a weight matrix and a bias vector for each layer and use ReLU as a nonlinear activation function; determine the number of model structure layers and the number of neurons in each layer: specifically, the number of hidden layers of the multi-layer perceptron (MLP) can be determined according to the complexity of the memory evaluation task to be processed; for example, for several types of tasks and the memory requirements are affected by fewer factors, fewer hidden layers, 2-3 layers, are set; and for complex situations involving multiple different types of tasks and the memory requirements are affected by many factors, such as processing diversified data processing and analysis tasks in large-scale data centers, more hidden layers, such as 4-6 layers or even more, are required, so that the model can learn sufficiently complex feature relationships to accurately evaluate memory requirements; then according to the input processed feature vector x i The dimension of the input layer determines the number of neurons in each layer; in this embodiment, the number of neurons in the input layer is consistent with the dimension of the feature vector. For example, if the relevant feature combination contains 10 features, 10 neurons are set in the input layer, and each neuron receives a feature value as input; for the hidden layer, the number of neurons is set according to the gradually decreasing number from the input layer to the output layer;

[0053] Weight matrix initialization: For each layer (from the first hidden layer to the Lth hidden layer and the final output layer), define the corresponding weight matrix W l ; The dimension of the weight matrix depends on the number of input neurons and output neurons of the layer. For example, for the first hidden layer, if the input layer has n neurons and the first hidden layer has m neurons, then W l The dimension is m×n; when initializing the weight matrix, the random initialization method is used to obtain the initial weight value, which provides a starting parameter state for the model. The subsequent training process continuously adjusts and optimizes these weight values ​​so that it can learn the appropriate feature mapping relationship. Bias vector initialization: Similarly, define the bias vector b for each layer l , whose dimension is consistent with the number of neurons in this layer; the initial value of the bias vector is generally initialized randomly; the role of the bias vector is to provide an additional adjustable offset for the neuron, which helps the model to better fit the data and enables the model to flexibly adapt to different input-output relationships;

[0054] Finally, in each layer except the output layer (i.e., hidden layer), ReLU (Rectified Linear Unit) is configured as the nonlinear activation function; for the neuron output of each hidden layer, after linear transformation (i.e., multiplication with the weight matrix and addition of the bias vector), the ReLU function is applied for activation; for example, for the jth neuron in the first hidden layer, its input is x ij, first calculate the result after linear transformation, and then obtain the output after activation through the ReLU function; finally, the output of the entire hidden layer is the vector composed of the outputs of all neurons after ReLU activation; the advantage of using the ReLU function in this embodiment is that it can introduce nonlinear factors to solve the problem that the linear model cannot handle complex nonlinear relationships. At the same time, the calculation is simple and efficient, and it can alleviate the gradient disappearance problem to a certain extent, which helps the model converge faster during the training process and better learn the complex nonlinear mapping relationship between memory-related features, thereby improving the accuracy of the model's memory demand assessment;

[0055] Step S320: Use the memory evaluation model to transform the memory-related features, and the output layer outputs the predicted memory space prediction result; specifically, the memory-related feature vector processed by the previous step is used as input to the input layer of the constructed multi-layer perceptron model, and the weight matrix, bias vector and activation function of each layer defined in step S310 are calculated in sequence to realize the layer-by-layer transformation of the features; at the output layer, according to Perform calculations, Represents the predicted memory value of the i-th historical memory task; get the predicted memory usage size, here the weight matrix W of the output layer l+1 and the bias vector b l+1 It is also a parameter defined when the model is built. Through the forward propagation process, combined with the input feature vector and the transformation of each hidden layer, the final output layer gives a predicted value of the memory requirement corresponding to the input feature; this predicted value is the model's estimate of the memory size required for the task based on the memory-related features of the input, after feature transformation and mapping, which can be used to recommend appropriate memory allocation for actual tasks. This embodiment constructs a memory evaluation model framework with feature learning and nonlinear mapping capabilities by constructing a multi-layer perceptron model structure, defining weight matrices and bias vectors for each layer, and configuring an effective nonlinear activation function ReLU, so that the model can dig out the deep relationship between complex memory-related features and memory requirements, breaking through the limitations of linear models and enhancing the adaptability and fitting ability to changes in memory requirements of diversified tasks; finally, the constructed model is used to perform forward propagation calculations and output predicted memory space prediction results, thereby realizing the efficient conversion of input memory-related features into quantitative estimates of memory requirements, providing an intuitive and critical reference basis for actual memory allocation decisions, helping to improve the accuracy of memory allocation, avoiding waste of memory resources or obstruction of task execution due to insufficient memory, and thus improving the efficiency and reliability of the memory management system as a whole.

[0056] Step S330: Use mean square error as regression loss function to calculate the error between the predicted value and the actual value; Collect sample data pairs: Specifically, in the model training phase, obtain the input feature vector x from the existing historical memory task data set that has been preprocessed and feature extracted. i The corresponding sample data pair (x i ,y i ), where x i is the previously processed feature vector, which contains various memory-related feature information, y i is the actual memory usage size corresponding to the historical task. For example, the feature vector of the historical data processing task contains relevant feature information such as the data volume and calculation operator of the task, and the corresponding y i It records the size of the memory space occupied by the task when it is actually executed. Such sample data pairs are collected to form a training data set; the feature vector x of each sample is i The samples are sequentially input into the built memory evaluation model based on multi-layer perceptron, and the predicted memory values ​​corresponding to all samples are calculated according to the forward propagation calculation process of the model (that is, through the linear transformation of each hidden layer and the effect of ReLU activation function, the predicted memory value is finally obtained in the output layer);

[0057] Step S340: Use the gradient descent method to update the parameters of the memory evaluation model to minimize the regression loss function; the expression of the regression loss function is:

[0058]

[0059] Where L(θ) represents the regression loss function; represents the predicted memory value of the i-th historical memory task; y i represents the actual memory value of the i-th historical memory task; n represents the number of samples of the historical memory task; specifically, according to the expression of the mean square error (MSE) loss function, the predicted memory value and the actual memory value corresponding to each sample are substituted into the formula for calculation; this loss value quantitatively reflects the degree of difference between the model prediction result and the actual situation. The smaller the loss value, the closer the model prediction is to the actual memory usage, and the better the model performance is;

[0060] This embodiment uses the back propagation algorithm to calculate the mean square error loss function L(θ) about the weight matrix W of each layer of the model l and the bias vector b l Gradient; starting from the output layer, according to the chain rule, gradually reverse the contribution of each layer parameter to the loss function, that is, the gradient. For example, for the weight matrix W of the output layer L+1 , first calculate the derivative of the loss function with respect to the output layer output, and then combine the output layer to calculate About WL+1 The derivative of , using the chain rule, we can get the loss function with respect to W L+1 Gradient; in the same way, back propagate to calculate the gradient of the weight matrix and bias vector of each hidden layer in turn; then based on the calculated gradient, use the gradient descent method to update the model parameters, update all the parameters of the model, and complete a parameter update iteration. Finally, repeat the above process, perform forward propagation to calculate the predicted value, calculate the loss function, back propagate to calculate the gradient, and update the parameters for many times, and continuously adjust the model parameters so that the value of the loss function gradually decreases until the preset stop condition is reached. At this time, the model has learned the appropriate parameters and can predict the memory usage size based on the input memory-related features; using the mean square error as the loss function, it can accurately quantify the difference between the memory value predicted by the model and the actual memory value, providing an intuitive and effective standard for measuring model performance; and using the gradient descent method to update the model parameters based on the gradient of the loss function, the model can continuously adjust its own key parameters such as weight matrix and bias vector in the direction of minimizing the loss function according to error feedback, gradually improve the ability to fit the relationship between memory-related features and actual memory requirements, and thus improve the accuracy of memory evaluation.

[0061] Step S400: Obtain real-time task data, and use the trained memory evaluation model to output the optimal memory allocation result; specifically, deploy a memory monitoring tool (such as Prometheus combined with relevant memory monitoring exporter) to monitor and collect the real-time memory usage of the system and memory-related indicators during task execution, including but not limited to the amount of memory currently occupied by the task, the growth rate of memory usage, etc. By collecting these data regularly (such as every second or every few seconds), it is possible to grasp the dynamic changes of memory during the execution of the task in real time, and provide more comprehensive input information for the memory evaluation model. It is also necessary to clean the acquired real-time task data. For example, if the task data volume information format obtained from the task management system is incorrect, it is corrected or marked as invalid data; at the same time, the data is verified for integrity to ensure that all necessary feature information has been obtained. If some key features are missing (such as the task type is not clearly specified), try to infer from other relevant information or use default values; then convert the cleaned task data into the feature format required by the model. For categorical variables, such as task type and data format, the same one-hot encoding method as used in model training is used; for numerical variables, such as data size and memory usage growth rate, normalization or standardization is performed to match the range of data features during model training. Then, the converted and encoded features are combined into feature vectors in the order required by the model, and the constructed real-time task feature vectors are input into the trained memory evaluation model. The model calculates based on the weight matrices and activation functions of each layer in its internal multi-layer perceptron. The feature vectors are first connected to the input layer neurons, and then sequentially undergo linear transformation and nonlinear activation of the hidden layer, and finally the predicted memory usage size is obtained at the output layer.

[0062] The following is an example of a video rendering farm, where there are a large number of video rendering tasks to be processed. These tasks have different video resolutions, frame rates, special effects, etc., so the memory requirements vary greatly. When a new video rendering task is submitted to the task management system, step S400 is started. First, the task management system sends the relevant information of the task (such as video resolution, frame rate, light and shadow effects, etc.) to the data acquisition module of the memory evaluation model. At the same time, the memory monitoring system starts to collect the memory usage of the current system and the initial memory occupancy information related to the task; then, the data preprocessing module cleans and converts this information, converts the video resolution and frame rate information into a feature format recognizable by the model, performs one-hot encoding on the task type (video rendering), normalizes the numerical information related to memory usage, and constructs a feature vector; then, the feature vector is input into the trained memory evaluation model. The model predicts that the video rendering task requires 8GB of memory after calculation. The system directly allocates 8GB of memory space for the task to ensure that the video rendering task has enough memory during execution to process high-resolution video data, complex special effects calculations, etc., to avoid rendering freezes, screen tearing, and even task failures due to insufficient memory. At the same time, it also avoids the situation where excessive memory allocation causes a reduction in available memory resources for other tasks, thereby improving the task processing efficiency and resource utilization of the entire video rendering farm.

[0063] For another example, in the order processing system of a large e-commerce platform, there is a massive amount of order data that needs to be processed, including tasks such as entering order information, updating order status, and statistical analysis of order data. These tasks have different types and data processing requirements, and their memory requirements are also different; when a new order statistical analysis task is triggered, the relevant data of the task is obtained from the task management module of the order processing system, including order data volume, task type, calculation operator, data format and other information. Then, the data is preprocessed, the data volume is converted into appropriate numerical units, and categorical variables such as task type and calculation operator are one-hot encoded to construct a feature vector. Next, the feature vector is input into the trained memory evaluation model. The model predicts that the task requires approximately 3GB of memory after calculation; finally, the order processing system allocates nearly 3GB of memory space to the task based on this prediction result to ensure that the task can run efficiently;

[0064] In summary, this memory allocation method based on deep learning can dynamically adapt to various scenarios, allocate memory differently based on different scenarios, and does not require manual adjustment, which is effective in daily periodic tasks. At the same time, memory allocation can be set to the subtask level, effectively improving resource utilization and task success rate. It realizes highly intelligent and precise memory allocation, and can accurately predict memory requirements based on the diverse characteristics of real-time tasks, effectively overcoming the limitations of traditional memory allocation methods. It has great advantages in dealing with complex and dynamically changing task environments, providing a reliable solution for memory management.

[0065] See also Figure 2 As shown, the technical solution of the second aspect of the present invention provides a memory evaluation system based on deep learning, which adopts the memory allocation method based on deep learning described in the technical solution of the first aspect of the present invention, and the system includes:

[0066] A data processing module, configured to obtain a historical memory task data set and preprocess the historical memory task data set;

[0067] a feature extraction module configured to extract memory-related features based on the preprocessed historical memory task dataset;

[0068] A memory evaluation module, configured to build a memory evaluation model based on deep learning and train the memory evaluation model using memory-related features;

[0069] The memory allocation module is configured to obtain real-time task data and output the optimal memory allocation result using the trained memory evaluation model.

[0070] The technical solution of the third aspect of the present invention provides an electronic device, comprising: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the deep learning-based memory allocation method described in the technical solution of the first aspect of the present invention.

[0071] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing a memory allocation method based on deep learning. The program for implementing a memory allocation method based on deep learning is executed by a processor to implement the steps of the memory allocation method based on deep learning described in the technical solution of the first aspect of the present invention.

[0072] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A memory allocation method based on deep learning, characterized in that: The method comprises: Acquire a historical memory task data set and preprocess the historical memory task data set; Extract memory-related features based on the preprocessed historical memory task dataset; Build a memory evaluation model based on deep learning and use memory-related features to train the memory evaluation model; Obtain real-time task data and use the trained memory evaluation model to output the optimal memory allocation result.

2. The memory allocation method based on deep learning as claimed in claim 1, characterized in that: Obtaining the historical task data set and preprocessing the historical task data set includes: Obtain relevant feature vectors and memory spaces of different historical memory tasks to obtain a historical memory task dataset; One-hot encoding is used to handle categorical variables in the historical memory task dataset.

3. The memory allocation method based on deep learning as claimed in claim 2, characterized in that: The relevant feature vectors include data size, calculation operator, data format, task type, data column set, data screening column set, data output column set, compression ratio and storage type.

4. The memory allocation method based on deep learning as claimed in claim 1, characterized in that: Memory-related features extracted from the preprocessed historical memory task dataset include: Extract memory-related feature combinations based on the target task type; The embedding layer is used to process the memory-related feature combination and extract the embedded representation of the memory-related feature combination.

5. The memory allocation method based on deep learning according to any one of claims 1 to 4, characterized in that: Building a memory evaluation model based on deep learning and using memory-related features to train the memory evaluation model include: A memory evaluation model is built based on a multi-layer perceptron. Each layer defines a weight matrix and a bias vector and uses ReLU as a nonlinear activation function. The memory evaluation model is used to transform memory-related features, and the output layer outputs the predicted memory space prediction results.

6. The memory allocation method based on deep learning as claimed in claim 5, characterized in that: Building a memory evaluation model based on deep learning and using memory-related features to train the memory evaluation model also includes: The mean square error is used as the regression loss function to calculate the error between the predicted value and the actual value; The parameters of the memory evaluation model are updated using gradient descent to minimize the regression loss function.

7. The memory allocation method based on deep learning as claimed in claim 6, characterized in that: The expression of the regression loss function is: Where L(θ) represents the regression loss function; represents the predicted memory value of the i-th historical memory task; y i represents the actual memory value of the i-th historical memory task; n represents the number of samples of the historical memory task.

8. A memory evaluation system based on deep learning, characterized in that: Using the deep learning-based memory allocation method according to any one of claims 1 to 7, the system comprises: A data processing module, configured to obtain a historical memory task data set and preprocess the historical memory task data set; a feature extraction module configured to extract memory-related features based on the preprocessed historical memory task dataset; A memory evaluation module, configured to build a memory evaluation model based on deep learning and train the memory evaluation model using memory-related features; The memory allocation module is configured to obtain real-time task data and output the optimal memory allocation result using the trained memory evaluation model.

9. An electronic device, characterized in that: The electronic device includes: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the deep learning-based memory allocation method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing a memory allocation method based on deep learning, and the program for implementing a memory allocation method based on deep learning is executed by a processor to implement the steps of the memory allocation method based on deep learning described in any one of claims 1 to 7.

Images