An elastic resource allocation method based on supervised learning and reinforcement learning

Abstract

The application discloses an elastic resource allocation method based on supervised learning and reinforcement learning, comprising the following steps: step one, based on the test data after crawling and processing, a first training set and a first test set are divided, and a second training set is constructed; step two, a first neural network is established, the first neural network is trained and parameter-adjusted based on a supervised learning technology, and a supervised model after training is optimized through an RMSprop optimization algorithm and the first test set; step three, a second neural network is established, the second neural network is initialized, the second neural network is trained and parameter-adjusted based on a reinforcement learning technology, and a reinforcement learning model after training is optimized through an Adam optimization algorithm; and step four, a real data set is subjected to resource allocation through the second neural network after training and optimization, and an optimal resource allocation result is output. The neural network is initialized by using the supervised learning technology, and then the reinforcement learning technology is combined, so that the accuracy of resource allocation is effectively improved.

Description

TECHNICAL FIELD

[0001] The application relates to an elastic resource allocation method based on supervised learning and reinforcement learning, and belongs to the field of data stream processing. BACKGROUND

[0002] With the massive growth of network data and the increasing real-time requirement for data processing, the technology of data stream processing has been continuously developed. In a distributed stream processing system (hereinafter referred to as DSPS), a data stream processing application (hereinafter referred to as DSPA) is usually represented as a directed acyclic graph (DAG). The points in the DAG represent operators that process data streams, and the edges represent data streams. An operator receives incoming data on one side and processes and converts it into a new data stream on the other side.

[0003] Each operator allocates a certain amount of resources (such as threads) to process data in parallel. The number of resources on an operator is called parallelism. If the parallelism is insufficient, the data processing capacity of the operator is weak, resulting in long processing delay and data loss; if the parallelism is too large, the resource utilization of the DSPS is low. In addition, due to the time-varying and bursty nature of stream data, static resource allocation cannot meet the real-time processing requirements. Therefore, an accurate and dynamic resource allocation method is necessary and crucial.

[0004] Common operator resource allocation methods are mainly based on threshold, queuing theory model and control theory model, etc., and have problems such as inaccurate resource adjustment and high algorithm complexity. In recent years, deep learning technology has been applied to resource scheduling, including deep reinforcement learning models that have shown good performance in resource placement problems. However, the dynamic operator resource allocation problem, i.e., the parallelism adjustment problem, has not been solved, and it cannot accurately and efficiently adapt to the changing processing requirements of stream data.

[0005] Popular supervised learning methods train models through labels, and the acquisition of labels is time-consuming and difficult. At the same time, if there is a large difference between the distribution of the training set and the real data distribution, the performance of the model will be affected; deep reinforcement learning belongs to model-free learning and can better adapt to the dynamic characteristics of data, but the initial parameters of the deep neural network are random, and if a good model performance is achieved, the training time is long and the non-convergence situation is prone to occur, which cannot achieve the training goal. SUMMARY

[0006] The application designs and develops an elastic resource allocation method based on supervised learning and reinforcement learning. The deep neural network is initialized through supervised learning, and combined with reinforcement learning, the parallel resource allocation of the operator is performed, thereby effectively ensuring the accuracy of resource allocation. The method can be deployed in an actual data stream processing platform as a resource scheduling component to improve system resource utilization and save system energy consumption with low resource adjustment cost.

[0007] The technical scheme provided by the application is:

[0008] An elastic resource allocation method based on supervised learning and reinforcement learning, comprising:

[0009] Step one, based on the test data crawled and processed by the web crawler, divide the first training set and the first test set, and construct the second training set;

[0010] Step two, establish the first neural network, based on the supervised learning technology, train and parameterize the first neural network through the first training set, and optimize the trained supervised learning model through the RMSprop optimization algorithm and the first test set;

[0011] Step three, establish the second neural network, initialize the second neural network, based on the reinforcement learning technology, train and parameterize the second neural network through the second training set, and optimize the trained reinforcement learning model through the Adam optimization algorithm;

[0012] Step four, perform resource allocation on the real data set through the second neural network optimized by training, and output the optimal resource allocation result.

[0013] Preferably, the step one comprises:

[0014] The test data is divided into a first part and a second part, the first part of test data is obtained by the optimal allocation algorithm according to the data processing capacity and cache occupancy of the operator, and the optimal parallelism of the operator is obtained as the optimal resource allocation value and the supervised learning label of the operator, and the cache occupancy of the operator is recalculated, and the optimal parallelism and cache occupancy of the operator are used as the algorithm output;

[0015] The first part of test data and the four-tuple set composed of the cache occupancy and the optimal solution returned by the optimal algorithm and the data processing capacity of a single thread CPU are used as a data set, and the data set is divided into a first training set and a first test set according to a ratio of 8:2;

[0016] The second part of test data and the two-tuple set composed of the data processing capacity of a single thread CPU are used as a second training set.

[0017] Preferably, the step two comprises:

[0018] Supervised learning is performed through the first training set, a data batch training method is adopted, the first neural network model is trained and parameterized, the first neural network satisfying the optimal resource allocation is obtained, and the network parameters are saved;

[0019] In the training process, the first neural network optimized each time is verified by the first test set to approach the optimal resource allocation and its target, and is optimized until the expected target is met, and the training is completed.

[0020] Preferably, step three includes:

[0021] A second neural network with the same structure as the first neural network was constructed, the optimal network parameters of the first neural network were overloaded, and the second neural network was initialized.

[0022] Based on the initialized second neural network, and combined with reinforcement learning techniques, the current data in the second training set, the CPU data processing volume of a single thread, and the cache usage after the previous data processing are used as input data for the operator to train and optimize the second neural network.

[0023] Preferably, in step one, the optimal number of threads corresponding to each piece of data is calculated using an optimal allocation algorithm, and the cache usage β after allocating T threads to process the data is obtained. The optimal algorithm model is as follows:

[0024]

[0025]

[0026] In the formula, T represents the optimal number of allocated threads, and C... cpu For the amount of data processed by a single thread, C cache β is the cache capacity of the operator, D is the amount of input data to be processed by the operator, and β- is the cache usage before this data processing.

[0027] Preferably, in step two, training the first neural network using supervised learning techniques includes:

[0028] Input (D, β, C) cpu The triple is fed into the first neural network, and the resource allocation result of the output operator of the second Dense layer of the first neural network is obtained. Will The error between the two is calculated by comparing it with T using the cross-entropy function, and then the network parameters of the first neural network are updated using the error value and the feedforward gradient descent algorithm.

[0029] Preferably, in step three, training the second neural network after supervised learning initialization using reinforcement learning techniques includes:

[0030] In each training step t, based on the operator's state s t Through the second neural network, a parallelism degree is selected from the given set of parallelism degrees. After executing its corresponding resource allocation, the operator's state transitions to the operator state s of the next training step t+1. t+1 Meanwhile, the operator receives a reward r t, guide the second neural network to select the resource allocation value and parallelism T' that is most optimal and the corresponding cache occupancy β' that is most optimal:

[0031]

[0032] In the formula, D' is the data to be processed, β' is the cache occupancy before the current data processing. -

[0033] Preferably, the step four sets a differentiated reward value r for the reinforcement learning t The calculation model of the reward value is:

[0034] When D'+β' - >C cache ,

[0035]

[0036]

[0037] r t =one_reward;

[0038] When D'+β' - ≤C cache ,

[0039]

[0040] In the formula, overflow_penalty is the reward value for lacking one resource, waste_penalty is the reward value for having one more resource, and one_reward is the reward value when the resource allocation is optimal.

[0041] The beneficial effects of the present application are:

[0042] The present application utilizes a deep neural network, combines model-based supervised learning technology and model-free reinforcement learning technology, thereby realizing dynamic self-adaptive allocation of operator resources, improving the accuracy of resource allocation, and reducing resource allocation time and cost.

[0043] The present application combines supervised learning and reinforcement learning, avoids the step of preparing labels compared to single supervised learning, and is more suitable for real-time processing of stream data; compared to single reinforcement learning, the training time is reduced, effectively improving the training efficiency and model accuracy.

[0044] In a stream computing platform, accurate resource allocation is beneficial to improve the performance of resource scheduling, save resource use, and further save system energy consumption. BRIEF DESCRIPTION OF DRAWINGS ​

[0045] Figure 1 The flowchart of the intelligent resource allocation method of the present application.

[0046] Figure 2 The principle diagram of the intelligent resource allocation model provided by the embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement the present application according to the description.

[0048] As shown in the drawings, the present application provides an elastic resource allocation method based on supervised learning and reinforcement learning, which initializes a deep neural network through supervised learning, and combines with reinforcement learning to perform parallel resource allocation of operators, so as to effectively ensure the accuracy of resource allocation, and greatly reduce the training time of deep reinforcement learning, and balance between the accuracy, efficiency and low cost of resource allocation, which comprises: Figures 1-2 Step one, collecting real data, pre-processing the real data, combining the distribution characteristics and range of the real data to appropriately expand the data volume, meeting the data requirements of the model, as test data, and then dividing the test data into two parts;

[0049] Among them, the first part of the test data is used as the input data of the operator, and according to the data processing capacity and cache occupancy of the operator, the optimal allocation algorithm is used to obtain the optimal parallelism of the operator, that is, the thread number T, at the same time, the cache occupancy β of the operator is recalculated, that is, the thread number T and the cache occupancy are used as the algorithm output together, T is the optimal value of the operator resource allocation, and also the label of the supervised learning;

[0050] Each input data of the operator and the cache occupancy and the CPU data processing amount of a single thread returned by the optimal algorithm and the optimal allocation thread array form a four-tuple, and then all the four-tuples are divided into a first training set and a first test set according to the ratio of 8:2;

[0051] Through the optimal allocation algorithm, the optimal allocation thread number corresponding to each data is calculated, and at the same time, the cache occupancy β after processing the data is obtained, and the optimal algorithm model is:

[0052]

[0053]

[0054]

[0055] In the formula, T is the optimal allocation thread number, C cpu is the data processing amount of a single thread, and C cache ​For the cache capacity of the operator, D is the input data amount to be processed by the operator, and β is the cache occupancy amount before this data processing.

[0056] Step two, establish the first neural network, input the first training set to train the network offline and adjust the parameters through supervised learning, and constantly optimize the network with the first test set during the training process, and save the network parameters that meet the expected target;

[0057] Using the first training set through supervised learning, the first neural network model is trained and adjusted by using the data batch training method, and the first neural network that meets the optimal resource allocation is obtained, and the network parameters are saved, including:

[0058] During the training process, for each optimized first neural network, verify its proximity to the optimal resource allocation expected target through the first test set and optimize it until the expected target is met, and the training is completed;

[0059] Input the (D, β, C cpu ) triplet to the first neural network, and the resource allocation result of the operator is output from the second Dense layer of the first neural network Then compare with T, calculate the error value between the two through the cross-entropy function, and update the network parameters of the first neural network through the error value and the feedforward gradient descent algorithm.

[0060] Step three, take the second part of the test data divided in step one as the second training set, and then establish the second neural network, reload the network parameters after supervised training as the initial parameters of the network;

[0061] Take the second part of the test data divided in step one as the second training set, and construct a second neural network with the same structure as the first neural network, and reload the optimal network parameters of the first neural network, i.e. initialize the second neural network.

[0062] Step four, based on the initialized second neural network after supervised learning, as a policy network of reinforcement learning, further combined with reinforcement learning technology, use the second training set to update and optimize the network parameters, and the reinforcement learning training includes:

[0063] In each training step t, according to the operator state S t , i.e. the input data and cache occupancy of the operator, select a parallelism from the given parallelism set through the second neural network, i.e. the policy network, after executing the resource allocation corresponding to the parallelism, the state of the operator transitions to S t+1 , i.e. the operator state of the next training step t+1), at the same time, the operator receives the reward r tThe reward value guides the second neural network to select a parallelism degree that is biased towards the optimal resource allocation value;

[0064] The second neural network initialized based on the parameters is further combined with a reinforcement learning technique, and the current data in the second training set, the CPU data processing amount of a single thread, and the cache occupancy amount after processing the previous data are taken as input data of an operator to train and optimize the second neural network, so that the second neural network close to the optimal resource allocation target is obtained;

[0065] The second neural network initialized based on the parameters is further combined with a reinforcement learning technique, and the current data in the second training set, the CPU data processing amount of a single thread, and the cache occupancy amount after processing the previous data are taken as input data of an operator to train and optimize the second neural network, so that the second neural network close to the optimal resource allocation target is obtained;

[0066] In each training step t, according to the state S t of the operator, a parallelism degree is selected from a given parallelism degree set through the second neural network, and after executing the corresponding resource allocation, the state of the operator is transferred to the operator state S t+1 of the next training step t+1, and the operator receives a reward r t , which guides the second neural network to select a parallelism degree T' that is biased towards the optimal resource allocation value, and the corresponding cache occupancy amount β' is:

[0067]

[0068] In the formula, D' is the data to be processed, and β' - is the cache occupancy amount before processing the data this time;

[0069] A differentiated reward value r t is set for the reinforcement learning, and the calculation model of the reward value is:

[0070] When D' + β' - > C cache ,

[0071]

[0072]

[0073] r t = one_reward: (3)

[0074] When D' + β' - ≤ C cache ,

[0075]

[0076] The formula (1) is a reward value when resource allocation is insufficient, wherein overflow_penalty is a reward value for lacking one resource;

[0077] Formula (2) is the reward value of resource allocation, wherein waste_penalty is the reward value of one more resource;

[0078] Formula (3) is the reward value of optimal resource allocation, and the value is equal to one_reward.

[0079] Step five, test the real data set using the trained second neural network, verify whether it can obtain the approximate optimal operator resource allocation result, that is, the operator parallelism.

[0080] Input the collected real data and the cache occupancy of the operator into the trained second neural network to obtain the parallelism of the operator, compare the allocation result with the optimal solution, and test the effect of the model resource allocation.

[0081] Embodiment

[0082] The neural network is initialized by using the supervised learning technology, and then the network parameters are further trained by combining the reinforcement learning technology, so that the accuracy of resource allocation is effectively improved, including:

[0083] Operating environment: in the present application, as a preferred, the python software is run, the version number is: python3.7.12, the deep learning framework is: theano 1.0.5 and lasagne 0.1, the operating system is: windows 10, the hardware configuration is: processor AMD Ryzen 94900HS Radeon Graphics 3.00GHz, memory 32G RAM, graphics card NVIDIAGeForce RTX 2060 with Max-Q design.

[0084] Initialization setting, according to the resource demand of flow data processing in the application scene and the total amount of system resources, the processing capacity of a single resource is set, specifically, in each data processing time step, the data value range is 1-40000 tuples, and the data processing capacity C of each thread resource is set cpu 5000 tuples, the cache capacity C of each operator is set to 2000 tuples, and the formula cache is used. The upper limit N of the allocatable resource allocation is calculated as 8.

[0085] In order to intuitively express, the intelligent resource allocation model is set to be composed of the first and second neural networks, which are the first neural network trained by using the supervised learning technology and the second neural network trained by using the deep reinforcement learning technology. When training the model, the first training set and the second training set generated based on the real data are used to train the first neural network and the second neural network, respectively.

[0086] Since the second neural network is initialized using the trained first neural network parameters, the two are in a progressive relationship, that is, before training the second neural network, the network parameters saved after the training of the first neural network is completed need to be loaded, and then the training of the second neural network is performed, which specifically includes:

[0087] Step one, establish training and test data sets;

[0088] Collect real data: through the network crawler tool tweepy provided by python, according to the tweet id number, the English tweet text related to COVID19 from twitter is crawled to generate the covid19_twitter real data set, the time range is: 2021-08-01 to 2021-10-01;

[0089] Among them, the tweet id number is obtained from the data text file in the corresponding time range in the GitHub project (https: / / github.com / thepanacealab / covid19_twitter). The downloaded tweet data includes the number of likes, platform, text, comments, forwarding information, number, creation time, in the present application, as a preferred, the original data is the number of tweet forwards.

[0090] The original data is preprocessed. The original data is sorted by time, the number of tweet forwards per minute is counted, and the data distribution is obtained: the data in the range of 0-300 accounts for 99.8%. The data in this part is expanded by 133 times, and the expanded data range is: 0-40000, according to the random distribution characteristics, the data amount is expanded to 10.01 million, which is used as test data.

[0091] Further, the test data is divided into 10,000 and 1,000,000 two parts: the first part of the test data is divided into four according to the optimal resource allocation algorithm, the parallelism of the operator and the cache occupation, and the data processing amount of the CPU single thread and the test data form a four tuple, and then divided into the first training set and the first test set according to the ratio of 8:2, which is used to train and optimize the first neural network; The second part of the test data is directly used as the second training set to train the second neural network.

[0092] Step two, establish two real data sets to test the performance of the intelligent resource allocation model;

[0093] 1) Real data set 1, the data source is the text in Beijing_2016_HourlyPM25 data set, the content is the air quality index obtained every hour in 2016, and the specific data can be downloaded from the website (http: / / www.stateair.net / web / assets / historical / 1 / Beijing_2016_HourlyPM25_created20170201.csv).

[0094] The downloaded data includes city, time, air quality index, and time length. In the present application, as a kind of preference, the original data is air quality index.

[0095] The original data is processed, first, 45 invalid data less than 0 are removed, then, every 10 data is summed, and the sum value is taken as the data of one unit of operator processing. After the operation is completed, then, the data greater than 4000 is removed, and the remaining data is enlarged by 10 times, to obtain the first real data set.

[0096] 2) Real data set 2, the data source is the text in SN m tot V2.0 data set, the content is the sunspot related data from January 1749 to December 2021, and the specific data can be downloaded from the website (https: / / github.com / karenlmasters / ASTR204JupyterAssignments / blob / master / SN_m_tot_V2.0.txt).

[0097] The downloaded data includes Gregorian year and month, year fraction in the middle of the corresponding month, monthly average total sunspot number, and monthly average sunspot number standard deviation. In the present application, as a kind of preference, the original data is monthly average total sunspot number.

[0098] The original data is processed, the data in the range of 0-300 accounting for 99.8% is retained, and enlarged by 133 times, to obtain the second real data set.

[0099] Step three, constructing intelligent resource allocation model;

[0100] Based on DNN, the first neural network and the second neural network are established, the network structure is shown in Figure 2 The first layer is Input layer, the second layer is Dense layer, the third layer is Dense layer, and also as output layer.

[0101] The first neural network: the first layer is an input layer with one neuron; the second layer is a Dense layer with 20 neurons; the third layer is a Dense layer with 9 neurons, which is an output layer, used to output the parallelism result of resource allocation of the operator.

[0102] Input format: [None, 3], where 3 represents 3 kinds of input data: streaming data, cache occupancy after the previous resource allocation execution, and CPU data processing capacity of a single thread, and None represents the number of groups of each input.

[0103] Output format: [None, 9], where 9 represents 9 probability values corresponding to parallelism output by the neural network after processing data, and None represents the number of groups of each output.

[0104] In the second Dense layer of the first neural network, the Softmax function is used to output the probability value of each parallelism, and the parallelism corresponding to the maximum probability value is selected as the resource allocation result of the operator. Then, the error value of the parallelism obtained by the optimal allocation algorithm is calculated according to the cross-entropy function, and the network parameters are adjusted combined with the backpropagation gradient descent method in deep learning.

[0105] The second neural network has the same structure as the first neural network and the same input and output formats.

[0106] Based on the second neural network, a reinforcement learning model is designed, including a state set, an action set, and a reward function.

[0107] State set: the resource allocation state set of the operator is {(D', β' - , C cpu )}, where the triple (D', β' - , C cpu ) represents the state of the system, composed of the data D' to be processed and the current cache occupancy β' - and the CPU data processing capacity of a single thread.

[0108] Action set: the resource allocation action set of the operator is N is the maximum resource allocation number, i.e., the maximum parallelism. When the action value is T', 1≤T'≤N, it represents that the parallelism of resource allocation is T'. Specifically, when the action is , no action is taken, that is, the parallelism of resource allocation remains unchanged.

[0109] Reward function: Due to the existence of matching, excessive and insufficient of resource allocation results and optimal allocation value, a differentiated reward value is set, and the specific reward function refers to formulas (1)-(4). Wherein, one_reward is set to 1, overflow_penalty and waste_penalty are both set to-1.

[0110] In the 2nd Dense layer of the second neural network, the Softmax function is used to output the probability value of all parallel degrees in the current state, and the parallel degree corresponding to the maximum probability value is selected as the resource allocation result of the operator. Then, the function logπ θ (s t , a t )v t is used as the loss function to calculate the loss value, wherein v t is the cumulative discounted reward value, The discount factor γ is 1, and finally the loss value, network parameters and learning rate are combined to adjust the network parameters by using the feedforward gradient descent method.

[0111] Step four, train the first neural network, combine the supervised learning technology with the data of the first training set to train and adjust the parameters of the first neural network, wherein the learning rate is set to 0.0004, the data batch size is set to 32, the training times are 40, the RMSprop algorithm is used to update the network parameters, the decay factor ρ is set to 0.9, and the fuzzy factor ε is set to 10 -9 .

[0112] For the trained first neural network, the first test set is used for testing, and the result meeting the expected performance is obtained, the accuracy rate of resource allocation reaches about 96.4%, that is, close to the optimal allocation parallel degree, and the trained network parameters are saved.

[0113] Step five, train the second neural network. Initialize the second neural network using the parameter reload of the first neural network, form a triple consisting of the data of the second training set and the cache occupancy of the operator as the state of the operator, and further combine the reinforcement learning technology to train and adjust the parameters; the goal of reinforcement learning is to maximize the cumulative reward value, that is, to encourage optimal operator resource allocation; specifically, according to the cumulative reward value of each training, the network parameters of the second neural network are adjusted and updated to improve the accuracy of resource allocation.

[0114] In the training process, the Adam algorithm is used to update the strategy network parameters, the learning rate is set to 0.0004, the training times are 500, and 20000 data are trained at a time.

[0115] Step six, compare the intelligent resource allocation model, the trained and optimized second neural network is used as the trained intelligent resource allocation model, and the resource allocation test is performed on two real data sets respectively to obtain the result set of resource allocation, that is, the parallel degree set of the operator.

[0116] Comparative example

[0117] In the present application, the intelligent resource allocation model is compared with two models, which are a supervised learning model based on the first neural network and a reinforcement learning model based on the second neural network. The parameter settings, training sets, test sets and training times of the above two models are the same as those of the model of the present application.

[0118] Performance measure, in the present application, two indicators are mainly used to measure the performance of the model: accuracy and cumulative reward value; wherein the accuracy indicator indicates the matching degree of the allocation results of each method and the required optimal allocation value, the greater the accuracy, the greater the probability of obtaining the optimal result in training; the cumulative reward value reflects the deviation between the resource allocation result and the required optimal allocation in the whole, the greater the reward value, the closer to the optimal resource allocation.

[0119] The evaluation results of the reinforcement learning model are as follows: the accuracy of the covid19_twitter real data set is 41.9%, and the reward value is -50.86; the accuracy of the Beijing_2016_HourlyPM25 real data set is 71.78%, and the reward value is 22.24; the accuracy of the SN_m_tot_V2.0 real data set is 63.99%, and the reward value is 5.55.

[0120] The evaluation results of the model of the present application are as follows: the accuracy of the covid19_twitter real data set is 91.95%, and the reward value is 83.87; the accuracy of the Beijing_2016_HourlyPM25 real data set is 95.4%, and the reward value is 90.6; the accuracy of the SN_m_tot_V2.0 real data set is 94.11%, and the reward value is 88.82.

[0121] From the above comparison results, it can be seen that compared with the reinforcement learning model, the accuracy and the reward value of the model of the present application are much higher than those of the reinforcement learning model, which shows that the error between the obtained resource allocation result and the optimal value is smaller compared with the reinforcement learning model, and at the same time, it also reflects that the data convergence speed of the model of the present application is faster, thereby verifying that the resource allocation model based on deep neural network combining supervised learning and reinforcement learning can effectively perform high-quality and high-efficiency resource allocation.

[0122] In summary, in the face of dynamic operator resource allocation problems in stream data processing, an intelligent resource allocation method based on supervised learning and reinforcement learning is proposed. First, the supervised learning technique is used to train and adjust the parameters of the deep neural network. Then, the model parameters trained by supervised learning are used to initialize the policy network of the deep reinforcement learning model. The policy network is trained and optimized by combining reinforcement learning technology, and the trained policy network is used as an intelligent resource allocation model. Compared with the method based on reinforcement learning, the method can obtain better resource allocation performance with less training cost for time-varying and burst stream data. In the stream computing platform, elastic intelligent resource scheduling helps to reduce the cost of resource scheduling, improve resource utilization, and reduce system energy consumption while meeting the resource demand of applications.

[0123] Although the embodiments of the present application have been disclosed as above, they are not limited only to the applications listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and therefore, the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A flexible resource allocation method based on supervised learning and reinforcement learning, characterized in that, include: Step 1: Based on the test data crawled and processed by the web crawler, divide the data into a first training set and a first test set, and simultaneously construct a second training set; The first part of the test data serves as the input data for the operator. Based on the operator's data processing capability and cache usage, the optimal allocation algorithm is used to obtain the optimal parallelism of the operator, i.e., the number of threads T. At the same time, the cache usage β of the operator is recalculated. The number of threads T and the cache usage are used together as the output of the algorithm. T is both the optimal value of operator resource allocation and the label for supervised learning. Each input data of the operator, the cache usage returned by the optimal algorithm, the CPU data processing volume of a single thread, and the optimal number of threads are combined into a quadruple. All quadruples are then divided into the first training set and the first test set. Step 2: Establish the first neural network. Based on supervised learning technology, train and tune the first neural network using the first training set. Optimize the trained supervised learning model using the RMSprop optimization algorithm and the first test set. Step 3: Establish a second neural network, initialize the second neural network, train and tune the parameters of the second neural network using the second training set based on reinforcement learning technology, and optimize the trained reinforcement learning model using the Adam optimization algorithm. In each training step t, according to the operator state S t The input data and cache usage of the operator are used by a second neural network, namely the policy network, to select a parallelism from a given set of parallelism degrees. After executing the resource allocation corresponding to that parallelism degree, the operator's state transitions to the operator state s of the next training step t+1. t+1 At the same time, the operator receives a reward r. t The reward value guides the second neural network to select a parallelism that is biased towards the optimal resource allocation value; Based on parameter initialization, the second neural network is further combined with reinforcement learning technology. The current data in the second training set, the CPU data processing amount of a single thread, and the cache usage after the previous data processing are used as the input data of the operator to train and optimize the second neural network, so as to obtain a second neural network that is close to the optimal resource allocation target. Step 4: Use the trained and optimized second neural network to allocate resources to the real dataset and output the optimal resource allocation result.

2. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 1, characterized in that, Step one includes: The test data is divided into two parts. The first part of the test data uses the optimal parallelism of the operator and the cache usage as the algorithm output. The quadruplets were divided into the first training set and the first test set in an 8:2 ratio; The set of two pairs consisting of the second part of the test data and the data processing capability of a single-thread CPU is used as the second training set.

3. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 2, characterized in that, Step two includes: Supervised learning is performed using the first training set, and the first neural network model is trained and its parameters are tuned using a batch training method to obtain the first neural network that satisfies the optimal resource allocation, and the network parameters are saved. During training, for each optimized first neural network, its proximity to the optimal resource allocation and its target is verified and fine-tuned using the first test set until the expected target is met, at which point training ends.

4. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 3, characterized in that, Step three includes: A second neural network with the same structure as the first neural network was constructed, the optimal network parameters of the first neural network were overloaded, and the second neural network was initialized.

5. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 4, characterized in that, In step one, the optimal number of threads for each piece of data is calculated using an optimal allocation algorithm, and the cache usage β after allocating T threads to process the data is obtained. The optimal allocation algorithm model is as follows: In the formula, T represents the optimal number of threads to allocate, and C... cpu For the amount of data processed by a single thread, C cache β is the buffer capacity of the operator, D is the amount of input data to be processed by the operator, and β is the buffer capacity of the operator. - This is the cache usage before this data processing.

6. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 5, characterized in that, In step two, the first neural network is trained using supervised learning techniques, including: Input (D,β,C) cpu The triple is fed into the first neural network, and the resource allocation result of the output operator of the second Dense layer of the first neural network is obtained. Will The error between the two is calculated by comparing it with T using the cross-entropy function, and then the network parameters of the first neural network are updated using the error value and the feedforward gradient descent algorithm.

7. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 6, characterized in that, In step three, the second neural network initialized through supervised learning is trained using reinforcement learning techniques, including: In each training step t, based on the operator's state s t Through the second neural network, a parallelism degree is selected from the given set of parallelism degrees. After executing its corresponding resource allocation, the operator's state transitions to the operator state s of the next training step t+1. t+1 Meanwhile, the operator receives a reward r t The second neural network is guided to select a parallelism T′ that is biased towards the optimal resource allocation value, and the corresponding cache usage β′ is: In the formula, D′ is the data to be processed, and β′ is the data to be processed. - This is the cache usage before this data processing.

8. The elastic resource allocation method based on supervised learning and reinforcement learning according to claim 7, characterized in that, In step four, a differentiated reward value r is set for reinforcement learning. t The reward value is calculated using the following model: When D′+β′ - >C cache hour, This formula represents the reward value when resource allocation is insufficient, where overflow_penalty is the reward value for missing one resource; This formula is the reward value for over-allocation of resources, where waste_penalty is the reward value for an extra resource; r t =one_reward; This formula represents the reward value when resource allocation is optimal, and its value is equal to one_reward; When D′+β′ - ≤C cache , In the formula, overflow_penalty is the reward value for missing one resource, waste_penalty is the reward value for having one extra resource, and one_reward is the reward value when the resource allocation is optimal.

Images