A method and system for scheduling concurrent cloud service requests based on deep reinforcement learning

Abstract

The present invention provides a method and system for scheduling concurrent requests for cloud services based on deep reinforcement learning, and the method comprises the following steps: obtaining a service request of a physical device, determining an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request; inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model; allocating the service request to a corresponding service according to the output sequence, and executing the service request; the deep learning model used in the technical solution of the present application has higher time efficiency than previous iteration-based methods, can obtain a scheduling sequence more quickly, and uses a reinforcement learning method; the solution makes it unnecessary for model training to have a large number of labeled data sets, reduces the time cost required for manual labeling, and the model can be quickly modified according to different problems, thereby improving the scalability of the model, and can be widely used in the field of computer technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for scheduling concurrent cloud service requests for deep reinforcement learning. Background Art

[0002] In recent years, the rise of the Industrial Internet has changed the traditional manufacturing industry. As an industrial cloud platform, the Industrial Internet platform aims to promote the digitalization, networking, and intelligence of the manufacturing industry. Through a new distributed microservice architecture, a service-oriented development environment is built.

[0003] However, the problem of connecting industrial equipment to the network is service delay. Generally speaking, both the supply and demand sides of cloud computing pursue a response in the shortest possible time after submitting a request. However, given the diversity of cloud computing resources and the different time sensitivities of requests, how to comprehensively consider and reduce response time has become a problem worth studying. This problem is essentially similar to a combinatorial optimization problem.

[0004] With the development of artificial intelligence technology, compared with the previous methods of solving combinatorial optimization problems, the use of machine learning algorithms such as deep learning and reinforcement learning has become a breakthrough in solving such problems. At present, some progress has been made in solving combinatorial optimization problems based on deep reinforcement learning. Therefore, the deep reinforcement learning method can be used to design a faster and more scalable model to solve service scheduling problems.

[0005] On the one hand, some related technical solutions use genetic algorithms to solve scheduling problems; however, in order to find a near-optimal solution, especially when the problem dimension increases, a large number of iterations are required to search, which usually leads to a long optimization run time; and when it comes to new problems or even new instances of similar problems, the algorithm needs to be modified to obtain good results. On the other hand, some related technical solutions use supervised learning; however, the performance of the model is related to the quality of the dataset labels; obtaining labels for combinatorial optimization problems is very labor-intensive, and once the problem changes, the dataset needs to be reconstructed, which is also inefficient. Summary of the invention

[0006] In view of this, in order to at least partially solve the above technical problems or one of the defects, the purpose of the embodiment of the present invention is to provide a cloud service concurrent request scheduling method for deep reinforcement learning. On the one hand, the method has higher time efficiency and can obtain the scheduling sequence more quickly; on the other hand, it reduces the time cost required for manual annotation, and the model can be quickly modified according to different problems, so as to improve the scalability of the model as much as possible. At the same time, the technical solution of this application also provides a cloud service concurrent request scheduling system that can implement this method.

[0007] On the one hand, the technical solution of the present application provides a method for scheduling concurrent requests for cloud services based on deep reinforcement learning, comprising the following steps:

[0008] Obtaining a service request of a physical device, and determining an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request;

[0009] Inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model;

[0010] Allocating the service request to a corresponding service according to the output sequence, and executing the service request;

[0011] The process of inputting the input sequence into the scheduling model and obtaining the output sequence through the output of the scheduling model includes:

[0012] Encoding the input sequence to obtain a high-dimensional vector;

[0013] Based on the selected input, obtain the first state in each output time step;

[0014] Obtaining a similarity sequence by calculating the similarity between the high-dimensional vector and the first state;

[0015] The similarity sequence is normalized into a conditional probability distribution on the input sequence, an output element is determined according to the conditional probability distribution, and the output elements are integrated to obtain the output sequence.

[0016] In a feasible embodiment of the solution of the present application, the training process of the scheduling model includes:

[0017] Construct a Critic network model and determine the Actor network model according to the scheduling model;

[0018] Inputting a training sequence of historical requests into an Actor network model, and determining a first weighted response time and a first expectation after the Actor network model outputs a scheduling decision through the Critic network model;

[0019] After the Actor network model outputs a scheduling decision, determining, through the Critic network model, a second weighted response time and a second expectation of an unscheduled request output through the Actor network model;

[0020] Parameters of the Actor network model are adjusted according to the first expectation and the second expectation to obtain the trained scheduling model.

[0021] In a feasible embodiment of the solution of the present application, the training process of the scheduling model further includes:

[0022] Determine the first initial value of the Actor network parameter through the training objective function in the public Actor network and the child thread in the training process;

[0023] The public critic network is trained until the global loss function reaches a minimum value, and the second initial value of the critic network parameter is output;

[0024] Synchronize the first initial value to the Actor network model, and synchronize the second initial value to the Critic network model;

[0025] Determine the first action value through the output results of the Critic network model and the Actor network model after synchronization;

[0026] Updating the first action value, constructing an advantage function through the updated first action value, and determining a first gradient of the Actor network parameter through the advantage function;

[0027] Calculate the second gradient of the critic network parameter according to the current state sequence, the next scheduling request and the global loss function;

[0028] Accumulate gradient updates of the first gradient and the second gradient in the child thread, and adjust the Actor network parameters and the Critic network parameters according to the gradient updates.

[0029] In a feasible embodiment of the solution of the present application, determining the first initial value of the Actor network parameter by using the training objective function in the common Actor network and the sub-thread in the training process includes:

[0030] Construct a first expression according to the training objective function and the sub-thread;

[0031] According to the first expression, a third gradient of the first initial value is calculated by outputting a second action value after the scheduling decision of the public Actor network and a first policy entropy term of the public Actor network;

[0032] According to the third gradient, the first initial value is determined by a policy gradient algorithm and a stochastic gradient descent algorithm.

[0033] In a feasible embodiment of the solution of the present application, determining the first action value through the output results of the synchronized Critic network model and the Actor network model includes:

[0034] Obtaining a third weighted response time and a third expectation corresponding to the predicted scheduling result by outputting the synchronized Critic network model and the Actor network model;

[0035] The first action value is determined according to the third expectation, where the first action value represents the weighted response time of the remaining unscheduled sequences after scheduling.

[0036] In a feasible embodiment of the solution of the present application, updating the first action value, constructing an advantage function through the updated first action value, and determining the first gradient of the Actor network parameter through the advantage function includes:

[0037] Determine an objective reward value by using a scheduling result at a previous moment and a scheduling result at a next moment in the scheduling, and update the first action value according to the objective reward value and the attenuation factor;

[0038] Constructing the advantage function according to the global optimization goal of the Actor network model and the updated first action value;

[0039] The first gradient of the Actor network parameter is determined according to the inverse of the advantage function combined with the single-step strategy entropy term.

[0040] In a feasible embodiment of the solution of the present application, the obtaining of the service request of the physical device, determining the input sequence according to the scheduling sequence corresponding to the service request and the keyword of the service request, includes:

[0041] Extracting the priority, request arrival time and latest response time of the service request to construct a sequence element corresponding to the service request;

[0042] The sequence elements are integrated and sorted to obtain the input sequence.

[0043] In a feasible embodiment of the solution of the present application, the obtaining of the similarity sequence by calculating the similarity between the high-dimensional vector and the first state includes:

[0044] Calculate the first state and the hidden state through learnable parameters to obtain a similarity vector;

[0045] The similarity vectors are integrated to obtain the similarity sequence.

[0046] In a feasible embodiment of the solution of the present application, allocating the service request to a corresponding service according to the output sequence and executing the service request includes:

[0047] When the number of idle services is not less than the number of the service requests, the service requests are allocated to corresponding services according to the order of arrival of the service requests;

[0048] Alternatively, when the number of idle services is less than the number of service requests, the service requests are allocated to corresponding services according to the output sequence.

[0049] On the other hand, the technical solution of the present application also provides a cloud service concurrent request scheduling system for deep reinforcement learning, which includes:

[0050] A request acquisition unit, used to acquire a service request of a physical device, and determine an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request;

[0051] A request scheduling unit, used for inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model;

[0052] a request execution unit, configured to allocate the service request to a corresponding service according to the output sequence, and execute the service request;

[0053] The step of inputting the input sequence into a scheduling model and obtaining an output sequence through output of the scheduling model comprises:

[0054] Encoding the input sequence to obtain a high-dimensional vector;

[0055] Based on the selected input, obtain the first state in each output time step;

[0056] Obtaining a similarity sequence by calculating the similarity between the high-dimensional vector and the first state;

[0057] The similarity sequence is normalized into a conditional probability distribution on the input sequence, an output element is determined according to the conditional probability distribution, and the output elements are integrated to obtain the output sequence.

[0058] The advantages and beneficial effects of the present invention will be partially given in the following description, and the other parts can be understood through the specific embodiments of the present invention:

[0059] The technical solution of this application is used to solve the problem of scheduling requests for cloud services in cloud computing industrial scenarios, so that in large-scale industrial scenarios, work efficiency will not be affected by cloud computing services as much as possible. On the one hand, the deep learning model used in the technical solution of this application has higher time efficiency than previous iteration-based methods and can obtain scheduling sequences more quickly. On the other hand, due to the use of reinforcement learning methods, model training does not require a large number of annotated data sets, reducing the time cost required for manual annotation, and the model can be quickly modified according to different problems to maximize the scalability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 A flowchart of the steps of a method for concurrent request scheduling of cloud services for deep reinforcement learning provided in the technical solution of this application;

[0062] Figure 2 This is a structural diagram of the scheduling module in the technical solution of this application;

[0063] Figure 3 This is a workflow diagram of the service scheduling module combined with the scheduling model in the technical solution of this application. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0065] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] As pointed out in the background technology, service scheduling is a combinatorial optimization problem, the essence of which is to sort an input so that the defined cost is the lowest or the benefit is the highest. In the related technical solutions, for combinatorial optimization problems, multi-objective evolutionary algorithms and manual heuristic algorithms have good effects, such as genetic algorithms for solving scheduling problems, which is a type of heuristic algorithm. This type of algorithm is based on iterative solvers and has obvious limitations. First, in order to find a near-optimal solution, especially when the dimension of the problem increases, a large number of iterations are required to search, which usually results in a long optimization run time. Secondly, when it comes to new problems encountered, or even new instances of similar problems, the algorithm needs to be modified to obtain good results. In addition, specially designed evolutionary strategies and heuristic algorithms can naturally improve the performance of problem solving, but with the development of machine learning algorithms, it has the ability to solve these problems that could only be solved by manually designed algorithms before.

[0068] In some other related technical solutions, machine learning is used; most successful machine learning techniques belong to the category of supervised learning, that is, learning a mapping from input to output from training data. This learning method is widely used. The earliest method of using machine learning to solve combinatorial optimization problems also used supervised learning for training. The supervised learning method is not applicable to combinatorial optimization problems, mainly for three reasons: 1. The performance of the model is related to the quality of the data set labels. 2. Obtaining labels for combinatorial optimization problems is very labor-intensive, and once the problem changes, the data set needs to be reconstructed. 3. If a data set is generated by other algorithms during the implementation process, only an approximation of the algorithm is obtained, and the goal of the technical solution of this application is not to copy an algorithm, but to have better results.

[0069] In view of the shortcomings of the prior art pointed out in the above content, firstly, Figure 1 As shown, the technical solution of the present application provides a method for scheduling concurrent cloud service requests for deep reinforcement learning, the method comprising steps S01-S03:

[0070] S01. Obtain a service request of a physical device, and determine an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request;

[0071] Specifically in the embodiment, a response module is designed to be responsible for accepting requests from physical devices; since the cloud platform is a request / response system, it is necessary to design a response module for transfer between various service modules and physical devices, and the scheduling model runs on this module. Furthermore, in the embodiment, a scheduling module can be used for each type of service. In the embodiment, the "service Definition" field in the JSON file of each request is read. Since different types of services run on different hosts and cannot perform each other's operations, different service types cannot be scheduled together, which requires distinction through the service Definition field.

[0072] S02, inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model;

[0073] Specifically, in the embodiment, the key data in a set of service request sequences is used as an input sequence and passed into the model. From a set of service request JSON files, the request ID field and the service Condition field group are taken out and an input sequence S is formed according to the input requirements in the model description. i , multiple S i Combine to form an input sequence S. This input sequence is passed to the model. Then, the requests are assigned to each service according to the output sequence. In the embodiment, the scheduling sequence C output by the model is obtained. S , a scheduling sequence for service requests is obtained, and these requests are processed in sequence. Since the requests cannot be accepted by the service at one time, the unprocessed requests are first saved in memory. When the model generates a new request scheduling sequence and the old request scheduling sequence has not been processed, the new request scheduling sequence is added after the old request scheduling sequence, following the idea of ​​first come first served.

[0074] S03, allocating the service request to a corresponding service according to the output sequence, and executing the service request;

[0075] Specifically in the embodiment, after a batch of service requests are processed, a new batch of service requests are read in for processing, and the operations of S1 and S2 are performed.

[0076] In some feasible implementation manners, the service request is allocated to the corresponding service according to the output sequence in the embodiment. Before executing the service request, a simple judgment module may be set to judge whether scheduling is required; that is, in the embodiment, steps S04-S05 may also be included:

[0077] S04. When the number of idle services is not less than the number of the service requests, the service requests are allocated to corresponding services according to the order in which the service requests arrive;

[0078] S05. When the number of idle services is less than the number of service requests, the service requests are allocated to corresponding services according to the output sequence.

[0079] Specifically in the embodiment, the number of idle service modules and the number of requests are maintained to determine whether the current service needs to be scheduled. The embodiment sets a simple judgment module to determine whether scheduling is needed. Here we need to combine the idea of ​​first come first served (FIFS). When the idle service can meet the current number of requests, these requests are directly distributed to each service on a first come first served basis. When the number of service requests is much greater than the available services, a group of service requests are taken out from the response module.

[0080] Before scheduling through the scheduling model in the embodiment, it is necessary to build and train the corresponding scheduling model. Figure 2 As shown, in the embodiment, the Seq2Seq model is used as the basis, the Attention mechanism is introduced, and the model is trained in combination with the reinforcement learning method.

[0081] Furthermore, in the embodiment, the step S02 of inputting the input sequence into the scheduling model and obtaining the output sequence through the output of the scheduling model may include steps S021-S024:

[0082] S021. Encode the input sequence to obtain a high-dimensional vector;

[0083] Specifically in the embodiment, ignoring some non-important parameters, the embodiment defines the request format sent by the physical device as approximately:

[0084] [requester id, target service name, request identifier, constraints]

[0085] When using the model for scheduling, the main elements used are request identifiers and constraints (constraints are a set of elements). The request identifier is used to identify a unique request. The simplest way to generate a request identifier is to combine the time and the service name. The constraint is the reference object for scheduling. Here we mainly consider two dimensions: priority and the latest response time. Because for the same type of service, no matter how it is scheduled, the total time required is the same, so we introduce the concept of priority. If there are more parameters to consider, the cost function can be redesigned. It should be pointed out that the processing time of the same type of cloud services is the same, so the processing time is not taken into consideration.

[0086] Furthermore, in the embodiment, step S021 may include sub-steps S0211-S0212:

[0087] S0211. Extract the priority of the service request, the request arrival time and the latest response time to construct a sequence element corresponding to the service request;

[0088] S0212. Integrate and sort the sequence elements to obtain the input sequence.

[0089] Specifically in the embodiment, the following provisions are made for the input and output of the network: the input is a set S = {S1, ..S n}. Output C S ={C1, .., C n} represents the sequence of solutions corresponding to the input S, C S The elements in are completely obtained by permuting the elements in S. We assume that the input set S is the request sequence, where S i =(x, y, z) represents a request, x is the priority of the request, y is the request arrival time, and z is the latest response time. Output C S As a scheduling sequence.

[0090] Within a certain time range, the number of requests that can be processed is limited. The implementation should ensure that high-priority requests are met as much as possible, while minimizing the number of requests that are discarded due to timeout. We define an input S and its scheduling sequence C S ={C1, .., C n}, the indicator of the scheduling sequence is defined as the weighted response time of the entire sequence. i ,y i , z i}∈C t , C t ∈C S , t i The time node for scheduling this request. The cost function is used in the embodiment to calculate the weighted response time of the scheduling result according to the current scheduling result:

[0091]

[0092] When t i More than z i , the response time of this service request is set to the maximum value. It should be noted that if the problem is modified, only this function needs to be changed without modifying the overall model.

[0093] It should be noted that the goal of the embodiment is to learn a random strategy π(C S |S) parameter π:

[0094]

[0095] Given an input S, output a sequence C S . Among them, the requests with high priority and about to time out will be given priority, and then the requests with low priority and a long time left before the timeout will be selected. Therefore, the goal of the task is to give priority to high-priority requests as much as possible, while ensuring that low-priority requests will not time out. The random strategy is approximated by the neural network model constructed in the technical solution of the present application. The neural network model in the embodiment mainly includes Encoder and Decoder.

[0096] S022. Obtain the first state in each output time step according to the selected input;

[0097] Specifically in the embodiment, the encoder in the embodiment is responsible for condensing the input sequence into a vector. Since there is no valuable information about the order of requests in the embodiment, RNN and its variants are not used as encoders in the embodiment, but a simple embedding layer is used to encode the input into a vector. A one-dimensional convolutional layer is used to encode the input sequence into a high-dimensional vector with a dimension of d. The number of input channels is equal to the dimension of the input. The request parameter in the present invention is defined as two dimensions, so the number of input channels is 3. The final encoding result is a 3×d vector.

[0098] S023. Obtain a similarity sequence by calculating the similarity between the high-dimensional vector and the first state;

[0099] Specifically, the modified Attention mechanism is introduced in the embodiment. When the model selects requests for scheduling, it needs to know the previously selected requests C1, ..., C t , so the Decoder uses LSTM network. However, the LSTM network is not directly used for output. What the model needs is the hidden state d of LSTM at the decoding step t. t , and the input code e1,…,e n Used together to calculate the conditional probability p of the next selection request π (C t+1 |C1,…,C t , P). In order to calculate the request for step t+1 selection, the embodiment first obtains the hidden state d of the Decoder t , that is, the first state. Combining e1,…,e n , calculated using the Attention mechanism.

[0100] In the embodiment, step S023 may include steps S0231 and S232:

[0101] S0231. Calculate the first state and the hidden state by using learnable parameters to obtain a similarity vector;

[0102] S0232. Integrate the similarity vectors to obtain the similarity sequence.

[0103] Specifically, in the embodiment, the embodiment calculates its difference with e at each output time step t. j The similarity is as follows, e j and d t Multiply with matrix W and add, then apply tanh to each element of the vector and calculate the inner product with vector v:

[0104]

[0105] Where v, W1 and W2 are learnable parameters.

[0106] S024. Normalize the similarity sequence into a conditional probability distribution on the input sequence, determine an output element according to the conditional probability distribution, and integrate the output elements to obtain the output sequence.

[0107] Specifically in the embodiment, for each state e of the Encoder j , we calculate the corresponding similarity and get the vector Use softmax to normalize it to the distribution on the input sequence. Where p is the conditional probability formula, which means that the input is P and C1, ..., C has been output. t In the case of t+1 The conditional probability distribution for selecting input elements when .

[0108] p π (C t+1 |C1,…,C t , S) = soft max(u t )

[0109] After normalization, select the one with the largest value The input element it points to is the current output element C t+1 .

[0110] In an embodiment, the model is trained by combining a reinforcement learning method, and the training process of the scheduling model in the embodiment may include steps S011-S014:

[0111] S011. Construct a Critic network model and determine an Actor network model according to the scheduling model;

[0112] Specifically in the embodiment, in order to let the model learn how to schedule, the Actor-Critic training method is introduced. First, the embodiment defines an Actor network. The Actor network is the aforementioned neural network model (including Encoder, Decoder and Attention modules). The task is to make decisions, also known as a decision network or scheduling model. Suppose the Actor network is:

[0113] π θ (S,C S )=P(C S |S,θ)

[0114] S represents an input of the model, θ represents the parameters of the network, which generates a scheduling sequence C based on the input S S π θ (S,C S ) is used to approximate the random strategy π(C S |S).

[0115] In addition, a function is introduced in the embodiment as follows:

[0116] p(C t+1 |C1,…,C t , S, θ)

[0117] It is used to represent a strategy for a certain step in the process of generating a sequence, which is based on the input sequence S and the currently scheduled requests C1, ..., C t , to select the next request C t+1 , where the parameter θ is related to P(C S |S, θ) is consistent.

[0118] Then the embodiment defines a Critic network. The structure of the Critic network is the same as the aforementioned Encoder, and a fully connected layer is added to map the input encoding to the output. Its task is to evaluate the current Actor's choice, also known as the state value network. The Critic network does not make decisions, and it does not select new outputs C based on the input. t .

[0119] S012. Input the training sequence of historical requests into the Actor network model, and determine the first weighted response time and the first expectation after the Actor network model outputs the scheduling decision through the Critic network model.

[0120] Specifically in the embodiment, let the critic network be V(S, φ); where S represents the input and φ represents the parameters of the critic network. Critic is used to estimate the expected weighted response time after the actor makes a complete decision scheduling under the input request sequence S; that is, the first weighted response time and the first expectation.

[0121] S013. After the Actor network model outputs a scheduling decision, determine, through the Critic network model, a second weighted response time and a second expectation of an unscheduled request output through the Actor network model.

[0122] Specifically in the embodiment, a function is introduced as follows:

[0123] v((C1,…,C t-1 , S), C t , φ)

[0124] It represents a single step calculation of the Critic network. S represents the input, C1,…,C t-1 represents the scheduled request, and serves as the input of the Critic network. φ represents the parameters of the Critic network. Critic is used to estimate the state (C1, ..., C t-1 , S), after Actor makes a choice C t , the weighted response time and expectation of the remaining unscheduled requests after scheduling through the Actor network decision; that is, the second weighted response time and the second expectation.

[0125] S014. Adjust parameters of the Actor network model according to the first expectation and the second expectation to obtain the trained scheduling model.

[0126] Specifically in the embodiment, corresponding calculation processing can be performed based on the weighted response time and expectation obtained after making a complete decision scheduling, as well as the weighted response time and expectation obtained after the remaining unscheduled requests are scheduled through the Actor network decision, and the model parameters of the global Actor and Critic network are updated according to the calculation processing results.

[0127] In an embodiment, in order to increase the speed of training and enable the network to converge better, an Asynchronous Advantage Actor-Critic algorithm is used, and the training process in the embodiment may further include steps S015-S021:

[0128] S015. Determine a first initial value of an Actor network parameter through a training objective function in a common Actor network and a sub-thread in a training process.

[0129] Specifically in the embodiment, s is recorded as the input of a thread in one training. First, for the common Actor network, its training objective function is combined with each sub-thread to obtain the formula (i.e., the first expression) as follows:

[0130]

[0131] In the embodiment, the parameter θ (the first initial value of the Actor network parameter) is optimized by the policy gradient method and the stochastic gradient descent method, and the gradient (third gradient) about θ is obtained as follows:

[0132]

[0133] Among them, Q(c s |s) represents the action value (second action value) of the Actor network after making a complete schedule. s |s, θ)) is the entropy term of the strategy P of the Actor network (the first strategy entropy term).

[0134] S016. Train the public critic network until the global loss function is minimized, and obtain the second initial value of the critic network parameter.

[0135] Specifically in the embodiment, a Critic network with parameter φ is used to combine the input and the decision of the Actor network to estimate the weighted response time. The loss function of the global Critic network is defined as follows:

[0136]

[0137] This global loss function is obtained by combining Temporal Difierence and Dueling DQN methods. The critic network parameters φ are trained by minimizing this function. The difference between the objective weighted response time and the expected predicted weighted response time is an unbiased estimate of the advantage function.

[0138] S017. Synchronize the first initial value to the Actor network model, and synchronize the second initial value to the Critic network model.

[0139] Specifically in the embodiment, for a training sub-thread, it obtains the parameters of the public network as θ, φ, resets the gradients dθ and dφ of the Actor network and the Critic network, and synchronizes the parameters of the public Actor-Critic network to the current thread. That is:

[0140] θ′=θ,φ′=φ

[0141] S018. Determine the first action value through the output results of the Critic network model and the Actor network model after synchronization.

[0142] Specifically in the embodiment, after the reset and synchronization are completed in step S017, the start time is set to t start , the end time is t end , let the current time step be t, based on the Actor network p(c t+1 |c1,…,c t , s, θ′) to get the next scheduling result c t+1 , and through c1,…,c t ,s,c t+1 Get the weighted response time and R(c1,…,c t+1 , s) (third weighted response time and), and the Critic network predicts from states c1, ..., c t+1 , s, the expected weighted response time V(c1,…,c t+1 , s, φ′). This continues until all requests are scheduled.

[0143] In the embodiment, the process of determining the first action value may include more detailed steps S0181-S0182:

[0144] S0181. Obtaining a third weighted response time and a third expectation corresponding to the predicted scheduling result by outputting the synchronized Critic network model and the Actor network model;

[0145] S0182. Determine the first action value according to the third expectation, where the first action value represents the weighted response time of the remaining unscheduled sequence after scheduling.

[0146] After completing all the scheduling, the embodiment obtains the V value (third expectation) in each case, and assigns an initial value to the action value Q (first action value) at time i, which represents the value of the action in state (c1, ..., c i-1 , s), select c i After scheduling, the weighted response time of the remaining unscheduled sequences after scheduling and the action value Q satisfy the following calculation formula:

[0147]

[0148] S019, updating the first action value, constructing an advantage function through the updated first action value, and determining a first gradient of the Actor network parameter through the advantage function;

[0149] In an embodiment, step S019 may include more detailed steps S0191-S193:

[0150] S0191, determining an objective reward value by using a scheduling result at a previous moment and a scheduling result at a next moment in the scheduling, and updating the first action value according to the objective reward value and the attenuation factor;

[0151] S0192. Construct the advantage function according to the global optimization goal of the Actor network model and the updated first action value.

[0152] More specifically, in the embodiment, from the end time t end -1 to t start Traverse backwards, assuming the current traversal is i. Update the action value Q through TD-error:

[0153] Q((c1,…,c i-1 , s), c i )=r((c1,…,c i-1 , s), c i )+γQ((c1,…,c i , s), c i+1 )

[0154] Among them, r represents the objective reward value, which is calculated by subtracting the weighted response time of the scheduling results at time i and time i-1 from R. γ is the attenuation factor, which takes a value between 0 and 1.

[0155] S0193. Determine the first gradient of the Actor network parameter according to the inverse of the advantage function combined with the single-step strategy entropy term.

[0156] More specifically, for the Actor network of the child thread, based on the global optimization goal, the advantage function A is introduced as follows, which represents the advantage of the Actor's scheduling selection compared to the average scheduling selection. However, since the higher the response time, the worse the scheduling effect, the embodiment needs to take the opposite of the advantage function to make the Actor network converge in the right direction. That is, we get:

[0157] A((c1,…,c i-1 , s), c i , φ′)=-Q((c1,…,c i-1 , s), c i )+v((c1,…,c i-1 , s), c i , φ′)

[0158] Combined with the advantage function, the gradient of the parameter θ′ of the Actor network is obtained as follows:

[0159]

[0160] Since entropy can measure the uncertainty of probability distribution, we hope that the action space of the model is as large as possible, so that the model can have better diversity and will not focus on the same action during training. Therefore, an entropy term is added to the single-step strategy p;

[0161] H(p(c i |c1,…,c i-1 , s, θ′))

[0162] After adding the entropy term, the gradient of the Actor parameter θ′ (Actor network parameter) on this thread is as follows:

[0163]

[0164] Where c is the coefficient.

[0165] S020, calculating the second gradient of the Critic network parameters according to the current state sequence, the next scheduling request and the global loss function;

[0166] Specifically in the embodiment, for the Critic network of the child thread, the current state c1, ..., c t , s and the next scheduled request c decided t+1 , and the global loss function, the gradient of the parameter φ′ (Critic network parameter) of the Critic network at time i is obtained as follows:

[0167]

[0168] After obtaining the gradient, the gradient updates of the Actor and Critic of this thread are accumulated:

[0169] dθ=dθ+dθ′

[0170] dφ=dφ+dφ

[0171] S021. Accumulate gradient updates of the first gradient and the second gradient in the sub-thread, and adjust the Actor network parameters and the Critic network parameters according to the gradient updates.

[0172] Specifically in the embodiment, the model parameters of the global Actor and Critic network are updated through the gradient of each sub-thread:

[0173] θ=θ+αdθ

[0174] φ=φ-βdφ

[0175] Where α and β are learning rates. Through the above steps S017-S021, training is performed, and each time the parameters of the public network are updated through the child thread, it is considered a training. Through multiple trainings, the Critic network makes more accurate value judgments on the current selection. The Actor network can make the best scheduling based on the evaluation of the Critic. After training, the Critic network is no longer used. The trained Actor network is used to solve practical problems.

[0176] In conjunction with the instruction manual Figure 3 , a more complete and detailed description of the complete implementation process of the technical solution of this application is given as follows:

[0177] The implementation example builds a cloud platform and a service scheduling module, which is responsible for scheduling service requests sent by physical layer devices to minimize time consumption. For the trained scheduling model, it is necessary to determine whether to use the service scheduling module based on the actual situation. When the number of service application requests is less than the idle service modules, we do not need to use the scheduling module, but directly take out the requests in the message queue and hand them over to the corresponding service module. However, when the number of requests is large and the number of available services is small in order to expand the production line and save computing resources, a scheduling model is needed.

[0178] The following is part of a json file of a service request, where clientID represents the client identifier, which is used to return information later. serviceDefinition represents the name of the service to be requested. serviceUri represents the Uri of the requested service. requestID represents the identifier of this request, serviceCondition represents the condition of the request, where endOfValidity represents the latest response time, and Priority represents the priority.

[0179]

[0180] When introducing the previously mentioned scheduling model, a series of operations need to be performed, as follows:

[0181] T1: Design the response module to accept requests from physical devices;

[0182] T2: Use one scheduling module for each type of service.

[0183] T3: Maintain the number of idle service modules and the number of requests to determine whether the current service needs to be scheduled;

[0184] T4: Take the key data in a set of service request sequences as input sequences and pass them into the model;

[0185] T5: Allocate requests to each service according to the output sequence;

[0186] T6: Process new service requests.

[0187] Among them, in T1, since the cloud platform is a request / response system, it is necessary to design a response module for transfer between various service modules and physical devices, and the scheduling model runs on this module.

[0188] In T2, the "serviceDefinition" field in the JSON file of each request is read. Since different types of services run on different hosts and cannot perform each other's operations, different service types cannot be scheduled together, so they must be distinguished through the serviceDefinition field.

[0189] In T3, the embodiment needs to set up a simple judgment module to determine whether scheduling is needed. Here we need to combine the idea of ​​first-come, first-served (FIFS). When the idle services can meet the current number of requests, these requests are directly distributed to each service according to first-come, first-served. When the number of service requests is much greater than the available services, a group of service requests are taken out from the response module.

[0190] In T4, the requestID field and serviceCondition field group are taken from a set of service request JSON files, and an input S is formed according to the input requirements in the model description. i , multiple S i Combine to form the input sequence S. Pass this input sequence to the model.

[0191] In T5, the scheduling sequence C output by the model is obtained S , a scheduling sequence for service requests is obtained, and these requests are processed in sequence. Since the requests cannot be accepted by the service at one time, the unprocessed requests are first saved in memory. When the model generates a new request scheduling sequence and the old request scheduling sequence has not been processed, the new request scheduling sequence is added after the old request scheduling sequence, following the idea of ​​first come first served.

[0192] In T6, after processing a batch of service requests, a new batch of service requests is read in for processing, and the operations from T3 to T5 are followed.

[0193] On the other hand, the technical solution of the present application also provides a cloud service concurrent request scheduling system for deep reinforcement learning, which includes:

[0194] A request acquisition unit, used to acquire a service request of a physical device, and determine an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request;

[0195] A request scheduling unit, used for inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model;

[0196] a request execution unit, configured to allocate the service request to a corresponding service according to the output sequence, and execute the service request;

[0197] The step of inputting the input sequence into a scheduling model and obtaining an output sequence through output of the scheduling model comprises:

[0198] Encoding the input sequence to obtain a high-dimensional vector;

[0199] Based on the selected input, obtain the first state in each output time step;

[0200] Obtaining a similarity sequence by calculating the similarity between the high-dimensional vector and the first state;

[0201] The similarity sequence is normalized into a conditional probability distribution on the input sequence, an output element is determined according to the conditional probability distribution, and the output elements are integrated to obtain the output sequence.

[0202] From the above specific implementation process, it can be concluded that the technical solution provided by the present invention has the following advantages or strengths compared with the prior art:

[0203] The technical solution of this application is different from the supervised learning method. The scheduling problem cannot be solved by constructing the optimal result from input to output. Even if it can be constructed, it is very time-consuming to manually design labels. If other algorithms are used to generate data sets, the model cannot learn a better scheduling result than the data set. However, the reinforcement learning method does not require setting labels. Through the model's own decisions and rewards from actual situations, it learns how to schedule services.

[0204] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0205] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0208] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for scheduling concurrent cloud service requests based on deep reinforcement learning, characterized in that: The following steps are involved: Obtaining a service request of a physical device, and determining an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request; Inputting the input sequence into a scheduling model, and obtaining an output sequence through the output of the scheduling model; Allocating the service request to a corresponding service according to the output sequence, and executing the service request; The process of inputting the input sequence into the scheduling model and obtaining the output sequence through the output of the scheduling model includes: Encoding the input sequence to obtain a high-dimensional vector; Based on the selected input, obtain the first state in each output time step; Obtaining a similarity sequence by calculating the similarity between the high-dimensional vector and the first state; Normalizing the similarity sequence into a conditional probability distribution on the input sequence, determining an output element according to the conditional probability distribution, and integrating the output elements to obtain the output sequence; The training process of the scheduling model includes: Construct a Critic network model, and construct an Actor network model based on the scheduling model; Inputting a training sequence of historical requests into an Actor network model, and determining a first weighted response time and a first expectation after the Actor network model outputs a scheduling decision through the Critic network model; After the Actor network model outputs a scheduling decision, determining, through the Critic network model, a second weighted response time and a second expectation of an unscheduled request output through the Actor network model; Adjusting parameters of the Actor network model according to the first expectation and the second expectation to obtain the trained scheduling model; The training process of the scheduling model further includes: Determine the first initial value of the Actor network parameter through the training objective function in the public Actor network and the child thread in the training process; The public critic network is trained until the global loss function reaches a minimum value, and the second initial value of the critic network parameter is output; Synchronize the first initial value to the Actor network model, and synchronize the second initial value to the Critic network model; Determine the first action value through the output results of the Critic network model and the Actor network model after synchronization; Updating the first action value, constructing an advantage function through the updated first action value, and determining a first gradient of the Actor network parameter through the advantage function; Calculate the second gradient of the critic network parameter according to the current state sequence, the next scheduling request and the global loss function; Accumulate gradient updates of the first gradient and the second gradient in the child thread, and adjust the Actor network parameters and the Critic network parameters according to the gradient updates.

2. According to claim 1, a method for scheduling concurrent requests for cloud services for deep reinforcement learning is characterized in that: The step of determining the first initial value of the Actor network parameter by using the training objective function in the common Actor network and the sub-threads in the training process includes: Construct a first expression according to the training objective function and the sub-thread; According to the first expression, a third gradient of the first initial value is calculated by outputting a second action value after the scheduling decision of the public Actor network and a first policy entropy term of the public Actor network; According to the third gradient, the first initial value is determined by a policy gradient algorithm and a stochastic gradient descent algorithm.

3. The method for scheduling concurrent requests for cloud services based on deep reinforcement learning according to claim 1, characterized in that: The determining of the first action value by outputting the synchronized Critic network model and the Actor network model comprises: Obtaining a third weighted response time and a third expectation corresponding to the predicted scheduling result by outputting the synchronized Critic network model and the Actor network model; The first action value is determined according to the third expectation, where the first action value represents the weighted response time of the remaining unscheduled sequences after scheduling.

4. The method for scheduling concurrent requests for cloud services based on deep reinforcement learning according to claim 1, characterized in that: The updating of the first action value, constructing an advantage function through the updated first action value, and determining a first gradient of the Actor network parameter through the advantage function includes: Determine an objective reward value by using a scheduling result at a previous moment and a scheduling result at a next moment in the scheduling, and update the first action value according to the objective reward value and the attenuation factor; Constructing the advantage function according to the global optimization goal of the Actor network model and the updated first action value; The first gradient of the Actor network parameter is determined according to the inverse of the advantage function combined with the single-step strategy entropy term.

5. The method for scheduling concurrent requests for cloud services based on deep reinforcement learning according to claim 1, characterized in that: The obtaining of the service request of the physical device, and determining the input sequence according to the scheduling sequence corresponding to the service request and the keyword of the service request, includes: Extracting the priority, request arrival time and latest response time of the service request to construct a sequence element corresponding to the service request; The sequence elements are integrated and sorted to obtain the input sequence.

6. The method for scheduling concurrent requests for cloud services based on deep reinforcement learning according to claim 1, characterized in that: The obtaining of a similarity sequence by calculating the similarity between the high-dimensional vector and the first state includes: The first state and the hidden state are calculated by using learnable parameters to obtain a similarity vector; The similarity vectors are integrated to obtain the similarity sequence.

7. A method for scheduling concurrent requests for cloud services for deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The allocating the service request to a corresponding service according to the output sequence and executing the service request comprises: When the number of idle services is not less than the number of the service requests, the service requests are allocated to corresponding services according to the order of arrival of the service requests; Alternatively, when the number of idle services is less than the number of service requests, the service requests are allocated to corresponding services according to the output sequence.

8. A cloud service concurrent request scheduling system based on deep reinforcement learning, characterized in that: include: A request acquisition unit, used to acquire a service request of a physical device, and determine an input sequence according to a scheduling sequence corresponding to the service request and a keyword of the service request; A request scheduling unit is used to input the input sequence into the scheduling model, and obtain an output sequence through the output of the scheduling model, wherein the training process of the scheduling model includes: constructing a Critic network model, and constructing an Actor network model according to the scheduling model; inputting a training sequence of historical requests into the Actor network model, and determining a first weighted response time and a first expectation after the Actor network model outputs a scheduling decision through the Critic network model; after the Actor network model outputs a scheduling decision, determining a second weighted response time and a second expectation after the unscheduled request is output through the Actor network model through the Critic network model; adjusting parameters of the Actor network model according to the first expectation and the second expectation to obtain the trained scheduling model; and determining the Actor network model through a training objective function in a common Actor network and a sub-thread in a training process. Actor network parameters are first initialized by training the public Critic network until the global loss function is the minimum, and the second initial value of the Critic network parameters is output; the first initial value is synchronized to the Actor network model, and the second initial value is synchronized to the Critic network model; the first action value is determined by the output results of the Critic network model and the Actor network model after synchronization; the first action value is updated, and the advantage function is constructed by the updated first action value, and the first gradient of the Actor network parameters is determined by the advantage function; the second gradient of the Critic network parameters is calculated according to the current state sequence, the next scheduling request and the global loss function; the gradient updates of the first gradient and the second gradient in the sub-thread are accumulated, and the Actor network parameters and the Critic network parameters are adjusted according to the gradient updates; a request execution unit, configured to allocate the service request to a corresponding service according to the output sequence, and execute the service request; The step of inputting the input sequence into a scheduling model and obtaining an output sequence through output of the scheduling model comprises: Encoding the input sequence to obtain a high-dimensional vector; Based on the selected input, obtain the first state in each output time step; Obtaining a similarity sequence by calculating the similarity between the high-dimensional vector and the first state; The similarity sequence is normalized into a conditional probability distribution on the input sequence, an output element is determined according to the conditional probability distribution, and the output elements are integrated to obtain the output sequence.

Images