Cloud computing task scheduling optimization method based on deep reinforcement learning

Abstract

The invention discloses a cloud computing task scheduling optimization method based on deep reinforcement learning, belongs to the technical field of cloud computing task scheduling, and establishes a cloud computing task scheduling architecture with minimized guidance of cloud computing task response time and cost consumption demand on the basis of a deep reinforcement learning algorithm. Scheduling optimization of cloud computing tasks is effectively realized, task scheduling efficiency in a cloud computing environment is improved, resource consumption is reduced, and service quality is improved; according to the method, dynamic optimization of a task scheduling strategy is realized by constructing a deep reinforcement learning model, so that dynamics and uncertainty in a cloud computing environment are effectively coped with.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing task scheduling, and specifically relates to a cloud computing task scheduling optimization method based on deep reinforcement learning. Background Art

[0002] Cloud computing is a distributed computing model that provides computing services to users through the Internet. Specifically, in cloud computing, hardware and software resources distributed in different geographical locations are always organized in the form of a shared resource pool. Only a small amount of management work and the participation of service providers can be quickly configured and released, which saves users a lot of costs in purchasing, operating and maintaining traditional physical data centers and servers, and has extremely high flexibility. Cloud platforms usually have large-scale computing resources, allowing users to choose on demand without incurring additional operation and maintenance costs. This flexible service model has greatly promoted users' dependence on cloud platforms, thereby prompting a rapid increase in the amount of computing tasks. With the rapid development of cloud computing technology, the task scheduling problem faced by cloud computing platforms has become increasingly prominent. In this context, an effective task scheduling strategy has become particularly critical because it is directly related to the optimal management of cloud resources and the improvement of service quality. In the past few years, although many task scheduling algorithms have been proposed, most of the existing task scheduling methods are based on static rules or heuristic algorithms, which are difficult to adapt to the dynamics and uncertainty in cloud computing environments. Therefore, how to design an efficient and adaptive cloud computing task scheduling method has become an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a cloud computing task scheduling optimization method based on deep reinforcement learning, which is used to solve the technical problems existing in the existing static cloud task scheduling algorithm.

[0004] A cloud computing task scheduling optimization method based on deep reinforcement learning, comprising: The state space of the deep reinforcement learning algorithm is constructed with the operating parameters of the cloud computing server, and the action space of the deep reinforcement learning algorithm is constructed with the selection actions of the cloud computing task on the cloud computing server; Taking the state space and action space of the deep reinforcement learning algorithm as a basis, obtaining a target action for a cloud computing task; Execute the target action of the cloud computing task and update the state space of the deep reinforcement learning algorithm to obtain the updated state space and the target reward at the same time; According to the original state space, target action, target reward and updated state space, a prior experience pool is constructed, and the target Q value corresponding to the target network is obtained according to the prior experience pool; According to the target action, target reward and updated state space, obtain the predicted Q value corresponding to the main network, and obtain the loss function value according to the predicted Q value and the target Q value; According to the loss function value, the network parameters of the main network are updated to obtain an updated main network, and when the next cloud computing task arrives, the updated main network is used for task scheduling; Wherein, after each network synchronization cycle, the network parameters of the main network are synchronized to the target network.

[0005] Furthermore, the state space of the deep reinforcement learning algorithm is constructed with the operating parameters of the cloud computing server, including: The queuing time, remaining computing resources and task size of queued cloud computing tasks corresponding to each cloud computing server are used as the state space of the deep reinforcement learning algorithm.

[0006] Furthermore, the action space of the deep reinforcement learning algorithm is constructed based on the selection action of the cloud computing task on the cloud computing server, including: For any cloud computing task to be scheduled, the action of each cloud computing server selected by the cloud computing task to be scheduled is used to construct the action space of the deep reinforcement learning algorithm.

[0007] Furthermore, the state space and action space of the deep reinforcement learning algorithm are used as a basis to obtain the target action for the cloud computing task, including: Based on the state space and action space of the deep reinforcement learning algorithm, determine the Q value corresponding to each action; wherein the Q value is obtained by querying the Q value table, and the Q value between the state and the action at the initial moment is initialized to a random number between (0, 0.1); The Softmax function is used to normalize the action function value corresponding to each action to obtain the probability value corresponding to each action; Based on the probability value corresponding to each action, a target action is selected to obtain the target action for the cloud computing task.

[0008] Furthermore, the target rewards are obtained at the same time: ;in, Represents the state space in time section t Select target action The rewards you can get, Represents the cost consumption after executing the target action, Indicates the time consumed after executing the target action.

[0009] Furthermore, based on the original state space, target action, target reward, and updated state space, a prior experience pool is constructed, including: Construct a priori experience sample based on the original state space, target action, target reward and updated state space, and add the prior experience sample to the prior experience pool each time a cloud computing task is executed; For any prior experience sample, the priority of the prior experience sample is determined according to the target reward: ;in, Represents a positive number less than 0.0001; When the number of prior experience samples in the prior experience pool is greater than or equal to a preset number threshold, the prior experience sample with the smallest priority is removed to obtain a prior experience pool of a fixed size.

[0010] Furthermore, according to the prior experience pool, the target Q value corresponding to the target network is obtained, including: According to the prior experience samples in the prior experience pool, the updated state space is used as the input of the target network to obtain the output of the target network, and the maximum Q value is determined as the target Q value based on the output of the target network.

[0011] Furthermore, according to the target action, target reward and updated state space, the predicted Q value corresponding to the main network is obtained as:

[0012] in, Indicates the main network selects the target action The corresponding predicted Q value is then, Indicates the Q value corresponding to the target action before the main network selects it. represents the learning rate, represents the discount factor, Represents the state space after the update Predicting selected actions Maximize the Q value; after obtaining the predicted Q value corresponding to the main network, use the predicted Q value to update the Q value table.

[0013] Furthermore, according to the predicted Q value and the target Q value, the loss function value is obtained as:

[0014] Among them, L represents the loss function value, E represents the expected value, Represents the target Q value.

[0015] Further, according to the loss function value, the network parameters of the main network are updated to obtain the updated main network, and when the next cloud computing task arrives, the task is scheduled with the updated main network, including: Based on the loss function value, the network parameters of the main network are updated using the gradient descent algorithm, and the update is repeated until the main network converges, and the action Q value is generated using the main network; According to the action Q value generated by the main network, the scheduling of cloud computing tasks is optimized.

[0016] The present invention provides a cloud computing task scheduling optimization method based on deep reinforcement learning. Based on the deep reinforcement learning algorithm, a cloud computing task scheduling architecture guided by minimizing the response time and cost consumption requirements of cloud computing tasks is established, thereby effectively realizing scheduling optimization of cloud computing tasks, improving the efficiency of task scheduling in the cloud computing environment, reducing resource consumption and improving service quality. The method realizes dynamic optimization of task scheduling strategies by constructing a deep reinforcement learning model, thereby effectively coping with the dynamics and uncertainty in the cloud computing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] Figure 1 A flowchart of a cloud computing task scheduling optimization method based on deep reinforcement learning is provided in an embodiment of the present invention.

[0019] The above drawings have shown clear embodiments of the present invention, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0020] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0021] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a cloud computing task scheduling optimization method based on deep reinforcement learning, including: S101, constructing a state space of a deep reinforcement learning algorithm using operating parameters of a cloud computing server, and constructing an action space of the deep reinforcement learning algorithm using a selection action of a cloud computing task on the cloud computing server; Optionally, the deep reinforcement learning algorithm described in the embodiment of the present invention can be preferably a deep Q network algorithm. The deep Q network (DQN) algorithm combines deep learning and Q-learning, and can process high-dimensional inputs, such as video game screens; the experience replay mechanism breaks data correlation and improves learning stability; the target network reduces the Q value estimation deviation and avoids training instability. These characteristics enable DQN to perform well in a variety of complex environments, learn efficient strategies, and become an important breakthrough in the field of reinforcement learning.

[0023] Therefore, the embodiment of the present invention is based on the deep Q network algorithm and is improved in combination with the characteristics of cloud computing tasks, so that the automatic allocation of cloud computing tasks can be realized, and as time goes by, the selection effect will be continuously strengthened, and the purpose of automatic optimization will be finally achieved.

[0024] S102, using the state space and action space of the deep reinforcement learning algorithm as a basis to obtain a target action for a cloud computing task; The Q value table may be queried through the state space first, so that an action may be selected for execution, namely, the target action described in this busy embodiment.

[0025] S103, executing the target action of the cloud computing task, and updating the state space of the deep reinforcement learning algorithm to obtain the updated state space, and simultaneously obtaining the target reward; The state space is essentially composed of some parameters that reflect the running state of the cloud computing service. Therefore, after allocating a cloud computing task to the cloud computing server, the state space can be updated.

[0026] S104, constructing a priori experience pool according to the original state space, target action, target reward and updated state space, and obtaining a target Q value corresponding to the target network according to the priori experience pool; The prior experience pool (Experience Replay) is a key component that has several important functions: In traditional online learning, successive experiences are often highly correlated, which can cause the neural network to fall into a local minimum or overfit during training. The prior experience pool breaks the correlation between data points by storing and randomly extracting past experiences, so that unrelated data samples are used each time the network is updated, thereby improving the generalization ability of learning. Without the prior experience pool, each experience can only be used once and then discarded. The prior experience pool allows the reuse of experience, which means that each experience can be sampled and learned in multiple training iterations, thereby improving the efficiency of data utilization. Since the data in the prior experience pool is randomly extracted, this helps reduce the variance in the learning process and makes learning more stable. Random sampling can reduce fluctuations when the network is updated and avoid training instability caused by continuous encounters with similar experiences. The prior experience pool allows DQN to perform off-policy learning, that is, the experience collected in the past under different policies can be used to update the current policy. This is an important feature for reinforcement learning because it allows the algorithm to learn from a range of policies, not just the current policy. In some environments, reward signals may have different time scales. For example, rewards for some actions may be obtained immediately, while rewards for other actions may be observed much later. The prior experience pool can help the algorithm deal with this time scale difference because it can store and review long-term and short-term experience. Therefore, the embodiment of the present invention can effectively improve the performance of the algorithm by constructing a dynamic prior experience pool.

[0027] S105. According to the target action, target reward and updated state space, obtain the predicted Q value corresponding to the main network, and obtain the loss function value according to the predicted Q value and the target Q value; The loss function value reflects the difference between the predicted Q value corresponding to the main network and the target Q value corresponding to the target network, which can effectively reflect the current parameter update effect. Therefore, a commonly used loss function can be used to obtain the loss function value, such as the mean square loss function.

[0028] S106, updating the network parameters of the main network according to the loss function value to obtain an updated main network, and when the next cloud computing task arrives, using the updated main network for task scheduling; Wherein, after each network synchronization cycle, the network parameters of the main network are synchronized to the target network.

[0029] Gradient descent is an optimization algorithm used to find the minimum value of a function. In machine learning and deep learning, gradient descent is widely used to train models, especially to minimize loss functions, thereby improving the prediction accuracy of the model. Therefore, the gradient descent method can be used to update the network parameters of the main network. After the main network is updated, fast action selection can be performed through the main network, ultimately achieving optimal scheduling of cloud computing tasks.

[0030] The embodiment of the present invention provides a cloud computing task scheduling optimization method based on deep reinforcement learning, which optimizes task scheduling in a cloud computing environment, thereby improving resource utilization, reducing task execution time, and improving overall cloud computing performance.

[0031] In an embodiment of the present invention, the state space of a deep reinforcement learning algorithm is constructed using the operating parameters of a cloud computing server, including: The queuing time, remaining computing resources and task size of queued cloud computing tasks corresponding to each cloud computing server are used as the state space of the deep reinforcement learning algorithm.

[0032] In an embodiment of the present invention, the action space of a deep reinforcement learning algorithm is constructed by selecting actions of cloud computing tasks for cloud computing servers, including: For any cloud computing task to be scheduled, the action of each cloud computing server selected by the cloud computing task to be scheduled is used to construct the action space of the deep reinforcement learning algorithm.

[0033] In an embodiment of the present invention, the state space and action space of the deep reinforcement learning algorithm are used as a basis to obtain a target action for a cloud computing task, including: Based on the state space and action space of the deep reinforcement learning algorithm, determine the Q value corresponding to each action; wherein the Q value is obtained by querying the Q value table, and the Q value between the state and the action at the initial moment is initialized to a random number between (0, 0.1); The Softmax function is used to normalize the action function value corresponding to each action to obtain the probability value corresponding to each action; Based on the probability value corresponding to each action, a target action is selected to obtain the target action for the cloud computing task.

[0034] In the embodiment of the present invention, the target rewards are obtained at the same time: ;in, Represents the state space in time section t Select target action The rewards you can get, Represents the cost consumption after executing the target action, Indicates the time consumed after executing the target action.

[0035] In an embodiment of the present invention, a prior experience pool is constructed based on the original state space, target action, target reward, and updated state space, including: Construct a priori experience sample based on the original state space, target action, target reward and updated state space, and add the prior experience sample to the prior experience pool each time a cloud computing task is executed; For any prior experience sample, the priority of the prior experience sample is determined according to the target reward: ;in, Represents a positive number less than 0.0001; When the number of prior experience samples in the prior experience pool is greater than or equal to a preset number threshold, the prior experience sample with the smallest priority is removed to obtain a prior experience pool of a fixed size.

[0036] In an embodiment of the present invention, obtaining a target Q value corresponding to a target network according to a priori experience pool includes: According to the prior experience samples in the prior experience pool, the updated state space is used as the input of the target network to obtain the output of the target network, and the maximum Q value is determined as the target Q value based on the output of the target network.

[0037] In this embodiment of the present invention, according to the target action, the target reward and the updated state space, the predicted Q value corresponding to the main network is obtained as follows:

[0038] in, Indicates the main network selects the target action The corresponding predicted Q value after that; after the prior experience pool is full, represents the Q value output by the main network; before the prior experience pool is full, represents the Q value obtained by looking up the table; represents the learning rate, represents the discount factor, Represents the state space after the update Predicting selected actions Maximize the Q value; after obtaining the predicted Q value corresponding to the main network, use the predicted Q value to update the Q value table.

[0039] The predicted Q value in the embodiment of the present invention is essentially the action Q value generated by the main network in the later stage of the algorithm. After the action Q value is updated by the reward function, the network parameters of the main network are updated, which can achieve better optimization effect and stronger robustness.

[0040] In the embodiment of the present invention, the loss function value is obtained according to the predicted Q value and the target Q value:

[0041] Among them, L represents the loss function value, E represents the expected value, Represents the target Q value.

[0042] It is worth noting that the above loss function value is only a preferred selection method in the embodiment of the present invention. Other loss functions can also be used to obtain the loss function to update the parameters of the main network and the target network according to the number of prior experience samples in the prior experience pool.

[0043] In an embodiment of the present invention, the network parameters of the primary network are updated according to the loss function value to obtain an updated primary network, and when the next cloud computing task arrives, the task is scheduled with the updated primary network, including: Based on the loss function value, the network parameters of the main network are updated using the gradient descent algorithm, and the update is repeated until the main network converges, and the action Q value is generated using the main network; According to the action Q value generated by the main network, the scheduling of cloud computing tasks is optimized.

[0044] After training the main network, the main network can quickly respond to the state space, thereby realizing action selection, that is, realizing the scheduling of cloud computing tasks. At the same time, with the update of Q value and prior experience pool, the performance of the main network can be continuously enhanced, and finally the deep strengthening effect can be effectively realized, so that the scheduling of cloud computing tasks can be optimized.

[0045] The present invention provides a cloud computing task scheduling optimization method based on deep reinforcement learning. Based on the deep reinforcement learning algorithm, a cloud computing task scheduling architecture guided by minimizing the response time and cost consumption requirements of cloud computing tasks is established, thereby effectively realizing scheduling optimization of cloud computing tasks, improving the efficiency of task scheduling in the cloud computing environment, reducing resource consumption and improving service quality. The method realizes dynamic optimization of task scheduling strategies by constructing a deep reinforcement learning model, thereby effectively coping with the dynamics and uncertainty in the cloud computing environment.

[0046] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

[0048] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0049] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0050] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0051] A person of ordinary skill in the art can understand that all or part of the steps in realizing the above-mentioned facts and methods can be completed by instructing the relevant hardware through a program, and the program involved or the program described can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: At this time, the corresponding method steps are derived, and the storage medium can be ROM / RAM, a disk, an optical disk, etc.

[0052] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cloud computing task scheduling optimization method based on deep reinforcement learning, characterized in that: include: The state space of the deep reinforcement learning algorithm is constructed with the operating parameters of the cloud computing server, and the action space of the deep reinforcement learning algorithm is constructed with the selection actions of the cloud computing task on the cloud computing server; Taking the state space and action space of the deep reinforcement learning algorithm as a basis, obtaining a target action for a cloud computing task; Execute the target action of the cloud computing task and update the state space of the deep reinforcement learning algorithm to obtain the updated state space and the target reward at the same time; According to the original state space, target action, target reward and updated state space, a prior experience pool is constructed, and the target Q value corresponding to the target network is obtained according to the prior experience pool; According to the target action, target reward and updated state space, obtain the predicted Q value corresponding to the main network, and obtain the loss function value according to the predicted Q value and the target Q value; According to the loss function value, the network parameters of the main network are updated to obtain an updated main network, and when the next cloud computing task arrives, the updated main network is used for task scheduling; Wherein, after each network synchronization cycle, the network parameters of the main network are synchronized to the target network.

2. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The state space of the deep reinforcement learning algorithm is constructed using the operating parameters of the cloud computing server, including: The queuing time, remaining computing resources and task size of queued cloud computing tasks corresponding to each cloud computing server are used as the state space of the deep reinforcement learning algorithm.

3. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 1 is characterized in that: The action space of the deep reinforcement learning algorithm is constructed based on the selection action of the cloud computing task on the cloud computing server, including: For any cloud computing task to be scheduled, the action of each cloud computing server selected by the cloud computing task to be scheduled is used to construct the action space of the deep reinforcement learning algorithm.

4. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 1 is characterized in that: Based on the state space and action space of the deep reinforcement learning algorithm, target actions for cloud computing tasks are obtained, including: Based on the state space and action space of the deep reinforcement learning algorithm, determine the Q value corresponding to each action; wherein the Q value is obtained by querying the Q value table, and the Q value between the state and the action at the initial moment is initialized to a random number between (0, 0.1); The Softmax function is used to normalize the action function value corresponding to each action to obtain the probability value corresponding to each action; Based on the probability value corresponding to each action, a target action is selected to obtain the target action for the cloud computing task.

5. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 4 is characterized in that: The target rewards are: ;in, Represents the state space in time section t Select target action The rewards you can get, Represents the cost after executing the target action, Indicates the time consumed after executing the target action.

6. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 5 is characterized in that: Based on the original state space, target action, target reward, and updated state space, a prior experience pool is constructed, including: Construct a priori experience sample based on the original state space, target action, target reward and updated state space, and add the prior experience sample to the prior experience pool each time a cloud computing task is executed; For any prior experience sample, the priority of the prior experience sample is determined according to the target reward: ;in, Represents a positive number less than 0.0001; When the number of prior experience samples in the prior experience pool is greater than or equal to a preset number threshold, the prior experience sample with the smallest priority is removed to obtain a prior experience pool of a fixed size.

7. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 6 is characterized in that: According to the prior experience pool, the target Q value corresponding to the target network is obtained, including: According to the prior experience samples in the prior experience pool, the updated state space is used as the input of the target network to obtain the output of the target network, and the maximum Q value is determined as the target Q value based on the output of the target network.

8. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 7 is characterized in that: According to the target action, target reward, and updated state space, the predicted Q value corresponding to the main network is obtained as follows: in, Indicates the main network selects the target action The corresponding predicted Q value is then, Indicates the Q value corresponding to the target action before the main network selects it. represents the learning rate, represents the discount factor, Represents the state space after the update Predicting selected actions Maximize the Q value; after obtaining the predicted Q value corresponding to the main network, use the predicted Q value to update the Q value table.

9. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 8 is characterized in that: According to the predicted Q value and the target Q value, the loss function value is obtained: Among them, L represents the loss function value, E represents the expected value, Indicates the target Q value.

10. The cloud computing task scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that: According to the loss function value, the network parameters of the main network are updated to obtain an updated main network, and when the next cloud computing task arrives, the updated main network is used for task scheduling, including: Based on the loss function value, the network parameters of the main network are updated using the gradient descent algorithm, and the update is repeated until the main network converges, and the action Q value is generated using the main network; According to the action Q value generated by the main network, the scheduling of cloud computing tasks is optimized.

Images