A method and device for collaborative optimization of energy and computing power based on reinforcement learning

Abstract

The present invention discloses a method and device for collaborative optimization of energy and computing power based on reinforcement learning, which defines the attributes and states of tasks and nodes to obtain definition data; minimizes the energy cost required by the scheduling system under the condition of accelerating the task completion time to obtain the minimum energy cost; designs constraints according to the supply and demand matching between resources and tasks; constructs an energy-computing collaborative scheduling model based on deep reinforcement learning technology and performs training; obtains the optimal "task-node" scheduling strategy through processing of the trained energy-computing collaborative scheduling model, and makes real-time adjustments to the operating state of the computing power node and the computing object according to it, generates a "task-node" scheduling sequence, and obtains a dynamic computing solution. The present invention helps to fully integrate computing resources, storage resources, and network resources, realize collaborative computing across devices, systems, and regions, reduce computing costs, and improve computing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of energy and computing power collaborative technology, and specifically to a method and device for energy and computing power collaborative optimization based on reinforcement learning. Background Art

[0002] Existing research and technologies related to energy and computing power mostly start from energy and computing power themselves, and rarely involve content related to energy-computing power synergy. In terms of computing power technology, current technical research mainly involves research on computing power scheduling, analysis, storage, allocation, measurement, perception, routing, and services; in terms of energy technology, current research mainly focuses on energy management, scheduling, and trading.

[0003] At present, there are few research methods on energy-computing power synergy. A small amount of research on energy-computing power synergy technology mainly studies energy-computing power synergy from the system architecture level. For example, the energy network computing power synergy system proposes specific system module architectures such as infrastructure, perception, model, control, intelligent brain, and control layer; the computing power and energy flow collaborative scheduling system proposes component architectures such as dynamic control, data processing, and waste heat heating modules. It lacks advanced algorithms related to energy-computing power synergy, and cannot achieve effective integration of energy and computing power. It is difficult to achieve the balanced development of efficient computing power scheduling and green energy saving.

[0004] Therefore, how to invent a method for collaborative optimization of energy and computing power to achieve collaborative computing across devices, systems, and regions, reduce computing costs, improve computing efficiency, and meet the ever-growing computing needs has become an urgent problem to be solved. Summary of the invention

[0005] To this end, the present invention provides a method and device for collaborative optimization of energy and computing power based on reinforcement learning, which forms a reasonable and effective collaborative scheduling of energy and computing power, helps to fully integrate computing resources, storage resources and network resources, realize cross-device, cross-system and cross-region collaborative computing, reduce computing costs, improve computing efficiency, meet the growing computing needs, and provide users with efficient, convenient and safe computing services. At the same time, while efficiently completing task load scheduling, it achieves the effect of green energy saving.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a method for collaborative optimization of energy and computing power based on reinforcement learning, comprising:

[0007] According to the process of task segmentation and assignment to nodes, the attributes and states of tasks and nodes are defined to obtain definition data;

[0008] Based on the comprehensive design perspective of the scheduling system, the energy cost required by the scheduling system is minimized under the condition of accelerating the task completion time to obtain the minimum energy cost;

[0009] Design constraints based on the supply and demand match between resources and tasks;

[0010] Based on deep reinforcement learning technology, an energy-computing collaborative scheduling model is constructed according to the definition data, the minimum energy consumption and the constraint conditions;

[0011] The energy-computing collaborative scheduling model is trained by a training data group to obtain the trained energy-computing collaborative scheduling model; and an optimal “task-node” scheduling strategy is obtained by processing the trained energy-computing collaborative scheduling model;

[0012] According to the optimal "task-node" scheduling strategy, the operating status and computing objects of the computing power nodes are adjusted in real time to generate a "task-node" scheduling sequence that changes over time, and obtain a dynamic computing solution under energy and computing collaboration.

[0013] As a preferred solution of a method for collaborative optimization of energy and computing power based on reinforcement learning, the definition data includes:

[0014] Node={Node1,Node2,…,Node N}: node list, the subscript is the node index, N is the total number of nodes;

[0015] (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space;

[0016] Job={Job1,Job2,…,Job M}: task list, the subscript is the task index, M is the total number of tasks;

[0017] (op j ,ene j ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements;

[0018] Pr={Pr1,Pr2,…,Pr M}: Task priority;

[0019] I j : Submit a job j The node index of the computing task Job j The required data is in the node Ij Initialize on;

[0020] [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order;

[0021] (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements;

[0022] A jk (t): Subtask O jk The node index assigned at decision time t should be the node that initially submitted the task before the decision is made, i.e., A jk =I j ;

[0023] T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation;

[0024] Stat={Stat1,Stat2,…,Stat N}: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated;

[0025] Tr={Tr1,Tr2,…,Tr N}: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation);

[0026] Use={Use1,Use2,…,Use N}: The remaining available energy of each node at the time of current decision (task allocation);

[0027] Store={Store1,Store2,…,Store N}: The remaining storage space of each node at the time of current decision (task allocation);

[0028] (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption;

[0029] (tc i ,tb i ):Node iThe cumulative computing time and the cumulative idle time;

[0030] (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

[0031] As a preferred solution for the energy and computing power collaborative optimization method based on reinforcement learning, in the process of minimizing the energy cost required by the scheduling system, a time cost function L is established for all current tasks. time , and minimize the time cost function to achieve the purpose of accelerating the task completion time; by establishing an energy cost function L for all current tasks ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure;

[0032] The expression of the time cost function is:

[0033] minimize L time (t) = ∑ j L time,j (t)

[0034] Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time;

[0035] The expression of the energy cost function is:

[0036] minimize L ene (t) = ∑ j L ene,j (t)

[0037] Where, L ene,j (t) is one of the tasks Job j The energy cost function.

[0038] As a preferred solution of a method for collaborative optimization of energy and computing power based on reinforcement learning, the constraints include:

[0039] The first constraint is:

[0040]

[0041] In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting;

[0042] Second constraint:

[0043]

[0044] The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources;

[0045] The third constraint is:

[0046]

[0047] According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that hour, vice versa;

[0048] The higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

[0049] As a preferred solution of the energy and computing power collaborative optimization method based on reinforcement learning, in the process of training the energy and computing power collaborative scheduling model through the training data group, the training data group generation step is:

[0050] The agent obtains observation data of the agent from the environment and inputs the observation data into the policy network;

[0051] The policy network generates a set action according to the input observation data, and uses the set action as the output of the intelligent agent;

[0052] Several agents input the set actions into the environment, and the environment feeds back the benefit values ​​to the agents;

[0053] The set action, the benefit value and the state of the agent generate a data group, and the data group is stored in an experience replay pool for training the energy-computing collaborative scheduling model.

[0054] The present invention also provides a device for collaborative optimization of energy and computing power based on reinforcement learning, based on the above method for collaborative optimization of energy and computing power based on reinforcement learning, comprising:

[0055] The attribute and state definition module is used to define the attributes and states of tasks and nodes and obtain definition data according to the process of task segmentation and allocation to nodes;

[0056] The minimum energy cost calculation module is used to minimize the energy cost required by the scheduling system under the condition of accelerating the task completion time based on the comprehensive design perspective of the scheduling system to obtain the minimum energy cost;

[0057] The constraint design module is used to design constraints based on the supply and demand matching between resources and tasks;

[0058] An energy-computing collaborative scheduling model construction module is used to construct an energy-computing collaborative scheduling model based on deep reinforcement learning technology according to the definition data, the minimum energy consumption and the constraint conditions;

[0059] The energy-computing collaborative scheduling model training and processing module is used to train the energy-computing collaborative scheduling model through a training data group to obtain the trained energy-computing collaborative scheduling model; and obtain the optimal "task-node" scheduling strategy by processing the trained energy-computing collaborative scheduling model;

[0060] The dynamic computing solution acquisition module under energy-computing collaboration is used to adjust the operating status and computing objects of the computing power nodes in real time according to the optimal "task-node" scheduling strategy, generate a "task-node" scheduling sequence that changes with time, and obtain a dynamic computing solution under energy-computing collaboration.

[0061] As a preferred solution of a device for collaborative optimization of energy and computing power based on reinforcement learning, in the attribute and state definition module, the definition data includes:

[0062] Node={Node1,Node2,…,Node N}: node list, the subscript is the node index, N is the total number of nodes;

[0063] (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space;

[0064] Job={Job1,Job2,…,Job M}: task list, the subscript is the task index, M is the total number of tasks;

[0065] (op j ,ene j ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements;

[0066] Pr={Pr1,Pr2,…,Pr M}: Task priority;

[0067] I j : Submit a job j The node index of the computing task Job j The required data is in the node Ij Initialize on;

[0068] [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order;

[0069] (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements;

[0070] A jk (t): Subtask O jk The node index assigned at the decision time t should be the node that initially submitted the task before the decision is made, that is, A jk =I j ;

[0071] T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation;

[0072] Stat={Stat1,Stat2,…,Stat N}: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated;

[0073] Tr={Tr1,Tr2,…,Tr N}: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation);

[0074] Use={Use1,Use2,…,Use N}: The remaining available energy of each node at the time of current decision (task allocation);

[0075] Store={Store1,Store2,…,Store N}: The remaining storage space of each node at the time of current decision (task allocation);

[0076] (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption;

[0077] (tc i ,tb i ):Node i The cumulative computing time and the cumulative idle time;

[0078] (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

[0079] As a preferred solution of the energy and computing power collaborative optimization device based on reinforcement learning, in the minimum energy cost calculation module, in the process of minimizing the energy cost required by the scheduling system, a time cost function L is established for all current tasks. time , and minimize the time cost function to achieve the purpose of accelerating the task completion time; by establishing an energy cost function L for all current tasks ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure;

[0080] The expression of the time cost function is:

[0081] minimize L time (t)=∑ j L time,j (t)

[0082] Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time;

[0083] The expression of the energy cost function is:

[0084] minimize L ene (t)=∑ j L ene,j (t)

[0085] Where, L ene,j (t) is one of the tasks Job j The energy cost function.

[0086] As a preferred solution of a device for collaboratively optimizing energy and computing power based on reinforcement learning, in the constraint design module, the constraints include:

[0087] The first constraint is:

[0088]

[0089] In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting;

[0090] Second constraint:

[0091]

[0092] The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources;

[0093] The third constraint is:

[0094]

[0095] According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that hour, vice versa;

[0096] The higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

[0097] As a preferred solution of an energy and computing power collaborative optimization device based on reinforcement learning, the energy and computing power collaborative scheduling model training and processing module includes:

[0098] The observation data acquisition and processing submodule is used for the agent to obtain the observation data of the agent from the environment and input the observation data into the policy network;

[0099] A set action generation submodule is used for the strategy network to generate a set action according to the input observation data, and use the set action as the output of the intelligent agent;

[0100] The benefit value acquisition submodule is used for a plurality of intelligent agents to input the set action into the environment, and the environment feeds back the benefit value to the intelligent agent;

[0101] The data group generation submodule is used to generate a data group from the set action, the benefit value and the state of the agent, and store the data group in the experience replay pool for the training of the energy-computing collaborative scheduling model.

[0102] The present invention has the following advantages: the present invention defines the attributes and states of tasks and nodes according to the process of task segmentation and assignment to nodes, and obtains definition data; based on the comprehensive design perspective of the scheduling system, under the condition of accelerating the task completion time, the energy expenditure required by the scheduling system is minimized to obtain the minimum energy expenditure; according to the supply and demand matching between resources and tasks, constraints are designed; based on deep reinforcement learning technology, according to the definition data, the minimum energy expenditure and the constraints, an energy-computing collaborative scheduling model is constructed; the energy-computing collaborative scheduling model is trained through a training data group to obtain the trained energy-computing collaborative scheduling model; the optimal "task-node" scheduling strategy is obtained by processing the trained energy-computing collaborative scheduling model; according to the optimal "task-node" scheduling strategy, the operating state and computing object of the computing power node are adjusted in real time, and a "task-node" scheduling sequence that changes with time is generated, and a dynamic computing scheme under energy-computing collaboration is obtained. The present invention fills the gap in energy-computing power collaborative technology, can realize the efficient integration of energy and computing power, and achieve the purpose of coordinated and balanced development of efficient computing power scheduling and green energy saving. The present invention can achieve fast decision-making and real-time optimization by using reinforcement learning, maximize reward expectations through training strategies, and is applicable to multi-step decision optimization problems. It has the advantages of fast decision-making and real-time optimization, can achieve high synergy between computing power and energy, and contributes to the efficient allocation and scheduling of energy and computing power. Through the synergistic integration of "computing power and energy", the present invention helps to promote the use of more renewable energy in data center clusters and promote the green development of the computing power economy. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0104] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0105] Figure 1 This is a schematic flow chart of a method for collaborative optimization of energy and computing power based on reinforcement learning provided in Example 1 of the present invention;

[0106] Figure 2 A schematic diagram of a solution framework of an energy and computing power collaborative scheduling model in a method for collaborative optimization of energy and computing power based on reinforcement learning provided in Example 1 of the present invention;

[0107] Figure 3 A schematic diagram of the training process of an energy-computing collaborative scheduling model in a method for collaborative optimization of energy and computing power based on reinforcement learning provided in Example 1 of the present invention;

[0108] Figure 4 A schematic diagram of a training algorithm flow of an energy-computing collaborative scheduling model in a method for collaborative optimization of energy and computing power based on reinforcement learning provided in Example 1 of the present invention;

[0109] Figure 5 A schematic diagram of the execution flow of an energy-computing collaborative scheduling model in a method for collaborative optimization of energy and computing power based on reinforcement learning provided in Example 1 of the present invention;

[0110] Figure 6 This is a schematic diagram of the architecture of an energy and computing power collaborative optimization device based on reinforcement learning provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0111] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0112] Example 1

[0113] See also Figure 1 Embodiment 1 of the present invention provides a method for collaborative optimization of energy and computing power based on reinforcement learning, comprising the following steps:

[0114] S1. According to the process of task segmentation and assignment to nodes, the attributes and states of tasks and nodes are defined to obtain definition data;

[0115] S2. Based on the comprehensive design perspective of the scheduling system, the energy cost required by the scheduling system is minimized under the condition of accelerating the task completion time to obtain the minimum energy cost;

[0116] S3. Design constraints based on the supply and demand matching between resources and tasks;

[0117] S4. Based on deep reinforcement learning technology, an energy-computing collaborative scheduling model is constructed according to the definition data, the minimum energy consumption and the constraint conditions;

[0118] S5. Train the energy-computing collaborative scheduling model through the training data group to obtain the trained energy-computing collaborative scheduling model; obtain the optimal “task-node” scheduling strategy through processing the trained energy-computing collaborative scheduling model;

[0119] S6. According to the optimal "task-node" scheduling strategy, the operating status and computing objects of the computing power nodes are adjusted in real time to generate a "task-node" scheduling sequence that changes over time, and obtain a dynamic computing solution under energy-computing collaboration.

[0120] In this embodiment, in a decentralized distributed computing power scheduling system with limited available resources, different computing power nodes are located at different positions in the physical space, and each node has a certain computing and storage capacity. Since there is no central node to coordinate task allocation, it may lead to energy waste caused by repeated communication and calculation, and the phenomenon of long-term incomplete tasks. The present invention constructs an energy-computing collaborative scheduling model based on the multi-objective constraint control of various available resources in the computing power nodes and the matching between them and the resource requirements of the tasks; and uses multi-agent reinforcement learning technology to solve the model, intelligently segment complex computing tasks, and generate intelligent agent strategies for distributed computing power networks, so as to achieve the effects of computing acceleration and green energy saving.

[0121] In this embodiment, in step S1, according to the process of task segmentation and assignment to nodes, the attributes and states of tasks and nodes are defined to obtain definition data;

[0122] Specifically, the definition data includes:

[0123] Node={Node1,Node2,…,Node N}: node list, the subscript is the node index, N is the total number of nodes;

[0124] (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space;

[0125] Job={Job1,Job2,…,Job M}: task list, the subscript is the task index, M is the total number of tasks;

[0126] (op j ,enej ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements;

[0127] Pr={Pr1,Pr2,…,Pr M}: Task priority;

[0128] I j : Submit a job j The node index of the computing task Job j The required data is in the node Ij Initialize on;

[0129] [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order;

[0130] (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements;

[0131] A jk (t): Subtask O jk The node index assigned at decision time t should be the node that initially submitted the task before the decision is made, i.e., A jk =I j ;

[0132] T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation;

[0133] Stat={Stat1,Stat2,…,Stat N}: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated;

[0134] Tr={Tr1,Tr2,…,Tr N}: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation);

[0135] Use={Use1,Use2,…,Use N}: The remaining available energy of each node at the time of current decision (task allocation);

[0136] Store={Store1,Store2,…,Store N}: The remaining storage space of each node at the time of current decision (task allocation);

[0137] (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption;

[0138] (tc i ,tb i ):Node i The cumulative computing time and the cumulative idle time;

[0139] (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

[0140] In this embodiment, in step S2, based on the comprehensive design perspective of the scheduling system, the energy cost required by the scheduling system is minimized under the condition of accelerating the task completion time to obtain the minimum energy cost;

[0141] Specifically, from the perspective of comprehensive design of the scheduling system, under the condition of accelerating the task completion time, the energy cost required for calculation is minimized. By establishing a time cost function L for all current tasks time , and minimize the time cost function so that the scheduling system can achieve the purpose of accelerating the task completion time;

[0142] The expression of the time cost function is:

[0143] minimize L time (t)=∑ j L time,j (t)

[0144] Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time;

[0145] L time,j (t)=∑ k L time,j,k (t)

[0146] Where, L time,j (t) is the task Job at the decision time tj The total time it takes for the k subtasks to be assigned to the corresponding nodes to perform calculations.

[0147] Among them, L time,j,k (t) measures O jk From the node at the last moment To the current node The transmission time, the calculation time at the current node, and the waiting time are three items in total:

[0148]

[0149] Among them, the waiting time Tw jk Depend on The running status of the node determines that if it is idle, there is no need to wait. Otherwise, check the remaining time of the task currently running on the node as the current waiting time.

[0150] In this embodiment, an energy cost function L is established for all current tasks. ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure;

[0151] The expression of the energy cost function is:

[0152] minimize L ene (t)=∑ j L ene,j (t)

[0153] Where, L ene,j (t) is one of the tasks Job j The energy cost function.

[0154] L ene,j (t)=∑ k L ene,j,k (t)

[0155] Where, L ene,j (t) is the task Job at the decision time t j The total time it takes for the k subtasks to be assigned to the corresponding nodes to perform calculations.

[0156]

[0157] L ene,j,k (t) measures O jk From the node at the last moment To the current node The transmission energy consumption and the computing energy consumption at the current node.

[0158] In this embodiment, in step S3, constraints are designed based on the supply and demand matching between resources and tasks;

[0159] Specifically, when executing task splitting and allocating computing nodes to subtasks at decision time t, if the running status value of a node is 1, the node is occupied and cannot be allocated; if the running status value is 0, allocation is allowed. At this time, it is necessary to observe node resources, calculate time cost and energy consumption cost, and ensure that the resource demand after task splitting matches the node resource supply. It should be noted that in addition to the constraints of control dependencies on subtasks, tasks also have their own priorities.

[0160] The design constraints include:

[0161] The first constraint is:

[0162]

[0163] In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting;

[0164] Second constraint:

[0165]

[0166] The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources;

[0167] The third constraint is:

[0168]

[0169] According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that hour, Vice versa; the higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

[0170] In this embodiment, in step S4, based on deep reinforcement learning technology, an energy-computing collaborative scheduling model is constructed according to the definition data, the minimum energy consumption and the constraint conditions;

[0171] Specifically, based on deep reinforcement learning technology, an energy-computing collaborative scheduling model is constructed according to the definition data, the minimum energy consumption and the constraint conditions;

[0172] In the reinforcement learning model, the three most important components are the state of the environment s(t), the action space a(t) of the agent, and the reward or benefit r(t) after executing the action. The energy-computing collaborative scheduling model of the present invention regards N nodes as N agents, the remaining resources of the nodes, the remaining computing time of the tasks, etc. as the state of the environment, the operation of the nodes, and the assigned task number as the action space of the agents, and uses the opposite number of the optimization target in the previous section to design the benefit after executing the action.

[0173] In addition, when multiple agents make unified decisions, the cumulative operating energy consumption (equivalent to the computing energy consumption of all tasks) and the cumulative idle energy consumption of all nodes can also be counted, that is, Pay attention to energy waste; and the final completion time when all tasks are calculated in parallel

[0174] When the number of nodes and task lists is large, the state space and action space of the collaborative scheduling model are large, and deep learning needs to be adopted and implemented as a neural network fitting, that is, deep reinforcement learning. For the policy network π in the multi-agent deep reinforcement learning framework, its goal is to maximize the benefits under the constraints and form the optimal strategy π * :

[0175]

[0176] In the formula, ω time ,ω ene ,ω t ,ω e They are the weights of task time cost, task energy cost, parallel computing completion time cost, and node energy cost.

[0177] In this embodiment, in step S5, the energy-computing collaborative scheduling model is trained by a training data group to obtain the trained energy-computing collaborative scheduling model; the trained energy-computing collaborative scheduling model is processed to obtain the optimal "task-node" scheduling strategy;

[0178] Specifically, the present invention uses computer simulation technology to build a small computing network with limited resources, and randomly generates tasks and their resource requirements, which are used to solve the "task-node" scheduling strategy in the energy computing collaborative scheduling model. The solution process is as follows Figure 2 As shown, it includes two major units: training and execution.

[0179] Among them, in the model training unit, the intelligent agent is each computing power node, the environment is the system environment simulated by the present invention for the scheduling process through the computer simulation platform, the state includes various types of resources available in the node and the set of tasks waiting to be calculated, and the action includes node operation (whether the node is in the calculation operation) and task allocation (selecting which task to calculate). In each step of learning, the state change after the calculation of the complex computing task is simulated, and the centralized training method of the policy network based on the multi-layer neural network is adopted to obtain the optimal node calculation and task allocation scheme, namely the "task-node" scheduling strategy. The system environment will return the status of each node and the task completion status in real time; then, according to the benefit value information returned by the system environment, the value network based on the multi-layer neural network is used to evaluate the pros and cons of the scheme, so as to optimize the strategy and obtain the optimal "task-node" scheduling strategy.

[0180] Specifically, each intelligent agent, that is, a computing power node, has a policy network and a value network. The policy network can quickly output an action that enables the intelligent agent to obtain the maximum expected benefit based on the observation information of the intelligent agent, while the value network of the intelligent agent only appears in the model training stage to evaluate the actions output by the intelligent agent's policy network and to assist in the training of the intelligent agent's policy network parameters.

[0181] Among them, Figure 3 As shown in the solid line part, the training data set generation steps are:

[0182] S51, the agent obtains the observation data of the agent from the environment, and inputs the observation data into the strategy network;

[0183] Specifically, agent i obtains its own observation data o from the environment i And input to the policy network π i ;

[0184] S52, the policy network generates a set action according to the input observation data, and uses the set action as the output of the intelligent agent;

[0185] Specifically, the policy network generates an action a based on the input information i As the output of agent i;

[0186] S53, a number of agents input the set action into the environment, and the environment feeds back the benefit value to the agent;

[0187] Specifically, all agents input their actions into the environment, and the environment feeds back a benefit value r to each agent. i ;

[0188] S54, generating a data group based on the set action, the benefit value and the state of the agent, and storing the data group in an experience replay pool for training the energy-computing collaborative scheduling model.

[0189] Specifically, each agent generates a data set (s i ,a i ,r i ) is stored in the experience replay pool P for the next model training; where s i Represents the state of agent i, which contains the information of agent i itself and the information observed from the environment.

[0190] In this embodiment, the specific training process is as follows: Figure 3 As shown in the dotted part of , the experience replay pool P is sampled to obtain n training data D, and the sampled data is used to train the multi-agent reinforcement learning network to complete the value network parameter φ=(φ1, φ2, …, φ N ) and policy network parameters θ=(θ1,θ2,···,θ N ) is updated. The specific training algorithm process is as follows Figure 4 As shown in the figure, the parameters and value parameters of each agent strategy network are initialized as input, and finally the optimal parameters after training are output through algorithm training; in the centralized training stage, the global state information needs to be obtained, that is, the observation information (assuming that o is equivalent to s) and action information of all agents are used as the input of each value network, that is, the input of the value network i of each agent i is the joint observation information o=(o1, o2,…, o N ) and joint action information a=(a1,a2,…,a N ), output agent i in the observation data is o i Take action a i Q value Then, according to the calculated network gradient dφ i The value network parameter φ for agent i i Similarly, the sampled data and the value network are used to calculate the policy network gradient dθ and update the parameter θ.

[0191] In this embodiment, in step S6, according to the optimal "task-node" scheduling strategy, the operating status and computing objects of the computing power nodes are adjusted in real time to generate a "task-node" scheduling sequence that changes with time, and obtain a dynamic computing solution under energy and computing collaboration.

[0192] Specifically, when using a computer simulation platform to simulate a series of complex computing tasks, the operating status and computing objects of each computing power node are adjusted in real time at regular intervals according to the "task-node" scheduling optimal strategy, generating a "task-node" scheduling sequence that changes over time, forming a sequence of task calculations, and obtaining a dynamic computing solution under energy-computing collaboration.

[0193] After the model is trained, the value network of the agent is discarded, and the policy network is used to execute the scheduling model. The agent can use its own policy network to quickly obtain an action that can maximize its expected benefits based on the information observed from the environment. The actions of each agent constitute the joint action of task scheduling a = (a1, a2, ..., a N ),like Figure 5 As shown in the figure, after inputting the joint action into the environment, the benefit value r = (r1, r2, ..., r N ) to evaluate the actions taken.

[0194] In summary, the present invention defines the attributes and states of tasks and nodes according to the process of task segmentation and assignment to nodes, and obtains definition data; based on the comprehensive design perspective of the scheduling system, under the condition of accelerating the task completion time, the energy expenditure required by the scheduling system is minimized to obtain the minimum energy expenditure; according to the supply and demand matching between resources and tasks, constraints are designed; based on deep reinforcement learning technology, according to the definition data, the minimum energy expenditure and the constraints, an energy-computing collaborative scheduling model is constructed; the energy-computing collaborative scheduling model is trained through a training data group to obtain the trained energy-computing collaborative scheduling model; the trained energy-computing collaborative scheduling model is processed to obtain the optimal "task-node" scheduling strategy; according to the optimal "task-node" scheduling strategy, the operating state and computing object of the computing power node are adjusted in real time, and a "task-node" scheduling sequence that changes with time is generated, and a dynamic computing scheme under energy-computing collaboration is obtained. The present invention fills the gap in energy-computing power collaborative technology, can realize the efficient integration of energy and computing power, and achieve the purpose of coordinated and balanced development of efficient computing power scheduling and green energy saving. The present invention can achieve fast decision-making and real-time optimization by using reinforcement learning, maximize reward expectations through training strategies, and is applicable to multi-step decision optimization problems. It has the advantages of fast decision-making and real-time optimization, can achieve high synergy between computing power and energy, and contributes to the efficient allocation and scheduling of energy and computing power. Through the synergistic integration of "computing power and energy", the present invention helps to promote the use of more renewable energy in data center clusters and promote the green development of the computing power economy.

[0195] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0196] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] Example 2

[0198] See also Figure 6 Embodiment 2 of the present invention further provides an energy and computing power collaborative optimization device based on reinforcement learning, comprising:

[0199] The attribute and state definition module 001 is used to define the attributes and states of tasks and nodes and obtain definition data according to the process of task segmentation and allocation to nodes;

[0200] The minimum energy cost calculation module 002 is used to minimize the energy cost required by the scheduling system under the condition of accelerating the task completion time based on the comprehensive design perspective of the scheduling system, so as to obtain the minimum energy cost;

[0201] The constraint design module 003 is used to design constraint conditions according to the supply and demand matching between resources and tasks;

[0202] The energy-computing collaborative scheduling model building module 004 is used to build an energy-computing collaborative scheduling model based on the deep reinforcement learning technology according to the definition data, the minimum energy consumption and the constraint conditions;

[0203] The energy-computing collaborative scheduling model training and processing module 005 is used to train the energy-computing collaborative scheduling model through a training data group to obtain the trained energy-computing collaborative scheduling model; and obtain the optimal "task-node" scheduling strategy through processing the trained energy-computing collaborative scheduling model;

[0204] The dynamic computing solution acquisition module 006 under energy-computing collaboration is used to adjust the operating status and computing objects of the computing power nodes in real time according to the optimal "task-node" scheduling strategy, generate a "task-node" scheduling sequence that changes with time, and obtain a dynamic computing solution under energy-computing collaboration.

[0205] In this embodiment, in the attribute and state definition module 001, the definition data includes:

[0206] Node={Node1,Node2,…,Node N}: node list, the subscript is the node index, N is the total number of nodes;

[0207] (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space;

[0208] Job={Job1,Job2,…,Job M}: task list, the subscript is the task index, M is the total number of tasks;

[0209] (op j ,ene j ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements;

[0210] Pr={Pr1,Pr2,…,Pr M}: Task priority;

[0211] I j : Submit a job j The node index of the computing task Job j The required data is in the node Ij Initialize on;

[0212] [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order;

[0213] (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements;

[0214] A jk (t): Subtask O jk The node index assigned at decision time t should be the node that initially submitted the task before the decision is made, i.e., A jk =I j ;

[0215] T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation;

[0216] Stat={Stat1,Stat2,…,Stat N}: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated;

[0217] Tr={Tr1,Tr2,…,Tr N}: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation);

[0218] Use={Use1,Use2,…,Use N}: The remaining available energy of each node at the time of current decision (task allocation);

[0219] Store={Store1,Store2,…,Store N}: The remaining storage space of each node at the time of current decision (task allocation);

[0220] (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption;

[0221] (tc i ,tb i ):Node i The cumulative computing time and the cumulative idle time;

[0222] (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

[0223] In this embodiment, in the minimum energy cost calculation module 002, in the process of minimizing the energy cost required by the scheduling system, a time cost function L is established for all current tasks. time , and minimize the time cost function to achieve the purpose of accelerating the task completion time; by establishing an energy cost function L for all current tasks ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure;

[0224] The expression of the time cost function is:

[0225] minimize L time (t)=∑ j L time,j (t)

[0226] Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time;

[0227] The expression of the energy cost function is:

[0228] minimize L ene (t)=∑ j L ene,j (t)

[0229] Where, L ene,j (t) is one of the tasks Job j The energy cost function.

[0230] In this embodiment, in the constraint design module 003, the constraint includes:

[0231] The first constraint is:

[0232]

[0233] In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting;

[0234] Second constraint:

[0235]

[0236] The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources;

[0237] The third constraint is:

[0238]

[0239] According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that hour, Vice versa; the higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

[0240] In this embodiment, the energy computing collaborative scheduling model training and processing module 005 includes:

[0241] The observation data acquisition and processing submodule 051 is used for the agent to obtain the observation data of the agent from the environment and input the observation data into the strategy network;

[0242] A set action generation submodule 052 is used for the strategy network to generate a set action according to the input observation data, and use the set action as the output of the intelligent agent;

[0243] The benefit value acquisition submodule 053 is used for a plurality of agents to input the set action into the environment, and the environment feeds back the benefit value to the agent;

[0244] The data group generation submodule 054 is used to generate a data group from the set action, the benefit value and the state of the agent, and store the data group in the experience replay pool for the training of the energy-computing collaborative scheduling model.

[0245] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0246] Example 3

[0247] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method for collaborative optimization of energy and computing power based on reinforcement learning is stored. The program code includes instructions for executing embodiment 1 or any possible implementation method of a method for collaborative optimization of energy and computing power based on reinforcement learning.

[0248] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0249] Example 4

[0250] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0251] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute a method for collaborative optimization of energy and computing power based on reinforcement learning in Example 1 or any possible implementation thereof.

[0252] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0253] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0254] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0255] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for collaborative optimization of energy and computing power based on reinforcement learning, characterized in that: include: According to the process of task segmentation and assignment to nodes, the attributes and states of tasks and nodes are defined to obtain definition data; Based on the comprehensive design perspective of the scheduling system, the energy cost required by the scheduling system is minimized under the condition of accelerating the task completion time to obtain the minimum energy cost; Design constraints based on the supply and demand match between resources and tasks; Based on deep reinforcement learning technology, an energy-computing collaborative scheduling model is constructed according to the definition data, the minimum energy consumption and the constraint conditions; The energy-computing collaborative scheduling model is trained by a training data group to obtain the trained energy-computing collaborative scheduling model; and an optimal "task-node" scheduling strategy is obtained by processing the trained energy-computing collaborative scheduling model; According to the optimal "task-node" scheduling strategy, the operating status and computing objects of the computing power nodes are adjusted in real time to generate a "task-node" scheduling sequence that changes over time, and obtain a dynamic computing solution under energy-computing collaboration.

2. The energy and computing power collaborative optimization method based on reinforcement learning according to claim 1, characterized in that: The definition data includes: Node={Node1,Node2,…,Node N }: node list, the subscript is the node index, N is the total number of nodes; (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space; Job={Job1,Job2,…,Job M }: task list, the subscript is the task index, M is the total number of tasks; (op j ,ene j ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements; Pr={Pr1,Pr2,…,Pr M }: Task priority; I j : Submit a job j The node index of the computing task Job j The required data is in the node Initialize on; [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order; (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements; A jk (t): Subtask O jk The node index assigned at decision time t should be the node that initially submitted the task before the decision is made, i.e., A jk =I j ; T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation; Stat={Stat1,Stat2,…,Stat N }: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated; Tr={Tr1,Tr2,…,Tr N }: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation); Use={Use1,Use2,…,Use N }: The remaining available energy of each node at the time of current decision (task allocation); Store={Store1,Store2,…,Store N }: The remaining storage space of each node at the time of current decision (task allocation); (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption; (tc i ,tb i ):Node i The cumulative computing time and the cumulative idle time; (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

3. The energy and computing power collaborative optimization method based on reinforcement learning according to claim 2 is characterized in that: In the process of minimizing the energy cost required by the scheduling system, a time cost function L is established for all current tasks. time , and minimize the time cost function to achieve the purpose of accelerating the task completion time; by establishing an energy cost function L for all current tasks ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure; The expression of the time cost function is: Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time; The expression of the energy cost function is: Where, L ene,j (t) is one of the tasks Job j The energy cost function.

4. The energy and computing power collaborative optimization method based on reinforcement learning according to claim 3 is characterized in that: The constraints include: The first constraint is: In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting; Second constraint: The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources; The third constraint is: (L time,j1 (t)-L time,j2 (t))·(Pr j1 -Pr j2 )≤0 According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that hour, vice versa; The higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

5. The energy and computing power collaborative optimization method based on reinforcement learning according to claim 4 is characterized in that: In the process of training the energy computing collaborative scheduling model through the training data group, the training data group generation step is: The agent obtains observation data of the agent from the environment and inputs the observation data into the policy network; The policy network generates a set action according to the input observation data, and uses the set action as the output of the intelligent agent; Several agents input the set actions into the environment, and the environment feeds back the benefit values ​​to the agents; The set action, the benefit value and the state of the agent generate a data group, and the data group is stored in an experience replay pool for training the energy-computing collaborative scheduling model.

6. A device for collaborative optimization of energy and computing power based on reinforcement learning, adopting a method for collaborative optimization of energy and computing power based on reinforcement learning according to any one of claims 1 to 5, characterized in that: include: The attribute and state definition module is used to define the attributes and states of tasks and nodes and obtain definition data according to the process of task segmentation and allocation to nodes; The minimum energy cost calculation module is used to minimize the energy cost required by the scheduling system under the condition of accelerating the task completion time based on the comprehensive design perspective of the scheduling system to obtain the minimum energy cost; The constraint design module is used to design constraints based on the supply and demand matching between resources and tasks; An energy-computing collaborative scheduling model construction module is used to construct an energy-computing collaborative scheduling model based on deep reinforcement learning technology according to the definition data, the minimum energy consumption and the constraint conditions; The energy-computing collaborative scheduling model training and processing module is used to train the energy-computing collaborative scheduling model through a training data group to obtain the trained energy-computing collaborative scheduling model; and obtain the optimal "task-node" scheduling strategy by processing the trained energy-computing collaborative scheduling model; The dynamic computing solution acquisition module under energy-computing collaboration is used to adjust the operating status and computing objects of the computing power nodes in real time according to the optimal "task-node" scheduling strategy, generate a "task-node" scheduling sequence that changes with time, and obtain a dynamic computing solution under energy-computing collaboration.

7. The energy and computing power collaborative optimization device based on reinforcement learning according to claim 6, characterized in that: In the attribute and state definition module, the definition data includes: Node={Node1,Node2,…,Node N }: node list, the subscript is the node index, N is the total number of nodes; (FLOP i ,MAX_USE i ,MAX_STORE i ):Node j The computing power scale (number of floating point operations per second), the maximum available energy, and the maximum storage space; Job={Job1,Job2,…,Job M }: task list, the subscript is the task index, M is the total number of tasks; (op j ,ene j ,data j ): Task Job j The computing power requirements (total number of floating-point operations), energy requirements, and data storage requirements; Pr={Pr1,Pr2,…,Pr M }: Task priority; I j : Submit a job j The node index of the computing task Job j The required data is in the node Ij Initialize on; [O j1 ,O j2 ,O j3 ,…]:Task Job j The second digit of the subscript in the segmented subtask sequence is the subtask execution order; (op jk ,ene jk ,data jk ): Subtask O jk The computing power requirements, energy requirements, and data storage requirements; A jk (t): Subtask O jk The node index assigned at decision time t should be the node that initially submitted the task before the decision is made, i.e., A jk =I j ; T jk :Subtask O jk The waiting time is the time difference from the completion of the previous subtask to the actual start of the transmission calculation; Stat={Stat1,Stat2,…,Stat N }: The running status of each node at the time of the current decision (task allocation). When the value is 1, it means that the calculation is in use, and when it is 0, it means that it is idle and can be allocated; Tr={Tr1,Tr2,…,Tr N }: The remaining computation time of the subtasks being processed by each node at the time of the current decision (task allocation); Use={Use1,Use2,…,Use N }: The remaining available energy of each node at the time of current decision (task allocation); Store={Store1,Store2,…,Store N }: The remaining storage space of each node at the time of current decision (task allocation); (ec i ,eb i ):Node i The energy consumption cost per second when participating in computing and when idle, that is, running energy consumption and idle energy consumption; (tc i ,tb i ):Node i The cumulative computing time and the cumulative idle time; (Ts a,b ,Te a,b ):Node a and Node b The data transmission time and data transmission energy consumption coefficient are the time cost and energy consumption cost corresponding to the transmission of each unit of data.

8. The energy and computing power collaborative optimization device based on reinforcement learning according to claim 7, characterized in that: In the minimum energy cost calculation module, in the process of minimizing the energy cost required by the scheduling system, a time cost function L is established for all current tasks. time , and minimize the time cost function to achieve the purpose of accelerating the task completion time; by establishing an energy cost function L for all current tasks ene , and minimize the energy cost function to achieve the purpose of reducing energy expenditure; The expression of the time cost function is: Where, L time,j (t) is one of the tasks Job j The time cost function of ; t is the set decision time; The expression of the energy cost function is: Where, L ene,j (t) is one of the tasks Job j The energy cost function.

9. The energy and computing power collaborative optimization device based on reinforcement learning according to claim 8, characterized in that: In the constraint design module, the constraints include: The first constraint is: In the task O j When splitting, the total resource requirements of the subtasks after splitting are consistent with the resource requirements before splitting; Second constraint: The subtask O jk The resource requirements cannot exceed the allocated nodes of remaining resources; The third constraint is: According to the priority ranking Pr between tasks, the completion time of any two tasks j1 and j2 is constrained to ensure that when Pr j1 >Pr j2 When L time,j1 ≤L time,j2 ,vice versa; The higher the task priority, the larger the Pr value, and the earlier the task is completed. That is, when the task starts execution at the same time t, the time cost L of task j is time，j The lower (t).

10. The energy and computing power collaborative optimization device based on reinforcement learning according to claim 9, characterized in that: The energy computing collaborative scheduling model training and processing module includes: The observation data acquisition and processing submodule is used for the agent to obtain the observation data of the agent from the environment and input the observation data into the policy network; A set action generation submodule is used for the strategy network to generate a set action according to the input observation data, and use the set action as the output of the intelligent agent; The benefit value acquisition submodule is used for a plurality of intelligent agents to input the set action into the environment, and the environment feeds back the benefit value to the intelligent agent; The data group generation submodule is used to generate a data group from the set action, the benefit value and the state of the agent, and store the data group in the experience replay pool for the training of the energy-computing collaborative scheduling model.

Images