A multi-resource scheduling method, system, storage medium and terminal

Abstract

The embodiment of the present invention discloses a multi-resource scheduling method, system, storage medium and terminal, and the method includes: obtaining multi-resource training data; configuring multi-resource scheduling model training conditions of value network and policy network; making the value network and policy network interact with the environment according to the multi-resource scheduling model training conditions to obtain an optimization sequence; obtaining the loss functions of the value network and policy network respectively according to the optimization sequence, updating the value network parameters and policy network parameters, and obtaining the optimization functions of the value network and policy network respectively from the optimization sequence through a random selection strategy; determining the model error according to the loss function and minimizing the model error according to the optimization function to obtain an optimal multi-resource scheduling model; selecting the optimal node from the multi-resource training data through the optimal multi-resource scheduling model, and assigning tasks to the node. By combining the decision-making ability of the value network and the perception ability of the policy network, the convergence of the multi-resource scheduling method is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource scheduling, and in particular to a multi-resource scheduling method, system, storage medium and terminal. Background Art

[0002] With the rapid development of Internet technology, cloud computing technology is becoming a strategic focus of the development of the information technology industry and is gradually entering people's lives. Cloud computing technology provides users with various IT (Internet Technology) resources (computing, storage, etc.) in data centers around the world through the Internet. Large-scale data centers currently have hundreds of thousands of servers, and effective management of resources in such large-scale data centers requires the use of resource scheduling technology in cloud computing technology. Resource scheduling technology is the most critical part of cloud computing technology. It can help cloud computing solve large-scale processing of resources and information, thereby achieving better resource allocation.

[0003] Existing resource scheduling algorithms use heuristic algorithms or genetic algorithms, which have inherent defects: first, slow convergence and poor real-time performance; second, they cannot guarantee the overall optimal solution for scheduling allocation, are unstable, and have poor calculation accuracy. Summary of the invention

[0004] Based on this, it is necessary to propose a multi-resource scheduling method, system, storage medium and terminal to address the above problems.

[0005] A multi-resource scheduling method, the method comprising:

[0006] Multi-resource training data is acquired, where the multi-resource training data includes node cluster information and task information.

[0007] Configure the training conditions of the multi-resource scheduling model for the value network and the policy network.

[0008] The value network and the policy network are made to interact with the environment according to the training conditions of the multi-resource scheduling model to obtain an optimization sequence, wherein the optimization sequence includes a state, an action, and a reward.

[0009] The loss functions of the value network and the policy network are respectively obtained according to the optimization sequence, the value network parameters and the policy network parameters are updated, and the optimization functions of the value network and the policy network are respectively obtained from the optimization sequence by randomly selecting a strategy.

[0010] The model error is determined according to the loss function and the model error is minimized according to the optimization function to obtain the optimal multi-resource scheduling model.

[0011] The optimal node is selected from the multi-resource training data through the optimal multi-resource scheduling model, and tasks are assigned to the node.

[0012] The multi-resource scheduling model training conditions for configuring the value network and the policy network specifically include:

[0013] Initialize the value network and the policy network, set the maximum number of rounds and samples for each selection, reset the environment and set the reward to zero.

[0014] The step of respectively obtaining the loss functions of the value network and the policy network according to the optimization sequence, updating the value network parameters and the policy network parameters, and respectively obtaining the optimization functions of the value network and the policy network from the optimization sequence by randomly selecting a strategy specifically includes:

[0015] The loss function of the value network is obtained according to the state and reward in the optimization sequence, and the value network parameters are updated.

[0016] The optimization function of the value network under different inputs is obtained from the optimization sequence through a random selection strategy, and the loss function of the value network is optimized.

[0017] The loss function of the policy network is obtained according to the state and reward in the optimization sequence, and the policy network parameters are updated.

[0018] The optimization function of the policy network is obtained from the optimization sequence by randomly selecting a strategy, and the loss function of the policy network is optimized.

[0019] Repeat the above steps until the maximum number of samples is reached, and then proceed to the next round of selection.

[0020] The step of respectively obtaining the loss functions of the value network and the policy network according to the optimization sequence, updating the value network parameters and the policy network parameters, and respectively obtaining the optimization functions of the value network and the policy network from the optimization sequence by randomly selecting a strategy specifically includes:

[0021] The random selection strategy is given by G t =R t+1 +γR t+2 +...+γ n-1 R t+n +γ n Q(S t+n ,a t+n ) is determined, where R t+n represents the reward of the nth step at time t, S t+n represents the state of the nth step at time t, a t+n Represents the action of step n at time t.

[0022] The step of obtaining the optimization function of the value network under different inputs from the optimization sequence by using a random selection strategy and optimizing the loss function of the value network specifically includes:

[0023] When the environment inputs the state to the value network, the optimization function of the value network is Determine, where L critic is the optimization function of the value network, N is the trajectory length, V St The current state value output by the value network of the state at time t.

[0024] When the environment inputs states and actions to the value network, the optimization function of the value network is given by Determine, where Q(s t ,a t ) is the value of the action taken in the current state output by the value network of the state at time t.

[0025] The step of obtaining the optimization function of the policy network from the optimization sequence by randomly selecting a strategy and optimizing the loss function of the policy network specifically includes:

[0026] The optimization function of the policy network is given by Determine, where π* is the optimal strategy.

[0027] The step of determining the model error according to the loss function and minimizing the model error according to the optimization function to obtain the optimal multi-resource scheduling model specifically includes:

[0028] At least one round of iteration is performed according to the optimization function to minimize the loss function, wherein the loss function includes a model error, until the round iteration reaches the maximum number of rounds of iteration, thereby obtaining an optimal multi-resource scheduling model.

[0029] A resource scheduling system, the system comprising:

[0030] The multi-resource training data acquisition module is used to acquire multi-resource training data, where the multi-resource training data includes node cluster information and task information.

[0031] Configuration module, used to configure the training conditions of the multi-resource scheduling model of the value network and the policy network.

[0032] The optimization sequence acquisition module is used to make the value network and the strategy network interact with the environment according to the training conditions of the multi-resource scheduling model to obtain the optimization sequence, and the optimization sequence includes state, action, and reward.

[0033] An updating module is used to obtain the loss functions of the value network and the policy network respectively according to the optimization sequence, update the value network parameters and the policy network parameters, and obtain the optimization functions of the value network and the policy network respectively from the optimization sequence through a random selection strategy.

[0034] The optimal multi-resource scheduling model acquisition module is used to determine the model error according to the loss function and minimize the model error according to the optimization function to obtain the optimal multi-resource scheduling model.

[0035] The task allocation module is used to select the best node from the multi-resource training data through the optimal multi-resource scheduling model and allocate tasks to the node.

[0036] A storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method described above.

[0037] A terminal, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0038] The embodiments of the present invention have the following beneficial effects:

[0039] The present invention provides a multi-resource scheduling method based on the value network and the policy network in deep reinforcement learning. By combining the decision-making ability of the value network and the perception ability of the policy network, the convergence of the multi-resource scheduling method is improved, and it can cope with quasi-real-time scenarios. The optimal multi-resource scheduling model can be obtained only by training the model, and then the overall optimal solution of multi-resource scheduling allocation is obtained. The calculation accuracy is high, and the random selection strategy is adopted to enhance the generalization ability and robustness of the optimal multi-resource scheduling model, ensuring its stable performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] in:

[0042] Figure 1 It is a flowchart of an embodiment of a multi-resource scheduling method provided by the present invention;

[0043] Figure 2It is a flowchart of another embodiment of a multi-resource scheduling method provided by the present invention;

[0044] Figure 3 It is a flow chart of an embodiment of a multi-resource scheduling system provided by the present invention;

[0045] Figure 4 It is a flow chart of another embodiment of a multi-resource scheduling system provided by the present invention;

[0046] Figure 5 is a schematic structural diagram of an embodiment of a terminal provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of an embodiment of the storage medium provided by the present invention. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] The development of information technology has promoted the rapid development of a series of Internet applications such as e-commerce and social networks. These emerging applications have the characteristics of abundant hardware service resources, large data storage capacity, and rapid business growth. Traditional computer computing models cannot solve these complex problems, and scheduling technology in cloud computing must be used to effectively manage these information and resources. Resource scheduling technology is the most critical part of cloud computing technology. It can help cloud computing solve large-scale processing of resources and information, thereby achieving better resource allocation. Traditional allocation scheduling algorithms such as heuristic algorithms cannot solve large, complex, dynamic, and real-time resource allocation problems because of their own limitations. Therefore, it is very necessary to design a multi-resource scheduling method to optimize and solve the above problems.

[0050] like Figure 1 As shown, Figure 1 1 is a flow chart of an embodiment of a multi-resource scheduling method provided by the present invention. A multi-resource scheduling method provided by the present invention comprises the following steps:

[0051] S101: Acquire multi-resource training data, where the multi-resource training data includes node cluster information and task information.

[0052] In one implementation scenario, the acquired multi-resource training data includes node cluster information and task information. Specifically, the node cluster information includes: the total number of node processors, the total node memory, the number of node remaining processors, the node remaining memory, and the node cluster information is as follows:

[0053] Mcluster={[node1_totoalcpu,node2_totoalcpu...noden_totoalcpu][node1_totalme m,node2_totalmem...noden_totalmem][node1_restcpu,node2_restcpu...noden_restcpu][node1_restmem,node2_restmem...noden_restmem]};

[0054] Among them, nodel_totoalcpu indicates the total number of processors of the node, nodel totalmem indicates the total memory of the node, nodel restcpu indicates the remaining processors of the node, and nodel restmen indicates the remaining memory of the node.

[0055] Task information includes the requirements for processor resources and memory resources. Task information is as follows:

[0056] Mjob={[job1_cpu,job2_cpu...jobn_cpu][job1_mem,job2_mem...jobn_mem]}.

[0057] S102: Configure the multi-resource scheduling model training conditions of the value network and the policy network.

[0058] In an implementation scenario, when configuring the training conditions of the multi-resource scheduling model of the value network and the policy network, the value network and the policy network are first initialized, the maximum number of iteration rounds and the maximum number of samples are set, the environment is reset and the reward is set to zero. Specifically, the learning rate of the value network and the learning rate of the policy network are initialized; the maximum number of iteration rounds is set, recorded as episode; the maximum number of samples is set, recorded as batch_size; the evaluation network parameters and the policy network parameters are initialized, where the evaluation network parameters are recorded as Qq and the policy network parameters are recorded as Qπ.

[0059] S103: According to the training conditions of the multi-resource scheduling model, the value network and the policy network interact with the environment to obtain an optimization sequence, where the optimization sequence includes states, actions, and rewards.

[0060] In a specific implementation scenario, after configuring the multi-resource scheduling model training conditions of the value network and the policy network, the value network and the policy network interact with the environment to obtain the optimization sequence. During the interaction, the value network and the policy network will obtain the state value obtained by observing the environment, and generate the actions of the value network and the policy network according to the multi-resource data input. The value network and the policy network can only decide the actions they make. The actions of the value network and the policy network are merged to generate joint actions. The environment transfers the state according to the actions selected by the value network and the policy network, and at the same time, the corresponding rewards are fed back to the value network and the policy network according to the changes in the state. The optimization sequence includes state, action, and reward. The optimization sequence is defined as {s0, a0, r1, s1, a1, r2....}, s is the state, a is the joint action, and r is the reward.

[0061] It should be noted that the environment is implemented using the gym library, and the input is multi-resource training data and running status information. When the environment inputs the environment state information to the value network, the value network evaluates the current state value and outputs it; when the environment inputs the environment state and the action taken to the value network, the value network evaluates the value of taking the action under the current state and outputs it. The environment inputs the current state to the policy network, and the output of the value network is the probability distribution of the current action. The value network then evaluates the quality of the action and adjusts the strategy accordingly.

[0062] S104: Obtain the loss functions of the value network and the policy network respectively according to the optimization sequence, update the value network parameters and the policy network parameters, and obtain the optimization functions of the value network and the policy network respectively from the optimization sequence by randomly selecting a strategy.

[0063] In one implementation scenario, the loss function of the value network is first obtained based on the state and reward in the optimization sequence, where the loss function of the value network is given by Determine; obtain the loss function of the policy network according to the state and return in the optimization sequence, where the loss function of the policy network is given by Determine and update the value network parameters and strategy network parameters.

[0064] Specifically, when updating the value network parameters and the policy network parameters, the loss function of the value network is used according to an iterative function Update the value network parameters and set the value network parameters Q q The value of Q is assigned to the current step q , continuously update in a loop; the loss function of the policy network is based on an iterative function Update the policy network parameters, assign the value of the policy network parameter Qπ in the previous step to the Qπ in the current step, and keep updating in a loop. A1 is the learning rate of the value network, and A2 is the learning rate of the policy network.

[0065] Secondly, the error value obtained by the loss function of the value network and the policy network is optimized by the optimization function of the value network and the policy network. The information input to the value network by the environment includes the state and the state and action. When the environment inputs the state to the value network, the optimization function of the value network is determined by the following formula:

[0066]

[0067] Where N is the trajectory length, V St The current state value output by the value network of the state at time t;

[0068] When the environment inputs states and actions into the value network, the optimization function of the value network is determined by the following formula:

[0069]

[0070] Among them, Q(s t ,a t ) is the value of the action taken in the current state output by the value network of the state at time t.

[0071] It should be noted that the random selection strategy is determined according to the following formula:

[0072] G t =R t+1 +γR t+2 +...+γ n-1 R t+n +γ n Q(S t+n ,a t+n );

[0073] Among them, R t+n represents the reward of the nth step at time t, S t+n represents the state of the nth step at time t, a t+n represents the action of the nth step at time t, and γ is the discount factor.

[0074] The value network and the policy network perform parameter updates and optimization of the loss functions of the value network and the policy network at least once until the preset maximum number of samples, batch_size, is reached and the next round of iteration is performed.

[0075] S105: Determine the model error according to the loss function and minimize the model error according to the optimization function to obtain the optimal multi-resource scheduling model.

[0076] In a specific implementation scenario, the value network and policy network after one round of iteration interact with the environment again to obtain an optimization sequence. The loss function and optimization function of the value network and policy network are obtained through the optimization sequence and the random selection strategy, and the value network parameters and policy network parameters are updated. The optimization function of the value network and the optimization function of the policy network continuously optimize the model error obtained by the loss function of the value network and the loss function of the policy network through round iteration until the preset maximum number of iterations episode is reached, and the optimization ends. The purpose of this method is to continuously interact with the environment so that the value network and the policy network can continuously adjust their strategies to obtain the optimal multi-resource scheduling model.

[0077] S106: Select the best node from the multi-resource training data through the optimal multi-resource scheduling model, and assign tasks to the node.

[0078] In a specific implementation scenario, multi-resource data is input into the optimal multi-resource scheduling model to generate actions of the value network and the policy network, and resources are allocated according to the actions of the value network and the policy network to make the actions effective. The optimal node is selected from the node cluster information and task information through the optimal multi-resource scheduling model, and tasks are assigned to the node.

[0079] From the above description, it can be seen that the present invention provides a multi-resource scheduling method based on the value network and the policy network in deep reinforcement learning. By combining the decision-making ability of the value network and the perception ability of the policy network, the convergence of the multi-resource scheduling method is improved, and it can cope with quasi-real-time scenarios. The optimal multi-resource scheduling model can be obtained only by training the model, and then the overall optimal solution of multi-resource scheduling allocation is obtained. The calculation accuracy is high, and the random selection strategy is adopted to enhance the generalization ability and robustness of the optimal multi-resource scheduling model, ensuring its stable performance.

[0080] like Figure 2 and Figure 3 As shown, Figure 2 is a flow chart of another embodiment of a multi-resource scheduling method provided by the present invention. Figure 3 The present invention provides a multi-resource scheduling method, which includes the following steps:

[0081] S201: Initialize the value network and the strategy network.

[0082] S202: Set the maximum number of iteration rounds and the maximum number of samples.

[0083] S203: Reset the environment and set the return equal to zero.

[0084] S204: The value network and the strategy network interact with the environment to obtain an optimization sequence, which includes states, actions, and rewards.

[0085] S205: Obtain the loss function of the value network according to the state and reward in the optimization sequence, and obtain the optimization function of the value network under different inputs according to the optimization sequence and the loss function of the value network.

[0086] S206: Obtain the loss function of the policy network according to the state and reward in the optimization sequence, and obtain the optimization function of the policy network according to the optimization sequence and the loss function of the policy network.

[0087] S207: Update the value network parameters and the strategy network parameters.

[0088] S208: Repeat steps S204-S207 until the preset maximum number of samples is reached, and then proceed to the next round of iteration.

[0089] S209: Repeat steps S203-S208 until the preset maximum number of iterations is reached, the training is completed, and the model training ends.

[0090] In an implementation scenario, the multi-resource scheduling system includes an environment module 11, a value network module 12, a policy network module 13, and an update module 14. The input of the environment module 11 includes the state of the environment and the action taken, the value network module 12 is used to evaluate the value of the action, and the policy network module 13 outputs the action and the probability distribution of the current action. The environment module 11 interacts with the value network module 12 and the policy network module 13, and the value network parameters and the policy network parameters are updated in the update module 14.

[0091] Specifically, when the environment inputs the environment state information to the value network, the value network evaluates the current state value and outputs it to the update module 14; when the environment inputs the environment state and the action taken to the value network, the value network evaluates the value of taking the action under the current state and outputs it to the update module 14. The environment inputs the current state to the policy network, and the output of the policy network is the probability distribution of the current action. The value network then evaluates the quality of the action and adjusts the strategy accordingly. The action output by the policy network is output to the environment module 11, and the feedback from the environment module 11 is directly output to the update module 14, in which the value network parameters and policy network parameters are updated.

[0092] It should be noted that the specific method for the multi-resource scheduling system to perform multi-resource scheduling through steps S201-S209 has been Figure 1 The implementation scenario shown is discussed in detail and will not be repeated here.

[0093] like Figure 4As shown, Figure 4 This is a flow chart of another embodiment of a multi-resource scheduling system provided by the present invention. Figure 4 , a multi-resource scheduling system 20 comprises:

[0094] The multi-resource training data acquisition module 21 is used to acquire multi-resource training data, where the multi-resource training data includes node cluster information and task information.

[0095] Configuration module 22 is used to configure the training conditions of the multi-resource scheduling model of the value network and the policy network.

[0096] The optimization sequence acquisition module 23 is used to make the value network and the policy network interact with the environment according to the training conditions of the multi-resource scheduling model to obtain the optimization sequence, and the optimization sequence includes state, action, and reward;

[0097] The updating module 24 is used to obtain the loss functions of the value network and the policy network respectively according to the optimization sequence, update the value network parameters and the policy network parameters, and obtain the optimization functions of the value network and the policy network respectively from the optimization sequence by randomly selecting a strategy.

[0098] The optimal multi-resource scheduling model acquisition module 25 is used to determine the model error according to the loss function and minimize the model error according to the optimization function to obtain the optimal multi-resource scheduling model.

[0099] The task allocation module 26 is used to select the best node from the multi-resource training data through the optimal multi-resource scheduling model and allocate tasks to the node.

[0100] In a specific implementation scenario, the multi-resource training data acquisition module 21 includes a data set, a task generator, a task queue and a machine resource table, and the multi-resource scheduling system also includes a node cluster. The data set in the multi-resource training data acquisition module 21 stores the scheduling task request information, and the scheduling task request information includes RDMS (Relational Data Management System) or message middleware. There are three types of information: the end of a single task scheduling, the arrival of a new task, and the completion of the task operation; the task generator reads data from the data set, generates corresponding tasks according to the rules, and finally submits it to the node cluster for scheduling and execution; the task queue is used to access the task information submitted by the task generator, and adopts the FIFO (First Input First Output) method; the machine resource table is used to store the information of each node in the node cluster, including the total processors, the total memory, and the remaining processors, the remaining memory and other information. The node cluster information and task information are obtained through the multi-resource training data acquisition module; the multi-resource scheduling model training conditions of the value network and the policy network are configured through the configuration module 22, and the configured value network and policy network interact with the environment to obtain the optimization sequence; the loss function and optimization function of the value network and the policy network are obtained according to the optimization sequence and the random selection strategy through the update module 24, and the value network parameters and the policy network parameters are updated; the optimization function of the value network and the optimization function of the policy network are obtained through the optimal multi-resource scheduling model acquisition module 25, and the model error obtained by the loss function of the value network and the loss function of the policy network are continuously optimized through round iteration until the optimization is completed, and the optimal multi-resource scheduling model is obtained; finally, the multi-resource data is input into the optimal multi-resource scheduling model through the task allocation module 26 to generate the actions of the value network and the policy network, and the resources are allocated according to the actions of the value network and the policy network to make the actions effective, and the optimal node is selected from the node cluster information and the task information through the optimal multi-resource scheduling model, and the task is assigned to the node.

[0101] It should be noted that the node cluster is composed of several independent hosts (nodes), and the resource configuration of each independent host may be heterogeneous. For each host's resources, the overall processor, memory, and remaining processors are considered. The processors include graphics processors and central processing units. In the scheduling process, each task to be scheduled includes the requirements for processor resources and memory resources. In the scheduling process, each host is selected as the target for task scheduling. When the resource requirements of the scheduled task can be met by the scheduled host, the scheduling is effective. At the same time, the labeling method in kubernetes is used to support task preferences, such as whether there is a graphics processor, SSD disk, etc.

[0102] like Figure 5 As shown, Figure 5 30 is a schematic diagram of a terminal according to an embodiment of the present invention. The terminal 30 includes a memory 31 and a processor 32. The memory 31 stores a computer program, and the processor 32 executes the computer program when working to implement the following. Figure 1 and Figure 2 The method shown.

[0103] The specific technical details of a multi-resource scheduling method implemented when the terminal 30 executes a computer program have been discussed in detail in the above method steps, so they will not be repeated here.

[0104] like Figure 6 As shown, Figure 6 The storage medium 40 stores at least one computer program 41, which is executed by the processor 32 to implement the following. Figure 1 and Figure 2 In one embodiment, the storage medium 40 may be a storage chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server, etc.

[0105] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0107] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0108] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0109] For the convenience of description, the above device is described by being divided into various units according to their functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0115] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0117] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0118] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A multi-resource scheduling method, characterized in that: The method comprises: Acquire multi-resource training data, where the multi-resource training data includes node cluster information and task information; Configure the training conditions of the multi-resource scheduling model for the value network and the policy network; The value network and the policy network are made to interact with the environment according to the training conditions of the multi-resource scheduling model to obtain an optimization sequence, wherein the optimization sequence includes a state, an action, and a reward; According to the state and reward in the optimization sequence, the loss function of the value network is obtained and the value network parameters are updated; the optimization function of the value network under different inputs is obtained from the optimization sequence through a random selection strategy, and the loss function of the value network is optimized, including: when the environment inputs the state to the value network, the optimization function of the value network is Determine, where L critic is the optimization function of the value network, N is the trajectory length, V(S t ) is the current state value output by the value network of the state at time t, G t is a random selection strategy; When the environment inputs states and actions to the value network, the optimization function of the value network is given by Determine, where Q(s t ,a t ) is the value of taking an action in the current state output by the value network of the state at time t; obtain the loss function of the policy network according to the state and reward in the optimization sequence and update the policy network parameters; obtain the optimization function of the policy network from the optimization sequence by randomly selecting a strategy, and optimize the loss function of the policy network; repeat the above steps until the maximum number of samples is reached, and continue the next round of selection; Determine the model error according to the loss function and minimize the model error according to the optimization function to obtain an optimal multi-resource scheduling model; The optimal multi-resource scheduling model is used to select the optimal node from the multi-resource training data, and tasks are assigned to the node.

2. A multi-resource scheduling method according to claim 1, characterized in that: The multi-resource scheduling model training conditions for configuring the value network and the policy network specifically include: Initialize the value network and the policy network, set the maximum number of rounds and the maximum number of samples for each selection, reset the environment and set the reward to zero.

3. A multi-resource scheduling method according to claim 1, characterized in that: The step of respectively obtaining the loss functions of the value network and the policy network according to the optimization sequence, updating the value network parameters and the policy network parameters, and respectively obtaining the optimization functions of the value network and the policy network from the optimization sequence by randomly selecting a strategy specifically includes: The random selection strategy is given by G t =R t+1 +γR t+2 +...+γ n-1 R t+n +γ n Q(S t+n ,a t+n ) is determined, where R t+n represents the reward of the nth step at time t, S t+n represents the state of the nth step at time t, a t+n It represents the action of the nth step at time t, γ represents the attenuation coefficient, which is configured through hyperparameter optimization and has a value range of [0, 1]. The smaller the γ value, the more emphasis is placed on immediate rewards, and the larger the value, the more emphasis is placed on long-term rewards.

4. A multi-resource scheduling method according to claim 1, characterized in that: The step of obtaining the optimization function of the policy network from the optimization sequence by randomly selecting a strategy and optimizing the loss function of the policy network specifically includes: The optimization function of the policy network is given by Determine, where π* is the optimal strategy.

5. A multi-resource scheduling method according to claim 1, characterized in that: Determining the model error according to the loss function and minimizing the model error according to the optimization function to obtain the optimal multi-resource scheduling model specifically includes: At least one round of iteration is performed according to the optimization function to minimize the loss function, wherein the loss function includes a model error, until the round iteration reaches the maximum number of rounds of iteration, thereby obtaining an optimal multi-resource scheduling model.

6. A resource scheduling system, characterized in that: The system comprises: A multi-resource training data acquisition module, used to acquire multi-resource training data, wherein the multi-resource training data includes node cluster information and task information; Configuration module, used to configure the training conditions of multi-resource scheduling models of value network and policy network; An optimization sequence acquisition module, used to enable the value network and the policy network to interact with the environment according to the training conditions of the multi-resource scheduling model to obtain an optimization sequence, wherein the optimization sequence includes a state, an action, and a reward; An updating module is used to obtain the loss function of the value network and update the value network parameters according to the state and reward in the optimization sequence; obtain the optimization function of the value network under different inputs from the optimization sequence through a random selection strategy, and optimize the loss function of the value network, including: when the environment inputs the state to the value network, the optimization function of the value network is Determine, where L critic is the optimization function of the value network, N is the trajectory length, V(S t ) is the current state value output by the value network of the state at time t, G t is a random selection strategy; When the environment inputs states and actions to the value network, the optimization function of the value network is given by Determine, where Q(s t ,a t ) is the value of taking an action in the current state output by the value network of the state at time t; obtain the loss function of the policy network according to the state and reward in the optimization sequence and update the policy network parameters; obtain the optimization function of the policy network from the optimization sequence by randomly selecting a strategy, and optimize the loss function of the policy network; repeat the above steps until the maximum number of samples is reached, and continue the next round of selection; An optimal multi-resource scheduling model acquisition module, used to determine a model error according to the loss function and minimize the model error according to an optimization function to obtain an optimal multi-resource scheduling model; The task allocation module is used to select the best node from the multi-resource training data through the optimal multi-resource scheduling model and allocate tasks to the node.

7. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

8. A terminal, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

Images