Heterogeneous computing power resource dynamic scheduling method based on deep reinforcement learning
By using a deep reinforcement learning-based approach, we achieved fine-grained scheduling control and long-term optimal strategy learning for heterogeneous resources. This solved the problem of low efficiency in traditional resource scheduling methods when facing dynamically changing computational task flows, and improved the system's stability and adaptability.
Patent Information
- Application Number
- CN202610171121.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-15
AI Technical Summary
Traditional resource scheduling methods lack a global optimization perspective and forward-looking decision-making ability when faced with dynamic, diverse, and strictly quality-of-service (QoS)-constrained computing task flows. This makes it difficult to achieve an effective balance between indicators such as resource utilization, task completion time, and energy consumption, resulting in low overall system efficiency.
By employing a deep reinforcement learning-based approach, a deep neural network is constructed by defining a continuous action space, designing an environment interaction mechanism and an experience replay system, and learning a long-term optimal strategy for multiple objectives. This enables refined and flexible scheduling control of heterogeneous resources. Furthermore, through temporal differential objectives and value network updates, accurate prediction and value guidance of long-term scheduling benefits are achieved.
It achieves intelligent mapping from complex system states to optimal scheduling actions, ensuring the stability and efficiency of model training, possessing strong generalization ability and robustness, and being able to cope with uncertainties such as task load fluctuations and resource failures, thereby improving the overall efficiency of the system.
Smart Images
Figure CN122044873A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed computing technology, specifically to a method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning. Background Technology
[0002] With the rapid development of applications such as artificial intelligence, big data analytics, and scientific computing, computing tasks are characterized by large scale, high complexity, and high real-time requirements, placing higher demands on the underlying computing infrastructure. Traditional resource scheduling methods often lack a global optimization perspective and forward-looking decision-making ability when facing dynamically changing, diverse computing task flows with strict quality-of-service constraints. They struggle to achieve an effective balance among multiple indicators such as resource utilization, task completion time, and energy consumption, resulting in overall system inefficiency. Existing scheduling algorithms suffer from high computational overhead, poor adaptability, and limited scalability when dealing with large-scale, high-dimensional, and state-space complex heterogeneous resource dynamic scheduling problems, and cannot respond in real time to rapid changes in task queues and resource states. Summary of the Invention
[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a dynamic scheduling method for heterogeneous computing resources based on deep reinforcement learning. Traditional resource scheduling methods often lack a global optimization perspective and forward-looking decision-making capabilities when facing dynamically changing, diverse computing task flows with strict quality-of-service constraints. This makes it difficult to achieve an effective balance among multiple indicators such as resource utilization, task completion time, and energy consumption, leading to overall system inefficiency. This solution defines a continuous action space to achieve refined and flexible scheduling control of heterogeneous resources. By constructing a deep neural network, it achieves intelligent mapping from complex system states to optimal scheduling actions. Furthermore, by designing an environmental interaction mechanism and establishing an experience playback system, it learns a long-term optimal strategy that considers multiple objectives. To ensure the stability, efficiency, and sample utilization of model training, this solution addresses the shortcomings of existing scheduling algorithms in handling large-scale, high-dimensional, and complex heterogeneous resource dynamic scheduling problems. These algorithms suffer from high computational overhead, poor adaptability, limited scalability, and an inability to respond in real time to rapid changes in task queues and resource states. This solution utilizes temporal differential objectives and value network updates to accurately predict long-term scheduling benefits and guide value. Through policy gradient theorem and policy network updates, it achieves a direct and efficient mapping from value assessment to policy improvement. A soft update mechanism for the objective network ensures the stability of the training process and the reliability of policy convergence. This solution can cope with uncertainties such as task load fluctuations and resource failures in real-world production environments, exhibiting strong generalization ability and robustness.
[0004] The technical solution adopted in this invention is as follows: The method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning provided by this invention includes the following steps:
[0005] Step S1: System modeling and state awareness, obtain heterogeneous computing power resource pool, collect the queue of computing tasks to be scheduled in real time, and construct a multi-dimensional system state vector at each scheduling moment;
[0006] Step S2: Policy definition and experience storage. Based on the DDPG (Deep Deterministic Policy Gradient) algorithm, construct the policy network and value network, calculate the specific action vector at each scheduling moment, and simulate it in the environment simulator. Store the system state vector, action vector, immediate reward and the system state vector at the next moment as experience samples in the experience replay buffer.
[0007] Step S3: Model training and update. Using the experience replay buffer, update the parameters of the policy network and value network through the DDPG algorithm to generate a scheduling policy.
[0008] Step S4: Online scheduling and dynamic adaptation. The trained model parameters are deployed in the actual scheduling service, and new experience samples are continuously collected to fine-tune the policy network and value network.
[0009] Furthermore, in step S1, the system modeling and state awareness specifically include the following steps:
[0010] Step S11: Resource node modeling. Extract resource nodes for each type of computing power resource in the heterogeneous computing power resource pool. The computing power resources include CPU, GPU, and FPGA. Define a resource description vector for each resource node in the following form: ;
[0011] In the formula, Indicates the type index of computing resources. Indicates the index of the resource node. Indicates the first The first of the various computing resources The resource description vector of each resource node. Indicates the type of computing resources. This represents the theoretical computational power of a resource node. This indicates the current load rate of the resource node. This indicates the current power consumption characteristics of the resource node. Indicates the communication overhead of resource nodes;
[0012] Step S12: Model the computational tasks, obtain the queue of computational tasks to be scheduled, and define the task description vector of the computational tasks in the following form: ;
[0013] In the formula, An index representing the computation task. Represents computational task The task description vector, This indicates the time when a computation task arrives in the computation task queue. Indicates the deadline for the calculation task. This indicates the computational complexity of the task. Indicates the type of computation task. Indicates the size of the data involved in the computation task. Indicates the priority of the computation task;
[0014] Step S13: System state modeling. Construct the system state vector at each scheduling moment, including the load rate of all resource nodes, the deadline, computational load and priority of all computational tasks, and system-level constraints, in the following form: ;
[0015] In the formula, Indicates the scheduling time. Represents the system state vector. This represents a vector concatenation operation. This represents the load rate vector for all resource nodes. Represents the feature vector of all computational tasks. This represents system-level constraints.
[0016] Furthermore, in step S2, the strategy definition and experience storage specifically include the following steps:
[0017] Step S21: Action vector definition. Based on the DDPG algorithm, a continuous action vector space is used. At each scheduling time, a task to be scheduled is selected from the task queue, and a resource node is allocated to the task. The action vector is defined as a three-dimensional vector, in the following form: ;
[0018] In the formula, Indicates the scheduling time. Represents the action vector. Indicates the computational tasks to be scheduled. This indicates the index of the computation task to be scheduled within the computation task queue. This indicates the resource type preference of the computational tasks to be scheduled. This represents the proportion of the load rate of the computation task to be scheduled relative to the current load rate of the allocated resource node;
[0019] Step S22: Neural network construction, including policy network and value network. The policy network outputs deterministic action vectors based on the system state vector, and the value network outputs value estimates based on the system state vector and action vectors.
[0020] Step S23: Environment interaction. The environment simulator is used to execute the action vector output by the policy network, and computing resources are allocated to the computing task to simulate the execution of the computing task. The system state vector at the next moment is obtained, and a composite reward function is designed to evaluate the immediate reward of the action vector under the system state vector.
[0021] Step S24: Experience storage. An experience replay buffer is constructed based on a first-in, first-out (FIFO) strategy to store experience samples generated from each decision. When the experience replay buffer is full, the last experience sample is removed, in the following form: ;
[0022] In the formula, Representing an empirical sample, Represents the action vector. This represents the system state vector at the next moment. This indicates an immediate reward.
[0023] Furthermore, in step S3, the model training and updating specifically includes the following steps:
[0024] Step S31: Initialize the experience replay buffer, initialize the policy network and value network with randomly initialized parameters respectively, and initialize the corresponding target policy network and target value network with the same parameters. Set the discount factor, soft update coefficient and exploration noise parameter, and reset the initial state of the environment simulator using the system state vector.
[0025] Step S32: Offline training. Randomly and uniformly sample a small batch of experience samples from the experience replay buffer. The policy network outputs action vectors based on the initial state and exploration noise parameters. After the environment simulator executes the action vectors, it obtains the immediate reward and the system state vector at the next moment. The experience samples are then stored in the experience replay buffer.
[0026] Step S33: Value network update. The temporal difference objective is calculated for the mini-batch sampled from the experience replay buffer. The parameters of the value network are updated by minimizing the mean squared error loss function. The specific steps are as follows:
[0027] Step S331: Calculate the target value of the time series difference, using the following formula: ;
[0028] In the formula, Indices representing empirical samples. This represents the time-series difference target value of the empirical sample. This represents the immediate reward for the experience sample. Indicates the discount factor. Represents the target value network. Represents the target policy network. Indicates the first The system state vector in an empirical sample;
[0029] Step S332: Value network update. Update the parameters of the value network using the Adam optimizer, using the following formula: ;
[0030] In the formula, The mean squared error loss function represents the value network. The batch size represents the sampling experience. This represents the value estimate of a value network. Represents a value network. and Let represent the system state vector and action vector in the i-th empirical sample, respectively;
[0031] Step S34: Policy network update. Accumulate the discounted reward of the policy network, construct the policy objective function, and calculate the gradient of the policy network. Update the policy network parameters by maximizing the value estimate of the value network, using the following formula: ;
[0032] In the formula, This represents the policy objective function. This represents the policy objective function with respect to the policy network parameters. gradient, Indicates value estimation regarding action gradient, This indicates the policy network with respect to its parameters. The gradient;
[0033] Step S35: Target network soft update. The parameters of the target value network and the target policy network are slowly updated using a soft update method to stabilize the training process. The formula used is as follows: ; ;
[0034] in, Indicates the update coefficients;
[0035] Step S36: State transition and iteration, update the current state of the environment simulator, repeat steps S32 to S35 until the policy network converges.
[0036] The beneficial effects achieved by the present invention using the above solution are as follows:
[0037] (1) Traditional resource scheduling methods often lack a global optimization perspective and forward-looking decision-making ability when facing dynamic, diverse, and strictly quality-of-service-constrained computing task flows. They are difficult to achieve an effective balance among multiple indicators such as resource utilization, task completion time, and energy consumption, resulting in low overall system efficiency. This solution defines a continuous action space to achieve refined and flexible scheduling control of heterogeneous resources. By constructing a deep neural network, it realizes intelligent mapping from complex system states to optimal scheduling actions. By designing an environment interaction mechanism and establishing an experience playback system, it learns a long-term optimal strategy that takes into account multiple objectives, ensuring the stability, efficiency, and sample utilization of model training.
[0038] (2) In view of the problems that existing scheduling algorithms have high computational overhead, poor adaptability and limited scalability when dealing with large-scale, high-dimensional and complex heterogeneous resource dynamic scheduling problems, and cannot respond to the rapid changes in task queue and resource status in real time, this scheme realizes accurate prediction and value guidance of long-term scheduling benefits through temporal differential target and value network update, realizes direct and efficient mapping from value assessment to policy improvement through policy gradient theorem and policy network update, and ensures the stability of training process and the reliability of policy convergence through target network soft update mechanism. It can cope with the uncertainty of task load fluctuation and resource failure in real production environment, and has strong generalization ability and robustness. Attached Figure Description
[0039] Figure 1 This is a flowchart illustrating the dynamic scheduling method for heterogeneous computing resources based on deep reinforcement learning proposed in this invention.
[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0042] Example 1, see Figure 1 The present invention provides a method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning, which includes the following steps:
[0043] Step S1: System modeling and state awareness, obtain heterogeneous computing power resource pool, collect the queue of computing tasks to be scheduled in real time, and construct a multi-dimensional system state vector at each scheduling moment;
[0044] Step S2: Policy definition and experience storage. Based on the DDPG algorithm, construct the policy network and value network, calculate the specific action vector at each scheduling moment, and simulate it in the environment simulator. Use the system state vector, action vector, immediate reward and the system state vector at the next moment as experience samples and store them in the experience playback buffer.
[0045] Step S3: Model training and update. Using the experience replay buffer, update the parameters of the policy network and value network through the DDPG algorithm to generate a scheduling policy.
[0046] Step S4: Online scheduling and dynamic adaptation. The trained model parameters are deployed in the actual scheduling service, and new experience samples are continuously collected to fine-tune the policy network and value network.
[0047] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, the system modeling and state awareness specifically includes the following steps:
[0048] Step S11: Resource node modeling. Extract resource nodes for each type of computing power resource in the heterogeneous computing power resource pool. The computing power resources include CPU, GPU, and FPGA. Define a resource description vector for each resource node in the following form: ;
[0049] In the formula, Indicates the type index of computing resources. Indicates the index of the resource node. Indicates the first The first of the various computing resources The resource description vector of each resource node. Indicates the type of computing resources. This represents the theoretical computational power of a resource node. This indicates the current load rate of the resource node. This indicates the current power consumption characteristics of the resource node. Indicates the communication overhead of resource nodes;
[0050] Step S12: Model the computational tasks, obtain the queue of computational tasks to be scheduled, and define the task description vector of the computational tasks in the following form: ;
[0051] In the formula, An index representing the computation task. Represents computational task The task description vector, This indicates the time when a computation task arrives in the computation task queue. Indicates the deadline for the calculation task. This indicates the computational complexity of the task. Indicates the type of computation task. Indicates the size of the data involved in the computation task. Indicates the priority of the computation task;
[0052] Step S13: System state modeling. Construct the system state vector at each scheduling moment, including the load rate of all resource nodes, the deadline, computational load and priority of all computational tasks, and system-level constraints, in the following form: ;
[0053] In the formula, Indicates the scheduling time. Represents the system state vector. This represents a vector concatenation operation. This represents the load rate vector for all resource nodes. Represents the feature vector of all computational tasks. This represents system-level constraints.
[0054] Example 3, see Figure 1 This embodiment is based on the above embodiment. In step S2, the strategy definition and experience storage specifically include the following steps:
[0055] Step S21: Action vector definition. Based on the DDPG (Deep Deterministic Policy Gradient) algorithm, a continuous action vector space is used. At each scheduling time, a task to be scheduled is selected from the computation task queue, and resource nodes are allocated to the task to be scheduled. The action vector is then defined. It is a three-dimensional vector, in the following form: ;
[0056] In the formula, Indicates the scheduling time. Represents the action vector. Indicates the computational tasks to be scheduled. This indicates the index of the computation task to be scheduled within the computation task queue. This indicates the resource type preference of the computational tasks to be scheduled. This represents the proportion of the load rate of the computation task to be scheduled relative to the current load rate of the allocated resource node;
[0057] Step S22: Neural network construction, including a policy network and a value network. The policy network outputs a deterministic action vector a based on the system state vector s, and the value network outputs a value estimate based on the system state vector s and the action vector a. The specific structure is as follows:
[0058] Policy Network A three-layer MLP network structure is adopted, with the input layer dimension being the same as the system state vector. The dimensions are the same; there are two hidden layers with dimensions of 256 and 128 respectively; the dimensions of the output layer correspond to the three dimensions of the action vector, and tanh is used as the activation function of the output layer.
[0059] Value Network A three-layer MLP network structure is adopted, with the input layer dimension being the system state vector. The sum of the dimensions of the action vector; two hidden layers are set with dimensions of 256 and 128 respectively; the output layer has a dimension of 1 and outputs the value estimate;
[0060] Step S23: Environment interaction, using an environment simulator to process the action vectors output by the policy network. The system executes the task, allocates computing resources to the computational task, simulates the execution of the computational task, and obtains the system state vector at the next time step. Design a composite reward function Evaluate action vectors In the system state vector Instant rewards below;
[0061] Step S24: Experience storage. An experience replay buffer is constructed based on a first-in, first-out (FIFO) strategy to store experience samples generated from each decision. When the experience replay buffer is full, the last experience sample is removed, in the following form: ;
[0062] In the formula, Representing an empirical sample, Represents the action vector. and These represent the system state vectors at the scheduling time and the next time step, respectively. This indicates an immediate reward.
[0063] By performing the above operations, this solution addresses the problem that traditional resource scheduling methods often lack a global optimization perspective and forward-looking decision-making ability when facing dynamically changing, diverse, and strictly quality-of-service-constrained computational task flows. This makes it difficult to achieve an effective balance among multiple indicators such as resource utilization, task completion time, and energy consumption, leading to overall system inefficiency. This solution defines a continuous action space to achieve refined and flexible scheduling control of heterogeneous resources. By constructing a deep neural network, it achieves intelligent mapping from complex system states to optimal scheduling actions. Furthermore, by designing an environmental interaction mechanism and establishing an experience playback system, it learns long-term optimal strategies that consider multiple objectives, ensuring the stability, efficiency, and sample utilization of model training.
[0064] Example 4, see Figure 1 This embodiment is based on the above embodiment. In step S3, the model training and updating specifically includes the following steps:
[0065] Step S31: Initialize the experience replay buffer with a capacity of 100,000. Initialize the policy network and value network using randomly initialized parameters, and initialize the corresponding target policy network and target value network using the same parameters. Set the discount factor, soft update coefficient, and exploration noise parameters, and use the system state vector. Reset the environment simulator to its initial state;
[0066] Step S32: Offline training. Randomly and uniformly sample a small batch of experience samples from the experience replay buffer. The policy network outputs action vectors based on the initial state and exploration noise parameters. After the environment simulator executes the action vectors, it obtains the immediate reward and the system state vector at the next moment. The experience samples are then stored in the experience replay buffer.
[0067] Step S33: Value network update. The temporal difference objective is calculated for the mini-batch sampled from the experience replay buffer. The parameters of the value network are updated by minimizing the mean squared error loss function. The specific steps are as follows:
[0068] Step S331: Calculate the target value of the time series difference, using the following formula: ;
[0069] In the formula, Indices representing empirical samples. This represents the time-series difference target value of the empirical sample. This represents the immediate reward for the experience sample. Indicates the discount factor. Represents the target value network. Represents the target policy network. Indicates the first The system state vector in an empirical sample;
[0070] Step S332: Value network update. Update the parameters of the value network using the Adam optimizer, using the following formula: ;
[0071] In the formula, The mean squared error loss function represents the value network. The batch size represents the sampling experience. This represents the value estimate of a value network. Represents a value network. and Let represent the system state vector and action vector in the i-th empirical sample, respectively;
[0072] Step S34: Policy network update. Accumulate the discounted reward of the policy network, construct the policy objective function, and calculate the gradient of the policy network. Update the policy network parameters by maximizing the value estimate of the value network, using the following formula: ;
[0073] In the formula, This represents the policy objective function. This represents the policy objective function with respect to the policy network parameters. gradient, Indicates value estimation regarding action gradient, This indicates the policy network with respect to its parameters. The gradient;
[0074] Step S35: Target network soft update. The parameters of the target value network and the target policy network are slowly updated using a soft update method to stabilize the training process. The formula used is as follows: ; ;
[0075] in, Indicates the update coefficients;
[0076] Step S36: State transition and iteration, update the current state of the environment simulator, repeat steps S32 to S35 until the policy network converges and a scheduling policy is generated.
[0077] By performing the above operations, this solution addresses the problems of existing scheduling algorithms, such as high computational overhead, poor adaptability, limited scalability, and inability to respond in real time to rapid changes in task queues and resource states, when dealing with large-scale, high-dimensional, and complex heterogeneous resource dynamic scheduling problems. It achieves accurate prediction and value guidance of long-term scheduling benefits through temporal differential target and value network updates. Through the policy gradient theorem and policy network updates, it achieves a direct and efficient mapping from value assessment to policy improvement. The target network soft update mechanism ensures the stability of the training process and the reliability of policy convergence. This solution can cope with uncertainties such as task load fluctuations and resource failures in real production environments, and possesses strong generalization ability and robustness.
[0078] Example 5, see Figure 1 This embodiment is based on the above embodiment. In step S4, the online scheduling and dynamic adaptation specifically involves: solidifying the parameters of the offline trained policy network and integrating them into the scheduling service of the actual production environment; at each scheduling moment, inputting the current system state vector into the policy network, outputting the action vector and parsing it to generate specific resource allocation instructions, which are then sent to the underlying resource manager for execution; continuously collecting new experience samples and adding them to the experience replay buffer; and periodically using the newly collected experience samples to fine-tune the parameters of the policy network and the value network to achieve continuous optimization.
[0079] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0081] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning, characterized in that: The method includes the following steps: Step S1: System modeling and state awareness, obtain heterogeneous computing power resource pool, collect the queue of computing tasks to be scheduled in real time, and construct a multi-dimensional system state vector at each scheduling moment; Step S2: Policy definition and experience storage. Based on the DDPG algorithm, construct the policy network and value network, calculate the specific action vector at each scheduling moment, and simulate it in the environment simulator. Use the system state vector, action vector, immediate reward and the system state vector at the next moment as experience samples and store them in the experience playback buffer. Step S3: Model training and update. Using the experience replay buffer, update the parameters of the policy network and value network through the DDPG algorithm to generate a scheduling policy. Step S4: Online scheduling and dynamic adaptation. The trained model parameters are deployed in the actual scheduling service, and new experience samples are continuously collected to fine-tune the policy network and value network.
2. The method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the system modeling and state awareness specifically include the following steps: Step S11: Resource node modeling, extracting resource nodes for each type of computing power resource in the heterogeneous computing power resource pool, and defining the resource description vector for each resource node; Step S12: Model the computational tasks, obtain the queue of computational tasks to be scheduled, and define the task description vector of the computational tasks; Step S13: System state modeling. Construct the system state vector at each scheduling moment, including the load rate of all resource nodes, the deadline, computational load and priority of all computational tasks, and system-level constraints, in the following form: ； In the formula, Indicates the scheduling time. Represents the system state vector. This represents a vector concatenation operation. This represents a vector of load rates for all resource nodes. Represents the feature vector of all computational tasks. This represents system-level constraints.
3. The method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the strategy definition and experience storage specifically include the following steps: Step S21: Action vector definition. Based on the DDPG algorithm, a continuous action vector space is adopted. At each scheduling time, a computing task to be scheduled is selected from the computing task queue, and a resource node is allocated to the computing task to be scheduled. The action vector is defined as a three-dimensional vector. Step S22: Neural network construction, including policy network and value network. The policy network outputs deterministic action vectors based on the system state vector, and the value network outputs value estimates based on the system state vector and action vectors. Step S23: Environment interaction. The environment simulator is used to execute the action vector output by the policy network, and computing resources are allocated to the computing task to simulate the execution of the computing task. The system state vector at the next moment is obtained, and a composite reward function is designed to evaluate the immediate reward of the action vector under the system state vector. Step S24: Experience storage. An experience replay buffer is constructed based on a first-in, first-out (FIFO) strategy to store experience samples generated from each decision. When the experience replay buffer is full, the last experience sample is removed, in the following form: ； In the formula, Representing an empirical sample, Represents the action vector. and These represent the system state vectors at the scheduling time and the next time step, respectively. This indicates an immediate reward.
4. The method for dynamic scheduling of heterogeneous computing resources based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the model training and updating specifically includes the following steps: Step S31: Initialize the experience replay buffer, initialize the policy network and value network with randomly initialized parameters respectively, and initialize the corresponding target policy network and target value network with the same parameters. Set the discount factor, soft update coefficient and exploration noise parameter, and reset the initial state of the environment simulator using the system state vector. Step S32: Offline training. Randomly and uniformly sample a small batch of experience samples from the experience replay buffer. The policy network outputs action vectors based on the initial state and exploration noise parameters. After the environment simulator executes the action vectors, it obtains the immediate reward and the system state vector at the next moment. The experience samples are then stored in the experience replay buffer. Step S33: Value network update. The temporal difference objective is calculated for the mini-batch sampled from the experience replay buffer. The parameters of the value network are updated by minimizing the mean squared error loss function. The specific steps are as follows: Step S331: Calculate the target value of the time series difference, using the following formula: ； In the formula, Indices representing empirical samples. This represents the time-series difference target value of the empirical sample. This represents the immediate reward for the experience sample. Indicates the discount factor. Represents the target value network. Represents the target policy network. Indicates the first The system state vector in an empirical sample; Step S332: Value network update. Update the parameters of the value network using the Adam optimizer, using the following formula: ； In the formula, The mean squared error loss function represents the value network. The batch size represents the sampling experience. This represents the value estimate of a value network. Represents a value network. and Let represent the system state vector and action vector in the i-th empirical sample, respectively; Step S34: Policy network update. Accumulate the discounted reward of the policy network, construct the policy objective function, and calculate the gradient of the policy network. Update the policy network parameters by maximizing the value estimate of the value network, using the following formula: ； In the formula, This represents the policy objective function. This represents the policy objective function with respect to the policy network parameters. gradient, Indicates value estimation regarding action gradient, This indicates the policy network with respect to its parameters. The gradient; Step S35: Target network soft update, slowly update the parameters of the target value network and the target policy network through soft update; Step S36: State transition and iteration, update the current state of the environment simulator, repeat steps S32 to S35 until the policy network converges.