Resource perception prediction and allocation method based on deep reinforcement learning

Abstract

The invention discloses a resource perception prediction and allocation method based on deep reinforcement learning. According to the method, by adopting a double-depth Q network structure, action selection and action evaluation in target Q value calculation are decoupled, the problems of unstable decision and difficult convergence caused by too high estimation of a Q value in a traditional Q learning algorithm are effectively solved, and more accurate and more efficient resource allocation is realized. By introducing the dynamic self-adaptive decision-making mechanism, the limitation that a traditional static or rule-based allocation strategy is low in resource utilization rate and high in response delay when facing complex and changeable application loads is overcome, and a large number of experiments prove that the method has the advantages of improving the overall throughput of a system and reducing the average turnover time of tasks. The method is obviously superior to the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer resource management technology, and in particular to a resource perception prediction and allocation method based on deep reinforcement learning and applied to server clusters or data centers. Background Art

[0002] Resource-aware prediction and allocation is a key technical challenge in data centers and cloud computing. The goal is to dynamically and intelligently make optimal resource allocation decisions based on the real-time, multi-dimensional operational status of server clusters (such as CPU / GPU utilization, memory usage, network load, and pending task queues), thereby achieving one or more goals: maximizing overall system throughput, minimizing average task turnaround time, optimizing resource utilization, and minimizing energy consumption.

[0003] This task is a typical sequential decision-making problem, and reinforcement learning (RL) provides an effective solution to this type of dynamic decision-making problem. In the reinforcement learning framework, an agent learns through interaction with the environment. At each time step, the agent observes the state of the environment and selects an action to execute. The environment then feeds back a reward to the agent and moves to a new state. The agent's goal is to learn an optimal policy to maximize the long-term cumulative reward it obtains from the environment. Q-learning is a classic algorithm in reinforcement learning. Its core is to learn an action-value function Q(s,a) to evaluate the expected reward of performing action a in state s.

[0004] At present, the technical solutions in this field can be mainly divided into the following categories:

[0005] 1. Static allocation strategies based on fixed rules: This is the most traditional approach, such as round-robin, first-come, first-served (FCFS), and shortest job first (SJF). These approaches offer the advantages of simple implementation and low computational overhead. However, they are unaware of dynamic changes in system load. When faced with sudden or complex application requests, they often lead to uneven resource allocation, low utilization, or long wait times for some tasks.

[0006] 2. Scheduling methods based on traditional machine learning or heuristic algorithms: These methods attempt to predict future system loads using more complex models and use them as a basis for resource allocation. Examples include time series analysis, support vector machines (SVMs), or genetic algorithms. These methods offer some improvement over static strategies, but they typically require offline training, rely heavily on historical data, and have limited model generalization capabilities, making them difficult to adapt to rapidly changing system conditions in real time. Furthermore, their computational complexity is relatively high.

[0007] 3. Dynamic scheduling methods based on standard Q-learning and its variants (such as DQN): With the development of deep learning, deep reinforcement learning (DRL) has been introduced into the field of resource scheduling. Among them, the Deep Q-Network (DQN) [1] uses deep neural networks to approximate complex Q-value functions, enabling the model to handle high-dimensional state spaces. Since then, researchers have further explored the application of DRL in resource management, such as using DRL for job scheduling and resource allocation [3], or building a hierarchical framework to simultaneously manage resource allocation and power consumption [4]. These works make it possible to make end-to-end dynamic decisions from the original state to the optimal action.

[0008] However, the above-mentioned existing technical solutions, especially the method of directly using standard DQN, face major challenges in dealing with complex server resource allocation problems. The core problem lies in the overestimation of Q-values ​​[2]. In DQN, the network used to calculate the target Q value (target network) and the network used to select the optimal action are the same. In the maximization operation, noise and inaccurate Q-value estimates can be easily amplified, resulting in the calculated target Q value being systematically higher than the true value. This overestimation will mislead the learning direction of the model, causing it to tend to choose non-optimal resource allocation strategies, resulting in unstable training process and difficulty in converging to the optimal solution. Ultimately, this defect makes it difficult for the scheduler based on standard DQN to achieve ideal performance in practical applications [5], and there are still problems such as the inability to further improve resource utilization and high task response delay. Summary of the Invention

[0009] The present application is to solve the problems existing in the prior art, that is, the model training instability and the non-optimal strategy caused by the overestimation of Q-values of the standard deep Q network (DQN) in resource allocation decision, so as to propose a more efficient and stable resource-aware prediction and allocation method based on deep reinforcement learning applied to server clusters or data centers. In the present application, by adopting a double deep Q network structure, the action selection and action evaluation in the target Q value calculation are decoupled, effectively solving the decision instability and convergence difficulty problems caused by the overestimation of Q values in the traditional Q learning algorithm, and realizing more accurate and efficient resource allocation; by introducing a dynamic adaptive decision mechanism, the limitations of low resource utilization and high response delay of the traditional static or rule-based allocation strategy in the face of complex and variable application load are overcome, and a large number of experiments prove that the present application is significantly superior to the prior art method in improving the overall throughput of the system and reducing the average turnaround time of tasks.

[0010] The technical scheme of the present application is specifically introduced as follows.

[0011] A resource-aware prediction and allocation method based on deep reinforcement learning, comprising the following continuously executed steps:

[0012] Step one, system state perception stage

[0013] Real-time collection of running data of the server cluster, construction of a multi-dimensional state vector to comprehensively describe the current system load and resource usage;

[0014] Step two, optimal allocation strategy selection stage

[0015] The state vector is input into a pre-trained double deep Q network (DDQN) model, which is composed of a policy network and a target network. The policy network and the target network are two deep neural networks with the same structure and independent parameters. By decoupling action selection and value evaluation, the resource allocation action that can maximize the expected cumulative reward in the current state is calculated;

[0016] Step three, policy execution and model optimization stage

[0017] The selected resource allocation action is executed, and an immediate reward value is calculated according to the task completion time, resource utilization and system energy consumption indicators. The parameters of the policy network and the target network are updated using the reward value and the new state of the system, so as to realize the continuous online optimization of the model. After the model is optimized, the final output result is the resource-aware prediction and allocation result.

[0018] In the present application, in step one, the construction method of the multi-dimensional state vector St at time step t is as follows:

[0019] St=[Ct,Gt,Mt,QLt,T1,...,T n ]

[0020] Among them, Ct represents the current average CPU usage, Gt represents the current average GPU usage, Mt represents the current memory usage, QLt represents the length of the pending task queue, T1,...,T n Represents the key attributes of the first n pending tasks in the queue, including task type, expected computational effort, and memory requirements.

[0021] In the present invention, in step 2, the method for selecting the optimal allocation strategy is as follows:

[0022] First, the policy network Q(S, a; θ) is used to select the action a* with the maximum Q value from all possible resource allocation actions a at the next state S{t+1}. The process is expressed as:

[0023] a*=argmax a Q(S{t+1},a;θ) (1)

[0024] Among them, θ is the parameter of the policy network;

[0025] Then, the target network Q(S,a;θ′) is used to evaluate the value of performing action a* at the next state S{t+1}, and the target Q value Yt is calculated in combination with the immediate reward R{t+1}. The process is expressed as:

[0026] Yt=R{t+1}+γ*Q(S{t+1},a*;θ′) (2)

[0027] Where θ′ is the parameter of the target network and γ is the discount factor used to balance long-term and short-term rewards;

[0028] By separating the action selection in formula (1) from the value evaluation in formula (2), overestimation of the Q value is avoided.

[0029] In the present invention, in step 3, the parameters θ of the policy network are updated by minimizing the following loss function L(θ):

[0030] L(θ)=E[(Yt-Q(St,At;θ))^2]

[0031] Where Q(St,At;θ) is the Q value predicted by the policy network for the action At executed in the current state St, and Yt is the target Q value calculated according to formula (2).

[0032] In the present invention, in steps 2 and 3, the calculation method of the instant reward Rt is as follows:

[0033] R{t+1} = w1*(1 / Tc) + w2*Ur - w3*Pe

[0034] Where Tc is the task completion time, Ur is the average utilization of critical resources, and Pe is the total energy consumption of the system; w1, w2, and w3 are preset weight coefficients for balancing different optimization objectives for task completion speed, resource utilization efficiency, and energy consumption control.

[0035] In the present application, the method further comprises an experience replay mechanism.

[0036] The state transition tuple (St, At, R{t+1}, S{t+1}) generated in each time step is stored in an experience replay pool; during training, a small batch of transition tuples are randomly extracted from the pool to update the network parameters, breaking the correlation between data samples and improving the stability and efficiency of training.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] 1. Solving the problem of overestimation of Q value, improving the accuracy of decision-making: the present application adopts a double deep Q network architecture to decouple action selection and value evaluation, which are performed by the policy network and the target network respectively. This mechanism fundamentally suppresses the problem of continuously high target Q value caused by the maximization operation in the standard DQN algorithm, making the model's value evaluation of resource allocation actions closer to the real situation, and thus making more accurate and optimal allocation decisions.

[0039] 2. Improving the convergence speed and stability of the model: since the misleading update signal caused by overestimation of Q value is eliminated, the framework proposed by the present application shows higher stability during training. The loss function decreases more smoothly, and the model can converge to a high-quality policy more quickly, reducing the sensitivity to hyperparameter tuning and enhancing the robustness and practicality of the method.

[0040] 3. Significantly improving system-level performance indicators: based on a more accurate decision-making model, the present application can more effectively allocate computing and storage resources in practical applications. A large number of simulation experiments prove that compared with scheduling methods based on fixed rules or standard DQN, the present application can significantly reduce the average turnaround time and waiting time of tasks in a high-load and dynamic environment, while effectively improving the overall utilization of critical resources such as CPU and GPU, achieving comprehensive optimization of system performance.

[0041] The application can perceive the multi-dimensional system states such as CPU, GPU, memory usage and task queue in the server cluster in real time, evaluate the current state by using a double deep Q network (DDQN) model, and continuously optimize the allocation decision model by using a reward signal quantified according to the key performance indicators such as task completion time, resource utilization and system energy consumption, so as to dynamically select and execute the optimal resource allocation strategy. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 : Schematic diagram of the basic framework of reinforcement learning.

[0043] Figure 2 : The resource allocation system architecture based on double deep Q network (DDQN) proposed by the application.

[0044] Figure 3 : Comparison bar chart of the average task turnaround time of the method of the application under different loads.

[0045] Figure 4 : Comparison bar chart of the average resource utilization of the method of the application under different loads. DETAILED DESCRIPTION

[0046] The technical solutions of the application will be described in detail below with reference to the drawings and examples.

[0047] The existing technical solutions, such as the method using standard deep Q network (DQN), although introduce reinforcement learning into the resource scheduling field, but in practical application, it is difficult to effectively solve the inherent defect of overestimation of Q value. This defect leads to unstable model training, and the resource allocation strategy learned finally is not optimal.

[0048] The application is to solve the above problems existing in the prior art, that is, the decision is not accurate and the performance bottleneck caused by overestimation of Q value, and thus a more accurate, more efficient and more stable dynamic resource allocation framework is proposed.

[0049] In order to fundamentally solve the problem of overestimation of Q value, the application introduces the core idea of double deep Q network (Double DQN), the key of which is to decouple the two links of "selection of the optimal action of the next state" and "evaluation of the value of the action" coupled together in the traditional DQN. The application executes the two tasks by setting two independent neural networks, a policy network (Policy Network) for decision and a target network (Target Network) for evaluation.

[0050] As Figure 1 shown, the overall framework of the present application takes an intelligent agent as the core. The agent interacts with the server cluster environment through a policy network. At each decision cycle t, the agent first perceives the state St from the environment, and then uses the policy network to select an action At (i.e. a specific resource allocation scheme) and execute it. After the action is executed, the environment moves to a new state S{t+1} and feeds back an immediate reward R{t+1} to the agent. This complete interaction experience tuple (St, At, R{t+1}, S{t+1}) is stored in an experience replay buffer.

[0051] The key to model optimization is how to use these experiences to train the network. Instead of directly using the maximum Q value of the next state as the update target, the present application completes it through the cooperation of the policy network and the target network.

[0052] The present application provides a resource perception prediction and allocation method based on deep reinforcement learning, which comprises the following continuously executed steps:

[0053] Step one, system state perception stage

[0054] Real-time collection of server cluster running data to construct a multi-dimensional state vector St to comprehensively describe the current system load and resource usage. The construction method of the state vector St at time step t is as follows: St = [Ct, Gt, Mt, QLt, T1,..., T n ]. Among them, Ct, Gt, Mt represent the current average usage rate of CPU, GPU and memory respectively, QLt represents the length of the task queue to be processed, T1,..., T n represent the attributes of key tasks in the queue, which include task type, expected computation amount and memory requirement.

[0055] Step two, optimal allocation strategy selection stage

[0056] The state vector St is input into a double deep Q network (Double DQN) model. The innovation of this model lies in that it decouples the two processes of action selection and action value evaluation in the traditional DQN, thereby inhibiting the overestimation of Q value. The specific implementation method is as follows:

[0057] First, a policy network (Policy Network) with parameters θ is used to select the action a* with the maximum estimated value in the next state S{t+1} from all possible resource allocation action sets A according to its understanding of the current state St. This process is only used to select actions:

[0058] a*=argmax a Q(S{t+1},a;θ) (1)

[0059] Then, a separate target network (Target Network) with parameters θ′ (usually an earlier version of the policy network θ, which performs a "hard update" of θ′←θ every fixed training cycle C) is used to evaluate the value of the action a*. Combined with the immediate reward R{t+1} obtained from the environment, the target Q value Yt is calculated. This process is only used to evaluate the value:

[0060] Yt=R{t+1}+γ*Q(S{t+1},argmax a Q(S{t+1},a;θ);θ′) (2)

[0061] Here, γ is a discount factor that is used to balance the importance of short-term rewards and long-term rewards.

[0062] Step 3: Strategy execution and model optimization

[0063] The system executes the optimal allocation action At = a* selected in step 2. Then, based on the changes in system state and the execution results of the task (such as task completion time Tc, resource utilization Ur, and system energy consumption Pe), the immediate reward R{t+1} of this decision is calculated.

[0064] Finally, using the target Q value Yt calculated according to formula (2), the policy network parameters θ are updated by minimizing the following loss function, so that the predicted Q value Q(St,At;θ) continuously approaches the more accurate target Q value Yt:

[0065] L(θ)=E[(Yt-Q(St,At;θ))^2] (3)

[0066] At the same time, the parameters θ′ of the target network are periodically or smoothly updated with the parameters θ of the policy network. This process is usually supported by an experience replay pool to improve the stability of training.

[0067] The specific implementation details of the present invention are as follows:

[0068] 1. Neural Network Structure

[0069] The policy network and target network used in the present invention are two deep neural networks with identical structures but independent parameters. They are used together to approximate the action-value function Q(S,A). In a specific embodiment, the network structure can be designed as follows:

[0070] Input Layer: The input layer receives a state vector St of dimension d. The number of neurons in the input layer, d, is determined by the composition of the state vector. For example, if the state vector contains 1 CPU utilization, 1 GPU utilization, 1 memory utilization, 1 queue length, and the attributes of the first n = 5 tasks in the queue (assuming each task is described by 3 key attributes), then the dimension d of the input layer is 1 + 1 + 1 + 1 + (5 * 3) = 19.

[0071] Hidden Layers: The state vector enters multiple fully connected hidden layers from the input layer for nonlinear feature extraction. In a typical embodiment, two or three hidden layers can be set. For example, in a two-layer structure, the first hidden layer can be set to 256 neurons and the second hidden layer can be set to 128 neurons. Each hidden layer uses the Rectified Linear Unit (ReLU) as the activation function because it can effectively prevent the vanishing gradient problem and accelerate model convergence.

[0072] Output Layer: The output of the hidden layer is fed into a fully connected output layer. The number of neurons in the output layer is m, where m is the total number of possible discrete actions in the entire action space (for example, each combination of assigning task i to server j is a separate action). This layer uses a linear activation function (LinearActivation) because the Q value is a direct estimate of the reward and has an unrestricted range of values ​​and should not be constrained by functions such as Sigmoid or Softmax. Each value in the output vector corresponds to the predicted Q value of executing the corresponding action At in the current state St.

[0073] In summary, the policy network's parameters are θ (i.e., the network's weights and biases), which are updated in real time via gradient descent at each training step and are responsible for selecting actions based on the current state understanding. The target network's parameters are θ′, which are not updated in real time but are copied periodically (every C steps) or smoothly from the policy network's parameters θ, providing a relatively stable, non-volatile value assessment metric during training.

[0074] 2. Core Mechanism: Decoupled Q-Value Target Calculation

[0075] When extracting a batch of experiences (S, A, R, S') from the experience replay pool for training, the method of calculating the target Q value Yt of the present invention is completely different from that of the standard DQN.

[0076] The target calculation of the standard DQN is: Yt_DQN = R + γ*max aQ(S, a; θ)

[0077] The problem with this formula is that both the action that selects the maximum Q value (max a ) and the network that computes this Q value (Q) use the same parameters θ, which directly leads to overestimation.

[0078] The present invention computes the target Q value Yt in two steps, as shown in Figure 2 :

[0079] Step 1: Action selection. Use the current policy network (parameters θ) to find the action a* that gives the maximum Q value in the next state S'. This step only determines the action, but does not use its corresponding Q value.

[0080] a* = argmax a Q(S', a; θ)

[0081] Step 2: Value evaluation. Use an independent, more stable target network (parameters θ') to evaluate the value of performing action a* in state S'. Use this value as part of the target Q value.

[0082] Yt = R + γ * Q(S', a*; θ')

[0083] In this way, the selection of actions and the evaluation of values are effectively decoupled, avoiding systematic bias caused by self-evaluation by the same network.

[0084] 3. Network training and updating

[0085] The parameters θ of the policy network are updated by minimizing the mean squared error (MSE) loss function between the predicted Q value and the target Q value:

[0086] L(θ) = E[(Yt - Q(S, A; θ))^2]

[0087] Where Yt is the more accurate target Q value calculated in the previous step. Training uses optimizers such as Adam for gradient descent.

[0088] After several steps of training, the parameters θ' of the target network will be completely overwritten by the parameters θ of the policy network, i.e. perform θ' <- θ. This regular update ensures that the target network provides a stable evaluation standard while keeping up with the learning process of the policy network.

[0089] Example 1

[0090] To verify the superiority of the method described in the present invention over the prior art, we built a simulation environment and conducted comparative experiments.

[0091] Experimental Environment Setup: The simulation environment simulates a data center with 20 servers, each with fixed CPU, GPU, and memory resources. Task flows are randomly generated using a Poisson distribution, with each task having random resource requirements and execution durations to simulate the complex and variable load conditions found in the real world.

[0092] Comparison method:

[0093] 1. FCFS (First Come First Served): A classic rule-based baseline approach.

[0094] 2. Standard DQN: A reinforcement learning method using a standard deep Q network.

[0095] 3. The method of the present invention: adopts the resource allocation method based on the dual-depth Q network described in detail in this article.

[0096] Evaluation Metrics:

[0097] 1. Average Task Turnaround Time (Tavg): The average time from task submission to completion. The lower the better.

[0098] 2. Average resource utilization (Uavg): The average usage percentage of key resources such as CPU and GPU. The higher the better.

[0099] 3. Cumulative Reward (Reward_acc): The cumulative reward value obtained by the reinforcement learning agent during the training process. The higher the reward, the better the strategy.

[0100] Experimental results:

[0101] During the training process, the cumulative reward curve of the method of the present invention not only converges to a higher level than the standard DQN, but also has smaller fluctuations during the process, proving that its learning process is more stable and the learned strategy is better.

[0102] Table 1 shows the comparison of average performance indicators after running 10,000 tasks in the simulation environment.

[0103] Table 1: Performance comparison between the present invention and existing disclosed technical methods

[0104] Technical approach Task average turnaround time (seconds) Average resource utilization (%) FCFS 152.4 58.2 Standard DQN 95.7 81.5 The invention 78.3 92.1

[0105] From Table 1, Figure 3 and Figure 4It can be seen that the method of the present invention significantly outperforms the comparison method in all key indicators. Compared with the traditional FCFS, the performance is improved several times. Compared with the standard DQN, which is also a reinforcement learning method, the present invention reduces the average task turnaround time by approximately 18.2%, while increasing the average resource utilization by more than 10 percentage points. The experimental results fully demonstrate that the present invention effectively solves the problem of overestimation of Q values ​​by decoupling action selection and value assessment, thereby being able to learn a better resource allocation strategy, significantly improving the operating efficiency and resource utilization level of complex systems such as data centers.

[0106] References

[0107] [1]Mnih, V., Kavukcuoglu, K., Silver, D., et al. Human-level control through deep reinforcement learning. Nature 518,529–533(2015).

[0108] [2]Van Hasselt,H.,Guez,A.,&Silver,D.(2016).Deep ReinforcementLearning with Double Q-learning.In AAAI Conference on ArtificialIntelligence.

[0109] [3]Mao, H., Alizadeh, M., Menache, I., & Kandula, S. Resource Management with Deep Reinforcement Learning. In Proceedings of the 15th ACM Workshop on HotTopicsin Networks (HotNets), 50–56 (2016).

[0110] [4] Liu, N., Li, Z., Xu, J., & Jin, H. AHierarchical Framework of CloudResource Allocation and Power Management Using Deep Reinforcement Learning. In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 372–382 (2017).

[0111] [5]Bao,W.,et al.ADeep-Reinforcement-Learning-Based Method forResource-Efficient and QoS-Aware Job Scheduling in Cloud Data Centers.FutureGeneration Computer Systems,139,137-148(2023)。

Claims

1. A resource perception prediction and allocation method based on deep reinforcement learning, characterized in that: It includes the following steps that are performed in succession: Step 1: System status perception stage Collect the running data of the server cluster in real time and build a multi-dimensional state vector to fully describe the current system load and resource usage; Step 2: Optimal Allocation Strategy Selection The state vector is input into a pre-trained Dual Deep Q Network (DDQN) model. By decoupling action selection and value evaluation, the resource allocation action that maximizes the expected cumulative reward under the current state is calculated. The Dual Deep Q Network (DDQN) model consists of a policy network and a target network. The policy network and the target network are deep neural networks with identical structures but independent parameters. Step 3: Strategy execution and model optimization stage Execute the selected resource allocation action and calculate an immediate reward value based on the task completion time, resource utilization and system energy consumption indicators; use this reward value and the new state of the system to update the parameters of the policy network and target network, thereby achieving continuous online optimization of the model. After the model is optimized, the final output result is the resource perception prediction and allocation result.

2. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 1 is characterized in that: In step 1, the multi-dimensional state vector St at time step t is constructed as follows: St=[Ct,Gt,Mt,QLt,T1,...,T n ] Among them, Ct represents the current average CPU usage, Gt represents the current average GPU usage, Mt represents the current memory usage, QLt represents the length of the pending task queue, T1,...,T n Represents the key attributes of the first n pending tasks in the queue, including task type, expected computational effort, and memory requirements.

3. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 1 is characterized in that: In step 2, the optimal allocation strategy is selected as follows: First, the policy network Q(S, a; θ) is used to select the action a* with the maximum Q value from all possible resource allocation actions a at the next state S{t+1}. The process is expressed as: a*=argmax a Q(S{t+1},a;θ) (1) Here, θ is the parameter of the policy network. Then, the target network Q(S,a;θ′) is used to evaluate the value of performing action a* at the next state S{t+1}, and the target Q value Yt is calculated in combination with the immediate reward R{t+1}. The process is expressed as: Yt=R{t+1}+γ*Q(S{t+1},a*;θ′) (2) Where θ′ is the parameter of the target network and γ is the discount factor used to balance long-term and short-term rewards; By separating the action selection in formula (1) from the value evaluation in formula (2), overestimation of the Q value is avoided.

4. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 3 is characterized in that: In step 3, the parameters θ of the policy network are updated by minimizing the following loss function L(θ): L(θ)=E[(Yt-Q(St,At;θ))^2] Where Q(St,At;θ) is the Q value predicted by the policy network for the action At executed in the current state St, and Yt is the target Q value calculated according to formula (2).

5. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 3 is characterized in that: In steps 2 and 3, the calculation method of the immediate reward R{t+1} is as follows: R{t+1}=w1*(1 / Tc)+w2*Ur-w3*Pe Among them, Tc is the task completion time, Ur is the average utilization rate of key resources, and Pe is the total energy consumption of the system; w1, w2, and w3 are preset weight coefficients used to balance different optimization goals of task completion speed, resource utilization efficiency, and energy consumption control.

6. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 1 is characterized in that: The method also includes an experience replay mechanism: The state transition tuple (St, At, R{t+1}, S{t+1}) generated in each time step is stored in an experience replay pool; during training, small batches of transition tuples are randomly extracted from the pool to update the network parameters to break the correlation between data samples and improve the stability and efficiency of training.

7. The resource perception prediction and allocation method based on deep reinforcement learning according to claim 1 is characterized in that: The network structure of the policy network and the target network includes an input layer, a hidden layer, and an output layer; the input layer is responsible for receiving the state vector St with a dimension of d. The number of neurons d in the input layer is determined by the composition of the state vector. The state vector enters multiple fully connected hidden layers from the input layer for nonlinear feature extraction; each hidden layer uses a rectified linear unit as the activation function, and the output of the hidden layer is sent to a fully connected output layer; the number of neurons in the output layer is m, where m is the total number of all possible discrete actions in the entire action space. This layer uses a linear activation function, and each value in the output vector corresponds to the predicted Q value of performing the corresponding action At under the current state St.

Images