Edge server dynamic activation method and system based on deep reinforcement learning

Abstract

The invention provides an edge server dynamic activation method and system based on deep reinforcement learning, and the method comprises the following steps: S1, building system models, including building a network model, an energy consumption model, a communication and service delay model and a state switching cost model, the network model including RES and MES; s2, optimizing a network model by calculating the sum of the cost of the energy consumption model, the communication and service delay model and the state switching cost model; s3, constructing a Markov decision process model, and improving RES stability; s4, predicting a traffic load; s5, calculating a baseline value; s6, training the network model through a centralized intelligent scheduling algorithm CDDS to obtain a trained strategy network model; s7, dividing the trained strategy network model through a federated distributed intelligent scheduling algorithm FDDS, and training the model based on federated learning to obtain a DDPG model; and S8, the DDPG model is deployed to each RES.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic activation of edge servers, and in particular to a method and system for dynamic activation of edge servers based on deep reinforcement learning. Background Art

[0002] With the rapid development of 6G communications and intelligent mobile terminals, Mobile Edge Computing (MEC), as an emerging computing architecture, is playing an increasingly important role in reducing communication latency and improving computing response efficiency. By deploying computing resources at the edge of the network close to terminal devices, MEC enables local processing of massive amounts of data, providing key support for scenarios such as industrial control, telemedicine, and intelligent manufacturing. However, the energy consumption of edge computing infrastructure is becoming increasingly prominent, especially in widely deployed edge servers. The system maintains a high load readiness state for extended periods, maintaining high energy consumption even during periods of sparse user requests or idle time, severely restricting the development of green computing.

[0003] Existing research has made initial progress in improving the energy efficiency of MEC systems, focusing primarily on task offloading optimization, resource scheduling, and network topology reconstruction. Some studies attempt to control the on / off switching of edge servers through static or empirical threshold-based methods, but these methods often lack sufficient awareness of the real-time environment and are unable to cope with the complex and changing user request traffic. Other studies focus on energy-saving strategies on the terminal side, ignoring energy consumption issues at the edge infrastructure level. Furthermore, traditional centralized scheduling strategies suffer from low efficiency and reliability when faced with large-scale heterogeneous deployments and inter-regional collaborative control. Untimely system responses and overly conservative or aggressive strategies often lead to reduced service quality or increased energy consumption.

[0004] Existing edge computing networks suffer from an imbalance between energy efficiency and service quality due to a mismatch between static resource deployment strategies and dynamic business demands. Existing research has largely focused on optimizing energy consumption in terminal devices, ignoring the energy-saving potential of edge infrastructure and lacking the ability to globally optimize the coordinated hibernation of multiple servers. Summary of the Invention

[0005] This invention addresses the aforementioned issues. Existing edge computing networks suffer from an imbalance between energy efficiency and service quality, caused by a mismatch between static resource deployment strategies and dynamic business demands. Existing research has primarily focused on optimizing the energy consumption of terminal devices, neglecting the energy-saving potential of edge infrastructure and lacking the ability to globally optimize the coordinated hibernation of multiple servers. This invention aims to provide a method and system for dynamic activation of edge servers based on deep reinforcement learning. This method, through intelligent state switching strategies, reduces energy consumption of edge servers while maintaining service quality, addressing the lack of adaptability of traditional methods in dynamic environments.

[0006] The present invention provides a method for dynamic activation of edge servers based on deep reinforcement learning, which has the following characteristics: specifically comprising the following steps: S1, constructing a system model, including constructing a network model, an energy consumption model, a communication and service delay model, and a state switching cost model, wherein the network model includes RES and MES; S2, optimizing the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model; S3, constructing a Markov decision process model to improve the stability of RES; step S3 includes the following sub-steps: S3-1, the state includes the traffic load of RES at time t and the working status of the previous cycle

[0007]

[0008] S3-2, action is the working mode of RES at time t

[0009]

[0010] S3-3, rewards are optimization targets and baseline values The difference between Improve the stability of the learning process,

[0011]

[0012] S4, predicted traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of S5, calculate baseline value According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm. S6, train the network model through the centralized intelligent scheduling algorithm CDDS to obtain the trained policy network model; S7, divide the trained policy network model through the federated distributed intelligent scheduling algorithm FDDS and obtain the DDPG model based on the federated learning training model; S8, deploy the DDPG model to each RES.

[0013] In the edge server dynamic activation method based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, step S1 includes the following sub-steps: S1-1 network model: construct an edge computing network consisting of a small base station and a macro base station, wherein: RES: an edge server deployed on the small base station, supporting dynamic activation / sleep, MES: an edge server deployed on the macro base station, always online, responsible for traffic takeover and model aggregation, each RES switches state according to the load in one cycle (t = 0.5 hours), and the network node sets are R = {1, 2, ..., n} and M = {1, 2, ..., m} respectively; S1-2 energy consumption model: the power consumption of each RES includes fixed power consumption and dynamic power consumption related to CPU utilization, wherein the activation power is

[0014]

[0015] The total power is

[0016]

[0017] in Represents the working state of RESi at time t, 1 is active, 0 is dormant, P sleep Indicates the fixed power in sleep state, P max Indicates the maximum power in working state, represents the server load factor (i.e., the CPU utilization at time t); S1-3, the communication and service delay model; S1-4, the state switching cost model. The server state switching between two cycles will generate energy consumption / migration / hardware wear overhead, which is defined as

[0018]

[0019] in represents the cost of switching from active state to dormant state, Indicates the cost of switching from dormant state to active state.

[0020] The edge server dynamic activation method based on deep reinforcement learning provided by the present invention may also have the following features: wherein step S1-3 includes the following sub-steps: S1-3-1, transmission delay: When the server is dormant or overloaded, the task will be forwarded to other RES or MES, and the transmission delay is:

[0021]

[0022] in is the task forwarding indicator variable, d(i,j) represents the geographical distance between servers, V i (t) Indicates the amount of data received by RES at time t, represents the electromagnetic wave propagation speed, θ ij represents the data transmission rate between servers; S1-3-2, service delay: the server adopts the M / M / c queuing model, the task arrival follows the Poisson distribution, the service time is exponentially distributed, and the service delay is:

[0023]

[0024] Where C(c,ρ) is the Erlang-C formula, which calculates the queuing probability.

[0025] The edge server dynamic activation method based on deep reinforcement learning provided by the present invention may also have the following features: wherein step S2 includes the following sub-steps: S2-1, the total system operation cost is defined as the sum of energy consumption, delay and switching cost,

[0026]

[0027] S2-2, the ultimate optimization goal is to minimize the total system operation cost

[0028]

[0029] The constraints include the binary nature of the server state and the service forwarding association matrix. The server's working state remains unchanged within a single cycle. The activated server service is associated with itself. Each dormant server is associated with at most one activated server. Other servers cannot be associated with dormant servers.

[0030] In the edge server dynamic activation method based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, step S6 includes the following sub-steps: S6-1, constructing a centralized deep reinforcement learning agent based on deep deterministic policy gradient DDPG; S6-2, constructing four networks: Actor, Critic and its target network; S6-3, Actor outputs action a t , and add noise to improve the agent's exploratory power; S6-4, Critic evaluates the Q value; S6-5, uses experience replay and soft update mechanisms to improve stability; S6-6, uses experience replay to update the Critic and Actor networks; S6-7, soft update of the target network; S6-8, obtains the trained policy network.

[0031] In the edge server dynamic activation method based on deep reinforcement learning provided by the present invention, it can also have the following characteristics: wherein, step S7 includes the following sub-steps: S7-1 distributed network division; step S7-1 includes the following sub-steps: S7-1-1 divides the network into K areas; S7-1-2, the representative RES of each area acts as a Worker to independently perceive the local state and train the local DDPG model; S7-2, based on federated learning training model; step S7-2 includes the following sub-steps: S7-2-1, all Workers initialize model parameters S7-2-2, Worker uses local data for training for several rounds and then uploads the model parameters Where o represents the number of iterations; S7-2-3, MES executes FedAvg as a federated server:

[0032]

[0033] Where |B i | indicates the sampling batch size; S7-2-4, MES sends the aggregated global parameters, and each Worker continues training; S7-2-5 supports asynchronous upload and parameter update to reduce communication overhead.

[0034] The edge server dynamic activation method based on deep reinforcement learning provided by the present invention may also have the following features: wherein, step S8 includes the following sub-steps: S8-1, deploying the trained DDPG model to each RES; S8-2, realizing autonomous decision-making and on-demand wake-up of the edge server according to actual environmental changes, and supporting online fine-tuning and model self-update.

[0035] The edge server dynamic activation method based on deep reinforcement learning provided by the present invention may also have the following features: S9, experimental verification, building a simulation environment based on an actual urban cellular network traffic data set, selecting 20 edge server nodes, with a time granularity of 0.5 hours, and setting server hardware parameters with reference to Huawei Atlas 500 edge device settings. The algorithm is trained and tested on the MATLAB platform, using unified parameter initialization to ensure reproducibility and fair comparison.

[0036] The present invention also provides an edge server dynamic activation system based on deep reinforcement learning, which has the following characteristics: a system modeling module, which constructs a system model, including constructing a network model, an energy consumption model, a communication and service delay model, and a state switching cost model, wherein the network model includes RES and MES; a target definition optimization module, which optimizes the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model; a Markov decision process modeling module, which constructs a Markov decision process model to improve the stability of RES, including the following steps: the state includes the traffic load of RES at time t and the working status of the previous cycle

[0037]

[0038] Action is the working mode of RES at time t

[0039]

[0040] The reward is the optimization target and the baseline value The difference between Improve the stability of the learning process,

[0041]

[0042] Traffic prediction module, predicting traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of Baseline value calculation module, calculates baseline value According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm. The centralized intelligent scheduling algorithm module trains the network through the centralized intelligent scheduling algorithm CDDS to obtain the trained policy network; the federated distributed intelligent scheduling algorithm module divides the trained policy network through the federated distributed intelligent scheduling algorithm FDDS and obtains the DDPG model based on the federated learning training model; the model deployment and inference module deploys the DDPG model to each RES.

[0043] Functions and effects of the invention

[0044] According to the method and system for dynamic activation of edge servers based on deep reinforcement learning involved in the present invention, the present invention significantly improves the energy efficiency performance of the mobile edge computing system by constructing a dynamic sleep decision mechanism for edge servers based on the fusion of deep reinforcement learning and federated learning. Compared with traditional static or rule-driven strategies, the proposed centralized and distributed intelligent scheduling algorithms can dynamically perceive the environmental status and autonomously learn the optimal server control strategy, effectively reducing system energy consumption and maintenance costs while ensuring service quality. Experimental results show that the scheme can achieve a significant reduction in the total cost of the system on actual data sets, has good generalization capabilities, deployment flexibility and engineering practical value, and provides a practical technical path for building a green, efficient and intelligent edge computing architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flowchart of the federated distributed intelligent scheduling algorithm FDDS in an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the algorithm convergence results in experimental verification in an embodiment of the present invention; and

[0047] Figure 3 This is a comparison of the algorithm in the experimental verification of the embodiment of the present invention with the comparison algorithm in terms of total system cost. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the edge server dynamic activation method and system based on deep reinforcement learning of the present invention.

[0049] The present invention aims to design an intelligent edge server operating status control mechanism for dynamic traffic perception. The core goal is to significantly improve the energy efficiency of edge computing systems without sacrificing the system's Quality of Service (QoS). With the help of artificial intelligence methods, especially Deep Reinforcement Learning (DRL) technology with adaptive learning and policy optimization capabilities, dynamic wake-up and sleep control of edge servers is achieved. At the same time, in response to the scalability and privacy protection issues brought about by distributed deployment, an algorithm framework integrated with Federated Learning (FL) is explored to collaboratively learn the optimal control strategy in multiple heterogeneous regions.

[0050] This paper focuses on the core issue of "dynamic sleep and activation decision optimization of edge servers" and builds a complete system modeling framework, covering multiple key modules such as network architecture, task traffic prediction, energy consumption and delay modeling, and server state switching cost modeling. The optimization problem is formalized as a Markov decision process, and then two scheduling algorithms are designed for different deployment scenarios: one is a centralized DRL decision algorithm (Centralized DDPG-based DynamicSleep, CDDS), and the other is a distributed DRL algorithm based on the federated learning mechanism (FederatedDRL-basedDynamicSleep, FDDS). The two types of algorithms can combine real-time perception information and historical experience to dynamically learn and output the optimal operating state decision of the edge server, realizing green, efficient, and scalable edge server resource management.

[0051] The present invention discloses a method and system for dynamic activation of an edge server based on deep reinforcement learning, which specifically includes the following steps:

[0052] S1, build the system model, including building the network model, energy consumption model, communication and service delay model and state switching cost model. The network model includes RES and MES.

[0053] Step S1 includes the following sub-steps:

[0054] S1-1 Network Model: Build an edge computing network consisting of small base stations and macro base stations, where:

[0055] RES: Edge server deployed in small base stations, supporting dynamic activation / sleep.

[0056] MES: An edge server deployed at a macro base station, always online, responsible for traffic takeover and model aggregation.

[0057] Each RES switches state according to the load in a period (t=0.5 hours), and the network node sets are R={1,2,...,n} and M={1,2,...,m} respectively.

[0058] S1-2 Energy Consumption Model: The power consumption of each RES includes fixed power consumption and dynamic power consumption related to CPU utilization, where the activation power is

[0059]

[0060] The total power is

[0061]

[0062] in Represents the working state of RESi at time t, 1 is active, 0 is dormant, P sleep Indicates the fixed power in sleep state, P max Indicates the maximum power in working state, Indicates the server load factor (i.e., CPU utilization at time t).

[0063] S1-3, Communication and service delay model.

[0064] Step S1-3 includes the following sub-steps:

[0065] S1-3-1, transmission delay: When the server is dormant or overloaded, the task will be forwarded to other RES or MES. The transmission delay is:

[0066]

[0067] in is the task forwarding indicator variable, d(i,j) represents the geographical distance between servers, V i (t) Indicates the amount of data received by RES at time t, represents the electromagnetic wave propagation speed, θ ij Indicates the data transfer rate between servers.

[0068] S1-3-2, service delay: The server adopts the M / M / c queuing model, task arrival follows the Poisson distribution, service time is exponentially distributed, and service delay is:

[0069]

[0070] Where C(c,ρ) is the Erlang-C formula, which calculates the queuing probability.

[0071] S1-4, state switching cost model.

[0072] The server state switching between two cycles will generate energy consumption / migration / hardware wear overhead, defined as

[0073]

[0074]

[0075] in represents the cost of switching from active state to dormant state, Indicates the cost of switching from dormant state to active state.

[0076] S2 optimizes the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model.

[0077] Step S2 includes the following sub-steps:

[0078] S2-1, the total system operation cost is defined as the sum of energy consumption, delay and switching cost.

[0079]

[0080] S2-2, the final optimization goal is to minimize the total system operation cost.

[0081]

[0082] The constraints include the binary nature of the server state and the service forwarding association matrix. The server's working state remains unchanged within a single cycle. The activated server service is associated with itself. Each dormant server is associated with at most one activated server. Other servers cannot be associated with dormant servers.

[0083] S3, constructs a Markov decision process model to improve the stability of RES.

[0084] Step S3 includes the following sub-steps:

[0085] S3-1, the state includes the traffic load of RES at time t and the working status of the previous cycle

[0086]

[0087] S3-2, action is the working mode of RES at time t

[0088]

[0089] S3-3, rewards are optimization targets and baseline values The difference between the two, where the baseline value is introduced The reason is that the fluctuation of traffic over time will cause the fluctuation of the objective function, but the deep reinforcement learning agent cannot distinguish whether the fluctuation is caused by changes in the environment or the quality of the strategy. Therefore, in order to reduce the impact of environmental changes, we introduce Improve the stability of the learning process,

[0090]

[0091] S4, predicted traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of

[0092] S5, calculate baseline value According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm.

[0093] S6, train the network model through the centralized intelligent scheduling algorithm CDDS to obtain the trained policy network model.

[0094] Step S6 includes the following sub-steps:

[0095] S6-1, Building a centralized deep reinforcement learning agent based on deep deterministic policy gradient DDPG.

[0096] S6-2, build four networks: Actor, Critic and its target network.

[0097] S6-3, Actor output action a t , and adding noise improves the agent's exploration.

[0098] S6-4, Critic evaluates Q value.

[0099] S6-5, use experience replay and soft update mechanisms to improve stability.

[0100] S6-6, using experience replay to update the Critic and Actor networks.

[0101] S6-7, target network soft update.

[0102] S6-8, obtain the trained policy network.

[0103] Figure 1 It is a flow chart of the federated distributed intelligent scheduling algorithm FDDS in an embodiment of the present invention.

[0104] S7, such as Figure 1 As shown in the figure, the trained policy network model is divided through the federated distributed intelligent scheduling algorithm FDDS and the DDPG model is obtained based on the federated learning training model.

[0105] Step S7 includes the following sub-steps:

[0106] S7-1 distributed network division.

[0107] Step S7-1 includes the following sub-steps:

[0108] S7-1-1 divides the network into K areas.

[0109] In S7-1-2, the representative RES of each region acts as a Worker to independently perceive the local state and train the local DDPG model.

[0110] S7-2, based on federated learning training model.

[0111] Step S7-2 includes the following sub-steps:

[0112] S7-2-1, all workers initialize model parameters

[0113] S7-2-2, Worker uses local data for training for several rounds and then uploads the model parameters Where o represents the number of iterations.

[0114] S7-2-3, MES executes FedAvg as a federation server:

[0115]

[0116] Where |B i | represents the batch size of sampling.

[0117] In S7-2-4, MES sends the aggregated global parameters and each worker continues training.

[0118] S7-2-5, supports asynchronous upload and parameter update to reduce communication overhead.

[0119] S8, deploy the DDPG model to each RES.

[0120] Step S8 includes the following sub-steps:

[0121] S8-1, deploy the trained DDPG model to each RES.

[0122] S8-2 enables edge servers to make autonomous decisions and wake up on demand based on actual environmental changes, and supports online fine-tuning and model self-updates.

[0123] S9,Experimental Verification,A simulation environment is built based on the actual urban cellular network traffic dataset.,Twenty edge server nodes are selected with a time granularity of 0.5 hours.,The server hardware parameters are set with reference to Huawei Atlas 500 edge,devices. The algorithm is trained and tested on the MATLAB platform,using a unified parameter initialization to ensure reproducibility and,fair comparison.

[0124] Algorithm convergence:

[0125] Figure 2 It is a schematic diagram of the algorithm convergence results in the experimental verification in the embodiment of the present invention.

[0126] Among them, PPO and REINFORCE are contrast algorithms.

[0127] like Figure 2 As shown in the figure, by comparing the training curves of different DRL algorithms, the CDDS and FDDS algorithms proposed in this invention both show good convergence performance and can converge to a stable strategy within a relatively small number of training rounds. FDDS has a faster convergence speed than CDDS, which verifies the advantage of the federated training mechanism in improving training efficiency.

[0128] Performance improvement effect:

[0129] Figure 3 This is a comparison of the total system cost of the algorithm in the experimental verification of the embodiments of the present invention with the comparison algorithm. TBCP stands for Traffic-Blind Control Policy, TTP stands for Two-Threshold Policy, DPM stands for Dynamic Partitioning Method, RSP stands for Random Sleep Policy, and No-sleep stands for No Sleep Policy.

[0130] like Figure 3 Experimental results show that CDDS and FDDS reduce the total system cost by approximately 11.06% to 45.36% and 8.71% to 43.77% respectively compared with the comparison algorithms, and show significant advantages in dimensions such as energy consumption, resource utilization and task response delay, fully demonstrating the comprehensive benefits of the present invention in green computing, service quality assurance and intelligent decision-making capabilities.

[0131] The present invention also provides an edge server dynamic activation system based on deep reinforcement learning, comprising:

[0132] The system modeling module builds a system model according to the method of step S1 above, including building a network model, an energy consumption model, a communication and service delay model, and a state switching cost model. The network model includes RES and MES.

[0133] The target definition optimization module optimizes the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model according to the method of step S2 above.

[0134] The Markov decision process modeling module builds a Markov decision process model according to the method of step S3 above to improve the stability of RES, including the following steps:

[0135] The state includes the traffic load of RES at time t and the working status of the previous cycle

[0136]

[0137] Action is the working mode of RES at time t

[0138]

[0139] The reward is the optimization target and the baseline value The difference between Improve the stability of the learning process,

[0140]

[0141] Traffic prediction module, according to the method of step S4 above, predicts the traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of

[0142] The baseline value calculation module calculates the baseline value according to the method of step S5 above According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm.

[0143] The centralized intelligent scheduling algorithm module, according to the method of step S6 above, trains the network through the centralized intelligent scheduling algorithm CDDS to obtain a trained policy network.

[0144] The federated distributed intelligent scheduling algorithm module, according to the method of step S7 above, uses the federated distributed intelligent scheduling algorithm FDDS to divide the trained policy network and obtain the DDPG model based on the federated learning training model.

[0145] The model deployment and inference module deploys the DDPG model to each RES according to the method of step S8 above.

[0146] Those skilled in the art will appreciate that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic activation of edge servers based on deep reinforcement learning, characterized in that: The specific steps include: S1, building a system model, including building a network model, an energy consumption model, a communication and service delay model, and a state switching cost model. The network model includes RES and MES; S2, optimizing the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model; S3, constructing a Markov decision process model to improve the stability of the RES; Step S3 includes the following sub-steps: S3-1, the state includes the traffic load of RES at time t and the working status of the previous cycle S3-2, action is the working mode of RES at time t S3-3, rewards are optimization targets and baseline values The difference between Improve the stability of the learning process, S4, predicting the traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of S5, calculating the baseline value According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm. S6, training the network model using a centralized intelligent scheduling algorithm CDDS to obtain a trained policy network model; S7, using a federated distributed intelligent scheduling algorithm FDDS, partitioning the trained policy network model and obtaining a DDPG model based on a federated learning training model; S8, deploying the DDPG model to each RES.

2. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: in, The step S1 includes the following sub-steps: S1-1 Network Model: Build an edge computing network consisting of small base stations and macro base stations, where: RES: Edge server deployed in small base stations, supporting dynamic activation / sleep, MES: An edge server deployed at a macro base station, always online, responsible for traffic takeover and model aggregation. Each RES switches state according to the load in a period (t = 0.5 hours), and the network node sets are R = {1, 2, ..., n} and M = {1, 2, ..., m}; S1-2 Energy Consumption Model: The power consumption of each RES includes fixed power consumption and dynamic power consumption related to CPU utilization, where the activation power is P active (t)=P sleep +(P max -P sleep )×ρ i R(t) The total power is in Represents the working state of RESi at time t, 1 is active, 0 is dormant, P sleep Indicates the fixed power in sleep state, P max Indicates the maximum power in working state, Indicates the server load factor (i.e., CPU utilization at time t); S1-3, communication and service delay model; S1-4, state switching cost model, The server state switching between two cycles will generate energy consumption / migration / hardware wear overhead, defined as in represents the cost of switching from active state to dormant state, Indicates the cost of switching from dormant state to active state.

3. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 2, characterized in that: in, The step S1-3 includes the following sub-steps: S1-3-1, transmission delay: When the server is dormant or overloaded, the task will be forwarded to other RES or MES. The transmission delay is: in is the task forwarding indicator variable, d(i,j) represents the geographical distance between servers, V i (t) Indicates the amount of data received by RES at time t, represents the electromagnetic wave propagation speed, θ ij Indicates the data transmission rate between servers; S1-3-2, service delay: The server adopts the M / M / c queuing model, task arrival follows the Poisson distribution, service time is exponentially distributed, and service delay is: Where C(c,ρ) is the Erlang-C formula, which calculates the queuing probability.

4. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: in, The step S2 includes the following sub-steps: S2-1, the total system operation cost is defined as the sum of energy consumption, delay and switching cost, S2-2, the ultimate optimization goal is to minimize the total system operation cost The constraints include the binary nature of the server state and the service forwarding association matrix. The server's working state remains unchanged within a single cycle. The activated server service is associated with itself. Each dormant server is associated with at most one activated server. Other servers cannot be associated with dormant servers.

5. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: in, The step S6 includes the following sub-steps: S6-1, Building a centralized deep reinforcement learning agent based on deep deterministic policy gradient DDPG; S6-2, build four networks: Actor, Critic and their target network; S6-3, Actor output action a t , and add noise to improve the agent's exploration ability; S6-4, Critic evaluation Q value; S6-5, using experience replay and soft update mechanisms to improve stability; S6-6, using experience replay to update the Critic and Actor networks; S6-7, target network soft update; S6-8, obtain the trained policy network.

6. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: in, The step S7 includes the following sub-steps: S7-1 distributed network division; Step S7-1 includes the following sub-steps: S7-1-1 divides the network into K areas; In S7-1-2, the representative RES of each region acts as a worker to independently perceive the local state and train the local DDPG model; S7-2, based on federated learning training model; Step S7-2 includes the following sub-steps: S7-2-1, all workers initialize model parameters S7-2-2, Worker uses local data for training for several rounds and then uploads the model parameters Where o represents the number of iterations; S7-2-3, MES executes FedAvg as a federation server: Where |B i | represents the batch size of sampling; S7-2-4, MES sends the aggregated global parameters, and each worker continues training; S7-2-5, supports asynchronous upload and parameter update to reduce communication overhead.

7. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: in, The step S8 includes the following sub-steps: S8-1, deploy the trained DDPG model to each RES; S8-2 enables edge servers to make autonomous decisions and wake up on demand based on actual environmental changes, and supports online fine-tuning and model self-updates.

8. The method for dynamic activation of edge servers based on deep reinforcement learning according to claim 1, characterized in that: Also includes: S9,Experimental Verification,A simulation environment is built based on the actual urban cellular network traffic dataset.,Twenty edge server nodes are selected with a time granularity of 0.5 hours.,The server hardware parameters are set with reference to Huawei Atlas 500 edge,devices. The algorithm is trained and tested on the MATLAB platform,using a unified parameter initialization to ensure reproducibility and,fair comparison.

9. The edge server dynamic activation system based on deep reinforcement learning according to claim 1, characterized in that: include: System modeling module, building a system model, including building a network model, energy consumption model, communication and service delay model, and state switching cost model. The network model includes RES and MES; a target definition optimization module, which optimizes the network model by calculating the sum of the costs of the energy consumption model, the communication and service delay model, and the state switching cost model; The Markov decision process modeling module constructs a Markov decision process model to improve the stability of the RES, including the following steps: The state includes the traffic load of RES at time t and the working status of the previous cycle Action is the working mode of RES at time t The reward is the optimization target and the baseline value The difference between Improve the stability of the learning process, Traffic prediction module, predicting the traffic load At the beginning of time slot t, the amount of tasks requested by RES in this time slot has not yet been obtained, which means that the prior information of task requests in time slot t is unknown. Therefore, the ARIMA model, LSTM and wavelet neural network methods can be used to predict The predicted value of Baseline value calculation module, calculates the baseline value According to the definition of reward, the strategy of deep reinforcement learning is to gradually approach or even surpass the baseline value. Therefore, the baseline value The closer to the optimal value, the better. The baseline value is obtained by enumeration, greedy strategy, and bionic intelligent algorithm. The centralized intelligent scheduling algorithm module trains the network through the centralized intelligent scheduling algorithm CDDS to obtain the trained policy network; A federated distributed intelligent scheduling algorithm module, which partitions the trained policy network using a federated distributed intelligent scheduling algorithm (FDDS) and obtains a DDPG model based on a federated learning training model. The model deployment and inference module deploys the DDPG model to each RES.

Images