A server deployment method based on deep reinforcement learning for edge computing

Abstract

The present invention discloses a server deployment method based on deep reinforcement learning for edge computing, which takes into account the heterogeneity between edge servers while constraining energy consumption, and balances the two optimization goals of placement cost and average access delay. First, the network architecture of edge computing is determined, and a system model and a computing model are constructed according to the network architecture, and then a problem optimization model is constructed according to the problem model. Secondly, a greedy-based strategy is used to preferentially select a subset of servers with the lowest unit resource cost price, and a subset with an energy consumption upper limit greater than the request sum and the lowest total cost is selected for subsequent deployment. Finally, the edge server deployment is modeled as a Markov decision process, and a deep reinforcement learning algorithm is used to solve a reasonable edge server deployment strategy to make full use of limited edge computing resources.

Description

Technical Field

[0001] The present invention belongs to the field of Internet of Things and artificial intelligence, and specifically relates to a server deployment method based on deep reinforcement learning in edge computing. Background Art

[0002] Edge computing technology is one of the emerging technologies to achieve 5G key performance indicators. Under the edge computing framework, mobile network and Internet services have been effectively integrated and further expanded to other application areas. Edge computing was originally proposed as a supplement and extension of cloud computing technology. Its basic idea is to provide computing and storage capabilities by placing servers on the user side. Its concept fully adapts to the basic form requirement of the Internet's development to the Internet of Things era - decentralization. Compared with cloud computing, edge computing can effectively alleviate backbone network congestion, explore the inherent capabilities of wireless networks, and has the advantages of low latency, high bandwidth, and strong flexibility. However, the performance of edge computing still needs to be improved. Compared with cloud central servers, edge servers have limited storage and computing resources. Therefore, making full use of limited edge computing resources is one of the ways to improve the performance of edge computing networks.

[0003] Existing task offloading strategies for edge computing have been widely studied. They usually skip the pre-step of edge server deployment and assume that the base station itself is a computing resource or that the servers are randomly distributed in the network. However, these are unrealistic: first, a base station is a type of radio station, and its function is completely different from that of a server and cannot be treated the same; second, from the perspective of mobile users, random deployment strategies will have an adverse effect on service response time, thereby causing a decline in user service quality in request-intensive areas; finally, from the perspective of edge providers, if an edge server is placed at each base station, it will inevitably increase placement costs and waste resources. Reasonable server deployment is the first key step to improve the performance of edge computing networks, which will help lay a solid foundation for subsequent research and improve the performance of edge computing networks.

[0004] In recent years, some studies have adopted clustering-based methods to deploy edge servers. This strategy usually places the server at the center of each user group, but since users away from the center of the cluster are not taken care of, these solutions cannot provide guarantees for the average user service quality in practical applications. Since the edge server placement problem is an NP-hard problem, that is, it is impossible to find all optimal feasible solutions in polynomial time, some studies have also used approximate algorithms and heuristic algorithms to solve such problems. Dixit Bhatta et al. focused on optimizing the cost of server deployment and user access delay, and proposed a two-factor approximation algorithm to separate the optimization objectives and independently analyze the worst case, thereby achieving the Pareto optimal state of multiple objectives, that is, the value of any objective function cannot be improved without reducing the values ​​of other objectives; Yuanzhe Li et al. added weight q on the basis of the traditional particle swarm algorithm to provide service quality assurance for base stations with large workloads and in core areas, thereby optimizing the energy consumption and average access delay of the server and maximizing the deployment profit of edge suppliers; considering the shortcomings of approximate algorithms and metaheuristic algorithms such as poor scalability, easy to fall into local optimal solutions and difficult parameter adjustment, Fei Luo et al. used reinforcement learning algorithms to solve the edge server placement problem, balance the workload between servers, and maximize resource utilization; for the application scenario of the Internet of Vehicles, Jiawei Lu et al. pre-set the number of servers to be placed in the edge network, and used deep reinforcement learning algorithms to solve the placement strategy that maximizes the coverage range, but the above studies are all based on the assumption that the server scale is consistent, and do not take into account the heterogeneity between servers in the actual edge network. Summary of the invention

[0005] Aiming at the problem of insufficient server utilization in edge computing networks, the present invention integrates the needs of users and suppliers, imposes energy consumption constraints while considering the heterogeneity between servers, balances the two optimization goals of placement cost and average access delay, and proposes a reasonable edge server deployment strategy to make full use of limited edge computing resources.

[0006] In order to solve the above problems, the present invention proposes a server deployment method based on deep reinforcement learning in edge computing. This method solves the two problems of server selection and layout according to the problem model on the basis of establishing a network model. First, determine the network architecture of edge computing, build a system model and a computing model according to the network architecture, and then determine the problem optimization model based on the computing model. Secondly, use a greedy-based strategy to preferentially select a subset of servers with the lowest unit resource cost price, and select a subset with an energy consumption upper limit greater than the request sum and the lowest total cost for subsequent deployment. Finally, model the edge server deployment as a Markov decision process, and use a deep reinforcement learning algorithm to solve a reasonable edge server deployment strategy to make full use of limited edge computing resources. Specifically comprising the following steps:

[0007] S1. Determine the network architecture for server deployment in edge computing;

[0008] The present invention relates to a server deployment strategy based on deep reinforcement learning in edge computing, such as Figure 2 As shown, the edge computing network usually consists of a three-layer architecture: a cloud computing center layer, an edge server layer, and a user layer. The present invention is mainly concerned with the edge server layer. There are several base stations and several edge servers in the network. It is stipulated that edge servers can only be placed at base stations, and each base station can be used as a candidate placement point for the server. Each server can choose to connect to one or more base stations. In order to meet the computing and network requirements of all users in edge computing, each base station must select one and only one server to connect to. When there are multiple servers to choose from around the base station, the most suitable one can be selected based on the computing model.

[0009] S2. Build a system model based on the edge computing network architecture;

[0010] The system model consists of a network topology model and a server layout relationship model.

[0011] The network topology model is as follows: The network topology relationship of the edge server layer is represented by a two-dimensional undirected graph G = (V, E). Where V = B ∪ S, B is the base station set, b i (i=1,2,3,...,m) represents the i-th base station, S is the server set; s j (j=1,2,3,...,n) represents the jth server;

[0012] Represents the connection relationship between the base station and the server, where Indicates base station b i Is it assigned to server s j .

[0013] The server layout relationship model is as follows: The layout relationship of the edge server is represented by Ω = (L, E), where Represents the location set where the server is located. Represents edge servers j Is it placed in b i Place.

[0014] S3. Build a computing model based on the edge computing network architecture;

[0015] The calculation model consists of three parts: server energy consumption model, cost model and delay model.

[0016] The energy consumption model is as follows: For server s j, its energy consumption w(s j ) is expressed as

[0017]

[0018] Among them, w(b i ) is base station b i The total number of user visits undertaken.

[0019] The cost model is as follows: The placement cost and C of all servers in the edge network are

[0020]

[0021] Among them, C(s j ) is the server s j The placement cost is as follows:

[0022] C(s j )=λ1W max (s j )+λ2G(s j ) (3)

[0023] Among them, W max (s j ) is the server s j The energy limit, G(s j ) is the server s j , λ1 and λ2 represent the proportionality coefficients. The placement cost of the server is positively correlated with its energy upper limit and the size of the floor space, so the proportionality coefficients are all positive.

[0024] The delay model is as follows: The average access delay in the network is

[0025]

[0026] Among them, x is the total number of servers selected in the network, d(b i ,s j ) represents base station b i and servers j Since all user data in the 5G system needs to be forwarded through the user plane function (UPF), the traditional method of using Euclidean distance to calculate the delay is no longer accurate. UPF should be taken into account when calculating the delay. This method only focuses on the access delay between the base station and the server, which is specifically composed of the access delay between the base station and UPF and the access delay between UPF and the edge server, as shown below:

[0027]

[0028] Among them, u krepresents UPFk (k=1, 2, 3…), and r represents the total number of UPFs.

[0029] S4. Based on the calculation model, establish an optimization model that considers the energy consumption limit and balances the cost and average delay;

[0030] The Z-Score normalization method is used to normalize the two competing optimization objectives of average delay and total cost, and to attach weights that sum to 1. j The energy consumption upper limit is optimized

[0031] min(θZ C +(1-θ)Z D ) (6)

[0032] Subject to:

[0033]

[0034] Among them, Z C is the result of normalizing the cost in formula (3), Z D is the result of normalizing the delay in formula (4). Formula (7) represents the energy consumption constraint of the server, that is, the energy consumption of each server cannot exceed its upper limit.

[0035] S5. Selecting an edge server includes the following steps:

[0036] Step 1: Select a server subset;

[0037] First, the sum of network requests received by all base stations in the network is counted. Then, among the sets of servers with different specifications, the servers with the lowest unit resource cost are preferentially selected to form a subset, and the subset with energy consumption cap greater than or equal to the sum of requests is retained.

[0038] Step 2: Among all subsets, select the subset with the lowest total cost price.

[0039] S6. The edge server deployment process is established as a Markov decision process, and a Markov decision model is constructed, which includes three elements: state space, action space, and reward and punishment function. The model is defined as follows:

[0040] State space: The state space S is represented by the server placement queue and the mapping relationship between the server and the base station, that is, S = (S1,…,S j ,…,S m ), where S j =(Location j ,Allocation j ), Location j =(lat j,lon j ) represents the edge server s j Location coordinates in a two-dimensional undirected graph, lat j Indicates j The dimension coordinate value, lon j Indicates j Longitude coordinate value; Allocation j For edge servers j The base station mapping queue is composed of j The server status is composed of the base station numbers to which it is connected. When the server location or the connection relationship with the base station changes, a new status is generated accordingly.

[0041] Action space: Each action is set to randomly change the placement of a server in the network, and each edge server can only move one step toward the neighboring base station. In a two-dimensional undirected graph, the server moving upward is equivalent to moving in the direction of the base station with increasing dimension, moving right is equivalent to moving in the direction of increasing longitude, and the same rules apply to moving left and downward. On the one hand, the action space A1 = (0,1,2,…,m) is defined as the set of edge servers, and a total of m edge servers are selected in the network; on the other hand, the action space A2 = (0,1,2,3) is defined as the movement direction of the edge server, where 0 means moving upward, 1 means moving downward, 2 means moving left, and 3 means moving right. Combining A1 and A2 into an action space A = (0,1,2,…,4*m), this means that for an input state sequence with m servers, there are a total of 4*m corresponding actions, and the optimal action is one of them.

[0042] Reward and Penalty Function: According to the optimization model established in step 2 of claim 1, the reward and penalty function is set as:

[0043] R=-(θZ C +(1-θ)Z D ) (8)

[0044] S7. Use the deep reinforcement learning algorithm to solve the deployment strategy to achieve the optimization goal, which specifically includes the following steps:

[0045] Step 1: Initialize parameters, including the number of algorithm iterations i, the number of executions T, the discount factor γ, the playback buffer capacity N, the batch size w, and the target Q network update frequency C;

[0046] Step 2: Construct an evaluation Q network with random weight θ and a target Q network with weight θ′, and set θ′ = θ;

[0047] Step 3: loop the iterative process for i times;

[0048] Step 4: Randomly generate the initial state s1 according to the defined state space;

[0049] Step 5: In order to explore more possibilities, according to the current state s t , use the ε-greedy algorithm to select action a t :

[0050]

[0051] That is, there is a probability of ε to randomly select an action, and a probability of 1-ε to select the action with the largest Q value corresponding to state s;

[0052] Step 6: Execute action a t , according to the reward and punishment function, the timely reward and punishment value r is calculated t , and relies on the evaluation Q network to obtain the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 ) into the playback buffer;

[0053] Step 7: Conduct the learning process;

[0054] Randomly select w tuples in the playback buffer and calculate the target Q value y for each tuple t :

[0055]

[0056] Minimize the loss function by backpropagation:

[0057] L=E[(y t -Q(s t ,a t ,θ)) 2 ]

[0058] Update θ;

[0059] Step 8: After each C actions, update θ′=θ;

[0060] Step 9: After executing T actions, proceed to the next round of iteration.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] The server deployment strategy based on deep reinforcement learning for edge computing can be applied to any edge computing network architecture, which is conducive to making full use of limited edge computing resources. In order to be closer to the actual application scenario, this strategy limits the energy consumption of edge servers while taking into account the heterogeneity between servers, and sets a set of servers with different computing and storage capabilities for selection. The deep reinforcement learning algorithm solution strategy avoids the shortcomings of traditional approximate algorithms and heuristic algorithms, such as poor scalability, easy to fall into local optimal solutions, and difficulty in parameter adjustment. It fully considers the needs of users and service providers in the edge computing environment and weighs the two optimization goals of placement cost and average access delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to make the purpose, structure and scheme of the present invention more understandable, the present invention will be further described below in conjunction with the accompanying drawings.

[0064] Figure 1 It is a flow chart of the present invention;

[0065] Figure 2 This is the edge computing network architecture diagram;

[0066] Figure 3 Schematic diagram of the deep reinforcement learning algorithm structure of the present invention. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0068] The present invention relates to a multi-level transaction scheduling and allocation strategy for edge-cloud heterogeneity, such as Figure 1 As shown, the following steps are included:

[0069] S1. Determine the network architecture for server deployment in edge computing;

[0070] The present invention relates to a server deployment strategy based on deep reinforcement learning in edge computing, such as Figure 2 As shown in Figure 1, edge computing networks are usually composed of three layers: cloud computing center layer, edge server layer, and user layer. This topic focuses on the edge server layer. There are several base stations and several edge servers in the edge server layer. It is stipulated that edge servers can only be placed at base stations and can be connected to one or more base stations. Each base station must select a server to connect to.

[0071] S2. Build a system model based on the edge computing network architecture;

[0072] The system model consists of two parts: the network topology model and the server layout relationship model;

[0073] The network topology is represented by a two-dimensional undirected graph G = (V, E). Where V = B ∪ S, B = b i (i=1,2,3,...,m) is the base station set; S=s j (j=1,2,3,...,n) is the server set; Indicates the connection relationship between the base station and the server. Indicates base station b i Is it assigned to server s j .

[0074] The server layout relationship is expressed as Ω = (L, E). Represents the location set where the server is located. Represents edge servers j Is it placed in b i Place.

[0075] S3. Build a computing model based on the edge computing network architecture;

[0076] The calculation model consists of three parts: energy consumption model, cost model and delay model.

[0077] For servers j , its energy consumption w(s j ) is expressed as w(b i ) is base station b i The total number of user visits undertaken; the placement cost is C(s j )=λ1W max (s j )+λ2G(s j ), W max (s j ) is the server s j The energy limit, G(s j ) is the server s j The size of the floor space, λ1 and λ2 represent the positive proportionality coefficients, and the placement cost of all servers in the edge network is The average access delay in the network is x is the total number of servers selected in the network, Indicates base station b i and servers j The access delay between the base station and the UPF is composed of the access delay between the UPF and the edge server. k represents UPFk (k=1,2,3…), r represents the total number of UPFs;

[0078] S4. Based on the calculation model, establish an optimization model that considers the energy consumption limit and balances the cost and average delay;

[0079] Establish optimization model min(θZ C +(1-θ)Z D ), Z C With Z D They are the cost and delay normalized by the Z-Score standardization method.

[0080] S5. Selecting an edge server includes the following steps:

[0081] Step 1: Select a server subset;

[0082] First, the sum of network requests received by all base stations in the network is counted. Then, among the sets of servers with different specifications, the servers with the lowest unit resource cost are preferentially selected to form a subset, and the subset with energy consumption upper limit greater than or equal to the sum of requests is retained.

[0083] Step 2: Among all subsets, select the subset with the lowest total cost price.

[0084] S6. Establish the edge server deployment process as a Markov decision process and construct a Markov decision model including three elements: state space, action space and reward and punishment function;

[0085] S7. Using a deep reinforcement learning algorithm to solve a deployment strategy for achieving an optimization goal includes the following steps:

[0086] Step 1: Initialize parameters;

[0087] Step 2: Construct an evaluation Q network with random weight θ and a target Q network with weight θ′, and set θ′ = θ;

[0088] Step 3: loop the iterative process for i times;

[0089] Step 4: Randomly generate the initial state s1 according to the defined state space;

[0090] Step 5: In order to explore more possibilities, according to the current state s t , use the ε-greedy algorithm to select action a t ;

[0091] Step 6: Store the immediate reward and punishment values ​​and next state data obtained after the action into the playback buffer;

[0092] Step 7: Randomly sample in the playback buffer, use back propagation to minimize the loss function, optimize the parameters, and train the neural network;

[0093] Step 8: After each C actions, update θ′=θ;

[0094] Step 9: After executing T actions, proceed to the next round of iteration.

[0095] For specific steps, please refer to the detailed description in the content of the invention.

Claims

1. A server deployment method based on deep reinforcement learning in edge computing, characterized in that: The following steps are involved: Step 1: Determine the network architecture for server deployment in edge computing; The edge computing network consists of three layers: cloud computing center layer, edge server layer and user layer. In the edge server layer, there are several base stations and several edge servers in the network. It is stipulated that edge servers can only be placed at base stations, and each server chooses to connect to one or more base stations. Each base station must choose a server to connect to. Step 2: Build a system model based on the edge computing network architecture; The system model consists of a network topology model and a server layout relationship model; The network topology model is as follows: The network topology relationship of the edge server layer is represented by a two-dimensional undirected graph G = (V, E); where V = B ∪ S, B is the base station set, b i (i=1,2,3,...,m) represents base station i, S is the server set, s j (j=1,2,3,...,n) represents server j; Represents the connection relationship between the base station and the server, where Indicates base station b i Is it assigned to server s j ; The server layout relationship model is as follows: The layout relationship of the edge server is represented by Ω = (L, E), where Represents the location set where the server is located. Represents edge servers j Is it placed in b i Department; Step 3: Build a computing model based on the edge computing network architecture; The calculation model considers three parts: the server's energy consumption model, cost model, and delay model; The energy consumption model is as follows: For server s j , its energy consumption w(s j ) is expressed as Among them, w(b i ) is base station b i Total number of user visits undertaken; The cost model is as follows: The placement cost of all servers in the edge network is Among them, C(s j ) is the server s j The placement cost is expressed as follows: C(s j )=λ1W max (s j )+λ2G(s j ) (3) Among them, W max (s j ) is the server s j The energy limit, G(s j ) is the server s j The size of the floor space, λ1 and λ2 represent the proportionality coefficients. The server placement cost is positively correlated with its energy limit and floor space size, so the proportionality coefficients are all positive numbers. The delay model is as follows: The average access delay in the network is Among them, x is the total number of servers selected in the network, d(b i ,s j ) represents base station b i and servers j The access delay between the base station and the server is composed of the access delay between the base station and the UPF and the access delay between the UPF and the edge server, as shown below: Among them, u k represents UPFk (k=1,2,3…), r represents the total number of UPFs; Step 4: Based on the calculation model, establish an optimization model that considers the energy consumption limit and balances the cost and average delay; The Z-Score normalization method is used to normalize the two competing optimization objectives of average delay and total cost, and add weights that sum to 1 respectively; the comprehensive server s j The energy consumption upper limit is optimized by: min(θZ C +(1-θ)Z D ) (6) Among them, Z C is the result of normalizing the cost in formula (3), Z D is the result of normalizing the delay in formula (4); Formula (7) represents the energy consumption constraint of the server, that is, the energy consumption of each server does not exceed its upper limit; Step 5, selecting an edge server, includes the following steps: Step 5.1, server subset selection; First, the sum of network requests received by all base stations in the network is counted. Then, from the set of servers with different specifications, the servers with the lowest unit resource cost are selected to form a subset, and the subset with energy consumption upper limit greater than or equal to the sum of requests is retained. Step 5.2: Among all subsets, select the subset with the lowest total cost price.

2. The server deployment method based on deep reinforcement learning in edge computing according to claim 1, characterized in that: The edge server deployment process is established as a Markov decision process, and the deployment strategy is solved using a deep reinforcement learning algorithm to achieve the optimization goal. The specific steps include: Step 1: Construct a Markov decision model consisting of three elements: state space, action space, and reward and punishment function. The model is defined as follows: State space: The state space S is represented by the server placement queue and the mapping relationship between the server and the base station, that is, S = (S1,…,S j ,…,S m ), where S j =(Location j ,Allocation j ), Location j =(lat j ,lon j ) represents the edge server s j Location coordinates in a two-dimensional undirected graph, lat j Indicates j The latitude coordinate value, lon j Indicates j Longitude coordinate value; Allocation j For edge servers j The base station mapping queue is composed of j The connected base station numbers; when the server location or the connection relationship with the base station changes, a new status is generated accordingly; Action space: Each action is set to randomly change the placement of a server in the network, and each edge server can only move one step toward the neighboring base station; in a two-dimensional undirected graph, the server moving upward is equivalent to moving toward the base station with increasing latitude, moving right is equivalent to moving in the direction of increasing longitude, and the rules for moving left and downward are the same; the action space A1 = (0,1,2,…,m) is defined as the set of edge servers, and a total of m edge servers are selected in the network; the action space A2 = (0,1,2,3) is defined as the movement direction of the edge server, where 0 means moving upward, 1 means moving downward, 2 means moving left, and 3 means moving right; A1 and A2 are combined into an action space A = (0,1,2,…,4*m). For an input state sequence with m servers, there are a total of 4*m actions, and the optimal action is one of them; Reward and Penalty Function: According to the optimization model established in step 2 of claim 1, the reward and penalty function is set as: R=-(θZ C +(1-θ)Z D ) (8) Step 2: Introduce the deep Q network DQN to solve the edge server deployment problem.

3. The server deployment method based on deep reinforcement learning in edge computing according to claim 2, characterized in that: The Q value is calculated using the Bellman optimality equation as follows: Among them, the discount factor γ∈[0,1] is the weight that balances the current reward and punishment value with the future reward and punishment value; the DQN algorithm obtains timely reward and punishment feedback when leaving the current state and the maximum Q value of the next state, and selects the optimal strategy to maximize the total expected return.

4. According to claim 2, a server deployment method based on deep reinforcement learning in edge computing is characterized in that: The DQN algorithm consists of the following steps: Step 1, initialize the parameters, initialize the algorithm iteration number i, execution number T, discount factor γ, playback buffer capacity N, batch size w and target Q network update frequency C; Step 2: construct an evaluation Q network with random weight θ and a target Q network with weight θ′, and set θ′=θ; Step 3, loop iteration process, each iteration loop execution process is as follows: Step 3.1, randomly generate the initial state s1 according to the state space defined in step 1 of the DQN algorithm; Step 3.2, according to the current state s t , use the ε-greedy algorithm to select action a t : That is, there is a probability of ε to randomly select an action, and a probability of 1-ε to select the action with the largest Q value corresponding to state s; Step 3.3, perform action a t , according to the reward and punishment function, the timely reward and punishment value r is calculated t , and relies on the evaluation Q network to obtain the next state s t+1 , the quaternion (s t ,a t ,r t ,s t+1 ) into the playback buffer; Step 3.4, carry out the learning process; Randomly select w tuples in the playback buffer and calculate the target Q value y for each tuple t : Minimize the loss function by backpropagation: L=E[(y t -Q(s t ,a t ,i)) 2 ] Update θ; Step 3.5, every time C actions are performed, update θ′=θ; Step 3.6, after executing T actions, proceed to the next round of iteration.

Images