A 5G core network user plane dynamic deployment method based on deep reinforcement learning

Abstract

The application belongs to the technical field of mobile communication, and particularly relates to a 5G core network user plane dynamic deployment method based on deep reinforcement learning. The application comprehensively considers three factors of edge server energy consumption, user plane data delay and base station traffic migration cost, establishes a 5G core network user plane dynamic deployment optimization model, so as to reduce the total operation cost in the operation period and ensure the service quality; the 5G core network user plane dynamic deployment optimization problem is converted into a Markov decision process model, and a DDQN algorithm is used for solving, so as to determine the optimal 5G core network user plane dynamic deployment strategy. The 5G core network user plane dynamic deployment method based on deep reinforcement learning has important significance for realizing the high energy efficiency and low delay of the 5G core network user plane and meeting the 5G low delay service demand.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology, specifically relating to a dynamic deployment method for the user plane of a 5G core network based on deep reinforcement learning. Background Technology

[0002] With the rapid development of 5G technology, mobile communication has entered a new era, bringing higher data transmission rates, lower latency, and greater connection density. To meet the explosive growth in data demand in 5G networks and support diverse application scenarios, the deployment of the 5G core network user plane is particularly important. Deploying the core network user plane on edge servers and optimizing base station traffic scheduling have become key technologies for improving network performance, reducing latency, and lowering energy consumption.

[0003] Traditional 5G core network user plane deployments often employ a static, fixed approach, which cannot adapt to dynamic fluctuations in network traffic and changes in user needs. Furthermore, traditional user plane deployments are typically centralized in a central data center, resulting in longer transmission paths for user plane data, increasing data transmission latency and impacting user experience. However, by introducing emerging technologies such as mobile edge computing and fog computing, deploying the user plane on edge servers closer to users can significantly reduce user plane data transmission latency, improve network response speed, and enhance user satisfaction.

[0004] However, the dynamic deployment of the user plane and the optimized scheduling of base station traffic face a series of challenges. First, the number and location of edge servers are limited; how to rationally deploy the user plane on edge servers to achieve optimal user experience and network performance is a complex optimization problem. Second, edge server deployment involves server energy consumption, requiring a balance between user experience and energy consumption to achieve efficient resource utilization. Simultaneously, with the dynamic changes in base station traffic, the network needs to be able to adjust the user plane deployment scheme in real time to meet performance requirements under different network loads, which necessitates the introduction of a dynamic scheduling mechanism into the network. Summary of the Invention

[0005] The purpose of this invention is to solve the aforementioned problems by proposing a dynamic deployment method for the 5G core network user plane based on deep reinforcement learning. This invention uses the minimization of the sum of edge server energy consumption, user plane data latency, and base station traffic migration costs within the operating cycle as the objective function. It establishes a dynamic deployment optimization model for the 5G core network user plane, transforms it into a Markov decision process, and employs a deep reinforcement learning algorithm—specifically, the DDQN algorithm—to learn and optimize the deployment decisions for the 5G core network user plane. This achieves intelligent user plane deployment, improving network performance and resource utilization efficiency.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] A dynamic deployment method for 5G core network user plane based on deep reinforcement learning includes the following steps: Step 1, establishing a dynamic deployment optimization model for 5G core network user plane, with the objective function being the minimization of the sum of edge server energy consumption, user plane data latency, and base station traffic migration cost within the operating cycle; Step 2, constructing a Markov decision process model for the dynamic deployment optimization problem of 5G core network user plane, wherein the Markov decision process model treats the dynamic deployment optimization problem of 5G core network user plane as a time-slot sequential decision problem, and divides the decision for each time slot into... Step 3: The traffic of one base station is scheduled in each step. Each decision is based on the index of the base station to be scheduled, the traffic load vector of the base station in the current time slot, the traffic scheduling result of the base station in the previous time slot, and the traffic scheduling result of the base station in the current time slot. The action set is the set of all edge servers. The reward function considers the impact of the selected action on the energy consumption of the edge server, the latency of user plane data, and the cost of flow migration, while also considering the violation of the processing capacity constraints of the edge server by the scheduling decision. Step 4: Solve the MDP model of the 5G core network user plane dynamic deployment optimization problem using the DDQN algorithm.

[0008] Furthermore, as a preferred embodiment of the present invention, the 5G core network user plane dynamic deployment optimization model considers constraints such as base station traffic scheduling status, edge server processing capacity, edge server activation status, and base station traffic migration status.

[0009] Furthermore, as a preferred embodiment of the present invention, the 5G core network user plane dynamic deployment optimization model is as follows:

[0010] (1)

[0011] In the objective function A set of time slots for one period. , and They are time slots Edge server energy consumption, user plane data latency, and base station traffic migration costs; In For a set of edge servers, For base station set, and Edge servers Peak power consumption and idle power consumption, For time slots base station Data traffic, For data packet length, For data packet processing time, For edge servers The number of processor cores, The duration of the time slot is a 0-1 variable. Indicates time slot base station Is the traffic being routed to the edge server? , Yes, Indicates no, 0-1 variable Indicates time slot Edge server Enable or disable? Indicates that it is turned on. This indicates that it is not enabled; In For base stations and edge servers Propagation delay between; In The time required for base station traffic to migrate from one edge server to another, a 0-1 variable. Indicates time slot base station Has the traffic shifted? This indicates that migration has occurred. This indicates that no migration has occurred;

[0012] Constraint (a) indicates that traffic from each base station can only be scheduled to one edge server; constraint (b) is a processing capacity constraint for the edge server; constraints (c) and (d) indicate that an edge server is enabled when traffic from at least one base station is scheduled to a certain edge server; constraint (e) indicates that traffic from a base station has migrated when traffic from a base station is scheduled to different edge servers in two adjacent time slots.

[0013] Furthermore, as a preferred embodiment of the present invention, the Markov decision process model is defined as follows:

[0014] Status: Time Slot No. The state of a step is defined as follows ; This is the index of the base station currently awaiting scheduling; This is the base station traffic load vector. ; For time slots The results of base station traffic scheduling Indicates time slot base station Is the traffic being routed to the edge server? ; For time slots forward The traffic scheduling results of each base station, in the time slot At the beginning, Initialized to all zeros; if the first Step base station Traffic is routed to edge servers Then Set the value to 1;

[0015] Action: Time Slot No. Step movement Indicates base station Which edge server is the traffic routed to, where the set of edge servers is... This is the action space; the decisions of all steps in a time slot constitute the complete base station traffic scheduling for that time slot and will be embedded into the state of the next time slot.

[0016] Reward function: Performs an action in the current state. The reward obtained is defined as:

[0017] (2)

[0018] , , These represent the increases in edge server energy consumption, user plane data latency, and stream migration costs, respectively. The penalty imposed for violating edge server processing capacity constraints; the specific definition is as follows:

[0019] (3)

[0020] (4)

[0021] (5)

[0022] (6)

[0023] It is a sufficiently large positive number.

[0024] Furthermore, as a preferred embodiment of the present invention, in the DDQN algorithm, the loss function is defined as:

[0025] (7)

[0026] in, For samples in the experience pool, For discount rate, and These represent the parameters of the online Q-network and the target Q-network, respectively; For the target Q value, Given the current Q value, the loss function reflects the estimation error of the DDQN model. The smaller the loss function, the better the estimation performance of the DDQN model. To optimize the loss function, the parameters of the Q-network are updated using the gradient descent algorithm, as follows:

[0027] (8)

[0028] in, and These represent the parameters of the online Q network before and after the update, respectively. This is the step size parameter.

[0029] Furthermore, as a preferred embodiment of the present invention, the training process of the DDQN algorithm is as follows:

[0030] (1) Initialize the experience pool and randomly initialize the parameters of the online Q network. And assign it to the target Q network, i.e. ;

[0031] (2) Initial state of the MDP model for the dynamic deployment optimization problem of 5G core network user plane , i.e., slot For base stations The state of the system during scheduling;

[0032] (3) Use the ε-greedy strategy to select the action to be performed in the current state, with probability. Randomly select an action With probability Select Action ;

[0033] (4) Performing actions Traffic from one base station is routed to an edge server to generate revenue. and transition to a new state. ;

[0034] (5) Take the sample Store in the experience pool;

[0035] (6) Randomly select a small batch of samples from the experience pool and update the parameters of the online Q-network using the gradient descent algorithm. ;

[0036] (7) Every certain number of steps, assign the parameters of the online Q-network to the target Q-network, i.e. ;

[0037] (8) Repeat (3)-(7) until the termination condition is met.

[0038] The 5G core network user plane dynamic deployment method based on deep reinforcement learning described in this invention has the following technical advantages compared with existing technologies:

[0039] This invention provides a dynamic deployment method for the 5G core network user plane based on deep reinforcement learning. It comprehensively considers three factors: edge server energy consumption, user plane data latency, and base station traffic migration costs. A dynamic deployment optimization model for the 5G core network user plane is established to reduce the total operating cost over the operating cycle while ensuring service quality. The optimization problem is transformed into a Markov decision process model, and the DDQN algorithm is proposed for solving it to determine the optimal dynamic deployment strategy. This deep reinforcement learning-based dynamic deployment method for the 5G core network user plane is of great significance for achieving a high-energy-efficiency, low-latency 5G core network user plane and meeting the low-latency service requirements of 5G. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the 5G core network user plane edge deployment according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the method steps in an embodiment of the present invention;

[0042] Figure 3 This is a diagram of the DDQN algorithm architecture according to an embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0044] like Figure 1 The diagram shown illustrates the edge deployment of a 5G core network user plane dynamic deployment method based on deep reinforcement learning, as provided in an embodiment of the present invention. The distribution of base stations and edge servers follows an independent Poisson point process. Base stations and edge servers are connected via millimeter-wave backhaul links, and the propagation delay between them is proportional to their geographical distance. The edge servers are used to deploy 5G core network user plane UPF network elements. Each base station connects wirelessly to multiple UEs, and the traffic of a base station is the sum of the traffic of the multiple UEs with which it communicates.

[0045] like Figure 2As shown, the present invention provides a dynamic deployment method for the user plane of a 5G core network based on deep reinforcement learning, comprising the following steps:

[0046] Step 1: Establish a dynamic deployment optimization model for the 5G core network user plane;

[0047] Taking into account edge server energy consumption, user plane data latency, and base station traffic migration costs, a dynamic deployment optimization model for the 5G core network user plane is established. The objective function of this model is to minimize edge server energy consumption, user plane data latency, and base station traffic migration costs within the operating cycle, expressed as:

[0048] (1-1)

[0049] In formula (1-1), A set of time slots for one period. , and They are time slots The energy consumption of edge servers, latency of user plane data, and cost of migrating base station traffic.

[0050] A linear power consumption model is used, with time slots The energy consumption of edge servers is expressed as follows:

[0051] (1-1-1)

[0052] In formula (1-1-1), For a collection of edge servers, For base station set, and Edge servers Peak power consumption and idle power consumption, For time slots base station Data traffic, For data packet length, For data packet processing time, For edge servers The number of processor cores, The duration of the time slot is a 0-1 variable. Indicates time slot base station Is the traffic being routed to the edge server? , Yes, Indicates no, 0-1 variable Indicates time slot Edge server Enable or disable? Indicates that it is turned on. This indicates that it is not enabled.

[0053] Time slot User plane data latency is:

[0054] (1-1-2)

[0055] In formula (1-1-2), For base stations and edge servers The propagation delay between them.

[0056] Time slot The cost of migrating base station traffic is:

[0057] (1-1-3)

[0058] In formula (1-1-3), The time required for base station traffic to migrate from one edge server to another, a 0-1 variable. Indicates time slot base station Has the traffic shifted? This indicates that migration has occurred. This indicates that no migration has occurred.

[0059] Meanwhile, constraints are considered regarding base station traffic scheduling status, edge server processing capacity, edge server activation status, and base station traffic migration status; specific constraints are shown in formula (1-2):

[0060] (1-2)

[0061] In formula (1-2), constraint (a) indicates that the traffic of each base station can only be scheduled to one edge server; constraint (b) is the processing capacity constraint of the edge server; constraints (c) and (d) indicate that when the traffic of at least one base station is scheduled to an edge server, the edge server is turned on; constraint (e) indicates that when the traffic of a base station is scheduled to different edge servers in two adjacent time slots, the traffic of that base station has been migrated.

[0062] Substituting the above formulas (1-1-1), (1-1-2), and (1-1-3) into formula (1-1) and combining them with formula (1-2), we obtain the 5G core network user plane dynamic deployment optimization model, as shown in formula (1):

[0063] (1)

[0064] Step 2: Construct a Markov decision process model for the dynamic deployment optimization problem of the 5G core network user plane;

[0065] The dynamic deployment optimization problem of the 5G core network user plane is viewed as a time-slot sequential decision problem, and the decision for each time slot is divided into... The MDP model schedules traffic for one base station at a time. Each decision is based on the index of the base station to be scheduled, the traffic load vector of the base station in the current time slot, the traffic scheduling result of the previous time slot, and the already determined traffic scheduling result for the current time slot. The action set is the set of all edge servers. The reward function considers the impact of executing the selected action on edge server energy consumption, user plane data latency, and flow migration costs, while also considering whether the scheduling decision violates the edge server processing capacity constraints. The specific definitions of the state, actions, and reward function of the MDP model are as follows:

[0066] Status: Time Slot No. The state of a step is defined as follows . This is the index of the base station currently awaiting scheduling. This is the base station traffic load vector. . For time slots The results of base station traffic scheduling Indicates time slot base station Is the traffic being routed to the edge server? . For time slots forward The traffic scheduling results of each base station, in the time slot At the beginning, Initialized to all zeros; if the first Step base station Traffic is routed to edge servers Then The value is set to 1.

[0067] Action: Time Slot No. Step movement Indicates base station Which edge server is the traffic routed to, where the set of edge servers is... This is the action space. The decisions made at all steps in a time slot constitute the complete base station traffic scheduling for that time slot and will be embedded into the state of the next time slot.

[0068] Reward function: Performs an action in the current state. The reward obtained is defined as:

[0069] (2)

[0070] , , These represent the increases in edge server energy consumption, user plane data latency, and stream migration costs, respectively. This is the penalty imposed for violating edge server processing capacity constraints. The specific definition is as follows:

[0071] (3)

[0072] (4)

[0073] (5)

[0074] (6)

[0075] It is a sufficiently large positive number.

[0076] Step 3: Use the DDQN algorithm to solve the MDP model for the dynamic deployment optimization problem of the 5G core network user plane;

[0077] In the DDQN algorithm, the loss function is defined as:

[0078] (7)

[0079] in, For samples in the experience pool, For discount rate, and These represent the parameters of the online Q-network and the target Q-network, respectively. For the target Q value, Given the current Q-value, the loss function reflects the estimation error of the DDQN model; the smaller the loss function, the better the estimation performance of the DDQN model. To optimize the loss function, the parameters of the Q-network are updated using the gradient descent algorithm, as follows:

[0080] (8)

[0081] in, and These represent the parameters of the online Q network before and after the update, respectively. This is the step size parameter.

[0082] like Figure 3 The diagram shown is of the DDQN algorithm architecture. The specific process is as follows:

[0083] (1) Initialize the experience pool and randomly initialize the parameters of the online Q network. And assign it to the target Q network, i.e. ;

[0084] (2) Initial state of the MDP model for the dynamic deployment optimization problem of 5G core network user plane , i.e., slot For base stations The state of the system during scheduling;

[0085] (3) Use the ε-greedy strategy to select the action to be performed in the current state, with probability. Randomly select an action With probability Select Action ;

[0086] (4) Performing actions Traffic from one base station is routed to an edge server to generate revenue. and transition to a new state. ;

[0087] (5) Take the sample Store in the experience pool;

[0088] (6) Randomly select a small batch of samples from the experience pool and update the parameters of the online Q-network using the gradient descent algorithm. ;

[0089] (7) Every certain number of steps, assign the parameters of the online Q-network to the target Q-network, i.e. ;

[0090] (8) Repeat (3)-(7) until the termination condition is met.

[0091] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for dynamic deployment of the user plane of a 5G core network based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Establish a dynamic deployment optimization model for the 5G core network user plane. The model's objective function is to minimize the sum of edge server energy consumption, user plane data latency, and base station traffic migration costs within the operating cycle. The model is as follows: （1） In the objective function A set of time slots for one period. , and They are time slots Edge server energy consumption, user plane data latency, and base station traffic migration costs; In For a set of edge servers, For base station set, and Edge servers Peak power consumption and idle power consumption, For time slots base station Data traffic, For data packet length, For data packet processing time, For edge servers The number of processor cores, The duration of the time slot is a 0-1 variable. Indicates time slot base station Is the traffic being routed to the edge server? , Yes, Indicates no, 0-1 variable Indicates time slot Edge server Enable or disable? Indicates that it is turned on. This indicates that it is not enabled; In For base stations and edge servers Propagation delay between; In The time required for base station traffic to migrate from one edge server to another, a 0-1 variable. Indicates time slot base station Has the traffic shifted? This indicates that migration has occurred. This indicates that no migration has occurred; Constraint (a) indicates that the traffic of each base station can only be scheduled to one edge server; constraint (b) is the processing capacity constraint of the edge server; constraints (c) and (d) indicate that when the traffic of at least one base station is scheduled to an edge server, the edge server is enabled; constraint (e) indicates that when the traffic of a base station is scheduled to different edge servers in two adjacent time slots, the traffic of that base station has been migrated. Step 2: Construct a Markov decision process model for the dynamic deployment optimization problem of the 5G core network user plane. Treat the 5G core network user plane dynamic deployment optimization problem as a time-slot sequential decision problem, and divide the decision for each time slot into... The process involves scheduling traffic for one base station at a time. Each decision step uses the index of the base station to be scheduled, the traffic load vector of the base station in the current time slot, the traffic scheduling result of the base station in the previous time slot, and the already determined traffic scheduling result of the base station in the current time slot as the state. The action set is the set of all edge servers. The reward function considers the impact of executing the selected action on the energy consumption of the edge servers, user plane data latency, and flow migration costs, while also considering whether the scheduling decision violates the processing capacity constraints of the edge servers. The Markov decision process model is defined as follows: Status: Time Slot No. The state of a step is defined as follows ; This is the index of the base station currently awaiting scheduling; This is the base station traffic load vector. ; For time slots The results of base station traffic scheduling Indicates time slot base station Is the traffic being routed to the edge server? ; For time slots forward The traffic scheduling results of each base station, in the time slot At the beginning, Initialized to all zeros; if the first Step base station Traffic is routed to edge servers Then Set the value to 1; Action: Time Slot No. Step movement Indicates base station Which edge server is the traffic routed to, where the set of edge servers is... This is the action space; the decisions of all steps in a time slot constitute the complete base station traffic scheduling for that time slot and will be embedded into the state of the next time slot. Reward function: Performs an action in the current state. The reward obtained is defined as: （2） , , These represent the increases in edge server energy consumption, user plane data latency, and stream migration costs, respectively. The penalty imposed for violating edge server processing capacity constraints; the specific definition is as follows: （3） （4） （5） （6） It is a sufficiently large positive number; Step 3: Solve the MDP model for the 5G core network user plane dynamic deployment optimization problem using the DDQN algorithm. In the DDQN algorithm, the loss function is defined as: （7） in, For samples in the experience pool, For discount rate, and These represent the parameters of the online Q-network and the target Q-network, respectively; For the target Q value, Given the current Q value, the loss function reflects the estimation error of the DDQN model. The smaller the loss function, the better the estimation performance of the DDQN model. To optimize the loss function, the parameters of the Q-network are updated using the gradient descent algorithm, as follows: （8） in, and These represent the parameters of the online Q network before and after the update, respectively. This is the step size parameter.

2. The method for dynamic deployment of 5G core network user plane based on deep reinforcement learning according to claim 1, characterized in that, The 5G core network user plane dynamic deployment optimization model considers constraints such as base station traffic scheduling status, edge server processing capacity, edge server activation status, and base station traffic migration status.

3. The method for dynamic deployment of 5G core network user plane based on deep reinforcement learning according to claim 2, characterized in that, The training process of the DDQN algorithm is as follows: (1) Initialize the experience pool and randomly initialize the parameters of the online Q network. And assign it to the target Q network, i.e. ; (2) Initial state of the MDP model for the dynamic deployment optimization problem of 5G core network user plane , i.e., slot For base stations The state of the system during scheduling; (3) Use the ε-greedy strategy to select the action to be performed in the current state, with probability. Randomly select an action With probability Select Action ; (4) Performing actions Traffic from one base station is routed to an edge server to generate revenue. and transition to a new state. ; (5) Take the sample Store in the experience pool; (6) Randomly select a small batch of samples from the experience pool and update the parameters of the online Q-network using the gradient descent algorithm. ; (7) Every certain number of steps, assign the parameters of the online Q-network to the target Q-network, i.e. ; (8) Repeat (3)-(7) until the termination condition is met.

Images