A deep reinforcement learning-based mobile edge computing dynamic service deployment method

Abstract

The application provides a mobile edge computing dynamic service deployment method based on deep reinforcement learning. In the learning process, a migration conflict resolution mechanism is introduced to maintain the service performance of the user. Specifically, not only the network state is predicted, but also the user connected to each edge node is calculated according to the user trajectory, and it is determined which node is the crowded node according to the user request and the service on the edge node, the resource reservation is carried out for the crowded node, and whether the conflict will be produced is detected. In order to achieve this goal, first, the network architecture of edge computing is determined, then the system model and the quality of service (QoS) model are established, and the problem to be solved is formulated. The application studies the joint optimization problem of delay and cost under the constraints of physical resources and operation cost, taking the minimum total cost of users as the target, and considering guaranteeing that multiple service requests can be responded in time under the sudden aggregation.

Description

Technical Field

[0001] This invention belongs to the fields of Internet of Things, wireless communication technology, and artificial intelligence, and specifically refers to a problem of dynamic service deployment for mobile edge computing based on deep reinforcement learning. Background Technology

[0002] In this era of information explosion on the internet, mobile network services have expanded to include various types of devices, such as tablets, mobile vehicles, and various sensors. Simultaneously, a plethora of new services are emerging. With the rapid development of the Internet of Things (IoT), smart transportation, smart cities, and smart agriculture are becoming increasingly common in our lives. The scale of mobile terminals will grow significantly in the near future. This continuous and rapid increase in data volume is driving the evolution of the entire computing paradigm. A series of resource-intensive and latency-sensitive applications, such as augmented / virtual reality, autonomous driving, and dynamic content delivery, have emerged and are widely used. These emerging IoT applications place higher demands on the real-time management and intelligent analysis of network edge services, making low latency, high performance, and high reliability particularly important. Traditional cloud computing, with its high latency due to long-distance communication, cannot meet these requirements. Therefore, a new computing paradigm has emerged: Mobile Edge Computing (MEC). MEC migrates the cloud computing platform from within the mobile core network to the mobile network edge, deploying computing and storage resources at the mobile network edge. This satisfies the expansion needs of terminal devices' computing capabilities while also solving the problem of long latency when accessing remote cloud data centers.

[0003] The placement of mobile edge computing services in scenarios with continuous provisioning for multiple mobile users has gradually become a focus of attention. Although this issue has been extensively studied, some studies still generally overlook the coupling relationship between service quality and operating costs caused by unstable user mobility. At the same time, existing research lacks in-depth research on the problems of untimely response and high overhead when multiple users suddenly gather, and does not take into account the issues of resource constraints and migration conflicts.

[0004] To address the issues of poor service quality and high operating costs caused by user mobility, and to minimize total user overhead (i.e., the sum of total latency and total cost) under constraints of physical resources and operating costs, while also ensuring service quality in the event of sudden clustering, a novel dynamic service deployment framework needs to be proposed. Summary of the Invention

[0005] Given the mobility and randomness of user activity, optimizing both total latency and total cost during sudden surges while adhering to physical resource and operational cost constraints is challenging. Finding an effective strategy to adapt to random movement is difficult. One solution is to migrate the service along with the user, resulting in lower latency. However, service migration also incurs higher operational costs. Another solution is to avoid service migration, saving operational costs, but this leads to higher latency due to long-distance communication.

[0006] To address the aforementioned issues, this invention investigates a joint optimization problem of latency and cost under constraints of physical resources and operational costs, aiming to minimize total user overhead, while also considering ensuring timely response to multiple service requests under bursty aggregation. This invention proposes a dynamic deployment framework based on deep reinforcement learning, introducing a migration conflict resolution mechanism during the learning process to maintain user service performance. Specifically, it not only predicts network state and calculates the users connected to each edge node for each time slot based on user trajectories, but also identifies congested nodes based on user requests and the services on their respective edge nodes, reserves resources for congested nodes, and detects potential conflicts. To achieve this goal, firstly, the network architecture of edge computing is determined, then a system model and a Quality of Service (QoS) model are established, and the problem to be solved is formulated. Based on the Deep Deterministic Policy Gradient (DDPG) algorithm, a dynamic service deployment algorithm (DRF-SF-V1) based on deep reinforcement learning is designed to solve this problem.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for dynamic deployment of mobile edge computing services based on deep reinforcement learning is applicable to a network architecture consisting of cloud servers, switches, and base stations. Each base station has limited computing and storage capabilities and connects to one MEC server. Each MEC has a service range. Services are deployed on virtual machines, and each user can only be served by one service on the MEC. The method is characterized by the following steps:

[0009] S1. Establish system and QoS models based on network architecture.

[0010] The system model is as follows: Assume there are a total of M MEC nodes, and the node set is M = {m j}, where m j This represents the j-th node; each MEC node connects to a base station with limited computing and storage capabilities, and the edge server m... jcomputing power This indicates that the edge server m j Storage capacity The set of users served by a MEC node is represented by U = {u i Let} represent, where u i Let represent the i-th user; the system runs continuously for a period of time [0, T], dividing the time axis into T+1 discrete time slots t, t∈T={0,1,2,…,T}; the set of user service requests is denoted by S={s h Let} represent, where s h Represents the h-th service; using Indicates user u i Whether to use the edge server in time slot t. j Deployed services h , if u i Use service s in time slot t h ,but otherwise, use This indicates that the edge server m is placed j The collection of services on, among which use Represents a set The set of users served;

[0011] For each edge server, adopt Represents edge server m j The preset total storage resources, i.e., edge server m j The service resource threshold is adopted. Represents edge server m j The storage resources currently in use, when When this happens, the node is defined as a crowded node;

[0012] The QoS model is as follows: using d c (i,t) represents user u i Calculate the delay within time slot t; α i (t) represents user u i The amount of computing resources required to serve a request within time slot t; Indicates user u i Whether to use the edge server in time slot t. j Deployed services h ; Represents edge server m j The computing power; the computing latency is:

[0013]

[0014] Communication latency occurs when a user and the requested service are located in different regions and communicate; maximum transmission rate. for:

[0015]

[0016] Among them, using γ represents the channel bandwidth of the physical link, where u is... i The transmission power; Indicate u i With m j The channel gain between, where Assuming service s h Deployed on edge node m j Above, then For user u i To edge node m j The distance between them; N is the noise power; This refers to the data size when passing through network devices between MEC nodes and service-providing nodes; data transmission is determined by the bandwidth of the physical link. and data size Therefore, the communication delay d l (i,t) is:

[0017]

[0018] Consider optimizing user experience through dynamic service migration; define a boolean variable. Indicates user u i services h Whether migration occurs within time slot t; ρ(s) h ) for service s h The migration latency includes the transfer of service configuration files, restarting software resources, etc.; the user's migration latency d u (i,t) is defined as:

[0019]

[0020] S3. Establish a joint optimization model oriented towards latency and cost;

[0021] Assume the cost of dynamic service placement is incurred by service migration across edge servers; to meet the Quality of Service (QoS) requirements of users with uncertain activity trajectories, the services required by users can be dynamically migrated to adapt to user mobility. h From edge server m a Migrate to m a′ The resulting costs It means that, among them For service s hThe specific formula for calculating the unit cost incurred during the migration process is as follows:

[0022]

[0023] The total cost within time slot t is:

[0024]

[0025] The total cost for users in set U over a continuous time period [0, T] is:

[0026]

[0027] The total delay within time slot t is:

[0028]

[0029] Where d c (i,t), d l (i,t), d u (i,t) represent the user's computation latency, communication latency, and migration latency, respectively;

[0030] The total latency for users in set U over a continuous time interval [0, T] is:

[0031]

[0032] The total delay d for users in set U within the continuous time interval [0,T] is given by... T And total cost c T The processing is mapped to the range [0,1], so the total latency and total cost are as follows:

[0033]

[0034]

[0035] in, and These represent the maximum delay and cost within the continuous time period [0, T], respectively. and These represent the minimum delay and cost within the continuous time period [0,T], respectively.

[0036] The optimization problem is represented as follows:

[0037] minimizeωD+(1-ω)C (12)

[0038]

[0039]

[0040]

[0041] The weight is ω, which represents the user's preference for latency or cost. The value of ω ranges from [0,1]. Equation (13) represents the cost constraint, where |S| represents the number of services. Indicates service s h From edge server m a Migrate to m a′ The costs incurred require that the total cost of providing the service cannot exceed a threshold Γ, where Γ represents the upper limit of the maximum total cost that the operator can provide; Equation (14) represents the physical resource constraint, where |s h | indicates that it is placed on the edge server m j The number of services on the edge server must not exceed the storage capacity. The computing resources required for a service request cannot exceed the computing capacity. Where α i (t) represents user u i The amount of computing resources required to serve a request within time slot t; Equation (15) represents the amount of computing resources required by user u. i Whether to use the edge server m within time slot t j Deployed services h .

[0042] The solution method for the joint optimization model in step S2 is the DRF-SF-V1 algorithm, and the execution process specifically includes:

[0043] Step 1: Predict the trajectory based on the Microsoft GPS trajectory dataset; observe the user's activity trajectory in the dataset and mark the origin center coordinates; divide the area with this location as the center point and r as the radius; traverse the user trajectory and find the trajectory within the area within 60 consecutive time slots;

[0044] Step 2: Randomly initialize the actor network (μ(S)) t |θ μ )) and critic network (Q(S,a|θ) Q ), where θ μ These are the actor network weights, θ Q These are the critic network weights; the weights are initialized to θ. μ′ and θ Q′ Target actor(μ(S) t |θ μ′ )) network and critic network (Q(S,a|θ) Q′ Initialize memory storage area B;

[0045] Step 3: Initialize the environment parameters of the edge server and users, including the action space dimension a_dim and the state space dimension s_dim, and generate the initial state S1; where the action space dimension and the state space dimension refer to the state space S1 and S2, respectively. t and action space A t The length of the vector;

[0046] Step 4: Select an action a t (S t |θ μ )+∈ t , where θ μ For the current policy network, ∈ t To explore noise, obtain immediate rewards and new states S t′ And store them in memory storage area B;

[0047] Step 5: Pre-training. When the storage capacity of memory storage area B is full, sampling training will be performed again.

[0048] Step 6: In each iteration, perform trajectory prediction based on the LSTM trajectory prediction model, perform pre-transfer, record the transfer trajectory, find congested nodes, and construct a set of congested nodes.

[0049] Step 7: Reduce the remaining available resources of servers around the congested node, reducing the capacity to the storage resource capacity currently used by the server, so as to ensure that the servers around the congested node can appear to have no available resources, thus reserving resources for the congested node.

[0050] Step 8: Perform conflict detection on edge nodes. If a resource request conflict occurs on the node, determine the set of conflicting users and calculate the estimated cost Y for each user in the set. Where d u (i,t) represents the migration delay, and d l (i,t) represents the communication delay; the user with the largest Y value in the conflicting user set is selected for migration. The selection principle for the migration location is to prioritize the edge node closest to the user. If the edge node has no spare resources, then the edge node closest to the user and with spare resources among the remaining nodes is selected.

[0051] Step 9: Calculate latency. If the service migrates within time slot t, calculate the migration latency and overhead. If no migration occurs in time slot t but the server hosting the service is not the server hosting the user, only calculate the communication latency. Otherwise, the total latency is 0.

[0052] Step 10: For user set U, output the current service placement strategy A. t Total overhead, release reserved resources.

[0053] Beneficial effects

[0054] Compared with the prior art, the present invention has the following advantages:

[0055] 1. This invention aims to minimize total user overhead by jointly optimizing total latency and total cost. Under the constraints of physical resources and operating costs, it studies the problem of untimely response to multiple service demands in the case of sudden aggregation of multiple users.

[0056] 2. This invention proposes a dynamic deployment framework based on deep reinforcement learning, introducing a migration conflict resolution mechanism during the learning process to maintain user service performance. A dynamic service deployment algorithm based on deep reinforcement learning (DRF-SF-V1) is designed to address this problem.

[0057] 3. Experiments show that the proposed dynamic service deployment algorithm DRF-SF-V1 based on deep reinforcement learning has a lower total cost than the comparison algorithms and can converge quickly. Attached Figure Description

[0058] To make the objectives and solutions of this invention more easily understood, the invention will be further described below with reference to the accompanying drawings.

[0059] Figure 1 This is a schematic diagram of a mobile edge computing network architecture;

[0060] Figure 2 Service deployment methods and steps;

[0061] Figure 3 A comparison chart showing the provided algorithm with traditional algorithms;

[0062] Figure 4 This is a schematic diagram illustrating the convergence of the provided algorithm. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0064] This invention relates to a dynamic service deployment algorithm for mobile edge computing based on deep reinforcement learning. In the mobile edge computing network architecture diagram, there are a total of M MEC nodes, each connected to a base station with limited computing and storage capabilities. Each user can only be served by one service on an MEC, and each MEC has a service range. To address the issue of timely response to sudden clustering of multiple users and their diverse service demands, a dynamic service deployment scheme is obtained using the DRF-SF-V1 algorithm based on the computational requirements provided by the users.

[0065] The specific steps are as follows:

[0066] Step 1: Establish the system model and QoS model based on the mobile edge computing network architecture diagram, and initialize the parameters;

[0067] There are a total of M MEC nodes, and the node set is M = {m j}, where m j Let U represent the number of users within the range of the base station connected to the j-th node by the MEC node, represented by the set U = {u...} i Let} represent, where u i Let represent the i-th user. Assuming the system runs continuously for a period of time [0, T], we divide the time axis into T+1 discrete time slots t∈T={0,1,2,…,T}. The set of user service requests is represented by S, S={s h}, where s h This represents the h-th service. Indicates user u i Whether to use the edge server in time slot t. j Deployed services h , if u i Use service s within time slot t. h ,but otherwise, use This indicates that the edge server m is placed j The service set on, of which use Represents a set The set of users served. For each edge server, the following is adopted: Represents edge server m j The preset total storage resources, i.e., edge server m j The service resource threshold is adopted. Represents edge server m j The storage resources currently in use, when At that time, the node is defined as a congested node. The QoS model consists of three parts: computation latency, communication latency, and migration latency. Computation latency uses... To indicate, the parameter α needs to be set. i (t) represents user u i The amount of computing resources required for a service request within time slot t. m j The computing power. Communication delay. To indicate, among which This indicates the transmission rate. γ represents the channel bandwidth of the physical link, and γ represents the user u. i The transmission power. Indicate ui With m j The channel gain between them, where N is the noise power. Indicates the size of the requested data. Migration delay is used. Similarly, parameters are set for it. This is a Boolean variable representing user u. i services h Whether to perform migration. ρ(s) h ) for service s h The migration delay time.

[0068] Step 2: Establish a joint optimization model for latency and migration cost;

[0069] Cross-server migration incurs migration costs, which are then used to calculate these costs. To indicate, For service s h The unit cost during migration. The optimization objective of this example is to minimize the weighted average of total latency and total cost for users in set U over a continuous time period [0, T]. Total latency and total cost are respectively represented by... and It means that, among them To adjust the coupling relationship between latency and cost, this invention performs normalization processing, which measures the total latency d of users in set U within a continuous time period [0,T]. T And total cost c T Mapped to the range [0,1] for processing, the total latency and total overhead are respectively and in, and These represent the maximum delay and cost within the continuous time period [0, T], respectively. and These represent the minimum latency and cost within the continuous time period [0, T]. That is, the optimization model in this example is minimizeωD + (1-ω)C, where ω represents the user's preference for latency.

[0070] Step 3: Build a dynamic service placement framework based on deep reinforcement learning

[0071] In this example, we need to set up the state and action spaces, and the reward function used in the reinforcement learning framework. State space S t It is by s t =[o t ,e t A vector composed of ] . Where o t It is the remaining storage vector, e t It is the position vector on each user's trajectory. Action space A tIt is A t =[a1,a2,…,a h ,…,a n ] t The vector formed by a, where a h (t)=[a h (t) - ,a h (t) + ] indicates service s h The range of candidate edge servers during the migration process in time slot t. Reward function. The baseline is defined as the total cost to users who consistently migrate to the new policy. The total user cost given for the DRF-SF-V1 algorithm.

[0072] Step 4: Use the DRF-SF-V1 algorithm to solve the optimization problem in the original model.

[0073] In this embodiment, preliminary parameters of the reinforcement learning agent need to be set. The destination of the migration is determined by running the current policy network and exploring noise. The algorithm is then used to iteratively solve the problem until the result converges. Specific steps are detailed above. As can be seen from the experimental figures, compared to the comparative algorithms, the proposed deep reinforcement learning-based dynamic service deployment algorithm DRF-SF-V1 has a lower total cost and converges faster.

Claims

1. A method for dynamic deployment of mobile edge computing services based on deep reinforcement learning, wherein the applicable network architecture is a model consisting of cloud servers, switches, and base stations; each base station has limited computing and storage capabilities, connects to one MEC server, and each MEC has a service range; services are deployed on virtual machines, and each user can only be served by one service on the MEC; characterized in that... Includes the following steps: S1. Establish system and QoS models based on network architecture. The system model is as follows: Assume there are a total of M MEC nodes, and the node set is M = {m j }, where m j This represents the j-th node; each MEC node connects to a base station with limited computing and storage capabilities, and the edge server m... j computing power This indicates that the edge server m j Storage capacity The set of users served by a MEC node is represented by U = {u i Let} represent, where u i This represents the i-th user; The system runs continuously for a period of time [0,T], dividing the time axis into T+1 discrete time slots t, where t∈T={0,1,2,…,T}; the set of user service requests is represented by S={s h Let} represent, where s h Represents the h-th service; using Indicates user u i Whether to use the edge server in time slot t. j Deployed services h , if u i Use service s in time slot t h ,but otherwise, use This indicates that the edge server m is placed j The collection of services on, among which use Represents a set The set of users served; For each edge server, adopt Represents edge server m j The preset total storage resources, i.e., edge server m j The service resource threshold is adopted. Represents edge server m j The storage resources currently in use, when When this happens, the node is defined as a crowded node; The QoS model is as follows: using d c (i,t) represents user u i Calculate the delay within time slot t; α i (t) represents user u i The amount of computing resources required to serve a request within time slot t; Indicates user u i Whether to use the edge server in time slot t. j Deployed services h ; Represents edge server m j The computing power; the computing latency is: Communication latency occurs when a user and the requested service are located in different regions and communicate; maximum transmission rate. for: Among them, using γ represents the channel bandwidth of the physical link, where u is... i The transmission power; Indicate u i With m j The channel gain between, where Assuming service s h Deployed on edge node m j Above, then For user u i To edge node m j The distance between them; N is the noise power; This refers to the data size when passing through network devices between MEC nodes and service-providing nodes; data transmission is determined by the bandwidth of the physical link. and data size Therefore, the communication delay d l (i,t) is: Consider optimizing user experience through dynamic service migration; define a boolean variable. Indicates user u i services h Whether migration occurs within time slot t; ρ(s) h ) for service s h The migration latency includes the transfer of service configuration files and the restart of software resources; the user's migration latency d u (i,t) is defined as: S3. Establish a joint optimization model oriented towards latency and cost; Assume the cost of dynamic service placement is incurred by service migration across edge servers; to meet the Quality of Service (QoS) requirements of users with uncertain activity trajectories, the services required by users can be dynamically migrated to adapt to user mobility. h From edge server m a Migrate to m a' The resulting costs It means that, among them For service s h The specific formula for calculating the unit cost incurred during the migration process is as follows: The total cost within time slot t is: The total cost for users in set U over a continuous time interval [0, T] is: The total delay within time slot t is: Where d c (i,t), d l (i,t), d u (i,t) represent the user's computation latency, communication latency, and migration latency, respectively; The total latency for users in set U over a continuous time interval [0, T] is: The total delay d for users in set U within the continuous time interval [0,T] is given by... T And total cost c T The processing is mapped to the range [0,1], so the total latency and total cost are as follows: in, and These represent the maximum delay and cost within the continuous time period [0, T], respectively. and These represent the minimum delay and cost within the continuous time period [0,T], respectively. The optimization problem is represented as follows: minimizeωD+(1-ω)C (12) The weight is ω, which represents the user's preference for latency or cost. The value of ω ranges from [0,1]. Equation (13) represents the cost constraint, where |S| represents the number of services. Indicates service s h From edge server m a Migrate to m a' The costs incurred require that the total cost of providing the service cannot exceed a threshold Γ, where Γ represents the upper limit of the maximum total cost that the operator can provide; Equation (14) represents the physical resource constraint, where |s h | indicates that it is placed on the edge server m j The number of services on the edge server must not exceed the storage capacity. The computing resources required for a service request cannot exceed the computing capacity. Where α i (t) represents user u i The amount of computing resources required to serve a request within time slot t; Equation (15) represents the amount of computing resources required by user u. i Whether to use the edge server m within time slot t j Deployed services h .

2. The method for deploying dynamic mobile edge computing services based on deep reinforcement learning according to claim 1, characterized in that: The solution method for the joint optimization model in step S3 is the DRF-SF-V1 algorithm, and the execution process specifically includes: Step 1: Predict the trajectory based on the Microsoft GPS trajectory dataset; observe the user's activity trajectory in the dataset and mark the origin center coordinates; divide the area with this location as the center point and r as the radius; traverse the user trajectory and find the trajectory within the area within 60 consecutive time slots; Step 2: Randomly initialize the actor network (μ(S)) t |θ μ )) and critic network (Q(S,a|θ) Q ), where θ μ These are the actor network weights, θ Q These are the critic network weights; the weights are initialized to θ. μ' and θ Q' Target actor(μ(S) t |θ μ' )) network and critic network (Q(S,a|θ) Q' Initialize memory storage area B; Step 3: Initialize the environment parameters of the edge server and users, including the action space dimension a_dim and the state space dimension s_dim, and generate the initial state S1; where the action space dimension and the state space dimension refer to the state space S1 and S2, respectively. t and action space A t The length of the vector; Step 4: Select an action a t (S t |θ μ )+∈ t , where θ μ For the current policy network, ∈ t To explore noise, obtain immediate rewards and new states S t' And store them in memory storage area B; Step 5: Pre-training. When the storage capacity of memory storage area B is full, sampling training will be performed again. Step 6: In each iteration, perform trajectory prediction based on the LSTM trajectory prediction model, perform pre-transfer, record the migration trajectory, find congested nodes, and construct a set of congested nodes. Step 7: Reduce the remaining available resources of servers around the congested node, reducing the capacity to the storage resource capacity currently used by the server, so as to ensure that the servers around the congested node can appear to have no available resources, thus reserving resources for the congested node. Step 8: Perform conflict detection on edge nodes. If a resource request conflict occurs on the node, determine the set of conflicting users and calculate the estimated cost Y for each user in the set. Where d u (i,t) represents the migration delay, and d l (i,t) represents the communication delay; the user with the largest Y value in the conflicting user set is selected for migration. The selection principle for the migration location is to prioritize the edge node closest to the user. If the edge node has no spare resources, then the edge node closest to the user and with spare resources among the remaining nodes is selected. Step 9: Calculate latency. If the service migrates within time slot t, calculate the migration latency and overhead. If no migration occurs in time slot t but the server hosting the service is not the server hosting the user, only calculate the communication latency. Otherwise, the total latency is 0. Step 10: For user set U, output the current service placement strategy A. t Total overhead, release reserved resources.

Images