Task migration and service placement method and system based on edge computing architecture

Abstract

The invention provides a task migration and service placement method and system based on an edge computing architecture, and the method comprises the steps: carrying out the modeling of communication channels of all devices and an edge layer, calculating channel parameters based on a modeled model, considering the total cost of all devices based on the calculated parameters, and formulating a joint resource optimization problem of task migration and service placement; a joint resource optimization problem is converted into a Markov decision process model, state spaces of intelligent agents are designed, a service placement allocation decision, a computing resource allocation decision, a computing resource allocation proportion and a wireless channel bandwidth proportion allocation decision are used as action spaces of each intelligent agent, and based on the goal of the joint resource optimization problem, the service allocation decision and the computing resource allocation proportion and the wireless channel bandwidth proportion allocation decision are calculated. Minimizing the total cost of each device as a reward function, and iteratively performing optimization solution by using a task migration and service placement optimization algorithm on the edge node to obtain a final task migration and service placement strategy. According to the invention, the robustness of the system can be improved while the total cost of task processing of the system is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of the integration of distributed computing and machine learning, and particularly relates to a task migration and service placement method and system based on an edge computing architecture. Background Technique

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, in the era of the unprecedented development of the Internet, a huge number of user terminals globally generate a large amount of task processing data every day. If the traditional cloud computing center model is used, all user task data will be transmitted to the central cloud server for execution, which not only causes a huge burden on the cloud server but also generates unexpected delays and computing costs. As an extended development technology of cloud computing, edge computing significantly reduces the response time of complex tasks of terminal devices by deploying high-performance computing servers close to IoT terminal devices, greatly optimizing the resource utilization rate of the overall network and improving the overall robustness of the system.

[0004] DRL (Deep Reinforcement Learning) combines DL (Deep Learning) and RL (Reinforcement Learning) and has made significant progress in many fields in recent years. RL aims to learn decision-making strategies by interacting with the environment, while DL can extract meaningful features from raw data. The application of DRL in wireless network resource allocation has received increasing research attention.

[0005] Edge computing technology can be traced back to 1997. In that year, Brian Noble et al. from the Massachusetts Institute of Technology (MIT) put forward the ideas of disconnected operation and proxy-based computing in a mobile computing environment, forming the initial theoretical basis of edge computing technology. The concept of "edge computing" was truly clearly put forward in 2009. In that year, based on considering the architecture of the Content Delivery Network (CDN), Akamai proposed to migrate computing tasks from the central cloud to the network edge closer to users to reduce latency and optimize computing efficiency. In 2014, the European Telecommunications Standards Institute (ETSI) officially proposed the concept of "Mobile Edge Computing (MEC)" in the ETSI white paper, laying the importance of MEC technology in 5G. Since then, edge computing technology has become a key part of the 5G standardization process, combined with 5G key technologies such as network slicing and ultra-reliable low-latency communication (URLLC) to jointly promote the development of edge computing.

[0006] Many early studies on edge computing adopted heuristic algorithms and greedy algorithms to solve this problem. For example, Wang et al. comprehensively considered the computing, caching, and communication issues in the Mobile Edge Computing (MEC) environment, focused on the application of heuristic algorithms in resource scheduling, and proposed a computing offloading scheme based on genetic algorithms. Mao et al. proposed a task offloading strategy based on the greedy algorithm based on the communication architecture of Mobile Edge Computing (MEC), and gradually selected the server with the lowest computing cost through the greedy strategy to optimize the computing resource allocation. Chen et al. focused on the energy consumption utilization rate of computing resource allocation and proposed an improved Ant Colony Optimization (ACO) for multi-user task offloading. These methods have relatively low computational complexity, but due to their dependence on local optimal solutions, it is difficult to adapt to system dynamic changes, and they are prone to falling into local optimal solutions and cannot effectively solve problems in large-scale systems.

[0007] In recent years, the research on Edge Computing (EC) technology has received increasing attention. Researchers in this field usually migrate the computing tasks in IoT terminal devices to edge devices for execution to achieve the purpose of reducing latency or energy consumption. With this as the research goal, the research directions of researchers in recent years can be roughly divided into the following three categories:

[0008] The first type of research focuses on optimizing the latency of devices in the edge computing (EC) environment as the key objective, mainly targeting latency-sensitive scenarios to meet the optimization requirements of latency-sensitive IoT devices. For example, Ren et al. proposed a joint computing resource allocation and wireless resource management strategy based on convex optimization to minimize the total latency of computing offloading in the mobile edge computing (MEC) environment. Le et al. designed a resource allocation model based on latency optimization and adopted a low-complexity heuristic algorithm to improve the task scheduling efficiency. The final experimental results can effectively reduce the total latency of computing tasks. Chi et al. studied an MEC task offloading framework suitable for heterogeneous network environments and designed a task offloading optimization strategy based on dynamic network conditions, which can improve the offloading efficiency and reduce latency under various network conditions.

[0009] The second type of research focuses on minimizing the energy consumption of devices in the edge computing (EC) environment as the main optimization objective. Researchers in this area mainly focus on latency-sensitive scenarios and usually optimize latency-sensitive IoT devices. For example, the authors Zhang et al. studied the task offloading problem in the fog computing environment, constructed a task offloading framework for minimizing energy consumption (FEMTO), and used a mixed integer programming method to well solve the problem of minimizing the energy consumption allocation of computing tasks among fog nodes. Dai et al. jointly optimized the computing offloading and user association framework and, based on the optimization algorithm of reinforcement learning, minimized the total energy consumption of tasks while ensuring the lowest latency requirement. Zhao et al. studied the user association and computing task offloading problem in the fog computing-supported device-to-device (D2D) network and proposed a low-complexity online algorithm based on Lyapunov optimization, taking into account both computing efficiency and the optimal allocation of network resources to achieve the goal of minimizing energy consumption.

[0010] The third type of research comprehensively considers the impacts of latency and energy consumption in the edge computing (EC) environment. Researchers usually use the weighted sum of latency and energy consumption as the optimization objective to meet the optimization requirements of IoT devices in comprehensive scenarios. For example, the authors Zhao et al. proposed a computing offloading strategy based on reinforcement learning in the computing offloading problem of the fog radio access network (Fog RAN) to minimize the weighted sum of the execution delay of computing tasks and the system energy consumption. Liu et al. proposed an optimization method based on game theory to solve the comprehensive optimization problem of tasks among different computing levels (terminals, fog nodes, cloud) in the fog computing and cloud computing network supported by non-orthogonal multiple access (NOMA) technology. Zhang et al. studied the task offloading problem in the mobile edge computing (MEC) environment and proposed a multi-objective optimization method that balances device energy consumption and latency, which well reduces the total cost of the system.

[0011] Although great progress has been made in the field of edge computing at home and abroad, there are still many problems that need to be studied more deeply in this area. For example, there is still much room for improvement in aspects such as the lack of effective utilization of the storage space of edge servers in edge systems, the lack of service placement algorithms, and the comprehensive consideration of cloud servers and edge servers. With the development of deep reinforcement learning (DRL) in recent years, new solutions have been provided for such problems. However, there are still many problems in the application of this new technology in the actual environment, such as difficult convergence and poor algorithm adaptability. More improved research is needed on deep reinforcement learning (DRL) in edge computing (EC) technology. Summary of the Invention

[0012] In order to solve the above problems, the present invention proposes a task migration and service placement method and system based on an edge computing architecture. The present invention can achieve the improvement of the robustness of the system while reducing the total cost of system task processing, and effectively improve the convergence effect and stability of the deep reinforcement learning algorithm in discrete task allocation and service placement scenarios.

[0013] According to some embodiments, the present invention adopts the following technical solutions:

[0014] A task migration and service placement method based on an edge computing architecture, comprising the following steps:

[0015] Model the communication channels of each device and the edge layer. Based on the modeled model, calculate the channel parameters. Based on the calculated parameters, consider the total cost of each device, and formulate a joint resource optimization problem for task migration and service placement;

[0016] Convert the joint resource optimization problem into a Markov decision process model. Design the service placement state in the previous time slot, the total cost in the previous time slot, and the channel gain between the edge node and all devices as the state space of the agent. Use the service placement allocation decision, computing resource allocation decision, computing resource allocation ratio, and wireless channel bandwidth ratio as the action space of each agent. Based on the objective of the joint resource optimization problem, use the minimum total cost of each device as the reward function, and use the task migration and service placement optimization algorithm on the edge node to iteratively optimize and solve to obtain the final task migration and service placement strategy.

[0017] As an alternative implementation, the modeled model includes three layers. The first layer is the IoT device layer, which includes multiple IoT devices. Each IoT device can generate computing tasks and offload them to the edge node or execute them locally;

[0018] The second layer is the edge layer, which includes a single edge node configured with an edge server connected to a cloud server;

[0019] The third layer is the cloud layer, which is configured with cloud servers and stores images of all types of services, and can migrate and place the images of corresponding types of services on edge servers;

[0020] The model time is discretized into multiple time slots. Within each time slot, computing tasks are randomly generated. The computing tasks can be executed locally, or offloaded to edge nodes or cloud servers for execution, and the execution location is decided by the agent;

[0021] The model includes multiple different types of heterogeneous services, and the service placement decision is used to determine the number and type of services to be placed on edge nodes.

[0022] As an alternative implementation, the process of calculating channel parameters includes the delay of local execution of computing tasks and the energy consumption required to process this task; when the task is migrated for execution, the transmission rate from the IoT layer device to the edge node, the delay and energy consumption of offloading the task to the edge node for execution; when the service placement is executed, the transmission rate from the cloud server to the edge node, the placement delay and energy consumption.

[0023] As an alternative implementation, the total cost of each device includes the total migration cost, total delay cost, and total energy consumption cost of migrating the service to the edge node within a set time slot.

[0024] As an alternative implementation, the process of formulating the joint resource optimization problem of task migration and service placement includes: the objective function of the joint resource optimization problem of task migration and service placement is to minimize the long-term average weighted total cost of each device;

[0025] The constraint conditions include the edge computing resource occupancy rate and wireless communication bandwidth occupancy rate constraints, the storage upper limit constraint of service images placed on edge nodes, the service placement constraint, and the weighted weight range constraint.

[0026] As an alternative implementation, the process of converting the joint resource optimization problem into a Markov decision process model includes: defining the state space of the Markov decision process model considering three parts: the channel gain in the current time slot, the service placement situation in the previous time slot, and the weighted sum of energy consumption, delay, and migration cost after the completion of the task set in the previous time slot;

[0027] Defining the action space as the set of service placement allocation decisions, computing resource allocation decisions, computing resource allocation ratio, and wireless channel bandwidth ratio;

[0028] Defining the reward as the negative of the weighted sum of energy consumption, delay, and migration cost after the completion of the task set within the corresponding time;

[0029] And the decision goal is to maximize the long-term cumulative reward.

[0030] As an alternative implementation, the process of using the task migration and service placement optimization algorithm on the edge node for iterative optimization includes: The task migration and service placement optimization algorithm on the edge node includes a main network and a target network. The target network is used to generate the target Q value, and the main network is used to make action decisions, evaluate the Q values of all actions in the current state, and select the optimal action according to the size of the Q value.

[0031] Furthermore, the main network is updated by minimizing the temporal difference error based on prioritized sampling. The parameters of the target network adopt a periodic soft update mechanism. Every time a set number of steps are reached, the parameters of the target network are updated to make them close to the parameters of the main network. Through the collaborative learning of the main network and the target network, the parameters converge to the optimal state, achieving the optimal resource allocation and task migration strategy.

[0032] A task migration and service placement system based on an edge computing architecture includes:

[0033] An optimization problem formulation module, configured to model the communication channels of each device and the edge layer, calculate channel parameters based on the modeled model, and consider the total cost of each device based on the calculated parameters to formulate a joint resource optimization problem for task migration and service placement;

[0034] An optimization problem solving module, configured to convert the joint resource optimization problem into a Markov decision process model, design the service placement state in the previous time slot, the total cost in the previous time slot, and the channel gains between the edge node and all devices as the state space of the agent, use the service placement allocation decision, the computing resource allocation decision, the computing resource allocation ratio, and the radio channel bandwidth ratio as the action space of the agent, and use the minimum total cost of each device as the reward function based on the objective of the joint resource optimization problem, and use the task migration and service placement optimization algorithm on the edge node for iterative optimization to obtain the final task migration and service placement strategy.

[0035] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps in the above method are completed.

[0036] An electronic device includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps in the above method are completed.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] The task migration and service placement method based on the edge computing (EC) architecture of the present invention. The proposed edge computing task migration and service placement algorithm based on deep reinforcement learning (hereinafter referred to as DRL-SPTM) is used for the collaborative optimization of computing task offloading decisions and service instance deployment in a dynamic environment, which can effectively reduce the total cost of IoT terminal devices, effectively improve the robustness of the system, and effectively improve the convergence effect and stability of the DRL algorithm in discrete task allocation and service placement scenarios.

[0039] The present invention can solve the discrete task allocation and service placement problems, provides a highly feasible solution for the application of deep reinforcement learning algorithms in edge computing, and has good application prospects.

[0040] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0042] Figure 1 It is a system model diagram of an embodiment;

[0043] Figure 2 It is a flowchart of an edge computing task migration and service placement algorithm of an embodiment;

[0044] Figure 3 It is a schematic diagram of an edge computing task migration and service placement algorithm model of an embodiment;

[0045] Figure 4 It is a comparison diagram of the convergence of the loss function under different learning rates of an embodiment;

[0046] Figure 5 It is a comparison diagram of the loss values of DRL-SPTM and DQN under different loads of an embodiment;

[0047] Figure 6 It is a comparison diagram of the total cost of different edge node computing capabilities of an embodiment;

[0048] Figure 7 It is a comparison diagram of the total cost of different task quantities of an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be further described below in conjunction with the drawings and embodiments.

[0050] It should be noted that the following detailed description is illustrative and aims to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention pertains.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] In the case of no conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0053] Embodiment 1

[0054] A task migration and service placement method based on an edge computing (EC) architecture, comprising the following steps:

[0055] Step (1): Model the direct communication channels between IoT devices and edge layer servers, calculate the signal-to-interference-plus-noise ratio and transmission rate, and then consider the local total cost, migration total cost, and service placement total cost of IoT devices. In this embodiment, the local total cost is the sum of the energy consumption and delay costs, and formulate a joint resource optimization problem for task migration and service placement.

[0056] Step (2): Design the key elements of the deep reinforcement learning algorithm. According to the joint resource optimization problem of task migration and service placement, design the service placement state in the previous time slot, the total cost in the previous time slot, and the channel gains between edge nodes and all IoT devices as the state space of the agent, while the service placement allocation decision, computing resource allocation decision, computing resource allocation ratio, and wireless channel bandwidth ratio are used as the action space of each agent; and design a reward function based on the objective of the joint optimization problem to minimize the total cost of IoT devices.

[0057] Step (3): Based on the task migration and service placement optimization algorithm (DRL-SPTM) on edge nodes, through multiple rounds of training, continuously optimize the main network and the target network to obtain a task migration and service placement strategy that minimizes the total cost of IoT devices.

[0058] Step (4): The edge computing (EC) system environment migrates the tasks of IoT devices according to the task migration and service placement strategy optimized in step (3), and deploys the services to the edge nodes according to the generated service placement decision.

[0059] The following is a detailed description:

[0060] First, in step (i), in this embodiment, as Figure 1 shown, in the system model, consider a three - layer edge computing system consisting of a single edge node. In the IoT device layer of the first layer, there are N IoT devices, denoted as the set N = {1, 2,..., N}. Each user device n ∈ N can generate a computing task and offload it to the edge node or execute it locally.

[0061] In the edge layer of the second layer, it contains a single edge node (ES), and a base station (or called an edge server) equipped with a high - performance server on the edge layer serves as the edge node.

[0062] The third layer is the cloud layer, where high - performance cloud servers are deployed and images of all types of services are stored. When the agent determines that offloading the service of type k to the edge node can bring better long - term benefits, the cloud server in the cloud layer will migrate and place the image of the service of type k on the edge server.

[0063] There are K different types of heterogeneous services in the system, denoted as the set K = {1, 2,..., K}. Each service k ∈ K requires a certain amount of storage space d k to store relevant software and data.

[0064] Time in the system is discretized into several time slots, indexed as T = {1, 2,…, T}. In each time slot t ∈ T, the user device n will randomly generate a computing task of type k, i ∈ k, and the total number of services of type k generated in time slot t is m k (t). The task can be executed locally, or offloaded to the edge node or the cloud server for execution. In order to execute the task of a certain service, the corresponding service must be deployed on the edge node in advance. The deployment and migration of services will incur additional overheads, such as data transmission and initialization time.

[0065] When the IoT device layer generates a task, the agent will decide where to offload each task for execution, that is, the task offloading decision, represented by the binary variable Y i (t), where Y i (t) indicates whether to migrate to the edge server for execution. When Y i (t) = 1, the IoT device needs to confirm whether the service k has been placed on the edge node. If the service of type k has been placed, the task migrates to the edge node for execution. If the service k has not been placed, the cloud server will place the service k on the edge server, which incurs additional overheads.

[0066] In the system model, the service placement decision determines which services to place on the edge node (ES). Represented by the binary variable X k(t) indicates whether service k is placed on the edge node, where X k (t) = 1 indicates that service k has been placed, X k (t) = 0 indicates that it has not been placed.

[0067] (1) Local execution

[0068] If task i is executed locally, that is, Y i (t) = 0. For a given decision, the execution delay and energy consumption of the task can be calculated. The delay of the task executed locally can be expressed as:

[0069]

[0070] The energy consumption required to process this task can be expressed as:

[0071]

[0072] where c i represents the number of CPU cycles required for task i, is the CPU frequency of the user device, and κ is the energy consumption coefficient. Since local computing does not require the migration process, there are no issues with task transmission delay and energy consumption.

[0073] (2) Migration execution (service has been placed)

[0074] If task i is executed on the edge node (ES), that is, Y i (t) = 1. The task system will first confirm that on the edge node, if service k (i ∈ k) has already been placed on the edge node, then task i will migrate to the edge node for execution.

[0075] Because the transmission rate of each device to the edge node is affected by factors such as bandwidth, transmission power, and noise power, the transmission rate of IoT layer device n to the edge node is as follows:

[0076]

[0077] where B represents the bandwidth of the edge node; is the transmission power when user n uploads data; H n represents the channel gain of user n in the wireless channel; N 0 represents the noise power of the channel; g up represents the target bit error rate, and Γ(g up ) represents the signal-to-noise ratio margin introduced to meet the target bit error rate of the uplink; d n represents the physical distance between user n and the edge node; ξ is the exponent describing the degree of transmission path loss of the signal; λ i represents the bandwidth ratio allocated by the system.

[0078] Under the assumption that the uplink and downlink have consistent channel characteristics and noise conditions, the transmission rate of the downlink can be expressed as:

[0079]

[0080] The delay of task offloading to the edge node for execution includes transmission delay and execution delay. Meanwhile, the required energy consumption includes transmission energy consumption and processing energy consumption. The transmission delay of task i offloaded to the edge node during migration execution can be expressed as:

[0081]

[0082] Meanwhile, the transmission energy consumption of the IoT device can be obtained:

[0083]

[0084] where p n is the transmission power of IoT device n in the uplink.

[0085] Similarly, after the edge node completes the computing task, it will return the computing result to the IoT terminal device. The transmission delay for the edge node to send the computing result to IoT device n is:

[0086]

[0087] where l i out is the size of the computing result returned after the edge node completes the calculation.

[0088] When the edge node receives the task data from IoT device n, the execution delay of task offloading to the edge node for execution can be expressed as:

[0089]

[0090] According to the CPU frequency allocated by the edge node to task i and the CPU cycles required by the task, η i represents the proportion of allocated resources. We can obtain the task execution energy consumption:

[0091]

[0092] Combining equations (5), (7), and (8), we can obtain the total delay of IoT device n migrating task i to the edge node as:

[0093]

[0094] Since there is standby power loss while the IoT terminal device n waits for the processing result, we can obtain the local standby energy consumption of the terminal device n:

[0095]

[0096] Where is the standby power when the IoT terminal device n waits for the processing result.

[0097] Combining equations (6), (8) and (11), we can obtain the total energy consumption of the terminal device during the migration of task i from the terminal device n to the edge node:

[0098]

[0099] (3) Service Placement

[0100] When the agent of the edge node evaluates the task scheduling strategy at the current moment, if it determines that offloading the service of type k to the edge node can bring better long-term benefits, the system will perform the task migration operation and place the image of the service of type k on the edge node (ES) to optimize the overall resource utilization and improve the system performance.

[0101] According to the current service placement decision, the edge node needs to migrate the required but unplaced services from the cloud server to the edge node (ES). The required and already placed services can be retained, while the unnecessary and already placed services can be removed. To migrate a specific service from the cloud, it takes a certain amount of time to download its relevant data and instantiate the container on the edge node, which is called the service placement delay.

[0102] The downlink transmission rate of the image of the service of type k from the cloud server to the edge node can be expressed as:

[0103]

[0104] Where P cloud is the transmission power of the cloud server for sending the image, and d cloud is the physical distance between the cloud server and the edge server (ES).

[0105] Since the service placement delay mainly depends on the transmission time of its relevant data between the cloud and the edge node (ES), based on the service placement decisions in the previous and current time slots (i.e., X k (t) and X k (t - 1)), the placement delay of service k can be expressed as:

[0106]

[0107] Due to the standby power loss of IoT terminal device n waiting for the processing result, the power loss of IoT devices with service placement delay is expressed as:

[0108]

[0109] where D k is the image size of service k, and I {·} is an indicator function. If the condition in the parentheses is satisfied, the value of the indicator function is equal to 1; otherwise, it is equal to 0.

[0110] Combining equations (14) and (15), the total placement cost of the service migrated to the edge node (ES) within time slot t can be obtained as:

[0111]

[0112] where α represents the weight of the delay cost in the total cost.

[0113] (4) Optimization problem description

[0114] Combining equations (1) and (10), the total delay cost within time slot t can be expressed as:

[0115]

[0116] Combining equations (2) and (12), the total energy consumption cost within time slot t can be expressed as:

[0117]

[0118] Based on the above system model and related definitions, under the constraint conditions of the variables, by jointly optimizing the migration decision Y i (t), the service placement decision X k (t), the proportion of edge node computing resource allocation η i , and the proportion of wireless channel bandwidth λ i , the following optimization problem is constructed:

[0119]

[0120]

[0121] Y i (t) ≤ X k (t)(19d)

[0122] α, ζ, η i , λ i ∈(0, 1)(19e)

[0123] Y i (t), X k(t) ∈ {0, 1}(19f)

[0124] In the optimization problem P1, the objective function represents minimizing the long - term average weighted cost sum, including delay, energy consumption, and migration cost. The constraint conditions (19a) and (19b) indicate that the occupancy rates of edge computing resources and wireless communication bandwidth cannot exceed 1, and the constraint condition (19c) indicates that the service images placed at the edge nodes cannot exceed their maximum storage limit. The constraint condition (19d) indicates that for task type k migration to be executed at the edge node, service k must be placed first. The constraint conditions (19e) and (19f) represent the value ranges of the relevant variables, where ζ represents the weight of the migration cost.

[0125] In step (ii), according to the joint resource optimization problem of task migration and service placement, the environmental state space, action space, reward function, etc. in the deep reinforcement algorithm are designed.

[0126] Specifically, (1) Markov decision process (MDP)

[0127] A typical Markov decision process model can be represented by a tuple containing five elements, namely <S, A, P, R, γ>. Among them, S and A represent the state space and action space respectively. The probability transition distribution P describes the probability that the current state of the environment transfers to another state given an action. Since the environment considered in this embodiment is an edge computing system and it is model - free, the transition probability is not required. R represents the reward function, and γ ∈ [0, 1] is a discount factor used to represent the degree of attention to future rewards. Next, in this embodiment, by defining each element in the tuple, the online computing resource allocation problem and service placement are formally modeled as a Markov decision process model.

[0128] (2) State space design

[0129] Each state S(t) ∈ S contains the current information observed from the edge computing system, and this information affects the reward obtained from the environment after taking an action. In the model of this embodiment, the state space is defined as:

[0130] S(t) = {X(t - 1), C(t - 1), H(t)} (20)

[0131] Among them, the state space we consider contains three parts. H(t) represents the channel gain in the current time slot, X(t - 1) represents the service placement situation in the previous time slot, and C(t - 1) represents the weighted sum of energy consumption, delay, and migration cost after the completion of the task set in the previous time slot, which can be expressed as:

[0132] C(t) = (ζT total (t)+(1 - ζ)E total (t))+ηCpla (t) (21)

[0133] (3) Action space design

[0134] In each time slot t, the edge node needs to make joint service placement and computing resource allocation decisions according to the current state, that is, A(t) ∈ A. The action space is defined as:

[0135] A(t) = {η i (t), λ i (t), Y(t), X(t)} (22)

[0136] where η i (t) represents the proportion of computing resource allocation for task i, λ i (t) represents the wireless channel bandwidth occupancy for task i, Y(t) represents the task migration decision, X(t) represents the service placement decision, and η i (t) represents the proportion of computing resource allocation for task i, λ i (t) represents the wireless channel bandwidth occupancy for task i.

[0137] (4) Reward function design

[0138] After taking action A(t) in state S(t), the edge node will obtain a reward R(t) from the edge computing system. In the model of this embodiment, we define the reward as:

[0139] R(t) = -C(t) (23)

[0140] where C(t) represents the weighted sum of energy consumption, delay, and migration cost after the completion of the task set in time slot t.

[0141] (5) Algorithm objective

[0142] The edge node aims to find an optimal policy π* to maximize the long-term cumulative reward:

[0143]

[0144] where γ ∈ [0, 1] is the discount factor, which is used to balance short-term and long-term benefits.

[0145] In the solution process of step (three), as Figure 2 、 Figure 3 shown, this embodiment uses the Dueling architecture to optimize the deep learning network, and the Q-value calculation can be decomposed into the state value function V(S) and the advantage function A(s, a).

[0146] The DRL-SPTM algorithm mainly consists of an online network and a target network. The online network is mainly responsible for evaluating the Q-values of all actions A(t) in the current state S(t) and selecting the optimal action. Let the parameters of the online network be θ, and its Q-value function is expressed as Q online (s t ,a t ; θ). The online network iteratively updates the Q-value through the Bellman equation, which can be expressed as:

[0147]

[0148] where s t represents the current state, a t ' represents the action at time step t+1, θ t represents the parameters of the online network, and θ - is the frozen parameter of the target network; as Figure 2 shown, in the Dueling architecture, the Q-value calculation can be decomposed into a state-value function V(S) and an advantage function A(s,a). To ensure the comparability of the values of the advantage function, the advantage function is normalized:

[0149]

[0150] where |A| represents the size of the action space. The online network calculates the Q-value of each action through forward propagation and selects the action with the maximum Q-value as the optimal action:

[0151]

[0152] In the DRL-SPTM algorithm, the update mechanisms of the online network and the target network combine the core ideas of DDQN and prioritized experience replay. In the prioritized experience replay mechanism, the sample priority p i is dynamically adjusted based on the temporal difference (TD) error δ. The calculation formula of the TD error δ is as follows:

[0153] δ = y t -Q online (s t ,a t ; θ t ) (28)

[0154] where y t is the target function. The DDQN algorithm uses the following target function to alleviate the problem of overestimation of Q-values:

[0155]

[0156] When a new sample is stored in the buffer, the priority is initialized to the current maximum priority value, i.e.:

[0157]

[0158] where ò is a very small constant to avoid zero priority. After each sampling, the priority of the corresponding sample is updated according to the latest temporal difference error (TD-error):

[0159] p i ←|δ i |+ò(31)

[0160] The main network parameters θ are updated by minimizing the temporal difference error based on priority sampling. Let the samples stored in the experience replay buffer D be (s t ,a t ,r t+1 ,s t+1 ,p i ), and its loss function is defined as:

[0161] L(θ)=E[ω i ·(y t -Q online (s t ,a t ;θ t )) 2 (32)

[0162] where the importance sampling weight ω i is used to correct the bias caused by priority sampling and is defined as:

[0163]

[0164] where N is the capacity of the experience replay buffer, σ∈[0,1] controls the influence degree of the priority, β∈[0,1] is the decay factor of importance sampling, and the importance sampling weight ω gradually increases the correction intensity of the bias through the parameter β, effectively reducing the estimation bias caused by non-uniform sampling.

[0165] The target network parameters θ - adopt a periodic soft update mechanism, and the parameters θ of the target network are updated every C steps - to make them close to the main network parameters:

[0166] θ - ←τθ+(1-τ)θ - (34)

[0167] Through the efficient collaborative learning of the main network and the target network, the system can effectively reduce the overestimation of the Q value, thereby improving the stability and convergence speed of the policy. The target network is used to generate the target Q value, and the main network is used to make action decisions. By periodically updating the parameters of the main network, it gradually approaches the optimal policy. Finally, the network parameters of the system converge to the best state, achieving the optimal resource allocation and task migration strategy.

[0168] To describe the above algorithm process more clearly, the detailed algorithm process is shown in Table 1 below.

[0169] Table 1 DRL-SPTM algorithm process

[0170]

[0171]

[0172] In summary, the agent first initializes the parameters θ of the main Q network, the parameters θ - , the experience replay pool, and sets the learning rate and discount factor. In each training episode, the system initializes the environmental state, and at each time step, it collects the current system state. Then, it uses the ε-greedy strategy to select the task migration action and obtains the corresponding reward and the state of the next time step by interacting with the environment. Subsequently, the system calculates the priority of the quadruple samples containing the state, action, reward, and the next state and puts them into the experience replay pool.

[0173] When the number of samples stored in the pool reaches the upper limit, the system replaces the sample with the lowest priority with a new sample and calculates its new priority. Subsequently, sampling is performed according to the priority of the samples, the importance sampling weights are calculated, and the target Q value is calculated for the selected samples. The sample priority is updated, and the loss function L(θ) is calculated from the target Q value, the Q value, and the importance sampling weights. The parameters θ of the main network are updated using the gradient descent algorithm to continuously optimize the policy. Every C steps in the time step, the system synchronizes the parameters θ of the main network to the target network θ - above to stabilize the training process. Finally, after multiple iterations, the trained network model is saved to obtain the optimal task migration and service placement strategy to minimize the total cost of IoT devices.

[0174] Finally, in step (4), in the edge computing (EC) system environment, the hardware devices are divided into the central cloud, edge nodes, and IoT end devices. The central cloud has powerful computing capabilities but is far from the device side, and the real-time interaction with the device side is not good. Therefore, the mirror data of the service is stored in the central cloud; the edge nodes are closer to the device side, and the interaction with the device side has lower latency. Therefore, the reinforcement learning agent containing the execution module is deployed on the edge nodes.

[0175] First, at each time step, the IoT devices generate a certain number of tasks. The EC system parses the optimized task migration decisions to determine which IoT device tasks need to be migrated. Then, according to the optimized service placement strategy, the system deploys the required services on the edge nodes. After the tasks are successfully migrated and the service deployment is completed, the target tasks are completed.

[0176] This embodiment also provides a comparative experiment:

[0177] Related experiments on the task migration and service placement method based on the edge computing (EC) architecture:

[0178] In this embodiment, computer simulation experiments are carried out to verify the performance of the proposed deep reinforcement learning (DRL)-based edge computing task migration and service placement algorithm (DRL-SPTM). The simulation scenario is based on the edge computing scenario, and the total cost (energy consumption + latency) of IoT terminal devices is optimized. Unless otherwise specified, the default settings of the environmental simulation parameters are shown in Table 1.

[0179] Table 1 Simulation environment parameters

[0180]

[0181]

[0182] The software and hardware specifications for the simulation experiments are shown in Table 2. The simulation environment is implemented using Python. The policy optimization algorithm and the network model are based on the Pytorch framework, and GPU acceleration is used with CUDA.

[0183] Table 2 Experimental platform specifications and software versions

[0184]

[0185]

[0186] The learning rate in the DRL algorithm determines the step size of parameter updates during the training process of the model and is an important parameter affecting the optimization performance. Figure 4 and Figure 5 The comparison of the loss functions of the DRL-SPTM algorithm under different learning rates is given. It can be seen from the figure that too small a learning rate will lead to slow convergence and it is difficult to obtain the optimal result, while too large a learning rate may lead to local optimality due to too large a policy update amplitude. Through simulation experiments, the learning rate of DRL-SPTM is 0.001, achieving good convergence performance. In addition, the experimental settings of DRL-SPTM include an experience replay pool capacity of 10000, a sampling batch size of 32 each time, a soft update coefficient of 0.005, a discount factor of 0.99, and the Adam optimizer is used.

[0187] In addition, Figure 6 It shows the comparison of the loss function convergence effects of the DRL-SPTM algorithm and the DQN algorithm under low load, default configuration, and high load. It can be observed that although the DQN algorithm converges faster in the first 30 rounds of training, the final loss value converges to around 300 and then no longer decreases and there are large fluctuations; the convergence effect of the DRL-SPTM algorithm after 30 rounds is significantly better than that of the DQN algorithm, with an overall lower loss value, tending to be stable, and smaller fluctuations. Judging from the convergence and stability of the loss function, the DRL-SPTM algorithm is significantly better than the traditional DQN algorithm.

[0188] In this embodiment, the DRL-SPTM algorithm will be compared with four other different algorithm mechanisms, namely "LocalFully", "Offloading Fully", "Greedy", and "DQN". Among them, the "Local Fully" algorithm means that all tasks generated by IoT devices are executed locally; "Offloading Fully" means that all tasks are migrated to the server on the edge node for execution; "Greedy" means the greedy algorithm, which makes the optimal choice in the current state at each step, but cannot guarantee the global optimal solution; "DQN" means using the basic deep Q network to make migration and service placement decisions.

[0189] Figure 6 It shows the influence degree of different algorithms on the total cost under different edge node computing capabilities. First, it can be seen from the figure that since all tasks of the "Local Fully" algorithm are executed locally, the change in the computing power of the edge node has no effect on it. Except for the "Local Fully" algorithm, the total costs of the other four algorithms generally decrease as the computing power of the edge node increases. Starting from the lowest 2Gcycle / s, the four algorithms all show a relatively fast downward trend, and then the rate of decrease in the total cost slows down. When the computing power increases to 4Gcycle / s or more, the total costs of all algorithms are less than that of the "Local Fully" algorithm. In the comparison of different algorithms, it can be seen that the DRL-SPTM algorithm proposed in this embodiment is significantly better than other algorithms when it is greater than or equal to 3Gcycle / s, followed by the "DQN" algorithm, and the "Greedy" algorithm, although it is worse, has a better effect than "Local Fully" and "Offloading Fully".

[0190] Figure 7It shows the comparison of the total costs of different algorithms with different numbers of tasks generated by each IoT device. It can be seen from the figure that when the number of tasks is very small, all the terminal devices have sufficient computing power and will choose to execute locally, and the differences among the five algorithms are small; when the number of tasks continues to increase, the total costs of all algorithms are rising. Generally speaking, the DRL-SPTM algorithm proposed in this embodiment has significantly lower total cost than other algorithms and is better than the other four algorithms under all numbers of tasks. The "DQN" algorithm ranks second only to the DRL-SPTM algorithm, and the "Greedy" algorithm has unsatisfactory results because it cannot guarantee the global optimal solution.

[0191] It can be seen that the present invention has good experimental results under different computing powers and different numbers of tasks, achieving a relatively low total cost consumption, indicating that the algorithm has good robustness under different computing powers and numbers of tasks and can minimize the total cost under different task conditions.

[0192] Embodiment 2

[0193] A task migration and service placement system based on an edge computing architecture, comprising:

[0194] An optimization problem formulation module, configured to model the communication channels of each device and the edge layer, calculate channel parameters based on the modeled model, and formulate a joint resource optimization problem for task migration and service placement considering the total costs of each device based on the calculated parameters;

[0195] An optimization problem solving module, configured to convert the joint resource optimization problem into a Markov decision process model, design the service placement state in the previous time slot, the total cost in the previous time slot, and the channel gains between the edge node and all devices as the state space of the agent, use the service placement allocation decision, the computing resource allocation decision, the proportion of computing resource allocation, and the proportion of wireless channel bandwidth as the action space of each agent, and use the minimization of the total costs of each device as the reward function based on the objective of the joint resource optimization problem, and iteratively optimize and solve using the task migration and service placement optimization algorithm on the edge node to obtain the final task migration and service placement strategy.

[0196] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made by those skilled in the art without creative efforts within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A task migration and service placement method based on edge computing architecture, characterized in that: The following steps are involved: Model the communication channels between each device and the edge layer, calculate the channel parameters based on the modeled model, and formulate the joint resource optimization problem of task migration and service placement based on the calculated parameters and considering the total cost of each device; The joint resource optimization problem is converted into a Markov decision process model. The service prevention state in the previous time slot, the total cost of the previous time slot, and the channel gain between the edge node and all devices are designed as the state space of the agent. The service placement allocation decision, computing resource allocation decision, computing resource allocation ratio and wireless channel bandwidth ratio are used as the action space of each agent. Based on the goal of the joint resource optimization problem, minimizing the total cost of each device is used as the reward function. The task migration and service placement optimization algorithm on the edge node is iteratively optimized and solved to obtain the final task migration and service placement strategy.

2. A method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The modeling model includes three layers. The first layer is the IoT device layer, which includes multiple IoT devices. Each IoT device can generate computing tasks and offload them to edge nodes or local execution; The second layer is the edge layer, which includes a single edge node, configured with edge servers connected to cloud servers; The third layer is the cloud layer, which is equipped with cloud servers and stores images of all types of services. It can migrate the images of corresponding types of services and place them on edge servers. The model time is discretized into multiple time slots. In each time slot, a computing task is randomly generated. The computing task can be executed locally or offloaded to an edge node or cloud server for execution. The execution location is decided by the agent. The model includes many different types of heterogeneous services, and the service placement decision is used to determine the number and type of services placed on the edge nodes.

3. The method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The process of calculating channel parameters includes calculating the latency of the task being executed locally and the energy consumption required to process this task; When task migration is executed, the transmission rate from IoT layer devices to edge nodes, the delay and energy consumption of task offloading to edge nodes are analyzed; when service placement is executed, the transmission rate, placement delay and energy consumption of migration from cloud servers to edge nodes are analyzed.

4. The method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The total cost of each device includes the total migration cost of the service migration to the edge node within the set time slot, the total delay cost and the total energy consumption cost.

5. The method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The process of formulating a joint resource optimization problem of task migration and service placement includes: the objective function of the joint resource optimization problem of task migration and service placement is to minimize the total cost of each device with a long-term average weighted average; The constraints include edge computing resource occupancy and wireless communication bandwidth occupancy constraints, service image storage upper limit constraints placed on edge nodes, service placement constraints, and weighted weight range constraints.

6. The method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The process of converting the joint resource optimization problem into a Markov decision process model includes: defining the state space of the Markov decision process model to consider the channel gain in the current time slot, the service placement in the previous time slot, and the weighted sum of the energy consumption, delay and migration cost after the completion of the task set in the previous time slot; The action space is defined as the set of service placement allocation decisions, computing resource allocation decisions, computing resource allocation ratios, and wireless channel bandwidth ratios; The reward is defined as the negative of the weighted sum of energy consumption, delay and migration cost after the task set is completed within the corresponding time; And the decision-making goal is to maximize the long-term cumulative rewards.

7. The method for task migration and service placement based on edge computing architecture as claimed in claim 1, characterized in that: The process of iteratively optimizing and solving using the task migration and service placement optimization algorithm on the edge node includes: the task migration and service placement optimization algorithm on the edge node includes a main network and a target network, the target network is used to generate a Q value of the target, the main network is used to execute action decisions, evaluate the Q values ​​of all actions in the current state, and select the optimal action according to the size of the Q value.

8. The method for task migration and service placement based on edge computing architecture as claimed in claim 7, characterized in that: The main network is updated by minimizing the timing difference error based on priority sampling. The target network parameters adopt a periodic soft update mechanism. The parameters of the target network are updated at each set step to make them close to the parameters of the main network. Through the collaborative learning of the main network and the target network, the parameters converge to the optimal state, realizing the optimal resource allocation and task migration strategy.

9. A task migration and service placement system based on edge computing architecture, characterized in that: include: an optimization problem formulation module, configured to model the communication channels between each device and the edge layer, calculate channel parameters based on the modeled model, and formulate a joint resource optimization problem of task migration and service placement based on the calculated parameters and considering the total cost of each device; The optimization problem solving module is configured to convert the joint resource optimization problem into a Markov decision process model, and designs the service prevention state in the previous time slot, the total cost of the previous time slot, and the channel gain between the edge node and all devices as the state space of the intelligent agent. The service placement allocation decision, computing resource allocation decision, computing resource allocation ratio and wireless channel bandwidth ratio are used as the action space of each intelligent agent. Based on the goal of the joint resource optimization problem, minimizing the total cost of each device is used as the reward function, and the task migration and service placement optimization algorithm on the edge node is iteratively optimized and solved to obtain the final task migration and service placement strategy.

10. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps in the method according to any one of claims 1 to 8 are completed.

Images