A method and device for joint computing offloading and resource allocation of MEC

Abstract

The present invention discloses a method and device for joint computing offloading and resource allocation of MEC. With minimizing the overall network delay as the optimization goal, an improved TD3 algorithm (ITD3) based on a deep reinforcement learning model is designed to form a joint scheduling strategy for task offloading decisions and edge server resource allocation, so as to ensure the high stability and high reliability of the user service scheduling process, realize flexible service scheduling and on-demand resource allocation, and improve user service quality.

Description

Technical Field

[0001] The present invention relates to the field of emerging information technology, and in particular to a method and device for joint computing offloading and resource allocation of Multi-Access Edge Computing (MEC, Mobile Edge Computing). Background Art

[0002] To meet the low-latency requirements of 5G application scenarios, multi-access edge computing (MEC) is becoming a trend. This technology provides computing services at the edge of the network. In MEC systems, MEC servers are closer to IoT devices than traditional cloud servers. Furthermore, the distributed structure of MEC servers prevents data congestion, significantly reducing transmission latency. Therefore, compared to cloud computing, MEC can support latency-critical services and a variety of IoT applications. Offloading computing tasks to resource-rich edge servers not only improves the quality of computing service (QoS) but also enhances the ability of end devices to meet resource-demanding applications. Compared to traditional cloud servers, MEC servers may have fewer resources and be more dynamic, forcing devices to compete for limited computing resources. Therefore, for such resource-constrained systems, resource allocation and scheduling (such as server selection, offload rate allocation, and local computing power) are crucial. To achieve efficient utilization of computing resources and meet the computing needs of devices, an intelligent compute offloading strategy is required. Consequently, compute offloading has attracted increasing research attention. Summary of the Invention

[0003] The embodiments of the present invention provide a method and apparatus for joint computing offloading and resource allocation of MEC, which are used to ensure high stability and high reliability of the user service scheduling process, realize flexible scheduling of services and on-demand allocation of resources, and improve user service quality.

[0004] An embodiment of the present invention provides a method for joint computing offloading and resource allocation of MEC, including:

[0005] Based on the real-time acquired task characteristics and the resource load of each edge server in the mobile edge network, the network state is abstractly modeled and a deep reinforcement learning model is established; the mobile edge network is composed of several user devices, several edge servers and a central controller;

[0006] After obtaining network status information in the mobile edge network at regular intervals, the deep reinforcement learning model is trained using the ITD3 algorithm;

[0007] Upon receiving the computing task uploaded by the user device, based on the latest updated network status information, with minimizing the processing delay of the computing task as the optimization goal, the deep reinforcement learning model trained with the ITD3 algorithm is used to make offloading and resource allocation decisions to obtain a decision result;

[0008] The offloading result of the computing task is sent to the edge server to which the computing task is connected, and the resource scheduling decision is sent to the edge server allocated for executing the computing task.

[0009] In one embodiment, the network status information includes network resource group information of each edge server;

[0010] The network resource group information of the edge server s is recorded as in, are the total amount of computing resources of the edge server s and the computing resource load of time slot t, are the total storage resources of the edge server s and the storage resource load of time slot t, respectively. s (t) is the communication resource of the edge server s in time slot t.

[0011] In one embodiment, the state space, action space, and reward function of the established deep reinforcement learning model are set as follows:

[0012] The state space Where, f(t)={f1(t),f2(t),…f S (t)} represents the computing resource allocation of each edge server at time t, q(t)={q1(t),q2(t),…q S (t)} represents the storage resource allocation of each edge server at time t, g(t)={g i,j (t)|i,j∈S,i≠j} represents the channel gain of the communication link between the edge servers at time t, h i,j (t) is the channel coefficient of channel (i, j) at time t, d i,j is the Euclidean distance between edge server i and edge server j, ζ is the path loss factor; (W k (t),D k (t),T k (t)) are the workload, data size and deadline of the computing task k(t);

[0013] The action space Among them, α k,s (t) is a binary unloading decision variable, when αk,s When (t) = 1, it means that the computing task k(t) is offloaded to the edge server s; f s (t) is the computing resources allocated by edge server s to perform computing tasks in time slot t;

[0014] The reward function in, is the objective function of the optimization problem, is the penalty function, C0 is the penalty factor, ∑ s∈S (1-α k,s (t)) represents the number of rejected tasks.

[0015] In one embodiment, the calculation formula for minimizing the processing delay of the computing task is as follows:

[0016]

[0017] Among them, D k (t) is the input data of computing task k, R k,s is the link rate, W k (t) is the workload, f s The computing resources allocated by the edge server s to the computing task k.

[0018] In one embodiment, the constraints of the calculation formula for minimizing the processing delay of the computing task are as follows:

[0019] C1:

[0020] C2:

[0021] C3:

[0022] C4:

[0023] C5:

[0024] C6:

[0025] Among them, constraint C1 limits the computing task to binary offloading; constraint C2 limits the deadline constraint of each computing task; constraint C3 limits the sum of computing resources occupied by the computing tasks being executed to no more than the computing resources of the edge server; constraint C4 limits the sum of storage resources occupied by all tasks offloaded to the edge server to no more than the storage resources of the edge server; constraint C5 limits a computing task to be offloaded to at most one edge server; constraint C6 limits the allocated computing resources to be always positive.

[0026] In one embodiment, the use of the ITD3 algorithm to train the deep reinforcement learning model specifically includes:

[0027] Construct a main network N and a target network N', each of which includes two critic sub-networks With an actor subnetwork π;

[0028] In the initial state, the parameters θ1, θ2 are randomly assigned to the main network N and the target network N', and the experience replay pool is initialized

[0029] When it is judged that the number of training executions t is greater than the set value, the initial state s is changed to the best state s* observed previously;

[0030] Add noise and select actions according to the determined strategy

[0031] After the next state is obtained according to the action execution, the sample (s, a, r, s') is stored in the experience replay pool middle;

[0032] From the Experience Replay Pool Randomly sample N samples (s, a, r, s') for review and update the action according to the quadratic update formula

[0033] In the two critic sub-networks, select the network update parameter with smaller Q-value and find the objective function

[0034] Update the parameters of the critic subnetwork according to the gradient descent formula

[0035] When the number of executions t reaches the set threshold d, the parameters of the actor subnetwork are updated according to the deterministic policy gradient formula

[0036] Use the soft update strategy to update the parameters of the main network to the parameters of the target network

[0037] Repeat the above training process until the parameters converge.

[0038] In one embodiment, the decision result includes the node mapping (task ID→node IP) of the computing task offloading edge server and the data mapping of resource allocation (node IP→{computing resources: X, storage resources: Y...}).

[0039] On the other hand, an embodiment of the present invention further provides a server, including:

[0040] A modeling unit is configured to abstractly model the network state and establish a deep reinforcement learning model based on the task characteristics acquired in real time and the resource load of each edge server in the mobile edge network; the mobile edge network is composed of a plurality of user devices, a plurality of edge servers, and a central controller;

[0041] A training unit, configured to, after obtaining network status information in the mobile edge network at regular intervals, train the deep reinforcement learning model using an ITD3 algorithm;

[0042] a processing unit configured to, upon receiving a computing task uploaded by the user equipment, make offloading and resource allocation decisions using the deep reinforcement learning model trained with the ITD3 algorithm based on the latest updated network status information, with minimizing the processing delay of the computing task as an optimization goal, and obtain a decision result;

[0043] The sending unit is configured to send the offloading result of the computing task to the edge server to which the computing task is connected, and send the resource scheduling decision to the edge server allocated for executing the computing task.

[0044] On the other hand, an embodiment of the present invention also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor; the instructions are executed by the at least one processor so that the at least one processor can execute the joint computing offloading and resource allocation method of MEC provided in an embodiment of the present invention.

[0045] On the other hand, an embodiment of the present invention further provides a computer storage medium, which stores a computer program, and the computer program is used to execute the joint computing offloading and resource allocation method of MEC provided by an embodiment of the present invention.

[0046] The beneficial effects of the present invention are as follows:

[0047] The embodiments of the present invention provide a method and device for joint computing offloading and resource allocation of MEC, which takes minimizing the overall network delay as the optimization goal. By designing an improved TD3 algorithm (ITD3) based on a deep reinforcement learning model, a joint scheduling strategy for task offloading decisions and edge server resource allocation is formed to ensure high stability and high reliability of the user service scheduling process, realize flexible scheduling of services and on-demand allocation of resources, and improve user service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A flowchart of the joint computing offloading and resource allocation method for MEC provided in an embodiment of the present invention;

[0050] Figure 2 A network topology diagram of a multi-access edge computing network provided by an embodiment of the present invention;

[0051] Figure 3 Flowchart of the ITD3 algorithm provided by an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the structure of a joint computing offloading and resource allocation device for MEC provided in an embodiment of the present invention;

[0053] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0055] In the embodiments of the present disclosure, the term "and / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0056] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Persons skilled in the art will appreciate that, as new application scenarios emerge, the technical solutions provided by the embodiments of the present disclosure will also be applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, "multiple" means two or more.

[0057] Numerous approaches have been proposed for computing offloading. Previous work has mostly employed optimization or game-based approaches to solve the computing offloading problem. These include approaches based on Lyapunov optimization, branch-and-bound algorithms and their improved algorithms (such as row / column generation and Benders decomposition), bioinspired algorithms (such as multi-particle swarm optimization and simulated annealing), and combinations of several traditional optimization algorithms. However, because the problem of joint computing offloading and resource allocation is inherently a mixed integer nonlinear programming (MINLP) problem, which is NP-hard, traditional optimization algorithms can only obtain approximate optimal solutions. Furthermore, it is difficult to accurately mathematically measure the upper and lower bounds between the feasible and optimal solutions. Furthermore, these traditional algorithms require prior information on environmental statistics, which is impossible to obtain in dynamic MEC systems in real-world scenarios. Furthermore, these traditional optimization algorithms fail to consider the impact of past service scheduling decisions on subsequent ones. Due to algorithmic limitations, they ignore the interdependencies between service scheduling strategies, resulting in independent scheduling and difficulty ensuring that the current decision is optimal over the long term.

[0058] To address the challenges of traditional algorithms, some researchers have modeled the computation offloading problem as a Markov decision process (MDP) and solved it using reinforcement learning (RL) or deep reinforcement learning (DRL) methods. Many researchers, both domestic and international, have used classic deep reinforcement learning models such as D3QN (Dueling Double DQN, value-based), TRPO (Trust Region Policy Optimization, value-based), and DDPG (Deep Deterministic Policy Gradient, actor-critic) to solve the joint computation offloading and resource allocation problem. These methods have achieved good performance without requiring prior knowledge of environmental statistics. However, they are modeled in either discrete or continuous action spaces, which limits the optimization of offloading decisions within a limited action space. In real-world offloading problems, the action space is often a mixture of continuous and discrete, requiring each device to jointly decide on both continuous and discrete actions to complete the offloading process. For example, a device must not only decide whether to offload a task or which server to select, but also select the offloading ratio or local computing capacity to balance time and energy consumption. Therefore, these methods may not perform well in fine discretization of continuous actions when the action space becomes large, and relaxing discrete actions into a continuous set may significantly increase the complexity of the action space.

[0059] Based on the above-mentioned technical problems existing in the prior art, the present invention provides a joint computing offloading and resource allocation method for MEC, with minimizing the overall network delay as the optimization goal, and designs an improved TD3 algorithm (ITD3) based on a deep reinforcement learning model to form a joint scheduling strategy for task offloading decisions and edge server resource allocation, so as to ensure the high stability and high reliability of the user service scheduling process, realize flexible scheduling of services and on-demand allocation of resources, and improve user service quality.

[0060] Specifically, an embodiment of the present invention provides a method for joint computing offloading and resource allocation of MEC, such as Figure 1 As shown, the following steps are included:

[0061] S1. Based on real-time task characteristics and the resource load of each edge server in the mobile edge network, an abstract model of the network state is constructed to establish a deep reinforcement learning model. In a mobile edge network consisting of user devices, edge servers, and a central controller, each user device generates computing tasks based on demand and sends them to the edge server for processing. The central controller abstractly models the network state based on real-time task characteristics and resource load.

[0062] Specifically, the mobile edge network is an end-edge-cloud network consisting of several user equipment UEs, several edge servers (MEC servers) and a central controller (core network), which operates in a time slot structure. Its basic topology is as follows: Figure 2 As shown in Figure 1, there are a total of K user devices, represented by K = {1, 2, ..., K}. The user devices themselves do not have the ability to process computing tasks and are only responsible for uploading computing tasks to the edge server connected to them. For user device k, the computing task generated by it in time slot t is recorded as k(t), which is represented by the tuple (W k (t),D k (t),T k (t)),where W k (t) is the workload, D k (t) is the input data size, T k (t) is the deadline. A computing task is indivisible. Without considering redundant offloading, each user device can upload at most one computing task to one edge server. There are S edge servers in total, denoted by S = {1, 2, …, S}. Each edge server has a certain amount of network resources and can simultaneously process computing tasks from multiple user devices.

[0063] S2. After obtaining network status information from the mobile edge network at regular intervals, the deep reinforcement learning model is trained using the ITD3 algorithm. Specifically, the central controller collects and caches the latest network status information at regular intervals, preparing it for invocation when computing tasks arrive.

[0064] Specifically, the network status information includes the network resource group information of each edge server. For edge server s, its network resource group information is recorded as in are the total computing resources of edge server s (reflected in CPU main frequency, unit MHz) and the computing resource load of time slot t, are the total storage resources of edge server s (reflected as hard disk capacity, unit GB) and the storage resource load of time slot t, B s (t) is the communication resource of edge server s in time slot t (expressed as bandwidth, unit Mbps).

[0065] Optionally, software-defined networking (SDN) technology can be enabled in the core cellular network to enable flexible routing and communication between edge servers. Software-defined networking technology centralizes control logic in a centralized entity called an SDN controller, simplifying network management. The SDN controller installs appropriate forwarding rules in the routing tables of all forwarding devices (such as those in OpenFlow switches), thereby obtaining the necessary information to apply network policies and gaining a global view of the network. The SDN controller can also provide monitoring information related to network performance.

[0066] Specifically, the Improved Twin Delayed Deep Deterministic Policy Gradient (ITD3) algorithm is an improved version of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and is a deep reinforcement learning algorithm. Reinforcement learning (RL) is suitable for solving real-time, related, and complex decision-making problems. It mainly includes three key elements: state, action, and reward. The agent interacts with the environment in a discrete time domain. At each time slot t, the agent performs an action a according to the strategy μ, that is, μ:s t →a t After the agent completes the action according to the strategy, the environment returns a reward value r t , and the system state s t Transition to state s t+1 Generally, we use the action-value function (i.e. Q function) Q μ(s,a) represents the expected long-term discounted reward (EDAR) of executing the strategy μ under the initial state s and the initial action a.

[0067] Therefore, before using deep reinforcement learning models to solve the computation offloading problem, it is necessary to first design the state space, action space, and reward function of the Markov decision process. Based on the main characteristics of mobile edge networks, the state space, action space, and reward function are set as follows:

[0068] Specifically, the state space consists of resource load, task information and channel state, namely Where, f(t)={f1(t),f2(t),…f S (t)} represents the computing resource allocation of each edge server at time t, that is, q(t)={q1(t),q2(t),…q S (t)} represents the storage resource allocation of each edge server at time t, that is, g(t)={g i,j (t)|i,j∈S,i≠j} represents the channel gain of the communication link between each edge server at time t, that is, Where h i,j (t) Channel coefficient of channel (i, j) at time t, d i,j is the Euclidean distance between edge server i and edge server j, ζ is the path loss factor, which defaults to 3.5. (W k (t),D k (t),T k (t)) is the basic information of computing task k, namely the workload, data size and deadline of computing task k(t).

[0069] Specifically, the action space consists of offloading decisions and resource allocation, i.e. where α k,s (t) is a binary unloading decision variable, when α k,s When (t) = 1, it means that the computing task k(t) is offloaded to the edge server s. s (t) is the computing resources allocated by edge server s to perform computing tasks in time slot t.

[0070] Specifically, the reward function is composed of the objective function and the penalty function, that is, in is the objective function of the optimization problem, i.e., the maximum execution delay of all computing tasks; is the penalty function, C0 is the penalty factor, ∑ s∈S (1-α k,s (t) represents the number of rejected tasks. The lower the latency, the fewer rejected tasks and the larger the reward function.

[0071] The TD3 algorithm is an improvement on the DDPG algorithm. It eliminates the parameter error problem of the traditional DDPG (Vanilla-DDPG) algorithm through the following three techniques: 1. Using two Q functions for learning, and using the parameters of the function with the smallest Q-value when updating model parameters to avoid over-estimation of the Q-value; 2. Delaying the update of the target network and policy, updating them once every d rounds, to avoid cumulative error caused by parameter updates; 3. Adding noise to the target action network to smooth the Q function and reduce the error in policy estimation.

[0072] Compared with the TD3 algorithm, the ITD3 algorithm borrows the idea of the greedy algorithm and modifies the steps at the beginning of each training set based on TD3. The initial state at the beginning of each set value (for example, 150) epoch is set to the best state among all previously observed states. This can help the original TD3 algorithm quickly find the optimal solution in the high-dimensional state space, thereby accelerating the algorithm convergence without affecting the algorithm performance, and improving the usability of the TD3 algorithm deployed in actual scenarios.

[0073] like Figure 3 As shown, the main steps of the ITD3 algorithm are as follows:

[0074] 1. Construct the main network N and the target network N', each network includes two critic sub-networks With an actor subnetwork π;

[0075] 2. Randomly assign parameters θ1, θ2, to each network in the initial state and initialize the experience replay pool

[0076] 3. If it is executed 150 times, change the initial state s to the best state s* observed previously, otherwise do nothing;

[0077] 4. Select actions based on the determined strategy and noise

[0078] 5. After the next state is obtained according to the action execution, the sample (s, a, r, s') is stored in the experience replay pool middle;

[0079] 6. After a period of time, randomly sample N samples (s, a, r, s') from the experience replay pool for review, and update action a according to the quadratic update formula. Ensure that the values around the target action are smooth to minimize the error (regularization).

[0080] 7. Select the network update parameter with smaller Q-value in the dual critic sub-network to find the objective function

[0081] 8. Update the critic network parameters according to the gradient descent formula

[0082] 9. When the number of executions t reaches a certain threshold d (t mod d), the actor network parameters are updated according to the deterministic policy gradient formula

[0083] 10. Use the soft update strategy to update the parameters of the main network to the parameters of the target network

[0084] 11. Repeat steps 3 to 10 above until the algorithm converges.

[0085] S3. When receiving computing tasks uploaded by user devices, based on the latest updated network status information, with minimizing the processing delay of computing tasks as the optimization goal, the deep reinforcement learning model trained by the ITD3 algorithm is used to make offloading and resource allocation decisions to obtain decision results.

[0086] Specifically, the network status information when calculating the offload request should be ensured to be the same as the network status information last updated by the global controller.

[0087] Specifically, the task processing delay includes the task transmission delay and execution delay, which are calculated as follows:

[0088]

[0089] Where, τ k,s (t) is the total delay of offloading, which is the total delay from the time slot t when the user device generates the computing task k to the time slot t when the computing task is executed on the edge server s. up is the input data upload delay, is the processing delay of edge server s executing computing task k, τ wait To calculate the waiting delay of task k, τ down The latency is the result download delay. Since each edge server allocates different resources to process multiple computing tasks simultaneously, the waiting delay is 0. Since the downlink quality is good, the download delay is negligible.

[0090] The calculation method for task processing delay is as follows:

[0091]

[0092] Among them, D k (t) is the input data of computing task k (unit: MB), R k,s is the link rate (in Mbps), W k (t) is the workload, f s is the computing resource (in MHz) allocated by edge server s to computing task k.

[0093] According to Shannon's formula, the link rate can be written as follows:

[0094]

[0095] Where B is the bandwidth, g k,s (t) is the channel gain, P k is the transmission power, is the noise power, I k,s (t) is the mutual interference between channels.

[0096] Therefore, the optimization goal of the computation offloading algorithm is to minimize the long-term consumption of the entire network. The consumption is defined as the delay in executing the computation task, which is calculated as follows:

[0097]

[0098] The calculation method for unloading constraint conditions is as follows:

[0099] C1:

[0100] C2:

[0101] C3:

[0102] C4:

[0103] C5:

[0104] C6:

[0105] Among them, constraint C1 limits task binary offloading; constraint C2 limits the deadline constraint of each computing task; constraint C3 limits the sum of computing resources occupied by the executing tasks to be no greater than the computing resources of the server; constraint C4 limits the sum of storage resources occupied by all tasks offloaded to the server to be no greater than the storage resources of the server; constraint C5 limits a task to be offloaded to at most one edge server; constraint C6 limits the allocated computing resources to be always positive.

[0106] Specifically, the decision result is a data structure that provides computing offloading and resource allocation results, including the node mapping of computing offloading (task ID→node IP) and the data mapping of resource allocation (node IP→{computing resources: X, storage resources: Y...}).

[0107] S4. Send the offloading result of the computing task to the edge server to which the computing task is connected, and send the resource scheduling decision to the edge server allocated to execute the computing task.

[0108] Specifically, the nearest edge server receiving the computing task determines the edge server address to which the computing task will be offloaded based on the node mapping and sends the task's key information to the designated edge server address. This key information includes the task ID, task type (which determines the algorithm to be executed), the task arrival node (used to transmit the result data after the computation is completed), and the task offloading node (destination IP address).

[0109] The edge server responsible for executing the computing task dynamically allocates the computing and storage resources required to execute the corresponding algorithm based on the server's own resource load, establishes a transmission link, and receives the task's input data. The corresponding algorithm is the artificial intelligence inference algorithm required to execute the computing task. The mapping between tasks and algorithms must be statically configured before the network is started. In fact, static measurement of the latency of the algorithm invoked in different network environments can also be used as auxiliary data to provide more accurate decision-making reference for computing offloading and resource allocation strategies. The edge server carrying out the computing task invokes the specified algorithm to process the computing task and transmits the output results back to the access server of the computing task via a communication link, completing the offloading of the computing task.

[0110] Compared with the prior art, the combined computing offloading and resource allocation method provided by the embodiment of the present invention has the following advantages:

[0111] In more complex scenarios, traditional computational offloading algorithms struggle to achieve both high performance and low complexity. Traditional computational offloading algorithms often only consider the offloading results of a single time slot, without considering the correlation between time slots. The computational offloading problem itself possesses two characteristics of reinforcement learning (reward maximization / Markov decision making) and can be solved using reinforcement learning. Deep reinforcement learning models are used to jointly solve computational offloading and resource allocation problems. Pre-training for different network scenarios makes the model ready-to-use. Furthermore, in MEC networks, each edge server can independently make decisions, including the target server for task offloading and resource allocation scheduling decisions based on time-varying wireless channels, to minimize the total cost of the network.

[0112] The ITD3 algorithm, an improved version of the DDPG algorithm, improves both the policy network and the value network, optimizing the overestimation of Q-values. It inherits the TwinCritic from the Double DQN algorithm to reduce overestimation errors. When calculating the Q-value used to update the TD-error, it adds noise to the action to help the critic fit a smoother Q-value estimation function. It also adds noise to the action to smooth the critic function. It also sets a timer to assign the optimal solution after 150 epochs as the initial state to accelerate the algorithm's convergence. In mobile edge network scenarios, with its continuous state and action spaces, the ITD3 algorithm is suitable for training deep reinforcement learning models. The ITD3 algorithm facilitates the exploration of deep reinforcement learning models and achieves faster convergence and higher accuracy than DDPG.

[0113] By completing the design of the state space, action space, and reward function of the deep reinforcement learning model, it can be trained in different mobile edge network scenarios to obtain a deep reinforcement learning model that is adapted to different mobile edge network scenarios. It is not restricted by specific application scenarios. Compared with traditional computational offloading and resource allocation schemes, it can achieve good performance without requiring prior knowledge of environmental statistics, can perform fine discretization of continuous actions well, does not significantly increase the complexity of the action space, and is easy to deploy in actual networking to achieve network load balancing.

[0114] Based on the same inventive concept, an embodiment of the present invention also provides a joint computing offloading and resource allocation device for MEC. Since the principle of solving the problem by the device is similar to that of the aforementioned joint computing offloading and resource allocation method for MEC, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0115] The embodiment of the present invention provides a joint computing offloading and resource allocation device for MEC, such as Figure 4 Shown, including:

[0116] Modeling unit 1 is used to abstractly model the network state based on the real-time acquired task characteristics and the resource load of each edge server in the mobile edge network and establish a deep reinforcement learning model. The mobile edge network consists of several user devices, several edge servers and a central controller;

[0117] Training unit 2 is used to obtain network status information in the mobile edge network at regular intervals and train the deep reinforcement learning model using the ITD3 algorithm;

[0118] Processing unit 3 is used to, upon receiving a computing task uploaded by a user device, make offloading and resource allocation decisions based on the latest updated network status information, with minimizing the processing delay of the computing task as the optimization goal, using the deep reinforcement learning model trained with the ITD3 algorithm to obtain a decision result;

[0119] The sending unit 4 is used to send the offloading result of the computing task to the edge server connected to the computing task, and send the resource scheduling decision to the edge server allocated to execute the computing task.

[0120] After introducing a method and apparatus for joint computing offloading and resource allocation of MEC according to an exemplary embodiment of the present invention, an electronic device according to another exemplary embodiment of the present invention is introduced next.

[0121] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0122] In some possible implementations, an electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps of the joint computation offloading and resource allocation method for MEC according to various exemplary embodiments of the present invention described above in this specification.

[0123] Refer to the following Figure 5 An electronic device 600 according to this embodiment of the present invention will be described. Figure 5 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0124] like Figure 5 As shown, electronic device 600 is a general electronic device. Components of electronic device 600 may include, but are not limited to, at least one processor 601, at least one computer storage medium 602, and a bus 603 connecting different system components (including computer storage medium 602 and processor 601).

[0125] Bus 603 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0126] Computer storage media 602 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 621 and / or cache storage media 622 , and may further include read-only computer storage media (ROM) 623 .

[0127] The computer storage medium 602 may also include a program / utility 625 having a set (at least one) of program modules 624, such program modules 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0128] The electronic device 600 may also communicate with one or more external devices 604 (e.g., a keyboard, pointing device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other electronic devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 605. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 606. As shown, the network adapter 606 communicates with other modules of the electronic device 600 via a bus 603. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] In some possible implementations, various aspects of the joint computing offloading and resource allocation method for MEC provided by the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the joint computing offloading and resource allocation method for MEC according to various exemplary embodiments of the present invention described above in this specification.

[0130] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination thereof.

[0131] The program product for joint computation offloading and resource allocation for MEC according to an embodiment of the present invention may be a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and may be run on an electronic device. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] A readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0134] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0135] It should be noted that although several modules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0136] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0137] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk computer storage media, CD-ROM, optical computer storage media, etc.) containing computer-usable program code.

[0138] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce an article of manufacture including an instruction device that implements the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0141] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A joint computing offloading and resource allocation method for multi-access edge computing MEC, characterized in that: include: Based on the real-time acquired task characteristics and the resource load of each edge server in the mobile edge network, the network state is abstractly modeled and a deep reinforcement learning model is established; the mobile edge network is composed of several user devices, several edge servers and a central controller; After obtaining network status information in the mobile edge network at regular intervals, the deep reinforcement learning model is trained using an improved TD3 algorithm (ITD3 algorithm); Upon receiving the computing task uploaded by the user device, based on the latest updated network status information, with minimizing the processing delay of the computing task as the optimization goal, the deep reinforcement learning model trained with the ITD3 algorithm is used to make offloading and resource allocation decisions to obtain a decision result; Sending the offloading result of the computing task to the edge server connected to the computing task, and sending the resource scheduling decision to the edge server assigned to execute the computing task; The deep reinforcement learning model is trained using the ITD3 algorithm, specifically including: Construct a main network N and a target network N', each of which includes two critic sub-networks and an actor sub-network π; randomly assign parameters θ1, θ2, to the main network N and the target network N' in the initial state, and initialize the experience replay pool B; When the number of training executions t is greater than the set value, the initial state s is changed to the best state s* observed previously; noise is added according to the determined strategy to select the action After the next state is obtained according to the action execution, the sample (s, a, r, s') is stored in the experience replay pool In; from the experience replay pool Randomly sample N samples (s, a, r, s') for review and update the action according to the quadratic update formula In the two critic sub-networks, select the network update parameter with smaller Q-value and find the objective function Update the parameters of the critic subnetwork according to the gradient descent formula When the number of executions t reaches the set threshold d, the parameters of the actor subnetwork are updated according to the deterministic policy gradient formula Use the soft update strategy to update the parameters of the main network to the parameters of the target network Repeat the above training process until the parameters converge.

2. The method according to claim 1, wherein The network status information includes network resource group information of each edge server; The network resource group information of the edge server s is recorded as in, are the total amount of computing resources of the edge server s and the computing resource load of time slot t, are the total storage resources of the edge server s and the storage resource load of time slot t, respectively. s (t) is the communication resource of the edge server s in time slot t.

3. The method according to claim 2, wherein The state space, action space, and reward function of the established deep reinforcement learning model are set as follows: The state space Where, f(t)={f1(t),f2(t),…f S (t)} represents the computing resource allocation of each edge server at time t, represents the storage resource allocation of each edge server at time t, g(t)={g i,j (t)|i,j∈S,i≠j} represents the channel gain of the communication link between the edge servers at time t, h i,j (t) is the channel coefficient of channel (i, j) at time t, d i,j is the Euclidean distance between edge server i and edge server j, ζ is the path loss factor; (W k (t),D k (t),T k (t)) are the workload, data size and deadline of the computing task k(t); The action space Among them, α k,s (t) is a binary unloading decision variable, when α k,s When (t) = 1, it means that the computing task k(t) is offloaded to the edge server s; f s (t) is the computing resources allocated by edge server s to perform computing tasks in time slot t; The reward function in, is the objective function of the optimization problem, is the penalty function, C0 is the penalty factor, ∑ s∈S (1-α k,s (t)) represents the number of rejected tasks.

4. The method according to claim 1, wherein The calculation formula for minimizing the processing delay of the computing task is as follows: Among them, D k (t) is the input data of computing task k, R k,s is the link rate, W k (t) is the workload, f s The computing resources allocated by the edge server s to the computing task k.

5. The method according to claim 4, wherein The constraints of the calculation formula for minimizing the processing delay of the computing task are as follows: Among them, constraint C1 limits the computing task to binary offloading; constraint C2 limits the deadline constraint of each computing task; constraint C3 limits the sum of computing resources occupied by the computing tasks being executed to no more than the computing resources of the edge server; constraint C4 limits the sum of storage resources occupied by all tasks offloaded to the edge server to no more than the storage resources of the edge server; constraint C5 limits a computing task to be offloaded to at most one edge server; constraint C6 limits the allocated computing resources to be always positive.

6. The method according to any one of claims 1 to 5, wherein: The decision result includes the node mapping (task ID→node IP) of the computing task offloading edge server and the data mapping of resource allocation (node IP→{computing resources: X, storage resources: Y...}).

7. A server, characterized in that: include: A modeling unit is configured to abstractly model the network state based on the real-time acquired task characteristics and the resource load of each edge server in the mobile edge network, thereby establishing a deep reinforcement learning model; the mobile edge network is composed of a plurality of user devices, a plurality of edge servers, and a central controller; A training unit, configured to obtain network status information in the mobile edge network at intervals and train the deep reinforcement learning model using an improved TD3 algorithm (ITD3 algorithm); a processing unit configured to, upon receiving a computing task uploaded by the user equipment, make offloading and resource allocation decisions using the deep reinforcement learning model trained with the ITD3 algorithm based on the latest updated network status information, with minimizing the processing delay of the computing task as an optimization goal, and obtain a decision result; a sending unit, configured to send the offloading result of the computing task to the edge server to which the computing task is connected, and send the resource scheduling decision to the edge server allocated for executing the computing task; The training unit is specifically used for: Construct a main network N and a target network N', each of which includes two critic sub-networks and an actor sub-network π; randomly assign parameters θ1, θ2, to the main network N and the target network N' in the initial state, and initialize the experience replay pool B; When the number of training executions t is greater than the set value, the initial state s is changed to the best state s* observed previously; noise is added according to the determined strategy to select the action After the next state is obtained according to the action execution, the sample (s, a, r, s') is stored in the experience replay pool In; from the experience replay pool Randomly sample N samples (s, a, r, s') for review and update the action according to the quadratic update formula In the two critic sub-networks, select the network update parameter with smaller Q-value and find the objective function Update the parameters of the critic subnetwork according to the gradient descent formula When the number of executions t reaches the set threshold d, the parameters of the actor subnetwork are updated according to the deterministic policy gradient formula Use the soft update strategy to update the parameters of the main network to the parameters of the target network Repeat the above training process until the parameters converge.

8. An electronic device, characterized in that: The invention comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor; the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 6.

Images