An edge computing offloading method based on dynamic user satisfaction in ultra-dense networks

Abstract

The present invention belongs to the field of mobile communication technology, and specifically relates to an edge computing offloading method based on dynamic user satisfaction in an ultra-dense network, the method comprising: building a multi-user ultra-dense network and initializing the network; user equipment in the network generates a computing task and sends a task offloading request to a macro base station; the macro base station obtains user equipment information and network environment information and builds a task model; the task model and network environment information are input into a trained task decision model to obtain a task offloading decision; the macro base station sends the task offloading decision to the user equipment and a micro base station respectively; the user equipment executes the task offloading according to the task offloading decision, and the micro base station allocates computing resources to perform task calculation according to the task offloading decision; the present invention improves the model of traditional tasks and measures the energy consumed by task execution so that it can meet the energy consumption requirements of users under different conditions.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology, and in particular relates to an edge computing offloading method based on dynamic user satisfaction in an ultra-dense network. Background Art

[0002] With the rapid development of wireless communication technology and the widespread adoption of smart devices, mobile applications have experienced explosive growth in recent years, including facial recognition, online mobile gaming, virtual reality (VR), and augmented reality (AR). Most of these applications are computationally intensive or latency-sensitive, yet mobile devices (such as smartphones and wearables) typically have limited computing power and battery life. The relationship between these applications and resource-constrained devices poses a significant challenge to improving the quality of experience (QoE) of mobile users.

[0003] Mobile Edge Computing (MEC) moves computing servers from cloud centers to the edge of the network, significantly shortening the distance between user devices and servers. Users can leverage computational offloading technology to offload tasks to edge servers for computation, meeting intensive computing needs. Furthermore, the Ultra-Dense Network (UDN) under the 5G architecture is a heterogeneous network solution that provides collaborative services across multiple base stations. By deploying a large number of micro and macro base stations in hotspots, it improves wireless network coverage, resolves network coverage blind spots, and increases network capacity, thereby improving overall network performance. However, due to the dense deployment of micro base stations and MEC servers, multiple users will be within the coverage of multiple micro base stations, and different base stations have different computing capabilities. Making offload decisions and allocating resources for users is a challenge.

[0004] A heuristic task offloading algorithm, HTOA, has been proposed for joint wireless resource management and task offloading in ultra-dense networks. This algorithm alternately updates wireless resource management in each iteration to maximize the total user offloading benefit, measured by task execution latency and energy consumption. A channel resource allocation algorithm, CRADE, based on differential evolution, has been proposed for task offloading and resource allocation in 5G ultra-dense networks based on mobile edge computing. This algorithm achieves optimal multi-user task offloading and resource allocation solutions.

[0005] However, these existing technologies are all performed in a static environment. Each optimization takes a snapshot of the current network state and performs optimization based on that snapshot. However, actual MEC networks are dynamic. Furthermore, these optimization efforts all aim to minimize energy consumption while meeting latency requirements, without fully considering the changing needs of users under varying conditions. Furthermore, these efforts improve user QoE indirectly through maximizing energy efficiency or minimizing energy consumption. Therefore, developing computational offloading methods that fully consider user needs and effectively improve user quality of experience in dynamic, ultra-dense networks is of great research value. Summary of the Invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes an edge computing offloading method based on dynamic user satisfaction in an ultra-dense network, the method comprising: constructing a multi-user ultra-dense network and initializing the network; the user equipment in the network generates a computing task and sends a task offloading request to the macro base station; after receiving the request, the macro base station obtains the user equipment information and the network environment information, and constructs a task model based on the obtained user equipment information, wherein the environment information includes the channel status between the user equipment and the micro base station and the computing resource information of the MEC server; the task model and the network environment information are input into the trained task decision model to obtain the task offloading decision, wherein the task decision model includes a policy network and an action value network; the macro base station sends the task offloading decision to the user equipment and the micro base station respectively; the user equipment performs the task offloading according to the task offloading decision, and the micro base station allocates computing resources for task calculation according to the task offloading decision.

[0007] Preferably, the multi-user ultra-dense network includes a macro base station and N micro base stations, each micro base station is configured with a MEC server to perform computing tasks; each micro base station uses orthogonal frequency division multiple access to user equipment.

[0008] Preferably, constructing the task model includes: obtaining user device information, which includes the size of task input data generated by the user device, the number of CPU cycles per task unit, the user's delay requirement for executing the task, the user's energy consumption requirement for the task, the maximum energy consumption that the user can accept when executing the task, and the remaining power of the user device in the current time slot; the task model constructed based on the user device information is:

[0009]

[0010] Among them, d u (t) represents the input data size of the task, c u (t) represents the number of CPU cycles per unit to execute the task, τ u (t) represents the user's execution delay requirement for the task, i.e., the maximum tolerable delay. It represents the user's energy consumption requirement for the task, that is, the ideal execution energy consumption. Indicates the maximum execution energy consumption that the user can accept. Indicates the remaining battery power of the user equipment in the current time slot.

[0011] Furthermore, the formula for determining the maximum execution energy consumption and ideal execution energy consumption acceptable to the user device is:

[0012]

[0013]

[0014] Among them, g represents the ideal energy consumption calculation function, is the maximum storage capacity of the user device, and ε is a hyperparameter.

[0015] Preferably, the process of training the task decision model includes:

[0016] S1: Initialize the parameters of the task decision model;

[0017] S2: In each time slot, the macro base station is used as an intelligent agent to obtain the current environment status information and the task information of the user equipment, and a task model is built based on the task information of the user task equipment;

[0018] S3: Input the current environment state information and task model into the policy network to obtain the task action, which includes the offloading decision, power control and computing resource allocation action of the current task;

[0019] S4: Calculate the current user's satisfaction based on the task action and obtain the immediate reward of the current macro base station;

[0020] S5: The macro base station obtains the next moment environment state information, and stores the current environment state information, task action, immediate reward, and next moment environment state information as a four-tuple into the priority experience replay array;

[0021] S6: A priority mechanism is used to sample the priority experience replay array, and the collected quadruple is input into the policy network and action value network for training respectively. When the immediate reward function of the macro base station converges, the model training is completed.

[0022] Furthermore, the immediate reward function is:

[0023]

[0024] in, Transmit reward function, s t Indicates the current environment state, a t represents an action, u represents a user, U represents the total number of users, Ou (t) represents the user's satisfaction with the macro base station's performance of the current task, € u (t) represents the penalty function.

[0025] Preferably, the process of sampling the priority experience replay array using a priority mechanism includes: using a priority score formula to calculate the priority score of each data in the priority experience replay array; sorting the data in descending order according to the calculated priority score, labeling the sorted data, and defining a sampling value for each data according to the label; calculating the sampling probability of each data according to the sampling value; and setting the learning rate of the task decision model training according to the sampling probability.

[0026] Furthermore, the formula for setting the task decision model training learning rate is:

[0027]

[0028] Where ψ represents the annealing variable, represents the number of all experience groups, α represents the initial learning rate, represents the sampling probability.

[0029] Preferably, the process of training the policy network includes: given the current state s t , the policy network outputs action a t =μ(s t θ μ ), the value network gives the action a score based on the current state: q t =Q(s t ,μ(s t θ μ );θ Q ); Calculate the expected value of the action with respect to the state according to the action score J(θ μ ); fix the value network parameter θ Q , update the parameters θ using the gradient ascent algorithm μ , when J(θ μ ) The training of the policy network is completed when convergence.

[0030] Preferably, the process of training the action value network includes:

[0031] Step 1: Extract a set of experience groups (s) from the experience replay array t ,a t ,r t ,s t+1 );

[0032] Step 2: Input the experience group into the action value network to obtain q t =Q(s t ,at θ Q ) and q t+1 =Q(s t+1 ,μ(s t+1 θ μ );θ Q ); where Q represents the value network, s t Indicates the current environment state, a t represents the action, θ Q represents the value network parameter, q t+1 represents the predicted output of the value network at the next moment, and μ represents the policy network parameter;

[0033] Step 3: According to q t+1 Calculate the TD target using the following formula:

[0034]

[0035] Among them, r t represents the immediate reward, γ represents the discount factor;

[0036] Step 4: According to TD target, q t and q t+1 Calculate the network's loss function;

[0037] Step 5: Use the gradient descent algorithm to calculate θ Q Updates are performed and the training of the action value network is completed when the network's loss function converges.

[0038] Beneficial effects of the present invention:

[0039] The present invention considers an edge computing offloading method based on dynamic user satisfaction in ultra-dense networks, improves the model of traditional tasks, and measures the energy consumed by task execution so that it can meet the energy consumption requirements of users under different conditions. Based on the improved task model, a related computational offloading problem of a dynamic satisfaction function is proposed, and the quality of user experience can be directly improved by optimizing the satisfaction function. When users cooperate, users with higher device power can set more relaxed energy consumption requirements, and more computing resources can be given up to users with lower device power to meet more stringent energy consumption requirements, so that the overall satisfaction of all users can be effectively improved. Finally, an algorithm based on deep reinforcement learning is used to obtain the optimal offloading strategy. Through simulation experiments, the algorithm effectively improves the user experience quality, and has better convergence and stability than existing algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a multi-user ultra-dense network scenario diagram of the present invention;

[0041] Figure 2 This is a diagram showing the influence of different sensitivity parameters on the satisfaction function of the present invention;

[0042] Figure 3 This is a framework diagram of the algorithm proposed in the present invention;

[0043] Figure 4 This is a flow chart of the algorithm proposed in the present invention;

[0044] Figure 5 This is a comparison chart of the effects of the algorithm proposed in this invention and the existing algorithms. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] An edge computing offloading method based on dynamic user satisfaction in an ultra-dense network, the method comprising: constructing a multi-user ultra-dense network and initializing the network; user equipment in the network generates a computing task and sends a task offloading request to a macro base station; after receiving the request, the macro base station obtains user equipment information and network environment information, and constructs a task model based on the obtained user equipment information, wherein the environment information includes the channel state between the user equipment and the micro base station and the computing resource information of the MEC server; the task model and the network environment information are input into a trained task decision model to obtain a task offloading decision, wherein the task decision model includes a policy network and an action value network; the macro base station sends the task offloading decision to the user equipment and the micro base station respectively; the user equipment performs task offloading according to the task offloading decision, and the micro base station allocates computing resources for task calculation according to the task offloading decision.

[0047] The multi-user ultra-dense network consists of a macro base station and N micro base stations. Each micro base station is equipped with an MEC server to perform computing tasks. Each micro base station uses orthogonal frequency division multiple access to access user devices. Specifically, Figure 1 As shown, the network includes 1 macro base station, under which N micro base stations are randomly distributed. The base station set is composed of Represents, where n=j represents the jth base station. There are areas where multiple micro base stations have partially overlapping coverage. Each base station is equipped with a MEC server to perform computing tasks. Different base station servers have different computing capabilities. Each base station uses orthogonal frequency division multiple access (OFDMA) to access user equipment. The channel is divided into K orthogonal sub-channels. The channel set is Therefore, the user equipment needs to select the appropriate base station and the appropriate channel when offloading tasks. The base station serves U user equipments, and the user equipment set is Where u=i represents the i-th user equipment. Since this work is carried out in a dynamic MEC system, the present invention adopts a discrete time model, and time is divided into a set of time slots of length l.

[0048] Building a task model includes: obtaining user device information, which includes the task input data size generated by the user device, the number of CPU cycles per task unit, the user's delay requirement for executing the task, the user's energy consumption requirement for the task, the maximum energy consumption that the user can accept when executing the task, and the remaining power of the user device in the current time slot.

[0049] The present invention adds energy consumption metrics to the traditional task model to meet the different energy consumption requirements of users for task execution under specific conditions. The task achieved by user device u in time slot t is represented by Indicates that d u (t) represents the input data size of the task (bit), c u (t) represents the number of CPU cycles per bit to execute the task, τ u (t) represents the user's execution delay requirement for the task, that is, the maximum tolerable delay, and the energy consumption measurement is different from the previous model. It represents the user's energy consumption requirement for the task, that is, the ideal execution energy consumption. Indicates the maximum execution energy consumption that the user can accept, and introduces Indicates the remaining power of the user device in the current time slot. The ideal energy consumption setting is related to the user's needs, the current remaining power of the user device, and the maximum execution energy consumption threshold of the current task. The formula is introduced as follows:

[0050]

[0051]

[0052] Among them, g represents the ideal energy consumption calculation function, is the maximum storage capacity of the user's device, ε∈[5,10] is a hyperparameter, and the user can adjust ε to meet his or her energy consumption requirements.

[0053] The formula for the maximum acceptable execution energy consumption for a user device shows that when the user device's current remaining battery power is low, the desired energy consumption is higher. As the battery power gradually increases, the user has sufficient battery power, so the desired energy consumption is reduced accordingly. Regarding the parameter ε, as ε increases, the user's demand for energy becomes higher when the battery power is low, and the desired energy consumption setting is lower. Therefore, different ε values ​​can meet the needs of different users. The formula for the user device's ideal execution energy consumption primarily serves as a constraint on the cooperative mechanism. Users cannot set the desired energy consumption too low, as this may affect all users. For rationality, the minimum desired energy consumption is set to half the maximum energy consumption for the task.

[0054] The present invention uses represents the uninstall decision of user device u, and defines When , it means that user u offloads the task to base station n for computation and transmits it using the k channel of that base station. Since the result of the task backhaul is much smaller than the original task data, this paper focuses on the task sending and computation stages.

[0055] The communication model in the multi-user ultra-dense network constructed by the present invention includes: when the user equipment selects a suitable base station and channel for transmission, the upload rate of the user can be obtained according to the Shannon formula:

[0056]

[0057] Among them B n represents the fixed bandwidth of base station n, p u,n (t) is the transmit power of user equipment u, σ 2 represents additive white Gaussian noise (AWGN), h u,n (t) is the channel gain between the user and the base station. In ultra-dense networks, since multiple base stations reuse the spectrum, the interference between devices cannot be ignored. u,n (t) is the interference received by the current device in the current channel, I u,n The calculation expression of (t) is:

[0058]

[0059] Among them, p u′,n′ (t) is the transmission power of other users using the current channel, h u′,n (t) represents the channel gain between the current channel user u′ and the base station n.

[0060] Based on the user's upload rate, the available uplink rate for user device u depends not only on the allocated channel and its own transmit power, but also on interference from other cellular user devices. Improving the user device's transmission rate requires not only appropriate offloading strategies and appropriate transmit power allocation, but also crucially, interference control between different cells.

[0061] The computing model of the present invention includes local computing and server computing.

[0062] For the sake of convenience, the present invention uses the energy consumption of local calculation as the maximum energy consumption criterion for executing tasks, so the calculation formula needs to be given:

[0063]

[0064] in is the CPU computing power of user device u, ρ = 10 -28 is the CPU coefficient, c u (t) represents the number of CPU cycles required for the computation task. The energy consumption of the local computation is set to the maximum energy consumption threshold of the task, that is:

[0065]

[0066] in, Indicates the maximum energy consumption threshold of the task, Indicates local computing energy consumption.

[0067] Server calculation involves the user device receiving the offload decision and selecting the appropriate base station and channel to process the task. Task processing involves two phases: task transmission and server calculation, so the latency incurred also includes these two phases. Regarding energy consumption, since this invention focuses solely on the user experience and MEC servers are generally powered by the grid, server computing energy consumption is ignored in this invention; only the energy consumption of the user device during task transmission is calculated.

[0068] The transmission delay obtained based on the user's upload rate is:

[0069]

[0070] where d u (t) is the input data size of the user task. After the task is transmitted to the base station, the server starts to calculate the task. The calculation delay is:

[0071]

[0072] Among them, c u (t) is the number of CPU cycles required to calculate the task, f u,n(t) is the computing resource allocated to user u by base station n, satisfying F n is the maximum computing capacity of base station n. The total computing resources allocated by the base station to all users cannot exceed this value.

[0073] The total delay of user equipment u can be obtained based on the transmission delay and calculation delay of the user equipment, and its calculation formula is:

[0074]

[0075] in, Indicates the transmission delay, Indicates the server calculation delay.

[0076] The user's transmission energy consumption is:

[0077]

[0078] Among them, p u,n (t) represents the transmission power.

[0079] Since the user only chooses to offload the calculation to the base station, the user's energy consumption is only the transmission energy consumption, that is:

[0080]

[0081] The process of training the task decision model includes:

[0082] S1: Initialize the parameters of the task decision model;

[0083] S2: In each time slot, the macro base station is used as an intelligent agent to obtain the current environment status information and the task information of the user equipment, and a task model is built based on the task information of the user task equipment;

[0084] S3: Input the current environment state information and task model into the policy network to obtain the task action, which includes the offloading decision, power control and computing resource allocation action of the current task;

[0085] S4: Calculate the satisfaction of the current macro base station based on the task action, and input the calculated satisfaction and task action into the action value network to obtain the immediate reward of the current macro base station;

[0086] S5: The macro base station obtains the next moment environment state information, and stores the current environment state information, task action, immediate reward, and next moment environment state information as a four-tuple into the priority experience replay array;

[0087] S6: A priority mechanism is used to sample the priority experience replay array, and the collected quadruple is input into the policy network and action value network for training respectively. When the immediate reward function of the macro base station converges, the model training is completed.

[0088] A user experience satisfaction model, which is:

[0089]

[0090] Among them, E u (t) represents the energy consumption of task execution, ω represents the sensitivity parameter of the satisfaction model, and They represent the ideal energy consumption and maximum energy consumption threshold for user u to perform this task respectively. is the median of the ideal energy consumption and the maximum energy consumption threshold for user u to perform this task, and its expression is:

[0091]

[0092] In a dynamic system, the user device's battery life decreases as the device's standby power consumption and task execution consumes energy. The satisfaction function for incoming tasks will change due to this change in battery life, resulting in a dynamic satisfaction model. ω∈[5,10] is the sensitivity parameter. By varying the value of ω, the function's curve can be controlled, making it unique and adaptable to user needs. As can be seen from the function, user satisfaction slowly decreases as energy consumption increases. When energy consumption exceeds the maximum energy consumption threshold, user satisfaction reaches 0.

[0093] Satisfaction function analysis: Before analyzing the satisfaction curve, we first introduce the concept of slope change point. A system's output sequence undergoes a sudden change at an unknown moment, called a change point. A slope change point is the point where the curve's slope accelerates (or decelerates) the most. Figure 2 Assuming that the ratio of maximum execution energy consumption to ideal energy consumption is 8:5, the ideal execution energy consumption formula is the one that satisfies the user's energy consumption requirements for the task. Figure 2is the impact of different sensitivity parameters within the value range on user satisfaction. From the perspective of a single curve trend, a curve has two slope change points. The present invention defines the change point on the left side of the figure as the first change point, and the change point on the right side as the second change point. When the energy consumption slowly increases from the ideal energy consumption, the user's satisfaction is slowly decreasing, but it can be seen from the slope of the curve that before reaching the first change point, the decline is relatively slow, which is also a reflection of the fact that the satisfaction function is closer to the user's feelings. When the energy consumption continues to increase and reaches the change point, it is already far away from the ideal energy consumption, and the satisfaction begins to drop sharply compared to before, until the second change point is encountered. At this time, the satisfaction has dropped to a lower level, and the amplitude of the change again is not large compared to before.

[0094] Moving from a single curve to multiple curves, the figure shows that as the sensitivity parameter ω increases, the first inflection point of the user satisfaction curve rises. This indicates that user satisfaction is higher when energy consumption is low. Once energy consumption begins to increase, the degree of decline in satisfaction increases. This setting can meet different needs in different situations. For example, if a user currently has stringent energy consumption requirements, they can obtain more resources by increasing the parameter. However, considering the computing needs of other users, the parameter value cannot be increased indefinitely, so the setting range of the parameter is limited.

[0095] In addition to being more responsive to user needs, this curve also offers another advantage. When solving optimization problems, such as maximizing overall user satisfaction, assuming all users are cooperative, it's possible to maximize overall user satisfaction by maximizing user satisfaction near the first change point rather than the ideal state. This allows for greater concessions to users with fewer resources, ultimately achieving greater satisfaction with fewer resources.

[0096] Optimization problem description: Based on the above description, the present invention proposes a long-term optimization problem to maximize the satisfaction of all users:

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] Among them, C1 and C2 are offloading variable constraints. C1 restricts the offloading decision value to be only in K channels, and C2 means that one channel of a base station can only be assigned to one user equipment. G {#} =1 indicates that # in the function is true. C3 is the user's transmit power constraint. The transmit power of each user cannot be greater than the maximum power supported by the user equipment. C4 represents the computing resource limit allocated to each base station by the user, that is, it cannot exceed the maximum computing resource of the base station. Constraint C5 is to ensure that the task can be completed within the maximum tolerable delay. U represents the total number of users, N represents the total number of base stations, u represents user u, n represents base station n, and λ u,n (t) represents the unloading decision variable, represents the uninstall decision representation function, p u,n (t) represents the transmit power, Indicates the maximum transmit power, f u,n (t) represents the computing resource allocation variable, F n represents the maximum computing resource of base station n, T u (t) represents the execution delay, τ u (t) represents the delay threshold.

[0104] The optimization problem P described above is a non-convex mixed-integer nonlinear programming problem involving the optimization of three variables. λ is a U×N-dimensional decision matrix, where the values ​​are discrete integers no greater than K, while p and f are continuous real vectors related to all users. The optimal solution to problem P under static conditions requires complete information about the mathematical model, such as the statistical distribution of channel conditions, which is not available in practical systems. Furthermore, the present invention is applied to dynamic MEC systems, where problem P is difficult to solve using traditional optimization algorithms under time-varying conditions. Reinforcement learning (RL) is a relatively advanced decision-making method that continuously learns through trial and error in a target environment, feeds back results, and modifies strategies to maximize rewards. Despite its many advantages, it also lacks scalability and is inherently limited to relatively low-dimensional problems. This is primarily because RL algorithms have the same memory, computational, and sample complexities as other algorithms. Therefore, to address high-dimensional decision-making problems that are difficult for RL, deep reinforcement learning combines the perception capabilities of deep learning with the decision-making capabilities of RL. By enhancing function approximation and deep neural networks, it addresses the problem of high-dimensional state and behavior spaces.

[0105] This paper proposes a deep reinforcement learning algorithm based on the RL framework and combines it with the aforementioned optimization problem to solve the multi-user computation offloading problem. The RL framework primarily consists of an agent, an environment, and three elements: a state space, an action space, and a reward.

[0106] In order to solve problem P in the reinforcement learning framework, the present invention defines the following elements based on the system model:

[0107] (1) State space: the set of all possible states At the beginning of each time slot, the macro base station observes the system status of the wireless network, including the task details of all devices requested, the available computing resources of each base station MEC server, the computing power of each user device, the remaining power of the user device, and the channel gain between each base station and the device. It can be defined as:

[0108]

[0109] Where V(t) is the task request feature matrix of all users, which contains the data features of all user devices reaching the task. H(t) is the channel gain matrix between each base station and each user. F(t) is the computing resource vector of all base station servers, which represents the computing resources available to the current server, and The current remaining power vector of the user device.

[0110] (2) Action space: the set of all possible actions of the agent Based on the currently observed system state s t , the agent selects different actions based on the decision variables of problem P, action a t It can be defined as:

[0111]

[0112] Among them, λ(t) is the base station selection and channel selection strategy action, p(t) is the user equipment's transmit power allocation action, and f(t) is the computing resource action allocated by the base station to the user.

[0113] (3) Reward: Reward function Refers to a reward returned to the agent by the environment after the agent performs an action: in, represents the state space, represents the action space, Denotes the reward function. The goal of the joint computation offloading and resource allocation problem P proposed in this paper is to maximize the satisfaction of all users, and its result can be directly used as the immediate reward. The immediate reward function is defined as:

[0114]

[0115] of which € u (t) is the penalty function, which is defined as follows:

[0116]

[0117] where υ 1 and υ 2 are two positive real numbers, and satisfy υ 1 ≤O u (t)≤υ 2 , O u (t) represents user satisfaction.

[0118] At each time slot t, the agent observes the environment and take action The action of the agent is determined by the policy function μ. After the action is executed, the environment returns a scalar reward r to the agent. t , and transition to the next state s t+1 .

[0119] The return refers to the sum of all rewards from the current moment to the end of the round, which can be expressed as:

[0120]

[0121] Where γ∈[0,1] is the discount factor, which indicates the importance of rewards. The further away from the current moment, the greater the discount. ε=(s0,a0,s1,a1,…) is the reward brought by a series of states and actions. Among them, s1 represents the state at that moment, and a1 represents the action at that moment.

[0122] The policy function μ in reinforcement learning refers to the strategy that the agent uses to take actions based on the current state. The action value function Q μ (s,a) is the expected reward of the agent taking the current action based on the current state. The learning goal of the agent is to learn the best strategy and take the best action in the current state to obtain the maximum value. The optimal action-value function can be written as:

[0123]

[0124] Among them, Q * (s t ,a t ) represents the action-value function at time t.

[0125] For RL problems with high-dimensional state space and state space, it is difficult to get an exact solution by directly maximizing the value function. One feasible method to solve RL problems is the well-known Q-learning algorithm, which is a type of Temporal Difference (TD) algorithm. Q-learning uses a table to store the best action value. Each state corresponds to a different action with a predicted action value. Each time an action is selected, the action with the largest value is selected.

[0126] As the state space and action space increase, the complexity of solving the priority score increases exponentially, and Q learning can no longer meet the demand. In order to solve this problem, the deep reinforcement learning (DRL) algorithm that uses a deep neural network (DNN) to approximate the function is an effective alternative. The DQN (Deep Q-network) algorithm uses the DNN network to approximate the action-value function to solve the optimal action-value function. Although DQN can successfully solve problems in high-dimensional state space, it can only handle discrete and low-dimensional action spaces. When there are a limited number of discrete actions, DQN can be solved first, but for continuous variables, such as the power allocation and computing resource allocation variables of problem P, the action space must be discretized before using this method. Discretization will cause a loss of accuracy, and as the number of discretization layers increases, its computational complexity increases exponentially. Therefore, it is necessary to find a deep reinforcement learning algorithm that is more suitable for the problem raised by the present invention.

[0127] Deep Deterministic Policy Gradient. Deep Deterministic Policy Gradient is the most commonly used continuous control method. It is an Actor-Critic based algorithm consisting of a policy network (actor) μ(s; θ μ ) to approximate the policy function, a value network (judge) Q(s,a;θ Q ) to approximate the action value function, θ μ and θ Q The policy network is mainly responsible for outputting actions based on the current state. The output of the policy network is the determined action, while the output of the value network is the current state and the value of the action output by the policy network. That is, the action is scored based on the current state to guide the actor to make better actions.

[0128] Experience replay is an important technique in reinforcement learning that can significantly improve the performance of reinforcement learning. The records (experiences) of the interaction between the agent and the environment are stored in an array and these experiences are used to train the agent. This array is called the experience replay array (Replay Buffer). The policy network is used to control the interaction between the agent and the environment, and the four-tuple (s t ,a t ,r t ,s t+1 ) into the experience replay array, and then a certain amount of experience is extracted from the array to train the policy network and the value network.

[0129] The process of training the policy network includes: given the current state s t , the policy network outputs action a t =μ(s t θ μ ), the value network gives the action a score based on the current state: q t =Q(s t ,a t θ Q ). The purpose of training the policy network is to improve the parameter θ μ , so that q t Become as large as possible. The action value can be written as: t =Q(s t ,μ(s t θ μ );θ Q ). The expectation of the action value with respect to the state is:

[0130]

[0131] The purpose of training the policy network parameters is to maximize J(θ μ ). During training, the value network parameters θ are fixed. Q , use gradient ascent to increase J(θ μ ), the update formula is as follows:

[0132]

[0133] Where β is the learning rate, is the number of experience groups drawn from the experience return array. Gradient ascent can gradually make J(θ μ ) increases, so that the judges give the actors higher scores; r B Represents the experience group r B , Indicates the gradient of the policy network parameters, ▽ a Indicates the gradient of the action, represents the policy network, represents the action-value network.

[0134] The process of training the action value network includes: The purpose of training the value network is to make its prediction closer to the true value function Q(s,a). The value network is like a judge. In order to make it more and more accurate in scoring the actors, its scoring needs to be calibrated according to the actual observed rewards. The training of the value network mainly uses the TD algorithm mentioned above to make the value network fit the TD target. Extract a set of experience groups (s from the experience replay array) t ,a t ,r t ,s t+1 ), first let the value network do the evaluation to get q t =Q(s t ,a t θ Q ) and q t+1 =Q(s t+1 ,μ(s t+1 θ μ );θ Q ), calculate the TD target The loss function is defined as:

[0135]

[0136] Update the parameters using the gradient descent method as follows:

[0137]

[0138] in is the TD error, α is the learning rate, Denotes the gradient of the action-value network parameters. Using gradient descent can make the loss function smaller, that is, make the value network evaluation closer to the TD target.

[0139] The target network is used to alleviate the overestimation problem. The target value network Q(s t ,a t θ Q′ ) and the target policy network μ(s t ,θ μ′ ) is used to calculate the TD target.

[0140] The algorithm proposed in this invention is based on the DDPG framework. The key parts are roughly the same as those introduced above. The algorithm framework is as follows: Figure 3 The present invention makes changes based on the original algorithm, improves the efficiency of the algorithm, and is more suitable for the problem raised by the present invention.

[0141] In traditional DDPG, uniform sampling is widely used, randomly extracting a small batch of experience samples from the experience replay array for network parameter training. This approach ignores the importance of experience groups. Different experience groups should have different importance, allowing for quick perception of successful or failed experiences to accelerate convergence. Therefore, some researchers have introduced experience replay methods based on PER to address this issue, in which more valuable experiences are replayed with a higher probability. Each experience group is associated with a priority, and the replay probability is calculated based on the priority values ​​of all experience groups in the replay array. In PER, experience groups that are more successful or unsuccessful are more likely to be included in the training batch, which helps shorten learning time and improve training stability.

[0142] When studying PER techniques, defining priority is a key issue. Absolute TD error is often used as an important metric for evaluating experience priority. Experience groups with higher absolute TD error values ​​are given higher priority, indicating that the neural network's estimate of the true value of the action is inaccurate. Giving them a higher weight helps the neural network reduce the probability of incorrect predictions.

[0143] Inspired by the above priority analysis, the present invention proposes a priority mechanism that is more suitable for the present problem. When resources are limited, the MEC network cannot achieve optimal energy consumption for all user tasks. Consequently, different users may receive different resources. Curve analysis shows that, as energy consumption decreases, after the second inflection point, satisfaction increases almost linearly with energy consumption, with a relatively rapid rate of increase. However, after the first inflection point, the increase in satisfaction begins to slow. This shows that, for the same cost, the magnitude of change in satisfaction before and after the first inflection point differs. Assuming all users are cooperative, since the present invention's optimization goal is to maximize the satisfaction of all users, if users with satisfaction above the first inflection point maintain their satisfaction near that point, more resources can be freed up for users with fewer resources, i.e., those with satisfaction below the first inflection point, providing them with more room for satisfaction improvement. This results in a higher overall satisfaction score. Based on the above analysis, the present invention proposes a priority mechanism based on maximizing user satisfaction. This prioritizes experience groups with a high level of user satisfaction near the first inflection point, facilitating rapid learning for the intelligent agent.

[0144] First, we propose a The priority score formula is:

[0145]

[0146] in, It is a score control parameter, which can be adjusted to control the importance of scores for different indicators. is the absolute value of the TD error, Is a group with this experience function related to user satisfaction.

[0147] Before putting the experience group into the experience replay array, the satisfaction curve formula can be derived based on the current environment, that is, the user's needs, and then the first slope change point of the curve is calculated using the second-order difference method of the regression coefficient. Based on the action output of the policy network, the specific satisfaction of each user can be calculated. u (t) and specific energy consumption E u (t). Assume that the value of the first slope change point calculated by the current user is And the satisfaction level corresponding to that point. Collect the energy consumption of users whose satisfaction level is greater than the first change point and calculate the difference between the two Based on these differences, the distribution of the difference between the user's energy consumption and the first change point can be calculated. The formula is as follows:

[0148]

[0149] The probability distribution of the difference can be used to derive the function

[0150]

[0151] The numerator of the above formula represents the entropy of the distribution of user energy consumption differences. The larger the value, the smaller the gap between user energy consumption differences. The denominator is the difference between the energy consumption of all users and the change point. The smaller the value, the closer the user's energy consumption is to the change point. In special cases, the network and computing resources are relatively abundant, and the energy consumption of all users stays near the ideal energy consumption. At this time, experience is not very helpful for the learning of the neural network, although the function Its numerator is relatively large, but the denominator is also relatively large, and the overall priority is not high, so the function Special circumstances are also considered.

[0152] After obtaining the priority scores, sort all experience groups in ascending order and give each experience group a serial number. Define a sampling value based on the sequence number:

[0153]

[0154] The higher the priority score, the larger the sampling value of the experience group. Finally, the sampling probability of each experience group is obtained:

[0155]

[0156] The parameter l can control the degree of influence of the priority on the sampling probability. However, the uneven sampling of the priority experience replay will affect the neural network. In order to correct the influence of the priority experience replay, the learning rate needs to be reset as follows:

[0157]

[0158] Among them, the parameter ψ is an annealing variable, is the number of all experience groups.

[0159] After the above optimization, the general framework of the reinforcement learning algorithm is as follows Figure 3 As shown in the figure. The algorithm first initializes the parameters of each neural network, and then lets the agent use the strategy main network to output an action according to the current environment. The environment returns a reward to the agent based on the action, and the environment enters the next state. The collected quadruple is added to the experience replay array, and the sampling probability is calculated according to the proposed priority experience replay strategy. After extracting a certain number of experience groups, the neural network is trained. The algorithm flow is as follows Figure 4 shown.

[0160] Figure 5 The figure compares the proposed algorithm with existing algorithms. The main simulation parameters are set as follows. The number of micro base stations in the network is 10, and the number of users is 30. The task parameters are c∈[300,500]kbits, the number of CPU cycles required per bit for all tasks is d is 1000 cycles / bit, and the maximum tolerable delay τ∈[20,50]ms. The maximum transmit power of the user equipment is p max =2W, the standby power is 1×10 -5 W. The computing power of the user equipment is f l = 1 GHz. Different base stations have different computing capabilities, F∈[15,25] GHz. The bandwidth of each base station is set to B = 10 MHz, and the channel is divided into K = 5 subchannels. The additive white Gaussian noise is set to -174 dBm / Hz. The channel gain between the user and the base station follows the free-space path loss model and is expressed as:

[0161]

[0162] The antenna gain A d =4.11, the carrier frequency is f c =900MHz, d u,n (t) is the distance between the user equipment and the base station. The maximum power of the user equipment The user's current power is randomly distributed.

[0163] The proposed algorithm contains four neural networks, including a value network, a policy network, and two corresponding target networks. Both the value network and the policy network are composed of four fully connected layers, including two hidden layers, one input layer, and one output layer. Each hidden layer contains 256 neurons. The number of neurons in the input layer of the policy network is the size of the state space, and the size of the output layer is the action space. The input layer size of the value network is the size of the action space, and the number of neurons in the output layer is 1. The hidden layers of the neural network use the Rectified Linear Unit (ReLU) as the activation function, and the output layer uses sigmoid as the activation function. The Adam optimizer is used to update the neural network parameters. Number of experience group samples The neural network learning rate is 0.01, and the size of the priority experience replay array is 512.

[0164] from Figure 5 It can be seen that the proposed algorithm reaches convergence after a certain number of iterations and achieves a relatively high satisfaction effect, and is superior to the existing reinforcement learning algorithm in terms of convergence.

[0165] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation plans of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An edge computing offloading method based on dynamic user satisfaction in ultra-dense networks, characterized in that: include: Build a multi-user ultra-dense network and initialize the network; A user device in the network generates a computing task and sends a task offloading request to the macro base station; After receiving the request, the macro base station obtains user device information and network environment information, and builds a task model based on the obtained user device information. The environment information includes the channel status between the user device and the micro base station and the computing resource information of the MEC server. The task model and network environment information are input into the trained task decision model to obtain the task offloading decision. The task decision model includes a policy network and an action value network. The macro base station sends the task offloading decision to the user device and the micro base station respectively. The user device executes the task offloading according to the task offloading decision, and the micro base station allocates computing resources according to the task offloading decision to perform task calculation. Building a task model includes: obtaining user device information, which includes the size of the task input data generated by the user device, the number of CPU cycles per task unit, the user's delay requirement for executing the task, the user's energy consumption requirement for the task, the maximum energy consumption that the user can accept when executing the task, and the remaining power of the user device in the current time slot; the task model built based on the user device information is: Among them, d u (t) represents the input data size of the task, c u (t) represents the number of CPU cycles per unit to execute the task, τ u (t) represents the user's execution delay requirement for the task, i.e., the maximum tolerable delay. It represents the user's energy consumption requirement for the task, that is, the ideal execution energy consumption. Indicates the maximum execution energy consumption that the user can accept. Indicates the remaining power of the user equipment in the current time slot; The formula for determining the maximum execution energy consumption and ideal execution energy consumption that the user device can accept is: Among them, g represents the ideal energy consumption calculation function, represents the maximum execution energy consumption, Indicates the remaining power of the device. is the maximum storage capacity of the user device, and ε is a hyperparameter.

2. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 1 is characterized in that: The multi-user ultra-dense network consists of a macro base station and N micro base stations. Each micro base station is equipped with an MEC server to perform computing tasks; each micro base station uses orthogonal frequency division multiple access to user devices.

3. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 1 is characterized in that: The process of training the task decision model includes: S1: Initialize the parameters of the task decision model; S2: In each time slot, the macro base station is used as an intelligent agent to obtain the current environment status information and the task information of the user equipment, and a task model is built based on the task information of the user task equipment; S3: Input the current environment state information and task model into the policy network to obtain the task action, which includes the offloading decision, power control and computing resource allocation action of the current task; S4: Calculate the current user's satisfaction based on the task action and obtain the immediate reward of the current macro base station; S5: The macro base station obtains the next moment environment state information, and stores the current environment state information, task action, immediate reward, and next moment environment state information as a four-tuple into the priority experience replay array; S6: A priority mechanism is used to sample the priority experience replay array, and the collected quadruple is input into the policy network and action value network for training respectively. When the immediate reward function of the macro base station converges, the model training is completed.

4. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 3 is characterized in that: The immediate reward function is: in, represents the reward function, s t Indicates the current environment state, a t represents an action, u represents a user, U represents the total number of users, O u (t) represents the user's satisfaction with the macro base station's performance of the current task, € u (t) represents the penalty function.

5. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 3 is characterized in that: The process of sampling the priority experience replay array using the priority mechanism includes: using the priority score formula to calculate the priority score of each data in the priority experience replay array; sorting the data in descending order according to the calculated priority score, labeling the sorted data, and defining a sampling value for each data according to the label; calculating the sampling probability of each data according to the sampling value; and setting the learning rate for task decision model training according to the sampling probability.

6. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 5, characterized in that: The formula for setting the learning rate of task decision model training is: Where ψ represents the annealing variable, represents the number of all experience groups, α represents the initial learning rate, represents the sampling probability.

7. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 3 is characterized in that: The process of training the policy network includes: given the current state s t , the policy network outputs action a t =μ(s t θ μ ), the value network gives the action a score based on the current state: q t =Q(s t ,μ(s t θ μ );θ Q ); Calculate the expected value of the action with respect to the state according to the action score J(θ μ ); fix the value network parameter θ Q , update the parameters θ using the gradient ascent algorithm μ , when J(θ μ ) The training of the policy network is completed when convergence.

8. The edge computing offloading method based on dynamic user satisfaction in an ultra-dense network according to claim 3 is characterized in that: The process of training the action-value network includes: Step 1: Extract a set of experience groups (s) from the experience replay array t ,a t ,r t ,s t+1 ); Step 2: Input the experience group into the action value network to obtain q t =Q(s t ,a t θ Q ) and q t+1 =Q(s t+1 ,μ(s t+1 θ μ );θ Q ); where Q represents the value network, s t Indicates the current environment state, a t represents the action, θ Q represents the value network parameter, q t+1 represents the predicted output of the value network at the next moment, and μ represents the policy network parameter; Step 3: According to q t+1 Calculate the TD target using the following formula: Among them, r t represents the immediate reward, γ represents the discount factor; Step 4: According to TD target, q t and q t+1 Calculate the network's loss function; Step 5: Use the gradient descent algorithm to calculate θ Q Updates are performed and the training of the action value network is completed when the network's loss function converges.

Images