A Multi-User Task Offloading Method Based on Deep Reinforcement Learning

Abstract

This invention discloses a multi-user task offloading method based on deep reinforcement learning. This method obtains the starting and ending points of the current device. A macro base station uses existing environmental information to predict the device's path and recommends a path with minimal interference to the user device. The user device actively obtains information about base stations that can provide services to it; other environmental information is mainly obtained from the serving base station and the macro base station. The device selects a base station from the available micro base stations as its serving base station. Using the information provided by the base station, interference factors are identified and used as the basis for determining whether to offload. Then, in the offloading process, DQN is used to determine the serving base station and make the offloading decision. This invention utilizes edge computing and deep reinforcement learning techniques, with base stations and devices cooperating. The base station recommends paths for the device, and the device optimizes the offloading method using the recommended paths, resulting in a lower offloading task failure rate and latency.

Description

Technical Field

[0001] This application relates to the field of edge computing, specifically to a multi-user task offloading method based on reinforcement learning. Background Technology

[0002] Mobile Edge Computing (MEC) is a network physically located close to the data source. It's an open platform integrating core capabilities such as networking, computing, storage, and applications, providing edge intelligence services locally. Edge computing processes data closer to the user. It allows software applications to leverage local content and real-time information about local network access conditions. By deploying various services and cached content at the network edge, the mobile core network can further alleviate congestion and effectively serve local devices. The location where edge computing occurs is called an edge node, which can be any node with computing and network resources between the data source and the cloud center. These edge device nodes provide rich service interfaces, working with the cloud computing center to provide collaborative computing services to users. This is because cloud computing's main characteristic is its ability to handle large amounts of data and perform in-depth analysis, rather than real-time data processing. Edge computing, on the other hand, focuses on local needs and can play a better role in small-scale, real-time intelligent analysis, such as meeting the real-time needs of local businesses. Therefore, compared to cloud computing networks, mobile edge networks have advantages such as lower latency, smaller bandwidth, higher power efficiency, closer service, and the ability to fully utilize environmental information.

[0003] In real-world scenarios, many factors can affect task offloading performance, including but not limited to mobility management, interference management, and resource allocation. Therefore, these related factors are discussed below.

[0004] Mobility management (MM) involves deciding whether to change the serving base station (handover) and whether to change the location of virtual machines (VM migration). Therefore, a key issue in mobility management is how to ensure service continuity.

[0005] The first method uses historical trajectory data to predict movement paths. The second method uses user preferences, goals, and analyzed spatial information to predict trajectories and destinations without any analysis of the user's historical movement trajectories.

[0006] Interference management refers to the potential for severe mutual interference when multiple user devices simultaneously select the same wireless channel to offload tasks, leading to reduced offloading efficiency and consequently higher latency and energy consumption. In networks with multiple MEC servers, the interplay between user transmission mode selection and offloading target server determination further complicates MEC scheduling.

[0007] The purpose of resource allocation is to allocate the limited resources within the edge node according to the needs of the devices as much as possible, thereby reducing latency and improving the user experience.

[0008] The core idea of ​​reinforcement learning is a trial-and-error mechanism. Based on the interaction between the agent and the environment, the agent autonomously trains and learns, exploring the entire environment as much as possible to learn the optimal action. The agent is an entity that can perceive the state of the environment and take corresponding actions. Clearly, reinforcement learning is a trial-and-error process, obtaining the maximum cumulative reward value by continuously trying different actions. Therefore, the key elements of reinforcement learning are: 1) Environment; 2) State: a description of the agent's current environment; 3) Action: the behavior taken by the agent based on the observed state; 4) Reward: the feedback given to the agent by the environment after the agent performs an action.

[0009] The Depth-Quantity (DQN) algorithm is an improvement on Q-Learning. Q-Learning requires maintaining a Q-Table, and the learning process involves updating this Q-Table. DQN replaces this Q-Table with a neural network. The input to the DQN neural network is state information, and the output is the value of each action. The goal of reinforcement learning is to obtain the maximum reward value through continuous learning. Therefore, the DQN algorithm can be used to solve problems in continuous state spaces and discrete action spaces, but it cannot solve problems in continuous action spaces.

[0010] DQN's modifications to Q-Learning are mainly reflected in three aspects: using deep convolutional neural networks to approximate the value function; using experience replay to train the reinforcement learning process; and setting up an independent target network to handle the TD bias in the temporal difference algorithm separately. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multi-user task offloading method based on reinforcement learning.

[0012] The objective of this invention is achieved through the following technical solution: a multi-user task offloading method based on deep reinforcement learning, the method comprising the following steps:

[0013] (1) Establish a network architecture, which consists of three layers: a cellular access layer, an edge computing layer, and a macro base station layer. User path recommendation is completed by the macro base station layer.

[0014] (2) When a user device enters the network, the device determines its starting point and destination and provides path recommendations to the user through the macro base station. The macro base station uses existing environmental information to make path recommendations. Specifically, the macro base station collects information on the devices in each micro base station and then determines whether the user device is on a predictable path. If the user enters the current network multiple times, the walking path can be predicted using historical trajectories. If the user does not enter the network multiple times, the user path can be inferred based on the user device's search record context information. When the user prohibits the base station from collecting information, the macro base station collects information on the devices in each micro base station to recommend the current device's path.

[0015] When the device adopts a predicted path, the task offloading scheme becomes a collaborative offloading scheme between the base station and the user equipment. For user offloading tasks, when multiple users enter the network, the information obtained by the user is not only the base station environment information, but may also be interfered with by other users. The user equipment can actively obtain the base station information that can provide services to the device, while the other environmental information is mainly obtained from the serving base station and the macro base station. The macro base station will inform the device of other devices in the current network and their predicted travel paths based on the existing information. The device trusts the macro base station's predicted path. The device selects a micro base station as the serving base station among the micro base stations that can provide services. It uses the information provided by the serving base station to determine the interference factors and uses this as the basis for deciding whether to offload. If the interference is too great and the local processing time is less than the transmission delay, it will not be offloaded to the serving base station for processing. Otherwise, it can be offloaded to the serving base station for processing. In the offloading process, DQN is used to determine the serving base station and make the offloading decision.

[0016] The data structure of DQN is as follows:

[0017] ACTION: An action consists of two parts: whether to uninstall and where to uninstall. Both parts are represented by the same vector, i.e., a = {a0, a1, ..., a...}. n}, where a0 = 0 indicates that the current part of the task is processed by the local device, a1, ..., a n This indicates that some tasks are currently being handled by the serving base station, a n This means offloading part of the current task to the nth base station a. j =j indicates that the serving base station for the current task is b. j ;

[0018] STATE: Status is This indicates the current location of the user equipment; the location of each user entering the network needs to be clearly defined; b j Indicates the base station currently connected to the device;

[0019] REWARD: Daley indicates the task processing delay.

[0020] Furthermore, suppose there are M users entering the network, u i This represents the i-th user entering the network, where i = {1, 2, ..., M}; the edge access layer has n micro base stations, denoted by B = {b1, b2, ..., b...}. n} indicates that time is divided into multiple time slots, and the user's connection information is determined in each time slot; δ i,j ={0,1 represents user u i Is it related to base station b? j Connected, where j = {0, 1, 2, ..., n}; ∑ j δ i,j =1 means that a mobile device is only allowed to connect to one base station within a time slot;

[0021] Q is used to describe the task, and user u i The task set is represented as Q i ={q i,1 ,q i,2 ,...,q i,K}, let q i,k ={τ i,k ,ω i,k ,σ i,k}, k={1,2,…,K}, where K is the total number of tasks, and τ i,k Indicates a user u i The kth task q of the device i,k The computational cost, ω i,k Represents computational task q i,k Size, σ i,k Indicates user equipment u i The task of the device q i,k The maximum time limit for completion; if this time limit is exceeded, the task is considered a failure.

[0022] Furthermore, assume the uplink data rate c when the user's mobile device offloads tasks to the serving base station in the edge cloud. i,j,t The calculation formula is as follows:

[0023]

[0024] Where B is the channel bandwidth, w j Let g(t) be the transmission power, g(t) be the channel gain of the user's mobile device, and σ be the transmission power. 2 Let I be the Gaussian white noise power, and let I represent the wireless interference caused to the base station by user equipment performing other transmission tasks. The formula for calculating I is as follows:

[0025]

[0026] The term represents the radio interference caused to the base station by user equipment performing other transmission tasks, g′(t) is the channel gain of other user equipment, and δ represents the possible obstruction interference.

[0027] To user u i Uninstall task q i,k The formula for calculating transmission delay is as follows:

[0028]

[0029] k represents the k-th task, i represents the number of users, j represents the number of base stations, t represents time, and e represents the time interval. T This represents the processing of data transmitted to the service base station in the edge cloud, where δ in the formula... j =1 indicates that the user has connected to the serving base station, and the transmission delay is calculated; u i Uninstall task q i,k The formula for calculating the energy consumption of the k-th task transmission to the edge is as follows:

[0030]

[0031] Furthermore, the factors influencing whether a task is unloaded are derived from the user device's u i From a certain perspective, it refers to the remaining battery energy of the user device. i And computing power f i From the perspective of the base station, it's about base station resource allocation; from the perspective of the task, it's about the energy required for task processing. k ;

[0032] There are two scenarios regarding where computing tasks are processed: local processing or offloading to a serving base station for processing.

[0033] Local computation:

[0034] When the device's remaining battery power is sufficient to support task processing, the task can be processed locally; the formula for calculating the local processing time of the task in this case is as follows:

[0035]

[0036] Where f i User Equipment u i The computing power; from this, the energy consumption of local task processing can be obtained, calculated as follows:

[0037]

[0038] ρ represents the energy consumption of the k-th task processed locally. k It is the power factor of the energy consumed per CPU cycle;

[0039] Unload to the serving base station:

[0040] When the user device's resources and remaining battery power are insufficient, the task will be offloaded. Factors affecting this include upload speed and the base station's processing capacity. The task duration is calculated using the following formula:

[0041]

[0042] in, The proportion of resources allocated by the current serving base station to connected devices for the current user's unloaded task. This represents the computing power of the edge cloud of the j-th serving base station.

[0043] Furthermore, the task placement problem involves deciding whether to handle the task on the user's local mobile device or at a serving base station in the edge cloud, taking into account the user's mobile location, and determining which edge cloud serving base station the task will be offloaded to; an integer decision variable, x, is defined. i,k ∈{0,1}, which represents task q i,k It is local (x) i,k =0) or in the edge cloud (x i,k =1) is processed; the formula for minimizing the total task duration is as follows:

[0044]

[0045] :j i ∈A(t)

[0046]

[0047]

[0048]

[0049] This formula is used to obtain the minimum total duration of the task unloading problem, which is then used to determine the optimal unloading solution; this formula is constrained by the four formulas mentioned above: j i ∈A(t) indicates that the edge server selected for the current device task offloading must be able to provide services, and A(t) represents the combination of base stations that can be offloaded; This represents the total energy consumption of tasks performed on a mobile device. The sum of these values ​​represents the energy consumed during transmission, and cannot exceed the total energy e of the device; δ i,j The sum of 1 indicates that the mobile device u in each time slot i Only one edge server can be connected; The sum of 1 is less than or equal to 1, which means that the total amount of computing resources allocated by the current serving base station to each user for each task cannot exceed the resource capacity of the base station.

[0050] Furthermore, the execution process of DQN involves first inputting action, state, and reward data into the Q-network, which then provides a predicted value Q. predict Then, based on the predicted value, the optimal action is selected to interact with the environment to obtain the latest state s′ and the network is trained; the results of each interaction are stored in the experience pool; a target network with the same network structure and parameters as the Q network is copied, and this target network samples data from the experience pool to output a stable target value Q. target =r+γQ target (s,s′,,), where γ represents the reward discount rate and r represents the reward; finally, the squared error loss is calculated using the target value and the output of the neural network to evaluate the degree of difference between the model's predicted value and the true value. The calculation formula is as follows:

[0051] L(θ)=E[(Q target -Q(s,a;θ)) 2 ].

[0052] Furthermore, to improve the scalability of the path recommendation algorithm, policy-based reinforcement learning is used for prediction, employing a proximal policy optimization (PPO) deep reinforcement learning algorithm; PPO is based on the action probability π of the current policy. θ (a|s) and the action probability π of the previous strategy θ′ The ratio of (a|s), the probability ratio r(θ) of choosing a certain Action by the old and new strategies, is calculated as follows:

[0053]

[0054] When r(θ)>1, it means that the current strategy is more inclined to choose action a;

[0055] Based on this, set the objective function L Clip (θ), which limits the update magnitude, is calculated using the following formula:

[0056]

[0057] Where θ is the policy parameter, r(θ) is the ratio of the probabilities of the new and old policies, A is the dominance function, and ∈ is a hyperparameter. It represents expected experience.

[0058] The beneficial effects of this invention: This invention proposes a multi-user task offloading method based on deep reinforcement learning. Utilizing edge computing and deep reinforcement learning techniques, the base station and the device collaborate, with the base station recommending paths for the device, and the device optimizing the offloading method using the recommended paths, resulting in a lower offloading task failure rate and latency. Attached Figure Description

[0059] Figure 1 This is a network diagram of the present invention.

[0060] Figure 2 This is a flowchart of an uninstallation method.

[0061] Figure 3 DQN neural network convergence graph.

[0062] Figure 4 DQN reward value convergence graph.

[0063] Figure 5 Task processing latency (uniformly distributed) under different algorithms.

[0064] Figure 6 Task processing latency (normal distribution) under different algorithms.

[0065] Figure 7 Number of task failures. Detailed Implementation

[0066] To make the mobile task unloading method of this application clearer, the technical solution of this application will be clearly and completely described below with reference to the accompanying drawings and specific embodiments:

[0067] like Figure 1 and 2 As shown, this invention provides a multi-user task offloading method based on deep reinforcement learning. This method establishes a network architecture consisting of three layers: a cellular access layer, an edge computing layer, and a macro base station layer. Path recommendation is completed by the macro base station layer.

[0068] When user equipment enters the network, interference may occur between users. This interference could be channel interference caused by multiple users connecting to the same base station, or interference caused by obstruction between users. Therefore, different interference issues need to be considered when making offloading decisions. Simultaneously, macro base stations can be used to recommend routes to users to avoid interference. Each device clearly defines its origin and destination, and macro base stations provide route recommendations. Macro base stations utilize existing environmental information to achieve route recommendations: specifically, macro base stations collect information from devices in each micro base station and then determine whether the device follows a predictable path; if a user enters the current network multiple times, the travel path can be predicted using historical trajectories; otherwise, the user's path can be inferred based on the user device's search history and contextual information. However, not all user equipment allows base stations to obtain device information; users may prioritize privacy. While such user equipment is permissible, it is not the focus of this invention. When a user prohibits base stations from collecting information, that user can only be treated as interference or noise. Then, the current device's path is recommended using other device roadmaps.

[0069] When the device adopts this path, the task offloading scheme transforms into a collaborative offloading scheme between the base station and the user equipment. However, the user can also choose not to take this path; in this case, the path recommendation is merely an auxiliary suggestion. For user offloading tasks, when multiple users enter the network, the information obtained by the user includes not only base station environmental information but also potential interference from other users. The user equipment can actively obtain information about base stations that can provide services to the device, while other environmental information is mainly obtained from the serving base station and macro base stations. The macro base station informs the device of other devices in the current network and their predicted travel paths based on existing information. The device trusts the macro base station's predicted path and selects a micro base station among the available micro base stations as the serving base station. Using the information provided by the serving base station, the device identifies interference factors and uses this as the basis for deciding whether to offload. If the interference is too great, the device will not offload to the edge server. DQN is used in the offloading process to determine the serving base station and make offloading decisions.

[0070] Suppose there are M users entering the network, and ui represents the i-th user entering the network, where i = {1, 2, ..., M}. The edge access layer has n micro base stations, denoted by B = {b1, b2, ..., b...}. n} represents this. We divide time into multiple time slots and determine the user's connection information in each time slot. δ i,j ={0,1 represents user u i Is it related to base station b? j Connected, where j = {0, 1, 2, ..., n}. Note: ∑ j δ i,j =1 indicates that a mobile device is only allowed to connect to one base station within that time slot.

[0071] Q is used to describe the task. i ={q i,1 ,q i,2 ,...,q i,K}, let q i,k ={τ i,k ,ω i,k ,σ i,k}, k={1,2,…,K}, where τ i,k Indicates a user equipment u i Task q i,k The computational cost, ω i,k Represents computational task q i,k Size, σ i,k Indicates user equipment u i Task q i,k The maximum time limit for completion; if this time limit is exceeded, the task is considered a failure.

[0072] Users select the nearest base station based on the distance between the base station and the mobile device. The formula for calculating the distance between the mobile device and the base station is as follows:

[0073]

[0074] Where two-dimensional vector Indicates user u i Current location Indicates user u k When the current location is known, the distance between users is calculated using the following formula:

[0075]

[0076] When users are too close together, interference between them must be considered. The distance between users is only measured within their mutual influence range. In other words, a user entering the network doesn't know the locations of all other users entering the network; measurements are only taken within their influence range. While a user knows the locations of all base stations, they don't calculate the distance between all devices and all base stations; the location of the user and base station is only calculated within the accessible range.

[0077] Assuming the uplink data rate c when a user's mobile device offloads tasks to the edge cloud i,j,t The calculation formula is as follows:

[0078]

[0079] Where B is the channel bandwidth, w j Let g(t) be the transmission power, g(t) be the channel gain, and σ be the channel gain. 2 Let I be the Gaussian white noise power, and let I represent the radio interference caused to the base station by user equipment performing other transmission tasks. The formula for calculating I is as follows:

[0080]

[0081] This represents the radio interference caused to the base station by user equipment (UEs) performing other transmission tasks, and δ represents the potential obstruction interference. Obstruction interference between UEs is difficult to calculate because it depends not only on the distance between users and their angle relative to the base station, but also on uncontrollable factors such as user size. Therefore, obstruction interference is randomly assigned based on the distance between users, and this interference only exists when users are sufficiently close.

[0082] To user u i Uninstall task q i,k The formula for calculating transmission delay is as follows:

[0083]

[0084] k represents the number of tasks, i represents the number of users, j represents the number of base stations, t represents time, and e represents the number of tasks. T This indicates that the data is being transmitted to the edge cloud for processing. i Uninstall task q i,k The formula for calculating the transmission energy consumption to the edge is as follows:

[0085]

[0086] Whether a task is uninstalled depends on factors related to the user device. i From this perspective, it refers to the remaining energy of the user equipment. i And computing power f i From the perspective of the base station, it's about base station resource allocation; from the perspective of the task, it's about the energy required for task processing. k .

[0087] There are two scenarios regarding where computational tasks are processed: local processing or offloading to the BS for processing.

[0088] Local computation:

[0089] When the device has sufficient remaining energy to support task processing, the task can be processed locally. The formula for calculating the local processing time of the task in this case is as follows:

[0090]

[0091] Where f i User Equipment u i The computing power. Therefore, the energy consumption of local task processing can be obtained, calculated using the following formula:

[0092]

[0093] e L ρ represents the energy consumption processed locally. k It is the power factor of the energy consumed per CPU cycle.

[0094] Uninstall to the edge:

[0095] When a user device's resources and remaining energy are insufficient, the task will be offloaded. Other factors can influence this offloading process, with the most significant impact being the selection of the base station. These influencing factors include upload speed and the base station's processing capacity. Based on the previous analysis, the task duration is calculated using the following formula:

[0096]

[0097] The proportion of resources allocated by the current serving base station to connected devices for the current user's unloaded task. This represents the computing power of the edge cloud of the j-th serving base station.

[0098] Task placement problem: Decide whether to handle mobile users locally or in the edge cloud, considering their location, and determine which edge cloud to offload the task to. An integer decision variable, x, is defined. i,k ∈{0,1}, which represents task q i,k It is local (x) i,k =0) or in the edge cloud (x i,k =1) is processed. The formula for minimizing the total task duration is as follows:

[0099]

[0100] subject to: j i ∈A(t)

[0101]

[0102]

[0103]

[0104] This formula is used to obtain the minimum total duration of the task unloading problem, which is then used to determine the optimal unloading solution. This formula is constrained by the four formulas mentioned above: j i ∈A(t) indicates that the edge server selected for the current device task unloading must be able to provide services; This represents the total energy consumption of tasks performed on a mobile device. The sum of these values ​​represents the energy consumed during transmission, and cannot exceed the total energy e of the device; δ i,j The sum of 1 indicates that the mobile device u in each time slot i Only one edge server can be connected; The sum of 1 is less than or equal to 1, which means that the total amount of computing resources allocated by the current serving base station to each user for each task cannot exceed the resource capacity of the base station.

[0105] The execution process of DQN involves first inputting data such as actions, states, and reward values ​​into the Q network, and then the network provides a predicted value Q. predict Then, based on this value, the optimal action is selected to interact with the environment to obtain the latest state s′, and the network is trained. The result of each interaction is stored in the experience pool. A target network with the exact same network structure and parameters as the Q network is copied. This target network samples data from the experience pool and outputs a stable target value Q. target =r+γQ target(s,s′,a,r), where γ represents the reward discount rate and r represents the reward. Finally, the squared error loss is calculated using the target value and the neural network output to evaluate the degree of difference between the model's predicted and actual values. The calculation formula is as follows:

[0106] L(θ)=E[(Q target -Q(s,a;θ)) 2 ]

[0107] The data structure of DQN is as follows:

[0108] ACTION: An action consists of two parts: whether to uninstall and where to uninstall. Both parts are represented by the same vector, i.e., a = {a0, a1, ..., a...}. n}, where a0 = 0 indicates that the current part of the task is being processed by the local device, a1, ..., a n This indicates that some tasks are currently being handled by the base station, a j =j indicates that the base station currently processing the task is b. j .

[0109] STATE: Status is This indicates the current location of the user equipment; the location of each user entering the network needs to be clearly defined. j This indicates the base station currently connected to the device.

[0110] REWARD: Daley indicates the task processing delay.

[0111] Therefore, when a path can be recommended to a device before it enters the network, this path is truly user-centric, taking into account the current network device information provided by the macro base station, and providing the user device with a path with minimal interference. When a device enters the network, it clearly identifies its starting and ending points. Path recommendation is performed at the macro base station, which can collect information on devices in each micro base station and then determine whether the device follows a predictable path. If the user has entered the current network multiple times, the historical trajectory can be used to predict the walking path; otherwise, the user's path can be inferred based on contextual information such as the user device's search records. Then, the current device's path is recommended using the roadmap of other devices. To improve the scalability of the path recommendation algorithm, this invention uses policy-based reinforcement learning for prediction, employing a proximal policy optimization (PPO) deep reinforcement learning algorithm. PPO uses the action probability π of the current policy. θ (a|s) and the action probability π of the previous strategy θ′The ratio of (a|s), the probability ratio r(θ) of choosing a certain Action by the old and new strategies, is calculated as follows:

[0112]

[0113] When r(θ)>1, it means that the current strategy is more inclined to choose action a.

[0114] Based on this, set the objective function L Clip (θ) limits the update amplitude when the motion amplitude is too large. The calculation formula is as follows:

[0115]

[0116] Where θ is the policy parameter, r(θ) is the ratio of the probabilities of the new and old policies, A is the dominance function, and ∈ is a hyperparameter. It represents expected experience.

[0117] To further illustrate the implementation method of the present invention, an implementation example is given below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0118] Imagine a 500m x 500m area with a dense network of 16 base stations. The base stations provide coverage randomly, but ensure that every path is covered by at least one base station. Every 100 meters, a user can change direction at a crossroads, but cannot freely change direction while walking in the middle. At this point, eight users simultaneously enter the network. They know their starting and ending points, but not their paths; therefore, interference between them may or may not occur.

[0119] Transmission power UE (w) j The channel bandwidth B is 10MHz, and the channel gain is 0.1w. The antenna gain is G = 4.11, and the carrier frequency is F. c =915MHz, path loss index is 3, noise power σ 2 10 -10 W, the local computing power is 10. 8 cycles / s, the base station's computing power is 10 10 cycles / s. The unit for task computation is cycles, and MB is the unit for task size. The current task is randomly generated according to a normal distribution, where the normal distribution of generated task sizes has a mean of 5 and a variance of 1, and the unit is MB. The computational cost of generating the task is τ. i,kThe normal distribution has a mean of 3 and a variance of 0.5, with units of 10. 8 cycles / s. Processing time σ i,k The normal distribution has a mean of 7 and a variance of 1, with units of seconds (s). The uniformly distributed task size ranges from (2, 8), with units of MB; the computational cost of the generated tasks ranges from (0.5, 5), with units of 10. 8 cycles; the task processing time limit is calculated in the range of (2, 10), and the unit is seconds (s).

[0120] The task offloading scheme based on DQN (Depth-Qi Shift Network) constructs different states based on changes in the user's position within the scene. The network is trained, and its parameters are updated via backpropagation of gradients. Once the parameters converge, we obtain an approximate method for calculating the Q-value. The DQN parameter settings are as follows:

[0121] The learning rate is the coefficient multiplied by the parameters during gradient descent optimization, and its magnitude is closely related to the model's convergence speed. The Q-network learning rate is set to 0.01; the experience pool stores information about state transitions with the environment, and when the number of samples in the pool reaches a certain threshold (51), model parameters are extracted for training (size 500); to ensure rapid convergence of the samples fed into the neural network each time, without memory limitations, the batch size is set to 32; the decay factor is set to 0.9; the Q-network has 2 layers and 10 neurons.

[0122] The convergence graph obtained by training using the above parameters is shown below. Figure 3 As shown in the figure. The convergence graph of the reward value is as follows. Figure 4 As shown in the figure, the model reaches a relatively stable state after 2000 iterations. However, the figure reveals significant noise. When we cannot determine the user's walking path, we can recommend a path for the user, forming a collaborative operation mode between the base station and the equipment.

[0123] The path scheme based on PPO (Progressive Point of Path) constructs different states based on the user's position in the scene, selects the turning point in the action selection network, and trains the network based on whether the destination has been reached and the interference encountered by other users along the path.

[0124] To better compare the performance of this method, it can also be compared with other different schemes.

[0125] 1) DQN-based offloading algorithm: random walk, offloading decision based on DQN network;

[0126] 2) Base station-equipment cooperative algorithm: The user's walking plan adopts the path recommended by the base station, and the rest is consistent with the DQN-based offloading algorithm;

[0127] 3) Random Algorithm: The user's walking plan is the same as DQN, but the connected base stations and whether to offload are randomly selected;

[0128] 4) Greedy Algorithm: The user's walking plan is the same as DQN, but the base station selection and offloading decision is based on the current base station and location information to select the optimal one.

[0129] Figure 5-6 This section compares the task processing latency under different algorithms. The task sets were randomly generated from uniform and normal distributions. The horizontal axis of both graphs represents the number of tasks (in units), and the vertical axis represents the total latency (in seconds). In the graphs, dashed lines represent the random scheme, lines composed of line segments represent the greedy scheme, lines composed of dashed lines and line segments represent the DQN offloading scheme using random paths, and solid lines represent the DQN offloading scheme using base station-recommended roadbeds, denoted as fixed-DQN. From the two comparison graphs above, it can be seen that the DQN offloading scheme using recommended roadbeds is the best in both task sets. While the DQN offloading scheme using random paths is slightly worse than the one using recommended roadbeds, the difference is not significant, which is related to the smaller experimental scenario.

[0130] Figure 7 This represents the number of task failures in the network when the task set is generated from a normal distribution. Since each task has a processing latency limit, exceeding this limit results in a failure. Failed tasks are due to unreasonable time constraints; therefore, obviously unreasonable tasks—those whose processing latency, whether local or offloaded, significantly exceeds the time limit—are removed during the experiment. Figure 7 The x-axis represents the total number of tasks, and the y-axis represents the number of failed tasks. In the graph, dashed lines represent the random algorithm, lines composed of line segments represent the greedy algorithm, lines composed of dashed lines and line segments represent the DQN offloading scheme using random paths, and solid lines represent the DQN offloading scheme using base station-recommended pathlines. The graph shows that the number of failed tasks is roughly the same for both the greedy and random algorithms. While the DQN algorithm is superior to the other two, its advantage is not significant. However, the base station-user collaborative fixed-DQN algorithm has a clear advantage. Regardless of the number of tasks, the fixed-DQN algorithm always has the fewest failed tasks, achieving the lowest task failure rate.

[0131] The experiments above show that the DQN offloading scheme using the base station-recommended path performs best in terms of task failure rate and latency. While the DQN offloading scheme using random paths performs slightly worse in latency than the one using the base station-recommended path, the difference is not significant. Therefore, even without using the base station-recommended path, the device's offloading scheme still performs well.

[0132] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for multi-user task offloading based on deep reinforcement learning, characterized in that, The method comprises the following steps: (1) establishing a network architecture, which is divided into three layers, one layer is a cellular access layer, one layer is an edge computing layer, and one layer is a macro base station layer, and the path recommendation of the user is completed by the macro base station layer; (2) when the user equipment enters the network, the user equipment determines the starting point and the ending point, and the macro base station provides path recommendation for the user, and the macro base station uses existing environmental information to realize path recommendation, specifically: the macro base station collects the information of the user equipment in each micro base station, and then judges whether the user equipment is a predictable path; if the user enters the current network multiple times, the walking path can be predicted by using the historical trajectory, and if the user does not enter the network multiple times, the user path can be inferred according to the search record scene information of the user equipment; when the user prohibits the base station to collect information, the macro base station collects the information of the user equipment in each micro base station to recommend the current user equipment path; When the user equipment takes the predicted path, the task offloading scheme is a base station and user equipment cooperative offloading scheme, for the user offloading task, when multiple users enter the network, the information obtained by the user is not only the base station environmental information, but also the interference of other users; the user equipment can actively obtain the base station information that can provide services for the user equipment, and the remaining environmental information is mainly obtained from the service base station and the macro base station; the macro base station informs the user equipment of the other user equipment and the predicted walking path of the user equipment in the current network according to the existing information, the user equipment trusts the predicted path of the macro base station, the user equipment selects a certain micro base station as a service base station in the micro base station that can provide services, uses the information provided by the service base station to determine the interference factor and uses it as the basis for judging whether to offload, if the interference is too large to cause the local processing time to be less than the transmission delay, the task is not offloaded to the service base station for processing, otherwise the task can be offloaded to the service base station for processing, and DQN is used to determine the service base station and make offloading decisions in the offloading process; The data structure of DQN is as follows: ACTION: The action consists of two parts, i.e., whether to offload and the selection of where to offload; the actions of the two parts are represented by a same vector, i.e., a = {a0, a1, …, an}, where a0= 0 represents that the current partial task is processed by the local user equipment, a1, …, anrepresent that the current partial task is processed by the service base station, and an= j represents that the service base station currently processing the task is b n . n} where a0= 0 represents that the current partial task is processed by the local user equipment, a1, …, an n represent that the current partial task is processed by the service base station, and an= j represents that the service base station currently processing the task is b j , j represents the number of base stations. j ​ STATE: State is Indicates the current location of the user equipment, the location of each user entering the network needs to be specified; b j Indicates the base station currently connected by the user equipment; REWARD: where daley represents the task processing delay.

2. The method of claim 1, wherein, Assume that there are M users entering the network, u i represents the i-th user entering the network, wherein i = {1, 2, …, M}, i represents the number of users; The edge access layer has n micro base stations, denoted by B = {b1, b2, ..., b}. n } indicates that time is divided into multiple time slots, and the user's connection information is determined in each time slot; δ i，j ={0, 1} represents user u i Is it related to base station b? j Connected, where j = {0, 1, 2, ..., n}; ∑ j δ i，j =1 means that a mobile device is only allowed to connect to one base station within a time slot; Q is used to describe the task, user u i The task set of user u is represented as Q i = {q i，1 , q i，2 ,..., q i,K}, let q i,k = {τ i,k , ω i，k , σ i，k}, k = {1, 2,..., K}, K is the total number of tasks, wherein τ i，k represents the computing amount of the kth task q i of a certain user equipment u i，k , ω i，k represents the size of the computing task q i，k , σ i，k represents the maximum time limit for the user equipment u i to complete the task q i，k , and if the time limit is exceeded, the task is determined to fail.

3. The method of claim 2, wherein, An uplink data rate c when a user mobile device offloads a task onto a service base station of an edge cloud is assumed ij，t The calculation formula is as follows: where B is the channel bandwidth, w j is the transmission power, g(t) is the mobile device channel gain of the user, σ 2 is the Gaussian white noise power, and I represents the wireless interference caused by the user equipment of other transmission tasks to the base station; the calculation formula of I is as follows: g'(t) is the channel gain of the other user equipment, and δ represents the blocking interference that can exist; User u i Offload task q i The transmission delay of k is calculated as follows: k represents the kth task, t represents time, e T represents the service base station processing transmitted to the edge cloud, δ i，j =1 in the formula, indicating that the user has connected to the service base station, and the transmission delay is calculated; u i offload task q i The transmission energy consumption calculation formula of the kth task to the edge is as follows:

4. The method of claim 3, wherein, The influencing factors of whether a task is offloaded are the remaining battery energy e i of the user equipment from the user equipment's perspective i and the computing power f i the energy e k needed for the task processing from the base station's perspective There are two cases for calculating where to process the task, local processing or offloading to the service base station for processing; Local calculation: When the remaining battery energy of the user equipment is sufficient to support the processing of the task, the task is processed locally; at this time, the local processing time of the task is calculated according to the following formula: where f i is the computing power of the user device u i ; thus obtaining the energy consumption of the local task processing, the calculation formula is as follows: denotes the energy consumption of the kth task processed locally, p k is the power coefficient of the energy consumed per CPU cycle; Offload to the service base station: When the resources and the remaining battery energy of the user equipment are insufficient, the task is selected to be offloaded, the influencing factors are the upload rate and the processing capacity of the base station, and the task duration is calculated according to the following formula: wherein, a proportion of resources allocated by the current serving base station to the connected user equipment for offloading tasks for the current user, denotes the computing capacity of the edge cloud of the jth serving base station.

5. The method of claim 4, wherein, Task placement problem: decide local user equipment or service base station in edge cloud processing, considering user mobile location, decide which edge cloud service base station to unload the task; integer decision variable x i,k ∈{0,1}, it represents task q i,k is processed in local (x u,k =0) or in edge cloud (x u,k =1); the minimum total task duration calculation formula is as follows: subject to: j i ∈A(t) The minimum total task duration calculation formula is used to obtain the minimum value of the total duration of the task offloading problem, so as to determine the optimal offloading scheme; the minimum total task duration calculation formula is constrained by the above four formulas: j i A(t) represents that the edge server selected by the current user equipment task offloading must be serviceable, and A(t) represents the base station combination that can be offloaded; represents the total energy consumption of the task executed on the mobile device, represents the transmission energy consumption, and the sum of the total energy consumption of the task executed on the mobile device and the transmission energy consumption cannot exceed the total energy e of the user equipment; δ i,j The sum of 1 represents that the user equipment u i can only be connected to one edge server; The sum of less than or equal to 1 represents that the total amount of computing resources allocated by the current service base station to each task of each user cannot exceed the resource amount of the base station.

6. The method of claim 1, wherein, The execution process of the DQN is to input the action, state and reward value data into the Q network, to obtain a predicted value Q from the network predict ; then select the optimal action according to the predicted value to interact with the environment to obtain the latest state s' and perform network training; the result obtained each time is stored in the experience pool; Copy a target network with the same network structure and parameters as the Q network, which samples data from the experience pool and outputs stable target value Q target = r + γQ target (s, s ′ , a, r), where γ represents the reward value discount rate, and r represents the reward; finally, the square difference loss is calculated using the target value and the output of the neural network to evaluate the degree of difference between the predicted value and the true value of the model, and the calculation formula is as follows: L(0) = E[(Q target - Q(s, a; 0)) 2 ] Where θ is the policy parameter, a is the action, s is the state, and s' is the latest state.

7. The method of claim 6, wherein, In order to make the path recommendation algorithm have better expansibility, prediction is carried out by using policy-based reinforcement learning, and a proximal policy optimization deep reinforcement learning algorithm (PPO) is adopted; the PPO is based on the action probability π θ (a|s) of the previous policy θπ (a|s) of the previous policy, and the probability ratio r(θ) of the new and old policies selecting a certain action is calculated according to the following formula: When r(θ)>1, it means that the current policy is more inclined to select action a. On this basis, the objective function L is set Clip (θ), limiting the update amplitude, the calculation formula is as follows: where r(θ) is the ratio of the probabilities of the new and old policies, A is the advantage function, and ∈ is a hyperparameter, denotes the empirical expectation.

Images