A mobile task offloading method for a single mobile user based on reinforcement learning

Abstract

The application provides a mobile task offloading method for a single mobile user based on reinforcement learning. The method mainly comprises two parts: firstly, the sensitivity of a large task to mobility is considered, the task is divided according to a certain threshold and the characteristics of the task from the perspective of "atom"; secondly, the existing learning condition and the mobility of the device are considered, a double-table Q-learning algorithm is used to select the connected base station and whether to offload, and the algorithm is used to optimize the delay and the comprehensive performance of the delay and energy consumption. The edge computing and the reinforcement learning technology are used, the threshold is used to judge whether the task needs to be divided, when the user equipment is moving, the user selects the connected base station and decides the offloaded task quantity, so that the total delay is reduced, and the comprehensive performance of the delay and the energy consumption is improved.

Description

Technical Field

[0001] This application relates to the field of edge computing, specifically to a method for unloading mobile tasks for a single mobile user based on reinforcement learning. Background Technology

[0002] Mobile Edge Computing (MEC) is a network physically located close to the data source. It's an open platform integrating core capabilities such as networking, computing, storage, and applications, providing edge intelligence services locally. Edge computing processes data closer to the user. It allows software applications to leverage local content and real-time information about local network access conditions. By deploying various services and cached content at the network edge, the mobile core network can further alleviate congestion and effectively serve local devices. The location where edge computing occurs is called an edge node, which can be any node with computing and network resources between the data source and the cloud center. These edge device nodes provide rich service interfaces, working with the cloud computing center to provide collaborative computing services to users. This is because cloud computing's main characteristic is its ability to handle large amounts of data and perform in-depth analysis, rather than real-time data processing. Edge computing, on the other hand, focuses on local needs and can play a better role in small-scale, real-time intelligent analysis, such as meeting the real-time needs of local businesses. Therefore, compared to cloud computing networks, mobile edge networks have advantages such as lower latency, smaller bandwidth, higher power efficiency, closer service, and the ability to fully utilize environmental information.

[0003] In real-world scenarios, many factors can affect task offloading performance, including but not limited to mobility management, interference management, and resource allocation. Therefore, these related factors are discussed below.

[0004] Mobility management (MM) involves deciding whether to change the serving base station (handover) and whether to change the location of virtual machines (VM migration). Therefore, a key issue in mobility management is how to ensure service continuity.

[0005] The first method uses historical trajectory data to predict movement paths. The second method uses user preferences, goals, and analyzed spatial information to predict trajectories and destinations without any analysis of the user's historical movement trajectories.

[0006] Interference management refers to the potential for severe mutual interference when multiple user devices simultaneously select the same wireless channel to offload tasks, leading to reduced offloading efficiency and consequently higher latency and energy consumption. In networks with multiple MEC servers, the interplay between user transmission mode selection and offloading target server determination further complicates MEC scheduling.

[0007] The purpose of resource allocation is to allocate the limited resources within the edge node according to the needs of the devices as much as possible, thereby reducing latency and improving the user experience.

[0008] Machine learning is generally categorized into three types of algorithms: Supervised Learning (SL), Unsupervised Learning (UL), and Reinforcement Learning (RL). Supervised learning involves learning from a labeled training set provided by an experienced external supervisor, allowing the system to infer or generalize its responses so that it can function correctly even when the training set is unavailable. Unsupervised learning seeks out hidden structures within unlabeled data sets. Reinforcement learning differs from both supervised and unsupervised learning; the goal of a reinforcement learning agent is to learn the optimal way to perform a task through repeated interactions with its environment.

[0009] Q-Learning is the most commonly used value-based reinforcement learning algorithm. It can learn to estimate the utility of individual state-action pairs. If a sufficient number of samples are obtained for each state-action pair, Q-Learning will learn the optimal state-action value; the optimal policy is to choose the action with the largest Q-value. The Q-Learning agent updates its Q-value according to the following update rule.

[0010]

[0011] α∈[0,1] is the learning rate that controls the updating of Q-values ​​at each time step, and γ∈[0,1] is the decay factor, indicating the correlation between the subsequent reward value and the current state and action. The main idea of ​​the Q-Learning algorithm is to construct a Q-Table to store Q-values ​​based on the state and actions, and then select the action that can obtain the maximum reward based on the Q-value. The learning process is to update this Q-Table. Summary of the Invention

[0012] The purpose of this invention is to address the shortcomings of existing technologies by providing a single-user mobile task offloading method based on reinforcement learning. This method considers the mobility sensitivity of large tasks, dividing tasks from an "atomic" perspective using certain thresholds and task characteristics. Then, from the device perspective, based on device mobility, a single-user task offloading method is proposed based on the dual-table Q-Learning algorithm, which reduces latency and energy consumption, and achieves better and more stable performance.

[0013] The technical solution adopted by this invention to solve its technical problem is as follows: a mobile task offloading method for a single mobile user based on reinforcement learning, the method comprising two steps: task partitioning and task offloading decision.

[0014] (1) Task division: When a user device enters the network, the device needs to collect information and determine the service base stations that can be connected. When the device generates a task, it first determines the task type based on the task weight threshold D, and then determines whether further division is needed based on the task type. Based on the above, the task can be divided into m independent atomic tasks and n interdependent atomic tasks. Interdependent atomic tasks are regarded as independent tasks, but the tasks must be executed in order. Independent atomic tasks are also regarded as independent tasks, and there are no constraints between the tasks. The execution order of independent atomic tasks can be changed at will.

[0015] (2) Task offloading decision: During actual movement, the user selects a serving base station to connect to based on the environment and current status of the device. If the user's mobile device has insufficient battery power or computing power, and the time for processing the task locally on the device is greater than the transmission latency, then the task is considered to be offloaded to a serving base station for processing; otherwise, it is processed locally on the device. After the user connects to a base station, the base station will only be switched after other base stations have better performance or the atomic-sized task has been processed. Since the decision in this problem involves two actions, one is base station selection and the other is whether to offload, the task offloading decision uses dual-table Q-Learning to solve the task offloading decision problem. The dual-table Q-Learning algorithm trains two Q-Tables simultaneously. The states and reward values ​​of these two Q-Tables are the same, but the action vectors are different. The two actions are base station selection and whether to offload. The action is selected from the state according to the Q value in the Q-Table, the action is executed, the immediate reward and the new state are obtained, and then the Q-Table is updated. The two tables are trained simultaneously to give the optimal solution, that is, whether to offload, and if so, which base station to offload to.

[0016] Furthermore, assume there are n serving base stations in the network, and each serving base station is equipped with an edge cloud; the edge cloud is represented as B = {b1, b2, ..., b}. n}, b n This represents the edge cloud of the nth serving base station; a user can only connect to one serving base station within a certain period of time. Therefore, the time is divided into multiple time slots, and the serving base station the user connects to is determined within each time slot: δ j ={0, 1} indicates whether user δ is associated with base station b j Connected, where j = {0, 1, 2, ..., n}; ∑δ j =1 indicates that the mobile device is only allowed to connect to one base station within this time slot;

[0017] Q is used to describe the movement task, Q = {q1, q2, ..., q} k , ..., q K}, let q k ={τ k ω k}, k = {1, 2, ..., K}, where K is the total number of moving tasks, q k Let τ represent the k-th movement task. k Represents a task q of a certain user device k The computational load, i.e., the CPU cycles required to complete the task, ω k Represents the computation task q k The size of the task is the amount of data delivered to the edge cloud. When a device generates a mobile task, the size of the mobile task cannot be determined, so the mobile task may need to be split. At this time, the task is divided into two types according to whether it exceeds the task weight threshold D: 1) light task, which is a task that does not need to be split; 2) heavy task, which is a task that needs to be split. The type of task is represented by tp, where tp=0 indicates that the task is a light task and tp=1 indicates that the task is a heavy task.

[0018] Since the task is divisible, it can be divided into atomic parts based on its characteristics. The atomic parts of the task cannot be further divided. In this case, the task can be divided into m independent atomic parts and n interdependent atomic parts. The lightweight task is an atomic task. The interdependent atomic parts need to be executed in sequence, while the independent atomic parts can make independent decisions and processes. The unloading ratio of task qk is the ratio of the number of tasks that need to be unloaded in the k-th task to the number of atomic tasks after splitting, where i = {0, 1, ...}. The sum of the number of task segments should be the actual total number of tasks.

[0019] Furthermore, the user selects the closer base station based on the distance between the serving base station and the device. The formula for calculating the distance d(t) between the device and the serving base station is as follows:

[0020]

[0021] In this case, the user is in a mobile state. The user's location is represented by a two-dimensional vector. The initial position is 0, and the j-th base station b j The location is determined as

[0022] Furthermore, the uplink data rate c when the user's mobile device offloads tasks to the edge cloud j，tThe calculation formula is as follows:

[0023]

[0024] Where B is the channel bandwidth, w j Let g(t) be the transmission power, g(t) be the channel gain, and σ be the channel gain. 2 The power is Gaussian white noise; the channel gain g(t) is affected by distance, and the distance changes over time.

[0025] It is possible to calculate the user uninstallation task q. k The transmission delay to the edge cloud is φ T The formula for calculating (k, i, j, t) is as follows:

[0026]

[0027] k represents the k-th task, i represents the number of tasks after the segmentation, j represents the number of base stations, t represents time, and e represents the time interval. T This indicates transmission to edge cloud processing, where δ in the formula... j =1 indicates that the user has connected to the serving base station, and the transmission delay is calculated;

[0028] The total transmission delay of a certain task is expressed by the formula φ. Tsum The calculation formula is as follows:

[0029]

[0030] Furthermore, the energy consumption of user-unloaded task qk transmitted to the edge cloud was obtained. The calculation formula is as follows:

[0031] .

[0032] Furthermore, the factors influencing whether a task should be uninstalled are as follows:

[0033] 1) Total energy e and computing power f of user equipment;

[0034] 2) Energy required for task processing e k ;

[0035] There are two scenarios regarding where computing tasks are processed: processing them locally on the device or offloading them to the serving base station for processing.

[0036] 1) When the device has sufficient remaining energy to support task processing, the task will be processed locally on the device, as shown in the following formula:

[0037] ∑e k <e

[0038] The formula for calculating the local processing time of a task is as follows:

[0039]

[0040] This indicates that what is processed locally could be a part of the task or the entire task; if the entire task is processed locally, then...

[0041] Energy consumption of processing the kth task locally on the device The calculation formula is as follows:

[0042]

[0043] e L ρ represents the energy consumption processed locally. k The power factor represents the energy consumed per CPU cycle;

[0044] 2) When the equipment's resources and remaining energy are insufficient to support task processing, the task needs to be offloaded to the edge cloud of the serving base station for processing; the computing power (CPU cycles per second) of the edge cloud should be utilized. E It is expressed as follows:

[0045]

[0046] This represents the computing power of the edge cloud at the nth serving base station;

[0047] The total task duration in the edge cloud mainly includes the time consumed by two processes, namely:

[0048] (1) The time consumed by the transfer when the user unloads the task;

[0049] (2) The time consumed when processing computing tasks on the edge cloud;

[0050] Because users may move, when a user leaves a serving base station to switch to another base station, the device may still need to seek service from that base station. In this case, the base station handover will involve a certain handover delay, which is defined as φ. S After the device switches base stations, the results of the user tasks processed by the previous base station also need to be transmitted back through the current serving base station. However, since the results are often small, this transmission delay is negligible. The formula for calculating the total task duration is as follows:

[0051] .

[0052] Furthermore, considering optimizing the total latency of the task unloading problem, we define an integer decision variable x. k x ∈{0,1} k Indicates task qk It is local (x) k =0) or in the edge cloud (x k =1) is handled in the following way; the formula for calculating the minimum total task duration to address the task unloading problem is as follows:

[0053]

[0054]

[0055]

[0056] ∑δj=1

[0057]

[0058] The minimum total duration of the task unloading problem is obtained using the above formula, which is then used to determine the optimal unloading solution. This indicates the placement location of all parts of the task, i.e., task q. k The edge cloud of the selected offloaded service base station is able to connect to the user during the time period; This represents the total energy consumption of tasks performed on a mobile device. The sum of these values ​​represents the energy consumed during transmission, and cannot exceed the total energy e of the device; δ j A sum of 1 indicates that a mobile device can only connect to one edge server in each time slot; The sum of 1 indicates that the total number of task cuts should be the actual total number of tasks.

[0059] From the user equipment's perspective, performance is also affected by the device's energy consumption. The formula for calculating the combined local latency and energy consumption performance is as follows:

[0060]

[0061] The formula for calculating the overall performance of unloading latency and energy consumption is as follows:

[0062]

[0063] This indicates the total remaining energy of the current user on the mobile device. Let represent the total remaining energy of the current user offloaded to the serving base station. The optimization problem described above is now updated as follows: the formula for maximizing the overall performance of total latency and energy consumption is as follows:

[0064]

[0065]

[0066]

[0067] ∑δj=1

[0068]

[0069] The above formula can obtain the maximum value of the combined performance of latency and energy consumption, and thus determine the optimal offloading scheme.

[0070] Furthermore, Q-Learning consists of a state (s), an action (a), and a reward value (r). In Q-Learning, the update rule formula for Q-value is as follows:

[0071]

[0072] α∈[0,1] is the learning rate that controls the update of Q value at each time step, and γ∈[0,1] is the decay factor, which indicates the correlation between the subsequent reward value and the current state and action;

[0073] The data structure of each parameter in the dual-table Q-Learning is as follows:

[0074] State: State consists of two parts in Indicates the location of the device, b j Indicates the user's current connected base station;

[0075] Reward: Where daley represents the total duration of tasks currently being processed; the shorter the total task duration, the greater the reward value.

[0076] The decision-making process in this problem involves two actions: base station selection and whether to offload the service; therefore, the actions of the two tables are as follows:

[0077] Action1: a1 = b j，j ∈{b1, b2, ..., b n} indicates that the base station currently processing the task is b. j ;

[0078] Action2: a2 = k, k ∈ {0, 1, 2, 3}, representing the unloading decision for the remaining subtasks in the current task; a2 = 0 indicates that both the remaining interdependent and independent subtasks make decisions simultaneously, and both are processed locally; a2 = 1 indicates that both the remaining interdependent and independent subtasks make decisions simultaneously, with interdependent subtasks processed locally and independent subtasks unloaded to the edge; a2 = 2 indicates that both the remaining interdependent and independent subtasks make decisions simultaneously, with independent subtasks processed locally and interdependent subtasks unloaded to the edge; a2 = 3 indicates that both the remaining interdependent and independent subtasks make decisions simultaneously, and both are unloaded.

[0079] The above solution provides a reward value based on minimizing latency. However, when the problem transforms into optimizing the overall latency-energy performance, only the reward value needs to be changed. The data structure for the reward value is as follows:

[0080] Reward: Where 'enger' represents the percentage of current remaining energy to total energy; according to the formula, the more remaining energy, the smaller the total latency, and the greater the reward value.

[0081] The beneficial effects of this invention are as follows: This invention proposes a mobile task offloading method for a single mobile user based on reinforcement learning. First, it determines whether tasks need to be partitioned based on a threshold. When the user equipment is moving, it guides the user on how to select the connecting base station and determine the amount of tasks to be offloaded, thereby reducing the total latency. Furthermore, the dual-table Q-Learning scheme improves the stability of services for mobile-sensitive users. Attached Figure Description

[0082] Figure 1 This is a network diagram of the present invention.

[0083] Figure 2 This is a flowchart of an uninstallation method.

[0084] Figure 3 A graph showing the task unloading latency (uniform distribution) under different strategies.

[0085] Figure 4 Plots showing the normal distribution of task unloading latency under different strategies.

[0086] Figure 5 Plots showing task unloading latency (Gamma distribution) under different strategies.

[0087] Figure 6 A graph showing the ratio of task unloading energy to latency under different strategies (uniform distribution).

[0088] Figure 7 Plot of the task unloading energy-latency ratio (normal distribution) under different strategies.

[0089] Figure 8 Plots showing the ratio of task unloading energy to latency (Gamma distribution) under different strategies. Detailed Implementation

[0090] To make the mobile task unloading method of this application clearer, the technical solution of this application will be clearly and completely described below with reference to the accompanying drawings and specific embodiments:

[0091] like Figure 1 and Figure 2As shown, this invention provides a method for mobile task offloading for a single mobile user based on reinforcement learning, the method comprising the following steps:

[0092] (1) Task Division: After a user device enters the network, it needs to collect information to determine the base stations it can connect to. When a task is generated, the task type is first determined based on a threshold, and then it is determined whether further division is needed. Tasks are divided into light tasks and heavy tasks. Distinguishing between light and heavy tasks is the basis for whether a task needs to be divided. For example, connecting to a smart car for autonomous driving is a heavy task, while facial recognition on a mobile phone is a light task. At this time, a task weight threshold D is given, which serves as a basis for determining the task type. This threshold is given according to the environment; for example, the threshold for facial recognition on a mobile phone is... Let's set the size to 12MB. Based on this, the task can be divided into m independent atomic parts and n interdependent atomic parts. Interdependent atomic tasks can be considered independent tasks, but they must be executed in a specific order. Independent atomic tasks are also considered independent tasks, and there are no constraints between them. In this case, the execution order of independent subtasks can be freely changed.

[0093] (2) Select the connecting base station and whether to unload tasks based on the device's current learning status. Tasks requiring unloading are unloaded to the serving base station for processing; those not requiring unloading are processed locally. During actual movement, the user selects the connecting serving base station based on the device's environment and current status. If the user's mobile device's battery power or computing power is insufficient, and the task's local processing time exceeds the transmission latency, then consider unloading the task to the serving base station for processing; otherwise, process it locally. After connecting to a base station, the user will only switch base stations after other base stations have better performance or after a task of atomic size has been processed. Since the decision in this problem involves two actions—base station selection and unloading—the decision uses dual-table Q-Learning to solve the task unloading decision problem. The dual-table Q-Learning algorithm trains two Q-Tables simultaneously. These two Q-Tables have the same state and reward values ​​but different action vectors. An action is selected from the state based on the Q-value in the Q-Table, the action is executed, an immediate reward and a new state are obtained, and then the Q-Table is updated. The two tables are trained simultaneously and closely integrated to provide the optimal solution: whether to unload, and if so, which base station to unload to.

[0094] Assume there are n serving base stations in the network, and each serving base station is equipped with an edge cloud; the edge cloud is represented as B = {b1, b2, ..., b}. n}, b nThis represents the edge cloud of the nth serving base station; a user can only connect to one serving base station within a certain period of time. Therefore, the time is divided into multiple time slots, and the serving base station the user connects to is determined within each time slot: δ j ={0, 1} indicates whether user δ is associated with base station b j Connected, where j = {0, 1, 2, ..., n}; ∑δ j =1 indicates that the mobile device is only allowed to connect to one base station within this time slot;

[0095] Q is used to describe the movement task, Q = {q1, q2, ..., q} k , ..., q K}, let q k ={τ k ω k}, k = {1, 2, ..., K}, where K is the total number of moving tasks, q k Let τ represent the k-th movement task. k Represents a task q of a certain user device k The computational load, i.e., the CPU cycles required to complete the task, ω k Represents the computation task q k The size of the task is the amount of data delivered to the edge cloud. When a device generates a mobile task, the size of the mobile task cannot be determined, so the mobile task may need to be split. At this time, the task is divided into two types according to whether it exceeds the task weight threshold D: 1) light task, which is a task that does not need to be split; 2) heavy task, which is a task that needs to be split. The type of task is represented by tp, where tp=0 indicates that the task is a light task and tp=1 indicates that the task is a heavy task.

[0096] Since the task is divisible, it can be divided into atomic parts based on its characteristics. The atomic parts of the task cannot be further divided. In this case, the task can be divided into m independent atomic parts and n interdependent atomic parts. The lightweight task is an atomic task. The interdependent atomic parts need to be executed in sequence, while the independent atomic parts can make independent decisions and processes. The unloading ratio of task qk is the ratio of the number of tasks that need to be unloaded in the k-th task to the number of atomic tasks after splitting, where i = {0, 1, ...}. The sum of the number of task segments should be the actual total number of tasks.

[0097] Furthermore, users select the nearest base station based on the distance between the serving base station and the device. The formula for calculating the distance d(t) between the mobile device and the base station is as follows:

[0098]

[0099] In this case, the user is in a mobile state. The user's location is represented by a two-dimensional vector. The initial position is 0, and each base station b j The location is determined as

[0100] Uplink data rate c when user mobile devices offload tasks to the edge cloud j，t The calculation formula is as follows:

[0101]

[0102] Where B is the channel bandwidth, w j Let g(t) be the transmission power, g(t) be the channel gain, and σ be the channel gain. 2 Let g(t) be the Gaussian white noise power. The channel gain g(t) is affected by distance, which changes over time.

[0103] You can get the user's uninstallation task q k Transmission delay φ to edge cloud T The formula for calculating (k, i, j, t) is as follows:

[0104]

[0105] k represents the k-th task, i represents the number of tasks after the segmentation, j represents the number of base stations, t represents time, and e represents the time interval. T This indicates transmission to edge cloud processing, where δ in the formula... j =1 indicates that the user has connected to the serving base station, and the transmission delay is calculated.

[0106] The total transmission delay of a certain task can be expressed as φ Tsum The calculation formula is as follows:

[0107]

[0108] It can be further obtained that the user uninstalled task q k Energy consumption for transmission to edge cloud The calculation formula is as follows:

[0109]

[0110] Factors influencing whether a task should be uninstalled:

[0111] 1) Total energy e and computing power f of user equipment;

[0112] 2) Energy required for task processing in the user's system (e) k ;

[0113] There are two scenarios regarding where computational tasks are processed: local processing or offloading to the base station for processing.

[0114] 1) When the device has sufficient remaining energy to support task processing, the task can be processed locally, as shown in the following formula:

[0115] ∑e k <e

[0116] The formula for calculating the local processing time of a task is as follows:

[0117]

[0118] This indicates that what is being processed locally could be a part of the task or the entire task. If the entire task is being processed locally, then...

[0119] Energy consumption of processing the kth task locally on the device The calculation formula is as follows:

[0120]

[0121] e L This represents the energy consumption during local processing, where ρk represents the power coefficient of the energy consumed per CPU cycle.

[0122] 2) When the resources and remaining energy of the device are insufficient to support task processing, the task needs to be offloaded to the edge for processing. The computing power of the edge cloud (CPU cycles per second) is expressed as follows:

[0123]

[0124] This represents the computing power of the edge cloud at the nth serving base station.

[0125] The total task duration in the edge cloud mainly includes the time consumed by two processes, namely:

[0126] (1) The time consumed by the user to unload the task.

[0127] (2) The time consumed when processing computing tasks on the edge cloud.

[0128] Because users may move, when a user leaves a serving base station to switch to another base station, the device may still need to seek service from that base station. In this case, the base station handover will involve a certain handover delay, which is defined as φ. SAfter a device switches base stations, the results of the user tasks processed by the previous base station also need to be transmitted back through the current serving base station. However, since the results are often small, this transmission delay is negligible. The formula for calculating the task duration is as follows:

[0129]

[0130] To optimize the total latency of task unloading, define an integer decision variable x. k x ∈{0,1} k Indicates task q k It is local (x) k =0) or in the edge cloud (x k =1) is handled in the following way. The formula for calculating the minimum total task duration to solve the task unloading problem is as follows:

[0131]

[0132]

[0133]

[0134] ∑δ j =1

[0135]

[0136] The minimum total duration of the task unloading problem is obtained using the above formula, which is then used to determine the optimal write solution. This formula is constrained by the four formulas mentioned above: This indicates the placement location of all parts of the task, i.e., task q. k The selected edge server to be offloaded must be an edge server that can provide services, that is, it can connect to the user's base station during the time period. This represents the total energy consumption of tasks performed on a mobile device. The sum of these values ​​represents the energy consumed during transmission, and cannot exceed the total energy e of the device; δ j A sum of 1 indicates that a mobile device can only connect to one edge server in each time slot; The sum of 1 indicates that the total number of task cuts should be the actual total number of tasks.

[0137] From the user's equipment perspective, energy consumption also affects performance. The formula for calculating the combined local latency and energy consumption performance is as follows:

[0138]

[0139] The formula for calculating the overall performance of unloading latency and energy consumption is as follows:

[0140]

[0141] in This indicates the total remaining energy of the current user on the mobile device. Let represent the total remaining energy of the current user unloaded to the serving base station. The optimization problem described above can then be updated as follows: the formula for maximizing the overall performance of total latency and energy consumption is as follows:

[0142]

[0143]

[0144]

[0145] ∑δ j =1

[0146]

[0147] The above formula can yield the maximum value of the sum of latency and energy consumption performance, which can be used to determine the optimal writing scheme.

[0148] In Q-Learning, a learner interacts with the environment and chooses an action to move to the next state based on a reward. Q-Learning consists of a state (s), an action (a), and a reward value (r). The update formula for the Q-value in Q-Learning is as follows:

[0149]

[0150] α∈[0,1] is the learning rate that controls the update of Q value at each time step, and γ∈[0,1] is the decay factor, which indicates the correlation between the subsequent reward value and the current state and action.

[0151] The data structure of each parameter in the dual-table Q-Learning is as follows:

[0152] State: State consists of two parts in Indicates the location of the device, b j This indicates the user's current connected base station.

[0153] Reward: Here, daley represents the total latency of the currently processed subtasks. The smaller the total latency, the larger the reward value.

[0154] The decision-making process in this problem involves two actions: base station selection and whether to offload the base station. Therefore, the actions of the two tables are as follows:

[0155] Action1: a1 = b j，j ∈{b1, b2, ..., bn} indicates that the base station currently processing the task is b. j .

[0156] Action2: a2 = k, k ∈ {0, 1, 2, 3}, represents the unloading decision for the remaining subtasks in the current task. a2 = 0 indicates that both the remaining interdependent and independent subtasks make decisions simultaneously, and both are processed locally; a2 = 1 indicates that both interdependent and independent subtasks make decisions simultaneously, with interdependent subtasks processed locally and independent subtasks unloaded to the edge; a2 = 2 indicates that both interdependent and independent subtasks make decisions simultaneously, with independent subtasks processed locally and interdependent subtasks unloaded to the edge; a2 = 3 indicates that both interdependent and independent subtasks make decisions simultaneously, and both are unloaded.

[0157] The above solution provides a reward value based on minimizing latency. However, when the problem transforms into optimizing the overall latency-energy performance, only the reward value needs to be changed. The data structure for the reward value is as follows:

[0158] Reward: Where 'enger' represents the percentage of current remaining energy to total energy. The formula shows that the more remaining energy, the lower the total latency, and the greater the reward value.

[0159] The dual-table Q-Learning scheme that optimizes latency is denoted as TQ-Learning, and the dual-table Q-Learning scheme that optimizes overall performance is denoted as ET-Q-Learning.

[0160] To further illustrate the implementation method of the present invention, an implementation example is given below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0161] Taking facial recognition applications as an example, assuming the user device's transmission power is 0.1W, the transmission channel bandwidth is 10MHz, and the channel gain in the free-space path loss model is... Where G = 4.11 is the antenna gain, F c =915MHz is the carrier frequency, and 3 is the path loss exponent. The noise power is σ. 2 =10 -10 W. The local device's computing power is 10. 8 cycles / s. The base station's computing power is 10 cycles / s. 10cycles / s. The unit for task computation is cycles, and MB is the unit for task size. The current task is randomly generated according to a normal distribution, where the normal distribution for generating task size has a mean of 30 and a variance of 8, and is expressed in MB; the normal distribution for generating task computation has a mean of 5 and a variance of 1, and is expressed in 10^6 MB. 8 cycles / s. Threshold Set it to 12MB.

[0162] Based on a dual-table Q-Learning task offloading decision scheme, when a single user enters the network, the system analyzes the user's location and the current serving base station to determine whether to offload and the offloading location. A reward value obtained from the Q-Table is then established, and this reward value is used to determine whether the trained model has reached a stable state; if so, the model has converged.

[0163] To better compare the performance of this method, it can also be compared with two other different schemes. Furthermore, due to the excessive uncertainty caused by the random path approach, the user's walking path selection is a deterministic path when comparing with other schemes.

[0164] (1) Stochastic model: The connection to the base station and whether to offload are randomly selected;

[0165] (2) Greedy Model: The base station selection is based on the distance between the user equipment and the base station when a switchable base station is available, selecting the base station closest to the current equipment. The decision on whether to offload a subtask is based on making the most of available resources, and local processing and offloading processing must be performed simultaneously.

[0166] Furthermore, for a more comprehensive comparison, in addition to being randomly generated from a normal distribution, the task set also includes two additional distributions: a uniform distribution and a Gamma distribution. The uniform distribution generates tasks ranging in size from (0, 100) MB, and tasks with computational complexity ranging from (0.5, 10) kilobytes per second. 8 Cycles. The shape parameter of the Gamma distribution for generating the task size is 1, the scale parameter is 8, and the unit is MB; the shape parameter of the computational cost for generating the task is 1, the scale parameter is 1, and the unit is 10. 8 cycles.

[0167] like Figure 3 , Figure 4 , Figure 5As shown in the figures, the comparison of three offloading schemes is presented under the condition of randomly generated tasks. The horizontal axis of all three graphs represents the number of tasks, i.e., the number of tasks processed by the device from the starting point to the ending point, while the vertical axis represents the cumulative latency of task processing, in seconds. In the graphs, the solid line represents the offloading scheme based on dual-table Q-Learning, which focuses on overall performance, denoted as ET-Q; the line composed of line segments and dots represents the offloading scheme based on dual-table Q-Learning, which focuses on latency, denoted as TQ; the line composed of line segments represents the greedy offloading scheme; and the line composed of dots represents the random offloading scheme. From the graphs, we can see that under normal and uniform distributions, the dual-table Q-Learning scheme significantly outperforms the other two schemes. Furthermore, as the number of tasks increases, the performance of the dual-table Q-Learning scheme becomes more prominent, with the latency-optimized dual-table Q-Learning offloading scheme performing optimally. However, under a Gamma distribution, the dual-table Q-Learning scheme outperforms the random scheme but is comparable to the greedy scheme. This is because tasks generated under the Gamma distribution are always micro-tasks, which are not sensitive to user mobility. This indirectly shows that the dual-table Q-Learning scheme has a good advantage in handling mobility-sensitive tasks.

[0168] The above comparison focuses solely on latency. To better compare the overall performance of each scheme, considering both energy consumption and latency, a performance comparison formula based on the energy-latency ratio is proposed as follows:

[0169]

[0170] The formula clearly defines the relationship between consumed energy, latency, and scheme performance. Dividing consumed energy by the original total energy is to avoid an excessively large energy value.

[0171] like Figure 6 , Figure 7 , Figure 8 As shown, the horizontal axis of the three graphs represents the number of tasks, and the vertical axis represents performance in terms of energy consumption and latency. The composition of the lines is consistent with the above. Similarly, under normal and uniform distributions, the dual-table Q-Learning scheme significantly outperforms the other two schemes, and the overall performance of ET-Q-Learning is always slightly better than TQ-Learning. However, it also shows no significant advantage under a Gamma distribution, which is because tasks produced by a Gamma distribution are always insensitive to movement.

[0172] Multiple experiments revealed that when tasks are randomly generated using a Gamma distribution, the performance of the greedy approach is sometimes comparable to that of the dual-table Q-Learning approach, and sometimes worse, exhibiting an unstable state. However, in the other two distributions, the dual-table Q-Learning approach consistently outperforms the other two. Therefore, the dual-table Q-Learning approach is generally more suitable for mobile-sensitive users, offering superior and more stable performance.

[0173] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for unloading mobile tasks from a single mobile user based on reinforcement learning, characterized in that, This method includes two steps: task partitioning and task unloading decision-making. (1) Task division: When a user device enters the network, the device needs to collect information to determine the service base stations that can be connected; when the device generates a task, it first determines the task weight threshold. Determine the task type, and then determine whether further subdivision is needed based on the task type; from the above, tasks can be divided into... A number of independent atomic tasks and Atom tasks are interdependent; interdependent atomic tasks are treated as independent tasks, but tasks must be executed in sequence; while non-dependent atomic tasks are also treated as independent tasks, and there are no constraints between the tasks. Non-dependent atomic tasks can change their execution order at will. Assume there are n serving base stations in the network, and each serving base station is equipped with an edge cloud; the edge cloud is represented as... , This represents the edge cloud of the nth serving base station; a user can only connect to one serving base station within a certain period of time. Therefore, the time is divided into multiple time slots, and the serving base station the user is connecting to is determined within each time slot. Indicates whether user 𝛿 is connected to the base station Connected, among which ; This means that a mobile device is only allowed to connect to one base station within that time slot; 𝑄 is used to describe movement tasks. ,make , Where K is the total number of mobile tasks. This represents the k-th movement task. Represents a task of a user device The computational load, i.e., the CPU cycles required to complete the task. Represents computational task The size refers to the amount of data delivered to the edge cloud; when a device generates a mobile task, the size of the mobile task cannot be determined, so the mobile task needs to be segmented; at this time, it depends on whether the task exceeds the task weight threshold. There are two types: 1) Light tasks, which are tasks that do not need to be segmented; 2) Heavy tasks, which are tasks that need to be segmented. The type of task is represented by 𝑡𝑝, where 𝑡𝑝=0 indicates that the task is a light task and 𝑡𝑝=1 indicates that the task is a heavy task. Since the task is divisible, it is divided into atomic parts based on its characteristics. The atomic parts cannot be further divided. In this case, the task is divided into 𝓂 independent atomic parts and 𝓃 interdependent atomic parts. The lightweight task is an atomic task. The interdependent atomic parts need to be executed sequentially, while the independent atomic parts make independent decisions and processes. Indicates task The unloading ratio is the ratio of the number of tasks that need to be unloaded in the k-th task to the number of atomic tasks after the split. ;∑ =1 indicates that the total number of task cuts should be the actual total number of tasks. (2) Task offloading decision: During actual movement, the user selects the serving base station to connect to based on the environment and current status of the device. If the user's mobile device has insufficient battery power or computing power, and the time for processing the task locally on the device is greater than the transmission delay, then the task is considered to be offloaded to the serving base station for processing; otherwise, it is processed locally on the device. After the user connects to a base station, the base station will only be switched after other base stations have better performance or the atomic-sized task has been processed. Since the decision in this problem involves two actions, one is base station selection and the other is whether to offload, the task offloading decision uses dual-table Q-Learning to solve the task offloading decision problem. The dual-table Q-Learning algorithm trains two Q-Tables at the same time. The states and reward values ​​of these two Q-Tables are the same, but the action vectors are different. The two actions are base station selection and whether to offload. The action is selected from the state according to the R value in the Q-Table, the action is executed, the immediate reward and the new state are obtained, and then the Q-Table is updated. The two tables are trained at the same time to give the optimal solution, that is, whether to offload, and if to offload, which base station to offload to.

2. The method for offloading mobile tasks for a single mobile user based on reinforcement learning according to claim 1, characterized in that, Users select the nearest base station based on the distance between the serving base station and the device. The formula for calculating the distance d(t) between the device and the serving base station is as follows: d(𝑡) = In this case, the user is in a mobile state. The user's location is represented by a two-dimensional vector. =0 indicates that the initial position is 0, and the j-th base station. The location is determined as .

3. The method for offloading mobile tasks for a single mobile user based on reinforcement learning according to claim 1, characterized in that, Uplink data rate when user mobile devices offload tasks to the edge cloud The calculation formula is as follows: Where 𝐵 is the channel bandwidth. For transmission power, For channel gain, The power is Gaussian white noise; where the channel gain is... It will be affected by distance, and distance changes over time; Able to calculate user uninstallation tasks The transmission latency to the edge cloud is The calculation formula is as follows: This represents the k-th task. This indicates the number of tasks after the split. Indicates the number of base stations. Indicates time, This indicates transmission to edge cloud processing, in the formula... =1 indicates that the user has connected to the serving base station, and the transmission delay is calculated. The total transmission delay of a certain task is expressed as: The calculation formula is as follows: Further, it was found that the user uninstalled the task. Energy consumption for transmission to edge cloud The calculation formula is as follows: 。 4. The method for offloading mobile tasks for a single mobile user based on reinforcement learning according to claim 3, characterized in that, The following factors influence whether a task should be uninstalled: 1) Total energy e and computing power f of user equipment; 2) Energy required for task processing ; There are two scenarios regarding where computing tasks are processed: processing them locally on the device or offloading them to the serving base station for processing. 1) When the device has sufficient remaining energy to support task processing, the task will be processed locally on the device, as shown in the following formula: The formula for calculating the local processing time of a task is as follows: This indicates that a part of the task or the entire task is being processed locally. If the entire task is processed locally, then =1; Energy consumption of processing the kth task locally on the device The calculation formula is as follows: This indicates the energy consumption processed locally. The power factor represents the energy consumed per CPU cycle; 2) When the equipment's resources and remaining energy are insufficient to support task processing, the task needs to be offloaded to the edge cloud of the serving base station for processing; The computing power of edge cloud The computing power is expressed as follows: CPU cycles per second. This represents the computing power of the edge cloud at the nth serving base station; The total task duration in the edge cloud mainly includes the time consumed by two processes, namely: (1) The time consumed by the transfer when the user unloads the task; (2) The time consumed when processing computing tasks on the edge cloud; Because users may move, when a user leaves a serving base station to switch to a new base station, the device still needs to seek service from that base station. This base station handover will involve a handover delay, which is defined as... After the device switches base stations, the results of the user tasks processed by the previous base station also need to be transmitted back through the current serving base station. However, since the results are often small, this transmission delay is negligible. The formula for calculating the total task duration is as follows: 。 5. A method for unloading mobile tasks from a single mobile user based on reinforcement learning according to claim 4, characterized in that, To optimize the total latency of task unloading, define integer decision variables. , Indicates task It is local ( = 0) or in the edge cloud ( = 1) is handled in the following way; the formula for calculating the minimum total task duration to address the task unloading problem is as follows: The minimum total duration of the task unloading problem is obtained using the above formula, which is then used to determine the optimal unloading solution. This indicates the placement location of all parts of the task, i.e., the task itself. The edge cloud of the selected offloaded service base station is able to connect to the user during the time period; This represents the total energy consumption of tasks performed on a mobile device. The sum of these values ​​represents the energy consumed during transmission, and neither can exceed the total energy consumption of the device. ; A sum of 1 indicates that a mobile device can only connect to one edge server in each time slot; The sum of 1 indicates that the total number of task cuts should be the actual total number of tasks. From the user equipment's perspective, performance is also affected by the device's energy consumption. The formula for calculating the combined local latency and energy consumption performance is as follows: The formula for calculating the overall performance of unloading latency and energy consumption is as follows: This indicates the total remaining energy of the current user on the mobile device. Let represent the total remaining energy of the current user offloaded to the serving base station. The optimization problem described above is now updated as follows: the formula for maximizing the overall performance of total latency and energy consumption is as follows: The above formula can obtain the maximum value of the combined performance of latency and energy consumption, and thus determine the optimal offloading scheme.

6. The method for offloading mobile tasks for a single mobile user based on reinforcement learning according to claim 1, characterized in that, Q-Learning consists of a state (q), an action (s), and a reward value (q). The formula for updating the Q-value in Q-Learning is as follows: It is the learning rate that controls the update of the Q-value at each time step. It is a decay factor, indicating the correlation between the subsequent reward value and the current state and action; The data structure of each parameter in the dual-table Q-Learning is as follows: State: State consists of two parts ,in Indicates the location of the equipment. Indicates the user's current connected base station; Reward: 𝑟 = Where daley represents the total duration of tasks currently being processed; The shorter the total task duration, the greater the reward value; The decision-making process in this problem involves two actions: base station selection and whether to offload the service; therefore, the actions of the two tables are as follows: Action1: 𝑎1= , , indicating that the base station currently processing the task is ; Action2: 𝑎2=𝑘, , indicates the unloading decision for the remaining subtasks in the current task; 𝑎2=0 indicates that both the remaining mutually dependent and mutually independent subtasks make decisions simultaneously, and both are processed locally; 𝑎2=1 indicates that both the remaining mutually dependent and mutually independent subtasks make decisions simultaneously, with the mutually dependent subtasks processed locally and the mutually independent subtasks unloaded to the edge; 𝑎2=2 indicates that both the remaining mutually dependent and mutually independent subtasks make decisions simultaneously, with the mutually independent subtasks processed locally and the mutually dependent subtasks unloaded to the edge; 𝑎2=3 indicates that both the remaining mutually dependent and mutually independent subtasks make decisions simultaneously, and both are selected for unloading. The above solution provides a reward value based on minimizing latency. However, when the problem transforms into optimizing the overall latency-energy performance, only the reward value needs to be changed. The data structure for the reward value is as follows: Reward: Where energy represents the percentage of current remaining energy to total energy; according to the formula, the more remaining energy, the smaller the total latency, and the greater the reward value.

Images