A service quality optimization method for allocating spectrum resources in mobile edge computing

Abstract

The application discloses a service quality optimization method for allocating spectrum resources in mobile edge computing, and comprises the following steps: 1) constructing a MEC system model; 2) constructing an offloading communication model; 3) constructing a task offloading calculation model; 4) constructing a time delay cost model; and 5) constructing an optimization model under multiple constraints. The method considers spectrum resource allocation, realizes higher-efficiency algorithm updating by using DDQN, and improves the convergence performance of the algorithm by using a priority replay buffer.

Description

Technical Field

[0001] This invention relates to the field of performance optimization applications for task offloading in mobile edge computing, specifically a method for optimizing the quality of service (QoS) of allocating spectrum resources in mobile edge computing. Background Technology

[0002] With the development and application of 5G communication technology and artificial intelligence, a large number of smart mobile devices have gradually become widespread. This has led to the booming development of computationally intensive mobile applications, such as online interactive games, facial recognition, and augmented / virtual reality. These emerging applications have high requirements for Quality of Service (QoS), resulting in higher resource consumption than traditional applications. However, due to the size limitations of smart devices, their performance and battery life are quite limited. Processing complex tasks takes a long time and consumes a lot of power, severely impacting the user experience of smart devices. To address these issues, mobile cloud computing provides abundant computing and storage resources to meet users' demands for high-quality services and address the performance limitations of smart devices. Smart devices can upload heavy-load tasks to cloud computing centers for processing via wireless networks and then send the results back to the smart devices, thus partially alleviating these problems.

[0003] Mobile edge computing (MEC), as an emerging computing paradigm, has greatly alleviated the pressure of insufficient terminal resources. MEC shortens the distance between resource centers and terminals by migrating caching and computing functions from the core network in the cloud to the edge network closer to the terminal devices. This system architecture allows edge servers to possess abundant resources similar to those in the cloud, including computing, communication, and storage resources. This not only significantly reduces application latency, improves real-time performance, and reduces the energy consumption of terminal devices, but also improves the computing efficiency and user experience of the terminals. Therefore, the optimization goal of MEC is primarily to provide end users with low-latency, high-bandwidth, and real-time availability services of wireless network information to improve service quality. Especially for computationally intensive applications with strict requirements on latency and energy consumption, MEC can enable applications that are difficult to implement in traditional cloud computing environments. By offloading complex computing tasks to edge servers, terminal applications can obtain more computing resources, addressing the high demands of computationally intensive tasks and reducing battery consumption for end users. Existing research focuses on solving current problems under single constraints, but the real-world environment is complex, so offloading tasks inevitably requires consideration of multiple constraints. Sun et al., in Sun Y X. Adaptive learning-based task offloading for vehicular edge computing systems[J]. IEEE Transactions on Vehicular Technology, 2019, 68(4): 3061-3074., studied the offloading system for vehicles and roadside units, considering the dynamics and uncertainties of the vehicle task offloading environment to minimize the average offloading delay. Liu et al., in Liu J, Mao YY, Zhang J, et al. Delay-Optimal Computation Task Scheduling for Mobile-EdgeComputing System[C] / / IEEE International Symposium on Information Theory(ISIT). Barcelona: IEEE,2016:1451-1455., analyzed the average delay generated by offloading tasks in MEC systems and the average power consumption of mobile devices, formulated a power-finite delay minimization problem, and then proposed an efficient one-dimensional search algorithm to find the optimal task offloading strategy.

[0004] The real-world environment is complex, and existing research has focused on optimizing offloading performance under multiple constraints. Chen et al., in Chen X, Jiao L, Li WZ, et al. Efficient Multi-User ComputationOffloading for Mobile-Edge Cloud Computing [J]. IEEE / ACM Transactions on Networking, 2016, 24(5): 2795-2808, studied the multi-user computation offloading problem in mobile edge computing under multi-channel wireless interference, and proved that it is NP-hard. Therefore, this paper formulates the distributed offloading problem in the system as a multi-user computation offloading game, and proposes a distributed offloading algorithm that achieves Nash equilibrium to achieve lower latency overhead. He et al., in their paper "He, Y. Tang, Z. Strategy for Task Offloading of Multi-user and Multi-server Based on Cost Optimization in Mobile Edge Computing Environment," *Journal of Information Processing Systems*, 17, 615-629, 10.3745 / JIPS.01.0078, studied the task offloading problem in mobile edge computing involving multiple users and multiple servers. They transformed the distributed offloading problem among mobile devices into an offloading game and finally proposed a distributed task offloading algorithm. While these methods address task offloading in complex environments using traditional approaches, none demonstrate good service quality, i.e., lower latency, energy consumption, or task drop rates. Therefore, reinforcement learning is increasingly favored for solving dynamic problems in MEC (Multi-User Computing). Huang et al., in Huang L, Bi S, Zhang Y. Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J]. IEEE Transactions on Mobile Computing, 2019, 19(11): 2581-2593, proposed an online offloading framework based on deep reinforcement learning for wireless powered MEC networks with binary offloading, which greatly reduces computational complexity.Zhou et al., in their paper "Deep Reinforcement Learning for Energy-Efficient Computation Offloading in Mobile-Edge Computing," published in IEEE Internet of Things Journal, vol. 9, no. 2, pp. 1517-1530, January 15, 2022, studied the joint optimization of computation offloading and resource allocation in dynamic MEC systems. To minimize the energy consumption of MEC systems, they proposed an algorithm based on Q-learning. To avoid the curse of dimensionality, they further proposed an algorithm based on DDQN.

[0005] In MEC systems, both mobile devices and edge nodes have limited resources, while edge nodes possess more resources to assist mobile processing tasks, thereby improving the quality of service (QoS) of applications. Existing research on task offloading rarely considers communication resource limitations. However, communication resources within a single cell are finite; ignoring this limitation during task offloading can cause transmission channel congestion, hindering timely task feedback and leading to poor QoS. Furthermore, real-world offloading environments are complex, requiring consideration of multiple constraints such as spectrum resources, edge computing resources, and latency limitations. Most research focuses solely on latency limitations, assuming edge servers have unlimited computing resources, which is unrealistic. Therefore, considering these factors, achieving good QoS through reasonable resource allocation in a multi-constrained MEC offloading environment is challenging. Advances in deep reinforcement learning, however, make high-QoS offloading feasible. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing task offloading performance by proposing an offloading decision and resource allocation method based on deep reinforcement learning for spectrum resource allocation. This method considers spectrum resource allocation, utilizes DDQN to achieve more efficient algorithm updates, and employs a priority replay buffer to improve the algorithm's convergence performance.

[0007] The technical solution for implementing the method of the present invention is as follows:

[0008] A method for optimizing the quality of service (QoS) of spectrum resource allocation in mobile edge computing includes the following steps:

[0009] 1) Constructing the MEC system model:

[0010] Consider a communication offloading scenario within a single cell, which includes one MEC server and multiple mobile devices (MDs). In the system, the MEC server provides computing services to all mobile devices within its communication range. The set of mobile devices is represented as... When the computing resources of a mobile device are insufficient to guarantee the quality of service for an application, the mobile device offloads tasks to an edge server via a wireless network. The MEC server then assists in processing the tasks and feeds the processing results back to the mobile device. The duration set of the offloading activity in the system is set to... Within each time slot, each MD generates only one computationally intensive task, and these tasks are indivisible. Therefore, the system's computational tasks are either processed at the edge or locally. When multiple tasks need to be offloaded simultaneously, the MEC server should allocate the system's spectrum and computing resources based on time-varying system conditions, task heterogeneity, and the overall system energy consumption. The task description in the MEC system is as follows: ,in This indicates the size of the computation task. This represents the computing resources required to complete the task, which can be quantified as the total number of CPU cycles required to process the task. Indicates the maximum tolerance time for the task. This indicates the resource limit for a task, meaning that regardless of whether the task is processed locally or offloaded, the execution time and resource consumption of the task cannot exceed [a certain limit]. and Furthermore, in the MEC system, MDs are randomly distributed within the communication range of the MEC server and undergo long-term offloading activities.

[0011] This technical solution only focuses on whether the task is computed locally or offloaded, without considering cloud servers or servers in other communities. Within each time slot, when MD has a new task... Upon arrival, the MD (Member of Decisions) will decide how to handle the task based on the system state. Each MD uses an integer variable for each newly arrived task. To represent task decisions, where This indicates that the task is processed by computing resources allocated by MD, and This indicates that the task is offloaded to the MEC server; therefore, the time slot is defined in the MEC system. The uninstallation decision is ;

[0012] 2) Constructing the offloading communication model:

[0013] In an MEC offloading system, when the MD's computing resources are insufficient to handle tasks, the MD offloads the tasks to the deployed MEC servers, which then provide the computing services. For the link transmission during task offloading, the Shannon model is used to describe the communication model. Since there is only one MEC server within a single cell, interference from other servers is ignored. During offloading activities in the MEC system, the rational allocation of communication resources determines the quality of application services. Each mobile device competes for spectrum resources during offloading, which can make it difficult for some mobile devices to achieve high-performance task offloading. [Definition...] A ratio vector for spectrum resource allocation, where Indicates mobile device Given the spectrum resource ratio allocated in time slot t, the mobile device can be calculated. Transfer rate of the unloading task:

[0014]

[0015] in, The bandwidth of the available spectrum. Indicates the transmission power of the channel transmission task. For channel gain during transmission between MD and MEC servers, This refers to channel noise during transmission.

[0016] 3) Construct a task unloading calculation model:

[0017] Tasks generated by mobile devices can be processed locally or offloaded to an MEC server for processing, depending on the unloading environment. Two computational models will then be introduced:

[0018] 3.1) Local computation model:

[0019] For generated tasks, if the task size is small, the task will be processed using MD's local computing resources and will not be unloaded but processed locally. In a MEC system, different mobile devices have different computing resource capabilities, therefore, the definition is... The computing power of MD is measured in CPU cycles per second, and is set as follows: The resource consumption per CPU cycle in MD is calculated as follows: Once the computation task is processed locally, the task execution time is as follows:

[0020]

[0021] The energy consumption calculation for processing the task locally is as follows:

[0022]

[0023] in, , This indicates the device's computing power parameters;

[0024] 3.2) Unloading calculation model:

[0025] Due to limited resources, MD cannot guarantee the quality of service for its devices. To ensure service quality, MD transmits computing tasks to the MEC server via a wireless network. The task offloading process includes three stages: task transmission, task processing, and result return. Because the downlink data transmission rate is high, the overhead of result return is ignored. Therefore, the main overhead of task offloading includes task transmission and task processing, i.e., task... Time of transmission to MEC server, task The time when the MEC server is processed;

[0026] Regarding the latency consumption during transmission, the latency cost is related to the data size and the uplink transmission rate, so the latency calculation is as follows:

[0027]

[0028] The corresponding transmission energy consumption for task offloading is calculated as follows:

[0029]

[0030] For the other part of offloading computation, namely the computational processing of the MEC server, the MEC server allocates different computing resources according to the computational requirements of the task; define ,in This represents the ratio of computing resources allocated by the MEC server to computing tasks in time slot t. This indicates the computing resources of the MEC server; when a task is allocated an excessively high resource ratio... When computing resources are allocated too intensively, task execution time will be shortened, but energy consumption will increase. Therefore, implementing a reasonable resource allocation scheme helps improve service quality. Meanwhile, the resources of the MEC server need to meet certain requirements. The constraint is that the allocated resources cannot exceed [a certain limit]. So, the task The execution latency cost of the MEC server is calculated as follows:

[0031]

[0032] And the mission When time slot t is processed by the MEC server, the energy consumption cost is calculated as follows:

[0033]

[0034] in, This represents the computing power of the MEC server under full load.

[0035] Combining the cost of the task unloading process, the total latency cost and energy cost of the MD unloading task are expressed as follows:

[0036]

[0037]

[0038] 4) Construct a latency cost model:

[0039] In each time slot, when a task is generated by the MD (Main Controller), the MD will decide how to handle the task. The execution time cost of each task, depending on the unloading method, is expressed as follows:

[0040]

[0041] Similarly, the task The energy consumption cost for the completed processing is calculated as follows:

[0042]

[0043] Finally, considering the computational tasks of all mobile devices in the MEC system within time slot t, the total latency cost of time slot t is expressed as:

[0044]

[0045] 5) Construct an optimization model under multiple constraints:

[0046] In the MEC offloading system, the latency and energy costs of offloading tasks in the entire system are analyzed, including local computation costs and offloading computation costs. The joint optimization problem of offloading decisions and resource allocation in the MEC system is also studied. Since service quality is a crucial factor affecting user experience in mobile devices, improving service quality will lead to a better user experience. Furthermore, service quality is determined by latency and task drop rate; therefore, minimizing latency and task drop rate optimizes service quality. The corresponding service quality optimization problem is formulated as follows:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] in, C1 indicates the average latency consumption of offloading tasks in the MEC system. C2 and C3 indicate that each MD can only choose to process tasks locally or offload them. C4 and C5 indicate that the total spectrum resources allocated to all MDs cannot exceed the total spectrum resources in the MEC system. C6 indicates that the total resources allocated to the MDs for offloading tasks cannot exceed the resources owned by the MEC server. C7 indicates that the processing latency of each task does not exceed the tolerance time.

[0056] This technical solution studies the offloading decision and resource allocation problem under latency and energy constraints. It considers the proportional allocation of spectrum resources and edge MEC computing resources in the system. The detailed allocation scheme enables more reasonable resource allocation in the MEC system and allows applications to provide better services. However, solving complex problems under multiple constraints is difficult, requiring the finding of offloading decisions within each time slot. Spectrum resource allocation vector and computational resource allocation vector To achieve the optimization goal of minimizing latency; among which, the offloading decision It is a binary variable, while the spectrum resource ratio and calculate resource allocation ratio The problem varies with the state of the MEC environment, making it difficult to solve using traditional methods such as game theory and some heuristic algorithms. Therefore, a deep reinforcement learning-based approach is proposed to solve the problem in dynamic environments.

[0057] For reinforcement learning methods, the problem needs to be approximated as a Markov Decision Process (MDP). An MDP is a mathematical framework for establishing sequential decision problems, typically used to describe the interaction between an agent and its environment. It has become a highly regarded model in machine learning, especially in reinforcement learning where it has become an essential foundational model. The basis for implementing an MDP model is defining three spatial elements: state space, action space, and reward space.

[0058] Markov decision model:

[0059] (1) State space:

[0060] The state in MDP is a space that reflects the MEC offloading environment. In MEC systems, spectrum resources and edge computing resources are the most important dynamic factors. Therefore, the state space is defined using the available resources in the system, where the spectrum resources for time slot t are... The computing resources are The purpose of observing both is to maintain the constraints of computing power and communication. In addition, it is also necessary to observe the energy consumption cost generated by each time slot. Therefore, the state space of time slot t is defined as follows:

[0061]

[0062] (2) Action space:

[0063] In reinforcement learning, the mobile device acts as the agent responsible for offloading actions. The goal is to map the state space to the action space. Implementing the offloading decision also requires determining the spectrum resources allocated to the mobile device and the computational resources of the MEC in each time slot t. Therefore, the vector of the action space consists of three parts: the offloading decision vector... Spectrum resource allocation ratio vector and computational resource allocation vector The action space is defined as:

[0064]

[0065] The specific action decision is represented as follows:

[0066]

[0067] in This indicates that in deep reinforcement learning, and The value output by the neural network, as a variable, is used to measure the quality of action decisions.

[0068] (3) Reward space:

[0069] The elements in the reward space are related to the objective function. The proposed optimization model defines an optimization problem that minimizes the average system latency. This is the inverse objective of reinforcement learning methods to achieve maximum reward, and therefore negatively correlated with the reward of reinforcement learning. The mobile device is in a state... Make an uninstall decision The reward is defined as:

[0070]

[0071] The reward value of MDP should meet the constraints in the optimization model. If the constraints are not met, the algorithm based on deep reinforcement learning will set the reward as a penalty value during the iteration process. That is, the mobile device uninstallation task will fail if the conditions are not met. After iteration through the deep reinforcement learning algorithm, the uninstallation failure rate of the task will be reduced.

[0072] In summary, we design a PERDDQN algorithm for unloading decisions and resource allocation that combines DDQN and Priority replay Buffer. This avoids the drawback of dimensionality explosion in Q-learning and utilizes Priority replay Buffer to implement a priority experience store. The specific algorithm content is as follows:

[0073] 1. Initialize the priority register

[0074] 2. Using random parameter vectors Initialize the main deep neural network

[0075] 3. Using random parameter vectors Initialize the main deep neural network

[0076] 4. Traverse the unloading cycle episode

[0077] 5. Obtain initial environment values

[0078] 6. Iterate through each time slot t in the cycle.

[0079] 7. Initially, with a random probability Select Action

[0080] 8. If Just randomly select an action.

[0081] 9. Otherwise: based on Select Action

[0082] 10. Mobile devices based on Uninstall tasks and receive uninstallation rewards. and enter the next state.

[0083] 11. The priority of initializing experience is... Maximum priority

[0084] 12. Incorporate experiential information Stored in priority memory

[0085] 13. From Experience information of sampling a batch according to priority Used for algorithm updates

[0086] 14. Traversing the empirical information from sampling

[0087] 15. Update the priority of each experience.

[0088] 16. Calculate the target Q value for each experience.

[0089] 17. Minimize the error function in the RL method. To update the weight parameters of the main network

[0090] 18. Regarding Perform gradient descent steps

[0091] 19. Utilization Update the weight parameters of the target network

[0092] 20. End the traversal.

[0093] Compared with existing technical solutions, this technical solution has the following characteristics:

[0094] 1. Considering Spectrum Resources to Optimize Service Quality: Many existing studies on task offloading in MEC systems do not consider spectrum resources, mostly focusing solely on binary offloading to directly optimize the target. While some studies consider the constraints of computing resources in the MEC server, these approaches are not suitable for real-world offloading environments. This technical solution incorporates spectrum resources into the optimization constraints, achieving optimal task offloading efficiency by setting different spectrum resource ratios for each mobile device.

[0095] 2. Algorithm Design Utilizing DDQN and Priority Replay Buffer: In MEC offloading applications, deep reinforcement learning is frequently used in complex offloading environments due to its excellent learning and simulation capabilities. However, existing reinforcement learning methods, such as Q-learning, often suffer from dimensionality explosion under complex decision-making. Therefore, DQN and DDQN methods are increasingly used in MEC, but their random sampling of empirical memories leads to poor convergence performance. This technical solution utilizes DDQN and Priority Replay Buffer to implement a deep reinforcement learning algorithm with a priority empirical memory, achieving latency minimization under multiple constraints.

[0096] 3. Delay optimization under consideration of latency and resources: In a single-cell offloading environment, this technical solution considers performance optimization under multiple constraints and establishes a one-to-many binary system model. Task offloading is completed based on mobile device resource conditions and channel status. Based on the local computational cost and offloading computational cost generated by the offloading method, a latency cost minimization problem under multiple constraints is formulated. Attached Figure Description

[0097] Figure 1 System model diagram;

[0098] Figure 2 This is a diagram of the training structure of the algorithm. Detailed Implementation

[0099] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0100] Example:

[0101] Reference Figure 1 A method for optimizing the quality of service (QoS) of spectrum resource allocation in mobile edge computing includes the following steps:

[0102] 1) Constructing the MEC system model:

[0103] Consider a communication offloading scenario within a single cell, which includes one MEC server and multiple mobile devices (MDs). In the system, the MEC server provides computing services to all mobile devices within its communication range. The set of mobile devices is represented as... When the computing resources of a mobile device are insufficient to guarantee the quality of service for an application, the mobile device offloads tasks to an edge server via a wireless network. The MEC server then assists in processing the tasks and feeds the processing results back to the mobile device. The duration set of the offloading activity in the system is set to... Within each time slot, each MD generates only one computationally intensive task, and these tasks are indivisible. Therefore, the system's computational tasks are either processed at the edge or locally. When multiple tasks need to be offloaded simultaneously, the MEC server should allocate the system's spectrum and computing resources based on time-varying system conditions, task heterogeneity, and the overall system energy consumption. The task description in the MEC system is as follows: ,in This indicates the size of the computation task. This represents the computing resources required to complete the task, which can be quantified as the total number of CPU cycles required to process the task. Indicates the maximum tolerance time for the task. This refers to the resource limit for a task, meaning that regardless of whether the task is processed locally or offloaded, the execution time and resource consumption of the task cannot exceed [a certain limit]. and Furthermore, in the MEC system, MDs are randomly distributed within the communication range of the MEC server and undergo long-term offloading activities.

[0104] This example only focuses on whether the task is computed locally or offloaded, without considering cloud servers or servers in other communities. Within each time slot, when MD has a new task... Upon arrival, the MD (Member of Decisions) will decide how to handle the task based on the system state. Each MD uses an integer variable for each newly arrived task. To represent task decisions, where This indicates that the task is processed by computing resources allocated by MD, and This indicates that the task is offloaded to the MEC server; therefore, the time slot is defined in the MEC system. The uninstallation decision is ;

[0105] 2) Constructing the offloading communication model:

[0106] In an MEC offloading system, when the MD's computing resources are insufficient to handle tasks, the MD offloads the tasks to the deployed MEC servers, which then provide the computing services. For the link transmission during task offloading, the Shannon model is used to describe the communication model. Since there is only one MEC server within a single cell, interference from other servers is ignored. During offloading activities in the MEC system, the rational allocation of communication resources determines the quality of application services. Each mobile device competes for spectrum resources during offloading, which can make it difficult for some mobile devices to achieve high-performance task offloading. [Definition...] A ratio vector for spectrum resource allocation, where Indicates mobile device Given the spectrum resource ratio allocated in time slot t, the mobile device can be calculated. Transfer rate of the unloading task:

[0107]

[0108] in, The bandwidth of the available spectrum. Indicates the transmission power of the channel transmission task. For channel gain during transmission between MD and MEC servers, This refers to channel noise during transmission.

[0109] 3) Construct a task unloading calculation model:

[0110] Tasks generated by mobile devices can be processed locally or offloaded to an MEC server for processing, depending on the unloading environment. Two computational models will then be introduced:

[0111] 3.1) Local computation model:

[0112] For generated tasks, if the task size is small, the task will be processed locally using MD's local computing resources and will not be unloaded but processed locally. In a MEC system, different mobile devices have different computing resource capabilities, therefore, the definition is... The computing power of MD is measured in CPU cycles per second, and is set as follows: The resource consumption per CPU cycle in MD is calculated as follows: Once the computation task is processed locally, the task execution time is as follows:

[0113]

[0114] The energy consumption calculation for processing the task locally is as follows:

[0115]

[0116] in, , This indicates the device's computing power parameters;

[0117] 3.2) Unloading calculation model:

[0118] Due to limited resources, MD cannot guarantee the quality of service for its devices. To ensure service quality, MD transmits computing tasks to the MEC server via a wireless network. The task offloading process includes three stages: task transmission, task processing, and result return. Because the downlink data transmission rate is high, the overhead of result return is ignored. Therefore, the main overhead of task offloading includes task transmission and task processing, i.e., task... Time of transmission to MEC server, task The time when the MEC server is processed;

[0119] Regarding the latency consumption during transmission, the latency cost is related to the data size and the uplink transmission rate, so the latency calculation is as follows:

[0120]

[0121] The corresponding transmission energy consumption for task offloading is calculated as follows:

[0122]

[0123] For the other part of offloading computation, namely the computational processing of the MEC server, the MEC server allocates different computing resources according to the computational requirements of the task; define ,in This represents the ratio of computing resources allocated by the MEC server to computing tasks in time slot t. This indicates the computing resources of the MEC server; when a task is allocated an excessively high resource ratio... When computing resources are allocated too intensively, task execution time will be shortened, but energy consumption will increase. Therefore, implementing a reasonable resource allocation scheme helps improve service quality. Meanwhile, the resources of the MEC server need to meet certain requirements. The constraint is that the allocated resources cannot exceed [a certain limit]. So, the task The execution latency cost of the MEC server is calculated as follows:

[0124]

[0125] And the mission When time slot t is processed by the MEC server, the energy consumption cost is calculated as follows:

[0126]

[0127] in, This represents the computing power of the MEC server under full load.

[0128] Combining the cost of the task unloading process, the total latency cost and energy cost of the MD unloading task are expressed as follows:

[0129]

[0130]

[0131] 4) Construct a latency cost model:

[0132] In each time slot, when a task is generated by the MD (Main Controller), the MD will decide how to handle the task. The execution time cost of each task, depending on the unloading method, is expressed as follows:

[0133]

[0134] Similarly, the task The energy consumption cost for the completed processing is calculated as follows:

[0135]

[0136] Finally, considering the computational tasks of all mobile devices in the MEC system within time slot t, the total latency cost of time slot t is expressed as:

[0137]

[0138] 5) Construct an optimization model under multiple constraints:

[0139] In the MEC offloading system, the latency and energy costs of offloading tasks in the entire system are analyzed, including local computation costs and offloading computation costs. The joint optimization problem of offloading decisions and resource allocation in the MEC system is also studied. Since service quality is a crucial factor affecting user experience in mobile devices, improving service quality will lead to a better user experience. Service quality is determined by latency and task drop rate; therefore, minimizing latency and task drop rate is equivalent to optimizing service quality. The corresponding service quality optimization problem is formulated as follows:

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] in, C1 indicates the average latency consumption of offloading tasks in the MEC system. C2 and C3 indicate that each MD can only choose to process tasks locally or offload them. C4 and C5 indicate that the total spectrum resources allocated to all MDs cannot exceed the total spectrum resources in the MEC system. C6 indicates that the total resources allocated to the MDs for offloading tasks cannot exceed the resources owned by the MEC server. C7 indicates that the processing latency of each task does not exceed the tolerance time.

[0149] This case study investigates offloading decisions and resource allocation under latency and energy constraints. It considers the proportional allocation of spectrum resources and edge MEC computing resources within the system. A detailed allocation scheme allows for more rational resource allocation within the MEC system and enables applications to provide better service. However, solving complex problems under multiple constraints is challenging, requiring the identification of offloading decisions within each time slot. Spectrum resource allocation vector and computational resource allocation vector To achieve the optimization goal of minimizing latency; among which, the offloading decision It is a binary variable, while the spectrum resource ratio and calculate resource allocation ratio The problem varies with the state of the MEC environment, making it difficult to solve using traditional methods such as game theory and some heuristic algorithms. Therefore, a deep reinforcement learning-based approach is proposed to solve the problem in dynamic environments.

[0150] For reinforcement learning methods, the problem needs to be approximated as a Markov model, or MDP. An MDP is a mathematical framework for establishing sequential decision problems, typically used to describe the interaction between an agent and its environment. It has become a highly regarded model in machine learning, especially in reinforcement learning where it has become an essential foundational model. The basis for implementing an MDP model is defining three spatial elements: state space, action space, and reward space.

[0151] Markov decision model:

[0152] (1) State space:

[0153] The state in MDP is a space that reflects the MEC offloading environment. In MEC systems, spectrum resources and edge computing resources are the most important dynamic factors. Therefore, the state space is defined using the available resources in the system, where the spectrum resources for time slot t are... The computing resources are The purpose of observing both is to maintain the constraints of computing power and communication. In addition, it is also necessary to observe the energy consumption cost generated by each time slot. Therefore, the state space of time slot t is defined as follows:

[0154]

[0155] (2) Action space:

[0156] In reinforcement learning, the mobile device acts as the agent responsible for offloading actions. The goal is to map the state space to the action space. Implementing the offloading decision also requires determining the spectrum resources allocated to the mobile device and the computational resources of the MEC in each time slot t. Therefore, the vector of the action space consists of three parts: the offloading decision vector... Spectrum resource allocation ratio vector and computational resource allocation vector The action space is defined as:

[0157]

[0158] The specific action decision is represented as follows:

[0159]

[0160] in This indicates that in deep reinforcement learning, and The value output by the neural network, as a variable, is used to measure the quality of action decisions.

[0161] (3) Reward space:

[0162] The elements in the reward space are related to the objective function. The proposed optimization model defines an optimization problem that minimizes the average system latency. This is the inverse objective of reinforcement learning methods to achieve maximum reward, and therefore negatively correlated with the reward of reinforcement learning. The mobile device is in a state... Make an uninstall decision The reward is defined as:

[0163]

[0164] The reward value of MDP should meet the constraints in the optimization model. If the constraints are not met, the algorithm based on deep reinforcement learning will set the reward as a penalty value during the iteration process. That is, the mobile device uninstallation task will fail if the conditions are not met. After iteration through the deep reinforcement learning algorithm, the uninstallation failure rate of the task will be reduced.

[0165] In summary, we design a PERDDQN algorithm for unloading decisions and resource allocation that combines DDQN and Priority replay Buffer. This avoids the drawback of dimensionality explosion in Q-learning and utilizes Priority replay Buffer to implement a priority experience store, such as... Figure 2 As shown, the specific content of the algorithm is as follows:

[0166] 1. Initialize the priority register

[0167] 2. Using random parameter vectors Initialize the main deep neural network

[0168] 3. Using random parameter vectors Initialize the main deep neural network

[0169] 4. Traverse the unloading cycle episode

[0170] 5. Obtain initial environment values

[0171] 6. Iterate through each time slot t in the cycle.

[0172] 7. Initially, with a random probability Select Action

[0173] 8. If Just randomly select an action.

[0174] 9. Otherwise: based on Select Action

[0175] 10. Mobile devices based on Uninstall tasks and receive uninstallation rewards. and enter the next state.

[0176] 11. The priority of initializing experience is... Maximum priority

[0177] 12. Incorporate experiential information Stored in priority memory

[0178] 13. From Experience information of sampling a batch according to priority Used for algorithm updates

[0179] 14. Traversing the empirical information from sampling

[0180] 15. Update the priority of each experience.

[0181] 16. Calculate the target Q value for each experience.

[0182] 17. Minimize the error function in the RL method. To update the weight parameters of the main network

[0183] 18. Regarding Perform gradient descent steps

[0184] 19. Utilization Update the weight parameters of the target network

[0185] 20. End the traversal.

Claims

1. A method for optimizing the quality of service (QoS) of spectrum resource allocation in mobile edge computing, characterized in that, Includes the following steps: 1) Constructing the MEC system model: Consider a communication offloading scenario within a single cell, which includes one MEC server and multiple mobile devices (MDs). In the system, the MEC server provides computing services to all mobile devices within its communication range. The set of mobile devices is represented as... When the computing resources of a mobile device are insufficient to guarantee the quality of service for an application, the mobile device offloads tasks to an edge server via a wireless network. The MEC server then assists in processing the tasks and feeds the processing results back to the mobile device. The duration set of the offloading activity in the system is set to... In each time slot, each MD generates only one computationally intensive task, and the task is indivisible, so the system's computational tasks are either processed at the edge or locally. When multiple tasks need to be offloaded simultaneously, the MEC server should allocate system spectrum and computing resources based on time-varying system conditions, task heterogeneity, and overall system energy consumption. Tasks in the MEC system are described as follows: ,in This indicates the size of the computation task. This represents the computing resources required to complete the task, which can be quantified as the total number of CPU cycles required to process the task. Indicates the maximum tolerance time for the task. This refers to the resource limit for a task, meaning that regardless of whether the task is processed locally or offloaded, the execution time and resource consumption of the task cannot exceed [a certain limit]. and Furthermore, in the MEC system, MDs are randomly distributed within the communication range of the MEC server and undergo long-term offloading activities. Within each time slot, when MD has a new task Upon arrival, the MD (Member of Decisions) will decide how to handle the task based on the system state. Each MD uses an integer variable for each newly arrived task. To represent task decisions, where This indicates that the task is processed by computing resources allocated by MD, and This indicates that the task is offloaded to the MEC server; therefore, the MEC system is defined in the time slot. The uninstallation decision is ; 2) Constructing the offloading communication model: In an MEC offloading system, when the MD's computing resources are insufficient to handle tasks, the MD offloads the tasks to the deployed MEC servers, which then provide the computing services. For the link transmission during task offloading, the Shannon model is used to describe the communication model. Since there is only one MEC server within a single cell, interference from other servers is ignored. During offloading activities in the MEC system, the rational allocation of communication resources determines the quality of application services. Each mobile device competes for spectrum resources during offloading, which can make it difficult for some mobile devices to achieve high-performance task offloading. [Definition...] A ratio vector for spectrum resource allocation, where Indicates mobile device Given the spectrum resource ratio allocated in time slot t, the mobile device can be calculated. Transfer rate of the unloading task: in, The bandwidth of the available spectrum. Indicates the transmission power of the channel transmission task. For channel gain during transmission between MD and MEC servers, This refers to channel noise during transmission. 3) Construct a task unloading calculation model: Tasks generated by mobile devices can be processed locally or offloaded to an MEC server for processing, depending on the unloading environment. Two computational models will then be introduced: 3.1) Local computation model: For generated tasks, if the task size is small, the task will be processed locally using MD's local computing resources and will not be unloaded but processed locally. In a MEC system, different mobile devices have different computing resource capabilities, therefore, the definition is... The computing power of MD is measured in CPU cycles per second, and is set as follows: The resource consumption per CPU cycle in MD is calculated as follows: Once the computation task is processed locally, the task execution time is as follows: The energy consumption calculation for processing the task locally is as follows: in, , This indicates the device's computing power parameters; 3.2) Unloading calculation model: Due to limited resources, MD cannot guarantee the quality of service for its devices. To ensure service quality, MD transmits computing tasks to the MEC server via a wireless network. The task offloading process includes three stages: task transmission, task processing, and result return. Because the downlink data transmission rate is high, the overhead of result return is ignored. Therefore, the main overhead of task offloading includes task transmission and task processing, i.e., task... Time of transmission to MEC server, task The time when the MEC server is processed; Regarding the latency consumption during transmission, the latency cost is related to the data size and the uplink transmission rate, so the latency calculation is as follows: The corresponding transmission energy consumption for task offloading is calculated as follows: For the other part of offloading computation, namely the computational processing of the MEC server, the MEC server allocates different computing resources according to the computational requirements of the task; define ,in This represents the ratio of computing resources allocated by the MEC server to computing tasks in time slot t. This indicates the computing resources of the MEC server; when a task is allocated an excessively high resource ratio... When computing resources are allocated too intensively, task execution time will be shortened, but energy consumption will increase. Therefore, implementing a reasonable resource allocation scheme helps improve service quality. Meanwhile, the resources of the MEC server need to meet certain requirements. The constraint is that the allocated resources cannot exceed [a certain limit]. So, the task The execution latency cost of the MEC server is calculated as follows: And the mission When time slot t is processed by the MEC server, the energy consumption cost is calculated as follows: in, This represents the computing power of the MEC server under full load. Combining the cost of the task unloading process, the total latency cost and energy cost of the MD unloading task are expressed as follows: 4) Construct a latency cost model: In each time slot, when a task is generated by the MD (Main Controller), the MD will decide how to handle the task. The execution time cost of each task, depending on the unloading method, is expressed as follows: Similarly, the task The energy consumption cost for the completed processing is calculated as follows: Finally, considering the computational tasks of all mobile devices in the MEC system within time slot t, the total latency cost of time slot t is expressed as: 5) Construct an optimization model under multiple constraints: In the MEC offloading system, the latency and energy costs of offloading tasks in the entire system are analyzed, including local computation costs and offloading computation costs. The joint optimization problem of offloading decisions and resource allocation in the MEC system is also studied. Since service quality is a crucial factor affecting user experience in mobile devices, improving service quality will lead to a better user experience. Furthermore, service quality is determined by latency and task drop rate; therefore, minimizing latency and task drop rate optimizes service quality. The corresponding service quality optimization problem is formulated as follows: in, C1 indicates the average latency consumption of offloading tasks in the MEC system. C2 and C3 indicate that each MD can only choose to process tasks locally or offload them. C4 and C5 indicate that the total spectrum resources allocated to all MDs cannot exceed the total spectrum resources in the MEC system. C6 indicates that the total resources allocated to the MDs for offloading tasks cannot exceed the resources owned by the MEC server. C7 indicates that the processing latency of each task does not exceed the tolerance time. Considering the proportional allocation of spectrum resources in the system and computing resources in edge MECs, a meticulous allocation scheme can achieve more rational resource distribution within the MEC system and enable applications to provide better services. However, solving complex problems under multiple constraints is difficult, requiring the discovery of offloading decisions within each time slot. Spectrum resource allocation vector and computational resource allocation vector To achieve the optimization goal of minimizing latency; among which, the offloading decision It is a binary variable, while the spectrum resource ratio and calculate resource allocation ratio The problem varies with the state of the MEC environment, making it difficult to solve using traditional methods. Therefore, a deep reinforcement learning-based approach is proposed to solve the problem in dynamic environments. For deep reinforcement learning methods, the problem needs to be approximated as a Markov model, i.e., MDP. The basis for implementing the MDP model is to define three spatial elements: state space, action space, and reward space. Markov decision model: (1) State space: The state in MDP is a space that reflects the MEC offloading environment. In MEC systems, spectrum resources and edge computing resources are the most important dynamic factors. Therefore, the state space is defined using the available resources in the system, where the spectrum resources for time slot t are... The computing resources are The purpose of observing both is to maintain the constraints of computing power and communication. In addition, it is also necessary to observe the energy consumption cost generated by each time slot. Therefore, the state space of time slot t is defined as follows: (2) Action space: As an agent in reinforcement learning, the mobile device is the implementer of the offloading action. The goal is to map the state space to the action space. Implementing the offloading decision also requires determining the spectrum resources allocated to the mobile device and the computational resources of the MEC in each time slot t. Therefore, the vector of the action space consists of three parts: the offloading decision vector... Spectrum resource allocation ratio vector and computational resource allocation vector The action space is defined as: The specific action decision is represented as follows: in This indicates that in deep reinforcement learning, and The value output by the neural network, as a variable, is used to measure the quality of action decisions. (3) Reward Space: The elements in the reward space are related to the objective function. The proposed optimization model defines an optimization problem that minimizes the average system latency. This is the inverse objective of reinforcement learning methods to achieve maximum reward, and therefore negatively correlated with the reward of reinforcement learning. The mobile device is in a state... Make an uninstall decision The reward is defined as: The reward value of MDP should meet the constraints in the optimization model. If the constraints are not met, the algorithm based on deep reinforcement learning will set the reward as a penalty value during the iteration process. That is, the mobile device uninstallation task will fail if the conditions are not met. The uninstallation failure rate of the task is reduced after iteration by the deep reinforcement learning algorithm. In summary, a PERDDQN algorithm for unloading decision-making and resource allocation, combining DDQN and Priority replay Buffer, is designed to avoid the drawback of dimensionality explosion in Q-learning. Furthermore, a Priority replay Buffer is used to implement a priority experience store. The specific algorithm content is as follows:

1. Initialize the priority register 2. Using random parameter vectors Initialize the main deep neural network 3. Using random parameter vectors Initialize the main deep neural network 4. Traverse the unloading cycle episode 5. Obtain initial environment values 6. Traverse each time slot t in the cycle.

7. Initially, with a random probability Select Action 8. If Just randomly select an action.

9. Otherwise: based on Select Action 10. Mobile devices based on Uninstall tasks and receive uninstallation rewards. and enter the next state.

11. The priority of initializing experience is... Maximum priority 12. Incorporate experiential information Stored in priority memory 13. From Experience information of sampling a batch according to priority Used for algorithm updates 14. Traversing the empirical information from sampling 15. Update the priority of each experience.

16. Calculate the target Q value for each experience.

17. Minimize the error function in the RL method. To update the weight parameters of the main network 18. Regarding Perform gradient descent steps 19. Utilization Update the weight parameters of the target network 20. End the traversal.

Images