Value maximization computing offloading method and system based on dynamic priority scheduling and reinforcement learning

Abstract

This invention discloses a value maximization computational offloading method and system based on dynamic priority scheduling and reinforcement learning. The invention constructs an optimization framework with the core objective of maximizing the system's "long-term average task value." By introducing an exponential value function to define task value, it addresses the "value neglect" problem inherent in traditional "average latency or energy consumption optimization" strategies. This value-oriented optimization mechanism enables multi-agent systems to autonomously learn strategies to prioritize high-value tasks during training when resources are limited. Furthermore, this invention proposes a dynamic priority function that accurately measures the actual urgency level of tasks at different times, thus overcoming the shortcomings of "static priority" models in handling real-time changes in urgency. Notably, this dynamic priority method is not only used for task selection in the waiting queue but is also innovatively introduced into the dynamic resource allocation mechanism of the running queue.

Description

Technical Field

[0001] This invention belongs to the field of industrial Internet of Things (IoT) technology, and relates to a value maximization computational offloading method and system based on dynamic priority scheduling and reinforcement learning. Background Technology

[0002] Driven by the Industrial Internet of Things (IIoT) technology, automated production processes are undergoing profound changes through the deployment of numerous intelligent devices such as sensors, actuators, and robots. This has generated an explosive demand for computing resources, often accompanied by stringent requirements for low latency and high reliability. Multi-access edge computing (MEC), by pushing computing power down to the network edge (e.g., factory floors), is widely regarded as a key technology for addressing this challenge.

[0003] In typical MEC scenarios, multiple terminal devices (users) share the computing resources of a single edge server. Each device may generate multiple tasks in each time slot, allowing for partial offloading at any ratio. However, due to the highly dynamic network state, the randomness of task arrival, and the resource competition and decision coupling among multiple users, designing a globally optimal offloading strategy constitutes an extremely complex distributed optimization problem. Multi-Agent Deep Reinforcement Learning (MADRL), with its ability to effectively handle distributed decision-making and multi-agent collaboration, has become a promising research direction in this context.

[0004] Although research on computational offloading based on MADRL is increasing, there are currently two main types of schemes most similar to this invention:

[0005] The first category is MADRL offloading schemes that aim for "optimal average latency or energy consumption." This type of method models multiple terminal devices as multiple agents, trains them using multi-agent reinforcement learning algorithms, and typically optimizes to minimize the average completion time or average energy consumption of all tasks. These schemes treat all tasks as equally important, and their scheduling strategies often implicitly assume the use of a First-In-First-Out (FIFO) queue, or simply simplify it.

[0006] The second type is the MADRL offloading scheme based on "static priority," which is an improvement on the first type and takes into account the differences between tasks. This type of method typically assigns a fixed static priority label (e.g., high, medium, low) to tasks based on their business attributes (such as data size or deadline) when the task is created. Subsequently, the MADRL agent is trained to prioritize the Quality of Service (QoS) of high-priority tasks. On the edge server side, the execution scheduling queue may also be sorted or preempted based on this static priority.

[0007] The aforementioned existing technical solutions have fundamental technical flaws in addressing the computation offloading problem in IIoT scenarios:

[0008] The first type of MADRL solution, which prioritizes "optimal average latency or energy consumption," adopts a task homogenization paradigm, neglecting the heterogeneity of task value in the IIoT environment. For example, the importance of device security early warning tasks differs significantly from that of routine data acquisition tasks; failure in the former can lead to serious losses, while the impact of a short delay in the latter is limited. Pursuing optimal average latency or energy consumption may sacrifice high-value critical tasks in resource competition, exhibiting a "value-blindness" flaw.

[0009] While the second type of "static priority" MADRL scheme introduces a value-aware mechanism, its static attributes cannot adapt to dynamic environments. This scheme confuses the inherent importance of tasks with their real-time urgency. In resource-contested edge computing environments, the urgency of tasks changes dynamically with factors such as remaining deadlines. For example, a medium-priority task nearing its deadline may be more urgent than a newly arrived high-priority task; the static model cannot capture this dynamic characteristic, causing scheduling decisions to fail to reflect real-time demands and exhibiting the defect of "inability to dynamically adapt to urgent changes."

[0010] To address the two major shortcomings of the existing technologies mentioned above—"value blindness" and "static priority failing to reflect real-time urgency"—this invention proposes a novel, value-aware computational offloading method and system. Summary of the Invention

[0011] This invention aims to solve a core technical problem in the Industrial Internet of Things (IIoT) multi-access edge computing (MEC) scenario: how to formulate optimal computing offloading decisions and power allocation strategies for terminal devices in a resource-constrained and highly dynamic environment with multiple terminal devices, multiple tasks, and limited resources.

[0012] Specifically, this invention aims to overcome the "value blindness" defect of existing technologies in optimizing objectives, namely, how to effectively distinguish and prioritize the Quality of Service (QoS) of security-critical high-value tasks, and prevent them from being squeezed out by low-value tasks in the process of resource competition; the ultimate goal is to maximize the long-term average task value of the system under the premise of meeting task latency constraints, rather than simply pursuing the minimization of average latency or energy consumption.

[0013] Firstly, this invention provides a value maximization computational offloading method based on dynamic priority scheduling and reinforcement learning, applied to industrial IoT edge computing scenarios, specifically:

[0014] S1. Construct a task model, a communication and task transmission model, an edge computing model, and a local computing model, wherein:

[0015] The task model is used to characterize multiple computing tasks generated by the terminal device in each time slot. Each task includes data volume, CPU cycle requirement, maximum tolerable latency and initial static priority.

[0016] The communication and task transmission model is used to calculate the transmission rate from the terminal device to the edge server based on Shannon's formula, and to determine the transmission order based on the static priority of the task, so that high static priority tasks can occupy the limited transmission capacity first.

[0017] The edge computing model is used to maintain a waiting queue and a running queue on the edge server side, schedule tasks in the waiting queue based on a dynamic priority function, and solve a convex optimization problem based on the dynamic priority weight of each task in the running queue to dynamically reallocate computing resources.

[0018] The local computing model is used to perform non-preemptive serial execution of the unloaded portion of tasks on the terminal device side based on the dynamic priority function.

[0019] S2. Define an optimization problem with the goal of maximizing the long-term average task value of the system, and constrain the total task delay to not exceed its maximum tolerable delay, the transmission power to not exceed the maximum transmit power of the device, and the offloading ratio to be between 0 and 1;

[0020] S3. The optimization problem is modeled as a partially observable Markov decision process. An independent agent is defined for each terminal device. An observation space is set up, which includes the remaining computational cost of the edge queue, the remaining computational cost of the local queue, the channel gain and the current task information. A composite action space is set up, which includes the transmission power and the task offloading ratio. A hybrid reward function is set up, which integrates the sparse reward of the value of the completed task and the dense reward of the value of the uncompleted task and the urgency penalty.

[0021] S4. A decoupled multi-agent deep reinforcement learning algorithm is adopted, which enables each agent to independently decide on transmission power and offloading ratio, and share a centralized value network. Through sparse-dense hybrid rewards, agents are guided to prioritize the service quality of high-value and high-urgency tasks in resource-constrained environments, thereby solving the optimization problem and maximizing the long-term average task value of the system.

[0022] Secondly, this invention provides a value maximization computational offloading system based on dynamic priority scheduling and reinforcement learning, applied to industrial IoT edge computing scenarios, including:

[0023] M terminal devices are used to generate multiple computing tasks in each time slot. Each task carries the amount of data, CPU cycle requirements, maximum tolerable latency and initial static priority, and independently decides the transmission power and task offloading ratio based on local observation.

[0024] A base station is used to provide uplink access to the terminal device via frequency division multiple access.

[0025] Edge servers, co-located with the base station, are used for:

[0026] Receive and cache edge subtasks unloaded by terminal devices, and maintain waiting queues and running queues;

[0027] Tasks in the waiting queue are scheduled based on a dynamic priority function, and the highest dynamic priority task is moved into the running queue.

[0028] When the number of tasks in the running queue changes, the convex optimization problem is solved by using the current dynamic priority of each task as the weight, and the server computing resources are dynamically reallocated.

[0029] The central coordination unit is used to model the problem of maximizing the long-term average task value of the system as a partially observable Markov decision process, and to configure an independent agent for each terminal device. The agents share the centralized value network and adopt a decoupled multi-agent deep reinforcement learning algorithm to collaboratively optimize the transmission power and offloading ratio decision under the guidance of sparse-dense hybrid reward, so that high-value and high-urgency tasks are completed first under resource-constrained conditions.

[0030] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the value maximization computation offloading method based on dynamic priority scheduling and reinforcement learning.

[0031] The beneficial effects of this invention are as follows: This invention constructs an optimization framework with the maximization of the system's "long-term average task value" as its core objective. By introducing an exponential value function to define task value, it addresses the "value neglect" problem inherent in traditional "average latency or energy consumption optimization" strategies. This value-oriented optimization mechanism enables multi-agent systems to autonomously learn strategies during training to prioritize the execution of high-value tasks when resources are limited, even if it may require sacrificing the performance of some low-value tasks to some extent. This perfectly aligns with the actual needs of industrial Internet of Things (IIoT) applications with extremely high security requirements.

[0032] Furthermore, this invention proposes a dynamic priority function that can accurately measure the actual urgency level of a task at different times, thus overcoming the shortcomings of the "static priority" model in dealing with real-time changes in urgency. It is particularly noteworthy that this dynamic priority method is not only used for task selection in the waiting queue, but is also innovatively introduced into the dynamic resource allocation mechanism of the running queue.

[0033] Finally, this invention also employs a decoupled multi-agent deep reinforcement learning structure that integrates sparse reward signals (based on the value of completed tasks) and dense reward signals (including the value of uncompleted tasks and urgency penalties) to form a hybrid reward guidance mechanism. This mechanism effectively promotes the agent's adaptive learning to the high-performance dynamic scheduler, improving training convergence speed while optimizing the overall system efficiency. Attached Figure Description

[0034] Figure 1 This is a block diagram illustrating the principle of MEC network system architecture.

[0035] Figure 2 This is an overall flowchart of the method of the present invention;

[0036] Figure 3 A flowchart of the edge computing model workflow;

[0037] Figure 4 This is a schematic diagram of the algorithm framework workflow. Detailed Implementation

[0038] The present application will be further described below with reference to the accompanying drawings.

[0039] The purpose of this invention is to: (1) establish an optimization framework aimed at maximizing the long-term average task value of the system, replacing the traditional paradigm of minimizing average latency or energy consumption, so that the agent learns to prioritize high-value tasks when resources are limited. (2) design a dynamic priority scheduling mechanism that can simultaneously perceive the "inherent importance" and "real-time urgency" of tasks, replacing FIFO or static priority scheduling, to ensure that truly critical and urgent tasks are given priority. (3) propose an innovative decoupled offloading and scheduling framework, in which the MADRL agent (device side) focuses on learning "partial offloading strategies" to actively adapt to and utilize a high-performance dynamic priority scheduler that runs independently on the edge server, thereby achieving significant advantages in ensuring the QoS of high-value tasks and improving the overall utility of the system.

[0040] Based on the above objectives, this application provides the following embodiments:

[0041] like Figure 1 The diagram illustrates a base station-centric multi-access edge computing (MEC) network. This system comprises one base station, one MEC edge server, and M terminal devices. The base station serves as the core wireless access point, with all terminal devices connected to it via their respective wireless communication links. The base station manages and relays all wireless communications. The MEC edge server is physically deployed near the base station, providing computing and storage capabilities at the network edge. The M terminal devices are MADRL agents within the system, connected to the base station via their respective wireless communication links, performing computing tasks or requesting network services.

[0042] In this application, computation offloading refers to the terminal device transmitting tasks to one or more nearby edge nodes (remote cloud nodes can also be added), and completing task processing with the help of the edge nodes, so as to solve the shortcomings of the device in terms of resource storage, computing performance and energy efficiency.

[0043] Based on the above system, the overall flowchart of the uninstallation method of this application is as follows: Figure 2 As shown, the main steps include the following:

[0044] Step 1: Based on the partial offloading scenario of a multi-device single edge server, construct the edge computing model and the local computing model.

[0045] Step 2: Based on the model described in Step 1, set the objective function and constraints for the optimization problem.

[0046] Step 3: Model the optimization problem as a partially observable Markov decision process, and set the observation space, state space, action space and reward function in multi-agent deep reinforcement learning.

[0047] Step 4: Use a multi-agent deep reinforcement learning method to find the optimal offloading strategy for each device and optimize the objective function.

[0048] In some embodiments, step 1: Based on the partial offloading scenario of a multi-device single edge server, construct an edge computing model and a local computing model, specifically:

[0049] First, based on M terminal devices (intelligent agents) A basic model is constructed based on a system consisting of one base station and an MEC edge server. This system is implemented in a discrete time-slot model. Running below, the duration of each time slot is Second.

[0050] Step 1.1: Construct the task model

[0051] In each time slot At the beginning, each terminal device Each generates N new computational tasks, forming a task set. }, The nth task generated by device m in time slot t is defined by a tuple:

[0052] (1)

[0053] in, Indicates the size of the task data (unit: bits). This indicates how many CPU cycles are required for a task to process 1 bit of data. This indicates the maximum tolerable time for the task (in seconds). The task's final delay must not exceed this time; otherwise, the task is considered a failure. This represents the initial static priority of the task; the larger the value, the higher the priority.

[0054] Step 1.2: Communication and Task Transfer Model

[0055] Suppose the total uplink bandwidth of the base station is W (unit: Hz), which can be evenly distributed to each device using Frequency Division Multiple Access (FDMA) technology. Considering that the devices can dynamically adjust their transmission power, we use... This represents the transmission power of device m within time slot t, where This is the maximum transmission power of the device. Within time slot t, according to Shannon's formula, the transmission rate of device m can be expressed as:

[0056] (2)

[0057] in It is the channel gain from device m to the base station, which varies independently between different time slots; This represents the noise power spectral density.

[0058] Further, define For the unloading decision of device m, where Indicates task The offloading ratio is used to divide the task into edge computing subtasks and local computing subtasks. In time slot t, there is an upper limit to the total amount of data that device m can transmit. Edge subtasks are constrained by this capacity during transmission. This embodiment adopts a priority-based transmission sorting mechanism: device m first attempts to transmit its offloaded data in descending order of priority among its N tasks. High-priority tasks occupy transmission capacity first. If capacity is insufficient, the actual offloaded data amount of low-priority tasks will be reduced. Task transmission latency... Depending on its order in the transmission queue and the actual amount of data transmitted, its calculation formula is as follows:

[0059] (3)

[0060] in This indicates the total amount of data that needs to be transferred before the task can be transmitted. This indicates the amount of data to be transmitted for this task.

[0061] Step 1.3: Construct the dynamic priority function

[0062] This embodiment provides an edge computing task scheduling method, wherein a dynamic priority function is constructed to achieve a comprehensive balance between task importance and timeliness. As shown in equation (4).

[0063] (4)

[0064] in Represents the static priority of the task. This represents the maximum static priority value defined by the system. Represents the current global timestamp of the system. This represents the global timestamp of the task arriving in the system. This represents a static priority factor, reflecting the inherent importance of the task; This represents the dynamic urgency factor, reflecting the timeliness requirements of the task. During scheduling, at the beginning of each scheduling cycle, the system obtains the current global timestamp, iterates through each task in the waiting queue, and calculates or updates the dynamic priority of each task according to the formula mentioned above.

[0065] Unlike existing technologies, the weighting coefficients in this embodiment... It is no longer a preset fixed value, but a dynamic adaptive weighting coefficient, with a value between [0, 1]. Furthermore, the calculation of this coefficient comprehensively considers the waiting queue. Average importance of all tasks Compared with average urgency And adaptively adjust based on their relative sizes, the specific calculation process is as follows:

[0066] (5)

[0067] (6)

[0068] (7)

[0069] This ensured The value can reflect the overall status of the queue in real time: when high-value tasks are piling up in the queue ( When (dominant), It will approach 1; while when there is a backlog of high-urgency tasks in the queue ( When (dominant), It will approach 0. If the queue is empty (K=0) or and If both are 0, then It is given a balanced default value to avoid the denominator being zero.

[0070] Finally, the scheduler will use this dynamically calculated... Substitute the values ​​into formula (4) to calculate the final dynamic priority for each task in the waiting queue, and then perform subsequent scheduling operations accordingly.

[0071] Step 1.4: Constructing an edge computing model

[0072] Based on the dynamic priority function described in step 1.3, an edge computing model is constructed. The detailed workflow of this model is as follows: Figure 3 As shown.

[0073] When the edge server receives an edge computing subtask from a terminal device, it first records its precise arrival time. And immediately place the task in the "waiting queue". This puts the system into a waiting-for-scheduling state. Meanwhile, the edge server continuously maintains a "running queue". It is used to store tasks that have been scheduled and are currently being executed while consuming computing resources.

[0074] Next, the scheduling trigger and task selection process will be executed. This process is not continuous, but is activated by specific pre-defined events, such as the "run queue". The scheduler is activated immediately once either a task completes and releases resources, or the number of tasks in the "run queue" is less than the system's maximum concurrency threshold, is reached. This activates the scheduler to iterate through the "wait queue." All tasks within it.

[0075] During this process, the scheduler calculates the current priority value of each waiting task in real time based on the dynamic priority function defined in step 1.3. Finally, the scheduler selects the task with the highest dynamic priority, removes it from the waiting queue, and adds it to the run queue. Simultaneously, the system records the task's "start execution time." Based on this, the waiting delay of its edge subtasks is calculated. .

[0076] (8)

[0077] Furthermore, when the scheduler is activated (i.e., the run queue) At the same time as a task change, the system will simultaneously initiate a dynamic resource reallocation mechanism. In this mechanism, the system first iterates through the updated run queue. All tasks are identified, and their dynamic priorities at the start of the resource allocation are obtained. This priority value is then used as the weight for resource allocation. Subsequently, the system determines the optimal resource allocation ratio for each running task by solving a pre-defined convex optimization problem. Where the subscript j represents the index number of the task in the run queue. Finally, based on the resource allocation ratio set obtained from the solution... Allocate computing resources to each task in the run queue.

[0078] After receiving resource allocation, the task enters the execution and delayed computation phase. Each edge computing subtask... Each has a total computational load determined by its data volume and the number of CPU cycles required. The execution of a task is a cumulative process; the computing power it acquires at any given moment equals the total capacity of the server. Its segmented constant resource ratio The product of the cumulative computational cost. When the cumulative computational cost reaches its required total computational cost. When the task is completed, it is considered finished. At this point, the system records its "global completion time". It also calculates the "execution latency" spent in the run queue. Since the amount of data resulting from the completed task is relatively small and the downlink bandwidth is sufficiently large, the latency of returning the task results from the edge server is ignored. Therefore, the total latency of this edge computing subtask is... It is calculated as the sum of transmission delay, waiting delay, and execution delay.

[0079] (9)

[0080] (10)

[0081] At the system level, this application embodiment also defines the total remaining computing power of the edge server. The evolution model. In each discrete time slot t (duration is...) At the end of the second, update the remaining computing power in the edge server:

[0082] first, Subtract the maximum service volume that the server can provide within that time slot ( ), and ensure the result is not less than zero; then, add the total computational cost of all new tasks arriving in that time slot. In this way, the total remaining computational load at the start of the next time slot t+1 can be obtained. That is, the total remaining computational load at the end of time slot t, whose evolution process can be formalized as follows:

[0083] (11)

[0084] Step 1.5: Build a local computing model

[0085] Task The portion of the computation not unloaded to the edge will be used as a local computation subtask. Execute on terminal device m:

[0086] First, the system makes an uninstallation decision. Calculate the total computational cost required for this local subtask. Place the local computation subtask in the local waiting queue. .

[0087] Then, terminal device m employs a non-preemptive serial strategy based on dynamic priority, calculating the dynamic priority of each waiting task according to the dynamic priority function defined in step 1.3. The task with the highest dynamic priority... It will be selected and removed from the queue, and then execution will begin on the local CPU. Its start time is recorded as follows. Based on this, the local waiting delay of this task... Defined as the time from task generation to the time its local execution begins. Time elapsed. Local wait delay. Local execution delay Local subtasks Total end-to-end delay The calculation formulas are as follows:

[0088] (12)

[0089] (13)

[0090] (14)

[0091] Finally, in each discrete time slot t (duration is...) At the end of the second, update the remaining total computation time in the local queue. During time slot t, the sum of newly generated local compute loads is defined as follows: The total remaining computational load of the local queue at the start of the next time slot t+1 Its update formula is as follows:

[0092] (15)

[0093] In some embodiments, step 2: Based on the model described in step 1, the objective function and constraints of the optimization problem are set, specifically:

[0094] Based on the model described in step 1, the task It adopts a cooperative parallel processing model and is divided into local subtasks. and edge subtasks Therefore, the task Total latency from generation to final completion (both local and edge-based) It depends on the maximum of the delays of the two subtasks:

[0095] (16)

[0096] Then the calculation task Value that can be obtained after execution .

[0097] (17)

[0098] Based on this, the objective function of the optimization problem is set as follows: under the condition of satisfying the task delay constraint, maximize the "long-term average task value" of the system, as shown in Equation (18):

[0099] (18)

[0100] The solution process for the above optimization objective is subject to three key constraints: First, there is a power constraint (C1), which specifies the transmission power of any device m in any time slot t. Must be within its permitted range Within this range, the power cannot be negative, nor can it exceed the device's maximum transmit power. Secondly, there is the offload ratio constraint (C2), which specifies the offload ratio for any task. The execution must be within the range [0, 1], where 0 corresponds to fully local execution and 1 corresponds to fully edge unloading. Thirdly, there is a delay constraint (C3): the total delay for task completion should not exceed the task's maximum tolerable time.

[0101] Therefore, the ultimate goal of the optimization problem is to find an optimal joint strategy that includes power ratio and offloading ratio, which maximizes the long-term average task value of the system while strictly satisfying the above constraints.

[0102] In some embodiments, step 3: modeling the optimization problem as a partially observable Markov decision process, and setting the observation space, state space, action space, and reward function in multi-agent deep reinforcement learning, specifically:

[0103] Based on the optimization problem and constraints described in step 2, this problem is modeled as a partially observable Markov decision process. In this model, each terminal device is first treated as an independent agent. Therefore, the original problem (P) is transformed into a multi-agent deep reinforcement learning (MADRL) problem, in which M agents (i.e., M terminal devices) interact with the environment in each time slot to collaboratively maximize the objective function defined in step 2.

[0104] Under partially observable settings, each agent cannot obtain the complete global state. It can only obtain its own local observations. In time slot t, the agent m's local observations... It can be expressed in the form of formula (16), mainly including: the total remaining computational load of the edge queue fed back from the edge server. Its own local queue remaining computation amount Its own channel gain Information on the N newly generated tasks.

[0105] (19)

[0106] Therefore, the global state can be defined as the joint observation of all agents, i.e. }

[0107] Next, the action space of agent m is defined. In each time slot t, agent m, based on its local observations... Generate an action This action It is a composite action, corresponding to the decision variable in step 2, specifically including: (1) the transmission power set for itself. (1) The action is subject to a power constraint (C1); (2) The unloading ratio is set for each of its N tasks, and the action is subject to an unloading ratio constraint (C2).

[0108] (20)

[0109] Finally, the reward function is set. To ensure that agent m's local decisions are guided to the global optimization objective defined in step 2 (i.e., maximizing the long-run average task value), the reward of agent m in time slot t is... It consists of two parts:

[0110] One is the cumulative task value of the tasks completed by the agent within the time slot t. These completed tasks may include tasks that the agent makes decisions before time slot t and completes in the current time slot t, or tasks that the agent makes decisions in the current time slot t and completes in the current time slot t. Therefore, this is a sparse and delayed reward, and its calculation formula is shown in formula (21). This represents the set of tasks completed by agent m within time slot t.

[0111] (twenty one)

[0112] Secondly, there are dense reward signals constructed based on task value and urgency. The specific calculation method is as follows:

[0113] (twenty two)

[0114] in This represents the set of tasks that agent m has not yet completed at the end of time slot t. U represents the tasks in the set of unfinished tasks, and the value calculation formula for unfinished tasks is shown in equation (17). Indicates that the task was not completed. The urgency level. By calculating such intensive rewards, the agent can prioritize tasks that are both high-value and high-urgency, which aligns with the optimization direction of step 2.

[0115] Finally, by combining sparse and dense rewards, the reward obtained by agent m in time slot t can be obtained through equation (23). for:

[0116] (twenty three)

[0117] This application accelerates convergence by combining sparse and dense rewards. By maximizing the long-term cumulative expectation of the local reward, the agent m will learn to autonomously optimize its power and offloading strategies while satisfying constraints, thereby collaboratively achieving the global optimization goal.

[0118] In some embodiments, step 4: using a multi-agent deep reinforcement learning method to find the optimal offloading and power strategies for each device, and optimizing the objective function, specifically:

[0119] To solve the optimization problem defined in step 2, this embodiment constructs a decoupled multi-agent deep reinforcement learning framework based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, aiming to maximize the long-term average task value. Specifically, each agent (i.e., the terminal device) is equipped with an independent decision network (Actor network), while all agents share the same centralized value network (Critic network). The agents focus solely on learning offload and power policies, and learn how to proactively adapt to an independently running high-performance scheduler through observation and rewards, ultimately finding the optimal offload and power policies to optimize the objective function in step 2. The workflow is as follows: Figure 4 As shown.

[0120] Step 4.1: Decision Generation and Action Execution

[0121] At the start of each decision time slot t, firstly, each agent m acquires its local observations. (As defined in step 3). Next, agent m will use this observation... The input is fed into its local policy network (i.e., the Actor network). This network then outputs a probability distribution representing the optimal action policy, and the agent m generates specific actions based on this distribution. Finally, after obtaining the actions of all agents, a joint action is executed. .

[0122] (twenty four)

[0123] Step 4.2: Collection and Storage of Experience Data

[0124] In joint operations After execution, the environment undergoes a state transition. Each agent m first receives its reward for time slot t. At the same time, new local observations were obtained. Subsequently, the observations of all agents are combined into a global state. and The global reward is calculated using formula (25). This reward will be used to guide the training of the agent.

[0125] (25)

[0126] Finally, the empirical tuples Store it in a global rollout buffer. This is for use in subsequent network updates.

[0127] Step 4.3: Network Training

[0128] During the update, the system from A mini-batch of empirical tuples is randomly sampled. First, a Critic network is used. Calculate time difference error :

[0129] (26)

[0130] in It is the global reward obtained by the system in time slot t. It is the global state. This is the discount factor. Then, the Generalized Advantage Estimation (GAE) method is used to calculate a global advantage function shared by all agents. and its target value :

[0131] (27)

[0132] (28)

[0133] in This is the smoothing parameter of GAE, and L is the length of the sampling trajectory.

[0134] Then update the network parameters. Crtic network. The value is updated by minimizing a value function loss to evaluate its worth. Approaching the target value .

[0135] (29)

[0136] For the Actor network of the agent, first calculate the policy probability ratio. Then, by maximizing a truncated objective function To update parameters .

[0137] (30)

[0138] in It is a preset truncation hyperparameter used to limit the magnitude of policy updates.

[0139] Finally, a gradient update method is used to optimize and update the parameters of the centralized Critic network and all decentralized Actor networks respectively. By continuously iterating through steps 4.1 to 4.3, the Actor networks of all agents will gradually converge, eventually finding a near-optimal joint policy for the system, thereby achieving the optimization of the global optimization objective in step 2.

[0140] In summary, the method embodiments of this application propose a dynamic priority quantification method that integrates the "inherent importance" and "real-time urgency" of tasks, and includes an adaptive weight adjustment mechanism. It innovatively establishes an edge computing model based on a dual scheduling and resource reallocation mechanism of dynamic priority. This model not only uses dynamic priority to select tasks in the waiting queue, but more importantly, it triggers dynamic resource reallocation when the state of the running queue changes: it recalculates the dynamic priority of all running tasks in real time, and achieves real-time reallocation of computing resources through convex optimization, ensuring that computing resources are always preferentially allocated to the most urgent tasks.

[0141] Furthermore, a hybrid signaling mechanism combining sparse rewards (based on the cumulative value of completed tasks) and dense rewards (based on the value and urgency assessment of uncompleted tasks) is proposed to effectively guide agents to prioritize high-value, high-urgency tasks, significantly improving learning efficiency. Based on the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, a decoupled learning framework is established with the goal of maximizing the long-term average task value. By defining task value through an exponential function, the optimization objective fundamentally shifts to "prioritizing high-value tasks." In this framework, the agent (terminal device) focuses on learning offload and power control strategies, autonomously learning how to adapt to an independently operating high-performance dynamic scheduler by observing environmental states and receiving hybrid reward signals.

[0142] Based on the same concept as the above method, embodiments of this application also provide a value maximization computational offloading system based on dynamic priority scheduling and reinforcement learning, including:

[0143] M terminal devices are used to generate multiple computing tasks in each time slot. Each task carries the amount of data, CPU cycle requirements, maximum tolerable latency and initial static priority, and independently decides the transmission power and task offloading ratio based on local observation.

[0144] A base station is used to provide uplink access to the terminal device via frequency division multiple access.

[0145] Edge servers, co-located with the base station, are used for:

[0146] Receive and cache edge subtasks unloaded by terminal devices, and maintain waiting queues and running queues;

[0147] Tasks in the waiting queue are scheduled based on a dynamic priority function, and the highest dynamic priority task is moved into the running queue.

[0148] When the number of tasks in the running queue changes, the convex optimization problem is solved by using the current dynamic priority of each task as the weight, and the server computing resources are dynamically reallocated.

[0149] The central coordination unit is used to model the problem of maximizing the long-term average task value of the system as a partially observable Markov decision process, and to configure an independent agent for each terminal device. The agents share the centralized value network and adopt a decoupled multi-agent deep reinforcement learning algorithm to collaboratively optimize the transmission power and offloading ratio decision under the guidance of sparse-dense hybrid reward, so that high-value and high-urgency tasks are completed first under resource-constrained conditions.

[0150] This application also provides a computer device. At the hardware level, the device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it. Of course, besides software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0151] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a value maximization computational offloading method based on dynamic priority scheduling and reinforcement learning as shown in any of the foregoing embodiments.

[0152] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0153] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A value maximization computation offloading method based on dynamic priority scheduling and reinforcement learning, applied to an industrial Internet of Things edge computing scene, characterized in that The method includes the following steps: S1. Construct a task model, a communication and task transmission model, an edge computing model, and a local computing model, wherein: The task model is used to characterize the multiple computing tasks generated by the terminal device in each time slot; The communication and task transmission model is used to calculate the transmission rate from the terminal device to the edge server and to determine the transmission order based on the static priority of the task. The edge computing model is used to maintain a waiting queue and a running queue on the edge server side, schedule tasks in the waiting queue based on a dynamic priority function, and solve a convex optimization problem based on the dynamic priority weight of each task in the running queue to dynamically reallocate computing resources. The local computing model is used to perform non-preemptive serial execution of the unloaded portion of tasks on the terminal device side based on the dynamic priority function. S2. Define an optimization problem with the goal of maximizing the long-term average task value of the system, and constrain the total task delay to not exceed its maximum tolerable delay, the transmission power to not exceed the maximum transmit power of the device, and the offloading ratio to be between 0 and 1; S3. The optimization problem is modeled as a partially observable Markov decision process. An independent agent is defined for each terminal device. An observation space is set up, which includes the remaining computational cost of the edge queue, the remaining computational cost of the local queue, the channel gain and the current task information. A composite action space is set up, which includes the transmission power and the task offloading ratio. A hybrid reward function is set up, which integrates the sparse reward of the value of the completed task and the dense reward of the value of the uncompleted task and the urgency penalty. S4. A decoupled multi-agent deep reinforcement learning algorithm is adopted, which enables each agent to independently decide on transmission power and offloading ratio, and share a centralized value network. Through sparse-dense hybrid rewards, agents are guided to prioritize the service quality of high-value and high-urgency tasks in a resource-constrained environment, thereby solving the optimization problem and maximizing the long-term average task value of the system. The weight coefficient a in the dynamic priority function is calculated in each scheduling period by the average static importance of all tasks in the waiting queue with the average urgency is calculated in real time and The dynamic priority function is expressed as: wherein a static priority of the task, a maximum static priority value defined by the system, a global timestamp of the system, a global timestamp of the system, a global timestamp of the system, a global timestamp of the system; In the hybrid reward function, sparse rewards are accumulated by an exponential value function to sum the value of completed tasks, while dense rewards negatively penalize uncompleted tasks. The exponential value function defines the task value.

2. The method according to claim 1, characterized in that, Each agent employs a multi-agent proximal policy optimization algorithm. The Actor network of each agent independently outputs the transmission power and offload ratio vectors, while the Critic network shares and outputs the global state value, which is used to calculate the advantage function and policy gradient update.

3. The method according to claim 2, characterized in that, The total task latency is the maximum of the local computing latency and the edge computing latency. The edge computing latency includes the transmission latency based on priority sorting, the waiting queue scheduling latency, and the execution latency after dynamic resource allocation.

4. The method according to claim 1, characterized in that, The uplink between the terminal device and the edge server uses FDMA to evenly distribute bandwidth, and the transmission rate is calculated by Shannon's formula. When transmission resources are limited, scheduling is performed according to the static priority of tasks to ensure that high-priority tasks are transmitted first.

5. The method according to any one of claims 2 to 4, characterized in that, Remaining computing power on edge servers Remaining computational load in the local queue The following equation is used to evolve at the end of each time slot, and is then used as part of the agent's observations in the next time slot: in For the computing power of device m, For discrete time slots Duration For discrete time slots The total computational cost of all newly arrived tasks. For discrete time slots The sum of newly generated local computing loads.

6. A value maximization computational offloading system based on dynamic priority scheduling and reinforcement learning, applied to industrial IoT edge computing scenarios, characterized in that... include: M terminal devices are used to generate multiple computing tasks in each time slot. Each task carries the amount of data, CPU cycle requirements, maximum tolerable latency and initial static priority, and independently decides the transmission power and task offloading ratio based on local observation. A base station is used to provide uplink access to the terminal device via frequency division multiple access. Edge servers, co-located with the base station, are used for: Receive and cache edge subtasks unloaded by terminal devices, and maintain waiting queues and running queues; Tasks in the waiting queue are scheduled based on a dynamic priority function, and the highest dynamic priority task is moved into the running queue. When the number of tasks in the running queue changes, the convex optimization problem is solved with the current dynamic priority of each task as the weight, and the server computing resources are dynamically reallocated. An optimization problem is set with the goal of maximizing the long-term average task value of the system, and the total task latency is constrained to not exceed its maximum tolerable latency, the transmission power does not exceed the maximum transmission power of the device, and the offloading ratio is between 0 and 1. The central coordination unit is used to model the problem of maximizing the long-term average task value of the system as a partially observable Markov decision process, and to configure an independent agent for each terminal device. The agents share the centralized value network and adopt a decoupled multi-agent deep reinforcement learning algorithm to collaboratively optimize the transmission power and offloading ratio decision under the guidance of sparse-dense hybrid reward, so that high-value and high-urgency tasks are completed first under resource-constrained conditions. The weighting coefficient α in the dynamic priority function is determined by the average static importance of all tasks in the waiting queue during each scheduling cycle. Compared with average urgency Calculated in real time, and The dynamic priority function Expressed as: in Representative task Static priority, This represents the maximum static priority value defined by the system. Represents the current global timestamp of the system. Representative task Reaching the system's global timestamp; In the hybrid reward function, sparse rewards are accumulated by an exponential value function to sum the value of completed tasks, while dense rewards negatively penalize uncompleted tasks. The exponential value function defines the task value.

7. A computer device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the value maximization computational offloading method based on dynamic priority scheduling and reinforcement learning as described in any one of claims 1 to 5.

Images