An edge computing-based hierarchical reinforcement learning task partial offloading method, system and storage medium based on adaptive constraints

Abstract

The application discloses a layered reinforcement learning task partial offloading method and system based on adaptive constraints under edge computing and a storage medium, belongs to the technical field of computer communication and artificial intelligence, and solves the problems of high coupling degree of a mixed action space, training shock and convergence difficulty caused by the prior art, and poor forward-looking of an existing action constraint mechanism and difficulty in effectively inhibiting invalid action selection. The application formally defines a task partial offloading process in an edge collaborative environment, constructs a trend prediction adaptive mask mechanism for invalid action constraints, generates a dynamic actionable action set according to a node state change trend, constructs a collaborative task partial offloading algorithm based on layered SAC, and trains in combination with the above steps. The model obtained after training completes selection of a task offloading node and allocation of an offloading ratio, and realizes collaborative optimization decision-making in the task partial offloading process. The application is used for layered reinforcement learning task partial offloading.

Description

Technical Field

[0001] A method, system, and storage medium for partially offloading hierarchical reinforcement learning tasks based on adaptive constraints under edge computing are disclosed. This method is used for partially offloading hierarchical reinforcement learning tasks and belongs to the field of computer communication and artificial intelligence technology. Specifically, it relates to a method for partially offloading tasks in an edge computing environment. Background Technology

[0002] With the development of information and communication technologies, computing task processing models are gradually evolving from centralized cloud computing to edge computing. Traditional cloud computing, relying on centralized resource pools, can provide sufficient computing power for large-scale data processing. However, with the rapid growth in the number of terminals and the continuous improvement in the real-time requirements of services, the link overhead and response latency caused by long-distance data transmission are becoming increasingly prominent, making it difficult to meet the actual needs of low-latency and high-reliability scenarios. To address this, edge computing provides a new path for task processing by moving computing resources closer to the terminal. In an edge computing environment, compared to completely local execution or overall offloading of tasks, partial task offloading can divide the same task into local execution and remote execution parts, thereby making more flexible use of heterogeneous resources on the terminal and edge sides. Therefore, it has high application value in computing-intensive and latency-sensitive services. At the same time, reinforcement learning methods, due to their strong environmental interaction and sequential decision-making capabilities, have been gradually used to solve task offloading problems, especially suitable for online optimization in dynamically changing scenarios. However, during the partial unloading of tasks, it is often necessary to determine the unloading node and the unloading ratio at the same time. The decision variables are highly coupled, and in complex dynamic environments, problems such as task allocation imbalance, invalid action selection, and congestion response lag are prone to occur. Therefore, a more efficient and stable partial unloading method is urgently needed.

[0003] In edge computing scenarios involving partial task offloading, reinforcement learning has gradually become an important technical approach for achieving dynamic offloading decisions. Existing technologies typically model the partial task offloading process as a Markov decision process, enabling agents to jointly decide on the offloading location and proportion based on the environmental state, and continuously optimize the strategy through ongoing interaction with the environment. Several representative technical paths have emerged for this type of problem. For example, policy optimization-based reinforcement learning methods can improve task offloading performance through iterative policy updates; methods based on the Actor-Critic architecture combine policy networks and value networks to evaluate and optimize decision-making behavior during partial task offloading; in edge computing scenarios involving multi-node collaboration, multi-agent reinforcement learning methods are also used to characterize the collaborative behavior among multiple computing agents; furthermore, considering the interrelationship between node selection and proportion allocation and the high decision complexity in partial task offloading, hierarchical reinforcement learning methods are also widely used, enhancing the modeling capability of the partial task offloading process by breaking down the complex decision-making process into different levels for separate processing. Meanwhile, as the task partial offloading problem increasingly exhibits stronger hybrid decision-making characteristics, reinforcement learning methods based on SAC have also been applied to related research, leveraging their maximum entropy mechanism and Actor-Critic structure to enhance the adaptability and stability of policy learning. Overall, the application of reinforcement learning in task partial offloading has evolved from early single-policy optimization to a comprehensive technical approach integrating policy learning, value assessment, multi-agent collaboration, and hierarchical decision-making.

[0004] Existing task partial unloading methods based on reinforcement learning typically require simultaneous decisions on both unloading node selection and unloading ratio allocation, making them a typical hybrid action space optimization problem. While these methods can achieve a certain degree of adaptive decision-making in dynamic edge computing environments, they still suffer from the following technical limitations in practical applications, hindering further improvements in task partial unloading performance:

[0005] 1. High spatial coupling in hybrid actions easily leads to training oscillations and convergence difficulties: During task unloading, the selection of unloading nodes and the allocation of unloading ratios are not independent but strongly coupled. The node selection result directly affects task transmission latency, queuing status, and remote execution overhead, while the allocation of unloading ratios further affects the load balance between the local and remote parts and the task completion efficiency. Existing methods often struggle to balance the synergy between discrete and continuous decisions when dealing with this type of hybrid action decision-making problem, easily leading to mutual interference in policy updates, significant training oscillations, and slow convergence. In complex dynamic environments, the strong coupling of decisions may also cause the system to get stuck in local optima, resulting in unstable task unloading effects and making it difficult to fully utilize the advantages of edge collaborative computing resources.

[0006] 2. Existing action constraint mechanisms lack foresight and struggle to effectively suppress ineffective action selection: In edge computing environments, node load, queue occupancy, and resource availability are typically constantly changing. During task offloading decisions, situations such as target nodes becoming unavailable, impending congestion, or increased execution risk can easily arise. Existing technologies typically employ action constraints based on the current state, masking or restricting actions that are no longer feasible. While this approach can reduce the participation of ineffective actions in decision-making to some extent, it is largely a reactive mechanism, lacking the ability to anticipate changes in node state trends. When a node has not yet reached an explicitly unavailable state but is already showing a clear congestion trend, existing methods may still include it in the candidate action set, leading to an increased probability of ineffective action selection. This not only affects the quality of training samples but also weakens the execution efficiency and stability during task offloading. Summary of the Invention

[0007] To address the problems mentioned above, the present invention aims to provide a method, system, and storage medium for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints under edge computing. This solves the problems of high coupling in the mixed action space, which easily leads to training oscillations and convergence difficulties in existing technologies; and the lack of foresight in existing action constraint mechanisms, which makes it difficult to effectively suppress invalid action selection.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment includes the following steps:

[0010] Step 1: Formalize the task offloading process in the edge collaborative environment, including the construction of the system model, MDP model, and reward function;

[0011] Step 2: Construct a trend prediction adaptive masking mechanism for invalid action constraints, and generate a dynamic set of actionable actions based on the trend of node state changes;

[0012] Step 3: Construct a collaborative task partial unloading algorithm based on hierarchical SAC, and train it in conjunction with Step 1 and Step 2. Based on the trained model, complete the selection of task unloading nodes and the allocation of unloading ratio, and realize collaborative optimization decision-making in the task partial unloading problem.

[0013] Furthermore, the specific steps of step 1 are as follows:

[0014] Step 1.1, System Model Construction:

[0015] Step 1.11: Based on a multi-access edge computing environment, the system consists of a mobile terminal set, an edge server set, and a cloud center. The system operates in discrete time slots, denoted as [time slot number missing]. For computing tasks generated by the terminal, the first... Task Represented as:

[0016] (1)

[0017] in, Indicates the generation of the first Task The source terminal device number, Indicates the first The amount of input data for each task Indicates the first The computational requirements for each task and They represent the first The arrival and deadline times for each task;

[0018] In the task-part unloading scenario, define discrete actions. Indicates task In the time slot Select the remote destination node and define an indicator variable. Indicates the first Does the task select edge nodes? As a remote execution location, i.e. when hour, ,otherwise At the same time, define continuous proportional variables Indicates task In the time slot The calculated unloading ratio is the percentage of data that is unloaded to the remote destination node, then the remaining percentage is... The computational load remains to be executed locally, and the local and remote parts of the task can be processed in parallel. The task is considered to have ended when both parts are completed.

[0019] Step 1.12, Transmission Model Construction:

[0020] No uplink transfer occurs when the task is executed locally;

[0021] When the remote portion of a task is offloaded to an edge node or the cloud, its input data needs to be transmitted via the uplink, numbered as follows: The source terminal equipment in the time slot To edge nodes The uplink instantaneous transmission rate is expressed using Shannon's formula as:

[0022] (2)

[0023] in, For the number Source terminal devices and edge nodes Link bandwidth between For the number The source terminal equipment in the time slot The transmission power, For the number The source terminal equipment in the time slot The equivalent channel gain, For noise power spectral density, For time slots To mitigate interference power, time slots are used for transmission links from the terminal to the cloud. equivalent constant rate This indicates its transmission capacity;

[0024] Due to the mission The distal part is proportional Unloaded, therefore, in the time slot The amount of data that needs to be uploaded is Therefore, the task The upload latency at the remote end is represented as:

[0025] (3)

[0026] in, This indicates that the remote portion is unloaded to the edge node. This indicates that the remote portion is offloaded to the cloud;

[0027] Step 1.13, Computational Model Construction:

[0028] edge nodes In the time slot The computing power is denoted as To characterize the fluctuations in node computing power over time, it is expressed as:

[0029] (4)

[0030] in, and Representing edge nodes respectively Minimum and maximum computing power, Indicates time slot The random perturbation term of the next node's computing power;

[0031] After proportionally dividing the task computation, the local portion is numbered as follows: The computation latency on the edge node corresponding to the source terminal device is:

[0032] (5)

[0033] in, Indicates the number is The source terminal equipment in the time slot The computing power, the remote part is at the edge node The computation delay is:

[0034] (6)

[0035] Step 1.14, Building the queuing model:

[0036] Using a first-come, first-served queue service method, let... Indicates time slot Internal mission At the edge node If the remaining computational workload of the preceding task that needs to wait is given, then its queuing delay can be expressed as the ratio of the remaining computational workload of the preceding task to the node's computing power.

[0037] Therefore, the queuing delays for the local portion and the remote portion are respectively:

[0038] (7)

[0039] (8)

[0040] Step 1.15, End-to-end delay:

[0041] Based on the above transmission model, calculation model, and queuing model, the total local latency is:

[0042] (9)

[0043] When the remote part of the task is proportional Unload to edge node When the total delay of its far end is , it can be written as:

[0044] (10)

[0045] When the remote part of the task is proportional When unloading to the cloud, the total latency of its remote portion can be written as:

[0046] (11)

[0047] in, Proportionately for the remote part of the task Similarly, the transmission latency when unloading to the cloud... To delay queuing, To calculate latency, since the local and remote parts of the task execute in parallel, the task... The end-to-end completion delay is defined as:

[0048] (12)

[0049] in, This indicates the total latency of the local execution portion of the task. This represents the total latency when the remote part of the task is executed on the edge node. This represents the total latency when the remote portion of a task is executed in the cloud.

[0050] Step 1.2: Construct the comprehensive cost function based on Step 1.1:

[0051] Step 1.21: Define the task based on the relationship between the task completion delay and the task deadline. In the time slot The timeout indicator function is:

[0052] (13)

[0053] in, For time slots Task The end-to-end completion delay, Indicates task The task is considered to have timed out if the completion time exceeds its allowed execution time limit; otherwise, the task is considered not to have timed out.

[0054] Step 1.22, Define the task In the time slot The discard indicator function is:

[0055] (14)

[0056] Specifically, when a task's local or remote portion triggers a capacity constraint during enqueuing, preventing the task from being accepted by the corresponding node, record... Otherwise, remember ;

[0057] To characterize the degree of collaboration between the local and remote execution parts of a task, a task is defined. In the time slot The differences in execution at the endpoints are as follows:

[0058] (15)

[0059] Based on the above definition, construct the task. In the time slot The instantaneous synthesis cost function is:

[0060] (16)

[0061] in, These are weighting coefficients, used to measure the relative importance of task completion latency, task timeout, task discarding, and edge execution differences in the overall cost;

[0062] Step 1.23: Based on the instantaneous synthesis cost function for time slots Internal arrival task set The system in this time slot The total immediate integrated cost can be expressed as:

[0063] (17)

[0064] Based on the total real-time comprehensive cost, the time slots will be... The single-step reward is defined as the negative of the immediate total cost, i.e.:

[0065] (18)

[0066] As can be seen from equation (18), when the task completion delay is small, the number of timeout tasks is small, the number of dropped tasks is small, and the execution of the local part and the remote part is more coordinated, the system can obtain a higher reward value; conversely, when the task completion delay increases, timeout or drop events occur frequently, or when there is an obvious imbalance between the end and the edge execution after the task is split, the system reward will decrease significantly.

[0067] Step 1.3: Construct the MDP model based on Steps 1.1 and 1.2:

[0068] Step 1.31: For any time slot The agent generates corresponding task partial unloading actions based on the current system state. After receiving the actions, the environment completes task allocation, resource state updates, and queue evolution, and provides feedback on the single-step reward and the state at the next moment. Thus, the edge collaborative task partial unloading problem can be formally represented as a quintuple:

[0069] (19)

[0070] in, Representing the state space, Represents the action space, Represents the state transition probability. Represents the reward function, Indicates the discount factor;

[0071] Step 1.32, State Space

[0072] For time slots The system status is composed of task information, link transmission status, node computing capability status, and queue status, denoted as:

[0073] (20)

[0074] in, Indicates time slot The set of task information for the decision-making task includes characteristics such as the amount of input data, computational requirements, task source, and deadline. Indicates time slot The current system link transmission status describes the transmission capability between the task source terminal and each candidate target node. Indicates time slot Each terminal node, edge node, and cloud node within the time slot The current state of computing power Indicates time slot The current queue status of each edge node reflects the remaining available capacity and queuing load of the node. Indicates time slot The state space within the system is used to describe the complete environmental information of the system at the moment of the task unloading decision.

[0075] Step 1.33, Motion Space

[0076] The task's unloading decision involves both remote destination node selection and unloading ratio allocation; therefore, its action space is a hybrid action space. Define its time slot The action is:

[0077] (twenty one)

[0078] in, Indicates the selected task The distant target edge node is a discrete action; Indicates task The remote calculation of the unloading ratio is a continuous operation, therefore, the time slot The overall operation of the internal system is as follows:

[0079] (twenty two)

[0080] in, Indicates time slot Arrival at the task set, actions The execution will directly determine the remote execution location of each task and the local-remote load balancing ratio, and further affect task transmission latency, computation latency and queue occupancy.

[0081] Step 1.34, Reward Function

[0082] The reward function is used to quantify the immediate feedback obtained by the agent after performing an action in the current state. Based on the reward function design in step 1.2, the system... The single-step reward is shown in formula (18). ;

[0083] Step 1.35, State Transition Probability

[0084] When the agent is in the time slot status Take action below Subsequently, the local and remote portions of the task will enter the execution queues of their respective nodes. Simultaneously, the link status, node computing power, queue load, and information on newly arrived tasks will change over time. Therefore, the system state will change from... Transition to the next state The transfer process can be represented as:

[0085] (twenty three)

[0086] in, Indicates the current state Next action Then transition to the next state The probability distribution;

[0087] Step 1.36, Optimize Objective

[0088] Set a strategy The decision rule for an agent to generate actions in any state, based on the negative of the instantaneous comprehensive cost, yields the optimization objective, which can be expressed as:

[0089] (twenty four)

[0090] in, Indicating in strategy The following are expectations regarding the system's evolution process. For time slots The discount factor is used to measure the relative weight between immediate rewards and long-term rewards.

[0091] Furthermore, the specific steps of step 2 are as follows:

[0092] Step 2.1, set Represents edge nodes In the time slot The current queue occupancy. Represents edge nodes The upper limit of the queue capacity is defined to characterize the changing trend of the node queue over a short period of time. In the time slot The queue growth rate is:

[0093] (25)

[0094] in, This represents the length of the trend estimation window, used to characterize the average rate of change in node queue occupancy over the first few time slots. When it is large, it indicates an edge node. The queue load has increased rapidly in the recent period, posing a risk of further congestion.

[0095] According to edge nodes Current queue occupancy and queue growth rate, for edge nodes Predicting the queue occupancy at the next moment yields:

[0096] (26)

[0097] in, The trend adjustment coefficient is used to control the degree of influence of the queue growth trend on the prediction result. As shown in equation (26), when the current queue of a node has not yet reached its capacity limit, but its growth rate is large, the predicted queue occupancy... It may have approached or even exceeded the acceptable threshold, indicating that the node faces a high risk of congestion in the next decision-making period;

[0098] Step 2.2: Based on edge nodes The prediction result of the queue occupancy at the next moment is used to determine whether a node is suitable as the remote destination node of the current task, and edge nodes are defined. In state The following trend feasibility indicator function is:

[0099] (27)

[0100] in, For nodes The acceptable threshold coefficient can be set to restrict a node from entering the candidate action set in advance based on the prediction results before the node reaches its capacity limit.

[0101] Based on the trend feasibility indicator function, the task can be... In state The set of feasible destination nodes is defined as follows:

[0102] (28)

[0103] in, This indicates that the local ID corresponding to the task source terminal is The current queue occupancy of the device, Equation (28) indicates that a node is included in the candidate unloading target set of the current task only when the local part of the task can still be accepted by the local node and the candidate remote node is still in an acceptable state under the trend prediction.

[0104] Based on the set of feasible destination nodes, a node-level binary mask is introduced, defined as follows:

[0105] (29)

[0106] Suppose a discrete policy network is in state The following outputs the unconstrained logits vector of each candidate node. , among which, the Weiwei Then the distribution of the discrete strategy without masking is:

[0107] (30)

[0108] in, Indicates the first dimension;

[0109] To ensure that the node selection result satisfies the feasible region constraint corresponding to equation (28), an adaptive node-level binary mask is applied to equation (30) and then re-normalized to obtain the discrete policy distribution after masking:

[0110] (31)

[0111] During training and task execution, firstly, based on the current state... Obtain each candidate edge node in the time slot queue usage The node queue growth rate is calculated using equation (25), and the predicted queue occupancy of each candidate node at the next moment is obtained based on equation (26). Subsequently, the acceptability of each node under the trend prediction meaning is judged according to equation (27), and the feasible target node set of the current task is generated by combining equation (28). On this basis, a node-level binary mask is constructed by equation (29), and the node selection distribution without constraints is output by equation (30). Then, the mask constraint and normalization are completed by equation (31), and the remote target node is sampled from the feasible node range. At the edge node of the remote destination Once determined, the continuous policy network outputs the corresponding offloading ratio. Thus forming a joint action .

[0112] Furthermore, the specific steps of step 3 are as follows:

[0113] Step 3.1: Construction of a hierarchical collaborative decision-making network. This involves constructing a hierarchical collaborative decision-making network, comprising a high-level discrete policy network (Actor), a low-level continuous policy network (Actor), and a shared dual-Q Critic network. The high-level discrete policy network is responsible for selecting the remote destination node, the low-level continuous policy network is responsible for providing the corresponding offloading ratio, and the dual-Q Critic network is responsible for comprehensively evaluating the joint actions under a unified value scale.

[0114] To characterize the hierarchical dependency relationship of "remote destination node - unloading ratio", the joint action is represented as a multiplicative integral solution:

[0115] (32)

[0116] in, This represents the discrete strategy distribution constrained by the trend prediction adaptive masking mechanism in step 2, used to output the remote destination node. The selection results Represents a low-level continuous policy network Actor, used to address a given state. And selected tasks Remote destination edge node selection Under these conditions, output unloading ratio The conditional distribution, through the decomposition of equation (32), can transform the originally coupled hybrid action learning problem into a hierarchical decision-making process of "first determining the target node, then determining the proportion", so that the proportion allocation always revolves around the currently selected target node;

[0117] The input to the high-level discrete policy network Actor is the state space defined in step 1.3. The output is the discrete strategy distribution of each candidate remote destination node; in the node selection stage, the trend prediction adaptive masking machine given in step 2 is applied to the discrete strategy output to ensure that the node selection is always limited to the current set of feasible destination nodes.

[0118] Subsequently, at the selected remote destination node Under these conditions, the offloading ratio is generated by the low-level continuous policy network, taking into account the offloading ratio. The domain of the network is the interval [0,1]. The continuous policy network uses a Beta distribution with the domain [0,1] to parameterize the scale, which is expressed as follows:

[0119] (33)

[0120] in, and These are the distributed parameters output by the low-level continuous policy network;

[0121] In the value assessment section, a shared dual QCritic network is constructed to evaluate state-action triples (...). To conduct an overall value assessment, two parallel Critic networks are defined as follows: and To ensure that the value objective aligns with the hierarchical joint strategy, and to retain necessary exploration at both the discrete and continuous decision-making levels, a soft state-value function is introduced, which takes the form:

[0122] (34)

[0123] in, and By controlling the exploration intensity of discrete decision branches and continuous decision branches respectively, Equation (34) shows that the soft state value not only considers the value assessment results of joint actions, but also incorporates the entropy regularization terms of the two-layer strategies.

[0124] During training, samples are transferred from the experience replay pool. ,in The termination marker indicates that, based on the soft state value function, the temporal difference objective of Critic can be constructed as follows:

[0125] (35)

[0126] in, Discount factor;

[0127] The regression loss function of the double-Q Critic is defined as:

[0128] (36)

[0129] in, The parameters of the Critic network are represented by the regression error given by equation (36). By minimizing the regression error, the double Q Critic can gradually approach the target value defined by equation (35), thereby evaluating the joint action composed of different remote target nodes and different unloading ratios under a unified value scale.

[0130] Step 3.2, Task Part Unloading Algorithm Execution Flow:

[0131] After completing the construction of the hierarchical collaborative decision-making network, based on the MDP constructed in step 1.3, the trend prediction adaptive masking mechanism in step 2, and the hierarchical collaborative decision-making network in step 3.1, the task part unloading algorithm execution flow based on hierarchical SAC is formed. The algorithm iteratively optimizes the task part unloading strategy through the method of "environment interaction - experience storage - value update - strategy update". Among them, the high-level discrete strategy branch is responsible for the selection of the remote destination node, the low-level continuous strategy branch is responsible for the generation of the unloading ratio, and the shared double Q Critic network is responsible for the unified value evaluation of the joint action.

[0132] During training, the current state is obtained for the computational tasks generated by the terminal. First, the high-level discrete policy network Actor outputs the unconstrained discrete policy distribution of each candidate remote target node according to equation (32); then, the trend prediction results, node-level binary mask, and the masked discrete policy distribution are calculated according to equations (25) to (31). And sample the remote destination nodes within the feasible destination node set. Finally, the unloading ratio is generated by the low-level continuous policy network Actor according to equation (33). Thus forming a joint action ( After applying this combined action to the environment, a single-step reward is obtained. Next state and termination mark and transfer samples Store in the experience replay pool;

[0133] Once the number of samples in the experience replay pool reaches the set condition, a small batch of transfer samples is randomly sampled from it. Target values ​​are constructed according to equations (34) and (35), and the shared dual-Q Critic network is updated according to equation (36). Subsequently, the high-level discrete policy network Actor and the low-level continuous policy network Actor are updated respectively, with their optimization objectives being:

[0134] (37)

[0135] (38)

[0136] in, and represents the parameters of the high-level discrete policy network and the low-level continuous policy network, respectively. After each round of updates, the target Critic network is smoothly updated.

[0137] (39)

[0138] in, To smoothly update the coefficients of the target network, Indicates the first The parameters of the current Critic network, Indicates the first The parameters of the target Critic network.

[0139] A hierarchical reinforcement learning task offloading system based on adaptive constraints under edge computing includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the hierarchical reinforcement learning task offloading method based on adaptive constraints under edge computing.

[0140] A computer-readable storage medium storing a computer program thereon, characterized in that: when the computer program is executed by a processor, it implements the steps of the described hierarchical reinforcement learning task offloading method based on adaptive constraints under edge computing.

[0141] Compared with the prior art, the beneficial effects of this invention are as follows:

[0142] This invention, while simultaneously optimizing the selection of unloading nodes and the allocation of unloading ratios, reduces training oscillations and convergence difficulties caused by the mixed action space, and enhances the constraint on ineffective actions, thereby improving the decision-making quality and execution effect of task unloading. Specifically, it is manifested in the following ways:

[0143] I. The trend prediction adaptive masking mechanism for invalid action constraints proposed in this invention is designed to address the problem of the dynamic change in the availability of candidate nodes as the queue evolves during the task unloading process. Unlike existing technologies that statically mask actions based solely on the current queue occupancy status, capacity limit, or instantaneous resource status of a node, this invention further introduces the short-term trend of node queue changes to predict the acceptability of candidate nodes in subsequent periods. Based on the prediction results, a dynamic feasible target node set and a node-level mask are constructed, and this mask is directly embedded into the discrete node selection process. Thus, action constraints no longer remain limited to fully loaded nodes. Instead of post-event restrictions, this method can identify and pre-constrain high-risk nodes that are currently acceptable but show a continuous congestion trend in advance. This solves the problem of pseudo-actions caused by the lag in constraint triggering in existing methods. This mechanism can reduce invalid action sampling, queuing failure, task discarding and subsequent congestion spread caused by high-risk nodes entering the decision-making and training process, improve the accuracy of candidate node selection and the effectiveness of training samples, and change the action constraints in the task unloading process from a passive response based on the current state to an active prevention approach oriented towards the subsequent execution state, thereby improving the effectiveness of node selection, congestion prevention capability and overall execution stability.

[0144] II. The task partial unloading method based on hierarchical collaborative reinforcement learning proposed in this invention is a collaborative decision-making mechanism built on the basis of the aforementioned dynamic feasible region constraints, addressing the problems of strong coupling between discrete node selection and continuous unloading ratio allocation in task partial unloading, high difficulty of joint optimization, and easy imbalance of execution results. This method adopts a hierarchical SAC network structure (i.e., a hierarchical collaborative decision-making network), where the high-level discrete network Actor is responsible for selecting the remote target node, and the low-level continuous network Actor generates the corresponding unloading ratio under the condition of the selected nodes in the high-level network. The discrete decision results serve as the preconditions and inputs for continuous decision-making, enabling the ratio allocation to be optimized in a targeted manner around the link status, computing power, and queue status of the target node; at the same time, a shared double Q is adopted. The Critic network performs a unified value evaluation on the joint action consisting of discrete node selection and continuous proportional allocation, establishing policy updates at both levels on the same value scale. This enhances the correlation learning ability between node selection and proportional allocation, reducing training disturbances and decision mismatches caused by separate optimization and disconnect between the two types of decisions. Furthermore, this invention introduces an edge execution difference term into the reward function, representing the latency difference between the local and remote execution parts of the task. This ensures that policy learning not only focuses on task completion latency, timeouts, and dropouts, but also constrains the execution balance after task splitting, avoiding situations where one-sided optimization is superior but the overall completion effect is limited. Through the synergistic effect of trend prediction constraints, hierarchical conditional decision-making, unified value evaluation, and edge execution balance constraints, this invention can simultaneously reduce search complexity and coupling interference in the mixed action space, reduce the cascading disturbances caused by invalid node selection to subsequent proportional learning, and improve the rationality of task splitting, the stability of the training process, and the overall effect of task unloading. Attached Figure Description

[0145] Figure 1 This is the network structure for the hierarchical reinforcement learning task unloading algorithm of this invention;

[0146] Figure 2 This is the algorithm flow for the hierarchical reinforcement learning task unloading part of the present invention. Detailed Implementation

[0147] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0148] To address the aforementioned technical problems, this invention proposes a hierarchical reinforcement learning task partial offloading method based on adaptive constraints in an edge collaborative environment. This method addresses issues such as the coupling between offloading node selection and offloading ratio allocation during task partial offloading, the lack of early detection of node congestion evolution by existing action constraints, and the potential for latency imbalance between local and remote execution parts after task splitting. It constructs a task partial offloading framework integrating state modeling, action constraints, collaborative decision-making, and value assessment. First, it uniformly models task arrival, link status, node computing power, and queue status in the edge collaborative environment, formalizing the task partial offloading process as an online solvable Markov decision process. Subsequently, by combining the changing trends of the node queue within a short-term window, the acceptability of candidate nodes in subsequent periods is predicted, and a dynamic set of feasible destination nodes and a node-level mask are generated accordingly. This allows the selection of discrete unloading nodes to no longer be based solely on static judgments of whether the current node is fully loaded. Instead, it enables early constraints on high-risk nodes that are not yet explicitly congested but show a continuous growth trend. This reduces the number of seemingly selectable but potentially ineffective actions that could lead to queuing failures, task abandonment, or subsequent congestion propagation during the actual execution phase, thus improving the effectiveness of node selection and the foresight of action constraints. Furthermore, this invention constructs a hierarchical SAC collaborative decision-making network. A high-level discrete Actor is responsible for selecting the remote target node, while a low-level continuous Actor generates the corresponding unloading ratio based on the selected nodes at the high level. The discrete decision results serve as preconditions and inputs for continuous decisions, ensuring that the unloading ratio allocation always revolves around the determined target node. Simultaneously, a shared dual QCritic network is used to perform a unified value evaluation of the joint action consisting of discrete node selection and continuous ratio allocation. This allows the decision results from both levels to be correlated under the same value scale, reducing training disturbances and decision mismatches caused by the separation between discrete selection and continuous allocation, and improving the stability and consistency of collaborative optimization in a hybrid action space. At the same time, an edge execution difference term is introduced into the reward function to characterize the latency difference between the local and remote execution parts of the task. This ensures that policy learning not only focuses on task completion latency, timeouts, and discards, but also constrains the execution balance after task splitting. Through these methods, the effectiveness of invalid action constraints, node congestion prevention capabilities, and task splitting coordination during task unloading can be improved, thereby enhancing the task unloading effect, training stability, and execution stability.

[0149] This step aims to offload the complex edge computing task scenario, abstract and formalize it into an MDP, and provide a theoretical basis for the application of subsequent reinforcement learning algorithms.

[0150] First, the system environment is defined as follows: a set of tasks to be processed. In mobile device collection It can be generated on the top and then offloaded to the edge server. In, among them, It is the total number of tasks. It represents the total number of mobile devices. This refers to the total number of edge servers; the tasks are independent of each other. Representing the A mobile device Representing the One edge server, Represents a cloud server. The intelligent agent decision-making in this invention primarily focuses on... and Between. Each task is processed through a quintuple. To describe, among which, As a unique identifier for the task, For the task arrival time, The deadline for the task. Task computational load (unit: ), Task data volume (unit: The agent needs to select the optimal execution location for task unloading based on observations of the current system state when a task arrives, with the aim of systematically optimizing key performance indicators such as task completion latency, timeout rate, and drop rate.

[0151] A method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment includes the following steps:

[0152] Step 1: Formalize the task offloading process in the edge collaborative environment, including the construction of the system model, MDP model, and reward function;

[0153] The specific steps are as follows:

[0154] Step 1.1, System Model Construction:

[0155] Step 1.11: Based on a multi-access edge computing environment, the system consists of a mobile terminal set, an edge server set, and a cloud center. The system operates in discrete time slots, denoted as [time slot number missing]. For computing tasks generated by the terminal, the first... Task Represented as:

[0156] (1)

[0157] in, Indicates the generation of the first Task The source terminal device number, Indicates the first The amount of input data for each task Indicates the first The computational requirements for each task and They represent the first The arrival and deadline times for each task;

[0158] In the task-part unloading scenario, define discrete actions. Indicates task In the time slot Select the remote destination node and define an indicator variable. Indicates the first Does the task select edge nodes? As a remote execution location, i.e. when hour, ,otherwise At the same time, define continuous proportional variables Indicates task In the time slot The calculated unloading ratio is the percentage of data that is unloaded to the remote destination node, then the remaining percentage is... The computational load remains to be executed locally, and the local and remote parts of the task can be processed in parallel. The task is considered to have ended when both parts are completed.

[0159] Step 1.12, Transmission Model Construction:

[0160] No uplink transfer occurs when the task is executed locally;

[0161] When the remote portion of a task is offloaded to an edge node or the cloud, its input data needs to be transmitted via the uplink, numbered as follows: The source terminal equipment in the time slot To edge nodes The uplink instantaneous transmission rate is expressed using Shannon's formula as:

[0162] (2)

[0163] in, For the number Source terminal devices and edge nodes Link bandwidth between For the number The source terminal equipment in the time slot The transmission power, For the number The source terminal equipment in the time slot The equivalent channel gain, For noise power spectral density, For time slots To mitigate interference power, time slots are used for transmission links from the terminal to the cloud. equivalent constant rate This indicates its transmission capacity;

[0164] Due to the mission The distal part is proportional Unloaded, therefore, in the time slot The amount of data that needs to be uploaded is Therefore, the task The upload latency at the remote end is represented as:

[0165] (3)

[0166] in, This indicates that the remote portion is unloaded to the edge node. This indicates that the remote portion is offloaded to the cloud;

[0167] Step 1.13, Computational Model Construction:

[0168] edge nodes In the time slot The computing power is denoted as To characterize the fluctuations in node computing power over time, it is expressed as:

[0169] (4)

[0170] in, and Representing edge nodes respectively Minimum and maximum computing power, Indicates time slot The random perturbation term of the next node's computing power;

[0171] After proportionally dividing the task computation, the local portion is numbered as follows: The computation latency on the edge node corresponding to the source terminal device is:

[0172] (5)

[0173] in, Indicates the number is The source terminal equipment in the time slot The computing power, the remote part is at the edge node The computation delay is:

[0174] (6)

[0175] Step 1.14, Building the queuing model:

[0176] Using a first-come, first-served queue service method, let... Indicates time slot Internal mission At the edge node If the remaining computational workload of the preceding task that needs to wait is given, then its queuing delay can be expressed as the ratio of the remaining computational workload of the preceding task to the node's computing power.

[0177] Therefore, the queuing delays for the local portion and the remote portion are respectively:

[0178] (7)

[0179] (8)

[0180] Step 1.15, End-to-end delay:

[0181] Based on the above transmission model, calculation model, and queuing model, the total local latency is:

[0182] (9)

[0183] When the remote part of the task is proportional Unload to edge node When the total delay of its far end is , it can be written as:

[0184] (10)

[0185] When the remote part of the task is proportional When unloading to the cloud, the total latency of its remote portion can be written as:

[0186] (11)

[0187] in Proportionately for the remote part of the task Similarly, the transmission latency when unloading to the cloud... To delay queuing, To calculate latency. Since the local and remote parts of the task execute in parallel, the task... The end-to-end completion delay is defined as:

[0188] (12)

[0189] in, This indicates the total latency of the local execution portion of the task. This represents the total latency when the remote part of the task is executed on the edge node. This represents the total latency when the remote portion of a task is executed in the cloud.

[0190] In summary, by establishing transmission models, computation models, and end-to-end latency models, the system model for the partial offloading scenario of edge collaborative tasks can be completed, providing a unified foundation for subsequent MDP model construction and reward function design.

[0191] Step 1.2: Construct the comprehensive cost function based on Step 1.1:

[0192] After completing the system model construction, in order for the agent to learn a reasonable task offloading strategy in the edge collaborative environment, it is necessary to further construct a reward function that matches the task offloading target. Since the task offloading scenario not only needs to consider the end-to-end completion latency of the task, but also whether the task times out, is dropped due to queue capacity limitations, and the collaborative execution relationship between the local and remote parts of the task, this invention adopts a reward construction method based on real-time comprehensive cost, and on this basis, introduces an edge execution difference term to characterize the latency difference between the local and remote execution parts, so that the reward function can simultaneously reflect the task execution efficiency, service success, and the rationality of task splitting.

[0193] Step 1.21: Define the task based on the relationship between the task completion delay and the task deadline. In the time slot The timeout indicator function is:

[0194] (13)

[0195] in, For time slots Task The end-to-end completion delay, Indicates task The task is considered to have timed out if the completion time exceeds its allowed execution time limit; otherwise, the task is considered not to have timed out.

[0196] Step 1.22: Further, to characterize the situation where a task cannot be accepted by the system during the unloading process because the local queue or the remote target node queue has reached its capacity limit, a task is defined. In the time slot The discard indicator function is:

[0197] (14)

[0198] Specifically, when a task's local or remote portion triggers a capacity constraint during enqueuing, preventing the task from being accepted by the corresponding node, record... Otherwise, remember ;

[0199] Considering that in a task unloading scenario, the local and remote parts execute in parallel, and the overall task completion time is determined by the slower of the two, if there is a significant difference in completion time between the local and remote parts, the faster part will be idle before the overall task is completed, resulting in insufficient resource utilization and an unreasonable allocation of unloading ratios. To characterize the degree of coordination between the local and remote execution parts of a task, a task is defined... In the time slot The differences in execution at the endpoints are as follows:

[0200] (15)

[0201] This item is used to quantify the latency difference between the local and remote parts of the task during parallel execution. The larger the value, the more unbalanced the execution of the two parts after the task is split, and the worse the coordination of the task offloading scheme.

[0202] Based on the above definition, construct the task. In the time slot The instantaneous synthesis cost function is:

[0203] (16)

[0204] in, , are weighting coefficients, used to measure the relative importance of task completion delay, task timeout, task discard and end-edge execution differences in the overall cost; through equation (16), multiple objectives that need to be optimized during the task unloading process can be uniformly mapped into a single instant overall cost framework.

[0205] Step 1.23: Based on the instantaneous synthesis cost function for time slots Internal arrival task set The system in this time slot The total immediate integrated cost can be expressed as:

[0206] (17)

[0207] Since reinforcement learning typically learns the optimal policy by maximizing cumulative reward, the time slots are calculated based on the total immediate integrated cost. The single-step reward is defined as the negative of the immediate total cost, i.e.:

[0208] (18)

[0209] As can be seen from equation (18), when the task completion delay is small, the number of timed-out tasks is small, the number of dropped tasks is small, and the execution of the local part and the remote part is more coordinated, the system can obtain a higher reward value; conversely, when the task completion delay increases, timeout or drop events occur frequently, or when there is a significant imbalance in execution between the end and the edge after task splitting, the system reward will decrease significantly; thus, the reward function designed in this invention can not only guide the agent to reduce the overall execution overhead in the task part unloading process, but also prompt it to actively suppress unreasonable splitting in the process of selecting unloading nodes and allocating unloading ratios, thereby improving the coordination and execution efficiency of task part unloading.

[0210] In summary, by introducing timeout indicator functions, discard indicator functions, and edge execution difference terms, and constructing instantaneous comprehensive cost and single-step reward functions accordingly, the reward function design for the partial offloading scenario of edge collaborative tasks can be completed, providing a reward basis for subsequent MDP model construction and hierarchical reinforcement learning algorithm solution.

[0211] Step 1.3: Construct the MDP model based on Steps 1.1 and 1.2:

[0212] Step 1.31: After completing the system model construction and reward function design, to characterize the sequential decision-making process of task unloading in an edge collaborative environment, this problem is further modeled as a Markov decision process. For any time slot... The agent generates corresponding task partial unloading actions based on the current system state. After receiving the actions, the environment completes task allocation, resource state updates, and queue evolution, and provides feedback on the single-step reward and the state at the next moment. Thus, the edge collaborative task partial unloading problem can be formally represented as a quintuple:

[0213] (19)

[0214] in, Representing the state space, Represents the action space, Represents the state transition probability. Represents the reward function, Indicates the discount factor;

[0215] Step 1.32, State Space

[0216] For time slots The system status is composed of task information, link transmission status, node computing capability status, and queue status, denoted as:

[0217] (20)

[0218] in, Indicates time slot The set of task information for the decision-making task includes characteristics such as the amount of input data, computational requirements, task source, and deadline. Indicates time slot The current system link transmission status describes the transmission capability between the task source terminal and each candidate target node. Indicates time slot Each terminal node, edge node, and cloud node within the time slot The current state of computing power Indicates time slot The current queue status of each edge node reflects the remaining available capacity and queuing load of the node. Indicates time slot The state space within the system is used to describe the complete environmental information of the system at the moment of the task partial unloading decision. Through the definition of the state space, the agent can fully perceive the task characteristics and edge environment state when making decisions, thereby providing a basis for the task partial unloading.

[0219] Step 1.33, Motion Space

[0220] The task's unloading decision involves both remote destination node selection and unloading ratio allocation; therefore, its action space is a hybrid action space. Define its time slot The action is:

[0221] (twenty one)

[0222] in, Indicates the selected task The distant target edge node is a discrete action; Indicates task The remote calculation of the unloading ratio is a continuous operation, therefore, the time slot The overall operation of the internal system is as follows:

[0223] (twenty two)

[0224] in, Indicates time slot Arrival at the task set, actions The execution will directly determine the remote execution location of each task and the local-remote load balancing ratio, and further affect task transmission latency, computation latency and queue occupancy.

[0225] Step 1.34, Reward Function

[0226] The reward function is used to quantify the immediate feedback obtained by the agent after performing an action in the current state. Based on the reward function design in step 1.2, the system... The single-step reward is shown in formula (18). This reward definition allows task completion delay, task timeout, task discarding, and execution differences between local and remote execution parts to be uniformly mapped as reward signals in reinforcement learning, thereby guiding the agent to learn better task offloading strategies.

[0227] Step 1.35, State Transition Probability

[0228] State transition probabilities are used to describe the environmental evolution of a system after it performs its current action. When an agent is in a time slot... status Take action below Subsequently, the local and remote portions of the task will enter the execution queues of their respective nodes. Simultaneously, the link status, node computing power, queue load, and information on newly arrived tasks will change over time. Therefore, the system state will change from... Transition to the next state The transfer process can be represented as:

[0229] (twenty three)

[0230] in, Indicates the current state Next action Then transition to the next state The probability distribution is as follows. Because task arrival, link transmission, node computing power, and queue status all have dynamic characteristics, this state transition process exhibits significant temporal and random characteristics.

[0231] Step 1.36, Optimize Objective

[0232] Based on the above definition, the task unloading problem can be transformed into a sequential decision problem based on maximizing cumulative discount rewards. Let the strategy be... The decision rule for an agent to generate actions in any state, based on the negative of the instantaneous comprehensive cost, yields the optimization objective, which can be expressed as:

[0233] (twenty four)

[0234] in, Indicating in strategy The following are expectations regarding the system's evolution process. For time slots The discount factor is used to measure the relative weight between immediate and long-term rewards. By solving the optimal policy corresponding to equation (24), the agent can obtain better task offloading decision results in a dynamic edge collaborative environment.

[0235] In summary, by formally defining the state space, action space, reward function, state transition probability, and optimization objective, we can construct an MDP model for the edge collaborative task partial unloading problem, thus providing a unified problem representation for the subsequent design of trend prediction adaptive masking mechanisms and the solution of collaborative task partial unloading algorithms based on hierarchical SAC.

[0236] Step 2: Construct a trend prediction adaptive masking mechanism for invalid action constraints, and generate a dynamic set of actionable actions based on the trend of node state changes;

[0237] During task unloading, the selection of remote destination nodes and the allocation of unloading ratios jointly affect the dynamic evolution of task transmission load, computation load, and node queue status. When a node's queue has reached its capacity limit, if it is still considered a candidate unloading target, the corresponding action will directly fail. Conversely, if a node, although not yet fully loaded, shows a significant increasing trend in queue occupancy, continuing to include it as a candidate target can easily cause congestion in subsequent time slots, leading to task enqueue failure, task dropping, or reduced unloading effectiveness. To address this, this invention proposes a trend prediction adaptive masking mechanism for invalid action constraints. During the node selection phase, it not only considers the queue occupancy status of the current time slot but also further combines the queue change trends of previous time slots to predict the acceptability of candidate nodes and dynamically generate a set of feasible destination nodes accordingly, thereby achieving proactive constraints on invalid actions.

[0238] The specific steps are as follows:

[0239] Step 2.1, set Represents edge nodes In the time slot The current queue occupancy. Represents edge nodes The upper limit of the queue capacity is defined to characterize the changing trend of the node queue over a short period of time. In the time slot The queue growth rate is:

[0240] (25)

[0241] in, This represents the length of the trend estimation window, used to characterize the average rate of change in node queue occupancy over the first few time slots. When it is large, it indicates an edge node. The queue load has increased rapidly in the recent period, posing a risk of further congestion.

[0242] Based on this, according to edge nodes Current queue occupancy and queue growth rate, for edge nodes Predicting the queue occupancy at the next moment yields:

[0243] (26)

[0244] in, The trend adjustment coefficient is used to control the degree of influence of the queue growth trend on the prediction result. As shown in equation (26), when the current queue of a node has not yet reached its capacity limit, but its growth rate is large, the predicted queue occupancy... It may have approached or even exceeded the acceptable threshold, indicating that the node faces a high risk of congestion in the next decision-making period;

[0245] Step 2.2: Based on edge nodes The prediction result of the queue occupancy at the next moment is used to determine whether a node is suitable as the remote destination node of the current task, and edge nodes are defined. In state The following trend feasibility indicator function is:

[0246] (27)

[0247] in, For nodes The acceptable threshold coefficient can be set to restrict a node from entering the candidate action set before it reaches its capacity limit, thereby enhancing the foresight of action constraints.

[0248] Based on the trend feasibility indicator function, the task can be... In state The set of feasible destination nodes is defined as follows:

[0249] (28)

[0250] in, This indicates that the local ID corresponding to the task source terminal is The current queue occupancy of the device, Equation (28) indicates that a node is included in the candidate unloading target set of the current task only when the local part of the task can still be accepted by the local node and the candidate remote node is still in an acceptable state under the trend prediction.

[0251] To facilitate use in discrete destination node selection networks, a node-level binary mask is introduced based on the set of feasible destination nodes, defined as:

[0252] (29)

[0253] Suppose a discrete policy network is in state The following outputs the unconstrained logits vector of each candidate node. , among which, the Weiwei Then the distribution of the discrete strategy without masking is:

[0254] (30)

[0255] in, Indicates the first dimension;

[0256] To ensure that the node selection result satisfies the feasible region constraint corresponding to equation (28), an adaptive node-level binary mask is applied to equation (30) and then re-normalized to obtain the discrete policy distribution after masking:

[0257] (31)

[0258] During training and task execution, firstly, based on the current state... Obtain each candidate edge node in the time slot queue usage The node queue growth rate is calculated using equation (25), and the predicted queue occupancy of each candidate node at the next moment is obtained based on equation (26). Subsequently, the acceptability of each node under the trend prediction meaning is judged according to equation (27), and the feasible target node set of the current task is generated by combining equation (28). On this basis, a node-level binary mask is constructed by equation (29), and the node selection distribution without constraints is output by equation (30). Then, the mask constraint and normalization are completed by equation (31), and the remote target node is sampled from the feasible node range. At the edge node of the remote destination Once determined, the continuous policy network outputs the corresponding offloading ratio. Thus forming a joint action .

[0259] Through the aforementioned mechanism, the node selection phase is always confined to the feasible region that satisfies trend constraints, thereby identifying high-risk targets and restricting their participation in action sampling before nodes become explicitly full. Compared to methods that only constrain based on whether the current queue has reached its capacity limit, this invention introduces queue change trend prediction, making invalid action constraints more forward-looking. It can transform passive congestion avoidance into proactive risk prevention, further reducing invalid action selection and improving the execution stability and node selection effectiveness during the task unloading process.

[0260] Step 3: Construct a collaborative task partial unloading algorithm based on hierarchical SAC, and train it in conjunction with Step 1 and Step 2. Based on the trained model, complete the selection of task unloading nodes and the allocation of unloading ratio, and realize collaborative optimization decision-making in the task partial unloading problem.

[0261] Since the task-part unloading action in this invention includes both remote destination node selection and unloading ratio allocation, it belongs to a typical hybrid action space. Remote destination node selection is a discrete decision, while unloading ratio allocation is a continuous decision. Directly performing unified learning on the joint action space can easily lead to training instability. Therefore, this invention uses SAC as the basic framework and constructs a hierarchical collaborative decision network, splitting the hybrid action into two levels: high-level discrete decision and low-level continuous decision, to improve the stability of the task-part unloading decision.

[0262] The specific steps are as follows:

[0263] Step 3.1: Construction of a hierarchical collaborative decision-making network. This involves constructing a hierarchical collaborative decision-making network, comprising a high-level discrete policy network (Actor), a low-level continuous policy network (Actor), and a shared dual-Q Critic network. The high-level discrete policy network is responsible for selecting the remote destination node, the low-level continuous policy network is responsible for providing the corresponding offloading ratio, and the dual-Q Critic network is responsible for comprehensively evaluating the joint actions under a unified value scale.

[0264] To characterize the hierarchical dependency relationship of "remote destination node - unloading ratio", the joint action is represented as a multiplicative integral solution:

[0265] (32)

[0266] in, This represents the discrete strategy distribution constrained by the trend prediction adaptive masking mechanism in step 2, used to output the remote destination node. The selection results Represents a low-level continuous policy network Actor, used to address a given state. And selected tasks Remote destination edge node selection Under these conditions, output unloading ratio The conditional distribution, through the decomposition of equation (32), can transform the originally coupled hybrid action learning problem into a hierarchical decision-making process of "first determining the target node, then determining the proportion", so that the proportion allocation always revolves around the currently selected target node;

[0267] The input to the high-level discrete policy network Actor is the state space defined in step 1.3. The output is the discrete strategy distribution of each candidate remote destination node; in the node selection stage, the trend prediction adaptive masking machine given in step 2 is applied to the discrete strategy output to ensure that the node selection is always limited to the current set of feasible destination nodes.

[0268] Subsequently, at the selected remote destination node Under these conditions, the offloading ratio is generated by the low-level continuous policy network, taking into account the offloading ratio. The domain of the network is the interval [0,1]. The continuous policy network uses a Beta distribution with the domain [0,1] to parameterize the scale, which is expressed as follows:

[0269] (33)

[0270] in, and These are the distributed parameters output by the low-level continuous policy network;

[0271] In the value assessment section, a shared dual QCritic network is constructed to evaluate state-action triples (...). To conduct an overall value assessment, two parallel Critic networks are defined as follows: and The smaller of the two values ​​is taken as the conservative evaluation result to suppress the bias caused by overestimation in policy improvement. To ensure that the value objective is consistent with the hierarchical joint policy and to retain necessary exploration at both the discrete and continuous decision-making levels, a soft state-value function is introduced, with the form:

[0272] (34)

[0273] in, and By controlling the exploration intensity of discrete decision branches and continuous decision branches respectively, Equation (34) shows that the soft state value not only considers the value assessment results of joint actions, but also incorporates the entropy regularization terms of the two-layer strategies.

[0274] During training, samples are transferred from the experience replay pool. ,in The termination marker indicates that, based on the soft state value function, the temporal difference objective of Critic can be constructed as follows:

[0275] (35)

[0276] in, Discount factor;

[0277] The regression loss function of the double-Q Critic is defined as:

[0278] (36)

[0279] in, The parameters of the Critic network are represented by the regression error given by equation (36). By minimizing the regression error, the double Q Critic can gradually approach the target value defined by equation (35), thereby evaluating the joint action composed of different remote target nodes and different unloading ratios under a unified value scale.

[0280] In summary, the hierarchical collaborative decision-making network constructed in this invention consists of a high-level discrete policy network, a low-level continuous policy network, and a shared dual-Q Critic network. The high-level discrete policy network is responsible for selecting the remote destination node, the low-level continuous policy network is responsible for providing the corresponding unloading ratio, and the dual-Q Critic network is responsible for the overall evaluation of joint actions under a unified value scale. This hierarchical collaborative decision-making structure can explicitly characterize the conditional dependency relationship between the "destination node and unloading ratio," and provides the network structure foundation for subsequent parameter updates based on SAC and the execution flow of the collaborative task's partial unloading algorithm.

[0281] Step 3.2, Task Part Unloading Algorithm Execution Flow:

[0282] After completing the construction of the hierarchical collaborative decision-making network, based on the MDP constructed in step 1.3, the trend prediction adaptive masking mechanism in step 2, and the hierarchical collaborative decision-making network in step 3.1, the task part unloading algorithm execution flow based on hierarchical SAC is formed. The algorithm iteratively optimizes the task part unloading strategy through the method of "environment interaction - experience storage - value update - strategy update". Among them, the high-level discrete strategy branch is responsible for the selection of the remote destination node, the low-level continuous strategy branch is responsible for the generation of the unloading ratio, and the shared double Q Critic network is responsible for the unified value evaluation of the joint action.

[0283] During training, the current state is obtained for the computational tasks generated by the terminal. First, the high-level discrete policy network Actor outputs the unconstrained discrete policy distribution of each candidate remote target node according to equation (32); then, the trend prediction results, node-level binary mask, and the masked discrete policy distribution are calculated according to equations (25) to (31). And sample the remote destination nodes within the feasible destination node set. Finally, the unloading ratio is generated by the low-level continuous policy network Actor according to equation (33). Thus forming a joint action ( After applying this combined action to the environment, a single-step reward is obtained. Next state and termination mark and transfer samples Store in the experience replay pool;

[0284] Once the number of samples in the experience replay pool reaches the set condition, a small batch of transfer samples is randomly sampled from it. Target values ​​are constructed according to equations (34) and (35), and the shared dual-Q Critic network is updated according to equation (36). Subsequently, the high-level discrete policy network Actor and the low-level continuous policy network Actor are updated respectively, with their optimization objectives being:

[0285] (37)

[0286] (38)

[0287] in, and represents the parameters of the high-level discrete policy network and the low-level continuous policy network, respectively. After each round of updates, the target Critic network is smoothly updated.

[0288] (39)

[0289] in, To smoothly update the coefficients of the target network, Indicates the first The parameters of the current Critic network, Indicates the first The parameters of the target Critic network.

[0290] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment, characterized in that, Includes the following steps: Step 1: Formalize the task offloading process in the edge collaborative environment, including the construction of the system model, MDP model, and reward function; Step 2: Construct a trend prediction adaptive masking mechanism for invalid action constraints, and generate a dynamic set of actionable actions based on the trend of node state changes; Step 3: Construct a collaborative task partial unloading algorithm based on hierarchical SAC, and train it in conjunction with Step 1 and Step 2. Based on the trained model, complete the selection of task unloading nodes and the allocation of unloading ratio, and realize collaborative optimization decision-making in the task partial unloading problem.

2. The method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1, System Model Construction: Step 1.11: Based on a multi-access edge computing environment, the system consists of a mobile terminal set, an edge server set, and a cloud center. The system operates in discrete time slots, denoted as [time slot number missing]. For computing tasks generated by the terminal, the first... Task Represented as: (1) in, Indicates the generation of the first Task The source terminal device number, Indicates the first The amount of input data for each task Indicates the first The computational requirements for each task and They represent the first The arrival and deadline times for each task; In the task-part unloading scenario, define discrete actions. Indicates task In the time slot Select the remote destination node and define an indicator variable. Indicates the first Does the task select edge nodes? As a remote execution location, i.e. when hour, ,otherwise At the same time, define continuous proportional variables Indicates task In the time slot The calculated unloading ratio is the percentage of data that is unloaded to the remote destination node, then the remaining percentage is... The computational load remains to be executed locally, and the local and remote parts of the task can be processed in parallel. The task is considered to have ended when both parts are completed. Step 1.12, Transmission Model Construction: No uplink transfer occurs when the task is executed locally; When the remote portion of a task is offloaded to an edge node or the cloud, its input data needs to be transmitted via the uplink, numbered as follows: The source terminal equipment in the time slot To edge nodes The uplink instantaneous transmission rate is expressed using Shannon's formula as: (2) in, For the number Source terminal devices and edge nodes Link bandwidth between For the number The source terminal equipment in the time slot The transmission power, For the number The source terminal equipment in the time slot The equivalent channel gain, For noise power spectral density, For time slots To mitigate interference power, time slots are used for transmission links from the terminal to the cloud. equivalent constant rate This indicates its transmission capacity; Due to the mission The distal part is proportional Unloaded, therefore, in the time slot The amount of data that needs to be uploaded is Therefore, the task The upload latency at the remote end is represented as: (3) in, This indicates that the remote portion is unloaded to the edge node. This indicates that the remote portion is offloaded to the cloud; Step 1.13, Computational Model Construction: edge nodes In the time slot The computing power is denoted as To characterize the fluctuations in node computing power over time, it is expressed as: (4) in, and Representing edge nodes respectively Minimum and maximum computing power, Indicates time slot The random perturbation term of the next node's computing power; After proportionally dividing the task computation, the local portion is numbered as follows: The computation latency on the edge node corresponding to the source terminal device is: (5) in, Indicates the number is The source terminal equipment in the time slot The computing power, the remote part is at the edge node The computation delay is: (6) Step 1.14, Building the queuing model: Using a first-come, first-served queue service method, let... Indicates time slot Internal mission At the edge node If the remaining computational workload of the preceding task that needs to wait is given, then its queuing delay can be expressed as the ratio of the remaining computational workload of the preceding task to the node's computing power. Therefore, the queuing delays for the local portion and the remote portion are respectively: (7) (8) Step 1.15, End-to-end delay: Based on the above transmission model, calculation model, and queuing model, the total local latency is: (9) When the remote part of the task is proportional Unload to edge node When the total delay of its far end is , it can be written as: (10) When the remote part of the task is proportional When unloading to the cloud, the total latency of its remote portion can be written as: (11) in, Proportionately for the remote part of the task Similarly, the transmission latency when unloading to the cloud... To delay queuing, To calculate latency, since the local and remote parts of the task execute in parallel, the task... The end-to-end completion delay is defined as: (12) in, This indicates the total latency of the local execution portion of the task. This represents the total latency when the remote part of the task is executed on the edge node. This represents the total latency when the remote portion of a task is executed in the cloud. Step 1.2: Construct the comprehensive cost function based on Step 1.1: Step 1.21: Define the task based on the relationship between the task completion delay and the task deadline. In the time slot The timeout indicator function is: (13) in, For time slots Task The end-to-end completion delay, Indicates task The task is considered to have timed out if the completion time exceeds its allowed execution time limit; otherwise, the task is considered not to have timed out. Step 1.22, Define the task In the time slot The discard indicator function is: (14) Specifically, when a task's local or remote portion triggers a capacity constraint during enqueuing, preventing the task from being accepted by the corresponding node, record... Otherwise, remember ; To characterize the degree of collaboration between the local and remote execution parts of a task, a task is defined. In the time slot The differences in execution at the endpoints are as follows: (15) Based on the above definition, construct the task. In the time slot The instantaneous synthesis cost function is: (16) in, These are weighting coefficients, used to measure the relative importance of task completion latency, task timeout, task discarding, and edge execution differences in the overall cost; Step 1.23: Based on the instantaneous synthesis cost function for time slots Internal arrival task set The system in this time slot The total immediate integrated cost can be expressed as: (17) Based on the total real-time comprehensive cost, the time slots will be... The single-step reward is defined as the negative of the immediate total cost, i.e.: (18) As can be seen from equation (18), when the task completion delay is small, the number of timeout tasks is small, the number of dropped tasks is small, and the execution of the local part and the remote part is more coordinated, the system can obtain a higher reward value; conversely, when the task completion delay increases, timeout or drop events occur frequently, or when there is an obvious imbalance between the end and the edge execution after the task is split, the system reward will decrease significantly. Step 1.3: Construct the MDP model based on Steps 1.1 and 1.2: Step 1.31: For any time slot The agent generates corresponding task partial unloading actions based on the current system state. After receiving the actions, the environment completes task allocation, resource state updates, and queue evolution, and provides feedback on the single-step reward and the state at the next moment. Thus, the edge collaborative task partial unloading problem can be formally represented as a quintuple: (19) in, Representing the state space, Represents the action space, Represents the state transition probability. Represents the reward function, Indicates the discount factor; Step 1.32, State Space For time slots The system status is composed of task information, link transmission status, node computing capability status, and queue status, denoted as: (20) in, Indicates time slot The set of task information for the decision-making task includes characteristics such as the amount of input data, computational requirements, task source, and deadline. Indicates time slot The current system link transmission status describes the transmission capability between the task source terminal and each candidate target node. Indicates time slot Each terminal node, edge node, and cloud node within the time slot The current state of computing power Indicates time slot The current queue status of each edge node reflects the remaining available capacity and queuing load of the node. Indicates time slot The state space within the system is used to describe the complete environmental information of the system at the moment of the task unloading decision. Step 1.33, Motion Space The task's unloading decision involves both remote destination node selection and unloading ratio allocation; therefore, its action space is a hybrid action space. Define its time slot The action is: (21) in, Indicates the selected task The distant target edge node is a discrete action; Indicates task The remote calculation of the unloading ratio is a continuous operation, therefore, the time slot The overall operation of the internal system is as follows: (22) in, Indicates time slot Arrival at the task set, actions The execution will directly determine the remote execution location of each task and the local-remote load balancing ratio, and further affect task transmission latency, computation latency and queue occupancy. Step 1.34, Reward Function The reward function is used to quantify the immediate feedback obtained by the agent after performing an action in the current state. Based on the reward function design in step 1.2, the system... The single-step reward is shown in formula (18). ; Step 1.35, State Transition Probability When the agent is in the time slot status Take action below Subsequently, the local and remote portions of the task will enter the execution queues of their respective nodes. Simultaneously, the link status, node computing power, queue load, and information on newly arrived tasks will change over time. Therefore, the system state will change from... Transition to the next state The transfer process can be represented as: (23) in, Indicates the current state Next action Then transition to the next state The probability distribution; Step 1.36, Optimize Objective Set a strategy The decision rule for an agent to generate actions in any state, based on the negative of the instantaneous comprehensive cost, yields the optimization objective, which can be expressed as: (24) in, Indicating in strategy The following are expectations regarding the system's evolution process. For time slots The discount factor is used to measure the relative weight between immediate rewards and long-term rewards.

3. The method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment according to claim 2, characterized in that, The specific steps of step 2 are as follows: Step 2.1, set Represents edge nodes In the time slot The current queue occupancy. Represents edge nodes The upper limit of the queue capacity is defined to characterize the changing trend of the node queue over a short period of time. In the time slot The queue growth rate is: (25) in, This represents the length of the trend estimation window, used to characterize the average rate of change in node queue occupancy over the first few time slots. When it is large, it indicates an edge node. The queue load has increased rapidly in the recent period, posing a risk of further congestion. According to edge nodes Current queue occupancy and queue growth rate, for edge nodes Predicting the queue occupancy at the next moment yields: (26) in, The trend adjustment coefficient is used to control the degree of influence of the queue growth trend on the prediction result. As shown in equation (26), when the current queue of a node has not yet reached its capacity limit, but its growth rate is large, the predicted queue occupancy... It may have approached or even exceeded the acceptable threshold, indicating that the node faces a high risk of congestion in the next decision-making period; Step 2.2: Based on edge nodes The prediction result of the queue occupancy at the next moment is used to determine whether a node is suitable as the remote destination node of the current task, and edge nodes are defined. In state The following trend feasibility indicator function is: (27) in, For nodes The acceptable threshold coefficient can be set to restrict a node from entering the candidate action set in advance based on the prediction results before the node reaches its capacity limit. Based on the trend feasibility indicator function, the task can be... In state The set of feasible destination nodes is defined as follows: (28) in, This indicates that the local ID corresponding to the task source terminal is The current queue occupancy of the device, Equation (28) indicates that a node is included in the candidate unloading target set of the current task only when the local part of the task can still be accepted by the local node and the candidate remote node is still in an acceptable state under the trend prediction. Based on the set of feasible destination nodes, a node-level binary mask is introduced, defined as follows: (29) Suppose a discrete policy network is in state The following outputs the unconstrained logits vector of each candidate node. , among which, the Weiwei Then the distribution of the discrete strategy without masking is: (30) in, Indicates the first dimension; To ensure that the node selection result satisfies the feasible region constraint corresponding to equation (28), an adaptive node-level binary mask is applied to equation (30) and then re-normalized to obtain the discrete policy distribution after masking: (31) During training and task execution, firstly, based on the current state... Obtain each candidate edge node in the time slot queue usage The node queue growth rate is calculated using equation (25), and the predicted queue occupancy of each candidate node at the next moment is obtained based on equation (26). Subsequently, the acceptability of each node under the trend prediction meaning is judged according to equation (27), and the feasible target node set of the current task is generated by combining equation (28). On this basis, a node-level binary mask is constructed by equation (29), and the node selection distribution without constraints is output by equation (30). Then, the mask constraint and normalization are completed by equation (31), and the remote target node is sampled from the feasible node range. At the edge node of the remote destination Once determined, the continuous policy network outputs the corresponding offloading ratio. Thus forming a joint action .

4. The method for partially unloading hierarchical reinforcement learning tasks based on adaptive constraints in an edge collaborative environment according to claim 3, characterized in that, The specific steps of step 3 are as follows: Step 3.1: Construction of a hierarchical collaborative decision-making network. This involves constructing a hierarchical collaborative decision-making network, comprising a high-level discrete policy network (Actor), a low-level continuous policy network (Actor), and a shared dual-Q Critic network. The high-level discrete policy network is responsible for selecting the remote destination node, the low-level continuous policy network is responsible for providing the corresponding offloading ratio, and the dual-Q Critic network is responsible for comprehensively evaluating the joint actions under a unified value scale. To characterize the hierarchical dependency relationship of "remote destination node - unloading ratio", the joint action is represented as a multiplicative integral solution: (32) in, This represents the discrete strategy distribution constrained by the trend prediction adaptive masking mechanism in step 2, used to output the remote destination node. The selection results Represents a low-level continuous policy network Actor, used to address a given state. And selected tasks Remote destination edge node selection Under these conditions, output unloading ratio The conditional distribution, through the decomposition of equation (32), can transform the originally coupled hybrid action learning problem into a hierarchical decision-making process of "first determining the target node, then determining the proportion", so that the proportion allocation always revolves around the currently selected target node; The input to the high-level discrete policy network Actor is the state space defined in step 1.

3. The output is the discrete strategy distribution of each candidate remote destination node; in the node selection stage, the trend prediction adaptive masking machine given in step 2 is applied to the discrete strategy output to ensure that the node selection is always limited to the current set of feasible destination nodes. Subsequently, at the selected remote destination node Under these conditions, the offloading ratio is generated by the low-level continuous policy network, taking into account the offloading ratio. The domain of the network is the interval [0,1]. The continuous policy network uses a Beta distribution with the domain [0,1] to parameterize the scale, which is expressed as follows: (33) in, and These are the distributed parameters output by the low-level continuous policy network; In the value assessment section, a shared dual QCritic network is constructed to evaluate state-action triples (...). To conduct an overall value assessment, two parallel Critic networks are defined as follows: and To ensure that the value objective aligns with the hierarchical joint strategy, and to retain necessary exploration at both the discrete and continuous decision-making levels, a soft state-value function is introduced, which takes the form: (34) in, and By controlling the exploration intensity of discrete decision branches and continuous decision branches respectively, Equation (34) shows that the soft state value not only considers the value assessment results of joint actions, but also incorporates the entropy regularization terms of the two-layer strategies. During training, samples are transferred from the experience replay pool. ,in The termination marker indicates that, based on the soft state value function, the temporal difference objective of Critic can be constructed as follows: (35) in, Discount factor; The regression loss function of the double-Q Critic is defined as: (36) in, The parameters of the Critic network are represented by the regression error given by equation (36). By minimizing the regression error, the double Q Critic can gradually approach the target value defined by equation (35), thereby evaluating the joint action composed of different remote target nodes and different unloading ratios under a unified value scale. Step 3.2, Task Part Unloading Algorithm Execution Flow: After completing the construction of the hierarchical collaborative decision-making network, based on the MDP constructed in step 1.3, the trend prediction adaptive masking mechanism in step 2, and the hierarchical collaborative decision-making network in step 3.1, the task part unloading algorithm execution flow based on hierarchical SAC is formed. The algorithm iteratively optimizes the task part unloading strategy through the method of "environment interaction - experience storage - value update - strategy update". Among them, the high-level discrete strategy branch is responsible for the selection of the remote destination node, the low-level continuous strategy branch is responsible for the generation of the unloading ratio, and the shared double Q Critic network is responsible for the unified value evaluation of the joint action. During training, the current state is obtained for the computational tasks generated by the terminal. First, the high-level discrete policy network Actor outputs the unconstrained discrete policy distribution of each candidate remote target node according to equation (32); then, the trend prediction results, node-level binary mask, and the masked discrete policy distribution are calculated according to equations (25) to (31). And sample the remote destination nodes within the feasible destination node set. Finally, the unloading ratio is generated by the low-level continuous policy network Actor according to equation (33). Thus forming a joint action ( After applying this combined action to the environment, a single-step reward is obtained. Next state and termination mark and transfer samples Store in the experience replay pool; Once the number of samples in the experience replay pool reaches the set condition, a small batch of transfer samples is randomly sampled from it. Target values ​​are constructed according to equations (34) and (35), and the shared dual-Q Critic network is updated according to equation (36). Subsequently, the high-level discrete policy network Actor and the low-level continuous policy network Actor are updated respectively, with their optimization objectives being: (37) (38) in, and represents the parameters of the high-level discrete policy network and the low-level continuous policy network, respectively. After each round of updates, the target Critic network is smoothly updated. (39) in, To smoothly update the coefficients of the target network, Indicates the first The parameters of the current Critic network, Indicates the first The parameters of the target Critic network.

5. A hierarchical reinforcement learning task offloading system based on adaptive constraints under edge computing, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-5.

Images