Server-free edge cloud collaborative arrangement system and method based on hierarchical reinforcement learning

Abstract

The invention discloses a server-free edge cloud collaborative arrangement system and method based on hierarchical reinforcement learning, and belongs to the technical field of server-free computing. The system comprises a three-layer control module and a socialized cross-layer coordination module, wherein a level I ready state manager adopts an A2C algorithm and outputs an enabled ready state set; the hierarchy II deployment scheduler adopts a GCN-A2C algorithm to determine the placement and activation of the working nodes in the edge server, and outputs the ready state of the working nodes and the activation of new nodes; the hierarchy III calls the gateway cluster to adopt an MAA2C algorithm, and outputs request-working node mapping according to the request and the edge resource state; and the socialized cross-layer coordination module realizes time synchronization, information intercommunication and resource matching of three layers of decisions through a time alignment unit, a decision interaction unit and a resource coupling unit. According to the system, the problems of state explosion, difficulty in cross-layer collaboration and the like of server-free edge cloud arrangement are solved, the service quality is improved, the cost is reduced, the load is balanced, and the overall performance of the system is optimized.

Description

Technical Field

[0001] This invention relates to the field of serverless computing technology, and in particular to a serverless edge cloud collaborative orchestration system and method based on hierarchical reinforcement learning. Background Technology

[0002] With the rapid development of cloud-native applications, serverless edge computing has become a crucial infrastructure for supporting event-driven execution. Mainstream platforms such as AWS Lambda, Azure Functions, and Google Cloud Functions abstract infrastructure management, allowing developers to focus on application logic development.

[0003] Serverless computing is widely used in dynamic workloads such as real-time analytics, AI inference, and microservices, placing stringent demands on high performance and cost-effectiveness. Modern serverless applications require complex orchestration and coordination across multiple system layers to ensure optimal Quality of Service (QoS). A typical serverless edge cloud architecture includes three core control layers: a ready-to-configure layer (responsible for managing the warm-up state of function containers), a worker node deployment layer (responsible for placing and activating worker nodes on edge servers), and a request scheduling layer (responsible for dispatching requests to appropriate worker nodes). Although these layers have different resource requirements and time scales, they collectively determine end-to-end latency, throughput, and resource efficiency. Therefore, cross-layer orchestration and coordination are crucial to system efficiency.

[0004] Existing serverless platforms suffer from fundamental flaws in resource preparation, characterized by coarse-grained and rigid configuration, leading to service quality degradation and inefficient resource utilization. Current serverless platforms employ fixed instance state patterns (e.g., cold, hot), failing to differentiate between requests based on specific Service Level Objectives (SLOs) constraints or execution urgency. This rigid configuration mechanism triggers unnecessary warm-up operations for low-priority requests while delaying latency-sensitive critical requests. Lacking an adaptive, multi-granularity readiness tuning mechanism based on real-time system status and request priorities, existing systems cannot guarantee service quality and limit resource efficiency improvements.

[0005] Meanwhile, existing serverless edge cloud systems suffer from severe deficiencies in cross-layer coordination, manifesting as fragmented control and unstable learning processes. Although serverless edge clouds employ a layered architecture, most solutions optimize each layer independently, leading to decision-making disruptions and inconsistent strategies. Under dynamic conditions, issues such as timeline misalignment, goal decoupling, and the lack of coordination protocols further exacerbate system instability and overall performance degradation. Each control layer makes isolated decisions based on local observations, which may achieve local optima at a single layer but results in global inefficiency. Especially when facing sudden workloads, diverse priority requirements, and strict latency constraints, this isolated control model cannot provide an effective coordination mechanism, severely limiting the system's adaptability and overall performance in dynamic environments. Summary of the Invention

[0006] The purpose of this invention is to provide a serverless edge cloud collaborative orchestration system and method based on hierarchical reinforcement learning, which solves problems such as state explosion and difficulty in cross-layer collaboration in serverless edge cloud orchestration, improves service quality, reduces costs, balances load, and optimizes the overall system performance.

[0007] To achieve the above objectives, this invention provides a serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning, comprising: Level III Call Gateway Cluster: Its function is to schedule requests and assign tasks to appropriate worker nodes. It uses the MAA2C ​​algorithm and outputs the request-worker node mapping X based on the request SLO and edge resource status. Tier II Deployment Scheduler: Deploys worker nodes using the GCN-A2C algorithm, determines the placement and activation of worker nodes on the edge server, and outputs the worker node readiness status Y and the new node activation Y'. Hierarchical I Ready State Manager: Employs the A2C algorithm to output the set of enabled ready states G based on the global SLO violation rate, average startup latency, and function ready cost; The socialized cross-layer coordination module includes a time alignment unit, a decision interaction unit, and a resource coupling unit, which realizes time synchronization, information exchange, and resource matching for three-layer decision-making.

[0008] Preferably, the Level III call gateway cluster consists of an independent intelligent agent for each gateway. In serverless environments, HRL is executed as follows: Under resource constraints, the gateway is invoked to schedule requests to satisfy SLOs, and then SLO violations are minimized and load balancing is achieved. ; ; ; in, This represents the short-term SLO violation rate, which is used as a measure of long-term Quality of Service (QoS). Real-time corresponding items; and As a weighting factor; This indicates that within the current scheduling period, the gateway node... e The number of requests received; For the request Corresponding function The service priority weight is used to distinguish the impact of different services in QoS violation statistics; For indicator functions; This is a load balancing metric for the system, used to measure the balance of resource usage among edge servers, and its value ranges from [0,1]. Indicates a request The end-to-end processing latency includes the sum of queuing latency, transmission latency, computation latency, and startup latency; Indicates a request SLO delay constraints; Indicates calling the gateway e The set of requests received during the current scheduling period; Represents edge server n The available storage resource capacity at the current moment; In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: the gateway application uses the MAA2C ​​algorithm to dynamically balance the workload among edge servers, while improving cost efficiency. state S D,t Each gateway e In time t Observation Request r j,e local state s j,e D,t , r j,e Indicates that the gateway is called e The first received j One request; status S D,t Specifically defined as: ; in, For the request The type of the objective function; The Service Level Objective (SLO) constraint for the requested service level is the latency constraint. Representation function Business priority weights; Indicates calling the gateway e The status of the request queue; For working nodes The queue state; Represents edge server n Available computing resources; For edge servers n Available storage resources; To call the gateway e Number of edge servers under management; Represents edge server n The number of working nodes on it; Global state aggregates local state: ; This indicates that the system calls the gateway set; action A D,t The decision space includes discrete actions. This indicates which worker node the request should be scheduled to; the joint decision is: ; This indicates the union calculation; award R D,t : at time step t Shared rewards guide collaboration across call gateways: ; Each floor Learning strategies t To optimize long-term rewards, where the expected return is: ; in, For control layer The initial system state, For control layer The initial action, , They are time points t state and action, Indicates at time t By control layer The immediate rewards received This indicates the strategy adopted by this layer, which determines the probability of selecting an action in each state. H The accumulated time span for rewards [·] represents the expectation operator for action sequences and environment transitions; The value function is: It facilitates scalable policy learning under multi-granular dynamics and heterogeneous resources.

[0009] Preferably, the Level II deployment scheduler, In serverless environments, the specific execution deployment scheduler sub-problem of HRL suffers from limited visibility into upper-layer request dynamics and a lack of direct observation of lower-layer execution. Therefore, an indirect metric is used: minimizing the request queue length of worker nodes. The optimization objective is defined as: ; ; ; ; ; in, , It balances the reduction of queue backlog with the deployment costs of maintaining multi-granularity ready configurations; This represents the set of edge servers in the system. For any one of the servers, Indicates the size of the requested data; Representation function In its initial ready state The required storage resources; Indicates server The available storage resource capacity at the current moment; Specifically implemented in the multi-granularity resource-driven HRL mechanism: Graph Convolutional Neural Network-A2C, namely GCN-A2C, is used to optimize the deployment of worker nodes in serverless edge clouds. Spatial and temporal correlations are captured through graph embedding, and scalable and adaptive policy learning is achieved under dynamic workloads. GCN Embedded Construction: Deploying the Scheduler for Each Worker Node Construct a virtual graph, encoding its local execution and resource states in time slot κ as follows: , Each worker node aggregates adjacent embeddings: ; in, This is the current readiness level; This refers to the type of function that the worker node is running; This is the current queue length of the worker node, i.e., the number of requests waiting. This is the business priority weight of the function; This refers to the startup delay in the current ready state; It is the maintenance cost in the current ready state; in, h0(·) and b0(·) are nonlinear transformations based on neural networks; It is a working node In the time slot Graph embedding vectors; yes Set Removal The set that remains afterward; It is a server n The set of all working nodes; Then, GCN is embedded in a hierarchical aggregation across three levels: Edge server embedding: Worker node-level embeddings are aggregated into edge server embeddings: ; in, Indicates in time slot Below, by the server All working nodes within the server-level graph are embedded and aggregated. Invoke gateway embedding: Edge-level embedding, then aggregation at the gateway level: : ; It is a gateway. e In the time slot Below, the gateway-level graph embedding is obtained by the aggregation of server-level embeddings under its jurisdiction; System embedding: Finally, the gateway-level embedding is called at the global level, through... Aggregation into system embedding; State S W,κ In each time slot κ, the input state includes all embeddings generated by GCN: ; Action A W,κ The deployment scheduler determines the deployment decision. ; award The deployment scheduler receives data from... The resulting reward: .

[0010] Preferably, the level I ready state manager, In serverless environments, the specific execution of the HRL's ready state manager sub-problem involves selecting a ready state to balance deployment flexibility and cost. A subset of [the previous layer]. Due to a lack of access to lower-level actions, this layer relies on aggregated performance metrics, including latency violations and startup latency, with the goal of maximizing cost-normalized responsiveness. ; in, , and This represents the average SLO violation rate and startup delay penalty within the scheduling window; Adjust the relative weights of illegitimate and non-illegitimate requests in the delay assessment. and It is a weighting factor; Compared to coarse-grained monolithic and two-layer designs, the layered framework decouples readiness, deployment, and scheduling into three layers, each with its own specific goals and strategies, reducing decision-making complexity and improving scalability under dynamic workloads. In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: ready state configuration is performed via A2C; the ready state manager selects the ready configuration. To balance readiness diversity and deployment costs, the agent employs an A2C learning strategy π R ; state Time frame The global state is defined as follows: in, and Represents the complete set and the current ready state set; and These represent the SLO violation rate and average startup latency, respectively. Encode the per-function readiness cost; It is the average response time, and Capture average startup latency, further divided into and These are used for violations and for fulfilling their SLO requests, respectively; the state also includes function weights. Ready distribution Function-specific traffic share and the resource utilization of each edge server. .

[0011] Preferably, the time alignment unit defines a "step level". t —Slot level κ—Frame level "Multi-scale structure, level III in step level" t Real-time scheduling is performed; Level II is at the slot level. κ Deployment is carried out at the layer I in the frame level. Configure the hierarchical time structure as follows: ; in It uses integer factors and nested time abstractions to balance long-term planning and short-term response capabilities. To align decision-making coupling among the three layers, a time-aligned feedback relay mechanism is introduced. This mechanism aggregates the state-action transitions and performance metrics of the fine-grained layers into high-level summary statistics, specifically within each time slot. At the end, the ready state manager observes the aggregated SLO violation rate. and average startup delay Then it propagates to the top level to optimize the ready state configuration in the next decision cycle. ; To support responsive adaptation under dynamic workloads, the time alignment unit introduces an event-driven mechanism. This mechanism triggers hierarchical control based on system metrics, activating ready-state reconfiguration under global degradation and triggering deployment adjustments under local overload. Two thresholds are defined: Used for global responsiveness. Used for local queue pressure; time step t The global metrics are: in, and express The system-wide SLO violation rate and average startup latency during the period; among which, This is the size of the statistical window, used to smooth out system performance fluctuations; , It is a non-negative weighting coefficient used to adjust the relative impact of default rate and startup delay on system indicators; The ready state reconfiguration is triggered when the following conditions are met: Each edge server monitors local queue pressure: ; A partial deployment adjustment is triggered if the following conditions are met: Define a control escalation mechanism and determine each decision-making moment. t The required intervention level, and the escalation logic formalized as follows: ; in, The control levels for activation are: 1 for ready state reconfiguration, 2 for deployment adjustment, and 0 for passive monitoring; the system accordingly calls the corresponding reinforcement learning agent to perform fine-grained policy updates. if The ready state manager updates by solving the following problem: ; if Optimize local deployments of overloaded edge servers: ; All actions are constrained by current resources and scheduling to ensure feasibility; the mechanism responds to anomalies by activating the control layer, balancing stability and agility to ensure QoS with minimal overhead.

[0012] Preferably, the decision interaction unit adopts a two-way coordination mechanism to achieve efficient cross-layer data and decision flow, so as to reduce overload and enhance overall performance. This includes a top-down guidance mechanism and a bottom-up reporting mechanism, in which the upper layer sends binary guidance vectors to the lower layer to refine the strategy logits, and the lower layer sends status information to the upper layer to adjust the upper layer's decision. Furthermore, the reward for each layer combines local intrinsic rewards and global extrinsic rewards. In a top-down guidance mechanism, to improve decision-making accuracy, the gateway is invoked. e routing decisions The vector is sent to the deployment scheduler, which generates a binary guidance vector. To indicate actions that cannot be taken; gateway policy logits Refinement is achieved through element-wise multiplication: ; The deployment scheduler uses transformations based on historical resources and queue states. Calculate the next state: ; in, This guidance mechanism reduces the need for continuous resource uploads, alleviates system load, and enhances lower-level decision-making. In a bottom-up reporting mechanism, to enhance the accuracy of the deployment scheduler's decisions, the deployment scheduler is allowed to utilize guidance from lower layers and invoke the gateway to process feedback from upper layers. and Generate binary vectors The deployment scheduler adjusts its logits in the following ways. : . This bottom-up reporting approach incorporates local context into upper-level decision-making, reducing misalignment and improving coordination. In cross-layer reinforcement rewards, each agent receives a shaping reward that combines local goals and global signals; each agent Receive shaping rewards Combined with intrinsic rewards and global external signals reflecting system-wide QoS and efficiency : ; in, It's weight. yes The external value function is adjusted to include a global reward aligned with the objective P0. This coupling avoids selfish optimization and promotes the evolution of cooperative strategies across multiple granular levels, improving training stability and system-wide efficiency.

[0013] Preferably, the resource coupling unit calculates the matching degree between the request distribution and the deployment distribution through Jensen-Shannon divergence (JSD), adjusts the request scheduling strategy through a bottom-up conformity mechanism, and adjusts the worker node deployment strategy through a top-down skew adjustment mechanism. The bottom-up approach mechanism aligns the request scheduling of the gateway with the deployment scheduler configuration, reducing cold starts and eliminating redundant reconfiguration caused by demand-supply mismatch; To schedule to edge servers function The number of requests; the aggregated request distribution is as follows: and ;set up Deploy distribution for the standardized function, where It is a function The number of active worker nodes; using JSD to evaluate the alignment between requests and deployment distributions, i.e.: ; A smaller JSD indicates better alignment between function supply and demand, promotes consistent configuration, and reduces cross-layer mismatches; The top-down skew adjustment mechanism enables the deployment scheduler to proactively reallocate resources based on real-time demand. This indicates that it has been scheduled and is on the edge server. Queuing function The number of requests; and the record of gateway routing decisions used for long-term alignment. different, The aggregated request strength across edge servers is defined as reflecting the short-term load dynamics observed by the deployment scheduler: ; use Quantify actual needs and current deployment Mismatches between functions, with larger values ​​indicating a deployment-requirement mismatch, prompt adjustments. These adjustments can include adding worker nodes for high-demand functions, increasing readiness status to reduce startup, and reducing the configuration of underutilized functions.

[0014] This invention also provides a serverless edge cloud collaborative orchestration method based on hierarchical reinforcement learning, comprising the following steps: S1: The Tier I Ready State Configuration Layer performs global resource readiness management: The ready state manager uses an A2C algorithm learning strategy to observe the global state; it outputs ready state configuration decisions to determine the set of enabled ready states; among them, the baseline cold start state and the maximum hot start state are always enabled, while intermediate ready states are selectively enabled based on decision variables; it updates the strategy according to the reward function to balance system service quality defaults, startup latency, and ready configuration overhead; and it distributes the configuration decisions to the Tier II Deployment Scheduler. S2: Level II worker node deployment layer performs fine-grained deployment scheduling: The deployment scheduler adopts the GCN-A2C algorithm learning strategy; constructs a virtual graph of worker nodes; generates hierarchical embeddings through a three-layer graph convolutional network, including worker node-level embeddings, edge server-level embeddings, call gateway-level embeddings, and system-level embeddings; outputs deployment decisions based on the graph convolutional network embedding state and the ready state set issued by Level I; the deployment decisions include the adjustment of the ready state of each worker node and the type of function newly started on each edge server; wherein the ready state of the worker node includes the deactivated state and multiple ready levels; according to the reward function update strategy, the ratio of worker node queue length to deployment cost is minimized; the deployment decisions are issued to the Level III call gateway; S3: Layer III Request Scheduling Layer performs real-time request allocation: Each call gateway uses the MAA2C ​​algorithm to learn a collaboration strategy; for each arriving request, it observes the local state; the local state includes the request's target function type, service level target constraints, function priority weight, call gateway queue state, worker node queue state, available computing resources and storage resources of each edge server, the set of edge servers under its jurisdiction, and the number of worker nodes on each server; based on the observed local state and the deployment decision issued by Layer II, it outputs a scheduling decision to allocate the request to the target worker node; according to the shared reward function update strategy, it minimizes the service level target default rate and load imbalance; the request is allocated to the target worker node for execution and execution feedback is collected; S4: A socialized cross-layer coordination mechanism enables synchronized optimization of three-layer decision-making: Multi-timescale decisions are synchronized through a time alignment mechanism; Layer III operates at the step level for millisecond-level response, Layer II at the time slot level for second-level scheduling, and Layer I at the frame level for minute-level configuration; at the end of each time slot, fine-grained layer state transitions and performance metrics are aggregated into high-level summary statistics; an event-driven adaptation mechanism is adopted, triggering ready state reconfiguration when the global performance indicator exceeds a threshold, and triggering local deployment adjustments when local queue pressure exceeds a threshold; a two-way coordination channel is established through a decision interaction mechanism, with upper layers sending guidance information to lower layers indicating inactive actions, and lower layers reporting execution feedback to upper layers for policy adjustments; a cross-layer reinforcement reward shaping mechanism aligns global goals, with each agent receiving shaping rewards combining intrinsic rewards and global extrinsic signals; a resource coupling mechanism bi-directionally adjusts supply and demand matching, evaluating the alignment between request allocation and function deployment from the bottom up, and triggering adjustments from the top down based on the mismatch between real-time needs and the current deployment.

[0015] Preferably, in S1, the global state includes a complete set of ready states, a set of currently enabled states, a service level target default rate, an average startup latency, a ready cost for each function, an average response time, a function priority weight, a ready state distribution, a function traffic share, and a resource utilization rate for each edge server.

[0016] Preferably, in S1, the local state encoding of the worker node includes the current ready state, the type of bound function, the available computing resources of the edge server, the available storage resources, the worker node queue length, the server identifier, the function priority, the startup delay, and the maintenance cost.

[0017] Therefore, the serverless edge cloud collaborative orchestration system and method based on hierarchical reinforcement learning described above have the following beneficial effects: 1) A resource-driven divide-and-conquer hierarchical reinforcement learning (HRL) framework is proposed, decomposing serverless orchestration into three sub-tasks: ready state configuration, worker node deployment, and request scheduling. This framework enables each control layer to handle task complexity at its native granularity while maintaining consistency between local and global objectives. By modeling fine-grained ready states and request priorities, the convergence process is accelerated, the state space explosion problem is mitigated, and scalable coordination under dynamic workloads is ensured. Specific technical contributions include: designing a multi-level ready state model to support fine-grained configuration from cold to hot states; establishing a request priority-aware resource allocation mechanism to dynamically adjust resource configuration according to service level objectives; and employing a divide-and-conquer strategy to reduce the computational complexity of multi-level coordination and improve the scalability of the system under large-scale deployment.

[0018] 2) A socialized coordination mechanism is introduced, connecting heterogeneous control layers through three core technologies: a time alignment mechanism for synchronizing multi-timescale decisions and resolving timescale differences at the millisecond level of the scheduling layer, the second level of the deployment layer, and the minute level of the configuration layer; a decision interaction mechanism for achieving bidirectional decision coordination through bottom-up reporting and top-down guidance; and a resource coupling mechanism for aligning supply and demand dynamics to achieve precise resource matching. These mechanisms collectively support adaptive orchestration under workload fluctuations. Specific implementations include: establishing an event-driven control activation mechanism that triggers corresponding level control adjustments based on global responsiveness indicators and local queue pressure; designing a cross-layer reinforcement reward shaping mechanism that combines global goals with local rewards to avoid selfish optimization; and proposing a bidirectional resource coordination mechanism that uses Jensen-Shannon divergence to quantify the degree of demand-supply matching and guide the dynamic reallocation of resources.

[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a multi-granularity and resource-driven HRL framework diagram for serverless environments according to an embodiment of the present invention; Figure 2 This is a diagram illustrating the bidirectional coordination and reward shaping of an embodiment of the present invention; Figure 3 This is a diagram of the bidirectional resource coordination mechanism according to an embodiment of the present invention; Figure 4 This is a comparison chart of the learning convergence performance of the Sandwich framework and the baseline method in this embodiment of the invention; Figure 5 This is a comparison chart of system performance metrics between the Sandwich framework of this invention and the baseline method. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0023] Example 1 This invention provides a serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning, using a multi-granularity and resource-driven hierarchical reinforcement learning (HRL) framework for serverless environments, such as... Figure 1 As shown, it includes: Level III Call Gateway Cluster: Its function is to schedule requests and assign tasks to appropriate worker nodes. It adopts the MAA2C ​​algorithm and outputs the request-worker node mapping X based on the request service level target SLO and edge resource status.

[0024] Level III calls the gateway cluster, with each gateway being an independent intelligent agent.

[0025] In serverless environments, HRL is executed as follows: Under resource constraints, the gateway schedules requests to satisfy the Service Limits (SLOs), and then SLO violations are minimized and load balancing is achieved. ; ; ; in, This represents the short-term SLO violation rate, which is used as a measure of long-term Quality of Service (QoS). Real-time corresponding items; and As a weighting factor; This indicates that within the current scheduling period, the gateway node... e The number of requests received; For the request Corresponding function The service priority weight is used to distinguish the impact of different services in QoS violation statistics; For indicator functions; This is a load balancing metric for the system, used to measure the balance of resource usage among edge servers, and its value ranges from [0,1]. Indicates a request The end-to-end processing latency includes the sum of queuing latency, transmission latency, computation latency, and startup latency; Indicates a request SLO delay constraints; Indicates calling the gateway e The set of requests received during the current scheduling period; Represents edge server n The available storage resource capacity at the current moment; In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: the gateway application uses the MAA2C ​​algorithm to dynamically balance the workload among edge servers, while improving cost efficiency. state S D,t Each gateway e In time t Observation Request r j,e local state s j,e D,t , r j,e Indicates that the gateway is called e The first received j One request; status S D,t Specifically defined as: ; in, For the request r j,e The type of the objective function; The Service Level Objective (SLO) constraint for the requested service level is the latency constraint. Representation function Business priority weights; Indicates calling the gateway e The status of the request queue; For working nodes The queue state; Represents edge server n Available computing resources; For edge servers n Available storage resources; To call the gateway e Number of edge servers under management; Represents edge server n The number of working nodes on it.

[0026] Global state aggregates local state: ; This indicates that the system calls the gateway set; action A D,t The decision space includes discrete actions. This indicates which worker node the request should be scheduled to; the joint decision is: ; This indicates the union calculation; award R D,t : at time step t Shared rewards guide collaboration across call gateways: ; Each floor Learning strategies t To optimize long-term rewards, where the expected return is: ; in, For control layer The initial system state, For control layer The initial action, , They are time points t state and action, Indicates at time t By control layer The immediate rewards received This indicates the strategy adopted by this layer, which determines the probability of selecting an action in each state. H The accumulated time span for rewards [·] represents the expectation operator for action sequences and environment transitions; The value function is: It facilitates scalable policy learning under multi-granular dynamics and heterogeneous resources.

[0027] Tier II Deployment Scheduler: Deploys worker nodes using the GCN-A2C algorithm, determines the placement and activation of worker nodes on the edge server, and outputs the worker node readiness status Y and the new node activation Y'.

[0028] The Tier II deployment scheduler, in the serverless HRL execution deployment scheduler subproblem, suffers from limited visibility into upper-layer request dynamics and a lack of direct observation of lower-layer execution. Therefore, it employs an indirect metric: minimizing the request queue length of worker nodes. The optimization objective is defined as: ; ; ; ; ; in, , It balances the reduction of queue backlog with the deployment costs of maintaining multi-granularity ready configurations; This represents the set of edge servers in the system. For any one of the servers, Indicates the size of the requested data; Representation function In its initial ready state The required storage resources; Indicates server The available storage resource capacity at the current moment; Specifically implemented in the multi-granularity resource-driven HRL mechanism: Graph Convolutional Neural Network-A2C, i.e. GCN-A2C, is used to optimize the deployment of worker nodes in serverless edge clouds. By capturing spatial and temporal correlations through graph embedding, scalable and adaptive policy learning is achieved under dynamic workloads.

[0029] GCN Embedded Construction: Deploying the Scheduler for Each Worker Node Construct a virtual graph, encoding its local execution and resource states in time slot κ as follows: , Each worker node aggregates adjacent embeddings: ; in, This is the current readiness level; This refers to the type of function that the worker node is running; This is the current queue length of the worker node, i.e., the number of requests waiting. This is the business priority weight of the function; This refers to the startup delay in the current ready state; It is the maintenance cost in the current ready state; in, h0(·) and b0(·) are nonlinear transformations based on neural networks; It is a working node In the time slot Graph embedding vectors; yes Set Removal The set that remains afterward; It is a server nThe set of all working nodes; Then, GCN is embedded in a hierarchical aggregation across three levels: Edge server embedding: Worker node-level embeddings are aggregated into edge server embeddings: ; in, Indicates in time slot Below, by the server All working nodes within the server-level graph are embedded and aggregated. Invoke gateway embedding: Edge-level embedding, then aggregation at the gateway level: : ; It is a gateway. e In the time slot Below, the gateway-level graph embedding is obtained by the aggregation of server-level embeddings under its jurisdiction; System embedding: Finally, the gateway-level embedding is called at the global level, through... Aggregation into system embedding; State S W,κ In each time slot κ, the input state includes all embeddings generated by GCN: ; Action A W,κ The deployment scheduler determines the deployment decision. ; award The deployment scheduler receives data from... The resulting reward: .

[0030] Hierarchical I Ready State Manager: Employs the A2C algorithm to output an enabled set of ready states G based on the global SLO violation rate, average startup latency, and function ready cost.

[0031] The Hierarchical I Ready State Manager, in serverless environments, specifically addresses the execution of the HRL's ready state manager sub-problem. To balance deployment flexibility and cost, the ready state manager selects ready states. A subset of [the previous layer]. Due to a lack of access to lower-level actions, this layer relies on aggregated performance metrics, including latency violations and startup latency, with the goal of maximizing cost-normalized responsiveness. ; in, , and This represents the average SLO violation rate and startup delay penalty within the scheduling window; Adjust the relative weights of illegitimate and non-illegitimate requests in the delay assessment. and It is a weighting factor; Compared to coarse-grained monolithic and two-layer designs, the layered framework decouples readiness, deployment, and scheduling into three layers, each with its own specific goals and strategies, reducing decision-making complexity and improving scalability under dynamic workloads.

[0032] In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: ready state configuration is performed via A2C; the ready state manager selects the ready configuration. To balance readiness diversity and deployment costs, the agent employs an A2C learning strategy π R ; State S R,ς The global state of time frame ς is defined as follows: in, and Represents the complete set and the current ready state set; and These represent the SLO violation rate and average startup latency, respectively. Encode the per-function readiness cost; It is the average response time, and Capture average startup latency, further divided into and These are used for violations and for fulfilling their SLO requests, respectively; the state also includes function weights. Ready distribution Function-specific traffic share and the resource utilization of each edge server. .

[0033] The socialized cross-layer coordination module includes a time alignment unit, a decision interaction unit, and a resource coupling unit, which realizes time synchronization, information exchange, and resource matching for three-layer decision-making.

[0034] Time alignment unit definition "step level" t —Slot level κ—Frame level "Multi-scale structure, level III in step level" t Real-time scheduling is performed; Level II is at the slot level. κ Deployment is carried out at the layer I in the frame level. Configure the hierarchical time structure as follows: ; in It uses integer factors and nested time abstractions to balance long-term planning and short-term response capabilities. To align decision-making coupling among the three layers, a time-aligned feedback relay mechanism is introduced. This mechanism aggregates the state-action transitions and performance metrics of the fine-grained layers into high-level summary statistics, specifically within each time slot. At the end, the ready state manager observes the aggregated SLO violation rate. and average startup delay Then it propagates to the top level to optimize the ready state configuration in the next decision cycle. ; To support responsive adaptation under dynamic workloads, the time alignment unit introduces an event-driven mechanism. This mechanism triggers hierarchical control based on system metrics, activating ready-state reconfiguration under global degradation and triggering deployment adjustments under local overload. Two thresholds are defined: Used for global responsiveness. Used for local queue pressure; time step t The global metrics are: in, and express The system-wide SLO violation rate and average startup latency during the period; among which, This is the size of the statistical window, used to smooth out system performance fluctuations; , It is a non-negative weighting coefficient used to adjust the relative impact of default rate and startup delay on system indicators; The ready state reconfiguration is triggered when the following conditions are met: Each edge server monitors local queue pressure: ; A partial deployment adjustment is triggered if the following conditions are met: Define a control escalation mechanism and determine each decision-making moment. t The required intervention level, and the escalation logic formalized as follows: ; in, The control levels for activation are: 1 for ready state reconfiguration, 2 for deployment adjustment, and 0 for passive monitoring; the system accordingly calls the corresponding reinforcement learning agent to perform fine-grained policy updates. if The ready state manager updates by solving the following problem: ; if Optimize local deployments of overloaded edge servers: ; All actions are constrained by current resources and scheduling to ensure feasibility; the mechanism responds to anomalies by activating the control layer, balancing stability and agility to ensure QoS with minimal overhead.

[0035] The decision-making interaction unit employs a two-way coordination mechanism to achieve efficient cross-layer data and decision flow, reducing overload and enhancing overall performance. This includes a top-down guidance mechanism and a bottom-up reporting mechanism. Upper layers send binary guidance vectors to lower layers to refine strategy logits, while lower layers send status information to upper layers to adjust upper-layer decisions. Furthermore, each layer's reward combines local intrinsic rewards with global extrinsic rewards. This two-way coordination and reward shaping mechanism is as follows: Figure 2 As shown.

[0036] In a top-down guidance mechanism, to improve decision-making accuracy, the gateway is invoked. e routing decisions The vector is sent to the deployment scheduler, which generates a binary guidance vector. To indicate actions that cannot be taken; gateway policy logits Refinement is achieved through element-wise multiplication: ; The deployment scheduler uses transformations based on historical resources and queue states. Calculate the next state: ; in, This guidance mechanism reduces the need for continuous resource uploads, alleviates system load, and enhances lower-level decision-making.

[0037] In a bottom-up reporting mechanism, to enhance the accuracy of the deployment scheduler's decisions, the deployment scheduler is allowed to utilize guidance from lower layers and invoke the gateway to process feedback from upper layers. and Generate binary vectors The deployment scheduler adjusts its logits in the following ways. : . This bottom-up reporting approach incorporates local context into upper-level decision-making, reducing misalignment and improving coordination.

[0038] In cross-layer reinforcement rewards, each agent receives a shaping reward that combines local goals and global signals; each agent Receive shaping rewards Combined with intrinsic rewards and global external signals reflecting system-wide QoS and efficiency : ; in, It's weight. yes The external value function is adjusted to include a global reward aligned with the objective P0. This coupling avoids selfish optimization and promotes the evolution of cooperative strategies across multiple granular levels, improving training stability and system-wide efficiency.

[0039] The resource coupling unit calculates the matching degree between request distribution and deployment distribution through Jensen-Shannon divergence (JSD), adjusts the request scheduling strategy through a bottom-up conformity mechanism, and adjusts the worker node deployment strategy through a top-down skew adjustment mechanism.

[0040] The bottom-up approach mechanism aligns the request scheduling of the gateway with the deployment scheduler configuration, reducing cold starts and eliminating redundant reconfiguration caused by demand-supply mismatch; To schedule to edge servers function The number of requests; the aggregated request distribution is as follows: and ;set up Deploy distribution for the standardized function, where It is a function The number of active worker nodes; using JSD to evaluate the alignment between requests and deployment distributions, i.e.: ; Smaller JSDs indicate better alignment between function supply and demand, promoting consistent configuration and reducing cross-layer mismatches.

[0041] The top-down skew adjustment mechanism enables the deployment scheduler to proactively reallocate resources based on real-time demand. This indicates that it has been scheduled and is on the edge server. Queuing function The number of requests; and the record of gateway routing decisions used for long-term alignment. different, The aggregated request strength across edge servers is defined as reflecting the short-term load dynamics observed by the deployment scheduler: ; use Quantify actual needs and current deployment Mismatches between functions, with larger values ​​indicating a deployment-requirement mismatch, prompt adjustments. These adjustments can include increasing worker nodes for high-demand functions, raising their readiness status to reduce startup, and reducing the configuration of underutilized functions. Two-way resource coordination mechanisms, such as... Figure 3 As shown.

[0042] The method for serverless edge cloud collaborative orchestration based on hierarchical reinforcement learning using this system includes the following steps: S1: The Tier I Ready State Configuration Layer performs global resource readiness management: The ready state manager uses an A2C algorithm learning strategy to observe the global state; the global state includes the complete ready state set, the currently enabled state set, the service level target default rate, the average startup latency, the ready cost of each function, the average response time, the function priority weight, the ready state distribution, the function traffic ratio, and the resource utilization of each edge server; it outputs a ready state configuration decision to determine the set of enabled ready states; the baseline cold start state and the maximum hot start state are always enabled, while intermediate ready states are selectively enabled based on decision variables; it balances system service quality defaults, startup latency, and ready configuration overhead according to the reward function update strategy; and it distributes the configuration decision to the Tier II deployment scheduler; S2: Level II worker node deployment layer performs fine-grained deployment scheduling: The deployment scheduler adopts the GCN-A2C algorithm learning strategy; it constructs a virtual graph of worker nodes, and the local state encoding of each worker node includes the current ready state, bound function type, available computing resources on the edge server, available storage resources, worker node queue length, server identifier, function priority, startup latency, and maintenance cost; it generates hierarchical embeddings through a three-layer graph convolutional network, including worker node-level embedding, edge server-level embedding, call gateway-level embedding, and system-level embedding; it outputs deployment decisions based on the graph convolutional network embedding state and the ready state set issued by Level I; the deployment decisions include the adjustment of the ready state of each worker node and the newly started function type on each edge server; the ready state of the worker node includes the deactivated state and multiple ready levels; it minimizes the ratio of worker node queue length to deployment cost according to the reward function update strategy; and it issues the deployment decisions to the Level III call gateway. S3: Layer III Request Scheduling Layer performs real-time request allocation: Each call gateway uses the MAA2C ​​algorithm to learn a collaboration strategy; for each arriving request, it observes the local state; the local state includes the request's target function type, service level target constraints, function priority weight, call gateway queue state, worker node queue state, available computing resources and storage resources of each edge server, the set of edge servers under its jurisdiction, and the number of worker nodes on each server; based on the observed local state and the deployment decision issued by Layer II, it outputs a scheduling decision to allocate the request to the target worker node; according to the shared reward function update strategy, it minimizes the service level target default rate and load imbalance; the request is allocated to the target worker node for execution and execution feedback is collected; S4: A socialized cross-layer coordination mechanism enables synchronized optimization of three-layer decision-making: Multi-timescale decisions are synchronized through a time alignment mechanism; Layer III operates at the step level for millisecond-level response, Layer II at the time slot level for second-level scheduling, and Layer I at the frame level for minute-level configuration; at the end of each time slot, fine-grained layer state transitions and performance metrics are aggregated into high-level summary statistics; an event-driven adaptation mechanism is adopted, triggering ready state reconfiguration when the global performance indicator exceeds a threshold, and triggering local deployment adjustments when local queue pressure exceeds a threshold; a two-way coordination channel is established through a decision interaction mechanism, with upper layers sending guidance information to lower layers indicating inactive actions, and lower layers reporting execution feedback to upper layers for policy adjustments; a cross-layer reinforcement reward shaping mechanism aligns global goals, with each agent receiving shaping rewards combining intrinsic rewards and global extrinsic signals; a resource coupling mechanism bi-directionally adjusts supply and demand matching, evaluating the alignment between request allocation and function deployment from the bottom up, and triggering adjustments from the top down based on the mismatch between real-time needs and the current deployment.

[0043] like Figure 4 As shown, the system and method described in this embodiment exhibit rapid convergence and the highest reward during training. Comparative experiments reveal that compared to four existing mainstream baseline methods: Demeter (a combination of graph neural networks and deep reinforcement learning for each layer), Cur-Coedge (using curiosity-driven deep reinforcement learning but with limited decision-making and resource interaction capabilities), EdgeTimer (using deep reinforcement learning but with limited time adaptability), and MAIC (using deep reinforcement learning but with limited decision-making interaction capabilities), this invention achieves a significant performance breakthrough. This embodiment achieves an average gain of 108.47% and a peak gain of 112.30% in cumulative reward. The training curves clearly show that traditional isolated design methods exhibit the highest instability during learning, with drastic reward fluctuations and slow convergence, while the scheme of this invention maintains a stable upward trend, ultimately reaching the highest cumulative reward value. This superior performance stems from the multi-granularity resource-driven hierarchical reinforcement learning framework effectively decomposing the complex orchestration problem, reducing learning difficulty, while the socialized cross-layer coordination mechanism ensures the consistency and stability of decisions across layers.

[0044] like Figure 5As shown, the multi-dimensional performance comparison results demonstrate improvements in key system performance indicators. In terms of overall system indicators, a 14.36% improvement was achieved compared to the optimal baseline method, reflecting the overall optimization effect of the system across multiple dimensions such as service quality, cost efficiency, and load balancing. This improvement is mainly attributed to the collaborative optimization of the three-layer control architecture, avoiding the local optima problem that may result from single-layer optimization. Regarding service quality assurance, a 1.07% QoS improvement was achieved, particularly in better meeting service level objectives when handling latency-sensitive requests. In terms of cost control, a 6.16% reduction in system costs was successfully achieved, primarily through intelligent readiness state configuration and precise resource supply and demand matching, avoiding resource preheating and idle waste. Regarding system response performance, startup latency was reduced by 10.74%, which is significant for addressing the cold start problem in serverless computing and can significantly improve the user's service experience.

[0045] Therefore, this invention adopts the aforementioned serverless edge cloud collaborative orchestration system and method based on hierarchical reinforcement learning, and proposes a resource-driven divide-and-conquer hierarchical reinforcement learning (HRL) framework, decomposing serverless orchestration into three sub-tasks: ready state configuration, worker node deployment, and request scheduling. This framework enables each control layer to handle task complexity at its native granularity while maintaining consistency between local and global objectives. By modeling fine-grained ready states and request priorities, the convergence process is accelerated, mitigating the state space explosion problem and ensuring scalable coordination under dynamic workloads. A social coordination mechanism is introduced to connect heterogeneous control layers through core technologies: a time alignment mechanism for synchronizing multi-timescale decisions, resolving timescale differences at the millisecond level for the scheduling layer, the second level for the deployment layer, and the minute level for the configuration layer; a decision interaction mechanism for achieving bidirectional decision coordination through bottom-up reporting and top-down guidance; and a resource coupling mechanism for aligning supply and demand dynamics to achieve precise resource matching. These mechanisms collectively support adaptive orchestration under workload fluctuations.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning, characterized in that, include: Level III Call Gateway Cluster: Its function is to schedule requests and assign tasks to appropriate worker nodes. It uses the MAA2C ​​algorithm and outputs the request-worker node mapping X based on the request service level target SLO and edge resource status. Tier II Deployment Scheduler: Deploys worker nodes using the GCN-A2C algorithm, determines the placement and activation of worker nodes on edge servers, and outputs the worker node readiness status Y and the new node activation Y'. Hierarchical I Ready State Manager: Employs the A2C algorithm to output the set of enabled ready states G based on the global SLO violation rate, average startup latency, and function ready cost; The socialized cross-layer coordination module includes a time alignment unit, a decision interaction unit, and a resource coupling unit, which realizes time synchronization, information exchange, and resource matching for three-layer decision-making.

2. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The Level III call gateway cluster, with each gateway being an independent intelligent agent; The specific implementation of hierarchical reinforcement learning HRL in serverless systems involves: under resource constraints, invoking the gateway to schedule requests to satisfy SLOs, and then minimizing SLO violations and load balancing. ； ； ； in, This represents the short-term SLO violation rate, which is used as a measure of long-term Quality of Service (QoS). Real-time corresponding items; and As a weighting factor; This indicates that within the current scheduling period, the gateway node... e The number of requests received; For the request Corresponding function The service priority weight is used to distinguish the impact of different services in QoS violation statistics; For indicator functions; This is a load balancing metric for the system, used to measure the balance of resource usage among edge servers, and its value ranges from [0,1]. Indicates a request The end-to-end processing latency includes the sum of queuing latency, transmission latency, computation latency, and startup latency; Indicates a request SLO delay constraints; Indicates calling the gateway e The set of requests received during the current scheduling period; Represents edge server n The available storage resource capacity at the current moment; In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: the gateway application uses the MAA2C ​​algorithm to dynamically balance the workload among edge servers, while improving cost efficiency. state S D,t Each gateway e In time t Observation Request r j,e local state s j,e D,t , r j,e Indicates that the gateway is called e The first received j One request; status S D,t Specifically defined as: ； in, For the request The type of the objective function; The Service Level Objective (SLO) constraint for the requested service level is the latency constraint. Representation function Business priority weights; Indicates calling the gateway e The status of the request queue; For working nodes The queue state; Represents edge server n Available computing resources; For edge servers n Available storage resources; To call the gateway e Number of edge servers under management; Represents edge server n The number of working nodes on it; Global state aggregates local state: ; This indicates that the system calls the gateway set; action A D,t The decision space includes discrete actions. This indicates which worker node the request should be scheduled to; the joint decision is: ; This indicates the union calculation; award R D,t : at time step t Shared rewards guide collaboration across call gateways: ； Each floor Learning strategies t To optimize long-term rewards, where the expected return is: ； in, For control layer The initial system state, For control layer The initial action, , They are time points t state and action, Indicates at time t By control layer The immediate rewards received This indicates the strategy adopted by this layer, which determines the probability of selecting an action in each state. H The accumulated time span for rewards [·] represents the expectation operator for action sequences and environment transitions; The value function is: It facilitates scalable policy learning under multi-granular dynamics and heterogeneous resources.

3. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The Level II deployment scheduler In serverless environments, the specific execution deployment scheduler sub-problem of HRL suffers from limited visibility into upper-layer request dynamics and a lack of direct observation of lower-layer execution. Therefore, an indirect metric is used: minimizing the request queue length of worker nodes. The optimization objective is defined as: ； ； ； ； ； in, , It balances the reduction of queue backlog with the deployment costs of maintaining multi-granularity ready configurations; This represents the set of edge servers in the system. For any one of the servers, Indicates the size of the requested data; Representation function In its initial ready state The required storage resources; Indicates server The available storage resource capacity at the current moment; Specifically implemented in the multi-granularity resource-driven HRL mechanism: Graph Convolutional Neural Network-A2C, namely GCN-A2C, is used to optimize the deployment of worker nodes in serverless edge clouds. Spatial and temporal correlations are captured through graph embedding, and scalable and adaptive policy learning is achieved under dynamic workloads. GCN Embedded Construction: Deploying the Scheduler for Each Worker Node Construct a virtual graph, encoding its local execution and resource states in time slot κ as follows: , Each worker node aggregates adjacent embeddings: ; in, This is the current readiness level; This refers to the type of function that the worker node is running; This is the current queue length of the worker node, i.e., the number of requests waiting. This is the business priority weight of the function; This refers to the startup delay in the current ready state; It is the maintenance cost in the current ready state; in, h0(·) and b0(·) are nonlinear transformations based on neural networks; It is a working node In the time slot Graph embedding vectors; yes Set Removal The set that remains afterward; It is a server n The set of all working nodes; Then, GCN is embedded in a hierarchical aggregation across three levels: Edge server embedding: Worker node-level embeddings are aggregated into edge server embeddings: ； in, Indicates in time slot Below, by the server All working nodes within the server-level graph are embedded and aggregated. Invoke gateway embedding: Edge-level embedding, then aggregation at the gateway level: : ； It is a gateway. e In the time slot Below, the gateway-level graph embedding is obtained by the aggregation of server-level embeddings under its jurisdiction; System embedding: Finally, the gateway-level embedding is called at the global level, through... Aggregation into system embedding; State S W,κ In each time slot κ, the input state includes all embeddings generated by GCN: ； Action A W,κ The deployment scheduler determines the deployment decision. ； award The deployment scheduler receives data from... The resulting reward: 。 4. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The Level I ready state manager In serverless environments, the specific execution of the HRL's ready state manager sub-problem involves selecting a ready state to balance deployment flexibility and cost. A subset of; lacking access to lower-level actions, this layer relies on aggregated performance metrics, including latency violations and startup latency, with the goal of maximizing cost-normalized responsiveness: ； in, , and This represents the average SLO violation rate and startup delay penalty within the scheduling window; Adjust the relative weights of illegitimate and non-illegitimate requests in the delay assessment. and It is a weighting factor; Compared to coarse-grained monolithic and two-layer designs, the layered framework decouples readiness, deployment, and scheduling into three layers, each with its own specific goals and strategies, reducing decision-making complexity and improving scalability under dynamic workloads. In the multi-granularity resource-driven HRL mechanism, the following is specifically implemented: ready state configuration is performed via A2C; the ready state manager selects the ready configuration. To balance readiness diversity and deployment costs, the agent employs an A2C learning strategy π R ; state Time frame The global state is defined as follows: in, and Represents the complete and currently ready sets; and These represent the SLO violation rate and average startup latency, respectively. Encode the ready cost per function; It is the average response time, and Capture average startup latency, further divided into and These are used for violations and for fulfilling their SLO requests, respectively; the state also includes function weights. Ready distribution Function-specific traffic share and the resource utilization of each edge server. .

5. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The time alignment unit is defined as "step". t —Slot level κ—Frame level "Multi-scale structure, level III in step level" t Perform real-time scheduling; Level II is at the slot level. κ Deployment is carried out at the layer I in the frame level. Configure the hierarchical time structure as follows: ； in It uses integer factors and nested time abstractions to balance long-term planning and short-term response capabilities. To align decision-making coupling among the three layers, a time-aligned feedback relay mechanism is introduced. This mechanism aggregates the state-action transitions and performance metrics of the fine-grained layers into high-level summary statistics, specifically in each time slot. At the end, the ready state manager observes the aggregated SLO violation rate. and average startup delay Then it propagates to the top level to optimize the ready state configuration in the next decision cycle. ; To support responsive adaptation under dynamic workloads, the time alignment unit introduces an event-driven mechanism. This mechanism triggers hierarchical control based on system metrics, activating ready-state reconfiguration under global degradation and triggering deployment adjustments under local overload. Two thresholds are defined: Used for global responsiveness. Used for local queue pressure; time step t The global metrics are: in, and express The system-wide SLO violation rate and average startup latency during the period; among which, This is the size of the statistical window, used to smooth out system performance fluctuations; , It is a non-negative weighting coefficient used to adjust the relative impact of default rate and startup delay on system indicators; The ready state reconfiguration is triggered when the following conditions are met: Each edge server monitors local queue pressure: ; A partial deployment adjustment is triggered if the following conditions are met: Define a control escalation mechanism and determine each decision-making moment. t The required intervention level, and the escalation logic formalized as follows: ； in, The control levels for activation are: 1 for ready state reconfiguration, 2 for deployment adjustment, and 0 for passive monitoring; the system accordingly calls the corresponding reinforcement learning agent to perform fine-grained policy updates. if The ready state manager updates by solving the following problem: ； if Optimize local deployments of overloaded edge servers: ； All actions are constrained by current resources and scheduling to ensure feasibility; the mechanism responds to anomalies by activating the control layer, balancing stability and agility to ensure QoS with minimal overhead.

6. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The decision interaction unit adopts a two-way coordination mechanism to achieve efficient cross-layer data and decision flow, thereby reducing overload and enhancing overall performance. This includes a top-down guidance mechanism and a bottom-up reporting mechanism. The upper layer sends binary guidance vectors to the lower layer to refine the strategy logits, and the lower layer sends status information to the upper layer to adjust the upper layer's decision. Furthermore, the reward for each layer combines local intrinsic rewards and global extrinsic rewards. In a top-down guidance mechanism, to improve decision-making accuracy, the gateway is invoked. e routing decisions The vector is sent to the deployment scheduler, which generates a binary guidance vector. To indicate actions that cannot be taken; gateway policy logits Refinement is achieved through element-wise multiplication: ; The deployment scheduler uses transformations based on historical resources and queue states. Calculate the next state: ； in, This guidance mechanism reduces the need for continuous resource uploads, alleviates system load, and enhances lower-level decision-making. In a bottom-up reporting mechanism, to enhance the accuracy of the deployment scheduler's decisions, the deployment scheduler is allowed to utilize guidance from lower layers and invoke the gateway to process feedback from upper layers. and Generate binary vectors The deployment scheduler adjusts its logits in the following ways. : . This bottom-up reporting approach incorporates local context into upper-level decision-making, reducing misalignment and improving coordination. In cross-layer reinforcement rewards, each agent receives a shaping reward that combines local goals and global signals; each agent Receive shaping rewards Combined with intrinsic rewards and global external signals reflecting system-wide QoS and efficiency : ； in, It's weight. yes The external value function is adjusted to include a global reward aligned with the objective P0. This coupling avoids selfish optimization and promotes the evolution of cooperative strategies across multiple granular levels, improving training stability and system-wide efficiency.

7. The serverless edge cloud collaborative orchestration system based on hierarchical reinforcement learning according to claim 1, characterized in that, The resource coupling unit calculates the matching degree between request distribution and deployment distribution through Jensen-Shannon divergence (JSD), adjusts the request scheduling strategy through a bottom-up conformity mechanism, and adjusts the worker node deployment strategy through a top-down skew adjustment mechanism. The bottom-up approach mechanism aligns the request scheduling of the gateway with the deployment scheduler configuration, reducing cold starts and eliminating redundant reconfiguration caused by demand-supply mismatch; To schedule to edge servers function The number of requests; the aggregated request distribution is as follows: and ;set up Deploy distribution for the standardized function, where It is a function The number of active worker nodes; using JSD to evaluate the alignment between requests and deployment distributions, i.e.: ； A smaller JSD indicates better alignment between function supply and demand, promotes consistent configuration, and reduces cross-layer mismatches; The top-down skew adjustment mechanism enables the deployment scheduler to proactively reallocate resources based on real-time demand. This indicates that it has been scheduled and is on the edge server. Queuing function The number of requests; and the record of gateway routing decisions used for long-term alignment. different, The aggregated request strength across edge servers is defined as reflecting the short-term load dynamics observed by the deployment scheduler: ； use Quantify actual needs and current deployment Mismatches between functions, with larger values ​​indicating a deployment-requirement mismatch, prompt adjustments. These adjustments can include adding worker nodes for high-demand functions, increasing readiness status to reduce startup, and reducing the configuration of underutilized functions.

8. A serverless edge cloud collaborative orchestration method based on hierarchical reinforcement learning, characterized in that, Includes the following steps: S1: The ready state configuration layer at level I performs global resource ready management: the ready state manager uses an A2C algorithm learning strategy to observe the global state; Output ready state configuration decisions to determine the set of ready states to be enabled; The baseline cold start state and the maximum hot start state are always enabled, while the intermediate ready state is selectively enabled based on the decision variables. Based on the reward function update strategy, balance system service quality defaults, startup latency, and ready configuration overhead; and distribute configuration decisions to the Tier II deployment scheduler. S2: Level II worker node deployment layer performs fine-grained deployment scheduling: The deployment scheduler adopts the GCN-A2C algorithm learning strategy; constructs a virtual graph of worker nodes; generates hierarchical embeddings through a three-layer graph convolutional network, including worker node-level embeddings, edge server-level embeddings, call gateway-level embeddings, and system-level embeddings; outputs deployment decisions based on the graph convolutional network embedding state and the ready state set issued by Level I; The deployment decision includes adjusting the readiness status of each worker node and the type of function newly started on each edge server; the readiness status of the worker node includes the deactivated state and multi-level readiness levels; the ratio of the worker node queue length to the deployment cost is minimized according to the reward function update strategy; the deployment decision is then sent to the Layer III call gateway. S3: Layer III Request Scheduling Layer performs real-time request allocation: Each call gateway uses the MAA2C ​​algorithm to learn a collaboration strategy; for each arriving request, it observes the local state; the local state includes the request's target function type, service level target constraints, function priority weight, call gateway queue state, worker node queue state, available computing resources and storage resources of each edge server, the set of edge servers under its jurisdiction, and the number of worker nodes on each server; based on the observed local state and the deployment decision issued by Layer II, it outputs a scheduling decision to allocate the request to the target worker node; according to the shared reward function update strategy, it minimizes the service level target default rate and load imbalance; the request is allocated to the target worker node for execution and execution feedback is collected; S4: A socialized cross-layer coordination mechanism enables synchronized optimization of three-layer decision-making: Multi-timescale decisions are synchronized through a time alignment mechanism; Layer III operates at the step level for millisecond-level response, Layer II at the time slot level for second-level scheduling, and Layer I at the frame level for minute-level configuration; at the end of each time slot, fine-grained layer state transitions and performance metrics are aggregated into high-level summary statistics; an event-driven adaptation mechanism is adopted, triggering ready state reconfiguration when the global performance indicator exceeds a threshold, and triggering local deployment adjustments when local queue pressure exceeds a threshold; a two-way coordination channel is established through a decision interaction mechanism, with upper layers sending guidance information to lower layers indicating inactive actions, and lower layers reporting execution feedback to upper layers for policy adjustments; a cross-layer reinforcement reward shaping mechanism aligns global goals, with each agent receiving shaping rewards combining intrinsic rewards and global extrinsic signals; a resource coupling mechanism bi-directionally adjusts supply and demand matching, evaluating the alignment between request allocation and function deployment from the bottom up, and triggering adjustments from the top down based on the mismatch between real-time needs and the current deployment.

9. The serverless edge cloud collaborative orchestration method based on hierarchical reinforcement learning according to claim 8, characterized in that, In S1, the global state includes a complete set of ready states, a set of currently enabled states, a service level target default rate, an average startup latency, the ready cost of each function, an average response time, a function priority weight, a ready state distribution, a function traffic share, and the resource utilization rate of each edge server.

10. The serverless edge cloud collaborative orchestration method based on hierarchical reinforcement learning according to claim 8, characterized in that, In S1, the local state encoding of the worker node includes the current ready state, the type of bound function, the available computing resources of the edge server, the available storage resources, the worker node queue length, the server identifier, the function priority, the startup delay, and the maintenance cost.

Images