A hybrid runtime function placement method based on deep reinforcement learning in edge-cloud collaborative environment

Abstract

This invention discloses a hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment. By constructing a weighted undirected graph network including cloud center and edge nodes and a dual-mode hybrid runtime environment of containers and WebAssembly, a Markov decision process model for service function chain placement is established. Combining a probability-aware cold start prediction model and a multi-objective composite reward function, a latency-aware near-end strategy optimization algorithm with temporal feature extraction and dynamic action masking is used for model training and solving. Intelligent node selection and runtime allocation are performed on dynamically arriving microservice functions, thereby effectively avoiding cold start latency under burst traffic, significantly reducing system end-to-end latency and total energy consumption, and achieving efficient utilization and load balancing of edge cluster resources.

Description

Technical Field

[0001] This invention relates to the fields of edge computing, serverless computing, and network resource allocation, and in particular to a hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment. Background Technology

[0002] The deep integration of IoT technology and 5G mobile communication networks has driven the widespread access of massive numbers of terminal devices at the network edge. To address the challenges of centralized cloud computing in processing large-scale edge data, such as limited backhaul bandwidth and excessive end-to-end response latency, the computing paradigm is shifting towards an edge-cloud collaborative architecture. Serverless computing, with its fine-grained resource allocation mechanism and automatic scaling capabilities, has become an ideal architecture for edge scenarios.

[0003] In related technologies, migrating serverless architectures to edge nodes faces the dual constraints of heterogeneous runtime and cold start problems. Traditional container technologies, such as Docker, involve cumbersome steps like image loading during cold start, often requiring hundreds of milliseconds or even seconds, failing to meet the real-time requirements of edge applications. While emerging lightweight sandbox technology WebAssembly can achieve millisecond-level startup, its performance is slightly inferior to native containers when performing computationally intensive tasks. Furthermore, most existing deep reinforcement learning-based scheduling schemes assume homogeneous container environments, treating cold start as a fixed penalty, lacking effective prediction of cold start probability under burst traffic, and struggling to find the optimal balance between latency, energy consumption, and load balancing. Summary of the Invention

[0004] To intelligently place service function chains in an edge-cloud collaborative environment, solve the problems of high cold start latency and uneven distribution of heterogeneous resources in traditional containers, and achieve low latency, low energy consumption and high stability of the system, this application provides a hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment.

[0005] This application provides a hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment, employing the following technical solution:

[0006] S1: Construct an edge-cloud collaborative network model and a hybrid runtime resource model, wherein the hybrid runtime includes container runtime and WebAssembly runtime, and establish computation latency and transmission latency models for functions under different nodes and different runtimes;

[0007] S2: Construct a probability-aware cold start latency model to calculate the probability that a function instance is in an active resident state on different nodes, and then obtain the expected cold start latency to quantify the impact of cold start on system performance.

[0008] S3: Model the function placement problem of the service function chain as a partially observable Markov decision process, and construct a joint state space that includes global resource state, service function chain request characteristics and probabilistic prediction characteristics;

[0009] S4: Construct an action space to simultaneously determine the target node and runtime type of the function placement, and introduce a dynamic action masking mechanism to generate a mask vector based on the real-time remaining resources of each node and the runtime constraints of the function to filter out invalid action explorations;

[0010] S5: Establish a multi-objective composite reward function with the goal of minimizing end-to-end latency and system energy consumption while also taking into account system load balancing;

[0011] S6: The agent is trained using a time-delay-aware proximal policy optimization algorithm. The state features are extracted and the action probability distribution is output using a time-aware Actor-Critic network structure. Through continuous interaction between the agent and the environment, the agent learns and outputs the optimal hybrid runtime placement strategy of the service function chain that satisfies resource and dependency constraints.

[0012] Optionally, in S1, the function At the node The above uses runtime Execution time computation delay Due to the influence of runtime efficiency coefficient, its calculation delay expression is shown in equation (1):

[0013]

[0014] In the formula: The number of CPU cycles required to process a unit of data; For nodes The calculation frequency; This is an efficiency coefficient used to reflect performance degradation under different operating environments.

[0015] Optionally, in S2, the probability that a function instance is in an active resident state. It follows an exponentially decaying distribution, and its expression is shown in equation (5):

[0016]

[0017] In the formula: The arrival time of the current function (or request). For function Last time at the node The execution end time, The maximum keep-alive window time for instances on the platform, where γ is the decay coefficient. Based on the aforementioned probability, the expected cold start latency. The expression is shown in equation (6):

[0018]

[0019] In the formula: For environment initialization time; This refers to the mirror fetch time.

[0020] Optionally, in S3, the joint state space ;in, It represents the global resource status, including the CPU utilization, memory usage, current queue length, and sparse indicator vectors that mark the warmed-up function instances on each node for all nodes. To identify the service function chain request characteristics, graph embedding technology is used to extract the topology and dependency edge weights of the current service function chain. These are probabilistic prediction features, including request arrival rate and cold start probability predicted based on historical data.

[0021] Optionally, in S5, the multi-objective composite reward function Taking into account latency, energy consumption, load balancing, and penalty terms, its expression is shown in equation (12):

[0022]

[0023] In the formula: This is the total end-to-end delay; This represents the total energy consumption of the system. and This is the moving average normalization factor; The standard deviation of CPU load for all edge nodes; Additional penalties for violating the service level agreement or resource overflow; α, β, δ, and ψ are all weight adjustment coefficients.

[0024] Optionally, in S6, the Actor network of the delay-aware near-end policy optimization algorithm adopts an objective function. To limit the policy update step size to avoid policy oscillation, its expression is shown in equation (13):

[0025]

[0026] In the formula: The ratio of new to old strategies; This is the clipping threshold; The advantage function is calculated using generalized advantage estimation.

[0027] Optionally, the advantage function The generalized advantage estimate of the exponentially weighted moving average of the discounted time-series difference error is expressed as Equation (17):

[0028]

[0029] In the formula: γ is the discount factor; This is a decay factor used to balance the accuracy of short-term return estimates with the stability of long-term strategies. For the first The time-series difference error includes both immediate rewards and state value.

[0030] Optionally, in S6, the Actor network and the Critic network share a low-level feature extraction module. The feature extraction module includes an LSTM unit, which is used to process historical load sequences to identify the tidal effect of traffic, and concatenates the LSTM output with static resource features before inputting it into the fully connected layer.

[0031] In summary, this application includes at least one of the following beneficial technical effects:

[0032] 1. This method embeds a hybrid architecture of containers and WebAssembly coexisting into the agent's decision-making action space. By introducing a probability-aware cold start prediction mechanism, when the agent perceives a high risk of cold start, it proactively routes long-tail requests to the WebAssembly environment, which enables millisecond-level startup. This mechanism transforms cold start from a passive system overhead into a proactively avoidable decision variable at the underlying architecture level. Combined with dynamic action masks, it effectively avoids ineffective decisions and accelerates algorithm convergence.

[0033] 2. This method embeds an LSTM module into the Actor-Critic network, utilizing its temporal hidden state to filter noise interference from burst traffic and ensure the stability of dynamic scheduling decisions. Furthermore, it incorporates the node CPU utilization variance into a composite reward and penalty mechanism, combined with WebAssembly's high-speed response and distribution capabilities for bursty loads, avoiding single-point resource overload. This synergy between deep reinforcement learning algorithms and a hybrid architecture ensures high concurrency throughput for regular requests while controlling system energy consumption fluctuations caused by load surges to 19.1%, achieving a high degree of unity between system latency, load balancing, and energy consumption stability under complex constraints. Attached Figure Description

[0034] Figure 1 This is a diagram of the edge-cloud collaborative hybrid runtime system architecture of an embodiment of this application.

[0035] Figure 2 This is a diagram of the latency-aware near-end policy optimization algorithm architecture in an embodiment of this application.

[0036] Figure 3 This is a comparison chart of the training convergence performance of various algorithms in the embodiments of this application.

[0037] Figure 4 This is a comparison chart of average response latency under different request arrival rates in the embodiments of this application.

[0038] Figure 5 This is a comparison chart of system energy consumption under different strategies in the embodiments of this application.

[0039] Figure 6 This is a comparison chart of the load balancing performance of various algorithms in the embodiments of this application.

[0040] Figure 7 This is a comparison chart of cold start mitigation performance in the embodiments of this application. Detailed Implementation

[0041] The present application will be further described in detail below with reference to all the accompanying drawings.

[0042] This application discloses a hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment. This method achieves intelligent resource scheduling by combining a hybrid architecture of WebAssembly and containers with a latency-aware near-end strategy optimization algorithm. Specifically, it includes the following steps and model construction:

[0043] I. System and Resource Modeling

[0044] like Figure 1 As shown, this application abstracts the edge-cloud collaborative network as a weighted undirected graph, and all edge nodes deploy a dual-mode hybrid runtime architecture that includes containers and WebAssembly.

[0045] (I) Computation and Transmission Model During Hybrid Runtime

[0046] function On the node Using runtime During execution, its deterministic computation time Due to the influence of runtime efficiency coefficient, its calculation delay expression is shown in equation (1):

[0047]

[0048] In the formula: The number of CPU cycles required to process a unit of data; For nodes The calculation frequency; This is an efficiency coefficient used to reflect performance degradation under different runtime environments. Generally, WebAssembly performs slightly less efficiently than native containers for computationally intensive tasks.

[0049] (II) Cold Start Delay Model Based on Probabilistic Prediction

[0050] To accurately quantify cold starts under burst traffic, the survival probability of function instances is modeled based on the interval characteristics of the Poisson arrival process. This model represents the probability of a function instance being in an active, persistent state. It follows an exponentially decaying distribution, as shown in equation (5):

[0051]

[0052] In the formula: The arrival time of the current function (or request). For function Last time at the node The execution end time, The maximum keep-alive window for instances on the platform. Based on this, the expected cold start latency. The expression is shown in equation (6):

[0053]

[0054] II. Construction of Markov Decision Processes

[0055] The scheduling problem is modeled as a partially observable Markov decision process. State space. It includes not only the global resource status such as CPU / memory Furthermore, the topological features of the service function chain were extracted through graph embedding. It also introduces cold start probability features based on historical data prediction. .

[0056] To improve exploration efficiency, the action space is designed with a dynamic action masking mechanism, which forces the probability of actions that are out of memory or do not support a specific runtime to 0, greatly reducing the search space.

[0057] (I) Multi-objective composite reward function

[0058] The reward function is the core of guiding agent optimization. This application designs a composite reward function that integrates latency, energy consumption, and load balancing. As shown in equation (12):

[0059]

[0060] In the formula: This is the total end-to-end delay; This represents the total energy consumption of the system. and This is the moving average normalization factor; The standard deviation of CPU load for all edge nodes; Additional penalties for violating the service level agreement or for resource overflow.

[0061] III. Delay-Aware Near-End Strategy Optimization Algorithm Architecture

[0062] like Figure 2 As shown, this application designs a time-aware Actor-Critic deep reinforcement learning network architecture, which includes an input and shared feature extraction layer (Stage 1), a reinforcement learning backbone network layer (Stage 2), and an action and evaluation output layer (Stage 3).

[0063] Stage 1: Input and Shared Feature Extraction Layer. State Space Includes resource status Service function chain request characteristics and probabilistic prediction features The three components are concatenated into a unified global observation vector, which is then input into the LSTM unit to smooth instantaneous noise in the traffic flow and capture long-term time-dependent temporal coding features, thereby achieving deep perception of the multidimensional heterogeneous state of the edge network.

[0064] Stage 2: Reinforcement Learning Backbone Network Layer. This layer consists of an Actor network and a Critic network, sharing the unified feature vector extracted in Stage 1. The shared feature vector first undergoes deep semantic extraction through two fully connected layers (FC 256 + ReLU). The network then adopts a dual-head structure: the Actor network output branch includes a node selection head (Softmax) and a runtime selection head (Softmax), outputting the edge node selection probability distribution and the runtime type selection probability distribution, respectively; and filters the output actions through a dynamic action masking mechanism, blocking invalid actions and outputting the optimal policy within the valid space. The Critic network calculates the state value in parallel through two fully connected layers (FC 256 + ReLU). It is used to evaluate the current strategy.

[0065] Stage 3: Action and Evaluation Output Layer. The network output maps the action probability distribution processed by the Actor network to the final scheduled actions. , indicating at node Select runtime Joint decision-making; simultaneously, utilizing the state value output by the Critic network. The generalized dominance estimate is obtained by combining the generalized dominance estimate with the calculation of the dominance value. It is used to guide the iterative update of Actor-Critic network parameters and achieves optimal placement of tasks in a heterogeneous hybrid runtime environment through end-to-end joint decision-making.

[0066] IV. Application Research of Delay-Aware Proximity Strategy Optimization Algorithm in Function Placement During Hybrid Runtime

[0067] To verify the effectiveness of the proposed algorithm, a simulation platform consisting of 10 heterogeneous edge nodes and 1 cloud center was built. Service function chain requests were generated using the Azure Functions Trace real dataset. The average cold start time for containers was set to 500ms, and the average cold start time for WebAssembly was set to 5ms.

[0068] (I) Training convergence and stability analysis

[0069] like Figure 3 As shown, the delay-aware proximal policy optimization algorithm exhibits high convergence efficiency in the early stages of training. Convergence is defined as the algorithm's reward value reaching 95% of the mean of the stable interval. Based on this, the delay-aware proximal policy optimization algorithm converges after 90,000 steps, with a final stable reward value of approximately -0.09. In contrast, while the standard proximal policy optimization algorithm achieves a similar convergence interval, it requires approximately 163,000 steps to converge due to its lack of temporal modeling capabilities, increasing the sample complexity by 1.8 times. Furthermore, the deep Q-network policy exhibits continuous oscillations throughout the training process, with a final stable reward value of -0.23, representing an absolute reward difference of approximately 0.14 compared to our method. This demonstrates that the delay-aware proximal policy optimization algorithm, which integrates LSTM temporal coding and dynamic masking mechanisms, possesses excellent convergence stability and decision quality when handling complex, non-stationary edge traffic.

[0070] (II) End-to-end delay, energy consumption, and load balancing performance analysis

[0071] like Figure 4 As shown, regarding response latency, as the system load intensity increases from 0.5 times the baseline load to 4.0 times the extremely high load, the latency-aware near-end policy optimization algorithm consistently maintains the lowest average response latency. Especially under 4.0 times the load pressure, compared to the deep Q-network strategy where latency increases sharply, the latency-aware near-end policy optimization algorithm keeps latency at a low level and significantly outperforms the greedy strategy and the standard near-end policy optimization algorithm. Figure 5 As shown, in terms of system energy consumption, the latency-aware near-end policy optimization algorithm, guided by a multi-objective composite reward function, achieves the optimal trade-off between response performance and energy consumption. Its energy consumption performance is significantly lower than that of the deep Q-network strategy, and it exhibits superior energy consumption suppression capabilities compared to the greedy strategy as the load increases. Figure 6 As shown, in terms of cluster load balancing performance, the latency-aware near-end strategy optimization algorithm exhibits the best stability, with a load standard deviation as low as [missing value]. It is significantly better than the greedy strategy. ), Proximity Policy Optimization Algorithm ( ) and deep Q network strategy ( The experimental data above fully demonstrates that the latency-aware near-end strategy optimization algorithm, through intelligent scheduling during hybrid runtime, can achieve low latency, low energy consumption, and excellent resource balance utilization in highly dynamic, large-span heterogeneous load environments, ensuring the overall service quality of the edge-cloud collaborative system.

[0072] (III) Verification of cold start mitigation mechanism

[0073] like Figure 7 As shown, in a scenario without preheating and without mitigation measures, the cold start rate is as high as 58.8%, and the average cold start latency is 500.0 ms. While the traditional fixed keep-alive strategy can reduce the cold start rate to 37.8%, its inability to intelligently schedule based on function characteristics means the average cold start latency remains high at 499.9 ms. In contrast, the probabilistic prediction hybrid runtime strategy proposed in this application, while maintaining a low cold start rate of 37.8%, accurately perceives the load probability distribution through a latency-aware near-end strategy optimization algorithm and, combined with the advantages of a hybrid runtime architecture, significantly reduces the average cold start latency to 156.2 ms.

[0074] (iv) Analysis of ablation experiments on core components

[0075] To verify the independent contribution and mechanism of action of each core component in the algorithm model proposed in this application, a systematic ablation experiment was conducted by constructing variant models with specific components removed. All variant models were run under the same training configuration and evaluation strategy, and the specific experimental comparison data are recorded in Table 1.

[0076] It should be noted that, Figure 6 The table shows the load standard deviation for a specific sampling period under the baseline load, while Table 1 shows the mean absolute load standard deviation over a longer period under the full load scenario. Therefore, the two have different numerical dimensions.

[0077] Table 1 Comparison of Ablation Experiment Performance Indicators of Core Components

[0078]

[0079] Experimental results show that the complete model of the latency-aware near-end policy optimization algorithm proposed in this application achieves the best overall performance in terms of end-to-end latency, energy consumption, and load standard deviation. When the hybrid runtime architecture is removed, the system loses its ability to actively avoid cold starts, resulting in an average increase of nearly four times in the number of cold starts and a significant increase in end-to-end latency of 46.4%. When the probabilistic prediction mechanism is removed, the agent, lacking state awareness, adopts centralized scheduling to avoid penalties, leading to a deterioration in the load standard deviation to 3.017 and an increase in latency of 28.4%. Removing the LSTM module causes the model to lose its temporal robustness, resulting in a 25.8% increase in latency and a 12.6% increase in energy consumption. The above ablation experiments verify the deep synergistic value and irreplaceable nature of core mechanisms such as hybrid runtime, probabilistic prediction, and temporal modeling in system scheduling.

[0080] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment, characterized in that, include: S1: Construct an edge-cloud collaborative network model and a hybrid runtime resource model, wherein the hybrid runtime includes container runtime and WebAssembly runtime, and establish computation latency and transmission latency models for functions under different nodes and different runtimes; S2: Construct a probability-aware cold start latency model to calculate the probability that a function instance is in an active resident state on different nodes, and then obtain the expected cold start latency to quantify the impact of cold start on system performance. S3: Model the function placement problem of the service function chain as a partially observable Markov decision process, and construct a joint state space that includes global resource state, service function chain request characteristics and probabilistic prediction characteristics; S4: Construct an action space to simultaneously determine the target node and runtime type of the function placement, and introduce a dynamic action masking mechanism to generate a mask vector based on the real-time remaining resources of each node and the runtime constraints of the function to filter out invalid action explorations; S5: Establish a multi-objective composite reward function with the goal of minimizing end-to-end latency and system energy consumption while also taking into account system load balancing; S6: The agent is trained using a time-delay-aware proximal policy optimization algorithm. The state features are extracted and the action probability distribution is output using a time-aware Actor-Critic network structure. Through continuous interaction between the agent and the environment, the agent learns and outputs the optimal hybrid runtime placement strategy of the service function chain that satisfies resource and dependency constraints.

2. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S1, the function On the node Using runtime Execution time computation delay Due to the influence of runtime efficiency coefficient, its calculation delay expression is shown in equation (1): In the formula: The number of CPU cycles required to process a unit of data; For nodes The calculation frequency; This is an efficiency coefficient used to reflect performance degradation under different operating environments.

3. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S2, the probability that a function instance is in an active resident state. It follows an exponentially decaying distribution, and its expression is shown in equation (5): In the formula: The arrival time of the current function (or request). For function Last time at the node The execution end time, The maximum time for keeping instances alive on the platform, where γ is the decay coefficient; Based on the aforementioned probability, the expected cold start latency The expression is shown in equation (6): E T c o l d i j k = 1 - P w a r m · T i n i t k + T p u l l k 6 In the formula: For environment initialization time; This refers to the mirror fetch time.

4. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S3, the joint state space S t = S r e s S r e q S p r o b ;in, It represents the global resource status, including the CPU utilization, memory usage, current queue length, and sparse indicator vectors that mark the warmed-up function instances on each node for all nodes. To identify the service function chain request characteristics, graph embedding technology is used to extract the topology and dependency edge weights of the current service function chain. These are probabilistic prediction features, including request arrival rate and cold start probability predicted based on historical data.

5. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S5, the multi-objective composite reward function Taking into account latency, energy consumption, load balancing, and penalty terms, its expression is shown in equation (12): In the formula: This is the total end-to-end delay; This represents the total energy consumption of the system. and This is the moving average normalization factor; The standard deviation of CPU load for all edge nodes; Additional penalties for violating the service level agreement or resource overflow; α, β, δ, and ψ are all weight adjustment coefficients.

6. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S6, the Actor network of the delay-aware near-end policy optimization algorithm adopts the objective function. To limit the policy update step size to avoid policy oscillation, its expression is shown in equation (13): L C L I P θ = E t [ m i n ρ t θ A t c l i p ρ t θ 1 - ε 1 + ε A t 13 In the formula: The ratio of new to old strategies; This is the clipping threshold; The advantage function is calculated using generalized advantage estimation.

7. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 6, characterized in that: The advantage function The generalized advantage estimate of the exponentially weighted moving average of the discounted time-series difference error is expressed as Equation (17): In the formula: γ is the discount factor; λ G A E ∈ 0 1 This is a decay factor used to balance the accuracy of short-term return estimates with the stability of long-term strategies. For the first The time-series difference error includes both immediate rewards and state value.

8. The hybrid runtime function placement method based on deep reinforcement learning in an edge-cloud collaborative environment according to claim 1, characterized in that: In S6, the Actor network and the Critic network share the underlying feature extraction module. The feature extraction module includes an LSTM unit, which is used to process historical load sequences to identify the tidal effect of traffic, and concatenates the LSTM output with static resource features before inputting it into the fully connected layer.

Images