Cloud edge collaborative reasoning and resource optimization method based on deep reinforcement learning

Abstract

The invention provides a cloud edge collaborative reasoning and resource optimization method based on deep reinforcement learning, which comprises the following steps: dividing E2E time delay of a DNN task in each time slot into transmission time delay, queuing time delay of an edge server and a cloud server and processing time delay, and obtaining influence factors of the E2E time delay; constructing an optimization function by taking the minimization of the E2E time delay as a target; carrying out DNN task partitioning by adopting a reinforcement learning algorithm based on the optimization function; and allocating edge server resources and cloud resources by adopting a resource dynamic optimization allocation algorithm. In a dynamic network and multi-task scene, end-to-end time delay reduction is taken as a target, an optimal DNN division scheme is given by observing a system operation environment in real time, and it is ensured that the system can adaptively adjust an operation strategy in a complex and changeable environment. The calculation resources are reasonably planned through the given partition points, the calculation amount and the time cost required by strategy training are reduced, and the overall operation efficiency of the algorithm is improved.

Description

Technical Field

[0001] This invention relates to the fields of cloud-edge collaboration and DNN task partitioning technology, and more specifically, to a cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of deep neural networks in fields such as computer vision and natural language processing is constantly expanding. However, their huge computing demands pose a severe challenge to resource-constrained edge devices. In many IoT scenarios, layered edge cloud systems that combine edge computing and cloud computing are gradually becoming a key paradigm for solving the problem. Resource-constrained edge devices can offload their deep learning tasks to edge servers or cloud servers, which can significantly reduce response time and further improve service levels.

[0003] With the increasing demand for DNN inference services, the layered deployment of deep neural network inference tasks has become a key technology for reducing latency. However, the limited computing resources of edge servers, the volatility of tasks in dynamic and random environments, and network bandwidth constraints severely limit the ability to provide optimal DNN inference services for various tasks. Against this backdrop, how to effectively improve DNN inference performance in resource-constrained cloud-edge systems has become a problem that needs to be solved in the current research field. Summary of the Invention

[0004] In view of this, the present invention proposes a cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning. It provides an optimal DNN partitioning scheme by dynamically training the system operating environment in real time, and rationally plans computing resources based on the given partitioning points, so as to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above objectives, this invention proposes a cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning, comprising: The E2E latency of the DNN task in each time slot is divided into transmission latency, queuing latency of edge servers and cloud servers, and processing latency, and the influencing factors of the E2E latency are obtained. An optimization function is constructed with the objective of minimizing the E2E delay; Based on the optimization function, a reinforcement learning algorithm is used to partition the DNN task; Based on the partitioning results and the optimization function, a dynamic resource optimization allocation algorithm is used to allocate resources between edge servers and the cloud.

[0006] Furthermore, the optimization function includes an optimization objective and constraints. The optimization objective is to minimize the E2E latency of the DNN task. The constraints include: constraining the E2E latency of each type of DNN task, ensuring that the computing resources allocated to each type of task are non-negative and bounded, and ensuring that the service queuing system is stable.

[0007] Furthermore, the process of partitioning DNN tasks using reinforcement learning algorithms includes: The state space is constructed based on the arrival rate of the DNN task, the wireless transmission rate, and the computing resources of edge servers and cloud servers. Construct the action space based on the partitioning results of the DNN task; The reward function is constructed based on the arrival rate of the DNN task, the wireless transmission rate, the computing resources of the edge server and cloud server, the E2E latency, and the optimization coefficients. Construct a policy gradient update objective based on the reward function and the optimization function; The Actor network and Critic network are updated based on the policy gradient update objective, and the partitioning results of the DNN task are output based on the updated Actor network.

[0008] Furthermore, during the process of updating the Actor network and Critic network based on the policy gradient update target, a buffer is used to store historical update information, and the update target value of the network parameters is obtained based on the historical update information.

[0009] Furthermore, the process of allocating edge server resources and cloud resources using a dynamic resource optimization allocation algorithm includes: Calculate the computing requirements of edge servers and cloud servers separately, and initialize the allocation of edge resources and cloud resources according to the computing requirements. Construct edge resource candidate solutions for the DNN task, update the E2E delay by iteratively perturbing the edge resource candidate solutions, and stop the loop when the delay error no longer changes; Based on the updated E2E latency, a resource allocation scheme is generated.

[0010] Furthermore, during the process of updating the E2E delay by cyclically perturbing the candidate solutions for edge resources, the following formula is used to determine whether the candidate solutions satisfy the resource capacity constraint:

[0011] in, Indicates a candidate solution for edge resources. This represents the cumulative computational cost of the k-th task at the edge (i=e) or in the cloud (i=c). This represents the task completion rate.

[0012] Furthermore, the method for initializing the allocation of edge resources and cloud resources according to computing demand is as follows:

[0013] in, This represents the computing resources allocated to the k-th type of task at the edge (i=e) or in the cloud (i=c). This represents the cumulative computational cost of the k-th type of task at the edge (i=e) or in the cloud (i=c).

[0014] Furthermore, the process of allocating edge server resources and cloud resources using a dynamic resource optimization allocation algorithm also includes: using the INT8 quantization method to quantize the intermediate transmission data in the resource allocation process from FP32 type to INT8 type.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a cloud-edge collaborative inference optimization model based on deep reinforcement learning. In dynamic networks and multi-task scenarios, aiming to reduce end-to-end latency, it achieves adaptive model partitioning and cloud-edge resource allocation. The DDPG-based DNN partitioning algorithm dynamically trains by observing the system's operating environment in real time to provide the optimal DNN partitioning scheme, thereby ensuring the system can adaptively adjust its operating strategy in complex and ever-changing environments and improve DNN inference performance. Furthermore, this invention employs the DRPA dynamic resource optimization algorithm for resource allocation, rationally planning computing resources based on given partition points, effectively reducing the computational load and time cost required for policy training, and improving the overall operating efficiency of the algorithm. Experiments show that, under maximum task arrival rate conditions, this scheme reduces latency by 13.7% compared to cloud-only inference and by 33.3% compared to edge-only inference. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a schematic diagram of the cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning of the present invention. Figure 2 This is a schematic diagram of the chain topology and DAG topology in the embodiments of the present invention; Figure 3 This is a network structure diagram of the DDPG algorithm in an embodiment of the present invention; Figure 4 This is a training graph with different learning rates in an embodiment of the present invention; Figure 5 These are training images from different batches in embodiments of the present invention; Figure 6 The training graphs are shown in the embodiments of the present invention under different optimization coefficients V; Figure 7 This illustrates the time delay under different optimization coefficients V in the embodiments of the present invention; Figure 8 This is a comparison chart of the total latency of different algorithms in the embodiments of the present invention; Figure 9 This is a comparison chart of cloud latency for different algorithms in this embodiment of the invention; Figure 10 This is a comparison chart of edge latency for different algorithms in embodiments of the present invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] This embodiment proposes a cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning, such as... Figure 1 As shown, it includes: The E2E latency of the DNN task in each time slot is divided into transmission latency, queuing latency of edge servers and cloud servers, and processing latency, and the influencing factors of E2E latency are obtained. Construct an optimization function with the objective of minimizing E2E latency; Based on the optimization function, a reinforcement learning algorithm is used to partition the DNN task. Based on the partitioning results and optimization functions, a dynamic resource optimization allocation algorithm is used to allocate resources between edge servers and the cloud.

[0019] The technical solution of the present invention will be described in detail below: This embodiment assumes the use of edge servers and a central cloud to jointly construct a hierarchical edge cloud system, with the edge servers and central cloud connected via a wireless network. This edge cloud system can provide various deep learning applications, each requiring a different type of DNN model. Examples include facial recognition systems, intelligent monitoring systems, infrared thermal imaging, and customer flow sensing in smart shopping malls. Each of these four applications requires a unique DNN model. To abstract the various types of DNN architectures, this chapter considers the logical layer as the atomic structure of the DNN model. For chain-topology DNN models, the logical layer can consist of actual layers or multiple layers connected in series. For DAG topology DNN models, the parallel layer is considered a logical layer. Figure 2 As shown, the DNN model is modeled as several logical layers connected sequentially. Therefore, the DNN model can be modeled as several logical layers connected sequentially. In this paper, the k-th DNN model is represented by formula (1).

[0020] (1)

[0021] in This represents the upper limit of E2E latency for the k-th deep learning task. It is the number of logical layers of the k-th type of DNN. This represents the k-th DNN model. This represents the l-th logical layer of the k-th DNN model, which can be specifically defined as a tuple. , express Input data size, express The computational overhead.

[0022] This embodiment takes into account the volatility of tasks in a dynamic and random environment, and intends to discretize the arrival time axis of the task into multiple time slots, with the length of each time slot defined as follows: The arrival of tasks within each time period follows a Poisson distribution, while the total available resources of the server fluctuate over time. The length of each time slot can vary depending on the task arrival rate, DNN deployment, and resource allocation costs. However, it is assumed that the available resources of the edge server and cloud server are fixed within each time slot.

[0023] make This represents the DNN partitioning decision at time slot t, when Time indicates from the first The layers are divided, and layers 1 through 2 are separated. One layer is deployed on edge servers, and other layers are deployed on cloud servers; when This indicates that the entire model has been deployed on a cloud server. This indicates that the entire model is deployed on edge servers. Based on the partitioning decision, the E2E latency of the DNN task in each time slot is divided into transmission latency and queuing latency and processing latency of edge / cloud services.

[0024] Transmission delay: The DNN task generated by the edge device is transmitted through the wireless network in each time slot. The wireless transmission delay formula of the DNN task at time slot t is shown in (2).

[0025] (2) in It is the DNN task number The size of the layer input data, This represents the wireless transmission rate at time slot t.

[0026] Queuing and processing delay: Since the arrival of tasks in each time period follows a Poisson distribution and the computing resources in each time slot are fixed, the service time of each task in the edge server is also fixed. Therefore, this chapter uses the M / D / 1 queue to model the queuing time of DNN tasks in each time period. The queuing and processing delay can be calculated using formula (3).

[0027] (3)

[0028] in This indicates the service rate of the edge server. This represents the edge computing resources allocated to the DNN task in time slot t, while... , This represents the arrival rate of the DNN task, when... The time interval t indicates that there are no DNN tasks to be processed at the edge server during time slot t.

[0029] For cloud server latency, queuing and processing latency can be expressed as formula (4).

[0030] (4) Similar to edge servers, Indicates the service speed of the cloud server. This represents the cloud computing resources allocated to the DNN task in time slot t, and at the same time , This represents the arrival rate of the DNN task, when... The time slot t indicates that there is no DNN task to be processed on the cloud server. Based on the three latency scenarios, the E2E latency of the DNN task in time slot t can be expressed as formula (5).

[0031] (5) From the above formula, we can see that... The value of is not only affected by DNN partitioning, but also by the allocation of computing resources on edge servers and cloud servers. This section mainly focuses on jointly optimizing DNN partitioning and computing resource allocation to minimize the E2E latency of DNN tasks. The overall formula for the optimization problem is shown in (6). Among them, constraint C2 ensures that the E2E latency of each type of DNN task cannot exceed the maximum value, and constraints C3 and C4 ensure that the computing resources allocated to each type of DL task are non-negative and bounded. Constraints C5 and C6 guarantee that the service queuing system is stable.

[0032] (6)

[0033] Constraints C7 and C8 ensure that the computing resources allocated to all DL tasks are equal to the total available computing resources of the edge servers and the central cloud.

[0034] DRPA resource dynamic optimization algorithm Since resource allocation to edge servers and cloud servers occurs after partitioning, this section proposes a dynamic resource allocation algorithm (DRPA) to address the computational resource allocation problem after DNN partitioning. See Algorithm 2 for details. This algorithm is applicable when partitioning is known. In this case, edge server resources and cloud resources The algorithm is divided into two phases: the initialization phase allocates initial resources based on the computational task requirements, and the joint optimization phase is mainly responsible for iteratively adjusting resources to further reduce latency. Through iterative optimization, the algorithm ensures the stability of the task queue while satisfying energy constraints.

[0035] (7)

[0036] (8)

[0037]

[0038] The core of the initialization phase is to calculate task requirements and allocate initial resources. This begins with a computational requirement analysis at the edge, analyzing the k-th type of task from layer 1 to the partition point. Cumulative computational load The specific formula is shown in (7), where This represents the computational cost of the i-th layer for the k-th task. Next, we analyze the computational requirements in the cloud, starting from the partitioning points. Cumulative computational load from +1 layer to the last layer The specific formula is shown in (8).

[0039] The next step is to initialize and allocate edge resources and cloud resources according to demand ratios, as shown in formula (9). i={e,c} , e This indicates that edge resources are being initialized and allocated. c This indicates the initial allocation of cloud resources. Finally, the latency is initialized according to formula (5), including transmission latency, queuing latency, and processing latency.

[0040] (9) The joint optimization phase primarily reduces latency by iteratively updating resource allocation. First, candidate resource allocations are generated; for each k task type, candidate solutions for edge resources are generated. By continuously perturbing the current value Then, we verify whether the candidate solution satisfies the resource capacity constraint. The specific formula is shown in (10), where This represents the task completion rate.

[0041] (10) If a candidate solution is feasible, calculate the new time delay according to formula (5). Repeat this process until the time delay error no longer changes, and then stop the loop. The stopping condition is shown in (11).

[0042] (11)

[0043] Output the final allocation scheme. Let... Let be the total number of iterations, then the algorithm's complexity is . Experiments have shown that it can converge after a finite number of iterations.

[0044] Finally, considering that the intermediate data to be transmitted is 32-bit floating-point type, which not only occupies a lot of memory but also requires a lot of computation, this embodiment performs quantization operation on the transmitted intermediate data. In the process of exploring quantization operation, compressing the data to 8 bits can show more superior performance compared with other quantization bit widths. In fact, INT8 quantization is a widely used quantization operation method. It can not only efficiently compress the data size, greatly reduce the storage requirements and computing resources, but also cause very little precision loss when quantizing data, which can be almost ignored. Based on experimental results and practical application experience in the industry, this experiment decided to use INT8 quantization as an effective method in the data compression process, quantizing FP32 type data into INT8 type as shown in formula (12). Conversely, the formula for dequantizing INT8 type data into FP32 type is shown in formula (13).

[0045] (12) (13)

[0046] in This represents the original intermediate data. This represents the intermediate data after quantization. Z This represents the quantized fixed-point value corresponding to a floating-point value of 0. S This is the smallest scale that can be represented after fixed-point quantization. S and Z It can be obtained through the following formulas (14) and (15).

[0047] (14)

[0048] (15)

[0049] in, Indicates the maximum floating-point value. Represents the smallest floating-point value. Represents the maximum fixed-point value. This represents the minimum fixed-point value.

[0050] DDPG-based DNN partitioning algorithm Since the problem of formula (6) is an NP-hard problem, we cannot find the optimal solution in polynomial time. This prompts us to propose a reinforcement learning-based DNN partitioning algorithm to find an approximate solution. This method has been proven to be an effective way to solve complex optimization problems.

[0051] The theoretical foundation of reinforcement learning (RL) is built upon the mathematical model of the Markov Decision Process (MDP). This framework formally describes the interaction between the agent and the environment through a quadruple: as shown in Equation (16), therefore, this paper first constructs an MDP M.

[0052] (16) in S It is the system state space. A It is the system action space. It is the state transition frequency. R It is a reward function. For specific states and actions, the reward function is defined as follows: 1) State: At the beginning of each time step, observe the state of the environment, including the arrival rate of the DNN task. wireless transmission rate edge and cloud computing resources That is, formula (17).

[0053] (17) (18) 2) Action: After observing the environmental state at time slot t, as shown in formula (18), the corresponding DNN partitioning is given. The specific partitioning points are... express.

[0054] 3) Reward: The reward function is shown in formula (19), where V is the optimization coefficient used to balance resources and latency.

[0055] (19)

[0056] Since the goal of actual reinforcement learning is to maximize the cumulative total reward of the exponential discount, problem P2 is transformed into problem P2, as shown in formula (20).

[0057] (20)

[0058] Then, an Actor-Critic-based deep reinforcement learning framework, DDPG, is used to learn action policies to determine the DNN partition for each time slot, such as... Figure 3 As shown, DDPG has four main neural networks.

[0059] 1) Main Actor Network By using the current state Mapping to Action The strategy for generating DNN partitions interacts with the environment.

[0060] 2) Main Critic Network The Q-table is trained to simulate the real state and action space, guiding the update direction of the Actor.

[0061] 3) Target Actor Network : Train the main participant network to calculate the target action for the next state and participate in the calculation of the Critic target value.

[0062] 4) Target Critic Network : Calculate the target Q value, which is used in the Critic network loss function.

[0063] In addition, DDPG uses a cache to store historical information, with each piece of historical information including the current state. Current action ,award and the next state of time slot t .

[0064] In order to satisfy the C1 constraint, the softmax function is applied to the output layer of each Actor network to ensure that each layer output has K units, as shown in formula (21).

[0065] (twenty one)

[0066] in It is the output of the softmax layer, representing the probability of each partition. The specific training principles are as follows.

[0067] 1) Training of the main Actor network: DDPG is a non-policy algorithm that generates an actual exploration policy by adding exploration noise during interaction, as shown in formula (22).

[0068] (twenty two) The main Actor network parameters The update is performed by sampling strategy gradient, as shown in formula (23).

[0069] (twenty three) In each training step, By randomly sampling N items from the buffer Perform a small batch update, and the update formula is shown in (24).

[0070] (twenty four)

[0071] in It is the learning rate of the main Actor network.

[0072] 2) Main Critic Network: By minimizing the loss function To update the parameters of the main Critic network ,in Defined as formula (25).

[0073] (25) The target value is obtained through the following formula (26): (26) The loss function is defined as shown in formula (27): (27)

[0074] (28)

[0075] In each training step, By randomly sampling N items from the buffer Perform a small batch update, and the update formula is shown in (28), where It is the learning rate of the main Critic network.

[0076]

[0077] 3) Target network training: Parameters of the target network The update is performed using a soft update algorithm, defined by formulas (29) and (30): (29) (30) The specific algorithm of the DNN partitioning algorithm based on DDPG is shown in Algorithm 3. It involves randomly adjusting the weights of the main network and the target network, then initializing the buffer. The training process of this algorithm is divided into multiple stages, each stage further divided into multiple time slots. At the beginning of each stage, the environment is reset. In each time slot, the main Actor network generates actions to determine the DNN partition, and then allocates computing resources between the edge and cloud. The environment observes the current reward... And update the next status. The data is stored in the cache and continuously extracted in batches to update the weights of the main Critic network. After a period of training, the weights of the target network are updated.

[0078] Specifically, end-to-end optimization of dynamic resource allocation is achieved through an Actor-Critic dual-network structure. The system inputs are the environment state space S, which includes information such as device computing resources, task arrival rate, and network bandwidth, and the action space A, which includes the partition decision set. The output is the policy network. Used to generate deterministic actions and commentator networks Used to evaluate the value of actions. During the initialization phase, the weight parameters of the main Actor network and the main Critic network... The exploration foundation is established through random initialization, and the experience replay buffer capacity is determined. It is then used to store historical state transition samples.

[0079] The training phase employs a nested double-loop structure to achieve iterative strategy optimization: the outer loop initializes the environment and loads the initial state in each training round. This simulates a complete resource allocation cycle. In each time slot t, the inner loop determines the policy network based on the current state. Generate Actions An edge-cloud resource allocation scheme is generated by combining the DRPA algorithm. and Environmental feedback rewards And update to the next state. To form a quadruple Store in the buffer. When the buffer reaches its capacity threshold... Then, a small batch of samples is randomly sampled to calculate the target value. The Critic network is updated, the Actor network is updated using policy gradients, and finally the target network is asymptotically synchronized with soft update coefficients τ.

[0080] Simulation Experiment Analysis To evaluate the experimental performance of the DRPA resource dynamic optimization algorithm and the DDPG-based DNN partitioning algorithm, an edge cloud system was built in this embodiment. The experimental equipment information is shown in the table below.

[0081] The Raspberry Pi 4B, launched by the Raspberry Pi Foundation in 2019, is a high-performance single-board computer. It features a Broadcom BCM2711 quad-core ARM Cortex-A72 processor with a 1.5GHz clock speed, 2GB / 4GB / 8GB LPDDR4 memory options, dual Micro HDMI ports for dual-screen 4K@60Hz output, and high-speed connectivity modules including USB 3.0, Gigabit Ethernet, dual-band Wi-Fi, and Bluetooth 5.0. The device is compatible with various sensors and peripherals via a 40-pin GPIO interface and MIPI CSI / DSI expansion ports. It supports booting from MicroSD cards or USB external storage and can run various operating systems such as Raspberry Pi OS and Ubuntu. Widely used in embedded development, IoT nodes, lightweight servers, smart home control, and STEM education, it combines low cost, low power consumption, and high scalability, making it an important tool platform for academic research and engineering practice.

[0082] Table 1

[0083] (1) Parameter settings The experimental parameters are mainly divided into two parts: environmental parameters and model parameters. The environmental parameters mainly cover the time slot length τ, task arrival rate λ, and edge computing resources in the edge cloud system. Cloud computing resources and network bandwidth The specific configuration information is shown in Table 2.

[0084] The average arrival rate λ(t) in each time slot is randomly sampled from λ, and other configuration information is determined based on the real-time situation.

[0085] Table 3 shows the core parameter configuration of the DDPG algorithm. The algorithm adopts an Actor-Critic dual network structure. The number of neurons in the hidden layer of the Actor network is (512, 256), and the number of neurons in the hidden layer of the Critic network is (256, 128). High-dimensional state features are extracted through a deep fully connected structure. In terms of activation function selection, Leaky ReLU is uniformly used in the hidden layer to avoid gradient vanishing. Leaky ReLU is retained in the output layer of the Actor network to support continuous action space, while the output layer of the Critic network uses the Tanh function to constrain the value prediction range.

[0086] Table 2

[0087] Maximum buffer capacity 1×10 4 The optimizer uses the Adam algorithm to adaptively adjust the learning rate to cope with the complex gradient dynamics of Actor-Critic joint training.

[0088] Table 3

[0089] (2) DNN model This experiment used four classic DNN models for different deep learning tasks: AlexNet, VGG19, ResNet101, and YOLOv2. VGG19 and YOLOv2 use chain topologies, while AlexNet and ResNet101 use direct algebra (DAG) structures. Detailed model descriptions are as follows: AlexNet is a landmark convolutional neural network model in the field of deep learning, which significantly outperformed traditional methods for the first time in the ImageNet large-scale visual recognition challenge. AlexNet is often used as a benchmark model for image classification tasks and has been applied to visual tasks such as medical image analysis and remote sensing image interpretation.

[0090] With its simple modular structure and transferability, VGG19 has become a pre-trained model for transfer learning, and is often used in high-precision feature extraction scenarios such as image style transfer, fine-grained classification, and medical image enhancement.

[0091] ResNet101 achieved the highest accuracy at the time in the ImageNet classification task and performed well in industrial inspection and autonomous driving perception systems that require processing high-resolution images or long-tailed data distributions.

[0092] YOLOv2 is a single-stage object detection model that predicts bounding boxes and class probabilities directly in images in an end-to-end manner. It is suitable for scenarios with high real-time requirements, such as pedestrian detection in video surveillance, obstacle recognition in autonomous driving, and real-time target tracking of drone aerial images, making it the preferred model for deployment on lightweight embedded devices.

[0093] Experimental Results Analysis This experiment first investigated the impact of different learning rates and batch processing on the average reward. Figure 4 This illustrates the convergence of average reward under different learning rates for training the model. The horizontal axis represents the number of training epochs, and the vertical axis represents the average reward value. In the four sets of parameter experiments, when the Actor network and the Critic network use the same learning rate, i.e. and The convergence curves exhibit better stability: the high learning rate combination converges fastest in the initial stage, with an average reward of -18.2 × 10⁻⁶ within 400 rounds. 3 The low-learning-rate combination only began to converge after 2000 training rounds, with an average reward of -17.8 × 10⁻⁶. 3 .

[0094] When the learning rates of the Actor and Critic networks differ significantly—for example, the shaded area of ​​the standard deviation becomes larger, and the average reward fluctuation is significantly higher than with other learning rate combinations—it indicates that imbalanced network parameter updates can compromise algorithm stability. This experiment comparatively verifies the impact of collaborative learning rate configuration on convergence efficiency in the DDPG network, providing a quantitative basis for parameter tuning.

[0095] Figure 5 This paper presents the average reward convergence of the DDPG algorithm under different batch sizes. Experiments show that batch size significantly affects convergence speed and stability: when Batchsize=64, the algorithm tends to converge after approximately 1500 training rounds, and the smallest shaded area of ​​the standard deviation indicates robust training. However, as the batch size decreases, the convergence speed gradually slows down, and with Batchsize=8, it only approaches convergence after 2500 training rounds. Furthermore, the larger shaded area of ​​the standard deviation for smaller batches reflects larger gradient estimation variance and significant policy update fluctuations during training. These results validate that appropriately increasing the batch size within the DDPG framework can effectively improve sample utilization in experience replay and balance the stability of exploration and utilization.

[0096] This experiment evaluated the weight V in the reward function formula (15). Figure 6This paper presents the convergence characteristics of the normalized cumulative reward with different trade-off factors V as the training series progresses. The horizontal axis represents the number of training rounds, and the vertical axis represents the normalized cumulative reward. Experimental results show that when V=1, the algorithm exhibits drastic fluctuations in the early stages, reaching an amplitude of 0.4 around round 200, but gradually converges to the optimal value as training progresses, verifying the high sensitivity of low V values ​​to resource constraints. When V=10, the convergence trajectory shows similar fluctuation characteristics, but the final stable value slightly decreases to 0.95, indicating that increasing V leads to an increase in the weight of latency optimization. The curves for V=100 and V=1000 show significant differences: the former stabilizes at 0.8 after 600 rounds, while the latter converges quickly to 0.6 with the smallest fluctuation amplitude throughout, indicating that high V forces the algorithm to prioritize minimizing the latency penalty term, but also limits the policy exploration space. Figure 7 This paper demonstrates the optimization effect of different weights V on the long-term average end-to-end latency of the DDPG algorithm in an edge cloud system. The horizontal axis represents the number of training sets, and the vertical axis represents the normalized latency. Experiments show that increasing the value of V significantly improves system stability and latency performance: when V=1, the latency fluctuates drastically in the first 200 rounds of training, with an amplitude of 0.15 seconds, eventually converging to a maximum latency value of 1.25 seconds; while the latency with V=1000 fluctuates the least throughout the entire process, with an amplitude of only 0.03 seconds, and quickly stabilizes at a minimum value of 1.10 seconds, a 12% reduction compared to the latency with V=1. The curves for intermediate parameters V=10 and V=100 converge to 1.15 seconds and 1.18 seconds, respectively, indicating that the latency performance improves with increasing V value. All curves tend to plateau after 800 rounds, indicating that the algorithm has reliable convergence. A smaller V value enhances the stability of resource constraints, enabling the strategy to achieve higher cumulative rewards under long-term resource budget constraints, but it carries the risk of high volatility in the early stages of training. Conversely, a larger V value, while enabling faster convergence and reducing variance, may lead to suboptimal solutions due to an overemphasis on delay optimization. Therefore, it is recommended to choose a compromise parameter such as V=10 to ensure resource constraints while providing a reliable decision-making basis for dynamic resource scheduling in edge cloud systems.

[0097] This result demonstrates that the V value directly influences the system's optimization direction by adjusting the weights of resource constraints and latency penalties. A larger V value strengthens the priority of the latency term, driving the policy network to converge quickly to a low-latency solution, but may come at the cost of stability; while a smaller V value, focusing on resource constraints, limits latency optimization. Therefore, it is recommended to set V=10. The system achieves a smooth reduction in latency while ensuring resource budget, verifying the balancing advantage of this parameter in dynamic resource scheduling.

[0098] In addition, this experiment compared our method with other baseline methods, and the methods compared are described below.

[0099] (1) Greedy algorithm: For each deep learning task, the DNN model selects the partition point with the lowest overall load.

[0100] (2) DNN deployed in the cloud (DDIC): DNN is not partitioned, but deployed entirely on cloud servers. Deep learning tasks are all run on cloud servers and cloud resources are allocated according to the DRPA algorithm.

[0101] (3) DNN deployed in the edge (DDIE): DNN is not partitioned, but deployed entirely on edge servers. Deep learning tasks are run on edge servers and edge resources are allocated according to the DRPA algorithm.

[0102] (4) DNN partition without resource optimization (DPNO): DNN partitioning adopts the same partitioning algorithm based on DDPG, but in terms of resource allocation at the edge and cloud, resource allocation is only based on the proportion of tasks. That is, the computing resources allocated to deep learning tasks are only proportional to the amount of computing power of the task in that time slot.

[0103] This experiment considered three common performance metrics in edge cloud systems: 1) Total inference latency, including transmission latency, queuing latency, and inference latency; 2) Edge inference latency, mainly the processing latency of deep learning tasks at the edge; and 3) Cloud inference latency, mainly the processing latency of deep learning tasks on the cloud server.

[0104] like Figure 8 As shown, when the task arrival rate increases from 15% to 25%, the total inference latency of all compared methods shows a significant upward trend. Among them, DPNO has the fastest latency increase, while our proposed method, through a dynamic resource scheduling mechanism, effectively controls the total latency to within 6 seconds under the same load, reducing it by about 20% compared to the suboptimal method DDIC. This demonstrates that the collaborative inference framework proposed in this paper can effectively alleviate the system overload problem in high-concurrency scenarios.

[0105] Figure 9 The comparison of cloud inference latency further reveals the differences in offloading strategies among different methods. Experimental data shows that the DPNO method, due to its over-reliance on cloud computing, exhibits a near-linear increase in cloud inference latency with increasing task rate. In contrast, our proposed method maintains cloud inference latency below 5.2 seconds even with a task arrival rate of 25%, a reduction of approximately 31% compared to the Greedy method, verifying the superiority of the hybrid offloading strategy designed in this invention in terms of cloud resource utilization efficiency.

[0106] Figure 10In terms of edge-side performance, this method stabilizes latency to within 7 seconds under the same load through a dynamic collaboration mechanism of edge nodes, which is an average reduction of 22% compared to the benchmark method. When the task rate exceeds 21%, the latency gap between this method and DDIE gradually widens, which confirms the robustness of the adaptive scheduling algorithm designed in this paper under high-load scenarios.

[0107] The combined results of the three sets of experiments demonstrate that our proposed method, through a cloud-edge collaborative optimization mechanism, exhibits significant advantages in three dimensions: total latency control, cloud resource utilization efficiency, and edge computing stability. Especially in high-load scenarios with task arrival rates exceeding 21%, our method achieves a performance improvement of 25%-32% compared to traditional methods, fully validating the effectiveness of the proposed framework in practical applications.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A cloud-edge collaborative reasoning and resource optimization method based on deep reinforcement learning, characterized in that, include: The E2E latency of the DNN task in each time slot is divided into transmission latency, queuing latency of edge servers and cloud servers, and processing latency, and the influencing factors of the E2E latency are obtained. An optimization function is constructed with the objective of minimizing the E2E delay; Based on the optimization function, a reinforcement learning algorithm is used to partition the DNN task; Based on the partitioning results and the optimization function, a dynamic resource optimization allocation algorithm is used to allocate resources between edge servers and the cloud.

2. The method according to claim 1, characterized in that, The optimization function includes an optimization objective and constraints. The optimization objective is to minimize the E2E latency of the DNN task. The constraints include: constraining the E2E latency for each type of DNN task, ensuring that the computing resources allocated to each type of task are non-negative and bounded, and ensuring that the service queuing system is stable.

3. The method according to claim 1, characterized in that, The process of partitioning DNN tasks using reinforcement learning algorithms includes: The state space is constructed based on the arrival rate of the DNN task, the wireless transmission rate, and the computing resources of edge servers and cloud servers. Construct the action space based on the partitioning results of the DNN task; The reward function is constructed based on the arrival rate of the DNN task, the wireless transmission rate, the computing resources of the edge server and cloud server, the E2E latency, and the optimization coefficients. Construct a policy gradient update objective based on the reward function and the optimization function; The Actor network and Critic network are updated based on the policy gradient update objective, and the partitioning results of the DNN task are output based on the updated Actor network.

4. The method according to claim 3, characterized in that, During the process of updating the Actor network and Critic network based on the policy gradient update objective, a buffer is used to store historical update information, and the update target value of the network parameters is obtained based on the historical update information.

5. The method according to claim 1, characterized in that, The process of allocating edge server resources and cloud resources using a dynamic resource optimization allocation algorithm includes: Calculate the computing requirements of edge servers and cloud servers separately, and initialize the allocation of edge resources and cloud resources according to the computing requirements. Construct edge resource candidate solutions for the DNN task, update the E2E delay by iteratively perturbing the edge resource candidate solutions, and stop the loop when the delay error no longer changes; Based on the updated E2E latency, a resource allocation scheme is generated.

6. The method according to claim 5, characterized in that, During the process of updating the E2E delay by cyclically perturbing the candidate solutions for edge resources, the following formula is used to determine whether the candidate solutions satisfy the resource capacity constraint: ， in, Indicates a candidate solution for edge resources. This represents the cumulative computational cost of the k-th task at the edge (i=e) or in the cloud (i=c). This represents the task completion rate.

7. The method according to claim 5, characterized in that, The method for initializing the allocation of edge and cloud resources according to computing demand is as follows: ， in, This represents the computing resources allocated to the k-th type of task at the edge (i=e) or in the cloud (i=c). This represents the cumulative computational cost of the k-th type of task at the edge (i=e) or in the cloud (i=c).

8. The method according to claim 1, characterized in that, The process of allocating edge server resources and cloud resources using a dynamic resource optimization allocation algorithm also includes: using the INT8 quantization method to quantize the intermediate transmission data in the resource allocation process from FP32 type to INT8 type.

Images