An improved unloading method for multi-agent deep reinforcement learning

Abstract

The application provides an improved multi-agent deep reinforcement learning unloading method, relates to the technical field of Internet of Vehicles and edge computing, and solves the technical problems of computing overhead, energy consumption and time delay in the task unloading process.The technical scheme comprises the following steps: S1: task unloading problem modeling;S2: improved unloading method design and implementation.The task queue smoothing algorithm designed by the application can accurately reflect the task load of the server and the mobile terminal, and assist the deep learning algorithm to make more accurate and stable unloading decisions.The application designs a smoothing penalty mechanism of a Sigmoid function in a reward function, and provides continuous and accurate gradient guidance for agent learning.The application corrects the action output by the deep reinforcement learning algorithm in real time in the early training stage through the closed-loop feedback mechanism of the PI controller, and improves the time delay performance of unloading.

Description

Technical Field

[0001] This invention relates to the fields of vehicle networking and edge computing technology, and in particular to an improved offloading method for multi-agent deep reinforcement learning. Background Technology

[0002] Mobile edge computing (MEC) enables near-field offloading and processing of tasks by deploying computing power on one side of a base station or roadside unit. To better accomplish computational tasks in MEC, task offloading primarily focuses on whether and how to offload tasks. Researchers typically model task offloading as an objective optimization problem and propose relevant offloading algorithms. However, in MEC scenarios such as vehicle-to-everything (V2X) networks, rapid terminal movement, sudden changes in task load, and wireless channel interference all pose challenges to traditional task offloading algorithms.

[0003] Because of its advantages of accurate and efficient decision-making and adaptability to dynamic environments, researchers have proposed using artificial intelligence (AI) to improve task offloading mechanisms. However, using machine learning in task offloading also faces some challenges. On the one hand, machine learning relies on comprehensively and accurately acquiring environmental parameters, but dynamic topologies can easily lead to inappropriate selection of environmental parameters or inaccurate detection, making it difficult to guarantee the performance of intelligent algorithms. On the other hand, deep reinforcement learning (DRL) algorithms randomly configure decisions and output actions in the early stages of learning, resulting in poor initial performance. Summary of the Invention

[0004] The purpose of this invention is to provide an improved offloading method for multi-agent deep reinforcement learning, which addresses the technical problem of poor performance of existing solutions in the early stages of training in high-speed, time-varying environments. An improved PI-MAPPO algorithm is implemented in the Internet of Vehicles (IoV) to achieve better offloading performance.

[0005] The inventive concept of this invention is as follows: First, it provides an improved offloading method for multi-agent deep reinforcement learning, offering more accurate task load information to the deep learning mechanism. Second, addressing the training instability problem caused by the "step" penalty of the traditional reward function, this invention designs a smoothing penalty term based on the Sigmoid function in the reward function, providing continuous and accurate gradient guidance for agent learning. Third, addressing the task timeout problem in the early stages of training for the MAPPO offloading algorithm, this invention proposes a PI-MAPPO task offloading algorithm combining classical control and MRL. Through the closed-loop feedback of the PI controller, the algorithm corrects the offloading ratio and transmit power of the DRL output in real time during the early stages of training.

[0006] To achieve the aforementioned objectives, the present invention employs the following technical solution: an improved offloading method for multi-agent deep reinforcement learning, comprising the following steps: S1: Task unloading problem modeling; Derive a multi-objective optimization mathematical model for task unloading latency, computational overhead, and energy consumption; S2: Design and implementation of an improved unloading method.

[0007] The problem is modeled as a Markov game problem. First, a MAPPO scheme is designed. Then, a proportional-integral control algorithm based on task queues is designed. Based on this, a PI-MAPPO task unloading algorithm is proposed.

[0008] Step S1 specifically includes the following: like Figure 5 As shown, in an MEC scenario, there is one base station, one edge server, and multiple vehicle terminals. The terminal devices traverse the base station's signal coverage area from west to east. Upon reaching the eastern edge of the coverage area, the vehicle turns around and continues moving from east to west until it reaches the western edge of the coverage area. When an in-vehicle application generates a computing task, the vehicle device can either complete it locally (i.e., local computing) or offload the data to the base station for the edge server to complete (i.e., edge computing).

[0009] terminal equipment m The set is The system uses discrete time slots, with each time slot... t The duration is Δ Define the terminal device. m In the t The first time slot generated at the start of the [number]th time slot n The task is Its expression is ,in for Data size, For the computational density of the task, This is the time delay threshold. (Definition) It is a task The computational load. Definition For equipment m In the t The set of tasks generated at the beginning of each time slot, expressed as: , N For the first t Maximum number of tasks in a time slot. Definition For equipment m In the t The characteristics of all tasks in each time slot include data size, computational density, and latency constraints.

[0010] S11, Edge Computing Modeling: If the mobile terminal decides to offload data to an edge server for computation, the following three steps are required. First, data (part or all of the task) is transmitted to the base station via a wireless channel. Second, arriving tasks are queued in the buffer of the base station-side server. Third, the server completes the computation tasks sequentially according to a "first-come, first-served" principle.

[0011] S111, Wireless Transmission Rate Modeling: Assume the origin O is at the left edge of the base station's signal coverage area. For equipment m The initial distance from the origin O, For base station height, This is the diameter of the base station's signal coverage area. (Define the device.) m movement speed Values ​​are positive from west to east and negative from east to west. Therefore, in the... t Each time slot, equipment m The distance to the base station is (1) definition W The total bandwidth of the channel. M For the number of devices, For terminal devices m The transmission power, This represents the noise spectral density. (Definition) ρ This serves as a reference coefficient for path loss. Sampling is performed from an exponential distribution with a mean of 1. ω This represents the path loss index. Maximum power of the equipment. The distance between the mobile device and the base station affects the data transfer rate during task offloading. For example, mobile terminals m In the t The wireless transmission rate of each time slot is (2) Refer to formula (1), The corresponding transmission rate can also be obtained under certain circumstances.

[0012] S112, Server queue modeling: Based on the "first-come, first-served" principle, when the number of arriving tasks exceeds the server's processing capacity, a task queue will appear in the server's task buffer. The length of this queue can be used as an indicator of the edge server's computational load.

[0013] No. t +1 time slot, the server's instantaneous task queue consists of two parts. The first part is the... t At the start of the time slot, the number of tasks already completed minus the server's workload. ΔThe task involves calculating the duration. Another part is the... t All tasks that are offloaded to the server by mobile devices within the time slot minus the server's processing time. Δ The task of calculating duration. Definition This refers to the actual amount of data unloaded to the server. (Definition) It is a task The uninstallation part, the expression is written as For ease of description, the present invention will... Called a time slot t Inner k The first task arrives at the server. Therefore, there is the first... k The computational load of each arriving task Therefore, the first t +1 slot server instantaneous queue length for: (3); in Calculate the server frequency, in units of , For the first The computation time obtained for each task For the first The number of tasks arriving at the MEC server within a time slot. For the first The instantaneous queue length of the edge server in the time slot.

[0014] Instantaneous queue length can reflect the server's computing load at a specific moment. However, in MEC scenarios, on the one hand, application layers can generate sudden tasks, and on the other hand, device movement can cause data transmission rates to vary over time. Affected by these two factors, the instantaneous task queue may oscillate rapidly. Therefore, the instantaneous queue length is difficult to accurately reflect the server's true computing load. The paper "Multi-agent deep reinforcement learning based multi-task partial computation offloading in mobile edge computing" chooses instantaneous queue length as one of the decision-making criteria, which can easily lead to the following two situations. First, based on the short-term sudden change in the instantaneous queue length, the algorithm is prone to making inappropriate offloading decisions. For example, the algorithm may judge a short-term sudden increase in the queue length as a large computing load on the edge server, forcing the agent to start local computation. Second, it may trigger frequent switching of offloading strategies, ultimately leading to a decrease in the stability of offloading decisions. Therefore, this invention introduces an exponentially weighted moving average mechanism to smooth the instantaneous queue length, in order to filter out instantaneous noise that occurs during queue observation, thereby more accurately reflecting the server's computing load. Therefore, the exponentially weighted moving average of the task queue length of the edge server (hereinafter referred to as "the server's average queue length") for: (4); in The weight factor for the instantaneous team leader can take any value within [0,1].

[0015] S113. Task latency modeling for edge computing: Edge computing task latency refers to the total time consumed by an edge server to complete a task. It includes transmission latency, computation latency, and waiting latency. First, let's analyze transmission latency. Transmitting data from the source end to the channel incurs transmission latency. Definition For the task The uninstallation ratio. Uninstallation decision. Therefore, the amount of data that the terminal devices plan to offload to the edge server is... It can be written as: (5) Define the device within each time slot m The maximum amount of data that can be sent is the initial upper limit of the transmission capacity. Therefore, in a complete time slot t ,equipment m The initial upper limit of the transmission capacity can be written as (6); The actual amount of data offloaded to the edge server This is constrained by two factors. Firstly, there may be data in the previous time slot that was not fully transmitted. Its value range is It will continue transmitting in the current time slot. Therefore, the actual transmission capacity of the current time slot is limited. for On the other hand, if the initial upper limit of transmission capacity is large enough, the actual offloading... That's equivalent to the planned uninstall volume. Therefore... Take the actual limit of transmission capacity With the planned unloaded data volume The minimum of the two is expressed as: (7) Therefore, the sending delay of the unloading task for (8) Next, we analyze the server's computation latency. For the first... k If task data arriving at the server is immediately delivered to the CPU for execution, the computation latency is... (9) Let's analyze the waiting latency again. According to the "first-come, first-served" rule, for the first... k The first task to arrive at the server needs to wait for the second task to arrive. k - One task must complete its calculation before processing can begin. The process must also be considered for the [missing information - likely a specific task or event]. k The sending latency and waiting latency of each task need to be the larger of the two factors mentioned above. .

[0016] Therefore, the task latency of edge computing It can be represented as (10) Among them, the first k The sending delay of each arriving task The initial task latency is... .

[0017] S114. Energy consumption modeling for edge computing: Terminal devices consume energy to transmit wireless signals to offload tasks, and edge servers also consume energy to perform computational tasks. However, since servers are typically powered by traditional wired connections, energy consumption can be temporarily ignored. We will only consider the energy consumption of wireless transmission from mobile terminals for edge computing. for (11) S115. Modeling the computational overhead of edge computing: Terminals are typically owned by the user, so terminal computing is free. However, server purchase and maintenance both incur economic costs. Offloading tasks from the terminal to edge servers usually incurs computing fees from the user. Therefore, considering only the computing overhead of edge servers, the computing overhead of edge computing... Can be written as (12) in, The cost of computing for edge servers.

[0018] S12, Local Computational Modeling: Compared to edge computing, local computing does not involve network transmission, so it is only necessary to model the computing latency, task queue, and computing power consumption of the local device.

[0019] S121. Queue modeling of terminal devices: Based on the task offloading ratio determined by the algorithm, a portion of the data for a given task is offloaded to an edge server, while the remaining data is computed locally. As tasks are continuously generated, if the workload received by a terminal device exceeds its computing capacity, a task queue will appear on the terminal device. If a terminal device is unable to complete task computation within the current time slot, the remaining tasks will continue to be computed in the next time slot.

[0020] In the t +1 time slot, the terminal's instantaneous queue length is the first... t The instantaneous queue length of the terminal device is calculated by adding the new task load to the slot's queue length, and then subtracting the task's computational load. for (13) in For equipment m CPU calculation frequency, unit: same. For equipment m No. n The local computational load of each task is expressed as follows: .

[0021] Referring to edge computing models, local queue length detection also introduces the EWMA smoothing queue mechanism. The weighted moving average of the task queue length on the terminal device (hereinafter referred to as "the device's average queue length") is used. for (14) S122. Local computation task latency modeling: Define the local computing task section ,in Indicates the size of the data calculated locally. This represents the local computation latency. Since there may be unfinished tasks from the previous time slot, the local computation latency needs to be considered in two parts. The first part is the latency of the previous time slot. t Task load that already exists at the start of the time slot The second part is the t New task load generated in time slots Therefore, the latency of locally computed tasks is... for (15) S123, Energy consumption modeling for local computation: When some of the task's data is computed locally, wireless transmission is unnecessary. Therefore, the energy consumption of local computation is... for (16) in, This is the energy consumption factor calculated locally.

[0022] It should also be noted that, based on the analysis above, this invention does not consider the overhead of local computation.

[0023] S13. Problem Statement: The goal of building task unloading is to optimize the problem. For the device... m The first n Task After the agent determines the unloading ratio, the task is calculated on both the edge server and the local device.

[0024] definition For the task The completion latency depends on the maximum latency of the two computing modes: edge computing and local computing. Therefore... Can be written as (17) in, and The expressions are shown in formulas (10) and (15) respectively.

[0025] definition For the task The total energy consumption. It includes edge computing energy consumption and local computing energy consumption. Therefore It can be represented as (18) in, and The expressions are shown in formulas (11) and (16) respectively.

[0026] To better reflect the timeliness of the task, latency should also be considered as a key indicator in the objective function. Therefore, this invention introduces a task latency term. This allows the system objective function to balance the timeliness of tasks with resource overhead (energy consumption, computation) in order to meet the task offloading requirements in the context of vehicle networking.

[0027] Considering latency, energy consumption, and computational overhead, the system objective function is defined. for (19) in, These represent the weights for latency, energy consumption, and computational overhead, with values ​​ranging from [0,1].

[0028] Ultimately, the task offloading problem in the MEC scenario can be expressed as follows: (20) Step S2 specifically includes the following: S2. Improved Unloading Method Design and Implementation: Treating each mobile terminal as an intelligent agent, therefore P 1 can be transformed into a fully cooperative multi-agent reinforcement learning problem. P 2. The mobile terminal randomly generates computing tasks. By observing node locations, wireless gain, task attributes, and task queues, the agents on the terminal can coordinate their respective offloading strategies to achieve... P The optimization objective is 1.

[0029] This invention utilizes Markov game pairs P 2. Modeling was performed, and the MAPPO algorithm was selected to optimize the offloading decision. However, some problems exist. First, MAPPO initially lacks an environmental model reference, and the learning process relies on interactive trial and error. This leads to the algorithm randomly determining action strategies in the early stages of training, resulting in a lack of strategic thinking in the exploration process. Early-stage deep networks struggle to cope with time-varying topology MEC scenarios. Second, for vehicle-to-everything (V2X) offloading scenarios, the high-dimensional state space increases the workload of initial exploration and feature learning, making it difficult for the agent to quickly traverse effective states in the early stages, thus affecting algorithm convergence. To address these issues, this invention proposes an improved offloading algorithm based on the multi-agent proximal policy optimization (MAPPO) algorithm and the proportion integration (PI) algorithm.

[0030] S21. Description of the unloading problem based on Markov game theory: Define the set of intelligent agents as The global state space is The joint action space is defined as ,in Represents intelligent agents m The individual action space of the intelligent agent. m The reward function is The global state transition function is This invention addresses the problem. P 2. Modeled as a Markov game process, its expression is: The specific definitions of each element are as follows.

[0031] No. t Time slot, intelligent agent m local state Including channel gain ,distance Task characteristics and average queue length of equipment Therefore, it can be written as (twenty one) Indicates the first t The global state of a time slot is determined by the edge server queue length. Composed of the local states of all agents, it can be written as (twenty two) Representation of intelligent agents m In the t Actions taken in time slots, including unloading ratios and transmission power , can be written as (twenty three) For intelligent agents m No. n The reward function for each task is used to guide the agent's policy learning. Represents intelligent agents m The individual reward is expressed as follows: .therefore M Joint reward function for each agent for (twenty four) According to formula (19), the system objective function This includes latency, power consumption, and computational overhead, but the orders of magnitude of these three factors can differ significantly. The reward function can be used directly. On the one hand, this can easily lead to parameter updates becoming more sensitive to features with large values, resulting in an imbalance in the model's learning direction; on the other hand, it may cause gradient descent to slow down. Therefore, for the objective function... Perform normalization. Define the normalized objective function as follows: ,have (25) in , This serves as a normalization reference scale. It should be a very small positive number to prevent division by zero.

[0032] The reward function designed in the paper "Multi-agent deep reinforcement learning based multi-task partial computation offloading in mobile edge computing" is discontinuous, thus resulting in time delay. Once the latency threshold is exceeded The reward function will switch abruptly. In a MEC environment, rapid vehicle movement or sudden mission changes can cause this. Frequent crossings The threshold makes it difficult for the gradient signal received by the policy network to be stable, which can easily cause oscillations in the unloading policy and thus reduce the unloading performance.

[0033] Therefore, this invention considers designing a smoothing penalty term, the Sigmoid function, in the reward function. Specifically, the time delay bias is first defined. This reflects Deviation The degree of [something]. Next, the normalized time delay deviation is used as the independent variable of the function to obtain the Sigmoid function value. When [something] exist When the gradient is near the target area, the sigmoid function provides a smooth penalty, effectively mitigating the abrupt changes in the original reward function and ensuring the policy network receives a relatively stable gradient signal. Through learning... and By understanding the error relationship between the two parameters, the agent can progressively adjust the offloading ratio or transmit power, avoiding frequent policy changes caused by hard switching of the original reward function at the latency threshold. To continue using latency, power consumption, and computational overhead as objectives in reinforcement learning, the normalized [reward function] will still be used. As part of the reward function, a normalized reward function is defined. for (26) The Sigmoid function , As a weighting factor, This is the normalization factor.

[0034] S22. Unloading Algorithm Design: S221, MAPPO algorithm: In order to make independent unloading decisions, each agent Each includes an actor network A network of commentators The Actor Network generates actions based on local observations. It outputs specific action policies based on the agent's current local environment observations (e.g., distance, channel gain, task queue length, task features). The Critics Network's core task is to evaluate the value of the agent's state, outputting a quantified value score to provide a basis for policy optimization within the Actor Network.

[0035] This invention employs a MAPPO structure with centralized commentators and distributed actors. Each agent... m Owns an independent actor network All agents share the same commentator network. During the execution phase, each agent relies solely on its own local observations to independently generate unloading decisions through its corresponding actor network, without needing access to the global state or decision information from other agents. The training and computation of all actor and commentator networks are performed by the MEC server. On the MEC server, the actor network is updated by sampling the dataset. and comment network .

[0036] During a training process, the agent first uses its current policy. Multiple rounds are generated. Then each agent... m Sample the dataset Uploaded to the MEC server.

[0037] (27) in For time steps, This is the set of all sampling time steps. For sampling action, This represents the probability of the corresponding sampling action.

[0038] Regarding actor networks. Definition Indicates the first k The actor network at the next update The parameters, where Training objective function The core lies in minimizing the loss function Its specific expression is (28) Among them, the shear loss term With entropy regularization term The definition can be found in the literature "Multi-agent deepreinforcement learning based multi-task partial computation offloading in mobile edge computing". This is the entropy regularization coefficient, used to balance strategy optimization and exploration capabilities.

[0039] Regarding commentator networks. Definition For the first k Commentators network at the time of the next update V Parameters ( It is trained by minimizing the following mean squared error loss. (29) in From time step Initial cumulative discount rewards. New actor network parameters after training update complete. It is distributed to each intelligent agent to replace its original parameters.

[0040] After interactive training using the MAPPO algorithm and an edge computing environment, the offloading ratio and transmit power output by the actor network were defined as follows: and .

[0041] S222, PI control algorithm: Terminal location changes and sudden tasks increase the dynamism and complexity of MEC environment parameters. When the MAPPO algorithm performs random exploration in the early stages of training, these two factors are more likely to interfere with algorithm convergence, leading to increased latency. To address this, a task queue algorithm based on proportional-integral control is designed. Specifically, when the task queue length of the terminal device... Exceeding the threshold At this time, the algorithm uses proportional-integral control theory to modify the offloading strategy (i.e., offloading ratio and transmit power) output by the MAPPO algorithm. By adjusting the task queue length, the algorithm alleviates the increased latency caused by the random strategy in the early stage of training.

[0042] definition For intelligent agents m The local task queue length threshold. This refers to the maximum computing frequency supported by the local device's CPU. ,have In the first t Each time slot defines the intelligent agent. m Captain error signal Average queue length With queue threshold difference. Can be written as (30) Since there is a task latency threshold, only considering Greater than In this case, the error signal It is a negative value.

[0043] definition For the integral element of the PI controller, By analyzing historical error signals The core function of this method, obtained through cumulative summation, is to reduce the steady-state deviation of the system. Until the... t Each time slot, captain error signal The cumulative expression is (31) definition This is the unloading ratio output by the PI controller. Based on the PI control principle, and to ensure... The interval is (0,1], therefore it can be written as: (32) in For proportional gain, This is the integral gain. According to control theory, the proportional gain gives the system a rapid response capability to instantaneous task queuing, and can suppress task queue fluctuations in real time. The integral gain, on the other hand, reduces the steady-state error of the system by accumulating historical deviations.

[0044] When the error signal of the local queue length When it exists, the intelligent agent m The system is in a task overload state. At this time, a PI controller is used to control the output of the DRL-MAPPO strategy network. The actual unloading ratio has been adjusted. for (33) After the PI controller adjusts the offload ratio of the DRL algorithm, to avoid task backlog on the device... m The transmit buffer, therefore the actual transmit power of the mobile device. for (34) in Power adjustment factor (usually) This is used to adjust the correction level of the PI controller. It ensures that the system can smoothly unload when a task overload occurs, while avoiding the energy consumption cost caused by aggressive power strategies.

[0045] Meanwhile, the present invention proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it implements the steps of the method described in the present invention.

[0046] Furthermore, the present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the method described in the present invention when invoked by a processor.

[0047] Finally, the present invention provides a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the present invention.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) This invention designs a task queue length smoothing algorithm to accurately predict the task load status of devices (i.e., servers and mobile terminals). Existing research uses instantaneous queue length as a load observation indicator, causing deep learning algorithms to make frequent action responses, affecting convergence performance. This invention introduces Exponential Weighted Moving Average (EWMA) to smooth the queue length. This mechanism can effectively filter out instantaneous noise in the queue length, providing a reliable and stable observation indicator for DRL offloading decisions.

[0049] 2) This invention designs a smoothing penalty term based on the Sigmoid mechanism in the reward function of the DRL algorithm. Existing algorithms' traditional reward functions set "step" penalties at latency constraint boundaries, leading to gradient discontinuities and training instability during the learning process. This invention introduces a smoothing penalty mechanism using the Sigmoid function, providing the agent with continuous and precise gradient guidance, enabling it to perceive the asymptotic risk when task latency approaches the threshold during training. The algorithm can more smoothly adjust the unloading strategy, accelerating the convergence of the agent's reward function.

[0050] 3) This invention proposes an improved task offloading (PI-MAPPO) method. Due to the random exploration mechanism, existing MAPPO offloading algorithms are prone to increased task latency in the early stages of training. Through the closed-loop feedback mechanism of the PI controller, the improved algorithm corrects the DRL output in real time during the early stages of training, alleviating the increased task latency and timeout phenomena. Attached Figure Description

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0052] Figure 1 This is a flowchart of the algorithm of the present invention.

[0053] Figure 2 This is a comparison chart of the computational overhead of the present invention and existing solutions.

[0054] Figure 3 This is a comparison chart of the total energy consumption of the present invention and existing solutions.

[0055] Figure 4 This is a comparison chart of the task latency of the present invention and existing solutions.

[0056] Figure 5 This is a diagram of the MEC system in this invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Example 1: See Figures 1-5 This embodiment provides an improved offloading method for multi-agent deep reinforcement learning, including the following steps: S1: Task unloading problem modeling; Derive a multi-objective optimization mathematical model for task unloading latency, computational overhead, and energy consumption; S2: Design and implementation of an improved unloading method.

[0059] The problem is modeled as a Markov game problem. First, a MAPPO scheme is designed. Then, a proportional-integral control algorithm based on task queues is designed. Based on this, a PI-MAPPO task unloading algorithm is proposed.

[0060] Step S1 specifically includes the following: like Figure 5 As shown, in an MEC scenario, there is one base station, one edge server, and multiple vehicle terminals. The terminal devices traverse the base station's signal coverage area from west to east. Upon reaching the eastern edge of the coverage area, the vehicle turns around and continues moving from east to west until it reaches the western edge of the coverage area. When an in-vehicle application generates a computing task, the vehicle device can either complete it locally (i.e., local computing) or offload the data to the base station for the edge server to complete (i.e., edge computing).

[0061] terminal equipment m The set is The system uses discrete time slots, with each time slot... t The duration is Δ Define the terminal device. m In thet The first time slot generated at the start of the [number]th time slot n The task is Its expression is ,in for Data size, For the computational density of the task, This is the time delay threshold. (Definition) For equipment m In the t The set of tasks generated at the beginning of each time slot, expressed as: , N For the first t Maximum number of tasks in a time slot. Definition For equipment m In the t The characteristics of all tasks in each time slot include data size, computational density, and latency constraints.

[0062] S11, Edge Computing Modeling: If the mobile terminal decides to offload data to an edge server for computation, the following three steps are required. First, data (part or all of the task) is transmitted to the base station via a wireless channel. Second, arriving tasks are queued in the buffer of the base station-side server. Third, the server completes the computation tasks sequentially according to a "first-come, first-served" principle.

[0063] S111, Wireless Transmission Rate Modeling: Assume the origin O is at the left edge of the base station's signal coverage area. For equipment m The initial distance from the origin O, For base station height, This is the diameter of the base station's signal coverage area. (Define the device.) m movement speed Values ​​are positive from west to east and negative from east to west. Therefore, in the... t Each time slot, equipment m The distance to the base station is (1) The distance between the mobile device and the base station affects the data transmission rate during task offloading. For example, mobile terminals m In the t The wireless transmission rate of each time slot is (2) Refer to formula (1), The corresponding transmission rate can also be obtained under certain circumstances.

[0064] S112, Server queue modeling: Based on the "first-come, first-served" principle, when the number of arriving tasks exceeds the server's processing capacity, a task queue will appear in the server's task buffer. The length of this queue can be used as an indicator of the edge server's computational load.

[0065] No. t +1 time slot, the server's instantaneous task queue consists of two parts. The first part is the... t At the start of the time slot, the number of tasks already completed minus the server's workload. Δ The task involves calculating the duration. Another part is the... t All tasks that are offloaded to the server by mobile devices within the time slot minus the server's processing time. Δ The task of calculating duration. Definition This refers to the actual amount of data unloaded to the server. (Definition) It is a task The uninstallation part, the expression is written as For ease of description, the present invention will... Called a time slot t Inner k The first task arrives at the server. Therefore, there is the first... k The computational load of each arriving task Therefore, the first t +1 slot server instantaneous queue length for (3); in Calculate the server frequency, in units of , For the first The computation time obtained for each task For the first The number of tasks arriving at the MEC server within a time slot. For the first The instantaneous queue length of the edge server in the time slot.

[0066] Instantaneous queue length can reflect the server's computing load at a specific moment. However, in MEC scenarios, on the one hand, application layers can generate sudden tasks, and on the other hand, device movement can cause data transmission rates to vary over time. Affected by these two factors, the instantaneous task queue may oscillate rapidly. Therefore, the instantaneous queue length is difficult to accurately reflect the server's true computing load. The paper "Multi-agent deep reinforcement learning based multi-task partial computation offloading in mobile edge computing" chooses instantaneous queue length as one of the decision-making criteria, which can easily lead to the following two situations. First, based on the short-term sudden change in the instantaneous queue length, the algorithm is prone to making inappropriate offloading decisions. For example, the algorithm may judge a short-term sudden increase in the queue length as a large computing load on the edge server, forcing the agent to start local computation. Second, it may trigger frequent switching of offloading strategies, ultimately leading to a decrease in the stability of offloading decisions. Therefore, this invention introduces an exponentially weighted moving average mechanism to smooth the instantaneous queue length, in order to filter out instantaneous noise that occurs during queue observation, thereby more accurately reflecting the server's computing load. Therefore, the exponentially weighted moving average of the task queue length of the edge server (hereinafter referred to as "the server's average queue length") for: (4); in The weight factor for the instantaneous team leader can take any value within [0,1].

[0067] S113. Task latency modeling for edge computing: Edge computing task latency refers to the total time consumed by an edge server to complete a task. It includes transmission latency, computation latency, and waiting latency. First, let's analyze transmission latency. Transmitting data from the source end to the channel incurs transmission latency. Definition For the task The uninstallation ratio. Uninstallation decision. Therefore, the amount of data that the terminal devices plan to offload to the edge server is... Can be written as (5) Define the device within each time slot m The maximum amount of data that can be sent is the initial upper limit of the transmission capacity. Therefore, in a complete time slot t ,equipment m The initial upper limit of the transmission capacity can be written as (6); The actual amount of data offloaded to the edge server This is constrained by two factors. Firstly, there may be data in the previous time slot that was not fully transmitted. Its value range is It will continue transmitting in the current time slot. Therefore, the actual transmission capacity of the current time slot is limited. for On the other hand, if the initial upper limit of transmission capacity is large enough, the actual offloading... That's equivalent to the planned uninstall volume. Therefore... Take the actual limit of transmission capacity With the planned unloaded data volume The minimum of the two is expressed as: (7) Therefore, the sending delay of the unloading task for (8) Next, we analyze the server's computation latency. For the first... k If task data arriving at the server is immediately delivered to the CPU for execution, the computation latency is... (9) Further analysis of the waiting delay shows that, according to the "first-come, first-served" rule, for the first... k The first task to arrive at the server needs to wait for the second task to arrive. k - One task must complete its calculation before processing can begin. The process must also be considered for the [missing information - likely a specific task or event]. k The sending latency and waiting latency of each task need to be the larger of the two factors mentioned above. ; Therefore, the task latency of edge computing It can be represented as (10) Among them, the first k The sending delay of each arriving task The initial task latency is... .

[0068] S114. Energy consumption modeling for edge computing: Terminal devices consume energy to transmit wireless signals to offload tasks, and edge servers also consume energy to perform computational tasks. However, since servers are typically powered by traditional wired connections, energy consumption can be temporarily ignored. We will only consider the energy consumption of wireless transmission from mobile terminals for edge computing. for (11) S115. Modeling the computational overhead of edge computing: Terminals are typically owned by the user, so terminal computing is free. However, server purchase and maintenance both incur economic costs. Offloading tasks from the terminal to edge servers usually incurs computing fees from the user. Therefore, considering only the computing overhead of edge servers, the computing overhead of edge computing... Can be written as (12) in, The cost of computing for edge servers.

[0069] S12, Local Computational Modeling: Compared to edge computing, local computing does not involve network transmission, so it is only necessary to model the computing latency, task queue, and computing power consumption of the local device.

[0070] S121. Queue modeling of terminal devices: Based on the task offloading ratio determined by the algorithm, a portion of the data for a given task is offloaded to an edge server, while the remaining data is computed locally. As tasks are continuously generated, if the workload received by a terminal device exceeds its computing capacity, a task queue will appear on the terminal device. If a terminal device is unable to complete task computation within the current time slot, the remaining tasks will continue to be computed in the next time slot.

[0071] In the t +1 time slot, the terminal's instantaneous queue length is the first... t The instantaneous queue length of the terminal device is calculated by adding the new task load to the slot's queue length, and then subtracting the task's computational load. for (13) in For equipment m CPU calculation frequency, unit: same. For equipment m No. n The local computational load of each task is expressed as follows: .

[0072] Referring to edge computing models, local queue length detection also introduces the EWMA smoothing queue mechanism. The weighted moving average of the task queue length on the terminal device (hereinafter referred to as "the device's average queue length") is used. for (14) S122. Local computation task latency modeling: Define the local computing task section ,in Indicates the size of the data calculated locally. This represents the local computation latency. Since there may be unfinished tasks from the previous time slot, the local computation latency needs to be considered in two parts. The first part is the latency of the previous time slot. t Task load that already exists at the start of the time slot The second part is the t New task load generated in time slots Therefore, the latency of locally computed tasks is... for (15) S123, Energy consumption modeling for local computation: When some of the task's data is computed locally, wireless transmission is unnecessary. Therefore, the energy consumption of local computation is... for (16) in, This is the energy consumption factor calculated locally.

[0073] It should also be noted that, based on the analysis above, this invention does not consider the overhead of local computation.

[0074] S13. Problem Statement: The goal of building task unloading is to optimize the problem. For the device... m The first n Task After the agent determines the unloading ratio, the task is calculated on both the edge server and the local device.

[0075] definition For the task The completion latency depends on the maximum latency of the two computing modes: edge computing and local computing. Therefore... Can be written as (17) in, and The expressions are shown in formulas (10) and (15) respectively.

[0076] definition For the task The total energy consumption. It includes edge computing energy consumption and local computing energy consumption. Therefore It can be represented as (18) in, and The expressions are shown in formulas (11) and (16) respectively.

[0077] To better reflect the timeliness of the task, latency should also be considered as a key indicator in the objective function. Therefore, this invention introduces a task latency term. This allows the system objective function to balance the timeliness of tasks with resource overhead (energy consumption, computation) in order to meet the task offloading requirements in the context of vehicle networking.

[0078] Considering latency, energy consumption, and computational overhead, the system objective function is defined. for (19) in, These represent the weights for latency, energy consumption, and computational overhead, with values ​​ranging from [0,1].

[0079] Ultimately, the task offloading problem in the MEC scenario can be expressed as follows: (20) Step S2 specifically includes the following: S2. Improved Unloading Method Design and Implementation: Treating each mobile terminal as an intelligent agent, therefore P 1 can be transformed into a fully cooperative multi-agent reinforcement learning problem. P 2. The mobile terminal randomly generates computing tasks. By observing node locations, wireless gain, task attributes, and task queues, the agents on the terminal can coordinate their respective offloading strategies to achieve... P The optimization objective is 1.

[0080] This invention utilizes Markov game pairs P 2. Modeling was performed, and the MAPPO algorithm was selected to optimize the offloading decision. However, some problems exist. First, MAPPO initially lacks an environmental model reference, and the learning process relies on interactive trial and error. This leads to the algorithm randomly determining action strategies in the early stages of training, resulting in a lack of strategic thinking in the exploration process. Early-stage deep networks struggle to cope with time-varying topology MEC scenarios. Second, for vehicle-to-everything (V2X) offloading scenarios, the high-dimensional state space increases the workload of initial exploration and feature learning, making it difficult for the agent to quickly traverse effective states in the early stages, thus affecting algorithm convergence. To address these issues, this invention proposes an improved offloading algorithm based on the multi-agent proximal policy optimization (MAPPO) algorithm and the proportion integration (PI) algorithm.

[0081] S21. Description of the unloading problem based on Markov game theory: Define the set of intelligent agents as The global state space is The joint action space is defined as ,in Represents intelligent agents m The individual action space of the intelligent agent. m The reward function is The global state transition function is This invention addresses the problem. P 2. Modeled as a Markov game process, its expression is: The specific definitions of each element are as follows.

[0082] No. t Time slot, intelligent agent m local state Including channel gain ,distance Task characteristics and average queue length of equipment Therefore, it can be written as (twenty one) Indicates the first t The global state of a time slot is determined by the edge server queue length. Composed of the local states of all agents, it can be written as (twenty two) Representation of intelligent agents m In the t Actions taken in time slots, including unloading ratios and transmission power , can be written as (twenty three) For intelligent agents m No. n The reward function for each task is used to guide the agent's policy learning. Represents intelligent agents m The individual reward is expressed as follows: .therefore M Joint reward function for each agent for (twenty four) According to formula (19), the system objective function This includes latency, power consumption, and computational overhead, but the orders of magnitude of these three factors can differ significantly. The reward function can be used directly. On the one hand, this can easily lead to parameter updates becoming more sensitive to features with large values, resulting in an imbalance in the model's learning direction; on the other hand, it may cause gradient descent to slow down. Therefore, for the objective function... Perform normalization. Define the normalized objective function as follows: ,have (25) in , This serves as a normalization reference scale. It should be a very small positive number to prevent division by zero.

[0083] The reward function designed in the paper "Multi-agent deep reinforcement learning based multi-task partial computation offloading in mobile edge computing" is discontinuous, thus resulting in time delay. Once the latency threshold is exceeded The reward function will switch abruptly. In a MEC environment, rapid vehicle movement or sudden mission changes can cause this. Frequent crossings The threshold makes it difficult for the gradient signal received by the policy network to be stable, which can easily cause oscillations in the unloading policy and thus reduce the unloading performance.

[0084] Therefore, this invention considers designing a smoothing penalty term, the Sigmoid function, in the reward function. Specifically, the time delay bias is first defined. This reflects Deviation The degree of [something]. Next, the normalized time delay deviation is used as the independent variable of the function to obtain the Sigmoid function value. When [something] exist When the gradient is near the target area, the sigmoid function provides a smooth penalty, effectively mitigating the abrupt changes in the original reward function and ensuring the policy network receives a relatively stable gradient signal. Through learning... and By understanding the error relationship between the two parameters, the agent can progressively adjust the offloading ratio or transmit power, avoiding frequent policy changes caused by hard switching of the original reward function at the latency threshold. To continue using latency, power consumption, and computational overhead as objectives in reinforcement learning, the normalized [reward function] will still be used. As part of the reward function, a normalized reward function is defined. for (26) The Sigmoid function , As a weighting factor, This is the normalization factor.

[0085] S22. Unloading Algorithm Design: S221, MAPPO algorithm: In order to make independent unloading decisions, each agent Each includes an actor network A network of commentators The Actor Network generates actions based on local observations. It outputs specific action policies based on the agent's current local environment observations (e.g., distance, channel gain, task queue length, task features). The Critics Network's core task is to evaluate the value of the agent's state, outputting a quantified value score to provide a basis for policy optimization within the Actor Network.

[0086] This invention employs a MAPPO structure with centralized commentators and distributed actors. Each agent... m Owns an independent actor network All agents share the same commentator network. During the execution phase, each agent relies solely on its own local observations to independently generate unloading decisions through its corresponding actor network, without needing access to the global state or decision information from other agents. The training and computation of all actor and commentator networks are performed by the MEC server. On the MEC server, the actor network is updated by sampling the dataset. and comment network .

[0087] During a training process, the agent first uses its current policy. Multiple rounds are generated. Then each agent... m Sample the dataset Uploaded to the MEC server.

[0088] (27) in For time steps, This is the set of all sampling time steps. For sampling action, This represents the probability of the corresponding sampling action. The agent will... Store in the replay buffer.

[0089] Regarding actor networks. Definition Indicates the first k The actor network at the next update The parameters, where Training objective function The core lies in minimizing the loss function Its specific expression is (28) Among them, the shear loss term With entropy regularization term The definition can be found in the literature "Multi-agent deepreinforcement learning based multi-task partial computation offloading in mobile edge computing". This is the entropy regularization coefficient, used to balance strategy optimization and exploration capabilities.

[0090] Regarding commentator networks. Definition For the first k Commentators network at the time of the next update V Parameters ( It is trained by minimizing the following mean squared error loss. (29) in From time step Initial cumulative discount rewards. New actor network parameters after training update complete. It is distributed to each intelligent agent to replace its original parameters.

[0091] After interactive training using the MAPPO algorithm and an edge computing environment, the offloading ratio and transmit power output by the actor network were defined as follows: and .

[0092] S222, PI control algorithm: Terminal location changes and sudden tasks increase the dynamism and complexity of MEC environment parameters. When the MAPPO algorithm performs random exploration in the early stages of training, these two factors are more likely to interfere with algorithm convergence, leading to increased latency. To address this, a task queue algorithm based on proportional-integral control is designed. Specifically, when the task queue length of the terminal device... Exceeding the threshold At this time, the algorithm uses proportional-integral control theory to modify the offloading strategy (i.e., offloading ratio and transmit power) output by the MAPPO algorithm. By adjusting the task queue length, the algorithm alleviates the increased latency caused by the random strategy in the early stage of training.

[0093] definition For intelligent agents m The local task queue length threshold. This refers to the maximum computing frequency supported by the local device's CPU. ,have In the first t Each time slot defines the intelligent agent.m Captain error signal Average queue length With queue threshold difference. Can be written as (30) Since there is a task latency threshold, only considering Greater than In this case, the error signal It is a negative value.

[0094] definition For the integral element of the PI controller, By analyzing historical error signals The core function of this method, obtained through cumulative summation, is to reduce the steady-state deviation of the system. Until the... t Each time slot, captain error signal The cumulative expression is (31) definition This is the unloading ratio output by the PI controller. Based on the PI control principle, and to ensure... The interval is (0,1], therefore it can be written as (32) in For proportional gain, This is the integral gain. According to control theory, the proportional gain gives the system a rapid response capability to instantaneous task queuing, and can suppress task queue fluctuations in real time. The integral gain, on the other hand, reduces the steady-state error of the system by accumulating historical deviations.

[0095] When the error signal of the local queue length When it exists, the intelligent agent m The system is in a task overload state. At this time, a PI controller is used to control the output of the DRL-MAPPO strategy network. The actual unloading ratio has been adjusted. for (33) After the PI controller adjusts the offload ratio of the DRL algorithm, to avoid task backlog on the device... m The transmit buffer, therefore the actual transmit power of the mobile device. for (34) in This is the power adjustment factor (typically) used to adjust the correction level of the PI controller. It ensures that the system can smoothly unload when a task overload occurs, while avoiding the energy consumption costs caused by aggressive power strategies.

[0096] Figure 2 This section compares the computational costs of the two algorithms. Figure 2 (a) Computational overhead of the MAPPO algorithm. The computational overhead of agents 1 through 4 only reached 0.07 after 1000 rounds. This indicates that in mobile scenarios, the MAPPO algorithm based on static topology design cannot quickly achieve edge offloading decisions. The computational overhead of agent 0 continuously decreased to approximately 0.02 after 1500 rounds. The lower edge server computational overhead indicates that agent 0 completes most tasks locally and does not choose to offload tasks to the edge. Figure 2 (b) Computational overhead of the PI-MAPPO algorithm. Each agent quickly achieves computational overheads ranging from 0.06 to 0.09, indicating that all mobile devices maintain a high offloading rate. This is because the PI controller corrects the random action decisions made in the early stages of DRL training, and the agents offload a suitable amount of tasks to edge servers. The computational tasks offloaded from mobile devices generate higher computational overhead on the servers.

[0097] Figure 3 A comparison of the total energy consumption of the two algorithms. Figure 3 (a) represents the total energy consumption of the MAPPO algorithm. Due to repeated triggering of latency exceedance penalties, agent 0 no longer tends to offload data to the edge, but instead chooses local computation more often. The total energy consumption of agent 0 continues to rise during the middle of training, eventually reaching 800mJ, while the total energy consumption of other agents, although relatively high in the early stages, can gradually decrease to 200 to 350mJ. Figure 3 (b) represents the total energy consumption of the PI-MAPPO algorithm. The mobile scenario also increases the algorithm's energy consumption in the early stages of training, but the energy consumption of each agent can be reduced to 200 to 350 mJ relatively quickly. Therefore, the PI-MAPPO algorithm can ensure that each agent reduces its own energy consumption in a synchronized manner, while maintaining the energy consumption at a low level.

[0098] Figure 4 This section compares the task latency of the two algorithms. Figure 4 (a) Task latency of the MAPPO algorithm. Agents 3 and 4 experienced latency exceeding 20 seconds in the early stages of training, requiring 750 rounds to reduce it to below 5 seconds, indicating a significantly slow convergence speed. Agent 0's latency continued to rise above 30 seconds after 1250 training rounds. Because it did not consider device movement and random exploration / unloading actions in the early stages of training, the MAPPO algorithm could not promptly change its local computing strategy. The agent's local task queue continued to grow. Subsequent tasks had to queue for a long time before being processed by the local CPU, ultimately leading to a significant increase in task latency. Figure 4 (b) Task latency of the PI-MAPPO algorithm. All five agents converged rapidly within 250 rounds, stabilizing the latency between 0.5 and 2.5 seconds. By using the PI controller to correct action outputs in real time and the continuous gradient guidance provided by the Sigmoid reward mechanism, the PI-MAPPO algorithm significantly improves the convergence speed while maintaining low latency performance. The simultaneous convergence of the five agents to the low latency range verifies the policy's synergy among multiple agents. Compared to the MAPPO algorithm, the PI-MAPPO algorithm offers even lower latency performance.

[0099] Example 3: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.

[0100] Example 4: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.

[0101] Example 5: This example proposes a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, they implement the steps of the method described in this invention, which will not be repeated here.

[0102] It should be noted that the processing flow of embodiments 3-5 corresponds to the specific steps of the method provided in embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in embodiment 1 of the present invention.

[0103] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An improved method for offloading multi-agent deep reinforcement learning, characterized in that, Includes the following steps: S1: Task unloading problem modeling; S2: Design and implementation of an improved unloading method.

2. The method of claim 1, wherein, Step S1 includes the following: S11, Edge Computing Modeling: If the mobile terminal decides to offload data to the edge server for computation, the following three steps are completed: First, the data, i.e., part or all of the task, is transmitted to the base station via the wireless channel; second, the arriving tasks are queued in the buffer of the base station server; and third, the server completes the above computation tasks in sequence according to the "first-come, first-served" principle. S12, Local Computational Modeling: Local computing does not involve network transmission; it models the computing latency, task queue, and computing power consumption of local devices. S13. Problem Statement: Constructing a target optimization problem of task offloading, for devices m The generated first n task , the agent determines the offloading ratio, and the task is calculated at the edge server and the local device respectively; The computational overhead of edge computing is ; definition For the task The completion latency depends on edge computing. With local computing The maximum latency corresponding to the two calculation modes, Written as: (17)； definition For the task Total energy consumption, including edge computing energy consumption and local computing power consumption , Represented as: (18)； Introducing task delay term This allows the system objective function to balance the timeliness of tasks with resource overhead, thus meeting the task offloading requirements in the context of the Internet of Vehicles. Considering latency, energy consumption, and computational overhead, the system objective function is defined. for: (19)； in, These are the weights for latency, energy consumption, and computational overhead, respectively, with values ​​ranging from [0,1]. The task offloading problem in the MEC scenario can be expressed in the following form: (20)。 3. The improved offloading method for multi-agent deep reinforcement learning according to claim 2, characterized in that, S11 includes the following steps: terminal equipment m The set is The system uses discrete time slots, each time slot t The duration is Δ Define terminal devices m In the t The first time slot generated at the start of the [number]th time slot n The task is Its expression is ,in for Data size, For the computational density of the task, Define the time delay threshold. It is a task The computational load is defined. For equipment m In the t The set of tasks generated at the beginning of each time slot, expressed as: , N For the first t The maximum number of tasks in a time slot is defined. For equipment m In the t The characteristics of all tasks in each time slot include data size, computation density, and latency constraints. S111, Wireless Transmission Rate Modeling: Assume the origin O is at the left edge of the base station's signal coverage area. For equipment m The initial distance from the origin O, For base station height, Define the device as the diameter of the base station signal coverage area. m movement speed Values ​​are positive from west to east and negative from east to west. In the [number]th [year]... t Each time slot, equipment m Distance to base station for: (1)； definition W The total bandwidth of the channel. M For the number of devices, For equipment m The transmission power, For noise spectral density, define This serves as a reference coefficient for path loss. Sampling is performed from an exponential distribution with a mean of 1. ω The path loss index represents the maximum power of the equipment. The distance between the mobile device and the base station affects the data transmission rate during task offloading. mobile terminal m In the t The wireless transmission rate for each time slot is: (2)； like Depending on the situation, obtain the corresponding transmission rate; S112, Server queue modeling: According to the "first-come, first-served" principle, when the number of arriving tasks exceeds the server's processing capacity, a task queue appears in the server's task buffer. t In the +1 time slot, the server's instantaneous task queue consists of two parts; the first part is the... t At the start of the time slot, the number of tasks already completed minus the server's workload. Δ The task of duration calculation, another part is the first t All tasks that are offloaded to the server by mobile devices within the time slot minus the server's processing time. Δ The task of duration calculation is defined. This refers to the actual amount of data unloaded onto the server, as defined. It is a task The uninstallation part, the expression is written as ,Will Called a time slot t Inner k The first task to arrive at the server, the... k The computational load of each arriving task , No. t +1 slot server instantaneous queue length for: (3)； in Calculate the server frequency, in units of , For the first The computation time obtained for each task For the first The number of tasks arriving at the MEC server within a time slot. For the first Instantaneous queue length of the edge server in a time slot; An exponentially weighted moving average mechanism is introduced to smooth the instantaneous queue length, thereby filtering out instantaneous noise that occurs during queue observation. This involves the exponentially weighted moving average of the task leader's value on the edge server. for: (4)； in The weight factor for the instantaneous leader is taken as a value within [0,1]. S113. Task latency modeling for edge computing: Edge computing task latency refers to the total time consumed by an edge server to complete a task, including transmission latency, computation latency, and waiting latency. First, we analyze transmission latency. Transmission of data from the source end to the channel generates transmission latency, which is defined as follows: For the task The uninstallation ratio, uninstallation decision The amount of data that terminal devices plan to offload to edge servers for: (5)； Define the device within each time slot m The maximum amount of data that can be sent is the initial upper limit of the transmission capacity. In a complete time slot t ,equipment m The initial upper limit of the transmission capacity is written as: (6)； The actual amount of data offloaded to the edge server Constrained by two factors: firstly, there is data in the previous time slot that has not been fully transmitted. Its value range is It will continue to transmit in the current time slot, and the actual transmission capacity of the current time slot is limited. for On the other hand, if the initial upper limit of transmission capacity is large, the actual offloading... Equal to the planned uninstall volume, Take the actual limit of transmission capacity With the planned unloaded data volume The minimum of the two is expressed as: (7)； Delay in sending unloading tasks for: (8)； Next, we analyze the server's computation latency, for the first... k If the task data arriving at the server is immediately delivered to the CPU for execution, the computational latency is: (9)； Analyzing the waiting latency, according to the "first-come, first-served" rule, for the first... k The first task to arrive at the server is waiting for the second... k -1 task completes computation, receives processing, and considers the ... k The sending delay and waiting delay of each task are determined by the larger of the two factors mentioned above. ; Edge computing task latency Represented as: (10)； Among them, the first k The sending delay of each arriving task The initial task latency is ; S114. Energy consumption modeling for edge computing: The energy consumed by terminal devices transmitting wireless signals to offload tasks, including the energy consumption of wireless transmission from mobile terminals and the energy consumption of edge computing. for: (11)； S115. Modeling the computational overhead of edge computing: Considering the computational overhead of edge servers, the computational overhead of edge computing Written as: (12)； in, The cost of computing for edge servers.

4. The improved offloading method for multi-agent deep reinforcement learning according to claim 2, characterized in that, S12 includes the following steps: S121. Queue modeling of terminal devices: When the workload of a terminal device exceeds its computing power, a task queue will appear on the terminal device. If the terminal device cannot complete the task calculation in the current time slot, the remaining tasks will continue to be calculated in the next time slot. In the t +1 time slot, the terminal's instantaneous queue length is the first... t The instantaneous queue length of the terminal device is calculated by adding the new task load to the time slot queue leader, minus the task's computational load. for: (13)； in For equipment m CPU calculation frequency, unit: same, For equipment m No. n The local computational load of each task is expressed as follows: ; Local queue length detection introduces the EWMA smoothing queue mechanism, which is a weighted moving average of the task queue length on the terminal device. for: (14)； S122. Local computation task latency modeling: Define the local computing task section ,in Indicates the size of the data calculated locally. This represents the local computation latency. Since there were unfinished tasks in the previous time slot, the local computation latency is considered in two parts: the first part is the latency of the previous time slot. t Task load that already exists at the start of the time slot The second part is the t New task load generated in time slots Local computing task latency for: (15)； S123, Energy consumption modeling for local computation: When some of the task's data is computed locally, the energy consumption of local computation... for: (16) in, This is the energy consumption factor calculated locally.

5. The improved offloading method for multi-agent deep reinforcement learning according to claim 2, characterized in that, The S2 improved unloading method design and implementation: treat each mobile terminal as an intelligent agent. P 1. Transform into a fully collaborative multi-agent reinforcement learning problem P 2. The mobile terminal randomly generates computing tasks. By observing node locations, wireless gain, task attributes, and task queues, the agents on the terminal coordinate their respective offloading strategies to achieve... P The optimization objective is 1; Includes the following steps: S21. The unloading problem based on Markov game theory: Define the set of intelligent agents as The global state space is The joint action space is defined as ,in Represents intelligent agents m Individual action space, intelligent agent m The reward function is The global state transition function is , the problem P 2. Modeled as a Markov game process, its expression is: The definitions of each element are as follows: No. t Time slot, intelligent agent m local state Including channel gain ,distance Task characteristics and average queue length of equipment for: (21)； Indicates the first t The global state of a time slot is determined by the edge server queue length. It consists of the local state of all agents: (22) Representation of intelligent agents m In the t Actions taken in time slots, including unloading ratios and transmission power : (23)； For intelligent agents m No. n The reward function for each task is used to guide the agent's policy learning. Represents intelligent agents m The individual reward is expressed as follows: , M Joint reward function for each agent for: (24)； According to formula (19), the system objective function Including latency, power consumption, and computational overhead, for the objective function Perform normalization, and define the normalized objective function as follows: ,have (25)； in , As a normalized reference scale, It is a very small positive number; In the reward function, design a smoothing penalty term, the Sigmoid function, and define the time delay bias. This reflects Deviation To determine the degree of time delay deviation, the normalized time delay deviation is used as the independent variable of the function, and the Sigmoid function value is obtained. exist When the signal is near the target area, the Sigmoid function provides a smoothing penalty, which will affect the normalized signal. As part of the reward function, a normalized reward function is defined. for: (26)； The Sigmoid function , As a weighting factor, Normalization factor; S22, Unloading Algorithm Design.

6. The improved multi-agent deep reinforcement learning offloading method according to claim 5, characterized in that, The S22 unloading algorithm includes the following steps: S221, MAPPO algorithm: In order to make independent unloading decisions, each agent Each includes an actor network A network of commentators Actor networks are used to generate actions based on local observations, with each agent... m Owns an independent actor network All agents share the same commentator network. During the execution phase, each agent relies solely on its own local observations to independently generate unloading decisions through its corresponding actor network, without needing to access the global state or decision information from other agents. The training and computation of all actor and commentator networks are performed by the MEC server. On the MEC server, the actor network is updated by sampling the dataset. and comment network ; During a training process, the agent first uses its current policy. Multiple rounds are generated, and then each agent... m Sample the dataset Uploaded to the MEC server; (27)； in For time steps, For the set of all sampling time steps, For sampling action, The corresponding sampling action probability; definition Indicates the first k The actor network at the next update The parameters, where Training objective function The core lies in minimizing the loss function Its specific expression is: (28)； in For shear loss, For entropy regularization, This is the entropy regularization coefficient, used to balance strategy optimization and exploration capabilities; definition For the first k Commentators network at the time of the next update V parameters It is trained by minimizing the following mean squared error loss: (29)； in From time step Initial cumulative discount rewards, new actor network parameters after training update. It is distributed to each intelligent agent to replace its original parameters; After interactive training using the MAPPO algorithm and an edge computing environment, the offloading ratio and transmit power output by the actor network were defined as follows: and ; S222, PI control algorithm: Design a task queue algorithm based on proportional-integral control, where the task queue length of the terminal device... Exceeding the threshold At the same time, the algorithm uses proportional-integral control theory to modify the unloading strategy output by the MAPPO algorithm; definition For intelligent agents m The local task queue length threshold, The maximum computing frequency supported by the local device's CPU, based on ,have In the t Each time slot defines the intelligent agent. m Captain error signal Average queue length With queue threshold The difference, for: (30)； Due to the existence of a task latency threshold, it is considered Greater than In this case, the error signal It is a negative value; definition For the integral element of the PI controller, By analyzing historical error signals The result is obtained by cumulative summation up to the nth. t Each time slot, captain error signal The cumulative expression is: (31)； definition The unloading ratio is the output of the PI controller. According to the PI control principle, to ensure... The interval is (0,1], written as: (32)； in For proportional gain, This is the integral gain; When the error signal of the local queue length When it exists, the intelligent agent m Under task overload conditions, a PI controller is used to control the output of the DRL-MAPPO policy network. Adjustments were made to the actual uninstallation ratio. for: (33)； After the PI controller adjusts the offload ratio of the DRL algorithm, to avoid task backlog on the device... m The transmit buffer, the actual transmit power of the mobile device. for: (34)； in This is the power adjustment factor, used to adjust the correction level of the PI controller.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is configured to implement the steps of the method according to any one of claims 1 to 6 when invoked by a processor.

9. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the method according to any one of claims 1 to 6.

Images