MAPPO edge computing task unloading method based on dominant value plus noise

Abstract

The invention discloses a GNN-MAPPO task unloading method based on dominant value noise addition, which is characterized in that an MLP is changed into a GNN on the basis of the existing MAPPO framework, a multi-agent system can be directly modeled into a graph structure, an interaction relationship among multiple agents can be better established, Gaussian noise is added on the dominant value, the exploration capability of a model is enhanced, and overfitting is reduced. According to the method, the powerful graph structure learning ability of GNN is combined with an innovative dominant value noise adding mechanism, and the mixed reward function is elaborately designed, so that the MAPPO algorithm can more effectively learn a cooperation strategy between agents and optimize time delay and energy consumption in the aspect of edge computing task unloading, and the efficiency of the MAPPO algorithm is improved. And the exploration capability of the strategy and the avoidance capability of the communication risk can be obviously enhanced, so that a more robust and efficient intelligent task unloading scheme adapting to a dynamic environment can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of distributed resource scheduling technology, and specifically relates to an edge computing task offloading method for MAPPO based on advantage value noise. Background Technology

[0002] With the commercialization of wireless communication technologies such as 5G, Wi-Fi 6, NB-IoT, and C-V2X (Cellular V2X), intelligent connected vehicles, as key mobile terminals, are experiencing an exponential increase in the amount of data and computing power required by their various sensors (cameras, LiDAR, millimeter-wave radar), onboard computing units (ECUs / domain controllers), and intelligent cockpit systems. Traditional cloud computing models, due to their centralized core nodes and long link distances, are unable to meet the numerous emerging business demands of the connected vehicle scenario.

[0003] 1. Millisecond-level end-to-end latency:

[0004] The closed loop of environmental perception, decision-making and planning, and control execution in advanced driver assistance systems (ADAS) and autonomous driving systems (AD) has extremely stringent latency requirements. For example, the latency budget for V2X applications such as cooperative collision warning and cooperative lane changing is usually less than 20-50ms; scenarios such as remote driving and platooning require ultra-low latency video transmission and control command issuance.

[0005] 2. Energy constraints:

[0006] ICV (Intelligent Connected Vehicle) relies on onboard batteries for power. High-intensity local computing (such as running complex deep learning models for target recognition or path planning) leads to a surge in energy consumption per unit time, which directly affects the vehicle's driving range.

[0007] 3. Network load and reliability:

[0008] Simultaneous uploading of raw sensor data (such as high-definition video streams and lidar point clouds) by a large number of ICVs to the cloud center can cause congestion in the cellular network core network or dedicated V2X network.

[0009] In scenarios such as tunnels, densely populated urban areas, and severe weather, wireless link interruptions or signal obstruction can easily lead to task unloading failures or excessive latency jitter, affecting driving safety.

[0010] Mobile edge computing (MEC) provides ICVs with local computing and storage resources by deploying small servers near the user side, such as wireless base stations, roadside units (RSUs), and access points. This significantly reduces transmission latency and alleviates pressure on the cloud side. A key technical challenge arising from this is how to intelligently determine whether each onboard computing task should be processed locally on the onboard computing unit or offloaded to a specific MEC server, under conditions of varying wireless channel conditions, random arrival of vehicle computing tasks, and uneven MEC server load. This aims to minimize the average task processing latency and energy consumption of the entire vehicle-to-everything (V2X) system while ensuring task processing reliability and meeting the QoS requirements of different applications.

[0011] Existing research mainly includes the following categories: 1) Centralized optimization based on integer programming or queuing theory, which can obtain the global optimum in small-scale vehicle and RSU problems, but the computational complexity increases exponentially with the number of vehicles, tasks, and RSUs; it requires prior knowledge of global information (such as the real-time location of all vehicles, task status, and channel quality), making it difficult to meet the real-time requirements of dynamic vehicle networking environments. 2) Heuristic or greedy algorithms (such as shortest delay offloading, highest computing power RSU priority, and nearest RSU priority), which are simple to implement and have low computational overhead, but are prone to getting trapped in local optima, lack the ability to adapt to dynamic traffic environments and network changes, and are difficult to achieve effective coordination among multiple vehicles. 3) Single-agent reinforcement learning (such as DQN, DDPG, etc.), which does not require building an accurate vehicle networking environment model and can be learned online, but is difficult to extend to complex vehicle networking scenarios with multiple vehicles, multiple RSUs, and multiple task types, and is more sensitive to the non-stationarity of the environment (caused by the decisions of other vehicles). 4) Multi-agent reinforcement learning (such as VDN, QMIX, MAPPO, etc.) has been validated on platforms like StarCraftII and MAMuJoCo through collaborative decision-making among multiple vehicle agents. However, MAPPO may still face problems such as insufficient exploration and policy degradation to local optima when dealing with computationally intensive unloading tasks requiring close cooperation among multiple vehicles (such as collaborative perception data sharing and fusion decision-making). MAPPO improves sample efficiency in multi-agent cooperative tasks by leveraging a centralized Critic, but it is also prone to overfitting to specific inter-vehicle cooperative unloading patterns due to over-reliance on global information and limited sampling advantages. Without intervention, the policy may be updated in suboptimal or even incorrect directions, missing better multi-vehicle cooperative unloading trajectories.

[0012] Existing MAPPO solutions for games suffer from insufficient exploration and susceptibility to local optima in high-dimensional collaborative vehicle network offloading scenarios. Traditional MAPPO uses MLP (Perceptron Neural Network), which loses a lot of structural information, forcing the algorithm to learn relationships that should be obvious from flattened data, resulting in performance degradation, poor generalization ability, and scalability issues. Traditional task offloading schemes also struggle to simultaneously address vehicle network system-level task processing latency, vehicle energy consumption, and the robustness of offloading decisions. Summary of the Invention

[0013] To address the shortcomings of existing MAPPO algorithms in high-dimensional collaborative vehicle-to-everything (V2X) offloading scenarios, such as insufficient exploration and susceptibility to local optima, traditional MAPPO uses a Multi-Level Perceptron (MLP), which loses a significant amount of structural information. This forces the algorithm to struggle to learn seemingly obvious relationships from flattened data, leading to performance degradation, poor generalization, and scalability issues. Furthermore, traditional task offloading schemes struggle to simultaneously balance V2X system-level task processing latency, vehicle energy consumption, and the robustness of offloading decisions. This application proposes a solution that replaces the MLP with a Graph Attention Network (GNN) (one type of GNN, GAT) within the existing MAPPO framework, enabling better modeling of interactions between multiple agents. Additionally, Gaussian noise is added to the advantage values ​​to enhance the model's exploration capabilities and reduce overfitting.

[0014] The technical solution provided by this invention is as follows:

[0015] A method for offloading edge computing tasks in MAPPO based on dominance value-based noise addition includes:

[0016] Step S1: Model the task unloading system;

[0017] The multi-agent task offloading process is modeled as a Markov decision process (MDP) or a multi-agent Markov decision process (MA-MDP), where each agent makes decisions in an environment that includes edge servers (MEC nodes), and a task offloading model that includes real-time energy consumption and latency constraints for agents is constructed.

[0018] Step S2: Design an immediate reward for task unloading and build an intelligent agent interaction environment that includes an immediate reward function for task unloading;

[0019] Based on the constructed task offloading system model, a global-differential hybrid task offloading instant reward is constructed that takes into account the total task completion latency of all agents and the total energy consumption of all agents.

[0020] Step S3: Build and train the GNN-MAPPO network;

[0021] By introducing a graph neural network into the existing MAPPO network framework, a GNN-MAPPO-based network consisting of an Actor-GNN subnetwork and a Critic-GNN subnetwork is obtained.

[0022] During training, the agent's local observation state is input into a GNN-MAPPO-based network. The Actor-GNN subnetwork provides the agent's decisions, and the Critic-GNN subnetwork evaluates these decisions to obtain an assessment of future expected value. Based on the agent's decisions and interactions with the environment, an immediate reward for task unloading is obtained. Combining this immediate reward with the Critic-GNN subnetwork's value assessment, the temporal difference (TD) error and baseline advantage value are calculated. Noise is added to the baseline advantage value. Based on the immediate reward for task unloading and the noisy baseline advantage value, a loss function based on the GNN-MAPPO-based network is calculated. The gradient of the loss function is then calculated via backpropagation, and gradient descent is used to update the parameters of the Actor-GNN and Critic-GNN subnetworks based on the GNN-MAPPO-based network. This training is repeated until a specified number of iterations is reached, resulting in a trained GNN-MAPPO-based network.

[0023] Step S3: Build and train the GNN-MAPPO network;

[0024] By introducing a graph neural network into the existing MAPPO network framework, a GNN-MAPPO-based network consisting of an Actor-GNN subnetwork and a Critic-GNN subnetwork is obtained.

[0025] During training, the agent's local observation state is input into a GNN-MAPPO-based network. The Actor-GNN subnetwork provides the agent's decisions, and the Critic-GNN subnetwork evaluates the agent's decisions to obtain an assessment of future expected value. Then, based on the designed task unloading immediate reward, the task unloading immediate reward, TD error, and baseline advantage value are calculated. Noise is added to the baseline advantage value, and the loss function based on the task unloading immediate reward and the noisy baseline advantage value is calculated. The gradient of the loss function is then calculated through backpropagation, and the parameters of the Actor-GNN and Critic-GNN subnetworks based on the GNN-MAPPO-based network are updated using gradient descent. Training is repeated until a specified number of iterations is reached to obtain a trained GNN-MAPPO-based network.

[0026] Step S4: Deployment and Inference;

[0027] The trained GNN-MAPPO network is used to make deployment decisions based on the agent's current observations, and the decisions are sent to the corresponding MEC nodes for task offloading.

[0028] Furthermore, the MLP in the existing MAPPO network framework is replaced with GNN, resulting in a GNN-MAPPO-based network composed of Actor-GNN subnetworks and Critic-GNN subnetworks;

[0029] The GNN used is GAT.

[0030] The core advantage of GNNs lies in their ability to directly process graph-structured data and learn the relationships and patterns between nodes. In multi-agent systems, agents and their interactions naturally form a graph.

[0031] Graph Construction: Nodes: Each agent They are all nodes in the graph.

[0032] Node Features (X): Local observations of each agent As the initial feature vector of the corresponding node.

[0033] Edges (A / Edge Index): The connections between agents.

[0034] Furthermore, the process of adding noise to the benchmark dominance value is as follows:

[0035] .

[0036] in, This represents the baseline advantage value of agent i after adding noise. This indicates adaptive exploration – utilizing a balance factor, dynamically adjusting based on training time t using cosine decay. This represents the baseline dominance value before noise is added. The risk aversion coefficient is a hyperparameter used to adjust the agent's overall sensitivity or aversion to risk. ; This represents Gaussian noise that follows a standard normal distribution. Indicates a penalty based on recent V2X communication status:

[0037] .

[0038] The weight representing the risk of packet loss is set to 0.7. The weight representing the risk of timeout is set to 0.3. A global variable representing time step t. Indicates joint action, and These represent penalties for packet loss and timeout, respectively.

[0039] The larger the value, the higher the agent's risk aversion, and the more inclined it is to choose even primitive methods. Actions with low risk but also very low risk.

[0040] The difference between the inline, action-specific risk adjustment item and the MAPPO noise-adding formula in games is a core consideration added to the connected vehicle scenario, and it is directly derived from the fundamental advantages. The values ​​are then subtracted or adjusted, and then weighted and noise is added.

[0041] Injecting structured noise into the dominant signal enhances exploration and mitigates risks.

[0042] This scheme chooses cosine decay because it decays slowly at the beginning, quickly in the middle, and then slows down again at the end, resulting in a very smooth decay that performs well in many deep learning tasks.

[0043] The role of advantage is to tell the Actor-GNN whether "your action is better or worse than the average," guiding the direction of policy updates. Adding noise to it is to encourage the policy to explore directions that may not currently appear optimal but could have potential value.

[0044] Furthermore, the baseline advantage value after adding noise is normalized and mini-batch segmented to obtain the segmented data used to calculate the loss function. The specific process is as follows:

[0045] Normalization: Calculate the mean and standard deviation of all the baseline dominance values ​​after adding noise, and then standardize them;

[0046] This keeps the scale of the advantage value consistent, resulting in more stable training.

[0047] Segmentation: The normalized data is randomly shuffled and then divided into multiple mini-batches of fixed size.

[0048] Furthermore, the total energy consumption global-differential hybrid instantaneous reward includes the agent's instantaneous reward and the global instantaneous reward, which are calculated using the following formulas:

[0049] Instant rewards for each agent:

[0050] .

[0051] in, This represents the instantaneous reward of agent i at time t. This represents the globally weighted cost calculated when all agents take the current joint action at time step t. It is the weighted sum of latency and energy consumption minimized under design constraints, where latency and energy consumption cannot exceed a set threshold. This refers to the hyperparameters that are set by the user and adjusted based on the training results. This represents the differential reward of agent i. Indicates a global soft constraint penalty;

[0052] ,

[0053] .

[0054] This means that "except for agent i, which takes a predefined baseline action to unload to the nearest MEC, other agents remain in..." If the actions remain unchanged, recalculate the global cost. , It's weight. Maintaining the same dimensions; under the "delay priority" condition, the initial value is... = 0.7, = 0.3; Under the "energy consumption priority" scenario, the initial value can be taken as 0.3. = 0.3, = 0.7, and It represents the total latency and total energy consumption of all tasks in the task unloading system at this time step.

[0055] .

[0056] in, Both α and denot represent positive weighting coefficients, 0 ≤ α ... ≤ 1, Used to balance the proportion of latency and energy consumption in cost calculations ; This represents the total system delay at time step t. This represents the total system energy consumption at time step t. This represents the total allowable latency threshold of the system. This represents the total energy consumption threshold allowed by the system. and All of these represent positive adjustable penalty coefficients, used to balance the penalty intensity when the constraint is exceeded;

[0057] The value of β is usually determined based on experience or experimental tuning to reflect the relative importance of time delay and energy consumption in the optimization objective;

[0058] Global instant reward formula:

[0059] .

[0060] in, Indicates global instant reward

[0061] The immediate reward for each agent is used to calculate the advantage value of each agent, thereby updating the Actor-GNN network;

[0062] The global instant reward is then used to calculate the TD error, thereby updating the Critic-GNN network;

[0063] Clear constraints and penalties ensure service quality:

[0064] By penalizing behaviors that exceed the system's total latency and total energy consumption thresholds through global soft constraints, the method ensures that task offloading is optimized while meeting basic service quality requirements. The adjustability of the weight coefficients α, β, and penalty coefficients allows the method to flexibly adjust the optimization focus according to different scenarios (such as "latency priority" or "energy consumption priority").

[0065] Secondly, a MAPPO edge computing task offloading system based on dominance value noise addition includes:

[0066] Task offloading system model: A Markov process of MEC node integrating agents and edge servers is used to construct a task offloading model of agents under real-time energy consumption-latency constraints.

[0067] Instant reward design module: Based on the task unloading system model, construct a global-differential hybrid instant reward for task unloading that takes into account both the total task completion latency of all agents and the total energy consumption of all agents.

[0068] Interactive environment building module: Builds the intelligent agent's interactive environment based on the designed task unloading instant reward;

[0069] GNN-MAPPO Network Construction and Training Module: A graph neural network is introduced into the existing MAPPO network framework to obtain a GNN-MAPPO-based network composed of Actor-GNN subnetwork and Critic-GNN subnetwork.

[0070] During training, the agent's local observation state is input into the GNN-MAPPO network. The Actor-GNN subnetwork provides the agent's decision, and the Critic-GNN subnetwork evaluates the agent's decision to obtain an assessment of future expected value. Then, based on the designed task unloading immediate reward, the task unloading immediate reward, TD error, and baseline advantage value are calculated. Noise is added to the baseline advantage value, and the loss function based on the task unloading reward and the noisy baseline advantage value is calculated. The gradient of the loss function is then calculated through backpropagation, and the parameters of the Actor-GNN and Critic-GNN subnetworks based on the GNN-MAPPO network are updated using gradient descent. The training is repeated until the specified number of iterations is reached to obtain the trained GNN-MAPPO network.

[0071] Deployment and Inference Module: Utilizes the trained GNN-MAPPO network to make deployment decisions based on the agent's current observations, and sends the decisions to the corresponding MEC nodes for task offloading.

[0072] Furthermore, the MLP in the existing MAPPO network framework is replaced with GNN, resulting in a GNN-MAPPO-based network composed of Actor-GNN subnetworks and Critic-GNN subnetworks;

[0073] The GNN used is GAT.

[0074] Furthermore, the process of adding noise to the benchmark dominance value is as follows:

[0075] .

[0076] in, This represents the baseline advantage value of agent i after adding noise. This indicates adaptive exploration – utilizing a balance factor, dynamically adjusting based on training time t using cosine decay. This represents the baseline dominance value before noise is added. The risk aversion coefficient is a hyperparameter used to adjust the agent's overall sensitivity or aversion to risk. > 0; This represents Gaussian noise that follows a standard normal distribution. Indicates a penalty based on recent V2X communication status:

[0077] .

[0078] The weight representing the risk of packet loss is set to 0.7. The weight representing the risk of timeout is set to 0.3. A global variable representing time step t. Indicates joint action, and These represent penalties for packet loss and timeout, respectively.

[0079] Thirdly, a computer storage medium stores a computer program, which is invoked by a processor to perform the following:

[0080] The steps of the above-described edge computing task offloading method for MAPPO based on dominance value noise addition.

[0081] Fourthly, a computer device comprising at least:

[0082] One or more processors;

[0083] A memory that stores one or more computer programs;

[0084] The processor calls the computer program to implement:

[0085] The steps of the above-described edge computing task offloading method for MAPPO based on dominance value noise addition.

[0086] Beneficial effects:

[0087] This invention introduces the GNN-MAPPO method based on advantage value noise, combining the powerful graph structure learning capability of GNN with the innovative advantage value noise mechanism, and carefully designing a hybrid reward function. This enables the MAPPO algorithm to not only learn the cooperation strategy between agents more effectively and optimize latency and energy consumption in the edge computing task offloading problem, but also significantly enhance the strategy exploration capability and the ability to avoid communication risks, thereby obtaining a more robust, efficient and dynamic intelligent task offloading solution.

[0088] Traditional MAPPO uses Multi-Agent Learning Platforms (MLPs) to handle local observations, which struggles to effectively capture complex dynamic interactions and dependencies between agents. This invention replaces MLPs with Generative Neural Networks (GNNs) with Generative Attraction and Interaction Modeling (GATs). By enhancing state representation and interaction modeling, GNNs can directly model multi-agent systems as graph structures, where agents are nodes and their interactions are edges. Through message passing and aggregation mechanisms, GNNs enable each agent to perceive and integrate state information from neighboring agents (and even more distant agents), thereby learning richer environmental representations and inter-agent interactions. This allows Actor-GNNs and Critic-GNNs to make decisions and evaluations based on more comprehensive information, improving the overall performance of multi-agent cooperative task offloading and reducing total latency and energy consumption.

[0089] In edge computing environments (especially V2X scenarios), network conditions, available resources, and task loads can all change dynamically. GNNs can flexibly handle changing graph structures (such as agents joining / leaving, and changes in communication links), making the trained policies more robust and adaptable to these dynamic changes, thus improving their adaptability to dynamic environments.

[0090] Introducing Gaussian noise into the dominant signal and adjusting it with an adaptive factor of cosine decay encourages the agent to explore more extensively in the early stages of training, trying offloading strategies that may seem suboptimal at present but have potentially high rewards. As training progresses, the exploration intensity gradually decreases, and the agent tends to utilize the learned superior strategies. This mechanism of adding noise to the dominant value to achieve better exploration—utilizing a balance and risk avoidance mechanism—helps to escape local optima and discover globally better offloading schemes, thus enhancing exploration capabilities.

[0091] Proactively mitigate potential risks and improve system reliability:

[0092] By introducing a risk penalty term based on V2X communication state, which directly adjusts the baseline advantage value, the agent proactively avoids offloading actions that could lead to high packet loss rates or high communication latency during decision-making. A risk aversion coefficient allows for adjustment of risk sensitivity based on the specific application scenario. This enables task offloading decisions to not only optimize latency and energy consumption but also consider communication reliability, making it particularly suitable for edge computing scenarios sensitive to communication quality (such as vehicle-to-everything (V2X) connections). It proactively avoids potential risks and improves system reliability.

[0093] Unlike simple random noise, the noise injection in this invention is structured, consisting of part random exploration (Gaussian noise) and part guidance targeting specific risks (V2X communication risks). This structured noise makes exploration and risk avoidance more targeted, improving learning efficiency and the practicality of the final strategy.

[0094] A global-differential hybrid immediate reward mechanism effectively solves the multi-agent credit allocation problem. Individual agent rewards include global cost, differential rewards, and global soft constraint penalties, incentivizing agents to contribute to the global objective (minimizing the weighted sum of total latency and energy consumption) while also considering the impact of their actions on the global picture. The global reward is used to train the Critic network, providing an accurate assessment of the value of the current state. This refined reward mechanism guides the optimization direction, improves overall system performance, and achieves an effective trade-off between energy consumption and latency. Attached Figure Description

[0095] Figure 1 This is a flowchart illustrating the task unloading method described in the technical solution of the present invention;

[0096] Figure 2This is a schematic diagram of the network training process constructed for the technical solution of this invention. Detailed Implementation

[0097] The present invention will now be further described in conjunction with the accompanying drawings and embodiments.

[0098] Example 1:

[0099] A method for offloading edge computing tasks in MAPPO based on dominance value-based noise addition, such as... Figure 1 As shown, it includes:

[0100] Step S1: Model the task unloading system;

[0101] Markov processes of MEC nodes that integrate agents with edge servers are used to construct a task offloading model for agents under real-time energy consumption and latency constraints.

[0102] Define system elements:

[0103] There are N vehicles, each representing an intelligent agent. Let i represent the i-th agent, which is the i-th vehicle, i = 1, ..., N;

[0104] There are M edge servers, and MEC node j represents the j-th edge server;

[0105] The local execution option is equivalent to MEC node j=0;

[0106] Each vehicle i is an independent decision-making agent, whose onboard unit is responsible for collecting information, running the Actor-GNN network, and making offloading decisions.

[0107] MEC servers can be deployed on roadside units (RSUs) or as edge computing nodes attached to cellular network base stations.

[0108] Define task attributes, that is, each task generated for vehicle i:

[0109] Input data size is (bits), that is, the amount of raw task data generated by vehicle i;

[0110] Output data size: The amount of data that needs to be sent back to vehicle i after the (bit) task is completed;

[0111] Required computation: (For example, CPU cycles) The number of instruction cycles required to complete the task;

[0112] Define node capabilities and states:

[0113] MEC Node j = 1, ..., M;

[0114] Computational ability: (Cycles / second), the total number of CPU cycles that MEC node j can process per second;

[0115] Current load rate: The current computing resource utilization of MEC node j. ;

[0116] Calculate the energy consumption factor: The energy consumed by MEC node j per CPU cycle;

[0117] Vehicle i ( ), i = 1,…,N:

[0118] Transmit power: The transmit power of the OBU communication module of vehicle i;

[0119] Receive power: The power consumption of the communication module of the OBU of vehicle i when receiving data via V2X communication;

[0120] Local computing power: The number of CPU cycles that the OBU local processing unit of vehicle i can process per second;

[0121] Local energy consumption factor (optional): The energy consumed per CPU cycle when the OBU of vehicle i performs calculations locally;

[0122] Channel uplink rate: (bits / second), the estimated data transmission rate of vehicle i uploading data to MEC node j via V2X;

[0123] Channel downlink rate: (bits / second), the estimated data transfer rate of MEC node j downloading data to vehicle i via V2X;

[0124] The channel rate is obtained by sending pilot sequences from the terminal to the MEC base station, and the base station measures and feeds back channel quality information (such as CQI), which is then combined with the historical throughput moving average.

[0125] Performance metrics for task i being unloaded to node j (if j=0, it means it is executed locally):

[0126] Transmission delay: (If j = 0, the transmission delay is 0).

[0127] , .

[0128] The computation delay is Queuing delay is ,

[0129] The latency of a task in the waiting queue of MEC node j or the local processing unit. This value depends on the current load of the node and the task scheduling policy, and is typically determined dynamically in the simulation environment or estimated through benchmarking.

[0130] Total latency: .

[0131] Transmission energy consumption is The chip's transmit power consumption is The calculated energy consumption is

[0132] Total energy consumption: .

[0133] Some of the latency and energy consumption parameters mentioned above (such as channel rate and queuing delay) are dynamically changing, while others (such as node computing power and task data volume) can be considered known or determined at the time of task generation. Latency and energy consumption are obtained during system operation through benchmarking using specialized tools and testing methods.

[0134] Step S2: Design the instant reward for task unloading, and build the intelligent agent interaction environment based on the designed instant reward for task unloading;

[0135] Based on the constructed task offloading system model, a global-differential hybrid task offloading instant reward is constructed that takes into account the total task completion latency of all agents and the total energy consumption of all agents.

[0136] The total energy consumption global-differential hybrid task unloading instant reward includes the agent's instant reward and the global instant reward, which are calculated using the following formulas:

[0137] Instant rewards for each agent:

[0138] .

[0139] in, This represents the instantaneous reward of agent i at time t. This represents the globally weighted cost calculated when all agents take the current joint action at time step t. It is the weighted sum of latency and energy consumption minimized under design constraints, where latency and energy consumption cannot exceed a set threshold. This refers to the hyperparameters that are set by the user and adjusted based on the training results. This represents the differential reward of agent i. Indicates a global soft constraint penalty;

[0140] ,

[0141] .

[0142] This means that, except for agent i, which takes a predefined baseline action to unload to the nearest MEC, other agents remain in their current MEC. If the actions remain unchanged, recalculate the global cost. , It's weight. Maintaining the same dimensions; under the "delay priority" condition, the initial value is... = 0.7, = 0.3; Under the "energy consumption priority" scenario, the initial value can be taken as 0.3. = 0.3, = 0.7, and It represents the total latency and total energy consumption of all tasks in the task unloading system at this time step.

[0143] .

[0144] in, and Each represents a positive weighting coefficient, 0 ≤ ≤ 1, Used to balance the proportion of latency and energy consumption in cost calculations ; This represents the total system delay at time step t. This represents the total system energy consumption at time step t. This represents the total allowable latency threshold of the system. This represents the total energy consumption threshold allowed by the system. and All of these represent positive adjustable penalty coefficients, used to balance the penalty intensity when the constraint is exceeded;

[0145] The value of β is usually determined based on experience or experimental tuning to reflect the relative importance of time delay and energy consumption in the optimization objective;

[0146] < 0 indicates that the current action of vehicle i reduces the overall cost compared to the baseline action (the contribution is positive because the reward is negative).

[0147] > 0 indicates that the current action of vehicle i increases the overall cost compared to the baseline action (the contribution is negative).

[0148] Global instant reward formula:

[0149] .

[0150] in, This indicates a global, immediate reward.

[0151] The immediate reward for each agent is used to calculate the advantage value of each agent, thereby updating the Actor-GNN network;

[0152] The global instant reward is then used to calculate the TD error, thereby updating the Critic-GNN network;

[0153] Clear constraints and penalties ensure service quality:

[0154] By using global soft constraints to penalize behaviors that exceed the system's total latency and total energy consumption thresholds, task offloading is optimized while still meeting basic quality of service requirements. Weighting coefficients , The adjustability of the penalty coefficient allows the method to flexibly adjust the optimization focus according to different scenarios (such as "latency priority" or "energy consumption priority").

[0155] To train and evaluate reinforcement learning agents, a dynamic simulation environment conforming to the OpenAI Gym interface needs to be built, which mainly includes:

[0156] `reset()`: Called at the start of each training round, responsible for initializing or resetting the environment state, including:

[0157] 1. Randomly generate the initial positions and speeds of N vehicles on the map.

[0158] 2. Initialize the status of M MEC servers (initial load is 0).

[0159] 3. Clear all task queues.

[0160] 4. Return the initial observations of all agents.

[0161] step(): Executed at each time step t, including:

[0162] 1. Receive joint action: Input the joint action of all agents at the current time step.

[0163] 2. Update environment status:

[0164] Channel changes: Based on the new vehicle-MEC distance, the channel gain is updated, thereby recalculating the transmission rate.

[0165] Task generation: Generate new computational tasks for some or all vehicles based on stochastic processes such as the Poisson distribution.

[0166] 3. Perform the action and calculate the result:

[0167] For each vehicle i, the latency and energy consumption generated by its task are calculated according to the set formula, and the load rate and task queue of the selected MEC server are updated.

[0168] 4. Calculate system-level metrics: Summarize the latency and energy consumption of all tasks;

[0169] 5. Calculate rewards: Calculate the global immediate reward and the final reward for each agent using a predefined formula;

[0170] 6. Generate the next observation: Generate a new local observation for each agent i;

[0171] 7. Return the results.

[0172] Step S3: Build and train the GNN-MAPPO network;

[0173] By introducing a graph neural network into the existing MAPPO network framework, a GNN-MAPPO-based network consisting of an Actor-GNN subnetwork and a Critic-GNN subnetwork is obtained.

[0174] The existing MLP in the MAPPO network framework is replaced by GNN, and a GNN-MAPPO-based network is composed of Actor-GNN subnetwork and Critic-GNN subnetwork.

[0175] The GNN used is GAT.

[0176] Actor-GNN (Policy Network):

[0177] Shared parameters: Each vehicle i has one They share parameters ;

[0178] Actor-GNN input: Local observations of vehicle i at time step t Local graph structure related to vehicle i , This includes the vehicle itself, its neighboring vehicles, and connectable MECs, etc. This indicates the connection relationships between these nodes.

[0179] Actor-GNN output: a probability distribution representing the probability that the vehicle will choose each possible action given the observations.

[0180] Critic-GNN (Value Network) The parameters are :

[0181] Input: Global state: , This represents the node feature matrix. Each node in the graph has a set of features, and the features of vehicle nodes are based on their local observations. The characteristics of an MEC node can be its load. wait, It can be an adjacency matrix or an edge list, representing the connection relationship between vehicles and MECs in the global graph.

[0182] like Figure 2 As shown, during training, the agent's local observation state is input into the GNN-MAPPO network. The Actor-GNN subnetwork provides the agent's decision, and the Critic-GNN subnetwork evaluates the agent's decision to obtain an assessment of future expected value. Then, based on the designed task unloading immediate reward, the task unloading immediate reward, TD error, and baseline advantage value are calculated. Noise is added to the baseline advantage value, and the loss function based on the task unloading immediate reward and the noisy baseline advantage value is calculated. The gradient of the loss function is then calculated through backpropagation, and the parameters of the Actor-GNN and Critic-GNN subnetworks based on the GNN-MAPPO network are updated using gradient descent. The training is repeated until the specified number of iterations is reached to obtain the trained GNN-MAPPO network.

[0183] The specific process is as follows:

[0184] Step S3-1. Roll-out (data collection phase).

[0185] Objective: To interact with the environment and accumulate... Step experience, a total of One cycle.

[0186] To acquire observations, each agent i acquires its current local observations from the environment. ;

[0187] Get action ,Will and local graph structure Input the Actor-GNN corresponding to each vehicle's intelligent agent Sampling obtains action and the corresponding log probability

[0188] Forming joint actions The total latency is calculated by interacting with the environment (by executing the step() function). Total energy consumption The global instant reward is then obtained from the reward function. Instant rewards for each agent Next step of observation With the end mark .

[0189] Build a global state and include all vehicle intelligent agents. and MEC load And the adjacency matrix of the nodes is concatenated into Similarly, the global state at the next time step can be obtained.

[0190] Store experience, tuples of experience for each agent. To the Buffer. Shared by all intelligent agents. It is unique to each intelligent agent.

[0191] If the environment is reset, If `= true`, the environment is reset, and the next round of interaction begins.

[0192] Step S3-2: Calculate the TD objective and advantage estimates (for each agent i);

[0193] This step prepares the necessary target and advantage values ​​for subsequent Critic-GNN and Actor-GNN network updates.

[0194] Calculate TD target (For future Critic-GNN updates), utilizing global instant rewards (As obtained from the environmental interaction and reward calculation in the previous step), Critic-GNN estimates the value of the current global state (obtained from the buffer). ; Critic-GNN's value estimation of the next global state (obtained from the buffer) Obtain the TD target of Critic-GNN .

[0195] Calculate advantage estimation (For each agent i, used in subsequent Actor-GNN updates): First, calculate the first-order TD error for agent i. : Utilizing the individual instant reward of agent i Critic-GNN estimates the value of the current global state (i.e., the expected global reward). Critic-GNN estimates the value of the next global state (i.e., the expected global reward). Calculate the TD error of agent i Then, the generalized advantage estimate of agent i is calculated using the GAE method. : Through a series of TD errors of agent i (from the current time step and several subsequent steps) Obtain the advantage value of agent i .

[0196] Using the individual instant reward calculated in the previous step and global instant rewards (used for construction) The TD target for updating the Critic-GNN network is calculated using the Critic-GNN network's value estimates for the current and next states. and the advantages of using Actor network updates .

[0197] Step S3-3 generates the noisy advantage;

[0198] a) Sampling noise: Before each PPO update begins (i.e., before processing the entire buffer), Gaussian noise is sampled independently for each agent i. .

[0199] b) Calculate the advantage with noise: For each piece of empirical data b (corresponding to time step τ) of agent i in the buffer:

[0200] The process of adding noise to the benchmark advantage value is as follows:

[0201] ;

[0202] in, This represents the baseline advantage value of agent i after adding noise. This indicates adaptive exploration – utilizing a balance factor, dynamically adjusting based on training time t using cosine decay. This represents the baseline dominance value before noise is added. The risk aversion coefficient is a hyperparameter used to adjust the agent's overall sensitivity or aversion to risk. > 0; This represents Gaussian noise that follows a standard normal distribution. Indicates a penalty based on recent V2X communication status:

[0203]

[0204] .

[0205] The weight representing the risk of packet loss is set to 0.7. The weight representing the risk of timeout is set to 0.3. A global variable representing time step t. Indicates joint action, and These represent penalties for packet loss and timeout, respectively.

[0206] The larger the value, the higher the agent's risk aversion, and the more inclined it is to choose even primitive methods. Actions with low risk but also very low risk.

[0207] The difference between the inline, action-specific risk adjustment item and the MAPPO noise-adding formula in games is a core consideration added to the connected vehicle scenario, and it is directly derived from the fundamental advantages. The values ​​are then subtracted or adjusted, and then weighted and noise is added.

[0208] Injecting structured noise into the dominant signal enhances exploration and mitigates risks.

[0209] This scheme chooses cosine decay because it decays slowly at the beginning, quickly in the middle, and then slows down again at the end, resulting in a very smooth decay that performs well in many deep learning tasks.

[0210] The role of advantage is to tell the Actor-GNN whether "your action is better or worse than the average," guiding the direction of policy updates. Adding noise to it is to encourage the policy to explore directions that may not currently appear optimal but could have potential value.

[0211] Step S3-4. Mini-batch processing and key value calculation:

[0212] The baseline advantage value after adding noise is normalized and mini-batch segmented to obtain the segmented data used to calculate the loss function. The specific process is as follows:

[0213] Normalization: Calculate the mean and standard deviation of all the baseline dominance values ​​after adding noise, and then standardize them;

[0214] Segmentation: The normalized data is randomly shuffled and then divided into multiple mini-batches of fixed size.

[0215] (a) Extract its sample data ( , , , ),

[0216] (b) Calculate the log probability of the new policy based on the current shared policy parameter θ. Calculate from the current Actor-GNN: Input the stored local observations into the current Actor-GNN (parameter θ), and then query for the stored actions. The network outputs the action probability, and then the logarithmic value is taken.

[0217] (c) Calculate the probability ratio: .

[0218] This is the action to be taken under the new strategy. The probability of performing the same action is the same as under the old strategy. The ratio of probabilities. It is a measure of the magnitude of policy change in the PPO algorithm.

[0219] Step S3-5. Calculate the loss function:

[0220] Actor-GNN Loss (PPO-Clip) - Shared Parameter θ: When the Actor-GNN parameter θ is shared by all agents, the Actor-GNN loss Lᴬᶜᵗᵒʳ(θ) is the average (or summed and normalized) loss of all agent samples in the mini-batch:

[0221] in: Indicates the clipping parameters. This represents the expectation of the empirical samples of all agents i in the mini-batch.

[0222] Critic-GNN Loss (Value Function Loss): The loss of Critic-GNN Using mean squared error (MSE): ;

[0223] in, It is the TD value in S3-2.

[0224] Step S3-6. Parameter Update:

[0225] Update the shared Actor-GNN parameters θ: Calculate the total Actor-GNN loss. The gradient ∇θ with respect to the shared parameter θ The parameters in the Actor-GNN network are updated using gradient descent.

[0226] Update the Critic-GNN parameters φ: Calculate the Critic-GNN loss. The gradient ∇φ with respect to the shared parameter φ .

[0227] Update the parameters in the Critic-GNN network using gradient descent.

[0228] Step S4: Deployment and inference of GNN-MAPPO in the Internet of Vehicles;

[0229] In this stage, the trained GNN-MAPPO model will be applied to the vehicle-to-everything (V2X) environment, using only the Actor-GNN subnetwork, so that the computing units on each vehicle can independently make computational offloading decisions.

[0230] Before the vehicle can make autonomous decisions, the following preparations need to be completed:

[0231] Distribution and loading of decision models:

[0232] Each vehicle receives parameters for the decision model (i.e., the Actor-GNN network) and saves these model parameters in an efficient format (such as PyTorch Mobile) for quick loading and execution on the vehicle's On-Board Unit (OBU).

[0233] Configure the runtime environment:

[0234] Ensure that each vehicle's OBU has the software required to run the decision model installed (including the corresponding inference engine and other necessary dependency libraries).

[0235] 2. Execute online:

[0236] For each vehicle that requires a decision:

[0237] a. Acquiring local observation information, similar to the acquisition of local observation information during the training process.

[0238] b. Calculate decision-making actions: The vehicle inputs the formatted local observation information into its already loaded self-decision model (Actor-GNN network);

[0239] The decision model outputs a probability distribution that represents the probability of choosing each possible action for the current observation.

[0240] Decision examples (e.g., in a connected vehicle task offloading scenario, possible actions include):

[0241] Action 0: Execute the task on the vehicle's local computing unit.

[0242] Action 1: Unload the task to MEC 1.

[0243] Action 2: Unload the task to MEC 2.

[0244] Action M: Unload the task to MEC M.

[0245] During the actual reasoning (deployment) phase, the vehicle will choose the action with the highest probability as the final decision.

[0246] c. Send the decision to the execution module:

[0247] Based on the calculated decision action, the vehicle will perform the corresponding operation: if the decision is to be executed locally: the task is assigned to the vehicle's computing unit for processing; if the decision is to be offloaded to a remote node: the vehicle will send the task data to the target execution node for remote execution: the MEC node uses its computing resources to execute the task; after the task is completed, the target node will return the calculation results to the vehicle that initiated the task through the communication link.

[0248] Example 2:

[0249] An edge computing task offloading system for MAPPO based on dominance value noise addition, comprising:

[0250] Task offloading system model: A Markov process of MEC node integrating agents and edge servers is used to construct a task offloading model of agents under real-time energy consumption-latency constraints.

[0251] Instant reward design module: Based on the task unloading system model, construct a global-differential hybrid instant reward for task unloading that takes into account both the total task completion latency of all agents and the total energy consumption of all agents.

[0252] Interactive environment building module: Builds the intelligent agent's interactive environment based on the designed task unloading instant reward;

[0253] GNN-MAPPO Network Construction and Training Module: A graph neural network is introduced into the existing MAPPO network framework to obtain a GNN-MAPPO-based network composed of Actor-GNN subnetwork and Critic-GNN subnetwork.

[0254] During training, the agent's local observation state is input into a GNN-MAPPO-based network. The Actor-GNN subnetwork provides the agent's decisions, and the Critic-GNN subnetwork evaluates the agent's decisions to obtain an assessment of future expected value. Then, based on the designed task unloading immediate reward, the task unloading immediate reward, TD error, and baseline advantage value are calculated. Noise is added to the baseline advantage value, and the loss function based on the task unloading immediate reward and the noisy baseline advantage value is calculated. The gradient of the loss function is then calculated through backpropagation, and the parameters of the Actor-GNN and Critic-GNN subnetworks based on the GNN-MAPPO-based network are updated using gradient descent. Training is repeated until a specified number of iterations is reached to obtain a trained GNN-MAPPO-based network.

[0255] Deployment and Inference Module: Utilizes the trained GNN-MAPPO network to make deployment decisions based on the agent's current observations, and sends the decisions to the corresponding MEC nodes for task offloading.

[0256] It should be understood that the specific implementation process of each module unit is described in the above method. This invention will not repeat it here. The above division of functional modules is only for illustrative purposes. In some embodiments, some functional modules can be merged and some functional modules can be split. Each functional module can be implemented in software, hardware, or a combination of software and hardware. The software and hardware devices include, but are not limited to, general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0257] Example 3:

[0258] A computer storage medium storing a computer program, which is invoked by a processor to implement:

[0259] The steps of the above-described edge computing task offloading method for MAPPO based on dominance value noise addition.

[0260] Please refer to the explanation of the method above for the specific implementation process of each step.

[0261] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the hardware and software device described in any of the foregoing embodiments, such as the hard drive or memory of the controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, or Flash Card equipped on the controller. Further, the readable storage medium can include both internal storage units and external storage devices of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0262] Example 4:

[0263] A computer device comprising at least:

[0264] One or more processors;

[0265] A memory that stores one or more computer programs;

[0266] The processor calls the computer program to implement:

[0267] The steps of the above-described edge computing task offloading method based on dominance value-based noise addition in GNN-MAPPO.

[0268] Please refer to the explanation of the method above for the specific implementation process of each step.

[0269] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0270] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0271] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application refers to flowchart illustrations and / or instructions executed by a processor of a method, apparatus (system), and computer program product according to embodiments of this application to create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0272] It should be emphasized that the examples described in this invention are illustrative rather than limiting. Therefore, this invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of this invention, without departing from the spirit and scope of this invention, whether modifications or substitutions, are also within the protection scope of this invention.

Claims

1. An edge computing task offloading method based on MAPPO with advantage value noise, characterized in that, Comprising: Step S1: task offloading system modeling; Modeling the multi-agent task offloading process as a Markov decision process or a multi-agent Markov decision process, wherein each agent makes decisions in an environment containing edge servers, and constructing a task offloading model containing real-time energy consumption and latency constraints of the agent; Step S2: task offloading immediate reward design, and building an agent interaction environment containing a task offloading immediate reward function; Based on the constructed task offloading system model, a global-difference hybrid task offloading immediate reward is constructed, which takes into account the total task completion latency of all agents and the total energy consumption of all agents; Step S3: building and training a GNN-MAPPO network; Introducing a graph neural network into the existing MAPPO network framework to obtain a GNN-MAPPO network composed of an Actor-GNN subnetwork and a Critic-GNN subnetwork; During training, the local observation state of the agent is input into the GNN-MAPPO network, the Actor-GNN subnetwork is used to give the decision of the agent, the Critic-GNN subnetwork is used to evaluate the decision of the agent, the evaluation of the future expected value is obtained, the task offloading immediate reward is obtained by interacting with the environment according to the decision of the agent, and the timing difference error and the baseline advantage value are calculated by combining the immediate reward and the value evaluation of the Critic-GNN subnetwork; The baseline advantage value is added with noise, the loss function of the GNN-MAPPO network is calculated based on the task offloading immediate reward and the noise-added baseline advantage value, the gradient of the loss function is calculated by back propagation, and the parameters of the Actor-GNN subnetwork and the Critic-GNN subnetwork of the GNN-MAPPO network are updated by gradient descent, and the training is repeated until a specified number of iterations is reached, and a trained GNN-MAPPO network is obtained; Step S4: deployment and reasoning; The trained GNN-MAPPO network is used to make deployment decisions for the current observation of the agent, and the decisions are sent to the corresponding MEC node for task offloading.

2. The method of claim 1, wherein, The GNN replaces the MLP in the existing MAPPO network framework, and the GNN-MAPPO network is composed of an Actor-GNN subnetwork and a Critic-GNN subnetwork. The GNN used is GAT.

3. The method of claim 1, wherein, The process of adding noise to the baseline advantage value is as follows: ； wherein, represents the baseline advantage value of the agent i after adding noise, represents the adaptive exploration-exploitation balance factor, which is dynamically adjusted according to the training time t using cosine decay, represents the baseline advantage value before adding noise, represents the risk aversion coefficient, which is a hyperparameter, used to adjust the overall sensitivity or aversion degree of the agent to risk, and ; represents the Gaussian noise, which is subject to a standard normal distribution, represents the penalty based on the recent V2X communication state: ； a weight representing the risk of packet loss, set to 0.7, a weight representing the risk of timeout, set to 0.3, a global variable representing the time step t, a joint action, and represent the risk of packet loss penalty and timeout risk penalty, respectively.

4. The method of claim 3, wherein, The noise-added baseline advantage value is normalized and Mini-batch split to obtain split data for calculating the loss function, and the specific process is as follows: Normalization: Calculate the mean and standard deviation of all noise-added baseline advantage values for standardization; Split: Randomly shuffle the normalized data, and then split it into multiple fixed-size Mini-batches.

5. The method of claim 1, wherein, The total energy consumption global-difference hybrid task offloading immediate reward includes the immediate reward of the agent and the global immediate reward, which are calculated by the following formulas respectively: The immediate reward of each agent: ； wherein, represents the immediate reward of the agent i at time t, represents the global weighted cost calculated when all agents take the current joint action at time step t, i.e., the weighted sum of the minimized delay and energy consumption under the designed constraint condition that the delay and energy consumption cannot exceed the set threshold, represents the hyperparameter set by itself, which is adjusted according to the training result, represents the differential reward of the agent i, represents the global soft constraint penalty; ， ； denotes "when all the agents but agent i take a pre-defined baseline action to offload to the nearest MEC, and agent i keeps its action unchanged in , the global cost is recalculated; , is the weight, is kept dimensionless; in the case of "latency-priority", the initial value of = 0.7, = 0.3; in the case of "energy-priority", the initial value of = 0.3, = 0.7, and are the total latency and total energy consumption of all tasks in the task offloading system at this time step; ； wherein, and both represent a positive weight coefficient, 0 ≤ ≤ 1, for balancing the proportion of latency and energy consumption in cost calculation, ; represents the total latency of the system at time step t, represents the total energy consumption of the system at time step t, represents the total latency threshold allowed by the system, represents the total energy consumption threshold allowed by the system, and both represent positive adjustable penalty coefficients for balancing the penalty strength when the constraints are exceeded.

6. An edge computing task offloading system based on MAPPO with advantage value plus noise, characterized in that, Comprising: A task offloading system model: a Markov process of a MEC node integrating agents and edge servers, to build a task offloading model of agents under real-time energy consumption-delay constraints; An instant reward design module: based on the task offloading system model, to construct a global-difference hybrid task offloading instant reward considering the total task completion delay of all agents and the total energy consumption of all agents; An interactive environment building module: to build an agent interaction environment according to the designed instant reward; A GNN-MAPPO network construction and training module: to introduce a graph neural network into an existing MAPPO network framework to obtain a GNN-MAPPO network composed of an Actor-GNN subnetwork and a Critic-GNN subnetwork; During the training process, the local observation state of the agent is input into the GNN-MAPPO network, the Actor-GNN subnetwork is used to give the decision of the agent, the Critic-GNN subnetwork is used to evaluate the decision of the agent to obtain the evaluation of the future expected value, and then the instant reward, the TD error and the baseline advantage value are calculated according to the designed task offloading instant reward; The baseline advantage value is added with noise, the loss function of the GNN-MAPPO network is calculated based on the task offloading instant reward and the noise-added baseline advantage value, the gradient of the loss function is calculated by back propagation, and the parameters of the Actor-GNN subnetwork and the Critic-GNN subnetwork of the GNN-MAPPO network are updated by gradient descent, the training is repeated until a specified number of iterations is reached, and a trained GNN-MAPPO network is obtained; A deployment and reasoning module: to use the trained GNN-MAPPO network to make deployment decisions for the current observation of the agent and send the decisions to the corresponding MEC node for task offloading.

7. The system of claim 6, wherein, The GNN replaces the MLP in the existing MAPPO network framework, and the GNN-MAPPO network is composed of an Actor-GNN subnetwork and a Critic-GNN subnetwork. The GNN used is GAT.

8. The system of claim 6, wherein, The process of adding noise to the baseline advantage value is as follows: ； wherein, represents the baseline advantage value of the agent i after adding noise, represents the adaptive exploration-exploitation balance factor, which is dynamically adjusted according to the training time t using cosine decay, represents the baseline advantage value before adding noise, represents the risk aversion coefficient, which is a hyperparameter, used to adjust the overall sensitivity or aversion degree of the agent to risk, and ; represents the Gaussian noise, which is subject to a standard normal distribution, represents the penalty based on the recent V2X communication state: ； a weight representing the risk of packet loss, set to 0.7, a weight representing the risk of timeout, set to 0.3, a global variable representing the time step t, a joint action, and respectively represent the penalty for the risk of packet loss and the penalty for the risk of timeout.

9. A computer storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the edge computing task offloading method based on advantage value noise MAPPO according to any one of claims 1-5.

10. A computer device, comprising: At least contains: One or more processors; A memory storing one or more computer programs; Wherein, the processor calls the computer program to implement: The steps of the edge computing task offloading method based on advantage value noise MAPPO according to any one of claims 1-5.

Images