MEC system multi-agent cooperation framework integrating AoI and internal excitation

Abstract

The invention discloses an MEC system multi-agent cooperation framework fusing AoI and internal excitation, and belongs to the technical field of cloud computing and edge computing. The method comprises the following steps: end-side heterogeneous resource generation and edge movement decision; modeling a communication link; collecting heterogeneous resources and limiting an AoI threshold value; isomerizing an age sensitive optimization problem; the MDP formulates an external reward function and an internal reward function; constructing a heterogeneous multi-agent Actor-Critic, and carrying out functional analysis on the heterogeneous multi-agent Actor-Critic; intrinsic rewards identified based on local information. According to the method, the computing power is extended to the edge of the network through the MEC, the real-time performance of data processing is improved, and the delay is reduced. The AoI concept is introduced to measure the data freshness, and efficient utilization of resources is realized and dynamic data requirements are met by optimizing resource management and scheduling strategies. A multi-agent deep reinforcement learning strategy is provided, the data freshness is ensured, and the cooperation efficiency between devices is improved.

Description

Technical Field

[0001] The present invention relates to a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation, and belongs to the technical field of cloud computing and edge computing. Background Art

[0002] Data sources in the manufacturing industry are very diverse, including various data from equipment, sensors, production lines, product tracking systems, etc., and involve different formats, structures, frequencies, and real-time requirements. These heterogeneous data require effective governance and integration to provide valuable insights for enterprises. Edge computing (EC) achieves lower latency and more efficient real-time data processing by migrating data processing capabilities from the cloud to the edge of the network. Mobile edge computing (MEC) further optimizes this process, especially for applications with mobility requirements. MEC pushes computing, storage, and network capabilities close to the data source, thereby solving the latency and bandwidth pressure caused by excessive data volume in cloud computing centers. In the manufacturing industry, MEC can effectively cope with the massive heterogeneous data generated by equipment, sensors, and systems, especially in industrial environments that require real-time response. For example, in smart factories, tasks such as equipment monitoring, production line optimization, and fault warning all rely on low-latency and high-reliability computing capabilities, and MEC provides an ideal solution.

[0003] MEC systems usually adopt a three-layer structure (such as Figure 1 As shown in the figure, this structure helps to provide efficient edge computing services. The bottom layer is the source of data, which refers to the device closest to the user device connected to the network, usually including sensors, smartphones, and smart home devices. The middle layer is the edge device, which can be a smart router, smart base station, unmanned aerial vehicle (UAV), and a vehicle with an onboard processor, which plays a relay role. Edge devices can collect data from edge sensors, assist in performing some computing tasks, and then relay them to the cloud center. The top-level cloud data center plays the role of central control in the edge computing solution. It integrates and analyzes data from edge nodes and assembles tasks according to actual service needs for processing at the edge. However, the resources of the edge server are not unlimited. If the tasks are unevenly distributed, it will cause the preemption of communication resources and edge server computing resources, resulting in increased latency and energy consumption in executing tasks.

[0004] This study focuses on the real-time and low-latency performance of image and video transmission processing in MEC systems. In key applications such as urban monitoring, data processing delay directly affects service quality. The age of information (AoI) is introduced to measure the freshness of data. In extreme cases, AoI thresholds are set, such as data exceeding the threshold in disaster response quickly loses value. Therefore, controlling AoI below the threshold and reducing the total AoI are the core goals.

[0005] In MEC systems, due to the limited computing resources of mobile edge devices and the dynamic changes in data tasks, coupled with communication bandwidth limitations, it is crucial to design effective scheduling strategies to optimize resource usage and meet data needs. This involves resource management, edge decision-making, and data offloading. The global optimal strategy requires cooperation and coordination among edge devices, but resource heterogeneity brings challenges. Different devices and cloud centers have different capabilities, resources, goals, and perception capabilities, and devices in different geographical locations generate data of different importance.

[0006] To address these challenges, the MEC system is considered as a multi-agent system, in which edge devices such as drones and onboard processors act as agents with intelligence and autonomy. Multi-agent deep reinforcement learning (MADRL) provides a new approach to solving sequential decision-making problems. Value-based methods such as Q-learning and DQN make decisions by predicting action values, while policy-based methods such as DDPG are suitable for tasks with large or continuous control spaces. Federated learning (FL) combined with MADRL allows agents to collaborate and learn while protecting privacy, thereby optimizing system performance.

[0007] In view of the above background and challenges, the present invention proposes a new MADRL strategy for heterogeneous AoI and resource collaboration problems of different edge devices in the MEC environment to ensure data freshness and collaboration efficiency between devices, and attempts to solve the problem of multi-agent dependence on global information under heterogeneous AoI and resource conditions. Make full use of Internet of Things technology and sensor networks to achieve real-time monitoring and data collection of various devices and sensors. At the same time, through the data processing capabilities of edge computing nodes, preliminary analysis and preprocessing of data can be achieved, reducing the data processing pressure on the cloud. Summary of the invention

[0008] In order to solve the problems existing in the background technology, the present invention provides a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: a multi-agent collaboration framework of a MEC system integrating AoI and intrinsic motivation, the method comprising the following steps:

[0010] S1: generation of heterogeneous resources on the edge and edge mobility decision-making;

[0011] S101: generating a data source for a device on the terminal side;

[0012] S10101: The terminal device collects computing task data generated by intelligent devices of distributed sensors in the Internet of Things to generate a data source;

[0013] S10102: Due to differences in geographical environment and hardware configuration, data sources form heterogeneous resource data and generate data packets in different forms;

[0014] S10103: For each data source S n Assign weights W Sn , to indicate the importance of different locations and assuming that the data sources are independently generated;

[0015] S10104: Define the data size of the data packet as d(t), the data generation time of the data packet as w(t), and the data source index as idx(t).

[0016] S102: Edge device E k Collect data packets from data sources on the move and offload the data to the cloud data center after processing it locally;

[0017] S103: Establishing mobility model of edge devices:

[0018] POS k (t+1)=pos k (t)+move k (t) (1)

[0019] In formula (1):

[0020] POS k (t) = [x k (t),y k (t),h k (t)] represents the edge device E k At the position of the tth time slot, x represents the horizontal distance difference, y represents the vertical distance difference, and h represents the height difference;

[0021] |move k (t) 2 ≤r k move represents the movement in each time slot;

[0022] r k move represents the edge device E k Moving radius;

[0023] | 2 Represents the 2-norm of a vector;

[0024] S104: Define movement, data collection, and local execution as core decision-making content in a multi-agent deep reinforcement learning framework.

[0025] S2: Communication link modeling;

[0026] S201: It is clarified that the transmission links between agents in the three-layer system environment include source-edge, edge-edge, and edge-cloud;

[0027] S202: Since edge devices only share states, observations, and learning parameters, data and task transmission is not involved. Therefore, the transmission cost of edge-to-edge communication is ignored;

[0028] S203: Modeling the source-edge and edge-cloud transmission processes;

[0029] S20301: Considering line-of-sight and non-line-of-sight path losses, an air-ground channel is introduced;

[0030]

[0031] In formula (2):

[0032] f represents the carrier frequency;

[0033] c represents the speed of light;

[0034] It is expressed as the distance between the edge device and the ground entity;

[0035] η ξ represents the path loss in line-of-sight and non-line-of-sight situations, ξ={0,1};

[0036] S20302: The average air-ground path loss of the communication channel between the edge device and the data source or between the edge device and the cloud center C in the tth time slot is obtained as:

[0037]

[0038] In formula (3):

[0039] p 1 (t) represents the probability of non-line-of-sight;

[0040] represents the probability of line of sight; where: a and b both represent parameters related to the environment;

[0041] Represents the angle between the edge link and the horizontal plane.

[0042] S204: Considering the frequency division mode with a total bandwidth of W, the transmission rate between the edge device and the data source or between the edge device and the cloud center C is calculated:

[0043]

[0044] In formula (4):

[0045] b k,n (t) represents the allocated bandwidth ratio;

[0046] N 0 represents the noise power spectral density;

[0047] Indicates the power required for transmission.

[0048] S3: Heterogeneous resource acquisition and AoI threshold limit;

[0049] S301: When the edge device collects heterogeneous resource data, the edge device runs to the vicinity of the data source to collect all data packets in the data source buffer and occupies the edge device data collection buffer A data block in

[0050] S302: Using the local processor to process the collected data buffer Perform pre-processing;

[0051] S303: Establish a time model for edge processing performed by edge devices in time slot t for data packets collected from data sources:

[0052]

[0053] In formula (5):

[0054] f c k (t) represents the data rate at which the edge device preprocesses the preset task in time slot t, which is related to the CPU cycle frequency;

[0055] S304: Assuming that in each time slot, the edge device allocates its edge computing resources to a data block in the buffer, the edge execution decision of the t-th time slot is represented by a one-hot vector as follows:

[0056]

[0057] In formula (6):

[0058] exe i (t)∈{0,1} represents the CPU allocation flag of each data block in the collection data buffer of the edge device;

[0059] S305: Edge device locally executes decision:

[0060]

[0061] S306: Cache the data in the executed data buffer and wait for unloading to the cloud center.

[0062] S307: Assuming that in each time slot, the edge device decides to unload a data packet in the data buffer, the unloading schedule is represented by a one-hot vector as follows:

[0063]

[0064] In formula (8):

[0065] off i (t)∈{0,1} represents the offloading decision of each data block;

[0066] S308: Local offloading scheduling of edge devices:

[0067]

[0068] S309: To quantify heterogeneity, HA-MAAC-Trans introduces weights for data sources to highlight the importance of different locations and corrects the packet size by weights:

[0069]

[0070] In formula (10):

[0071] d(t) represents the data size of the data packet;

[0072] Indicates the weight of the data source;

[0073] S3010: Establish a data value depreciation model:

[0074]

[0075] In formula (11):

[0076] d n,0 (t) represents the size of the initial data packet collected from the data source;

[0077] λ represents the penalty factor of the depreciation model, which is responsible for adjusting the speed of value reduction after exceeding the threshold;

[0078] χ represents the AoI threshold;

[0079] AoI(t) represents the information delay of each data packet at time t;

[0080] S3011: Data value is determined by the information delay of the data packet.

[0081] S3012: The edge device determines the subsequent route based on the value of the depreciated data.

[0082] S4: Heterogeneous age-sensitive optimization problem;

[0083] S401: Define the age of the data source in the tth time slot as the difference Δ between the current time and the generation time of the latest data at the receiving end n (t):

[0084]

[0085] In formula (12):

[0086] Indicates the generation time of the latest data packet of the data source received by the cloud center;

[0087] S402: Define the total number of data sources as N s , then a total data source number dimension vector can be used to record the age of each data source;

[0088] S403: Taking the system model as a constraint, we get the NP-hard optimization problem:

[0089]

[0090]

[0091] C2:pos k (t+1)=pos k (t)+move k (t)

[0092] C3: move k (t) 2 ≤r k move

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] S404: Discuss solutions based on MADRL.

[0104] S5: MDP formulates external reward function and intrinsic reward function;

[0105] S501: MDP is represented by a quaternion as follows:

[0106] M{S,A,R,P}(13)

[0107] In formula (13)

[0108] S stands for status;

[0109] A indicates action;

[0110] R stands for reward;

[0111] P represents the transfer strategy;

[0112] S502: Edge agent represents the local state M of the quadruple K {S K ,A K ,R K ,P K};

[0113] S503: The central agent represents the global state M of the quadruple C {S C ,A C ,R C ,P C};

[0114] S504: The actions of the edge agent consist of movement, execution decision and unloading scheduling:

[0115] {a k (t)}={[move k (t),exe k (t),off k (t)]} (14)

[0116] S505: Action a of the central agent c (t) is the allocated bandwidth ratio b(t);

[0117] S506: The synergy between extrinsic and intrinsic rewards.

[0118] S50601: Agent collaboration in HA-MAAC-Trans to minimize the average age of data sources for the goal;

[0119] S50602: Define the amount of data received by the cloud center as S(t), which is used to evaluate the data processing efficiency of the system at the current moment and motivate the system to process the backlog data;

[0120] S50603: The external reward of each agent in the tth time slot Described as a weighting of the amount of data and the average age of the data source:

[0121]

[0122] In formula (15):

[0123] α, β are weight coefficients;

[0124] S50604: Calculate intrinsic rewards

[0125]

[0126] In formula (16):

[0127] P μ represents a personalized classifier, which is used to perform personalized classification on the local information of each edge agent;

[0128] P μ (ko k ) represents the probability that the current observation range generated by the personalized classifier is suitable for the edge device;

[0129] ω LISIM represents the adjustment weight;

[0130] S50605: HA-MAAC-Trans defines a global reward function and combines it with a decaying long-term reward to study the global optimality of the system. The decaying long-term reward is as follows:

[0131]

[0132] In formula (17):

[0133] γ∈[0,1] represents reward / penalty decay;

[0134] S6: Heterogeneous multi-agent Actor-Critic construction and function analysis;

[0135] S601: constructing a heterogeneous multi-agent Actor-Critic, including a central Actor network, a central Critic network, an edge Actor network, and an edge Critic network; the edge Actor network and the edge Critic network both contain a Transformer-enhanced neural network to improve the ability of spatial data feature enhancement;

[0136] S602: Actor Network A k (s k (t);θ k ), taking the state of each agent as input and outputting the current action a k (t);

[0137] S603: For each agent, design a Critic network C based on the Actor network framework k (s k (t),a k (t); φ k ), combined with the value assessment module, input the current state and action estimation state-action value function Q k (s k (t),a k (t));

[0138] S604: Design edge actor network;

[0139] S60401: Build a multi-input-output neural network to integrate various edge observation information, learn and process diverse operations;

[0140] S6040101: The inputs to the multi-input-output neural network include local observations from data sources, edge buffer states, and offload channel states;

[0141] S6040102: Format the local observation data of the data source as A feature map of the image, extracting edge observation information based on the features of the feature map;

[0142] S60402: Build a CNN network to extract key spatial features of areas with large data packets and high AoI;

[0143] S60403: Convert feature maps into sequences and input them into the Transformer encoder, using the Transformer network's multi-head self-attention mechanism to capture long-distance dependencies;

[0144] S60404: Combine feature maps with CNN features to output device trajectories and action decisions;

[0145] S60405: Use MLP to process the execution buffer, completion buffer and bandwidth allocation to extract the edge state.

[0146] S605: Design center Actor network;

[0147] The central Actor network takes the status information of the edge device as input, and the Actor network outputs a one-sum vector representing the bandwidth allocation ratio. It uses MLP to combine the status of multiple devices to optimize the communication scheduling between the center and the edge and allocate bandwidth ratios for edge-center communication.

[0148] S606: Design edge critic network;

[0149] The input of the edge critic network includes the state information s of the edge devicek (t) and the action strategy a of the edge device output by the edge Actor network k (t);

[0150] S60601: The edge observation information is preliminarily processed through the convolution layer to extract local features;

[0151] S60602: Input local features into the Transformer network to capture the temporal characteristics of the observation information, thereby enhancing the understanding of the dynamic changes in the edge node environment;

[0152] S60603: Use multiple fully connected layers and ReLU activation functions to process the time series characteristics and mine the complex features of the state information layer by layer, so that the edge critic network can more accurately judge the scene in the current state and extract the edge state;

[0153] S60604: Different types of edge actions are processed separately through independent FC-ReLU branches to ensure effective extraction of different action features;

[0154] S60605: The processed action, state and observation information are connected and integrated in the network, and the estimate of the action value function is finally output To evaluate the value of each action in the current state;

[0155] S607: Design Center Critic Network;

[0156] The central critic network inputs edge device status information and central bandwidth control, and estimates the value function through the fully connected Softmax layer.

[0157] S7: Intrinsic rewards based on local information recognition.

[0158] S701: HA-MAAC-Trans introduces a local information self-identification module to explore the personalized characteristics of edge devices through self-supervised individual classifiers;

[0159] S702: Local information self-identification module as the intrinsic reward function of HA-MAAC-Trans The goal is to train a global probability classifier p with μ as a parameter μ , let the input be the observation o k , the output is the probability p of the observation belonging to each edge device μ (·|o k ), so that from the given different observations o k An edge device is identified in the model to model the independent actions of each edge device; the intrinsic reward p is defined μ (k|ok ) represents the possibility of accurately predicting / identifying a certain edge device from the observation. It can be seen that the total reward of the edge device is expressed as a combination of external reward and intrinsic reward:

[0160]

[0161] In formula (18):

[0162] ∑ k p μ (k|o k )=1;

[0163] ω LISIM represents the adjustment weight, which is used to measure the importance between intrinsic rewards and extrinsic rewards;

[0164] Represents extrinsic rewards from the environment.

[0165] Compared with the prior art, the present invention has the following beneficial effects:

[0166] 1. The present invention extends computing power to the edge of the network through mobile edge computing (MEC) to improve the real-time performance of data processing and reduce latency. In particular, the present invention focuses on the low-latency requirements of real-time data processing, introduces the concept of age of information (AoI) to measure data freshness, and strives to maintain AoI below an acceptable threshold. In addition, in the face of the challenges of limited edge device resources and dynamic changes in system load, the present invention optimizes resource management and scheduling strategies to achieve efficient resource utilization and meet dynamic data needs. Finally, the present invention proposes a new multi-agent deep reinforcement learning (MADRL) strategy to solve the problems of heterogeneous AoI and resource collaboration in the MEC environment, ensure the freshness of data and improve the efficiency of collaboration between devices.

[0167] 2. The present invention proposes the HA-MAAC-Trans framework, which models data offloading as a non-convex and NP-hard optimization problem to address the heterogeneous AoI problem existing in edge devices. HA-MAAC-Trans constructs a multi-agent reinforcement learning framework by integrating the heterogeneous AoI threshold optimization mechanism, and realizes the collaborative learning of edge and central models, which not only ensures the freshness of data, but also maximizes the cumulative rewards of the system. In response to the problem of agent collaboration efficiency in multi-agent reinforcement learning, HA-MAAC-Trans introduces the Local Information Self-Identification Module (LISIM). The LISIM module models the cooperative behavior of heterogeneous agents by realizing the self-identification of each agent's state, thereby overcoming the limitation of traditional multi-agent reinforcement learning on global information dependence. BRIEF DESCRIPTION OF THE DRAWINGS

[0168] Figure 1 It is a schematic diagram of a three-layer mobile edge computing system;

[0169] Figure 2 This is a schematic diagram of the reinforcement learning collaboration framework based on HA-MAAC-Trans;

[0170] Figure 3 Schematic diagram of the neural network model of each Actor-Critic agent, where: (a) is the edge Actor network, (b) is the center Actor network, (c) is the edge Critic network, and (d) is the center Critic network;

[0171] Figure 4 It is a schematic diagram of the impact of different AOI thresholds and depreciation factors on the amount of data received by the cloud center, where: (a) is the AoI threshold, (b) is the depreciation factor;

[0172] Figure 5 This is a comparison diagram of the worst AoI of different Actor-Critc networks;

[0173] Figure 6 Schematic diagrams comparing Actor-Critc networks designed with different network structures, where: (a) is a schematic diagram comparing the average age of different network structures, (b) is a schematic diagram comparing the amount of data received by different network structures, and (c) is a schematic diagram comparing the packet count received by different network structures;

[0174] Figure 7 It is a comparison diagram of different weighted parameters, where: (a) is a comparison diagram of the average age of each parameter, α represents the weight of the number of tasks completed by the performance indicator system; β represents the weight of the average age of the total sample of the performance indicator; (b) is a comparison diagram of the amount of data received by the cloud center of each parameter, the curve of α=0.5, β=0.5 indicates that the final amount of data received is second only to the highest value;

[0175] Figure 8 Schematic diagram of the comparison of edge device trajectories obtained by different RL-based MEC collaboration algorithms, where the red trajectory represents edge device No. 1, the green trajectory represents edge device No. 2, the blue trajectory represents edge device No. 3, and the magenta trajectory represents edge device No. 4; the data points enclosed by the dotted circle are data points that the edge device may not be able to collect; (a) is DDPG, (b) is MADDPG, (c) is H-MAAC, and (d) is HA-MAAC-Trans;

[0176] Fig. 9It is a 200×200 map of four edge devices and a comparative diagram of RL-based MEC collaboration algorithms for 30 data sources, where (a) is a diagram showing the average age change of the MEC system, (b) is a diagram showing the worst (maximum average) AoI change in the data source, (c) is a diagram showing the amount of data received by the cloud center, and (d) is a diagram showing the count of data packets received by the cloud center. DETAILED DESCRIPTION

[0177] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0178] A multi-agent collaboration framework for a MEC system integrating AoI and intrinsic motivation, the method comprising the following steps:

[0179] S1: generation of heterogeneous resources on the edge and edge mobility decision-making;

[0180] S101: generating a data source for a device on the terminal side;

[0181] S10101: The terminal device collects computing task data generated by intelligent devices of distributed sensors in the Internet of Things to generate a data source;

[0182] S10102: Due to differences in geographical environment and hardware configuration, data sources form heterogeneous resource data and generate data packets in different forms;

[0183] S10103: For each data source S n Assign weights to each To indicate the importance of different locations, assuming that the data sources are independently generated;

[0184] S10104: Define the data size of the data packet as d(t), the data generation time of the data packet as w(t), and the data source index as idx(t).

[0185] S102: In the MEC system, edge device E k It has mobility and computing capabilities, and can collect data packets from data sources while on the move, process the data locally, and then offload the data to the cloud data center;

[0186] S103: Establishing mobility model of edge devices:

[0187] POS k (t+1)=pos k (t)+movek (t) (1)

[0188] In formula (1):

[0189] POS k (t) = [x k (t),y k (t),h k (t)] represents the edge device E k At the position of the tth time slot, x represents the horizontal distance difference, y represents the vertical distance difference, and h represents the height difference;

[0190] |move k (t) 2 ≤r k move represents the movement in each time slot;

[0191] r k move represents the edge device E k Moving radius;

[0192] | 2 Represents the 2-norm of a vector;

[0193] S104: Define the three main actions of edge devices - movement, data collection, and local execution as the core decision-making content in the multi-agent deep reinforcement learning framework.

[0194] S2: Communication link modeling;

[0195] S201: It is clarified that the transmission links between agents in the three-layer system environment include source-edge, edge-edge, and edge-cloud;

[0196] S202: Since edge devices only share states, observations, and learning parameters, and do not involve the transmission of data and tasks, the transmission cost of edge-to-edge communication is ignored; this feature simplifies the communication model of the system and reduces the computational complexity.

[0197] S203: Modeling the source-edge and edge-cloud transmission processes;

[0198] S20301: Considering the line-of-sight (LoS) and non-line-of-sight (NLoS) path losses, the air-to-ground (A2G) channel is introduced;

[0199]

[0200] In formula (2):

[0201] f represents the carrier frequency;

[0202] c represents the speed of light;

[0203] It is expressed as the distance between the edge device and the ground entity (a selected data source or cloud center);

[0204] η ξ represents the path loss in line-of-sight and non-line-of-sight situations, ξ={0,1};

[0205] S20302: The average air-ground path loss of the communication channel between the edge device and the data source or between the edge device and the cloud center C in the tth time slot is obtained as:

[0206]

[0207] In formula (3):

[0208] p 1 (t) represents the probability of non-line-of-sight;

[0209] Indicates the probability of line of sight, which can approximately express the dynamic changes of the path loss model; where: a and b both represent parameters related to the environment; Represents the angle between the edge link and the horizontal plane.

[0210] S204: Considering the frequency division mode with a total bandwidth of W, the transmission rate between the edge device and the data source or between the edge device and the cloud center C is calculated:

[0211]

[0212] In formula (4):

[0213] b k,n (t) represents the allocated bandwidth ratio;

[0214] N 0 represents the noise power spectral density;

[0215] Indicates the power required for transmission.

[0216] This process takes into account factors such as the channel's signal-to-noise ratio (SNR), transmission power, and noise power spectral density, providing a theoretical basis for evaluating the communication performance of source-edge and edge-cloud links. This modeling approach effectively describes the characteristics of the transmission process and optimizes the system's communication and computing collaboration capabilities.

[0217] S3: Heterogeneous resource acquisition and AoI threshold limit;

[0218] S301: When the edge device collects heterogeneous resource data, the edge device runs to the vicinity of the data source to collect all data packets in the data source buffer and occupies the edge device data collection buffer A block of data (collected but unprocessed data)

[0219] S302: Using the local processor to process the collected data buffer Perform pre-processing;

[0220] S303: Due to the determinism of data packet formatting and edge preprocessing algorithms, a time model for the edge device to perform edge processing on data packets collected from the data source in time slot t is established based on the data size and edge computing rate:

[0221]

[0222] In formula (5):

[0223] It represents the data rate at which the edge device preprocesses the preset task in time slot t, which is related to the CPU cycle frequency;

[0224] S304: Unprocessed data packets are stored in the buffer of the edge device, waiting for the allocation of computing resources. After processing, they are cached in the executed buffer, and finally decide whether to offload to the cloud center to ensure the reasonable allocation and processing of data blocks in each time slot. Assuming that in each time slot, the edge device allocates its edge computing resources to a data block in the buffer, the edge execution decision of the tth time slot is represented by a one-hot vector as follows:

[0225]

[0226] In formula (6):

[0227] exe i (t)∈{0,1} represents the CPU allocation flag of each data block in the collected data buffer of the edge device, which obviously satisfies the condition;

[0228] S305: Edge device locally executes decision:

[0229]

[0230] S306: Cache the data in the executed data buffer and wait for unloading to the cloud center.

[0231] S307: Assuming that in each time slot, the edge device decides to unload a data packet in the data buffer, the unloading schedule is represented by a one-hot vector as follows:

[0232]

[0233] In formula (8):

[0234] off i (t)∈{0,1} represents the offloading decision of each data block, and they also satisfy the condition;

[0235] S308: Local offloading scheduling of edge devices:

[0236]

[0237] S309: In actual scenarios, due to the heterogeneity of end-side devices, different types of devices (such as drones, cameras, and IoT sensors) will generate data of different quantities and importance due to differences in geographical locations and computing power. This heterogeneity leads to differences in data processing time and age of information (AoI). Therefore, different data source acquisition models are needed to describe the data acquisition process. In order to quantify heterogeneity, HA-MAAC-Trans introduces weights for data sources, which are affected by factors such as economic activities, population density, and traffic conditions to highlight the importance of different locations. Location weights are affected by factors such as economic activities, traffic conditions, population density, and natural disaster risks. For example, areas with rich economic activities may be more important than areas with relatively less economic activities because they have greater demand for business and services. After investigation, it was found that the most important locations in a region account for 5% of the total, and the more important locations account for 25% of the total. The data packet size is corrected by weights:

[0238]

[0239] In formula (10):

[0240] d(t) represents the data size of the data packet;

[0241] Indicates the weight of the data source;

[0242] S3010: The age of information (AoI) of data is an important goal of system optimization. However, due to the mobility of edge devices, strict hard constraints are difficult to achieve. To this end, HA-MAAC-Trans proposes a soft constraint mechanism. When the AoI value exceeds the set threshold, a penalty factor is introduced through a depreciation model to reduce the effective value of the data. This model dynamically adjusts the data value based on information delay, thereby guiding edge devices to optimize the route and resource allocation to achieve a balance between data processing and transmission efficiency. Establish a data value depreciation model:

[0243]

[0244] In formula (11):

[0245] d n,0 (t) represents the size of the initial data packet collected from the data source;

[0246] λ represents the penalty factor of the depreciation model, which is responsible for adjusting the speed of value reduction after exceeding the threshold;

[0247] χ represents the AoI threshold;

[0248] AoI(t) represents the information delay of each data packet at time t;

[0249] S3011: Data value is determined by the information delay of the data packet.

[0250] S3012: The edge device determines the subsequent route based on the value of the depreciated data.

[0251] S4: Heterogeneous age-sensitive optimization problem;

[0252] S401: In age-sensitive situations, the freshness of data should be highly valued. Drawing on the concept of age of information (AoI), in the system introduced above, the age of the data source at the tth time slot is defined as the difference between the current time and the generation time of the latest data at the receiving end, Δ n (t):

[0253]

[0254] In formula (12):

[0255] Indicates the generation time of the latest data packet of the data source received by the cloud center;

[0256] The difference Δ is different from the delay of each packet. n (t) is the duration metric of each data source, indicating the frequency with which the data from the data source is collected, executed at the edge, and offloaded. This is particularly important for such time-sensitive MEC systems because it directly affects the validity of the data and the real-time nature of decision making.

[0257] S402: Define the total number of data sources as N s , then a total data source number dimension vector can be used to record the age of each data source;

[0258] S403: In order to maintain data timeliness, the optimization goal of the system is to effectively allocate heterogeneous data resources of edge devices and minimize the average age of data resources collected by all edge devices. Therefore, taking the system model as a constraint, the NP-hard optimization problem is obtained:

[0259]

[0260]

[0261] C2:pos k (t+1)=pos k (t)+move k(t)

[0262] C3: move k (t) 2 ≤r k move

[0263]

[0264]

[0265]

[0266]

[0267]

[0268]

[0269]

[0270]

[0271]

[0272]

[0273] S404: Before discussing the optimization problem, the present invention re-examines the system model and problem from three dimensions. First, due to the significant hardware differences between edge devices and the fact that the optimization target is divided into two stages, edge and cloud, the constraints are heterogeneous and cannot be solved directly, requiring an iterative algorithm. Second, the optimization problem p 1 For instantaneous optimization, the environment state is random, the strategy changes dynamically, and the agent decision can be either competitive or cooperative. Finally, the optimization problem p 1 It is NP-hard and difficult to solve by traditional methods, so it is converted into MDP and the MADRL solution is explored.

[0274] MADRL can dynamically process random states in the system and optimize the behavioral decisions of each agent, thereby effectively coping with the complexity and real-time requirements of the system. This method provides a feasible approach to solving the resource allocation problem in age-sensitive MEC systems.

[0275] S5: MDP formulates external reward function and intrinsic reward function;

[0276] S501: HA-MAAC-Trans realizes the interaction and collaboration of heterogeneous multi-agents by constructing a Markov decision process (MDP) model. The MDP is represented by a quadruple as follows and extended to a multi-agent version to adapt to edge devices and cloud centers in edge computing (MEC) scenarios.

[0277] M{S,A,R,P}(13)

[0278] In formula (13)

[0279] S represents the state; the state of the edge agent includes local observations such as buffer status and allocated offloading bandwidth, while the state of the central agent integrates the states of all edge devices.

[0280] A indicates action;

[0281] R stands for reward;

[0282] P represents the transfer strategy;

[0283] S502: Edge agent represents the local state M of the quadruple K {S K ,A K ,R K ,P K};

[0284] S503: The central agent represents the global state M of the quadruple C {S C ,A C ,R C ,P C};

[0285] The HA-MAAC-Trans framework adopts a heterogeneous multi-agent Actor-Critic approach to model the state input and output of the edge and center to achieve collaborative learning. The construction and learning process of the framework are as follows Figure 2 shown.

[0286] S504: The edge agent is responsible for device movement, data collection, local execution and task offloading. The actions of the edge agent are composed of movement, execution decision and offloading scheduling:

[0287] {a k (t)}={[move k (t),exe k (t),off k (t)]} (14)

[0288] S505: Coordinate the offloading resource allocation of the edge device by allocating bandwidth ratio vectors, that is, the action a of the central agent c (t) is the allocated bandwidth ratio b(t);

[0289] S506: HA-MAAC-Trans introduces a composite reward mechanism including external and intrinsic rewards to optimize heterogeneous AoI and resource allocation. Synergy of external and intrinsic rewards.

[0290] S50601: Agent collaboration in HA-MAAC-Trans to minimize the average age of data sources The goal is to ensure maximum freshness of data;

[0291] S50602: Define the amount of data received by the cloud center as S(t), which is used to evaluate the data processing efficiency of the system at the current moment and motivate the system to process the backlog data quickly and effectively;

[0292] S50603: External rewards comprehensively consider the average age of the data source and the amount of data received by the cloud center, and use a weighted strategy to achieve a balance between data freshness and system performance. Described as a weighting of the amount of data and the average age of the data source:

[0293]

[0294] In formula (15):

[0295] α and β are weight coefficients. HA-MAAC-Trans can optimize the overall performance and resource utilization of the system while maintaining data freshness by designing weights.

[0296] S50604: Calculate intrinsic rewards

[0297]

[0298] In formula (16):

[0299] P μ represents a personalized classifier, which is used to perform personalized classification on the local information of each edge agent;

[0300] P μ (ko k ) represents the probability that the current observation range generated by the personalized classifier is suitable for the edge device;

[0301] ω LISIM represents the adjustment weight;

[0302] S50605: HA-MAAC-Trans defines a global reward function and combines it with a decaying long-term reward to study the global optimality of the system. The decaying long-term reward is as follows:

[0303]

[0304] In formula (17):

[0305] γ∈[0,1] represents reward / penalty decay;

[0306] This multi-agent reinforcement learning framework not only optimizes the collaboration efficiency between the edge and the center, but also improves the overall performance and stability of the system through a compound reward mechanism.

[0307] S6: Heterogeneous multi-agent Actor-Critic construction and function analysis;

[0308] S601: constructing a heterogeneous multi-agent Actor-Critic, including a central Actor network, a central Critic network, an edge Actor network, and an edge Critic network; the edge Actor network and the edge Critic network both contain a Transformer-enhanced neural network to improve the ability of spatial data feature enhancement (abstraction and modeling);

[0309] S602: Actor Network A k (s k (t);θ k ), taking the state of each agent as input and outputting the current action a k (t); Since the input state and output action of each agent are multi-dimensional, the Actor network needs to be designed according to the specific data structure, such as Figure 3 (a) and Figure 3 (b) as shown.

[0310] S603: For each agent, design a Critic network C based on the Actor network framework k (s k (t),a k (t); φ k ), combined with the value assessment module, input the current state and action estimation state-action value function Q k (s k (t),a k (t)); Figure 3 (c) and Figure 3 (d) as shown.

[0311] S604: Design edge actor network;

[0312] S60401: Build a multi-input-output neural network to integrate various edge observation information, learn and process diverse operations;

[0313] S6040101: The input of the multi-input-output neural network includes information such as local observations from the data source, edge buffer status, and offload channel status;

[0314] S6040102: Format the local observation data of the data source as A feature map of the image, extracting edge observation information based on the features of the feature map;

[0315] S60402: Build a CNN network to extract key spatial features of areas with large data packets and high AoI;

[0316] S60403: Convert feature maps into sequences and input them into the Transformer encoder, using the Transformer network's multi-head self-attention mechanism to capture long-distance dependencies;

[0317] S60404: Combine feature maps with CNN features to output device trajectories and action decisions;

[0318] S60405: Use MLP to process information such as execution buffer, completion buffer, and bandwidth allocation to extract edge status.

[0319] CNN extracts key feature maps through convolution and pooling layers. To enhance the adaptability of the model, the present invention introduces the Transformer network, uses its multi-head self-attention mechanism to capture global contextual information, and enhances feature expression through a feedforward neural network. Transformer effectively models long-distance dependencies, which is crucial for key information coordination and resource optimization in edge computing scenarios. The present invention serializes the CNN feature map and inputs it into the Transformer encoder (middle layer dimension 2, number of heads 2), adjusts it back to the image shape after processing, and splices it with the original feature map to generate a more representative feature map. Observation map projected onto On this basis, the edge state extraction is mainly realized through MLP (Multi-layer Perceptron). Input information such as execution buffer, completion buffer and allocated bandwidth are formatted as lists and scalar inputs, and the output is the execution and offloading scheduling decision of the edge device. Finally, the concatenated feature map is further processed by a fully connected layer, and the observation map is projected onto the movement map to determine the trajectory and action of the edge device, such as Figure 3 (a) shown.

[0320] S605: Design center Actor network;

[0321] The central Actor network takes the state information of the edge devices as input, including the execution and completion data resource buffers of each edge device and its location in the environment. Based on these inputs, the Actor network outputs a one-sum vector representing the bandwidth allocation ratio, which represents the bandwidth allocation ratio of each edge device. Using MLP combined with multi-device status, the communication scheduling between the center and the edge is optimized, and the bandwidth ratio is allocated for edge-to-center communication, such as Figure 3 (b)

[0322] S606: Design edge critic network;

[0323] The input of the edge critic network includes the state information s of the edge device k (t) (location information, buffer status, allocated offload bandwidth) and the action strategy of the edge device output by the edge actor network a k (t)(Move, Execute and Uninstall);

[0324] S60601: The edge observation information is preliminarily processed through the convolution layer (including convolution and average pooling) to extract local features;

[0325] S60602: Input local features into the Transformer network to capture the temporal characteristics of the observation information, thereby enhancing the understanding of the dynamic changes in the edge node environment;

[0326] S60603: Use multiple fully connected layers and ReLU activation functions to process the time series characteristics and mine the complex features of the state information layer by layer, so that the edge critic network can more accurately judge the scene in the current state and extract the edge state;

[0327] S60604: Different types of edge actions (movement actions, execution actions, and unloading actions) are processed separately through independent FC-ReLU branches to ensure effective extraction of different action features;

[0328] S60605: The processed action, state and observation information are connected and integrated in the network, and the estimate of the action value function is finally output To evaluate the value of each action in the current state, such as Figure 3 (c)

[0329] S607: Design Center Critic Network;

[0330] The central critic network inputs the edge device status information (execution / completion buffer, location of each edge device) and the central bandwidth control, and obtains the estimate of the value function through the fully connected Softmax layer

[0331] The central critic network is used to calculate and evaluate the state value of the entire system, and provide the central actor with value assessments of different actions, thereby helping the central actor network to prefer high-value actions and optimize decision-making results. By integrating the state information, position, and related actions of each edge node, the value of the entire system in its current state is uniformly evaluated. The central critic network is designed to be lightweight, using only a fully connected layer to process the location information of the central control, thereby achieving global estimation while ensuring low computational complexity. This design enables the central critic network to effectively integrate information from each edge node, provide accurate value feedback to the central actor network, and promote the coordination of the system at the global level. Through this global value assessment, the system can ensure the consistency of overall goals (such as minimizing the average information age), such as Figure 3 (d) as shown.

[0332] S7: Intrinsic rewards based on local information recognition.

[0333] S701: Due to the uneven distribution of data sources in the task area and each edge device can only observe its own local information, the efficiency of collaboration between agents is limited under the MADRL framework. The lack of local information makes it difficult for agents to fully understand the global state, which affects the optimization of global decisions and reduces the effectiveness of multi-agent collaboration. Therefore, considering that devices are constantly moving in the MEC environment, a reasonable division of labor in space is achieved, while avoiding the waste of resources caused by multiple devices arriving in the same area. In addition, considering that the task needs to deal with the generation of heterogeneous data and the management of the age of information (AoI), HA-MAAC-Trans introduces a local information self-identification module (LISIM), which explores the personalized characteristics of edge devices through self-supervised individual classifiers. Through sufficient individual exploration, the device can complete the data collection task more effectively, thereby improving the overall performance when facing heterogeneous data and information delay problems;

[0334] S702: Local information self-identification module as the intrinsic reward function of HA-MAAC-Trans The goal is to train a global probability classifier p with μ as a parameter μ , let the input be the observation o k , the output is the probability p of the observation belonging to each edge device μ (·|o k ), so as to accurately obtain the given different observations o k An edge device is identified in the model to model the independent actions of each edge device; the intrinsic reward p is defined μ (k|o k) represents the possibility of accurately predicting / identifying a certain edge device from the observation. It can be seen that the total reward of the edge device is expressed as a combination of external reward and intrinsic reward:

[0335]

[0336] In formula (18):

[0337] ∑ k p μ (k|o k )=1;

[0338] ω LISIM represents the adjustment weight, which is used to measure the importance between intrinsic rewards and extrinsic rewards;

[0339] represents extrinsic rewards from the environment;

[0340] Intrinsic rewards help edge devices achieve individual recognition through partial environmental observation, promote division of labor, enhance strategy differences, and improve classifier training. LISIM discovers new data sources and trajectory patterns in data collection, increases cumulative external rewards, strengthens exploration enthusiasm, optimizes agent task performance, and balances exploration and task completion through personalized recognition capabilities to improve system efficiency.

[0341] Example 1

[0342] The present invention implements the HA-MAAC-Trans learning framework, and constructs a CNN-MLPs hybrid network of edge Transformer-enhanced Actor networks and MLP-based networks as the central network through TensorFlow. In order to compare the performance of the proposed framework with other RL methods, the present invention selects two Actor-Critic based algorithms, DDPG (centralized) and the popular MADDPG (multi-agent), in which dual neural networks are also used to learn hybrid strategies. In addition, algorithms similar to joint collaboration through local information sharing, such as EdgeFed H-MAAC, are also used as baselines. For fairness, the present invention sets the same random seed for the MEC environment for all methods, and also fixes the random seed for the training process. Therefore, the results are general and easy to reproduce.

[0343] Fig. 9 and Table 3 show the comparison results of the proposed algorithm and the baseline in an environment with 4 edge devices and 30 data sources on a 200×200 map with randomly initialized edge device and data source locations.

[0344] Table 3 AoI metric data comparison

[0345]

[0346] Average age during online interactions like Fig. 9 As shown in (a), it can be found that mixed DDPG leads to The highest, until 5K epochs, the curve stabilizes. However, under hybrid MADDPG, EdgeFed H-MAAC, and HA-MAAC-Trans, the average age remains at a lower value after sufficient iterations. Specifically, The statistics are listed in Table 3. HA-MAAC-Trans not only obtains the lowest average age, but also obtains the lowest variance, which means that the HA-MAAC-Trans method is better than DDPG and MADDPG in terms of system penalty and learning stability. In addition, it is better than the ordinary EdgeFed H-MAAC method in terms of convergence speed. It reflects the superiority of the intrinsic incentive mechanism of Transformer and LISIM in terms of MEC system latency. The right part of Table 3 is a comparison of peak AoI. Obviously, the HA-MAAC-Trans algorithm proposed in this paper obtains the lowest peak PAoI and the lowest average This means that HA-MAAC-Trans can greatly reduce the peak value of AoI through the AoI threshold and improve the efficiency of data processing in MEC. The minimum variance of the peak value also shows that the updates of all data sources are frequent and fair. Fig. 9 (b) shows the worst AoI (the AoI of the sample point with the largest AoI mean) among the four methods. HA-MAAC-Trans also performs best. It can be found that under centralized DDPG, some sources are ignored for a long time. The explanation for this is that the centralized collaborative algorithm requires a larger neural network model with a complex structure to extract the relationship between too many global input states and the local strategy of each individual agent, which also leads to the difficulty of training.

[0347] also, Fig. 9 (c) and Fig. 9 (d) shows the number and size of aggregated data packets received at the cloud center (i.e., the last hop of the MEC system). As can be seen from the figure, the HA-MAAC-Trans collaborative algorithm improves the data utility of the MEC system by completing more data processing in the same amount of time.

[0348] Example 2

[0349] In S5, the present invention introduces a compound reward mechanism to simultaneously optimize multiple key performance indicators of the system, including the number of tasks completed by the system and the average age of the total samples.

[0350] In this experiment, we firstly change the proportion of different performance indicators in the reward mechanism based on different configurations of α and β weight coefficients, and experimentally evaluate the average age of the system at the 5000th time slot, such as Figure 7 (a) shown.

[0351] From the box plot data, the experimental group with α=0.1, β=0.9 and α=0.2, β=0.8 has the lowest median of 11.5, which shows that the system can maintain a low average age, but the outliers are high, indicating that there may be a situation where information is updated slowly, affecting performance stability. The median of the other group with α=0.5, β=0.5 is similar, only 0.5 higher, but the interquartile range is the lowest, with the 75% quantile being 22 and the 25% quantile being 6, showing that the data is highly concentrated, the performance fluctuation is small, and it can maintain a stable low average age level in various situations, and the outliers are low, even in extreme cases, it can avoid a significant drop in performance. Therefore, this group of configurations performs best in terms of average age performance.

[0352] Figure 7 (b) shows the cumulative data volume received by the cloud center during the time slots from 4995 to 5000 under different weight coefficients α and β configurations. The cumulative data volume received is a key indicator for measuring the data transmission performance of the system. A higher received packet count means that the system can transmit data more efficiently and maintain better throughput. By analyzing the curves under different configurations, the optimal configuration under the experimental conditions is found.

[0353] The figure shows that the received packet counts of all weight combinations increase linearly as the time slot progresses. The two curves of α=1, β=0 and α=0, β=1 are at the highest and lowest positions respectively, proving that α and β can affect the change of rewards. α=1, β=0 indicates that the reward only focuses on the number of completed tasks, so the amount of data received by the cloud center is obviously the largest. While α=0, β=1 only focuses on the average age of the samples, and the amount of data received by the cloud center is the smallest, which is also expected. It is worth noting that the suboptimal configuration groups are α=0.6, β=0.4, α=0.3, β=0.7 and α=0.5, β=0.5, which are second only to α=1, β=0, where the reward mechanism is completely based on the number of completed tasks, and the amount of data received by the cloud center is at a higher level. The cloud center can receive data packets at a higher rate, and the system has the highest data transmission efficiency and throughput.

[0354] From the above two experimental results, it can be seen that the most noteworthy set of parameter weights is α=0.5, β=0.5. This set of weights not only performs best in terms of average age, but also achieves a level second only to α=1, β=0 in terms of the performance indicator of the amount of data received by the cloud center. It can be seen that by configuring the parameter weights to α=0.5, β=0.5, the MEC system not only focuses on the number of completed tasks to achieve high-speed data reception. It can also pay equal attention to the average age of the samples, ensuring that the system can respond quickly to data changes and avoid the accumulation of information delays.

[0355] In summary, the experimental results show that the parameter weights of α=0.5, β=0.5 can achieve the optimal optimization of system performance. Compared with the configurations of α=1, β=0 and α=0, β=1 which only focus on a single performance indicator, the comprehensive consideration of the weight distribution of α and β can more effectively improve the overall performance of the system.

[0356] Example 3

[0357] MEC collaboration algorithms such as DDPG and MADDPG often lead to similar access areas because edge devices are started from the same location, resulting in resource waste and insufficient access to data sources. These algorithms do not consider device differences, have similar behaviors, and cannot effectively collaborate to access more data sources. LISIM processes individual differences, avoids centralized access, and encourages devices to explore diverse areas, thereby reducing AoI and increasing data collection. This paper selects several common collaboration algorithms and verifies the superiority of the LISIM algorithm by comparing the movement trajectories of edge devices on the data source map under different algorithms. Figure 8 shown.

[0358] According to the experimental results, sharing the same neural network parameters or using a centralized network is not a good choice (DDPG, MADDPG). The paths obtained by the two algorithms have too many round trips and are limited to obtaining rewards only from neighboring data sources. The H-MAAC algorithm can alleviate the path duplication problem to a certain extent by introducing the ε-exploration method. Edge devices can actively explore data sources that have not been visited and avoid the similarity of paths between devices by learning the parameters of other devices. However, since H-MAAC does not consider the internal and external characteristics of rewards between devices, there is still a problem of path duplication. In contrast, the HA-MAAC-Trans algorithm proposed in the present invention introduces the LISIM mechanism, which effectively distinguishes the influence of physically adjacent homogeneous neighbors on the task path and avoids the problem of path duplication. Figure 8As shown in the figure, the task coverage of the path using LISIM is close to 100%, and the trajectory of each agent is circular, which successfully solves the problem of spatial division of labor. However, the path without LISIM (such as H-MAAC) has some unexplored areas. The reason is that the data source is unevenly distributed, the observable range of each edge device is limited, and spatial division of labor must be achieved through continuous movement.

[0359] LISIM introduces personality into edge devices, providing intrinsic motivation to make them aware of their responsibilities to move to remote areas, improve mission data source coverage, and ultimately increase efficiency.

[0360] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0361] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-agent collaboration framework for MEC systems that integrates AoI and intrinsic motivation, characterized by: The method comprises the following steps: S1: generation of heterogeneous resources on the edge and edge mobility decision-making; S2: Communication link modeling; S3: Heterogeneous resource acquisition and AoI threshold limit; S4: Heterogeneous age-sensitive optimization problem; S5: MDP formulates external reward function and intrinsic reward function; S6: Heterogeneous multi-agent Actor-Critic construction and function analysis; S7: Intrinsic rewards based on local information recognition.

2. According to claim 1, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S1 comprises the following steps: S101: generating a data source for a device on the terminal side; S102: Edge device E k Collect data packets from data sources on the move and offload the data to the cloud data center after processing it locally; S103: Establishing mobility model of edge devices: pos k (t+1)=pos k (t)+move k (t) (1) In formula (1): POS k (t) = [x k (t),y k (t),h k (t)] represents the edge device E k At the position of the tth time slot, x represents the horizontal distance difference, y represents the vertical distance difference, and h represents the height difference; |move k (t)2≤r k move represents the movement in each time slot; r k move represents the edge device E k Moving radius; |·2 represents the 2-norm of the vector; S104: Define movement, data collection, and local execution as core decision-making content in a multi-agent deep reinforcement learning framework.

3. According to claim 2, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S101 includes the following steps: S10101: The terminal device collects computing task data generated by intelligent devices of distributed sensors in the Internet of Things to generate a data source; S10102: Due to differences in geographical environment and hardware configuration, data sources form heterogeneous resource data and generate data packets in different forms; S10103: For each data source S n Assign weights W respectively Sn , to indicate the importance of different locations and assuming that the data sources are independently generated; S10104: Define the data size of the data packet as d(t), the data generation time of the data packet as w(t) and the data source index as idx(t).

4. According to claim 3, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S2 comprises the following steps: S201: It is clarified that the transmission links between agents in the three-layer system environment include source-edge, edge-edge, and edge-cloud; S202: Since edge devices only share states, observations, and learning parameters, and do not involve the transmission of data and tasks, the transmission cost of edge-to-edge communication is ignored; S203: Modeling the source-edge and edge-cloud transmission processes; S204: Considering the frequency division mode with a total bandwidth of W, the transmission rate between the edge device and the data source or between the edge device and the cloud center C is calculated: In formula (4): b k,n (t) represents the allocated bandwidth ratio; N0 represents the noise power spectral density; Indicates the power required for transmission.

5. According to claim 4, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S203 comprises the following steps: S20301: Considering line-of-sight and non-line-of-sight path losses, an air-ground channel is introduced; In formula (2): f represents the carrier frequency; c represents the speed of light; It is expressed as the distance between the edge device and the ground entity; η ξ represents the path loss in line-of-sight and non-line-of-sight situations, ξ={0,1}; S20302: The average air-ground path loss of the communication channel between the edge device and the data source or between the edge device and the cloud center C in the tth time slot is obtained as: In formula (3): p1(t) represents the probability of non-line-of-sight; represents the probability of line of sight; where: a and b both represent parameters related to the environment; Represents the angle between the edge link and the horizontal plane.

6. According to claim 5, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S3 comprises the following steps: S301: When the edge device collects heterogeneous resource data, the edge device runs to the vicinity of the data source to collect all data packets in the data source buffer and occupies the edge device data collection buffer A data block in S302: Using the local processor to process the collected data buffer Perform pre-processing; S303: Establish a time model for edge processing performed by edge devices in time slot t for data packets collected from data sources: In formula (5): It represents the data rate at which the edge device preprocesses the preset task in time slot t, which is related to the CPU cycle frequency; S304: Assuming that in each time slot, the edge device allocates its edge computing resources to a data block in the buffer, the edge execution decision of the t-th time slot is represented by a one-hot vector as follows: In formula (6): exe i (t)∈{0,1} represents the CPU allocation flag of each data block in the collection data buffer of the edge device; S305: Edge device locally executes decision: S306: Cache the data in the executed data buffer and wait for unloading to the cloud center; S307: Assuming that in each time slot, the edge device decides to unload a data packet in the data buffer, the unloading schedule is represented by a one-hot vector as follows: In formula (8): off i (t)∈{0,1} represents the offloading decision of each data block; S308: Local offloading scheduling of edge devices: S309: To quantify heterogeneity, HA-MAAC-Trans introduces weights for data sources to highlight the importance of different locations and corrects the packet size by weights: In formula (10): d(t) represents the data size of the data packet; Indicates the weight of the data source; S3010: Establish a data value depreciation model: In formula (11): d n,0 (t) represents the size of the initial data packet collected from the data source; λ represents the penalty factor of the depreciation model, which is responsible for adjusting the speed of value reduction after exceeding the threshold; χ represents the AoI threshold; AoI(t) represents the information delay of each data packet at time t; S3011: Determine data value through information delay of data packets; S3012: The edge device determines the subsequent route based on the value of the depreciated data.

7. According to claim 6, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S4 comprises the following steps: S401: Define the age of the data source in the tth time slot as the difference Δ between the current time and the generation time of the latest data at the receiving end n (t): In formula (12): Indicates the generation time of the latest data packet of the data source received by the cloud center; S402: Define the total number of data sources as N s , then a total data source number dimension vector can be used to record the age of each data source; S403: Taking the system model as a constraint, we get the NP-hard optimization problem: C2:pos k (t+1)=pos k (t)+move k (t) C3:move k (t)2≤r k move S404: Discuss solutions based on MADRL.

8. According to claim 7, a MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation is characterized by: The S5 comprises the following steps: S501: MDP is represented by a quaternion as follows: M{S,A,R,P}(13) In formula (13) S stands for status; A indicates action; R stands for reward; P represents the transfer strategy; S502: Edge agent represents the local state M of the quadruple K {S K ,A K ,R K ,P K }; S503: The central agent represents the global state M of the quadruple C {S C ,A C ,R C ,P C }; S504: The actions of the edge agent consist of movement, execution decision and unloading scheduling: {a k (t)}={[move k (t),exe k (t),off k (t)]} (14) S505: Action a of the central agent c (t) is the allocated bandwidth ratio b(t); S506: The synergy between external and intrinsic rewards; S50601: Agent collaboration in HA-MAAC-Trans aims to minimize the average age Δ(t) of data sources; S50602: Define the amount of data received by the cloud center as S(t), which is used to evaluate the data processing efficiency of the system at the current moment and motivate the system to process the backlog data; S50603: The external reward of each agent in the tth time slot Described as a weighting of the amount of data and the average age of the data source: In formula (15): α, β are weight coefficients; S50604: Calculate intrinsic rewards In formula (16): P μ represents a personalized classifier, which is used to perform personalized classification on the local information of each edge agent; P μ (ko k ) represents the probability that the current observation range generated by the personalized classifier is suitable for the edge device; ω LISIM represents the adjustment weight; S50605: HA-MAAC-Trans defines a global reward function and combines it with a decaying long-term reward to study the global optimality of the system. The decaying long-term reward is as follows: In formula (17): γ∈[0,1] represents reward / penalty decay.

9. The MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation according to claim 8, characterized in that: The S6 comprises the following steps: S601: constructing a heterogeneous multi-agent Actor-Critic, including a central Actor network, a central Critic network, an edge Actor network, and an edge Critic network; the edge Actor network and the edge Critic network both contain a Transformer-enhanced neural network to improve the ability of spatial data feature enhancement; S602: Actor Network A k (s k (t);θ k ), taking the state of each agent as input and outputting the current action a k (t); S603: For each agent, design a Critic network C based on the Actor network framework k (s k (t),a k (t); φ k ), combined with the value assessment module, input the current state and action estimation state-action value function Q k (s k (t),a k (t)); S604: Design edge actor network; S60401: Build a multi-input-output neural network to integrate various edge observation information, learn and process diverse operations; S6040101: The inputs to the multi-input-output neural network include local observations from data sources, edge buffer states, and offload channel states; S6040102: Format the local observation data of the data source as A feature map of the image, extracting edge observation information based on the features of the feature map; S60402: Build a CNN network to extract key spatial features of areas with large data packets and high AoI; S60403: Convert feature maps into sequences and input them into Transformer encoders, using the multi-head self-attention mechanism of the Transformer network to capture long-distance dependencies; S60404: Combine feature maps with CNN features to output device trajectories and action decisions; S60405: extract edge status using MLP processing execution buffer, completion buffer and bandwidth allocation; S605: Design center Actor network; The central Actor network takes the status information of the edge device as input, and the Actor network outputs a one-sum vector representing the bandwidth allocation ratio. It uses MLP to combine the status of multiple devices to optimize the communication scheduling between the center and the edge and allocate bandwidth ratios for edge-center communication. S606: Design edge critic network; The input of the edge critic network includes the state information s of the edge device k (t) and the action strategy a of the edge device output by the edge Actor network k (t); S60601: The edge observation information is preliminarily processed through the convolution layer to extract local features; S60602: Input local features into the Transformer network to capture the temporal characteristics of the observation information, thereby enhancing the understanding of the dynamic changes in the edge node environment; S60603: Use multiple fully connected layers and ReLU activation functions to process the time series characteristics and mine the complex features of the state information layer by layer, so that the edge critic network can more accurately judge the scene in the current state and extract the edge state; S60604: Different types of edge actions are processed separately through independent FC-ReLU branches to ensure effective extraction of different action features; S60605: The processed action, state and observation information are connected and integrated in the network, and the final output is the estimate of the action value function To evaluate the value of each action in the current state; S607: Design Center Critic Network; The central critic network inputs edge device status information and central bandwidth control, and estimates the value function through the fully connected Softmax layer.

10. The MEC system multi-agent collaboration framework integrating AoI and intrinsic motivation according to claim 9, characterized in that: The S7 comprises the following steps: S701: HA-MAAC-Trans introduces a local information self-identification module to explore the personalized characteristics of edge devices through self-supervised individual classifiers; S702: Local information self-identification module as the intrinsic reward function of HA-MAAC-Trans The goal is to train a global probability classifier p with μ as a parameter μ , let the input be the observation o k , the output is the probability p of the observation belonging to each edge device μ (·o k ), so that from the given different observations o k An edge device is identified in the model to model the independent actions of each edge device; the intrinsic reward p is defined μ (ko k ) represents the possibility of accurately predicting / identifying a certain edge device from the observation. It can be seen that the total reward of the edge device is expressed as a combination of external reward and intrinsic reward: In formula (18): ∑ k p μ (is k )=1; ω LISIM represents the adjustment weight, which is used to measure the importance between intrinsic rewards and extrinsic rewards; Represents extrinsic rewards from the environment.

Images