Multi-agent reinforcement learning method

Abstract

The invention discloses a multi-agent reinforcement learning method. The method comprises the following steps: step 1, system initialization and edge device modeling; 2, generating a data source and constructing a task package; step 3, edge equipment trajectory planning and movement acquisition; 4, local task execution and unloading strategy decision making; 5, communication link modeling and bandwidth resource allocation are carried out; step 6, scheduling optimization driven by an information age and depreciation mechanism; step 7, multi-agent strategy optimization based on Actor-Critic is carried out; and step 8, reward function design and reinforcement learning process. The technical problems that an existing centralized scheduling or heuristic algorithm cannot perform efficient learning and scheduling in a resource heterogeneous and data dynamic environment, and cannot still have good learning ability and generalization ability under the condition of lack of global information are solved; the method is suitable for system modeling and optimization work of efficient strategy learning and collaborative decision making of multiple agents in a complete cooperation task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cloud computing and edge computing, and relates to a multi-agent reinforcement learning method, in particular to a multi-agent cooperation method for a mobile edge computing (MEC) system fusing AoI and intrinsic motivation. BACKGROUND

[0002] With the development of Internet of Things (IoT), intelligent transportation, smart city and other application scenarios, mobile edge computing (MEC) has become one of the key technologies to support low-latency and high-bandwidth services. MEC can effectively alleviate network congestion and improve real-time performance by sinking computing resources from remote clouds to network edges, so that tasks can be processed nearby. Therefore, MEC is particularly suitable for scenarios sensitive to time delay, such as video monitoring, autonomous driving and emergency response.

[0003] To meet the growing demand for edge computing, researchers are constantly exploring more efficient scheduling and decision-making methods. In traditional methods, computing tasks are usually allocated and scheduled through centralized scheduling or heuristic algorithms, but such methods have the following shortcomings: (1) Difficulty in dealing with device heterogeneity and dynamic data changes: The computing power, energy consumption constraints and communication states of edge nodes in the MEC system differ significantly, and the data sources have spatial distribution and time sensitivity. Traditional static or semi-static strategies are difficult to adapt to. (2) Lack of intelligent decision-making ability: Centralized control faces communication bottlenecks and single-point failure problems, and heuristic methods generally rely on fixed rules, which are difficult to generalize to complex and dynamic real-world scenarios.

[0004] In recent years, Multi-Agent Reinforcement Learning (MARL) methods have been widely introduced into the edge computing field, as they can autonomously learn collaborative strategies in dynamic environments and have gradually become a research hotspot. For example, MADDPG (Multi-Agent Deep Deterministic Policy Gradient) and other methods model the strategies of multiple agents jointly, which are suitable for continuous action space tasks. QMIX, VDN, and other algorithms based on value function decomposition are used for multi-agent collaborative control problems and have achieved good results in tasks such as StarCraft and robot path planning. OpenAI Five and AlphaStar projects have successfully demonstrated the application potential of MARL in high-dimensional and non-stationary environments. Although the above methods perform well in simulation environments, they still face the following challenges when directly applied to MEC systems: (1) Strong dependence on global information, limiting distributed execution capabilities: Most MARL algorithms assume that agents can access global states, but in edge computing scenarios, agents can only access local observations; (2) Difficulty in handling data timeliness and value decay: Edge computing tasks have strong timeliness, and existing MARL methods generally ignore the modeling of data "freshness"; (3) Lack of modeling ability for agent heterogeneity: Edge nodes exhibit different behavior characteristics due to location and capability differences, and existing methods are difficult to support personalized decision-making; (4) Low collaboration efficiency under resource constraints: Under bandwidth, computing power, and other constraints, how to achieve efficient collaboration and task allocation remains a challenge.

[0005] Therefore, there is an urgent need for a new multi-agent reinforcement learning method for edge computing systems that can efficiently learn and schedule in a resource-heterogeneous and data-dynamic environment, taking into account the age of information (AoI) and individual characteristics of edge devices, while still having good learning ability and generalization ability in the absence of global information, improving the overall data processing efficiency and collaboration level between agents. SUMMARY

[0006] To solve the technical problems that existing centralized scheduling or heuristic algorithms cannot efficiently learn and schedule in a resource-heterogeneous and data-dynamic environment, and cannot still have good learning ability and generalization ability in the absence of global information, the present application provides a multi-agent reinforcement learning method suitable for efficient learning and collaborative decision-making scenarios of multiple agents in fully cooperative tasks.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] A multi-agent reinforcement learning method, comprising the following steps:

[0009] Step 1, system initialization and edge device modeling:

[0010] Constructing a multi-agent environment in the edge computing system, initializing the state information of multiple edge nodes, including location information, cache capacity, computing resources and communication conditions;

[0011] Step two, data source generation and task package construction:

[0012] Receiving heterogeneous task data from multiple data sources, identifying the data volume, generation time, data type and priority weight of each task;

[0013] Step three, edge device trajectory planning and mobile collection:

[0014] According to the current location of the edge node, data source distribution and task value, generate a mobile path and collect target task data, and store the collected data in the local task buffer;

[0015] Step four, local task execution and unloading strategy decision:

[0016] Based on the local computing resource state and the urgency of the task, select part of the task from the buffer for local processing, and schedule the remaining tasks to the upload queue;

[0017] Step five, communication link modeling and bandwidth resource allocation:

[0018] Combined with the link state and global bandwidth allocation information, select the appropriate time window to send the task data to be uploaded to the cloud center;

[0019] Step six, scheduling optimization driven by information age and depreciation mechanism:

[0020] Calculate the information age of all tasks, and adjust the processing priority of the task in the local queue according to the task timeout condition;

[0021] Step seven, multi-agent strategy optimization based on Actor-Critic:

[0022] Use local state information to input into the policy network, output scheduling actions including movement, processing and uploading, to control the next step of the edge node behavior;

[0023] Step eight, reward function design and reinforcement learning process:

[0024] Receive environmental feedback, including task processing success and data timeliness indicators, generate external rewards, and combine internal incentives generated by identifying behavior differences to update the policy network.

[0025] The multi-agent reinforcement learning device for implementing the above method comprises a system initialization and edge device modeling module, a data source generation and task package construction module, an edge device trajectory planning and mobile collection module, a local task execution and unloading strategy decision module, a communication link modeling and bandwidth resource allocation module, an information age and devaluation mechanism driven scheduling optimization module, an Actor-Critic based multi-agent strategy optimization module, a reward integration and strategy updating module of global scheduling target, wherein:

[0026] The system initialization and edge device modeling module is responsible for constructing the multi-agent environment in the edge computing system and initializing the state information of the plurality of edge nodes.

[0027] The data source generation and task package construction module is responsible for receiving heterogeneous task data from a plurality of data sources, identifying the data volume, generation time, data type and priority weight of each task.

[0028] The edge device trajectory planning and mobile collection module is responsible for generating a mobile path and collecting target task data according to the current position of the edge node, the data source distribution and the task value, and storing the collected data in the local task buffer module.

[0029] The local task execution and unloading strategy decision module is responsible for selecting part of the tasks in the buffer for local processing based on the local computing resource state and the urgency of the task, and scheduling the remaining tasks to a to-be-uploaded queue.

[0030] The communication link modeling and bandwidth resource allocation module is responsible for selecting a suitable time window to send the to-be-uploaded task data to the cloud center in combination with the link state and global bandwidth allocation information.

[0031] The information age and devaluation mechanism driven scheduling optimization module is responsible for calculating the information age of all tasks and adjusting the processing priority of the tasks in the local queue according to the task timeout condition.

[0032] The Actor-Critic based multi-agent strategy optimization module is responsible for inputting the local state information into the strategy network to output scheduling actions including movement, processing and uploading, which are used to control the next behavior of the edge node.

[0033] The reward integration and strategy updating module of the global scheduling target is responsible for receiving environment feedback to generate external rewards, and updating the strategy network in combination with the internal incentives generated by identifying the behavior difference.

[0034] A computer storage medium for storing a computer program, when the computer program is read by a computer, the computer executes the above multi-agent reinforcement learning method.

[0035] A computer comprises a processor and a storage medium, and the computer executes the multi-agent reinforcement learning method when the processor reads a computer program stored in the storage medium.

[0036] A computer program product, as a computer program, implements the multi-agent reinforcement learning method when the computer program is executed.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] 1、The present application introduces information age (AoI) modeling and devaluation mechanism, the core definition of AoI is the time difference from the data generation time to the current time, which directly quantifies the "freshness" of information, and realizes the accurate control of data real-time. Compared with the traditional scheduling strategy which only optimizes the delay or queue length, the AoI modeling can directly reflect the "freshness" of data in the transmission and processing process, and punish the value of overdue data through the devaluation model, so as to guide the edge device to preferentially process high-value data, effectively reduce redundant transmission and improve the timeliness of the system.

[0039] 2、The present application adopts edge device mobility modeling and location-driven collection strategy, combines task importance and data generation distribution, optimizes the mobile path and scheduling efficiency of edge node. Compared with static or random moving strategy, this way can dynamically guide the edge device to approach the high-value task source, improve the task processing density of unit resource, and significantly reduce the task loss rate of uncollected area.

[0040] 3、The present application designs a reward function mechanism that combines external and internal incentives in the multi-agent reinforcement learning architecture, the external reward provides global guidance through the system data volume and AoI minimization target, and the internal incentive enhances the strategy learning through the individualized identification ability of edge agent. Compared with the traditional MARL method which only uses external reward, this mechanism enhances the model's ability to identify individual differences, improves the learning efficiency and strategy stability in non-homogeneous environment.

[0041] 4、The present application adopts the Transformer-based Actor-Critic network design in the model structure, combining CNN for spatial feature extraction and Transformer for capturing temporal dependence, which improves the understanding ability of agent to complex environment state. Compared with the traditional Actor-Critic structure using MLP or RNN, this method has stronger expression ability and generalization performance, especially in the scene of multiple data sources and large task scale, it shows better convergence speed and strategy accuracy.

[0042] 5、The local information self-identification module (LISIM) effectively alleviates the dependence of multi-agent reinforcement learning on global information. By training a classifier to identify the local observation characteristics of each edge device, the module enables the agent to have self-recognition and differentiated learning ability, supporting distributed execution. Compared with existing methods that rely on centralized control or share global state, the LISIM mechanism improves the scalability and deployment flexibility of the method in actual deployment scenarios.

[0043] 6、The task scheduling problem is modeled as a Markov decision process and combined with resource constraints to construct an NP-hard optimization problem, which is approximately solved by reinforcement learning strategy. Compared with the dimension disaster faced by traditional heuristic algorithms on large-scale problems, the method can adaptively trade off communication overhead, data value and task time limit in policy learning, thereby obtaining better global scheduling performance.

[0044] 7、The present application is suitable for multi-agent in the complete cooperation task carries on the efficient strategy learning and the collaborative decision-making system modeling and optimization work. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The method framework is shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions of the present application will be further described below in conjunction with the drawings, but are not limited thereto. Any modification or equivalent replacement of the technical solutions of the present application without departing from the spirit and scope of the present application shall be covered in the protection scope of the present application.

[0047] Specific embodiment one: the present embodiment provides a multi-agent reinforcement learning method, which comprises the following steps:

[0048] Step one, system structure design and edge device modeling:

[0049] A multi-agent environment in an edge computing system is constructed, and the state information of multiple edge nodes is initialized, including position information, cache capacity, computing resources and communication conditions. The specific steps are as follows:

[0050] Step one, constructing a three-layer MEC system, including a bottom layer of distributed data sources, a middle layer of edge devices and a top layer of cloud centers. The edge device has the ability of perception, calculation, caching and communication, and can independently plan trajectory and make task decision in the form of agent.

[0051] Step two, initializing the spatial position information, maximum moving range, cache capacity, computing capability and available bandwidth resources of each edge device.

[0052] Step one three, the data source is assigned a weight according to the physical location, the type of perception task and the emergency level, which is used as a reference for scheduling sorting. The edge device will make a mobile path and task priority based on this.

[0053] Step two, the data source generates a task package:

[0054] Receive heterogeneous task data from multiple data sources, identify the data volume, generation time, data type and priority weight of each task, wherein: the priority weight of the task is dynamically calculated based on the category, geographical location and task generation frequency of the data source to which the task belongs. The specific steps are as follows:

[0055] Step two one, in the Internet of Things environment, the data source periodically collects environmental information, generates a data package with timestamp, data size and source index. Due to different geographical locations and device types, data shows heterogeneity in spatial structure and timeliness.

[0056] Step two two, the MEC system encapsulates the data generated in step two one into a task package for edge device collection.

[0057] Step two three, to depict the trend of data value change, the information age (AoI) index is introduced, and a data devaluation model is constructed combined with the data source weight to clarify the value change of the task package at different times.

[0058] Step two four, each data package is assigned to a collection queue and waits for a nearby edge device to respond.

[0059] Step three, edge device trajectory planning and mobile collection:

[0060] According to the current location of the edge node, the distribution of data sources and the value of the task, a mobile path is generated and the target task data is collected, and the collected data is stored in the local task buffer, wherein: the mobile path of the edge node is optimized according to the target function constructed by the task density distribution heat map and the unit movement cost. The specific steps are as follows:

[0061] Step three one, the edge device updates the trajectory based on the current location and the distribution of nearby data sources. The movement is limited by the preset maximum movement radius, and needs to be combined with energy consumption, path planning and dynamic weight of data value for strategy optimization.

[0062] Step three two, when the edge device enters the effective collection range, it automatically obtains the task package from the connected data source and transfers it to the local collection buffer.

[0063] Step three, the edge device continuously updates the perception data during the journey, evaluates the priority between new data sources and existing tasks, and dynamically adjusts the path direction to improve collection efficiency and coverage.

[0064] Step four, local task execution and offloading strategy decision:

[0065] Based on the local computing resource status and the urgency of the task, select part of the tasks from the buffer for local processing, and schedule the remaining tasks to the uploading queue, wherein: the local processing decision is based on the current processing capacity of the edge node and the remaining tolerance delay of the task, when the processing resource is insufficient or the task is of high urgency, it is preferentially offloaded to the cloud center. The specific steps are as follows:

[0066] Step four one, the edge device judges whether each to-be-processed task meets the local processing condition.

[0067] Step four two, if the current processing unit is idle or the task AoI is low, it is allocated to the execution buffer for local processing. The local execution process is constrained by the edge CPU frequency and cache space, and a task blocking processing strategy is adopted to improve processing efficiency. After processing is completed, the task result is transferred to the completed queue and uploaded to the cloud center.

[0068] Step four three, if the local resource is insufficient or the task delay tolerance is low, the task will be scheduled to the offloading buffer. The output of this stage is the current task processing state of the device and the to-be-communicated data packet.

[0069] Step five, communication link modeling and bandwidth resource allocation:

[0070] Combine link state and global bandwidth allocation information to select the appropriate time window to send the to-be-uploaded task data to the cloud center, wherein: the link state includes signal strength, path loss and bandwidth occupation, and the bandwidth allocation is dynamically adjusted by the center node based on the task congestion degree of the device. The specific steps are as follows:

[0071] Step five one, the system establishes the communication channel model between the edge device and the data source, and between the edge device and the cloud center, and calculates the link loss and reachable rate. In actual deployment, consider the non-line-of-sight and line-of-sight path loss model, and introduce environmental influence factor to adjust the channel transmission capacity.

[0072] Step five two, after the center agent obtains the state and communication demand of all edge devices, it generates a global bandwidth proportion vector to allocate the proportion of available communication channels for each edge device.

[0073] Step five three, the device decides whether to upload according to the allocation proportion, task packet size and channel state.

[0074] Step six, scheduling optimization driven by information age and depreciation mechanism:

[0075] Calculate the information age of all tasks, and adjust the processing priority of tasks in the local queue according to the task timeout, wherein: the calculation of information age is based on the difference between the current system clock and the task generation time, and the priority adjustment adopts a preset task devaluation curve for mapping. The specific steps are as follows:

[0076] Step six, the system continues to track the time span of each data packet from generation to the present, and calculates its information age in real time.

[0077] Step six, according to the set AoI threshold and devaluation factor, calculate the current remaining value of the data packet.

[0078] Step six, the edge device uses the devaluation function result to judge whether the task has the significance of continuing processing, and preferentially schedules high-value data for processing and unloading.

[0079] Step six, when the edge device sorts the task packet queue, it comprehensively considers the data packet size, AoI, remaining value and location distance, and dynamically selects the scheduling target among multiple candidate tasks.

[0080] Step seven, Actor-Critic-based multi-agent strategy optimization:

[0081] Local state information is input into the policy network, and the output includes scheduling actions such as moving, processing and uploading, which are used to control the next behavior of the edge node. The specific steps are as follows:

[0082] Construct an Actor-Critic-based multi-agent policy learning structure. The input of the edge device's Actor network includes local state (position, cache state, task characteristics) and observed task packet information, and the output includes three parts of decision-making including moving strategy, local execution strategy and unloading strategy. The edge Critic network combines the input state and the output action of the edge Actor to estimate the action value function, which is used to train the Actor to update the behavior strategy. The central Actor takes the state of all edge devices as input, and outputs the bandwidth allocation strategy; the central Critic is used to estimate the global action value. All networks use feature extraction modules containing convolution and Transformer structure to enhance the expression ability of spatial distribution and time sequence change.

[0083] Step eight, reward function design and reinforcement learning process:

[0084] Receive environmental feedback, including task processing success and data timeliness indicators, generate external rewards, and update policy networks combined with internal incentives generated by identifying behavior differences. The specific steps are as follows:

[0085] Step eight, internal incentive mechanism and individual identification fusion:

[0086] Step eight one, in order to improve the cooperation ability between heterogeneous intelligent agents, a local information self-identification module is constructed.

[0087] Step eight two, the local information self-identification module realizes the probability mapping from the observation state to the device identity through the self-supervised training method. Each edge device obtains the matching probability of its own characteristics in the system by individual identification of its own observation data. The higher the matching degree, the stronger the internal incentive obtained, so as to prompt the intelligent agent to adopt a strategy more matched with its own ability and state. This mechanism effectively avoids strategy conflict and information misdirection in the multi-agent learning process, and improves the stability of distributed collaborative learning.

[0088] Step eight two, reward integration and strategy update of global scheduling target:

[0089] Step eight two one, the system takes minimizing the average AoI and maximizing the effective data processing amount as external targets, and takes the device identification matching degree as an internal reward factor, to form a multi-objective optimization function.

[0090] Step eight two two, all edge devices sample environmental data in parallel, and use the experience replay mechanism to build a training data set, and update the network through the policy gradient method in reinforcement learning (such as distributed DDPG or MAPPO). After the training process is completed in the simulation environment, it can be deployed to the actual edge device for real-time operation. The system finally outputs a set of multi-agent task cooperation strategies suitable for different resource conditions and data distribution scenarios.

[0091] Specific implementation method two: combined with Figure 1 In this embodiment, the technical solutions provided in the specific implementation scheme one are further described in detail through specific examples.

[0092] I. System initialization and edge device modeling:

[0093] The multi-agent reinforcement learning method proposed in the present application is constructed based on the mobile edge computing (MEC) system architecture, and adopts a three-layer heterogeneous structure as a whole, which is respectively a bottom layer distributed data source, an intermediate layer edge device and a top layer cloud center. This structure not only meets the actual needs of modern distributed computing systems, but also provides a good physical basis for the cooperation modeling between intelligent agents.

[0094] In the bottom layer distributed data source, multiple intelligent data source devices with sensing capability are deployed, such as environmental monitoring sensors, cameras, traffic collection nodes, etc. These data source devices usually have real-time task generation capability, and can continuously generate diversified raw task data according to the geographical environment and task characteristics. These data often have obvious heterogeneity, which is manifested in different task granularity, different processing complexity and large differences in time sensitivity. Each data source has an independent identifier and can carry a weight parameter to reflect its importance ranking in the entire system task flow.

[0095] The middle layer edge device is the execution unit optimized by the application. Each edge device is modeled as an independent reinforcement learning agent and has the following core functional modules: spatial positioning module, communication interaction module, computing processing module, cache management module and action decision module, wherein:

[0096] Spatial positioning module: responsible for real-time location tracking and path planning;

[0097] Communication interaction module: manages multi-directional network connections and monitors channel quality;

[0098] Computing processing module: dynamically allocates computing power to execute local task processing;

[0099] Cache management module: intelligently schedules multi-level storage space;

[0100] Action decision module: as the intelligent hub, generates joint optimization decisions of movement / processing / offloading through the strategy network.

[0101] Each module cooperates to realize the closed-loop control of autonomous perception-decision-execution, wherein the positioning module provides spatial reference, the communication module guarantees data transmission, the computing and cache modules optimize resource utilization, the decision module integrates all state information to output the optimal strategy, and together they ensure efficient operation of the device in a dynamic environment.

[0102] The edge device not only has local data processing capability, but also can offload tasks to the cloud center for remote computing as needed, thereby flexibly balancing delay and load. In addition, the edge device has mobility and can shuttle between task-intensive areas through trajectory planning, further improving data coverage capability and task processing efficiency.

[0103] The top layer cloud center mainly undertakes model parameter aggregation, strategy learning, global resource scheduling and other functions. The cloud center keeps periodic synchronization with the edge device through the communication network, and can allocate computing resources or issue model update instructions according to device requests. In the training phase, the cloud center can also act as the main coordinator of joint training, responsible for strategy synchronization and differentiated modeling between multiple edge devices.

[0104] The MEC system is designed to follow the principles of heterogeneity, high dynamics and scalability, and can adapt to complex network topology, diverse task types and high-density device deployment scenarios. On this basis, the edge devices realize distributed collaborative control through reinforcement learning mechanism, effectively improving the overall processing efficiency and response speed of the system under resource constraints.

[0105] Among them, the edge device modeling, specifically establishing the mobility model of the edge device:

[0106]

[0107] In the formula, The edge device At the position of the tth time slot, x represents the lateral distance difference, y represents the longitudinal distance difference, and h represents the height difference; Indicates the movement in each time slot; move represents the movement radius of the edge device ; Indicates the 2-norm of the vector.

[0108] II. Data source generation and task package construction:

[0109] In the running process of the MEC system, the underlying data source node continuously generates computing tasks, and these tasks come from various distributed sensors, image acquisition devices and control trigger units in the Internet of Things. Each task will be accompanied by specific context information when generated, including generation timestamp, data type, data volume, geographic location and source number, etc. In order to reasonably express the importance and processing priority of data, the present application introduces a weight factor for each data source, which is used to distinguish the urgency and strategic significance of the task.

[0110] This design allows the system to prioritize high-value data acquisition and processing under resource constraints, thereby effectively improving the overall service level and user experience of the system. For example, in the urban security monitoring scenario, video data located in important areas (such as densely populated commercial streets or transportation hubs) will be given higher weights, and the system will prioritize scheduling edge devices to collect and process them.

[0111] The heterogeneity of the data package is reflected in multiple dimensions, including:

[0112] Task source heterogeneity: Different tasks may come from different types of sensors, such as temperature and humidity sensors, image cameras, pressure gauges, etc., resulting in diverse data types and complex structures;

[0113] Time difference: Some tasks such as abnormal alarm information have very low processing delay requirements, while regular log-type tasks have higher time delay tolerance;

[0114] Task scale difference: the data volume and computational complexity of tasks are different, some tasks may only need simple parsing, while others may need model inference, target recognition and other deep calculations;

[0115] Spatial position isomerism: data sources are distributed in different geographical locations, and there are natural differences in collection cost and scheduling path;

[0116] Priority difference: at the same time, task priority changes due to application goals (safety, efficiency, energy consumption), and the system must have dynamic adjustment capability.

[0117] All task data will be encoded into standardized task package form after generation and stored in the local cache of the data source node, waiting for collection access by adjacent edge devices. The task package structure is uniform, which is beneficial for edge device parsing and strategy evaluation. The basic information of the task package includes: data generation time, task index number, data volume, original priority, remaining time budget, required computing resource estimate and other fields.

[0118] Through unified task modeling, the system realizes the abstraction of task type, importance, timeliness, processing method and other dimensions, so that subsequent scheduling, execution and offloading strategies can be operated and compared with consistent data structures. This design provides good heterogeneous accommodation capability and strategy adaptive flexibility for the system, and is the basis for the invention to run stably in a high dynamic and multi-task environment.

[0119] Among them, considering the path loss of line of sight and non-line of sight, the air-ground channel is introduced:

[0120]

[0121] In the formula: The carrier frequency is denoted by c, and the speed of light is denoted by c: The distance between the edge device and the ground entity is denoted by d; The path loss in the line of sight and non-line of sight conditions is denoted by , The total transmission loss of the channel is denoted by. The air-ground channel model is a bridge from the physical layer to the scheduling layer of the task package, which determines whether the task can be collected / transmitted, the channel state participates in task value calculation and resource allocation, and is jointly optimized with AoI, device mobility, etc. to ensure the stability of the system in a high dynamic environment.

[0122] III. Edge device trajectory planning and mobile collection:

[0123] In the multi-agent reinforcement learning scheduling system, the active movement capability of edge devices is a key technical point of the invention. Unlike traditional static receiving task scheduling methods, the system supports edge devices to autonomously plan movement paths based on environmental perception and system strategies, actively moving to task-intensive areas or high-priority data source locations to achieve active acquisition and efficient distributed execution of tasks.

[0124] The edge device updates its current position in each scheduling period, and comprehensively evaluates its next target area based on the position state at the last time, movement capability parameters, data source distribution state, and other factors. The movement range of the device is limited by constraints such as maximum speed, terrain environment, flight restrictions, etc., so the movement strategy not only ensures task coverage efficiency, but also considers system resource conservation and device life management.

[0125] The planning process of the movement path considers the following key factors:

[0126] Task density heat map: the system constructs a task density distribution map for each area in real time, and the edge device dynamically adjusts the movement direction based on the density gradient, and preferentially approaches high task accumulation areas;

[0127] Task value priority: by analyzing factors such as task remaining time, data source weight, and current AoI, a task priority ranking is generated to assist devices in formulating target points;

[0128] Current device resource state: including device cache capacity, remaining computing resources, current execution load, etc., used to limit whether to continue collecting or switch to the unloading phase;

[0129] Neighbor device behavior prediction: to avoid multiple edge devices converging on the same task point and causing redundancy, path duplication can be achieved through local communication or strategy inference;

[0130] Energy and path cost evaluation: combining the current energy budget, the candidate path is modeled for cost, and the resource efficiency optimal path is selected for advancement.

[0131] When the device approaches an area where a task source is located, the system performs data collection confirmation based on the device's collection range radius, channel state, and data source response. Successfully collected task data is stored in the device's "collection buffer" and waits for the next processing decision. Data collection behavior is strongly associated with device location information, so the accuracy of path planning will directly affect the data effectiveness and task timeliness of the entire system.

[0132] In addition, the system supports the implementation of a "data centralized collection strategy" in special cases, i.e., multiple edge devices cooperatively collecting high-density, large task source areas. This strategy generally implements task broadcasting and role allocation through a central controller to ensure that large tasks are not missed and to reduce the risk of single-device load.

[0133] The system also introduces a "dynamic path adjustment" mechanism. During the movement of the device, if a system state mutation occurs (such as high-frequency generation of new tasks, expiration of a certain task, etc.), the current path can be interrupted and re-planned to improve the response flexibility to sudden tasks.

[0134] Through the mechanism design in this paragraph, the edge device has the ability to actively navigate and task collection in a complex and dynamic environment, maximizing the capture rate of high-value information and the balance of task resource coverage, and also provides more efficient and more targeted input data flow for the subsequent task processing strategy of the scheduling system.

[0135] Among them, the time model for the edge device to accumulate the data packets collected from the data source to perform edge processing at time slot t is established:

[0136]

[0137] In the formula: represents the data rate of the edge device for pre-processing the preset task at time slot t, which is related to the CPU cycle frequency, represents an data block collected by the edge device in the data buffer . .

[0138] This model dynamically adjusts the moving path in trajectory planning by real-time calculation of task processing time consumption, prioritizes high-value tasks (AoI-driven), and jointly optimizes mobile energy consumption and computing resource allocation to achieve rapid response to sudden tasks and efficient use of resources.

[0139] Four, local task execution and unloading strategy decision:

[0140] After the edge device completes data collection and stores task data in the local buffer, the system needs to make a decision on the processing path for each task packet. This decision includes two core options: one is local execution by the edge device, and the other is to unload the task to the cloud center for processing. This part is the core embodiment of the edge device strategy execution capability, and its rationality and efficiency directly affect the service timeliness and resource allocation balance of the entire system.

[0141] Local processing is the key path for the system to realize edge intelligence. When the computing resources of the device are in a usable state and the computing intensity of the task is low or the real-time requirement is strong, local processing is preferred to effectively reduce system delay and avoid excessive occupation of communication resources. When making a local execution decision, the system will evaluate the following key factors:

[0142] Task urgency: evaluate whether the timeliness of the task has approached the deadline threshold through the AoI (information age) of the task;

[0143] Local computing resource margin: including current CPU load, available thread number, occupied memory, etc.

[0144] Current task queue length: when there are many device tasks in the queue, to prevent congestion, the local entry frequency of new tasks should be properly controlled.

[0145] Historical processing efficiency record: combined with the past time-consuming records of the device when processing similar tasks, the execution time is predicted.

[0146] Data packet size and processing cost estimation: larger data packets require longer processing time and may be preferentially offloaded to the central node with stronger computing capability.

[0147] When the above factors are comprehensively evaluated and meet the conditions, the system will send the task to the "local execution module" and allocate computing resources to start the processing flow. The processed data will enter the "result cache queue" and wait for the next communication upload processing.

[0148] If the device is in a high load state, or the task complexity is too high, or the task remaining time is too short, the system will preferentially consider offloading it to the cloud center. Task offloading decisions usually have higher flexibility and processing resource redundancy, but are affected by communication link state and bandwidth resource limitations, so the decision strategy needs to be more detailed.

[0149] The system will consider the following information when making offloading decisions:

[0150] Current task processing queue length and feedback delay of the cloud center;

[0151] Transmission capacity and stability of the current communication link;

[0152] Data size and estimated transmission time of the task to be offloaded;

[0153] Whether the central server has a model and computing module for the corresponding type of task;

[0154] The degree of acceptance of the central scheduling strategy for task priority (e.g., low-priority tasks may be delayed for processing).

[0155] Once a task is determined to be offloaded, it will be transferred to the "to-be-transmitted queue" by the device, and the upload operation will be initiated after the communication scheduling mechanism allocates bandwidth resources. The system supports the "task priority marking mechanism", which allows the device to attach a priority label when uploading the task, so that the cloud center can quickly filter and prioritize processing in the task queue.

[0156] It is worth mentioning that in some resource-constrained or unstable communication environment, the system supports the "task forwarding" function, that is, the edge device can transfer the task to the nearby device with good communication state, realizing asynchronous offloading. This mechanism further enhances the robustness and task transmission success rate of the system.

[0157] Through the dual-channel fusion processing mechanism of local execution and cloud offloading, the system effectively alleviates the bottleneck risk of single processing path, realizes dynamic classification processing of different types of tasks, and effectively improves the task completion rate, delay control ability and resource allocation efficiency of the overall system.

[0158] Among them, the age of the data source at the tth time slot is defined as the difference between the current time and the generation time of the latest data received by the receiver

[0159]

[0160] In the formula: The generation time of the latest data packet of the data source received by the cloud center.

[0161] Five, communication link modeling and bandwidth resource allocation:

[0162] In the edge computing system, the data exchange between the edge device and the cloud center is the key path to ensure the uploading of task results and the synchronization of models, so the modeling and scheduling mechanism of the communication link becomes an important aspect of the system efficiency and stability optimization. The present application constructs a complete set of scheduling system considering communication quality, task urgency and resource fairness from the aspects of link modeling and bandwidth resource allocation.

[0163] Firstly, the system models the wireless channel state between the edge device and the cloud center. The modeling process considers the following communication characteristics:

[0164] Path loss model: considering the straight-line distance between the edge device and the cloud center, terrain shielding conditions and other factors, distinguishing the signal attenuation under line of sight (LOS) and non-line of sight (NLOS) conditions;

[0165] Noise interference modeling: introducing the environmental noise density estimation value, evaluating the channel quality according to the device working frequency band;

[0166] Communication power control mechanism: the edge device can dynamically adjust the transmission power according to the link state, balancing the transmission speed and energy consumption;

[0167] Transmission rate evaluation: based on the link state and bandwidth constraint, the achievable data transmission rate is predicted as the input of task uploading time estimation.

[0168] ​After the communication modeling is completed, the system enters the bandwidth resource allocation phase. In the scenario of multiple edge devices sharing limited communication bandwidth, reasonable allocation of the uplink bandwidth ratio of each device becomes the key to scheduling efficiency. Therefore, the present application adopts a distributed bandwidth allocation mechanism with task priority labels, and the core steps include:

[0169] Task level identification: edge devices mark task priority when uploading tasks, such as "urgent", "high", "medium", and "low", to guide resource allocation;

[0170] Device state reporting mechanism: edge devices periodically report the current task buffer pressure, task average AoI, communication status, etc. to the scheduling center;

[0171] Central scheduling strategy optimization: the cloud center scheduling module constructs a global scheduling graph according to the states of all devices, and preferentially allocates higher bandwidth to nodes with high task urgency and good device channel quality;

[0172] Bandwidth ratio dynamic updating mechanism: to avoid single device occupying bandwidth for a long time, the system sets a ratio updating threshold to ensure fairness;

[0173] Backoff and retransmission mechanism: when uploading fails or bandwidth scheduling is abnormal, the device can enter a backoff and retry state to ensure that the data is eventually delivered.

[0174] In addition, to improve the robustness and environmental adaptability of the communication link, the present application introduces a "predictive scheduling" mechanism. In certain scenarios (such as known UAV trajectories and fixed vehicle paths), the system can evaluate the link quality fluctuations in the future time period based on historical communication states and path prediction models, and schedule task upload windows in advance. This mechanism is particularly important in high-speed mobile edge nodes, effectively reducing data loss and delay fluctuations caused by temporary disconnection.

[0175] It is worth mentioning that under extreme resource constrained conditions, the present system supports task compression transmission mechanism, and the device can compress and encode low priority tasks, and only upload summary information to the cloud center, and then perform supplementary transmission when the communication conditions improve. This mechanism significantly improves the data utilization efficiency per unit bandwidth, while reducing the risk of system communication congestion.

[0176] By constructing a dual mechanism of link modeling and bandwidth allocation, the system realizes dynamic optimization control of task uploading, so that each bit of communication resource can produce benefit on the most valuable data transmission. This strategy not only improves the timeliness and success rate of task execution, but also enhances the anti-pressure ability and cooperation stability of the system in the communication quality fluctuation scene. In the actual method, the interaction and cooperation of heterogeneous multi-agents are realized by constructing a Markov decision process (MDP) model. The Markov decision process (MDP) is represented by four tuples as follows: M(S,A,R,P), wherein: S represents the state, the state of the edge agent includes the buffer state, the allocated offloading bandwidth and other local observations, and the state of the central agent integrates the states of all edge devices; A represents the action; R represents the reward; and P represents the transition strategy.

[0177] Six, scheduling optimization driven by information age and devaluation mechanism:

[0178] In the distributed edge computing environment, the real-time performance of task data is directly related to its processing value and system feedback ability. Traditional task scheduling methods mostly use task queue length, average waiting time and other indicators as the basis for processing priority judgment, but these indicators cannot accurately reflect the freshness of the data itself and the urgency of the system response. Therefore, the present application proposes to introduce "information age (AoI, Age of Information)" as the core scheduling parameter, and establishes a data devaluation evaluation mechanism, which effectively guides the edge device to make better task processing and offloading decisions.

[0179] Information age refers to the interval length from the generation time of a data packet to the current time. This indicator dynamically increases in time units, reflecting the "aging" process of the data. In system operation, the smaller the AoI value, the newer the data and the higher the processing value; the larger the AoI value, the more likely the data has lost reference significance or will expire soon.

[0180] The system maintains the AoI value of each data packet in real time and uses it as an important input variable for task priority evaluation. To improve the accuracy and response ability of the scheduling decision, the system further constructs a "task value devaluation model". This model defines how the residual value of a task decreases over time within different AoI ranges. This devaluation function can be customized according to the application scenario, for example, in the industrial quality detection scenario, data delay exceeding several seconds is invalid; while in weather monitoring, several minutes of delay may still be acceptable.

[0181] The system uses the following method for task value evaluation:

[0182] Dynamic calculation of task AoI: In each scheduling period, the system updates the AoI value of all tasks to be processed and uploaded, which is automatically calculated based on the current system clock and data generation time;

[0183] Devaluation factor modeling: Set the time-sensitive threshold corresponding to the task type, when the AoI exceeds the threshold, the system gradually reduces its priority and value coefficient according to the devaluation function;

[0184] Multi-dimensional task weight fusion: In addition to the influence of AoI, the original importance of the task package (such as from a critical area), data type (such as image, log, alarm), etc. also participate in the final value scoring;

[0185] Value-guided queue sorting: When the task is processed locally or uploaded by the edge device, the priority is sorted according to the comprehensive value, ensuring that the most meaningful data is prioritized for scheduling;

[0186] Deadline processing mechanism: For data packets that will exceed the maximum AoI threshold, the system will perform emergency processing scheduling, or directly discard to save resources and avoid invalid processing.

[0187] Through the above mechanisms, the system can achieve dynamic value evaluation at different task life cycle stages, so that the scheduling strategy is no longer limited to static rules or first-in-first-out methods, but is truly oriented to the "effectiveness" of the task. In this way, even in the case of extremely limited resources, the system can still prioritize the most meaningful data to be processed or uploaded, maximizing the response value density of the system.

[0188] In addition, the present application supports a task value visualization module, and the scheduling center can grasp the task value distribution of the whole system in real time through a heat map and the like, providing a reference basis for high-level management scheduling or strategy fine-tuning.

[0189] The introduction of information age and task devaluation mechanism is a core innovation in the technical route of the present application. It not only makes up for the defects of existing scheduling models in data "freshness" identification, but also provides a dynamic objective function for the reward and punishment design of the reinforcement learning strategy, and effectively improves the intelligent response ability of the whole multi-agent system under the condition of heterogeneous tasks.

[0190] Seven, multi-agent strategy optimization based on Actor-Critic:

[0191] In the edge computing system, multiple distributed edge devices participate in data collection, processing and uploading tasks at the same time, and there are both cooperation needs and autonomous judgment based on local information among them. The traditional centralized scheduling architecture often faces communication bottlenecks, high delay and single point failure when dealing with large-scale devices, and it is difficult to meet the real-time and system robustness requirements. Therefore, the invention proposes a strategy optimization method based on a multi-agent reinforcement learning (Multi-Agent Reinforcement Learning, MARL) architecture, which realizes individualized learning and collaborative control of edge devices in a completely distributed state.

[0192] Each edge device is regarded as an agent (Agent), which has independent observation ability, decision-making ability and experience memory mechanism. The agent learns to autonomously master various behavior strategies such as task collection, mobile path selection, task processing path, and upload timing control. Although each agent is independent, it can achieve system-level collaborative optimization under the guidance of global goals through certain degree of information exchange or strategy sharing.

[0193] To enhance the adaptability of the system under the condition of heterogeneous tasks and device differences, the system designs a heterogeneous multi-agent strategy structure with the following specific features:

[0194] Edge agent local strategy network (Local Actor): Each agent generates its next behavior action according to its own state input (including current position, cache state, task queue information, communication state, etc.), such as whether to move, moving direction, whether to process, whether to upload, etc. The strategy network can dynamically adjust the network structure according to the device type, for example, different structure parameters may be used for aircraft, vehicle terminal and fixed node.

[0195] Edge agent local evaluation network (Local Critic): used to evaluate the value of a state-action pair and provide learning feedback for the Actor network. This network can combine task value, communication cost, processing delay and other indicators to form a comprehensive reward reference.

[0196] Central coordinator (optional): The system allows the introduction of a central scheduler as an auxiliary role, which can provide global suggestions or state summary information to some edge devices in some task scenarios, but does not participate in specific strategy execution to ensure the independence of the agent.

[0197] Shared strategy optimization framework: Although each agent strategy network can be updated independently, the system allows devices with similar attributes to share some network parameters or training experience to speed up the training convergence speed and enhance the generalization ability.

[0198] Personalized differential expression mechanism: Introduce edge device label or task preference vector, embedded in the policy network, to capture the preference of different devices for task type, mobile cost, execution ability, etc., effectively improving the policy adaptation degree.

[0199] Policy input information construction: The input state of each agent not only includes the basic state vector, but also includes multi-dimensional information such as task heat map, historical trajectory summary, task AoI statistical distribution, and channel state encoding, to enhance the state space expression ability.

[0200] Action output structure design: The output action of the agent is a multi-dimensional combined action, including discrete (whether to execute the task) and continuous variables (movement vector, power adjustment value), and the system uses a multi-head policy structure to model and fuse the output of different dimensional actions.

[0201] Through the above structure design, the edge device can realize autonomous adaptation and personalized scheduling in complex environment, avoiding the non-scalable problem caused by the dependence of traditional centralized policy on global information. In real scenarios with high task dynamics and strong device differences, this architecture has good flexibility and deployment convenience, and can achieve high system intelligence level under low communication overhead.

[0202] During policy training, the system realizes policy optimization through the centralized training and distributed execution (CTDE) mechanism. All agents can share experience pool and update policy parameters in the training phase, while running independently in actual deployment without relying on global state or other agent information, realizing truly autonomous intelligent control of edge.

[0203] Among them, the local information self-recognition module as the internal reward function target of HA-MAAC-Trans is to train a global probability classifier with parameters p , so that the input is observation , and the output is the probability that the observation belongs to each edge device , so as to identify an edge device from different given observations , for modeling the independent action of each edge device; define the internal reward , which represents the possibility of accurately predicting / identifying a certain edge device from the observation. Therefore, the total reward of the edge device is represented as the combination of the external reward and the internal reward:

[0204]

[0205] In the formula: ; represents the adjustment weight, which measures the importance between the internal reward and the external reward; represents the external reward from the environment, represents the total reward.

[0206] The physical meaning of intrinsic reward is that, since edge devices can only partially observe the environment, reaching a distant area can help an edge device distinguish from others' observations and identify itself, thus obtaining a higher intrinsic reward encourages clearer division of labor. Since edge devices maximize expected future returns, the differences between edge device strategies are strengthened, which makes them more identifiable and self-interested, thus promoting the training of the classifier p. Therefore, the learning process is a closed-loop process with positive feedback.

[0207] Adding LISIM is of great significance in data collection. When edge devices collect data sources from various places, new local observations mean discovering and accessing previously uncontacted data sources, which can trigger new trajectory patterns and obtain higher cumulative external rewards. Intrinsic rewards can be seen as a positive reinforcement signal for exploration, encouraging agents to optimize task performance rather than weakening their motivation to focus on completing tasks. Through the personalized recognition ability of LISIM, agents can more effectively adjust their strategies to balance exploration and task completion goals, improving the overall system efficiency.

[0208] Eight, reward function design and reinforcement learning process:

[0209] In the reinforcement learning process of multi-agent system, the learning direction and convergence quality of the strategy are highly dependent on the design of the reward function. A reasonable reward function can not only accurately reflect the optimization goal of the system, but also guide each agent to evolve towards a collaborative and efficient behavior. Therefore, the present application adopts a multi-level mechanism of "external reward + intrinsic reward" in reward modeling, constructs optimization goals from the system as a whole and the individual agent, and realizes multiple guidance on task value, resource efficiency and strategy difference.

[0210] External reward design (system level)

[0211] The external reward is mainly constructed based on the key performance indicators of the entire system to measure the overall operation effect of the edge computing system in a specific scheduling period. Specifically, it includes:

[0212] Task processing reward: the more the number of tasks successfully completed by edge devices (local processing or cloud completion) in the current time period, the higher the reward, guiding agents to improve task completion rate;

[0213] Average information age penalty: the system calculates the AoI of all tasks, if the overall AoI is low, the reward is high, otherwise, if there is serious delay, negative incentive is given to guide agents to process fresh data first;

[0214] Communication efficiency reward: considering the value density of data uploaded per unit of communication bandwidth, if the agent chooses to upload a task with high value or low bandwidth occupation, it will receive higher rewards;

[0215] Task discard penalty: if a task is invalidated due to delay, congestion, or improper scheduling, the system will impose a penalty on the corresponding device or scheduling strategy to reduce the system burden of invalid data;

[0216] Energy consumption control reward: considering the trend of energy consumption in device movement, computation, and communication processes, guide the device to choose a more energy-efficient behavior path.

[0217] Intrinsic reward design (individual level of agent)

[0218] To overcome the possible coordination confusion or strategy homogenization problems of multi-agent systems in the early stages of training, the invention introduces a "local information recognition module" (LISIM) to build an intrinsic motivation mechanism for agents, enabling agents to form independent behavior characteristics during the learning process and improve their autonomy in recognizing their own state and behavior selection. The intrinsic reward design includes:

[0219] Identity distinguishability reward: a lightweight classification network is used to determine whether the current state and behavior of the agent have sufficient differences (i.e., "whether it can be correctly identified as a unique device"), and the more accurate the identification, the higher the intrinsic reward;

[0220] Behavioral diversity incentive: if an agent chooses to take different actions from others in the same state (non-imitative behavior), the system gives positive rewards to encourage strategy exploration;

[0221] Local optimal feedback: when an agent locally processes tasks, if its scheduling strategy results are better than the historical average performance, it can obtain individual reinforcement signals;

[0222] Individual behavior persistence reward: the system observes the consistency of the agent's strategy, and if it maintains an efficient personalized strategy behavior for a long time, it will receive stable reinforcement signals to avoid frequent strategy shocks.

[0223] Reward fusion mechanism and weight dynamic adjustment

[0224] External rewards and intrinsic rewards will be fused into the final objective function during training. To meet the needs of different stages of training, the system sets the reward fusion coefficient to be dynamically adjustable:

[0225] In the early stages of training, the system increases the proportion of intrinsic rewards to encourage agents to fully explore the strategy space and form strategy differences;

[0226] In the middle and later stages, gradually increase the proportion of external rewards to guide the strategy to converge to the global optimal solution;

[0227] In extreme scenarios (such as task bursts, communication malfunctions), the system can quickly adjust the weight structure according to the policy failure feedback to cope with sudden target changes.

[0228] Reinforcement learning training process and optimization method

[0229] The system uses a distributed reinforcement learning framework for policy training, and the process is as follows:

[0230] Each agent samples an action according to the current local state and executes it;

[0231] The environment feedback includes the next state, task changes, reward signals, etc.

[0232] The agent records the experience (state, action, reward, next state) to the experience replay pool;

[0233] Use policy gradient method or value function approximation method for policy update;

[0234] Update the policy every few cycles to the device running end to complete policy iteration.

[0235] The system supports common multi-agent reinforcement learning algorithm frameworks, including DDPG, MADDPG, MAPPO, QMIX, etc. The specific algorithm can be flexibly selected according to the task type, state space complexity, and policy generalization requirements.

[0236] Nine, system convergence training and deployment optimization

[0237] This step is to ensure that the multi-agent scheduling policy can run stably and effectively in actual deployment. The present invention not only proposes innovative points in modeling and policy structure design, but also builds a complete set of engineering processes for policy training and system deployment, ensuring that the system has good convergence performance, migratability, and online update capability.

[0238] Policy training and convergence guarantee mechanism

[0239] In the training phase, the system carries out multi-agent reinforcement learning process through centralized training and decentralized execution mechanism. This process improves training efficiency and ensures stable convergence of the policy through the following key strategies:

[0240] Step-by-step policy iteration mechanism

[0241] A relatively loose exploration strategy is used at the beginning of training to guide the agent to fully explore the possible state-action space; a centralized evaluation module is introduced in the middle period to moderately converge the exploration range according to external performance indicators; and in the later period, the policy skeleton is fixed, only the details of the behavior are fine-tuned, and the risk of policy shock is reduced.

[0242] Multi-stage environmental disturbance training

[0243] In order to enhance the robustness of the strategy in real deployment, the system designs multiple disturbance scenarios such as communication interruption, task burst increase, and computing resource fluctuation, and trains the agent to adapt to abnormal states in advance through "adversarial disturbance training".

[0244] Experience fusion update strategy

[0245] The experience pool formed by each edge device during the training process can be periodically fused, and the public strategy backbone is updated through joint learning, while the individual fine-tuning weight is reserved to support the unity of system strategy generalization ability and individual difference expression.

[0246] Convergence evaluation criteria

[0247] The system adopts a series of evaluation indexes such as task completion rate improvement amplitude, average information age decline trend, strategy stable fluctuation range, training round convergence curve, etc., to comprehensively judge whether the training enters a stable state, and if the indicators converge continuously for multiple rounds, the current model is saved as a deployment version.

[0248] Model simplification and inference acceleration

[0249] After completing the training, the system automatically performs structure pruning and weight quantization operations on the strategy network to optimize the running efficiency of the model in the edge device, and ensures that the low-power device can also stably execute the intelligent strategy.

[0250] Strategy deployment and online update mechanism

[0251] After completing the training phase, the system distributes the trained strategy to each edge device and deploys it according to the actual scene. The present application considers the following key engineering elements in the deployment process:

[0252] Strategy packaging and edge distribution

[0253] The trained strategy model is packaged and compressed, and then distributed to all participating devices through a secure channel. Each device loads the corresponding model parameters or fine-tuning branches according to the label to ensure deployment consistency.

[0254] Lightweight execution framework

[0255] The strategy model loaded by the edge device at runtime only retains the necessary inference part, and the background environment maintains a resource release mechanism to avoid task delays caused by intelligent control.

[0256] Strategy health state monitoring

[0257] Each device periodically feeds back the strategy running results to the central system, including task completion success rate, average AoI, resource usage, etc. The system judges whether the current strategy is stable and effective through a dynamic scoring mechanism.

[0258] Online fine-tuning and model hot update

[0259] During system operation, if phenomena such as task structure changes and external interference persistently increase occur, the system can collect the latest experience of edge devices through the federated learning mechanism and complete retraining and fine-tuning in the cloud center. The new strategy can replace the original strategy through the hot update mechanism without stopping the system operation, realizing self-optimization of the agent behavior.

[0260] Multi-scene expansion capability

[0261] The deployment mechanism supports rapid migration to different scenes such as intelligent transportation, emergency response, and industrial quality inspection. The strategy network can flexibly adjust the input and output dimensions according to the task dimension changes, minimizing the scene switching cost.

[0262] Actual deployment example

[0263] In real scenarios, such as a "multi-UAV environmental data patrol system", each UAV acts as an edge device and autonomously selects a cruise trajectory, a collection station, and a processing path according to the multi-agent reinforcement learning method of the present application. The system has successfully deployed the strategy obtained through training in different environments, and the results show that the task coverage rate has increased by about 30%, the data delay has been reduced by about 45%, and the average resource utilization has been improved by more than 50%.

Claims

1. A multi-agent reinforcement learning method, characterized in that... The method includes the following steps: Step 1: System Initialization and Edge Device Modeling Construct a multi-agent environment in the edge computing system and initialize the state information of multiple edge nodes, including location information, cache capacity, computing resources and communication conditions; Step 2: Data Source Generation and Task Package Construction It receives heterogeneous task data from multiple data sources and identifies the data volume, generation time, data type, and priority weight of each task. Step 3: Edge Device Trajectory Planning and Motion Data Acquisition Based on the current location of the edge node, the distribution of data sources, and the value of the task, a movement path is generated and target task data is collected. The collected data is stored in the local task buffer. Step 4: Local task execution and uninstallation strategy decision: Based on the status of local computing resources and the urgency of tasks, select some tasks from the buffer for local processing and schedule the remaining tasks to the queue to be uploaded. Step 5: Communication Link Modeling and Bandwidth Resource Allocation Based on the link status and global bandwidth allocation information, select an appropriate time window to send the data of the task to be uploaded to the cloud center; Step Six: Scheduling Optimization Driven by Information Age and Depreciation Mechanisms Calculate the information age of all tasks and adjust the processing priority of tasks in the local queue based on task timeout status. Step 7: Optimization of multi-agent strategy based on Actor-Critic: Local state information is input into the policy network, and the output includes scheduling actions such as moving, processing, and uploading, which are used to control the next behavior of edge nodes. Step 8: Reward Function Design and Reinforcement Learning Process: Receive environmental feedback, including task success status and data timeliness indicators, generate external rewards, and update the policy network by combining the intrinsic incentives generated by identifying behavioral differences.

2. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step one are as follows: Step 11: Construct a three-tier MEC system, including a bottom-layer distributed data source, a middle-layer edge device, and a top-layer cloud center; Step 1 and Step 2: Initialize the spatial location information, maximum mobility range, cache capacity, computing power, and available bandwidth resources of each edge device; Step 13: The data source is assigned weights based on physical location, sensing task type, and urgency level, which are used as a scheduling and sorting reference. The edge device will then use these weights to determine the movement path and task priority.

3. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step two are as follows: Step 21: In the Internet of Things (IoT) environment, the data source periodically collects environmental information and generates data packets with timestamps, data size, and source indexes. Step 22: The MEC system encapsulates the data generated in Step 21 into task packages for edge devices to collect. Steps 2 and 3: To depict the trend of data value change, an information age indicator is introduced, and a data devaluation model is constructed in combination with data source weights to clarify the value change of task packages at different times. Step 24: Each data packet is assigned to the collection queue and waits for a response from a nearby edge device.

4. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step three are as follows: Step 31: Edge devices update their trajectories based on their current location and the distribution of nearby data sources; Step 32: When the edge device enters the effective acquisition range, it automatically obtains the task package from the connected data source and transfers it to the local acquisition buffer. Step 3: During the process of moving, the edge device continuously updates the sensing data, evaluates the priority between new data sources and existing tasks, and dynamically adjusts the path direction to improve collection efficiency and coverage.

5. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step four are as follows: Step 41: The edge device determines whether each task to be processed meets the local processing conditions. Step 42: If the current processing unit is idle or the task AoI is low, it is allocated to the execution buffer for local processing. The local execution process is constrained by the edge CPU frequency and cache space. A task block processing strategy is adopted to improve processing efficiency. After processing is completed, the task result is transferred to the completed queue to be uploaded to the cloud center. Step 43: If local resources are insufficient or the task delay tolerance is low, the task will be scheduled to the unload buffer.

6. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step five are as follows: Step 51: Establish communication channel models between edge devices and data sources, and between edge devices and cloud centers, and calculate link loss and reachable speed; Step 52: When allocating bandwidth, the central agent obtains the status and communication requirements of all edge devices, generates a global bandwidth ratio vector, and allocates the proportion of usable communication channels to each edge device. Step 53: The device decides whether to upload based on the allocation ratio, task packet size, and channel status.

7. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step six are as follows: Step 61: The system continuously tracks the time span of each data packet from its generation to the present and calculates its information age in real time; Step 62: Calculate the current remaining value of the data packet based on the set AoI threshold and devaluation factor; Step 63: Edge devices use the devaluation function result to determine whether the task is worth continuing to process, and prioritize the processing and unloading of data with high value. Step 64: When the edge device sorts the task packet queue, it comprehensively considers the packet size, AoI, remaining value and location distance, and dynamically selects the scheduling target from multiple candidate tasks.

8. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step seven are as follows: Construct a multi-agent policy learning structure based on Actor-Critic; the input of the Actor network of the edge device includes local state and observed task package information, and the output includes three parts of decision: movement policy, local execution policy and unloading policy; The edge Critic network combines the input state with the output actions of the edge actors to estimate the action value function, which is used to train the actors to update their behavior policies. The central actor takes the state of all edge devices as input and outputs a bandwidth allocation policy. The central Critic is used to estimate the global action value. All networks use feature extraction modules containing convolutional and Transformer structures to enhance their ability to express spatial distribution and temporal changes.

9. The multi-agent reinforcement learning method according to claim 1, characterized in that... The specific steps of step eight are as follows: Step 81: Integrating Intrinsic Motivation Mechanisms with Individual Identification Step 811: To enhance the collaborative capabilities among heterogeneous intelligent agents, the system constructs a local information self-identification module; Step 812: The local information self-identification module realizes the probability mapping from observation state to device identity through self-supervised training. Each edge device performs individual identification on its own observation data and obtains the matching probability of its own features in the system. Step 82: Reward integration and strategy update for global scheduling targets: Step 821: The system takes minimizing the average AoI and maximizing the effective data processing volume as external objectives, while taking the device identification matching degree as an internal reward factor, thus forming a multi-objective optimization function; Step 822: All edge devices sample environmental data in parallel and use the experience replay mechanism to build a training dataset, and update the network through the policy gradient method in reinforcement learning.

10. A multi-agent reinforcement learning device for implementing the method of any one of claims 1-9, characterized in that... The device includes a system initialization and edge device modeling module, a data source generation and task package construction module, an edge device trajectory planning and mobile acquisition module, a local task execution and unloading strategy decision module, a communication link modeling and bandwidth resource allocation module, a scheduling optimization module driven by information age and depreciation mechanism, an Actor-Critic-based multi-agent strategy optimization module, and a global scheduling target reward integration and strategy update module, wherein: The system initialization and edge device modeling module is responsible for constructing the multi-agent environment in the edge computing system and initializing the state information of multiple edge nodes. The data source generation and task package construction module is responsible for receiving heterogeneous task data from multiple data sources and identifying the data volume, generation time, data type and priority weight of each task. The edge device trajectory planning and mobile acquisition module is responsible for generating a mobile path and collecting target task data based on the current location of the edge node, the distribution of data sources and the value of the task. The collected data is stored in the local task buffer. The local task execution and unloading strategy decision module is responsible for selecting some tasks from the buffer for local processing based on the status of local computing resources and the urgency of the tasks, and scheduling the remaining tasks to the queue to be uploaded. The communication link modeling and bandwidth resource allocation module is responsible for combining link status and global bandwidth allocation information to select an appropriate time window to send the task data to be uploaded to the cloud center. The scheduling optimization module driven by the information age and devaluation mechanism is responsible for calculating the information age of all tasks and adjusting the processing priority of tasks in the local queue according to the task timeout status. The multi-agent policy optimization module based on Actor-Critic is responsible for using local state information as input to the policy network and outputting scheduling actions including movement, processing, and uploading to control the next behavior of edge nodes. The global scheduling target reward integration and policy update module is responsible for receiving environmental feedback to generate external rewards, and updating the policy network by combining the intrinsic incentives generated by identifying behavioral differences.

Images