Heterogeneous node charging scheduling method and system based on multi-agent deep reinforcement learning

Abstract

The invention discloses a heterogeneous node charging scheduling method and system based on multi-agent deep reinforcement learning, and relates to the field of power grid inspection scheduling, and the method comprises the steps: obtaining an energy change result of each unmanned plane at any time slot, and a transmission parameter of a ground device for carrying out the communication service with each unmanned plane or unmanned vehicle; calculating a first parameter and a second parameter according to the energy change result of each unmanned aerial vehicle in any time slot; calculating a third parameter and a fourth parameter according to the transmission parameters; establishing an objective function for maximizing the minimum cumulative throughput of the communication service between the unmanned aerial vehicle or the unmanned vehicle and the ground equipment, and determining constraint conditions of the objective function by combining the first parameter, the second parameter, the third parameter and the fourth parameter; and under the constraint condition, an MADDPG algorithm is adopted to solve the target function, and an optimal strategy of cooperative charging scheduling of the unmanned aerial vehicle and the unmanned vehicle is obtained.

Description

Technical Field

[0001] This invention relates to the field of power grid inspection and dispatching, and more specifically, to a heterogeneous node charging dispatching method and system based on multi-agent deep reinforcement learning. Background Technology

[0002] With the continuous expansion of the deployment scale of grid-side facilities such as distribution automation terminals, smart meters, and fault indicators, coupled with the widespread access of power application terminals such as distributed energy resources, electric vehicle charging facilities, and user-side smart electrical equipment, the number of power grid data collection points and communication demands have surged dramatically. Massive numbers of smart terminals continuously generate multi-dimensional data on operating status, electricity consumption behavior, and environmental parameters at high frequencies, leading to an exponential increase in the data volume carried by power communication systems. This poses a more severe challenge to power supply reliability and Quality of Service (QoS). Traditional terrestrial communication facilities have inherent limitations such as high deployment costs and long construction periods. In areas with complex geographical environments or significant load fluctuations, their coverage and connection reliability are difficult to guarantee, ultimately hindering the efficient improvement of intelligent operation and maintenance and real-time monitoring capabilities of the power grid.

[0003] Unmanned Aerial Vehicles (UAVs), with their high mobility, low deployment costs, and reliable line-of-sight (LOS) transmission capabilities, have become an important technological means to enhance power grid communication and inspection systems. Aerial communication nodes built by UAVs can flexibly supplement existing ground communication networks or quickly restore basic communication capabilities in power areas during disaster emergencies, significantly improving the power grid's emergency response level. Furthermore, the high mobility of UAVs helps establish reliable line-of-sight (LoS) links in complex terrain, thereby improving communication conditions at remote substations, transmission lines, and other locations. However, UAVs still face many challenges in practical power grid applications, with the frequent charging requirements due to limited battery life being a major bottleneck restricting their long-term autonomous operation.

[0004] Currently, rapid battery replacement technology can alleviate range anxiety, but this method still relies heavily on manual intervention and ground support, making true autonomous operation difficult. Frequent task interruptions and personnel reassignment will lead to soaring maintenance costs and decreased response efficiency in remote areas or large-scale fleet applications. Therefore, how to achieve intelligent scheduling of UAVs during power grid inspection and communication assurance missions, while balancing regional coverage fairness and service continuity, is a pressing issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to provide a heterogeneous node charging scheduling method and system based on multi-agent deep reinforcement learning, which solves the problems in the prior art of drone scheduling lacking a collaborative charging strategy and being difficult to balance coverage fairness and communication service continuity.

[0006] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0007] In a first aspect, the present invention provides a heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning, the method comprising:

[0008] Obtain the energy change results of each drone in any time slot, as well as the transmission parameters for communication services between ground equipment and each drone or unmanned vehicle;

[0009] Based on the energy change results of each drone in any time slot, a first parameter characterizing the charging urgency of the drone and a second parameter characterizing the health status of the drone communication service are calculated.

[0010] The third parameter characterizing the performance of the communication service and the fourth parameter characterizing the fair allocation of resources are calculated based on the transmission parameters.

[0011] Establish an objective function that maximizes the minimum cumulative throughput for communication services between drones or unmanned vehicles and ground equipment, and determine the constraints of the objective function by combining the first, second, third, and fourth parameters;

[0012] Under constraints, the MADDPG algorithm is used to solve the objective function and obtain the optimal strategy for collaborative charging scheduling between UAVs and unmanned vehicles.

[0013] In one implementation, the transmission parameters for communication services between ground equipment and drones or unmanned vehicles are obtained, including:

[0014] Acquire the real-time location information of drones, unmanned vehicles, and ground equipment, as well as the channel parameters for drones and unmanned vehicles to transmit data to ground equipment;

[0015] Based on real-time location information and channel parameters, the transmission parameters for communication services between ground equipment and drones and unmanned vehicles are calculated.

[0016] In one implementation, a first parameter characterizing the urgency of charging the drone is calculated based on the energy change of the drone in any time slot, specifically:

[0017] Obtain the minimum and maximum stored energy for each drone;

[0018] Based on the energy change results, minimum reserve energy, and maximum reserve energy of each drone in any time slot, a first parameter characterizing the charging urgency of the drone is calculated.

[0019] In one implementation, a second parameter characterizing the health status of the UAV's communication service is calculated based on the energy variation of the UAV in any time slot, specifically:

[0020] Based on the relationship between the energy change results of each UAV in any time slot and the minimum reserve energy, the first binary variable of the health status of the communication service of each UAV is determined;

[0021] The first binary variable for each drone is summed and then divided by the total number of drones to obtain the second parameter characterizing the health status of the drone communication service.

[0022] In one implementation, a third parameter characterizing the performance of the communication service is calculated based on the transmission parameters, specifically:

[0023] Determine a second binary variable for each UAV to provide communication services to ground equipment in any time slot, and a third binary variable for each unmanned vehicle to provide communication services to ground equipment in any time slot; wherein the states of the second binary variable and the third binary variable are mutually exclusive.

[0024] Based on the second and third binary variables, and combined with the transmission parameters of the communication service between the ground equipment and each UAV or unmanned vehicle, the cumulative throughput of each ground equipment is calculated.

[0025] The minimum cumulative throughput is taken as the third parameter characterizing the performance of the communication service.

[0026] In one implementation scheme, a fourth parameter characterizing the fair allocation of resources is calculated based on the transmission parameters. Specifically, the fourth parameter characterizing the fair allocation of resources is calculated based on the second binary variable, the third binary variable, and the transmission parameters of the communication service between the ground equipment and each drone or unmanned vehicle, combined with the Jain fairness index.

[0027] In one implementation, the constraints also include that each UAV can only be charged by one unmanned vehicle in any time slot, each ground device can only be communicated by one UAV or unmanned vehicle, and at least one unmanned vehicle is used to perform charging services within the maximum horizontal coverage radius of the UAV.

[0028] In one implementation, before using the MADDPG algorithm to solve the objective function and obtain the optimal strategy for charging scheduling of UAVs and unmanned vehicles, the method further includes: determining the state space, action space, and reward function of the MADDPG algorithm.

[0029] In one implementation, the state space includes the state of the drone and the state of the unmanned vehicle, wherein the state of the drone includes the drone's location, energy level, and throughput, and the state of the unmanned vehicle includes the location information and throughput of the mobile charging station.

[0030] The action space includes the actions of drones and unmanned vehicles. The actions of drones include flying to ground equipment to provide communication services and flying to unmanned vehicles to provide charging services. The actions of unmanned vehicles include moving to the drone's coverage area to wait for charging services and providing communication services to ground equipment.

[0031] The reward function is the sum of the minimum throughput of all ground devices up to time t.

[0032] A second aspect of the present invention provides a heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning, the system comprising:

[0033] The acquisition module is used to acquire the energy change results of each UAV in any time slot, as well as the transmission parameters for communication services between ground equipment and each UAV or unmanned vehicle.

[0034] The parameter calculation module is used to calculate a first parameter characterizing the charging urgency of the drone and a second parameter characterizing the health status of the drone's communication service based on the energy change results of each drone in any time slot; and to calculate a third parameter characterizing the performance of the communication service and a fourth parameter characterizing the fair allocation of resources based on the transmission parameters.

[0035] The function construction module is used to establish an objective function that maximizes the minimum cumulative throughput of communication services between drones or unmanned vehicles and ground equipment, and to determine the constraints of the objective function by combining the first, second, third and fourth parameters;

[0036] The strategy solving module is used to solve the objective function under constraints using the MADDPG algorithm to obtain the optimal strategy for collaborative charging scheduling between UAVs and unmanned vehicles.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This invention introduces unmanned vehicles (UAVs) to construct a joint communication network architecture for UAVs and UGVs. By combining the energy change results of UAVs and the transmission parameters of communication services, four parameters are calculated for evaluating and optimizing the communication network architecture: a first parameter for charging urgency, a second parameter for network health, a third parameter for global throughput, and a fourth parameter for fair resource allocation. Based on these four parameters as constraints of the objective function that maximizes the minimum cumulative throughput of all ground devices, the MADDPG algorithm is used to solve the objective function, thereby obtaining the optimal strategy for collaborative charging scheduling of UAVs and unmanned vehicles. This enables intelligent scheduling and energy replenishment of UAVs, effectively alleviating the battery life bottleneck of UAVs and enhancing the emergency response capability of the power grid. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart illustrating the heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning provided in an embodiment of the present invention.

[0041] Figure 2 A communication network block diagram between the drone, the unmanned vehicle, and the ground equipment provided in an embodiment of the present invention;

[0042] Figure 3 The flowchart of the MADDPG algorithm for solving the optimal strategy provided in the embodiments of the present invention is shown. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0044] It should be noted that the terms "comprising" or "may include" used in the various embodiments of this application indicate the presence of the claimed function, operation, or element, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in the various embodiments of this application, the terms "comprising," "having," and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or adding one or more combinations of the foregoing.

[0045] It should be understood that terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0046] As described in the background section, unmanned aerial vehicles (UAVs), with their high mobility, low deployment cost, and reliable line-of-sight (LOS) transmission capabilities, have become an important technological means to enhance power grid communication and inspection systems. Aerial communication nodes built by UAVs can flexibly supplement existing ground communication networks or quickly restore basic communication capabilities in power areas during disaster emergencies, significantly improving the power grid's emergency response level. Furthermore, the high mobility of UAVs helps establish reliable line-of-sight (LoS) links in complex terrain, thereby improving communication conditions at remote substations, transmission lines, and other locations. However, UAVs still face many challenges in practical power grid applications, with the frequent charging requirements due to limited battery life being a major bottleneck restricting their long-term autonomous operation.

[0047] Currently, rapid battery replacement technology can alleviate range anxiety, but this method still relies heavily on manual intervention and ground support, making true autonomous operation difficult. Frequent mission interruptions and personnel reassignment will lead to soaring maintenance costs and decreased response efficiency in remote areas or large-scale fleet applications. Therefore, how to achieve intelligent scheduling of UAVs (Unmanned Aerial Vehicles) during power grid inspection and communication support mission cycles, while ensuring regional coverage fairness and service continuity, is a pressing issue that needs to be addressed.

[0048] To address the technical shortcomings described above, this embodiment provides a heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning. It constructs a joint communication network architecture for UAVs and Unmanned Ground Vehicles (UGVs), establishes communication transmission and energy calculation models, and proposes system optimization indices to characterize an optimization function that aims to maximize the minimum cumulative throughput of all ground equipment. The objective function is a multi-objective optimization problem, and the optimal charging scheduling strategy is obtained by solving the objective function using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.

[0049] The method provided in this embodiment can be applied to a terminal device, on which a heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning is run. For example, a personal computer, tablet computer, or information processing center can all be considered a terminal device. This terminal device can also be referred to as a user equipment, and also includes a cloud server. The terminal device connects to the cloud server via wireless communication. This wireless communication method includes, but is not limited to, Bluetooth, Wi-Fi, ZigBee, GPRS, 3G, 4G, 5G, and WiMAX.

[0050] The aforementioned cloud server, acting as an information hub, receives, processes, and stores user information. Users send information retrieval commands to the cloud server via the aforementioned terminal devices. Upon receiving the command, the cloud server sends the corresponding information to the terminal devices. The terminal devices then receive the information for the users to view.

[0051] Figure 1 This is a flowchart illustrating the heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0052] S101, obtain the energy change results of each UAV in any time slot, as well as the transmission parameters for communication services between ground equipment and each UAV or unmanned vehicle.

[0053] Specifically, before performing step S101, a system model of UAVs, UGVs and ground equipment is established, and the sets of UAV, UGV and ground equipment quantities are determined.

[0054] like Figure 2 As shown, in the system model composed of UAVs, UGVs, and ground equipment, the UAV set is defined as follows: .

[0055] Ground power equipment collection .

[0056] UGV collection is Data tasks from ground equipment can be offloaded to UAVs or UGVs for processing.

[0057] To ensure network coverage performance, the system adopts a discrete time slot model, with the time slot set as follows: Each time slot has a length of t. A Cartesian three-dimensional coordinate system is used to construct the set of spatial coordinates. UAV Location in time slot The coordinates are The coordinates of UGV are The coordinates of the ground equipment are fixed as follows: UAV Flying at a constant altitude H and speed V, further assuming the UAV... The maximum elevation angle is That is, the maximum horizontal coverage radius of the signal service is All UAVs and UGVs must meet the following requirements. and The movement constraints, and This represents the maximum side length of the rectangular region. To avoid collisions, a minimum safe distance constraint is introduced between the UAV and UGV groups. and ,Right now: ;

[0058] In the communication network architecture, UGV In time slot t, it can be used as a mobile charging station for UAVs. Provides charging services and serves as a temporary mobile base station for ground equipment. Provide communication services; UAV The executable hovering service in time slot t is for ground equipment. Provides communication coverage or allows you to visit a mobile charging station. UGV Perform charging replenishment. Define charging state variables. If and only if UAV In time slot t for UGV During charging, ,otherwise .

[0059] By UAV UGV With ground equipment The three-dimensional coordinates yielded the horizontal coordinates as follows: , and UAV With UGV Together they form a correlation model, meaning that within time slot t, one UGV serves only one UAV, and the UAV... Within the maximum horizontal coverage radius, there must be at least one UGV to guarantee charging service, that is: ,in, For UGV With UAV Horizontal distance.

[0060] Specifically, definition For a binary variable, when When indicating UAV Ground equipment within time slot t When providing communication coverage services, otherwise ;definition For a binary variable, when When indicating UAV During time slot t, charging is required but ground equipment cannot be accessed. Provides communication services, UGV Provide them with temporary communication services, otherwise .

[0061] Secondly, the calculation process of transmission parameters for communication services between ground equipment and drones or unmanned vehicles is as follows: First, obtain the real-time location information of the drone, unmanned vehicle, and ground equipment, as well as the channel parameters for transmitting data from the drone and unmanned vehicle to the ground equipment; then, calculate the transmission parameters for communication services between ground equipment and drones and unmanned vehicles based on the real-time location information and channel parameters.

[0062] First, the calculation process for the transmission parameters of the communication service from ground equipment to UAV is as follows:

[0063] Assuming UAV With ground equipment The communication time and data volume of the downlink are negligible (primarily considering the uplink). A free-space path loss model is adopted, and the following assumptions are made regarding UAVs. With ground equipment The line-of-sight link dominates the communication process. UAV With ground equipment Channel gain in time slot t for:

[0064] ,in, This represents the channel gain coefficient of the ground-to-air link.

[0065] Assuming the downlink transmit power p of all UAVs is constant, then the ground equipment The received signal-to-noise ratio is:

[0066] ,in, This represents the power of additive white Gaussian noise. This indicates interference caused by other UAVs. Ground equipment. The achievable data rate in time slot t is:

[0067] ,in, For the bandwidth of the transmission link, To describe UAV With ground equipment A binary variable relating the relationships: assuming that only one user can be served per time slot t. Indicates when UAV For ground equipment When providing services, otherwise .

[0068] The calculation process for the transmission parameters of the communication service from ground equipment to UGV is as follows:

[0069] When ground equipment Offload the data task to UGV When the channel gain is such that:

[0070] ,in, This represents the channel gain coefficient from the ground equipment to the UGV.

[0071] Assuming the transmit power p of all UGVs is constant, then the ground equipment The received signal-to-noise ratio is:

[0072] ,in, This represents the power of additive white Gaussian noise. This indicates interference caused by other UGVs. Ground equipment. The achievable data rate in time slot t is:

[0073] ,in, This refers to the bandwidth of the transmission link. To describe UGV With ground equipment A binary variable relating the relationships: assuming that only one user can be served per time slot t. Indicates when UGV For ground equipment When providing services, otherwise .

[0074] For unmanned aerial vehicles (UAVs), their main energy consumption includes communication energy consumption and flight energy consumption. Since communication energy consumption is much lower than flight energy consumption, the power consumption of communication components is relatively low. Fixed. UAV The energy consumption during constant speed flight is as follows:

[0075] ,in, , and UAV correlation constant, They represent UAV The thrust of flight. Therefore, UAV The energy consumption at time t can be expressed as: Assume the UAV is equipped with an antenna for receiving energy, while the UGV is equipped with a transmitting antenna. When the UGV approaches the UAV, the wireless charging device activates, and the UGV transfers beamforming energy to the UAV. For UGVs at mobile charging stations... UAV The amount of energy replenished during duration t can be characterized as:

[0076] ,in, For energy conversion efficiency, For the duration of the time slot, denoted as UGV's energy emission power, and h as the charging height.

[0077] In any time slot t, each UAV The energy state is updated as follows:

[0078] ,in, This indicates the maximum energy capacity of each UAV.

[0079] In the current communication network architecture, UGVs, as mobile charging stations, are considered ideal devices that do not take into account energy changes and are only used as temporary communication service devices that may be used during mobile charging stations and drone charging.

[0080] S102, based on the energy change results of each drone in any time slot, calculate the first parameter characterizing the charging urgency of the drone and the second parameter characterizing the health status of the drone communication service.

[0081] Specifically, based on the content described in step S101 above, a first parameter characterizing the charging urgency of the drone is calculated according to the energy change results of the drone in any time slot. Specifically, the minimum and maximum reserve energy of each drone are obtained; and the first parameter characterizing the charging urgency of the drone is calculated based on the energy change results, minimum and maximum reserve energy of each drone in any time slot.

[0082] It is understandable that the energy capacity of the batteries installed in drones is common knowledge, so the rated parameters of the drone can be obtained directly.

[0083] The first parameter is calculated based on the energy change, minimum reserve energy, and maximum reserve energy of each UAV in any time slot. The calculation method is as follows: ,in, It is the minimum reserve energy required to extend battery life.

[0084] Based on the energy change results of the UAV in any time slot, a second parameter characterizing the health status of the UAV communication service is calculated. Specifically, based on the relationship between the energy change results of each UAV in any time slot and the minimum reserve energy, a first binary variable characterizing the health status of the UAV communication service is determined. The first binary variable of each UAV is summed and then divided by the total number of UAVs to obtain the second parameter characterizing the health status of the UAV communication service.

[0085] The network health index is defined as the ratio of the number of UAVs with energy above a minimum energy threshold to the total number of UAVs, representing the network's energy state. Let... As a binary variable, when UAV energy satisfy hour, ,otherwise, .

[0086] The second parameter for the health of the standard unmanned aerial vehicle (UAV) communication service network within time slot t. The formula for calculation is: ,in, Indicates the total number of UAVs. When When the network falls below a certain threshold, it is considered to be in failure.

[0087] S103, calculate the third parameter characterizing the performance of the communication service and the fourth parameter characterizing the fair allocation of resources based on the transmission parameters.

[0088] Specifically, a third parameter characterizing the communication service performance is calculated based on the transmission parameters. This involves: First, determining a second binary variable representing the communication service provided by each UAV to ground equipment in any time slot, and a third binary variable representing the communication service provided by each unmanned vehicle to ground equipment in any time slot; wherein the states of the second and third binary variables are mutually exclusive; second, based on the second and third binary variables, and combined with the transmission parameters of the communication service between the ground equipment and each UAV or unmanned vehicle, calculating the cumulative throughput of each ground device; finally, taking the minimum value of the cumulative throughput as the third parameter characterizing the communication service performance.

[0089] The third parameter represents the minimum cumulative throughput of the mission across all ground equipment. This parameter reflects the lowest rate experienced by any ground equipment in the communication network, characterizing the performance aimed at maximizing the achievable data rate under worst-case conditions. This third parameter is used... It is represented, and its definition is as follows: .

[0090] The fourth parameter characterizing the fair allocation of resources is calculated based on the transmission parameters. Specifically, it is calculated by combining the second binary variable, the third binary variable, and the transmission parameters of the communication service between the ground equipment and each drone or unmanned vehicle with the Jain fairness index.

[0091] To analyze the fair allocation of resources, the Jain Fairness Index (JFI) is used, which is defined as follows:

[0092] , where K represents the set of ground devices in the network. The value ranges from 0 to 1, where 0 indicates extremely unfair resource allocation and 1 indicates completely fair resource allocation.

[0093] S104, establish an objective function that maximizes the minimum cumulative throughput of communication services between UAVs or unmanned vehicles and ground equipment, and determine the constraints of the objective function by combining the first parameter, the second parameter, the third parameter and the fourth parameter.

[0094] Specifically, based on the four parameters described above and the existing communication network architecture, the goal is to maximize the minimum cumulative throughput for all users while ensuring network health (i.e., a minimum number of available UAVs). Simultaneously optimize the charging scheduling strategy for UAVs, namely... The designed optimization objective function is as follows: ;

[0095] The constraints are as follows:

[0096] .

[0097] Among them, constraints The binary characteristic variables for charging decision and communication range decision are specified; constraints are defined. and Ensure that the number of healthy UAVs in the network can be maintained at a defined threshold. The above are the constraints. Limiting the emergency charging level of the UAV to ensure its battery health; constraints JFI must be higher than a preset threshold. Constraints and Ensure no collisions occur between UAV or UGV groups; constraints Ensure that at least one UGV is available within the maximum horizontal coverage radius of the UAV to guarantee its charging service; constraints Ensure that each UAV can be charged by at most one UGV in time slot t; finally, constraints. and Ensure that a user equipment is serviced by only one UAV or UGV.

[0098] S105. Under the constraints, the MADDPG algorithm is used to solve the objective function and obtain the optimal strategy for the coordinated charging scheduling of UAVs and unmanned vehicles.

[0099] After establishing the objective function and relevant constraints based on step S104 above, given that charging scheduling is a time-series decision problem, and the arrival times of UAVs and UGVs, as well as the initial positions of ground equipment, are random, this problem can be modeled as a Multi-Agent Markov Decision Process (MMDP). Then, MADDPG is used to obtain the charging scheduling strategy for UAVs and to maximize the minimum cumulative throughput for all users.

[0100] Establishing the six-tuple in MMDP Elements, among which, Represents a set of intelligent agents. and These represent the state space and the behavior space, respectively. Represents the state transition probability. For the reward function, This represents the discount factor.

[0101] Collection of intelligent agents UAVs and UGVs serve as intelligent agents for learning charging scheduling schemes and optimizing system metrics, among which... ,

[0102] state space This reflects the environmental conditions under which the agent operates in time slot t. Therefore, for a UAV, the state space... The state space of a UGV includes its location, energy level, and throughput. Includes the location information and throughput of mobile charging stations, namely:

[0103] ;and

[0104] .

[0105] Action space Dispatching UAV flights to serve ground equipment And to provide charging services to UGVs; while UGVs move to areas covered by UAVs to wait for charging services and temporarily serve ground equipment. Provides communication services. That is, the action space is represented as:

[0106] ;

[0107] .

[0108] reward function The agents should cooperate to minimize the objective function while satisfying the constraints. The reward for time slot t is defined as the sum of the minimum throughput of all ground devices up to time t.

[0109] .

[0110] The MADDPG algorithm is a reinforcement learning algorithm for multi-agent systems, whose core architecture follows a "centralized training, distributed execution" paradigm. In this paradigm, each agent (UAV and UGV) is equipped with its own policy network (Actor) and value evaluation network (Critic). The policy network generates control actions based on its local observations, while the value network evaluates the value of selected actions using global state information—the joint state and actions of all agents—during training. Notably, during the deployment phase, each agent makes independent decisions based solely on its local observations and its own policy network, without needing real-time communication or information exchange with other agents. The solution process of the MADDPG algorithm is as follows: Figure 3 As shown below:

[0111] First, each agent initializes its Actor network and Critic network. The Actor network parameters are denoted as follows: The parameters of the Critic network are: Simultaneously, initialize the target Actor network. and target Critic network Its weights are the same as the main network. Create a buffer. tuples containing the interactive experience of the stored intelligent agent ,in, This is the global state. For joint operations, As a reward, each agent is rewarded based on its current local state. Select action: ,in, It is a deterministic strategy for generating Actor networks.

[0112] Next, we will carry out joint operations. Afterwards, the environment returns a reward. and the next state . Empirical tuples Store in buffer .from Random sampling in small batches Sample Calculate the target Q value: ,in, As a discount factor, For the target Actor network.

[0113] Then, the Critic loss function is obtained based on the objective function: .

[0114] Then, the Critic parameters are updated using gradient descent: .

[0115] Furthermore, the policy gradient that maximizes the expected cumulative reward is:

[0116] .

[0117] Update Actor parameters: .

[0118] Finally, based on the previous neural network parameters of the target network and the main network, a soft update strategy (update rate) is adopted. ):

[0119] .

[0120] This invention also provides a heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning, the system comprising:

[0121] The acquisition module is used to acquire the energy change results of each UAV in any time slot, as well as the transmission parameters for communication services between ground equipment and each UAV or unmanned vehicle.

[0122] The parameter calculation module is used to calculate a first parameter characterizing the charging urgency of the drone and a second parameter characterizing the health status of the drone's communication service based on the energy change results of each drone in any time slot; and to calculate a third parameter characterizing the performance of the communication service and a fourth parameter characterizing the fair allocation of resources based on the transmission parameters.

[0123] The function construction module is used to establish an objective function that maximizes the minimum cumulative throughput of communication services between drones or unmanned vehicles and ground equipment, and to determine the constraints of the objective function by combining the first, second, third and fourth parameters;

[0124] The strategy solving module is used to solve the objective function under constraints using the MADDPG algorithm to obtain the optimal strategy for collaborative charging scheduling between UAVs and unmanned vehicles.

[0125] This application provides an embodiment of a heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning, which is similar to the above-mentioned... Figure 1The heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning shown is a technical solution under the same inventive concept. Through the detailed description of the heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning provided in the above embodiments, those skilled in the art can clearly understand the implementation process of a heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning in this embodiment. Therefore, for the sake of brevity, it will not be described again here.

[0126] Accordingly, the heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning provided by this invention introduces unmanned vehicles to construct a joint communication network architecture for UAVs and UGVs. Combining their respective energy change results and communication service transmission parameters, four parameters are calculated for evaluating and optimizing the communication network architecture: a first parameter for charging urgency, a second parameter for network health, a third parameter for global throughput, and a fourth parameter for fair resource allocation. Based on these four parameters as constraints of the objective function that maximizes the minimum cumulative throughput of all ground devices, the MADDPG algorithm is used to solve the objective function, thereby obtaining the optimal strategy for collaborative charging scheduling of UAVs and unmanned vehicles. This achieves intelligent scheduling and energy replenishment of UAVs, effectively alleviating the battery life bottleneck of UAVs and enhancing the emergency response capability of the power grid.

[0127] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous node charging scheduling method based on multi-agent deep reinforcement learning, characterized in that, The method comprises: obtaining the energy change result of each unmanned aerial vehicle at any time slot, and the transmission parameter of the communication service between the ground equipment and each unmanned aerial vehicle or unmanned vehicle; calculating a first parameter representing the charging emergency degree of the unmanned aerial vehicle and a second parameter representing the health state of the communication service of the unmanned aerial vehicle according to the energy change result of each unmanned aerial vehicle at any time slot; calculating a third parameter representing the performance of the communication service and a fourth parameter representing the fair allocation of resources according to the transmission parameter; establishing a target function of maximizing the minimum cumulative throughput of the communication service between the unmanned aerial vehicle or unmanned vehicle and the ground equipment, and determining the constraint condition of the target function in combination with the first parameter, the second parameter, the third parameter and the fourth parameter; solving the target function under the constraint condition by using the MADDPG algorithm to obtain the optimal strategy of the cooperative charging scheduling of the unmanned aerial vehicle and the unmanned vehicle.

2. The method of claim 1, wherein, The transmission parameter of the communication service between the ground equipment and the unmanned aerial vehicle or unmanned vehicle comprises: obtaining the real-time position information of the unmanned aerial vehicle, the unmanned vehicle and the ground equipment respectively, and the channel parameter of the transmission of data from the unmanned aerial vehicle to the ground equipment; calculating the transmission parameter of the communication service between the ground equipment and the unmanned aerial vehicle and the unmanned vehicle according to the real-time position information and the channel parameter.

3. The method of claim 1, wherein, According to the energy change result of the unmanned aerial vehicle at any time slot, the first parameter representing the charging emergency degree of the unmanned aerial vehicle is calculated, specifically: obtaining the minimum reserve energy and the maximum reserve energy of each unmanned aerial vehicle; calculating the first parameter representing the charging emergency degree of the unmanned aerial vehicle according to the energy change result of each unmanned aerial vehicle at any time slot, the minimum reserve energy and the maximum reserve energy.

4. The method of claim 3, wherein, According to the energy change result of the unmanned aerial vehicle at any time slot, the second parameter representing the health state of the communication service of the unmanned aerial vehicle is calculated, specifically: determining the first binary variable of the communication service health state of each unmanned aerial vehicle according to the size relationship between the energy change result of each unmanned aerial vehicle at any time slot and the minimum reserve energy; summing up the first binary variable of each unmanned aerial vehicle and then dividing by the total number of the unmanned aerial vehicles to obtain the second parameter representing the health state of the communication service of the unmanned aerial vehicle.

5. The method of claim 1, wherein, According to the transmission parameter, the third parameter representing the performance of the communication service is calculated, specifically: determining the second binary variable of each unmanned aerial vehicle providing the communication service for the ground equipment at any time slot and the third binary variable of each unmanned vehicle providing the communication service for the ground equipment at any time slot; wherein the states of the second binary variable and the third binary variable are mutually exclusive; based on the second binary variable and the third binary variable, and in combination with the transmission parameter of the communication service between the ground equipment and each unmanned aerial vehicle or unmanned vehicle, calculating the cumulative throughput of each ground equipment; taking the minimum value of the cumulative throughput as the third parameter representing the performance of the communication service.

6. The method of claim 5, wherein, According to the transmission parameter, the fourth parameter representing the fair allocation of resources is calculated, specifically: based on the second binary variable, the third binary variable and the transmission parameter of the communication service between the ground equipment and each unmanned aerial vehicle or unmanned vehicle, and in combination with the Jain fairness index, the fourth parameter representing the fair allocation of resources is calculated.

7. The method of claim 1, wherein, The constraint condition further comprises that each UAV can only be charged by one unmanned vehicle in any time slot, each ground device can only be served by one UAV or one unmanned vehicle in communication, and at least one unmanned vehicle in the maximum horizontal coverage radius of the UAV is used to perform the charging service.

8. The method of claim 1, wherein, Before the MADDPG algorithm is used to solve the objective function and obtain the optimal strategy of the UAV and the unmanned vehicle charging scheduling, the method further comprises: determining a state space, an action space and a reward function of the MADDPG algorithm.

9. The method of claim 8, wherein, The state space comprises a state of the UAV and a state of the unmanned vehicle, wherein the state of the UAV comprises a position, an energy level and a throughput of the UAV, and the state of the unmanned vehicle comprises position information and a throughput of the mobile charging station. The action space comprises an action of the UAV and an action of the unmanned vehicle, wherein the action of the UAV is to fly to the ground device for communication service and fly to the unmanned vehicle for charging service, and the action of the unmanned vehicle is to move to the coverage area of the UAV to wait for charging service and provide communication service for the ground device. The reward function is the cumulative sum of the minimum throughput of all ground devices up to time t.

10. A heterogeneous node charging scheduling system based on multi-agent deep reinforcement learning, characterized in that, The system comprises: An acquisition module is configured to acquire energy change results of each UAV in any time slot and transmission parameters of communication service of the ground device with each UAV or unmanned vehicle; A parameter calculation module is configured to calculate a first parameter representing the charging urgency of the UAV and a second parameter representing the health status of the communication service of the UAV according to the energy change results of each UAV in any time slot; A third parameter representing the performance of the communication service and a fourth parameter representing the fair allocation of resources are calculated according to the transmission parameters; A function construction module is configured to establish an objective function maximizing the minimum cumulative throughput of the communication service of the UAV or the unmanned vehicle with the ground device, and determine a constraint condition of the objective function in combination with the first parameter, the second parameter, the third parameter and the fourth parameter; A strategy solving module is configured to use the MADDPG algorithm to solve the objective function under the constraint condition, and obtain the optimal strategy of the UAV and the unmanned vehicle cooperative charging scheduling.

Images