A multi-unmanned aerial vehicle power grid cache cooperative scheduling method based on federal deep reinforcement learning

Abstract

This invention relates to the field of smart grid processing technology, specifically to a multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning, belonging to the field of smart grid processing technology. For scenarios where ground communication is disrupted during disasters, UAVs, with their advantages of flexibility, mobility, and convenient deployment, have become an effective solution. However, UAVs suffer from limited cache capacity and problems such as missing key information and mismatched coverage resources due to dynamically changing user request distribution. Therefore, this invention proposes a hierarchical collaborative scheduling scheme. This scheme constructs a collaborative system between a high-altitude platform and multiple UAVs, with maximizing cache hit rate as the optimization objective. It adopts a hierarchical reinforcement learning architecture, performing centralized global cache decision-making based on a deep Q-network at the high-altitude platform end, and distributed real-time trajectory planning based on a dual-delay deep deterministic policy gradient algorithm at each UAV end. A federated learning mechanism is introduced to aggregate UAV trajectory model parameters, achieving global optimization while protecting data privacy. This invention effectively reduces the computational load on UAVs and significantly improves cache hit rate and emergency communication service quality.

Description

Technical Field

[0001] This invention relates to the field of smart grid processing technology, specifically to a multi-UAV grid cache collaborative scheduling method based on federated deep reinforcement learning. Background Technology

[0002] During disasters such as earthquakes and floods, fixed communication infrastructure such as ground base stations, fiber optic links, and substation communication modules are directly damaged, causing the previously stable "user-server" communication topology to collapse instantly, resulting in the failure of power grid emergency communications. To quickly restore communication links in disaster-stricken areas, drones, with their flexibility, mobility, and ease of deployment, have become a core technological solution for addressing this scenario. However, in practical applications, drone networking faces some key challenges:

[0003] In the event of a ground communication disruption, drones rely on onboard caches to pre-cache critical emergency data, enabling them to directly provide services to rescue personnel when backhaul links are unavailable. However, limited by the physical size of onboard equipment, drone cache capacity is extremely limited. Furthermore, power grid emergency data exhibits significant dynamic priority and timeliness characteristics—the data types required evolve rapidly at different stages. For example, in the early stages of a disaster, the focus is on quickly assessing the extent of damage and locating severely affected areas. During the repair phase, the emphasis shifts to fault isolation, controlling the extent of damage, and restoring critical loads, while historical load curves and equipment maintenance records are considered low-priority data. Expired data quickly becomes invalid. Based on the power grid topology generated at the time, if the network structure has changed due to repairs, old solutions are not only useless but may also mislead operations. This makes it difficult for limited cache space to match dynamically changing data needs, easily leading to missing critical information or redundant cached content.

[0004] The communication and service topology of drone networks exhibits significant dynamism, and the distribution of users in disaster-stricken areas constantly changes as people are relocated and rescue efforts progress. If drones fly along fixed trajectories, resource misallocation is highly likely to occur—drones continue patrolling along predetermined routes after users in the original service area are evacuated, resulting in idle resources; while newly formed rescue hotspots lack drone coverage and cannot obtain necessary communication support. Simultaneously, covering large-scale disaster-stricken power grids requires the coordinated operation of multiple drones. In fixed-trajectory mode, multiple drones are prone to overlapping coverage in the same area, leading to resource waste, or creating coverage gaps at area junctions, resulting in service loss. More importantly, in dynamic communication topologies, drones need to acquire dynamic status information such as user distribution and link quality in real time to adjust their trajectories. However, the surge in the proportion of non-line-of-sight links and increased transmission latency in disaster scenarios further exacerbates the difficulty of trajectory planning. Summary of the Invention

[0005] To address the above problems, this invention provides a multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning, comprising the following steps:

[0006] S1. Construct a multi-UAV air-ground-space emergency communication system model, which includes a high-altitude platform and an UAV layer;

[0007] S2. Construct a problem model with maximizing cache hit rate as the system optimization objective;

[0008] S3. At the high-altitude platform end, a global caching decision is made based on a deep Q-network;

[0009] S4. At each UAV terminal, after receiving the cached decision instructions generated by the high-altitude platform terminal, each UAV performs distributed trajectory planning based on the dual-delay depth deterministic strategy gradient algorithm.

[0010] S5. Introducing a federated learning framework, each UAV trains its trajectory planning model locally, uploads the model parameters to the high-altitude platform, aggregates the model parameters of all UAVs, and distributes them to achieve the collaborative evolution of trajectory strategies of multiple UAVs.

[0011] The beneficial effects of this invention are:

[0012] This invention moves the computationally complex cache decision-making task (DQN), which requires a global perspective, to a resource-rich high-altitude platform (HAP), while retaining the real-time-critical trajectory planning task (TD3) on the UAV, thus reducing the onboard computing burden of a single UAV. The UAV's local cache state vector is used as the core state input to the TD trajectory planning module, enabling the UAV to perceive the value of its carried data and intelligently fly towards areas with high demand for its cached content. Reward functions are set for both the HAP and UAV sides, with cache hits as the core, ensuring that their optimization goals are fundamentally aligned and jointly driving improvements in cache hit rate and service efficiency. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the multi-UAV power grid emergency communication system model of the present invention;

[0014] Figure 2 This invention provides a dual-module reinforcement learning framework.

[0015] Figure 3 This is a framework diagram of the PER+Noisy DQN algorithm of this invention;

[0016] Figure 4 This is a diagram of the TD3 algorithm framework of the present invention. Detailed Implementation

[0017] This invention provides a multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning, comprising the following steps:

[0018] S1. Construct a multi-UAV air-ground-space emergency communication system model, which includes a high-altitude platform and an UAV layer;

[0019] Specifically, the system model includes a high-altitude platform (HAP) and N unmanned aerial vehicles (UAVs); such as Figure 2 As shown, the high-altitude platform, acting as the global cache decision center, federated learning parameter server, and system coordination node, possesses centralized computing capabilities. It is used to run a global cache decision model based on deep reinforcement learning and is responsible for the federated aggregation and distribution of model parameters uploaded by UAVs. Each UAV is equipped with a mobile edge computing server and a local cache unit with limited capacity. It receives and executes cache instructions from the high-altitude platform and performs real-time trajectory planning based on the local reinforcement learning model to receive power grid task requests initiated by ground users.

[0020] S2. Construct a problem model with maximizing cache hit rate as the system optimization objective;

[0021] Based on the above system model, to clarify the optimization direction, this step defines the cache hit rate η as...

[0022]

[0023] Where T represents the total number of time slots, and N represents the number of drones. This represents the set of user requests within the communication coverage area of ​​UAV i in time slot t. This is an indicator function that is 1 when user u's request is hit by drone i's local cache, and 0 otherwise.

[0024] This metric directly measures the system's service efficiency. With the goal of maximizing cache hit rate, the problem model is constructed as follows:

[0025]

[0026] This represents the caching strategy for the i-th drone, which determines how to replace low-value data within a limited cache capacity. This represents the trajectory planning strategy for the i-th drone, which determines its flight path to maximize service success rate; Let C1 represent the set of cache and trajectory strategies for all UAVs, serving as the decision variable for this optimization problem; C1 represents the cache capacity constraint for the UAVs. This represents the set of locally cached content for the i-th UAV in time slot t, which includes cached power grid task data types. C represents the number of cache slots occupied by the i-th drone in time slot t; C represents the maximum cache capacity of each drone; C2 and C3 represent the speed constraints of the drone. Let represent the three-dimensional acceleration vector of the i-th UAV in time slot t. Let C1 represent the three-dimensional velocity vector of the i-th UAV in time slot t; C4 represents the service feasibility constraint of the UAV. This indicates whether user u is within the communication coverage area of ​​the i-th UAV in time slot t. It is a binary variable, where the horizontal distance does not exceed the communication radius. The value is 1 if the condition is met, and 0 otherwise. C5 indicates whether the data required by user u exists in the local cache of the i-th drone. It is a binary variable, with a value of 1 when the data type is cached on the drone, and 0 otherwise. C5, C6, and C7 represent the spatial position constraints of the drone. , Let represent the horizontal coordinates of the i-th UAV in time slot t, respectively. C represents the flight altitude of the i-th UAV in time slot t; C8 represents the energy constraint of the UAV. This represents the energy of the i-th drone in time slot t.

[0027] S3. At the high-altitude platform end, a global caching decision is made based on a deep Q-network;

[0028] To solve the aforementioned complex problem, this invention employs a hierarchical reinforcement learning architecture. First, a global cache decision model is deployed on a resource-rich high-altitude platform. Specifically, the global cache decision module deployed on the high-altitude platform uses a deep Q-network as the decision model, which makes centralized decisions on cached content based on the aggregated state information of each UAV and issues instructions to the UAVs for execution. To improve model performance, the network adopts a mechanism of prioritizing experience replay and noise exploration.

[0029] Define the state space of DQN Define the cache decision-related state of each drone in time slot t as follows:

[0030]

[0031] This represents the cache details vector, which records the type, cache duration, last access time, and currently calculated priority value of each data item in the cache. It represents a detailed list of user requests within the coverage area, including the data type, generation time, and priority weight of each request; This indicates the emergency phase.

[0032] Define cache action space Where k is the total number of cache slots; Action This indicates a caching decision, when an action... This indicates that no cache replacement will be performed for the k-th cache slot. This indicates that the data in the k-th cache slot will be replaced.

[0033] The reward function r(t) is expressed as

[0034]

[0035] in, The priority weight of the business to which data item d belongs is a static weight that is pre-set based on the importance of the business to which the power grid data belongs; Indicates the generation time of data item d; This is the time decay coefficient, used to reflect the decrease in the value of cached data over time.

[0036] like Figure 3 As shown, the caching strategy module of this invention adopts a reinforcement learning framework that integrates priority experience replay and noisy exploration deep Q-network. Specifically, the Q-Network is used to evaluate the value of actions in the current state; NoisyNet adds Gaussian noise to the weights of the Q-Network to achieve continuous exploration without ε-greedy, improving sample utilization efficiency in sparse reward scenarios; the Target Q-Network is used to stabilize the training process, and its parameters are synchronized from the main network through soft updates; the ExperienceReplay buffer stores historical experiences (S, A, R, S') and prioritizes sampling key samples based on TD error to enhance learning efficiency; all experiences are generated through environmental interaction and fed back to the buffer to complete closed-loop training.

[0037] The caching strategy module uses a Noisy DQN network, whose network parameters are derived from learnable mean values. , with standard deviation , It is constructed and superimposed with independent Gaussian noise during each forward propagation. and Specifically, the weights W and biases b in the network are dynamically generated as follows:

[0038]

[0039]

[0040] in This indicates element-wise multiplication.

[0041] Let the hidden state of the l-th layer be Then its forward propagation process is as follows:

[0042]

[0043] The final Q-value is calculated from the forward computation function of DQN. Output:

[0044]

[0045] in Indicates the state from input The nonlinear mapping relationship to the Q value This is the parameter set for the noisy network, containing all parameters.

[0046] Although Noisy DQN enhances its exploration capabilities through parameter noise, it still suffers from uneven sample utilization during training: some key experiences may be overlooked. To address this, this invention further introduces a Priority Experience Replay (PER) mechanism to enable repeated learning of important experiences, thereby improving the model's convergence speed and stability.

[0047] Specifically, each experience Stored in the experience replay buffer and assigned different priorities based on the corresponding TD error size.

[0048] Let the TD error of the j-th experience be...

[0049]

[0050] in This indicates the current state of the Q network. The output value under action a, and its network parameters are: ; Indicates the target Q-network in the next state Take action below The output value, whose network parameters are: ; This is the discount factor.

[0051] The priority of this experience is defined as follows:

[0052]

[0053] in Control priority intensity, It is a small constant to prevent the priority from being zero.

[0054] To mitigate the distribution bias caused by priority sampling, importance sampling weights are introduced:

[0055]

[0056] Where N is the size of the buffer, To control the intensity of the bias correction, the hyperparameter is gradually increased to 1 during training.

[0057] Ultimately, network parameters The update is performed by minimizing the following weighted loss function:

[0058]

[0059] Where, N b The number of sampling experiences selected from the experience pool. This is the target value for the Q-network.

[0060] During the training phase, since the Noisy DQN network itself contains parameter noise, there is no need to introduce an additional ε-greedy exploration strategy; during the inference phase, a deterministic strategy is adopted.

[0061]

[0062] To achieve optimal cache replacement decisions.

[0063] The high-altitude platform will generate a cache decision vector The command is distributed to each corresponding UAV via the downlink. After receiving the command, each UAV performs a specified cache replacement operation to update its local cache contents. The updated cache state vector will be used as part of the input for its local TD3 trajectory planning module to make subsequent flight decisions.

[0064] S4. At each UAV terminal, distributed trajectory planning is performed based on the dual-delay depth deterministic policy gradient algorithm;

[0065] After receiving the cache update instruction and refreshing the local cache, each UAV enters the distributed trajectory planning phase. Specifically, the distributed trajectory planning module deployed on each UAV adopts the dual-delay deep deterministic strategy gradient algorithm (TD3).

[0066] The position of the i-th drone in time slot t can be represented as: ,in , Let represent the horizontal coordinates of the i-th UAV in time slot t, respectively. Let represent the flight altitude of the i-th UAV in time slot t.

[0067] Therefore, in time slots The spatial location of a drone can be represented as:

[0068]

[0069] This represents the velocity of the i-th drone in time slot t. This represents the velocity of the i-th drone in time slot t. Indicates the time step.

[0070] The state space of the TD3 algorithm is the trajectory state of the i-th UAV in time slot t. Represented as

[0071]

[0072] Let represent the set of user requests within the communication coverage area of ​​the i-th drone in time slot t; This represents the local cache of the i-th drone in time slot t;

[0073] Define the trajectory action space of time slot t ,in This is represented as a three-dimensional acceleration command in time slot t;

[0074] TD3 reward function Represented as

[0075]

[0076] in, This represents the set of users successfully served by the drone in time slot t. , represents the priority weight of the data item d requested by user u, which is dynamically allocated based on the importance of the device and the urgency of the business; This indicates the weight of the current emergency phase, reflecting the differences in the importance of power grid operations at different times after the disaster.

[0077] like Figure 4 As shown, the trajectory planning module of this invention employs the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, and its core structure includes:

[0078] Policy Network (Actor): Based on the current state s, it outputs acceleration commands for the UAV in three-dimensional space to achieve continuous action decision-making;

[0079] Two value networks (Twin Critics): each evaluates the value of an action separately, and the Q-value is suppressed by taking the minimum value;

[0080] Target Value Network 1 and Target Value Network 2: These correspond to the target networks of the two Critics. The parameters are synchronized through soft updates to improve training stability.

[0081] Target policy network: The target network, corresponding to the policy network, is used to generate target actions and participates in the calculation of the target Q-value.

[0082] The network parameters of the two Critic networks are updated by minimizing the following loss function:

[0083]

[0084] Where, N b This represents the number of samples taken from the experience replay pool. This indicates that the i-th Critic network has certain network parameters. ,state ,action The output and target value are as follows. Defined as

[0085]

[0086] in, It is in state Execute action Immediate feedback received afterwards; Indicates the target Actor network parameters are In state The network executing the target policy is in state Down-output actions The output of .

[0087] When updating the Actor network every d steps, the goal is to maximize the expected reward.

[0088]

[0089] Indicates policy parameters How to adjust to improve performance is ultimately used for updates. The direction; The number of samples taken from the experience replay pool; Indicates the current state Down, in which direction of movement? It can make the Q value rise the fastest; Indicates when parameter When changes occur, output action How it changes.

[0090] S5. Introducing a federated learning framework, each UAV trains its trajectory planning model locally, uploads the model parameters to the high-altitude platform, aggregates the model parameters of all UAVs, and distributes them to achieve the collaborative evolution of trajectory strategies of multiple UAVs.

[0091] Specifically, after each K rounds of local training, a global aggregation is performed. Each UAV uploads the TD3 network parameters to the high-altitude server for aggregation to obtain the global model parameters:

[0092]

[0093] in, Weighting of drones; Let be the TD3 network parameters of the i-th UAV;

[0094] After aggregation, the high-altitude platform will allocate global model parameters. The data is distributed to each drone, which then uses it to replace the parameters of its local trajectory planning model as the initial model for the next training round. This enables the collaborative evolution of trajectory strategies across multiple drones while protecting the privacy of each drone's local observation data.

Claims

1. A multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning, characterized in that, Includes the following steps: S1. Construct a multi-UAV air-ground-space emergency communication system model, which includes a high-altitude platform and an UAV layer; S2. Construct a problem model with maximizing the system cache hit rate as the system optimization objective; S3. At the high-altitude platform end, a global caching decision is made based on a deep Q-network; S4. At each UAV terminal, after receiving the cached decision instructions generated by the high-altitude platform terminal, each UAV performs distributed trajectory planning based on the dual-delay depth deterministic strategy gradient algorithm. S5. Introducing a federated learning framework, each UAV trains its trajectory planning model locally, uploads the model parameters to the high-altitude platform, aggregates the model parameters of all UAVs, and distributes them to achieve the collaborative evolution of trajectory strategies of multiple UAVs.

2. The multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, The system model includes a High Altitude Platform (HAP) and N Unmanned Aerial Vehicles (UAVs). The HAP serves as the global cache decision center, federated learning parameter server, and system coordination node, possessing centralized computing capabilities to run a deep reinforcement learning-based global cache decision model and to perform federated aggregation and distribution of model parameters uploaded by the UAVs. Each UAV is equipped with a mobile edge computing server and a local cache unit with limited capacity, receiving and executing cache instructions from the HAP and performing real-time trajectory planning based on its local reinforcement learning model to receive power grid task requests initiated by ground users. The cache state vector output by the cache decision module serves as part of the state input to the trajectory planning module, achieving coupled optimization of the cache strategy and flight trajectory.

3. The multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, Based on the above system model, to clarify the optimization direction, this step defines the cache hit rate η as: ； Where T represents the total number of time slots, and N represents the number of drones. This represents the set of user requests within the communication coverage area of ​​UAV i in time slot t. This is an indicator function that is 1 when user u's request is hit by drone i's local cache, and 0 otherwise.

4. The multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, Step S3 aims to maximize cache hit rate, and the problem model is represented as follows: ； This represents the caching strategy for the i-th drone, which determines how to replace low-value data within a limited cache capacity. This represents the trajectory planning strategy for the i-th drone, which determines its flight path to maximize service success rate; Let C1 represent the set of cache and trajectory strategies for all UAVs, serving as the decision variable for this optimization problem; C1 represents the cache capacity constraint for the UAVs. This represents the set of locally cached content for the i-th UAV in time slot t, which includes cached power grid task data types. C represents the number of cache slots occupied by the i-th drone in time slot t; C represents the maximum cache capacity of each drone; C2 and C3 represent the speed constraints of the drone. Let represent the three-dimensional acceleration vector of the i-th UAV in time slot t. Let C1 represent the three-dimensional velocity vector of the i-th UAV in time slot t; C4 represents the service feasibility constraint of the UAV. This indicates whether user u is within the communication coverage area of ​​the i-th UAV in time slot t. It is a binary variable, where the horizontal distance does not exceed the communication radius. The value is 1 if it is true, and 0 otherwise. C5 indicates whether the data required by user u exists in the local cache of the i-th drone. It is a binary variable, with a value of 1 when the data type is cached on the drone, and 0 otherwise. C5, C6, and C7 represent the spatial position constraints of the drone. , Let represent the horizontal coordinates of the i-th UAV in time slot t, respectively. C represents the flight altitude of the i-th UAV in time slot t; C8 represents the energy constraint of the UAV. This represents the energy of the i-th drone in time slot t.

5. A multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, The global cache decision module deployed on the high-altitude platform uses a deep Q-network as the decision model. It makes centralized decisions on cached content based on the aggregated state information of each UAV and sends instructions to the UAVs for execution. To improve model performance, the network adopts a priority experience replay and noise exploration mechanism. Define state space The cache decision-related state of each drone in time slot t is defined as follows: ； This represents the cache details vector, which records the type, cache duration, last access time, and currently calculated priority value of each data item in the cache. It represents a detailed list of user requests within the coverage area, including the data type, generation time, and priority weight of each request; Indicates the emergency phase; Define cache action space Where k is the total number of cache slots; Action This indicates a caching decision, when an action... This indicates that no cache replacement will be performed for the k-th cache slot. This indicates that the data in the k-th cache slot will be replaced; The reward function r(t) is expressed as: ； in, The priority weight of the business to which data item d belongs is a static weight that is pre-set based on the importance of the business to which the power grid data belongs; Indicates the generation time of data item d; This is the time decay coefficient, used to reflect the decrease in the value of cached data over time.

6. The multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, The distributed trajectory planning module deployed on each UAV terminal uses the dual-delay deep deterministic strategy gradient algorithm (TD3) to define the trajectory state of the i-th UAV in time slot t. Represented as: ； Let represent the three-dimensional position coordinates of the i-th UAV in time slot t; Let represent the three-dimensional velocity vector of the i-th UAV in time slot t; Let i represent the set of user requests within the communication coverage area of ​​the i-th drone in time slot t; This represents the local cache of the i-th drone in time slot t; Define the trajectory action space of time slot t ,in This is represented as a three-dimensional acceleration command in time slot t; TD3 reward function Represented as: ； in, This represents the set of users successfully served by the drone in time slot t. , represents the priority weight of the data item d requested by user u, which is dynamically allocated based on the importance of the device and the urgency of the business; This indicates the weight of the current emergency phase, reflecting the differences in the importance of power grid operations at different times after the disaster.

7. A multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, Step S3 specifically includes: The caching strategy model uses a Noisy DQN network to calculate the Q-value, and its network parameters are derived from learnable mean values. , with standard deviation , It is constructed and superimposed with independent Gaussian noise during each forward propagation. and Specifically, the weights W and biases b in the network are dynamically generated as follows: ； ； in This represents element-wise multiplication; Let the hidden state of the l-th layer be Then its forward propagation process is as follows: ； The final Q-value is calculated from the forward computation function of DQN. Output: ； in Indicates the state from input The nonlinear mapping relationship to the Q value This is the parameter set for the noisy network, containing all parameters. Although Noisy DQN enhances exploration capabilities through parameter noise, it still suffers from uneven sample utilization during training: some key experiences (such as high-reward or high-error samples) may be ignored. To address this, this invention further introduces a Priority Experience Replay (PER) mechanism to enable repeated learning of important experiences, thereby improving model convergence speed and stability. Specifically, each experience Stored in the experience replay buffer and assigned different priorities according to the size of the corresponding TD error; Let the TD error of the j-th experience be: ； in, The immediate reward in the j-th experience; This indicates the current state of the Q network. The output value under action a, and its network parameters are: ; Indicates the target Q-network in the next state Take action below The output value, whose network parameters are: ; Discount factor; The priority of this experience is defined as follows: ； in Control priority intensity, It should be a small constant to prevent the priority from being zero; To mitigate the distribution bias caused by priority sampling, importance sampling weights are introduced: ； Where N is the size of the buffer, The hyperparameter for controlling the strength of bias correction is gradually increased to 1 during training; Finally, the Q network parameters The update is performed by minimizing the following weighted loss function: ； Where, N b The number of sampling experiences selected from the experience pool. The target value for the Q network; During the training phase, since the Noisy DQN network itself contains parameter noise, no additional random exploration strategy is needed; during the inference phase, a deterministic strategy is adopted. ； To achieve optimal cache replacement decisions; The high-altitude platform will generate a cache decision vector The command is distributed to each corresponding UAV via the downlink. After receiving the command, each UAV performs a specified cache replacement operation to update its local cache contents. The updated cache state vector will be used as part of the input for its local TD3 trajectory planning module to make subsequent flight decisions.

8. A multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, Step S4 specifically includes: TD3 employs an Actor-Critic structure. It has two Critic networks with identical structures but different network parameters. The network parameters of the two Critic networks... It updates by minimizing the following loss function: ； Where, N b The number of samples taken in the experience replay pool This indicates that the i-th Critic network has certain network parameters. ,state ,action The output and target value are as follows. Defined as: ； in, It is in state Execute action The instant reward obtained afterward; The target Critic network parameters are In state Under these conditions, the network executing the target policy is in state Down-output actions The output; When updating the Actor network every d steps, the goal is to maximize the expected return: ； Indicates policy parameters How to adjust to improve performance is ultimately used for updates. The direction; The number of samples taken from the experience replay pool; Indicates the current state Down, in which direction of movement? It can make the Q value rise the fastest; Indicates when the parameter When changes occur, output action How it changes.

9. A multi-UAV power grid cache collaborative scheduling method based on federated deep reinforcement learning according to claim 1, characterized in that, Each UAV uploads its TD3 network parameters to a high-altitude server for aggregation, resulting in global model parameters. ； in, Weighting of drones; Let be the local model parameters for the i-th UAV; After aggregation, the high-altitude platform will allocate global model parameters. The data is distributed to each drone, which then uses it to replace the parameters of its local trajectory planning model as the initial model for the next training round. This enables the collaborative evolution of trajectory strategies across multiple drones while protecting the privacy of each drone's local observation data.

Images