An unmanned aerial vehicle cluster cooperative target tracking strategy acquisition and online updating method and device

Abstract

The application discloses a method and device for acquiring and online updating of a UAV cluster cooperative target tracking strategy, and the method comprises the following steps: deploying a strategy network Actor network and an evaluation network Critic network to each UAV flight control system, and creating a mirror agent for each UAV; acquiring observation information of each UAV in real time, inputting the observation information into the strategy network Actor network and the mirror Actor network of each UAV, and generating a corresponding flight strategy; after driving the UAV to perform a flight action according to the flight strategy, acquiring a first cumulative reward value of the UAV, and after the mirror agent performs a virtual flight action according to a mirror flight strategy, estimating a reward value and acquiring a second cumulative reward value; and updating parameters of the strategy network Actor network, the evaluation network Critic network and the mirror network of the UAV after driving the UAV to perform the flight strategy for a preset number of times. The application can improve the accuracy of the tracking strategy.

Description

Technical Field

[0001] This invention relates to the field of autonomous collaborative decision-making and control technology for unmanned aerial vehicle (UAV) swarms, specifically to a method and apparatus for acquiring and online updating a collaborative target tracking strategy for UAV swarms. Background Technology

[0002] In collaborative target search and tracking tasks involving UAV swarms, UAVs must adjust their positions based on their own perception information to ensure the target remains within the swarm's perception and control range. However, since the number and initial positions of targets are usually unknown, and the target's movement patterns and intentions are also unclear, UAV swarms face significant challenges in target tracking. To adapt to these uncertainties, UAV swarms must adjust their positions in real time based on their local observation information to simultaneously track multiple dynamic targets. This process requires efficient swarm collaboration and precise decision-making and control capabilities. In recent years, reinforcement learning-based UAV swarm target tracking methods have been widely studied. Through reinforcement learning, agents can continuously optimize their strategies during interaction with the environment, thereby improving the swarm's task execution efficiency and tracking accuracy.

[0003] Existing reinforcement learning-based target tracking methods typically rely on simplified modeling of the real-world environment to construct the training environment, while the real-world environment is often more complex than the training environment. Directly applying the policies learned by the agent in the training environment to the actual control system may lead to a decline in control performance or even suboptimal decisions. Summary of the Invention

[0004] This invention provides a method and apparatus for acquiring and online updating a cooperative target tracking strategy for unmanned aerial vehicle (UAV) swarms. Implementing this invention improves the generalization ability of the strategy and avoids suboptimal decision-making problems caused by differences between the training environment and the real-world environment.

[0005] An embodiment of the present invention provides a method for acquiring and updating an unmanned aerial vehicle (UAV) swarm cooperative target tracking strategy online, comprising:

[0006] Obtain an initial network generated through training in a preset target tracking simulation environment; wherein the initial network includes a first Actor network and a first Critic network;

[0007] The initial network is deployed to each UAV flight control system, and a mirror agent is created for each UAV; the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache;

[0008] Real-time acquisition of observation information from each drone;

[0009] After acquiring the observation information of each UAV, the observation information is input into the first Actor network and the mirror Actor network corresponding to each UAV to generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; after driving each UAV to perform flight actions according to the corresponding first flight strategy, the first cumulative reward value of each UAV is obtained; after each mirror agent performs virtual flight actions according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value.

[0010] After each drone executes the first flight strategy m times, the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network corresponding to each drone are updated based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone.

[0011] After each drone executes its first flight strategy, the parameters of the corresponding mirror Actor network and mirror Critic network are updated based on the mirror experience replay cache.

[0012] Furthermore, obtaining the initial network generated through training in a preset target tracking simulation environment includes:

[0013] Acquire the hardware parameters of the drone, the motion parameters of the target to be tracked, and the location information of the mission area;

[0014] Based on the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the tracking area, a target tracking simulation environment is constructed; wherein, the target tracking simulation environment includes an environmental constraint model, a target model, and a UAV model;

[0015] In the target tracking simulation environment, the state space, action space, reward function, and training objective of the agent are defined based on the Markov decision process framework; an Actor network and a Critic network are constructed; and an experience replay cache is built.

[0016] Based on the Advantage Actor-Critic algorithm, the UAV is treated as an intelligent agent. The agent interacts with the target tracking simulation environment to train until a preset number of training iterations are reached, generating a trained policy network (Actor network) and an evaluation network (Critic network). During training, the agent generates state transition experience samples by interacting with the simulation environment and stores them in an experience replay cache. The network parameters of the Actor network and Critic network are optimized by sampling the experience in the experience replay cache. All agents share the same set of Actor and Critic networks during training.

[0017] The trained policy network (Actor network) is denoted as the first Actor network; the trained evaluation network (Critic network) is denoted as the first Critic network; the first Actor network and the first Critic network are denoted as the initial network.

[0018] Furthermore, the target model includes the target's motion pattern;

[0019] The motion pattern of the target is specifically as follows:

[0020]

[0021] in, For target T i The position coordinates at time t; ground point Height; C cone The number of equivalent cones within the task area; Let J be the height of the j-th cone; The radius of the base of the j-th cone; The coordinates of the center of the base of the j-th cone; For target T i Maximum speed on the X-axis; For target T i Maximum speed on the Y-axis; It is a uniform distribution; It follows a normal distribution; D X D represents the lateral extent of the tracking area. Y The longitudinal range of the tracking area.

[0022] Furthermore, the drone model includes the drone's motion mode, the drone's target tracking and locking capability range, the drone's maximum scanning and detection capability range, the drone's effective communication range, and the drone's maximum communication range.

[0023] The movement mode of the drone is as follows:

[0024]

[0025] D(a,b)=sgn(a)-sgn(b)

[0026]

[0027] in, For U drones i The position coordinates at time t; s U sgn represents the unit flight step size of the UAV; sgn is the sign function. This is the command for the drone to move forward along the X-axis. This is the command for the drone to move backward along the X-axis. This is the command for the drone to move forward along the Y-axis. The command for the drone to move backward along the Y-axis; This is the command for the drone to move forward along the Z-axis. This is the command for the drone to move backward along the Z-axis. For point ground height; h min h is the minimum flight altitude of the drone relative to the ground. max This refers to the maximum flight altitude of the drone relative to the ground.

[0028] The target tracking and locking capability range of the UAV is specifically as follows:

[0029]

[0030] in, For U drones i Target locking range; P max β is the radius of the drone target locking range; β is the maximum angle of the drone target locking range.

[0031] The maximum scanning and detection capability range of the UAV is as follows:

[0032]

[0033] in, For U drones i Maximum detection range; P dec α is the radius of the drone's maximum detection range; α is the maximum angle of the drone's maximum detection range.

[0034] The effective communication range of the UAV is specifically as follows:

[0035]

[0036] in, For U drones i Effective communication range; C e The effective communication radius of the drone;

[0037] The maximum communication range of the drone is as follows:

[0038]

[0039] in, For U drones i Maximum communication range; C max This represents the maximum communication radius of the drone.

[0040] Furthermore, the state space includes local information, target information within the target tracking and locking capability range, and friendly machine information within the effective communication range;

[0041] The local information is specifically as follows:

[0042]

[0043] in, For U drones i Local information at time t;

[0044] The target information within the target tracking and locking capability range specifically includes:

[0045]

[0046] in, For U drones i Regarding target T within the target tracking and locking capability range k Target information at time t; For U drones i At time t relative to target T k Relative position on the X-axis; For U drones i At time t relative to target T k Relative position on the Y-axis; For U drones i At time t relative to target T k Relative position on the Z-axis; For U drones i With target T k The distance between times t; For target T k The velocity on the X-axis at time t; For target T k The velocity on the Y-axis at time t;

[0047] The information on friendly devices within the effective communication range specifically includes:

[0048]

[0049] in, For U drones i Friendly machine U within the relatively effective communication range j Information about the friendly machine at time t; For U drones i Friendly machine U within the relatively effective communication range j The relative position on the X-axis at time t; UAV i Friendly machine U within the relatively effective communication range j The relative position on the Y-axis at time t; UAV i Friendly machine U within the relatively effective communication range j The relative position on the Z-axis at time t; For U drones i With friendly machine U j The distance at time t.

[0050] Furthermore, the reward function includes environmental rewards, target rewards, and friendly machine rewards;

[0051] The environmental rewards are specifically:

[0052]

[0053] in, For U drones i Environmental reward at time t;

[0054] The target reward is specifically as follows:

[0055]

[0056] in, For U drones i The target reward at time t; For U drones i At time t relative to target T k Target rewards;

[0057] The aforementioned friendly aircraft reward is as follows:

[0058]

[0059] in, For U drones i Rewards for friendly aircraft at time t For U drones i Friendly machine U within the relatively effective communication range j Rewards for friendly aircraft at time t.

[0060] Furthermore, the Advantage Actor-Critic algorithm treats the drone as an intelligent agent, which interacts with the target tracking simulation environment to train the agent until a preset number of training iterations are reached, generating a trained policy network (Actor network) and an evaluation network (Critic network), including:

[0061] Initialize the experience replay cache and the network parameters of the policy network (Actor network) and the evaluation network (Critic network);

[0062] Repeat the parameter update operation until the training count is reached, and generate the trained policy network (Actor network) and evaluation network (Critic network).

[0063] The parameter update operation includes:

[0064] For each agent, current observation information is acquired, and corresponding state transition experience samples are generated based on the interaction process between the agent and the target tracking simulation environment. The observation information includes local information defined in the state space, target information within the target tracking locking capability range, and friendly agent information within the effective communication range. The state transition experience samples include the current observation information, the executed action, the environmental feedback reward, and the observation information at the next moment.

[0065] Store the state transition experience samples into the experience replay cache and update the current experience replay cache;

[0066] Calculate the timing error and sampling probability of each experience in the current experience replay buffer;

[0067] For each agent, K experience samples are drawn from the current experience replay buffer according to probability; the policy gradient is calculated, and the network parameters of the policy network (Actor network) are updated using the gradient descent method; the network parameters of the evaluation network (Critic network) are updated according to the calculated time-series error value.

[0068] Given the number of training iterations, the current policy network (Actor network) and evaluation network (Critic network) are denoted as the trained policy network and evaluation network.

[0069] Furthermore, updating the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network corresponding to each drone based on the difference between the first cumulative reward value and the corresponding second cumulative reward value includes:

[0070] Drones with a first cumulative reward value greater than the corresponding second cumulative reward value are classified as Class 1 drones, and the remaining drones are classified as Class 2 drones. The mirror Actor network and mirror Critic network of Class 1 drones are updated to the corresponding first Actor network and first Critic network according to the preset first probability. The first Actor network and first Critic network of Class 2 drones are updated to the corresponding mirror Actor network and mirror Critic network according to the preset second probability.

[0071] Furthermore, the step of updating the parameters of the corresponding mirror Actor network and mirror Critic network for each mirror agent based on the mirror experience replay cache includes:

[0072] For each mirror agent, calculate the temporal error and sampling probability of each experience in the corresponding mirror experience replay cache; extract K second experiences from the corresponding mirror experience replay cache according to the probability; calculate the policy gradient, and use the gradient descent method to update the network parameters of the corresponding mirror Actor network and the corresponding mirror Critic network.

[0073] The network parameters of the mirror Actor network are updated in the following manner:

[0074]

[0075] δ t =r t +γv ω (o t+1 )-v ω (o t )

[0076]

[0077] p t =|δ t |+∈

[0078] Where θ represents the network parameters of the mirror Actor network; η a Δ is the learning rate; a The gradient change of the mirror Actor network; K is the sampling size; w t The importance sampling weight; δ t This represents the time-series difference value; The gradient of the mirror Actor network; r t For instant rewards; γ is the discount factor; v ω (o t+1 ) is the value function for the next state; v ω (o t ) represents the value function of the current state; N represents the mirror experience replay cache. The capacity of P(b) t ) for experience b t The sampling probability; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t Priority;

[0079] Update the network parameters of the mirrored Critic network as follows:

[0080]

[0081] Where ω represents the network parameters of the mirrored Critic network; Δ c This represents the gradient update amount for the mirrored Critic network. This represents the gradient of the mirrored Critic network.

[0082] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0083] An embodiment of the present invention provides a device for acquiring and updating a collaborative target tracking strategy for a drone swarm, comprising: an initial network acquisition module, a network deployment module, a real-time observation information acquisition module, a cumulative reward value acquisition module, and a network update module;

[0084] The initial network acquisition module is used to acquire the first Actor network and the first Critic network generated through training in a preset target tracking simulation environment.

[0085] The network deployment module is used to deploy the first Actor network and the first Critic network to each UAV flight control system, and to create a mirror agent for each UAV. The mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache.

[0086] The real-time observation information acquisition module is used to acquire the observation information of each UAV in real time.

[0087] The cumulative reward value acquisition module is used to input the observation information of each UAV into the first Actor network and the mirror Actor network corresponding to each UAV after acquiring the observation information of each UAV, and generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; to acquire the first cumulative reward value of each UAV after driving each UAV to perform flight actions according to the corresponding first flight strategy; and to estimate the reward value and acquire the second cumulative reward value after each mirror agent performs virtual flight actions according to the mirror flight strategy.

[0088] The network update module is used to update the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each UAV based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each UAV after each UAV executes the first flight strategy m times; and to update the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent based on the mirror experience replay cache after each UAV executes the first flight strategy once.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] This invention provides a method and apparatus for acquiring and online updating a collaborative target tracking strategy for a drone swarm. The method acquires an initial network trained in a target tracking simulation environment, comprising a first Actor network and a first Critic network, and deploys it to the drone flight control system. A mirror agent is created for each drone, containing a mirror Actor network, a mirror Critic network, and a mirror experience replay cache. During task execution, observation information from each drone is acquired in real time, and corresponding flight strategies are generated by inputting this observation information into the drone's first Actor network and mirror Actor network. By executing these flight strategies, cumulative reward values ​​are calculated and differential updates are performed to optimize the agent's network parameters.

[0091] This invention deploys an initial strategy to the UAV flight control system and introduces a mirror agent for online correction, adjusting the strategy in real time. This enables the agent to better adapt to complex real-world environments when performing tasks, improving the generalization ability of the strategy and avoiding suboptimal decision-making problems caused by differences between the training environment and the real environment. Attached Figure Description

[0092] Figure 1 This is a flowchart illustrating a method for acquiring and updating a collaborative target tracking strategy for unmanned aerial vehicle (UAV) swarms, according to an embodiment of the present invention.

[0093] Figure 2 This is a schematic diagram of the communication and sensing capabilities of an unmanned aerial vehicle (UAV) according to an embodiment of the present invention.

[0094] Figure 3 This is a schematic diagram of the deployment of a policy network provided in an embodiment of the present invention.

[0095] Figure 4 This is a flowchart illustrating a method for acquiring and updating a collaborative target tracking strategy for a drone swarm, provided by another embodiment of the present invention.

[0096] Figure 5 This is a schematic diagram of the structure of a device for acquiring and updating a collaborative target tracking strategy for a drone swarm, according to an embodiment of the present invention. Detailed Implementation

[0097] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0098] like Figure 1 As shown, an embodiment of the present invention provides a method for acquiring and updating an unmanned aerial vehicle (UAV) swarm cooperative target tracking strategy online, comprising at least the following steps:

[0099] Step S1: Obtain the initial network generated by training in a preset target tracking simulation environment.

[0100] Specifically, the initial network includes a first Actor network and a first Critic network;

[0101] In a preferred embodiment, obtaining the initial network generated through training in a preset target tracking simulation environment includes:

[0102] Acquire the hardware parameters of the drone, the motion parameters of the target to be tracked, and the location information of the mission area;

[0103] Based on the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the tracking area, a target tracking simulation environment is constructed; wherein, the target tracking simulation environment includes an environmental constraint model, a target model, and a UAV model;

[0104] In the target tracking simulation environment, the state space, action space, reward function, and training objective of the agent are defined based on the Markov decision process framework; an Actor network and a Critic network are constructed; and an experience replay cache is built.

[0105] Based on the Advantage Actor-Critic algorithm, the UAV is treated as an intelligent agent. The agent interacts with the target tracking simulation environment to train until a preset number of training iterations are reached, generating a trained policy network (Actor network) and an evaluation network (Critic network). During training, the agent generates state transition experience samples by interacting with the simulation environment and stores them in an experience replay cache. The network parameters of the Actor network and Critic network are optimized by sampling the experience in the experience replay cache. All agents share the same set of Actor and Critic networks during training.

[0106] The trained policy network (Actor network) is denoted as the first Actor network; the trained evaluation network (Critic network) is denoted as the first Critic network; the first Actor network and the first Critic network are denoted as the initial network.

[0107] Specifically, the process involves acquiring the drone's hardware parameters, the target's motion parameters, and the location information of the mission area. These hardware parameters include the drone's flight capabilities, sensor performance, and communication capabilities; the target's motion parameters include its speed, acceleration, trajectory, and prediction model; and the location information of the mission area includes the size, shape, geographical features, and obstacle information of the tracking area.

[0108] Based on this acquired information, a target tracking simulation environment is constructed. The simulation environment comprises three main parts: an environmental constraint model, a target model, and a UAV model. The environmental constraint model simulates the constraints within the mission area; the target model describes the motion patterns, behavior modes, and interactions with the environment of the target to be tracked, ensuring that the target's behavior and trajectory conform to reality; the UAV model includes the UAV's dynamic characteristics, control algorithms, and communication systems, simulating the UAV's flight behavior and collaborative operation during mission execution. These models work together to form a complete target tracking simulation environment that realistically reflects the performance of UAVs in actual missions.

[0109] The problem of cooperative target tracking by a swarm of unmanned aerial vehicles (UAVs) under the condition that the communication range and observation range are bounded can be described as: target swarm A drone swarm contains n targets that can move randomly within the mission area. It's difficult to directly obtain all target information, such as quantity, location distribution, and speed. Drones can maintain continuous communication with other drones within effective communication range and share their location and movement information. Drones make action decisions based on their own information, target information within their tracking range, and information from friendly drones within effective communication range, ensuring the target remains continuously within the drone swarm's tracking range. Simultaneously, the drone swarm doesn't know how many targets need to be continuously observed to complete its mission; it simply assumes that the more targets observed simultaneously, the better.

[0110] In a preferred embodiment, the environmental constraint model includes tracking area boundaries and topographic model factors;

[0111] The target model includes the target's range of motion and the target's motion pattern;

[0112] The drone model includes the drone's motion mode, the drone's target tracking and locking capability range, the drone's maximum scanning and detection capability range, the drone's effective communication range, and the drone's maximum communication range.

[0113] The state space includes local information, target information within the target tracking and locking capability range, and friendly machine information within the effective communication range;

[0114] The motion space includes the forward and backward movement of the UAV along the X, Y, and Z axes;

[0115] The reward function includes environmental rewards, target rewards, and friendly machine rewards;

[0116] The boundary of the tracking region is specifically:

[0117] D X ×D Y ×D Z

[0118] Among them, D X D represents the lateral extent of the tracking area. Y D represents the longitudinal range of the tracking area. Z To limit the height of the tracking area;

[0119] Optionally, the boundary of the tracking area can be set to a 2000×2000×1000 three-dimensional space. The terrain features within the mission area are areas that the UAV cannot traverse, but the target can move on the surface of the terrain features.

[0120] The specific geomorphological model factors are:

[0121]

[0122] Where z is the height coordinate of the point (x, y) on the base of the cone; (x0, y0) is the coordinate of the center of the base of the cone; r is the radius of the base circle of the cone; h cone The height of the cone is the vertical distance from the vertex to the center of the bottom circle; here, the terrain objects in the environment are modeled as cones.

[0123] The target's range of motion is the bottom surface of the cone;

[0124] The motion pattern of the target is specifically as follows:

[0125]

[0126] in, For target T i The position coordinates at time t; ground point Height; C cone The number of equivalent cones within the task area; Let J be the height of the j-th cone; The radius of the base of the j-th cone; The coordinates of the center of the base of the j-th cone; For target T i Maximum speed on the X-axis; For target Ti Maximum speed on the Y-axis; It is a uniform distribution; It follows a normal distribution; D X D represents the lateral extent of the tracking area. Y The longitudinal range of the tracking area;

[0127] Specifically, if a target is within the target locking range of any drone in the drone swarm at time t, then the target is considered to be tracked by the drone swarm at time t.

[0128] The movement mode of the drone is as follows:

[0129]

[0130] D(a,b)=sgn(a)-sgn(b)

[0131]

[0132] in, For U drones i The position coordinates at time t; s U sgn represents the unit flight step size of the UAV; sgn is the sign function. This is the command for the drone to move forward along the X-axis. This is the command for the drone to move backward along the X-axis. This is the command for the drone to move forward along the Y-axis. The command for the drone to move backward along the Y-axis; This is the command for the drone to move forward along the Z-axis. This is the command for the drone to move backward along the Z-axis. For point ground height; h min h is the minimum flight altitude of the drone relative to the ground. max This refers to the maximum flight altitude of the drone relative to the ground.

[0133] The target tracking and locking capability range of the UAV is specifically as follows:

[0134]

[0135] in, For U drones i Target locking range; P max β is the radius of the drone target locking range; β is the maximum angle of the drone target locking range.

[0136] The maximum scanning and detection capability range of the UAV is as follows:

[0137]

[0138] in, For U drones i Maximum detection range; P dec α is the radius of the drone's maximum detection range; α is the maximum angle of the drone's maximum detection range.

[0139] The effective communication range of the UAV is specifically as follows:

[0140]

[0141] in, For U drones i Effective communication range; C e The effective communication radius of the drone;

[0142] The maximum communication range of the drone is as follows:

[0143]

[0144] in, For U drones i Maximum communication range; C max This represents the maximum communication radius of the drone.

[0145] It needs to be explained here that if the drone U i and U drones j Spatial distance d between i,j Not greater than the effective communication radius C e Then the UAV i Location and motion information can be linked to the UAV. j To share communications; if the UAV U i and U drones j Spatial distance d between i,j Greater than the effective communication radius C e However, it is less than the maximum communication radius C. max Then the UAV i Location information can only be associated with the UAV. j To share communications; if the UAV U i and U drones j Spatial distance d between i,j Greater than the maximum communication radius C max Then the UAV i Unable to communicate with U drones j Establish communication connections. Specifically, a diagram illustrating the drone's communication and sensing capabilities is shown below. Figure 2 As shown.

[0146] The local information is specifically as follows:

[0147]

[0148] in, For U drones i Local information at time t;

[0149] The target information within the target tracking and locking capability range specifically includes:

[0150]

[0151] in, For U drones i Regarding target T within the target tracking and locking capability range k Target information at time t; For U drones i At time t relative to target T k Relative position on the X-axis; For U drones i At time t relative to target T k Relative position on the Y-axis; For U drones i At time t relative to target T k Relative position on the Z-axis; For U drones i With target T k The distance between times t; For target T k The velocity on the X-axis at time t; For target T k The velocity on the Y-axis at time t;

[0152] The information on friendly devices within the effective communication range specifically includes:

[0153]

[0154] in, For U drones i Friendly machine U within the relatively effective communication range j Information about the friendly machine at time t; For U drones i Friendly machine U within the relatively effective communication range j The relative position on the X-axis at time t; UAV i Friendly machine U within the relatively effective communication range j The relative position on the Y-axis at time t; UAVi Friendly machine U within the relatively effective communication range j The relative position on the Z-axis at time t; For U drones i With friendly machine U j The distance at time t.

[0155] The environmental rewards are specifically:

[0156]

[0157] in, For U drones i Environmental reward at time t;

[0158] The target reward is specifically as follows:

[0159]

[0160] in, For U drones i The target reward at time t; For U drones i At time t relative to target T k Target rewards;

[0161] The aforementioned friendly aircraft reward is as follows:

[0162]

[0163]

[0164] in, For U drones i Rewards for friendly aircraft at time t For U drones i Friendly machine U within the relatively effective communication range j Rewards for friendly aircraft at time t.

[0165] It should be noted here that the swarm reward of a drone swarm is the sum of the rewards received by all drones, and the drone reward includes the environmental reward, target reward, and friendly drone reward in the reward function.

[0166] The training objective is specifically as follows:

[0167]

[0168] st

[0169]

[0170] in, For target T iAt time t; For drone swarms; T is the time sequence length.

[0171] It should be explained here that, among the constraints of the training objectives mentioned above, the first and second terms constrain the target's range of movement, while the third term represents the performance constraints of the UAV.

[0172] In a preferred embodiment, the step of using the Advantage Actor-Critic algorithm, treating the UAV as an intelligent agent, and training the agent through interaction with the target tracking simulation environment until a preset number of training iterations are reached, generates a trained policy network (Actor network) and an evaluation network (Critic network), including:

[0173] Initialize the experience replay cache and the network parameters of the policy network (Actor network) and the evaluation network (Critic network);

[0174] Repeat the parameter update operation until the training count is reached, and generate the trained policy network (Actor network) and evaluation network (Critic network).

[0175] The parameter update operation includes:

[0176] For each agent, current observation information is acquired, and corresponding state transition experience samples are generated based on the interaction process between the agent and the target tracking simulation environment. The observation information includes local information defined in the state space, target information within the target tracking locking capability range, and friendly agent information within the effective communication range. The state transition experience samples include the current observation information, the executed action, the environmental feedback reward, and the observation information at the next moment.

[0177] Store the state transition experience samples into the experience replay cache and update the current experience replay cache;

[0178] Calculate the timing error and sampling probability of each experience in the current experience replay buffer;

[0179] For each agent, K experience samples are drawn from the current experience replay buffer according to probability; the policy gradient is calculated, and the network parameters of the policy network (Actor network) are updated using the gradient descent method; the network parameters of the evaluation network (Critic network) are updated according to the calculated time-series error value.

[0180] Given the number of training iterations, the current policy network (Actor network) and evaluation network (Critic network) are denoted as the trained policy network and evaluation network.

[0181] Specifically, the network is trained using a parameter-sharing method, and once training is complete, a target tracking strategy with a certain degree of universality is obtained.

[0182] Optionally, calculating the sampling probability of each experience in the current experience replay cache includes:

[0183] Calculate the temporal difference value of each experience in the current experience replay cache, and generate the temporal difference value of each experience;

[0184] Based on the temporal difference values ​​of each experience and a preset constant, the priority of each experience is generated;

[0185] The priorities of each experience are summed to generate a cumulative priority.

[0186] The sampling probability of each experience is determined based on the ratio of the priority of each experience to the cumulative priority.

[0187] The timing difference value is calculated using the following formula:

[0188] δ t =r t +γv ω (o t+1 )-v ω (o t )

[0189] In the formula, δ t The time difference value; r t For instant rewards; γ is the discount factor; v ω (o t+1 ) is the value function for the next state; v ω (o t ) is the value function of the current state.

[0190] Priority is calculated using the following formula:

[0191] p t =|δ t |+∈

[0192] In the formula, p t Priority; ∈ is a constant;

[0193] The sampling probability is determined using the following formula:

[0194]

[0195] In the formula, P(b) t ) for experience b t The sampling probability; l is the compensation parameter;

[0196] Step S2: Deploy the initial network to each UAV flight control system and create a mirror agent for each UAV; the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache. Their deployment relationships are as follows: Figure 3 As shown.

[0197] It should be explained here that each drone is equipped with a microcomputer, on which the mirrored agent can be deployed.

[0198] Step S3: Acquire observation information from each UAV in real time.

[0199] Step S4: After acquiring the observation information of each UAV, the observation information is input into the first Actor network and the mirror Actor network corresponding to each UAV to generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; after driving each UAV to perform flight actions according to the corresponding first flight strategy, the first cumulative reward value of each UAV is obtained; after each mirror agent performs virtual flight actions according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value.

[0200] Specifically, after acquiring observation information from each UAV, this information is first input into the first Actor network and mirror Actor network corresponding to each UAV. Through calculations in these two networks, a first flight strategy and a corresponding mirror flight strategy are generated for each UAV. The first flight strategy represents the action plan chosen by the UAV in the current state, while the mirror flight strategy is a similar strategy generated for the corresponding mirror agent, aiming to enhance the system's generalization ability through simulated training with the mirror agent.

[0201] Subsequently, by driving each UAV to execute its corresponding first flight strategy, the UAV adjusts its behavior based on environmental feedback during actual execution. After execution, each UAV receives a first cumulative reward value based on its deviation from the target and environmental conditions. This reward value reflects the UAV's performance in a specific task and is evaluated based on the aforementioned reward function.

[0202] At the same time, when the mirror agent interacts with the real environment, it will also execute the corresponding mirror flight strategy. Through this training simulation in the virtual environment, the mirror agent will obtain a second cumulative reward value, which indicates the effect of the mirror agent executing the mirror flight strategy in the real environment, serving as a reference and aid for the execution effect of the real drone flight strategy.

[0203] These reward function values ​​are used to update the networks of drones and mirror agents, ensuring that each agent gradually improves the accuracy and effectiveness of its policies through continuous feedback and learning.

[0204] Step S5: After each drone executes the first flight strategy m times, update the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network corresponding to each drone based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone.

[0205] In a preferred embodiment, updating the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network corresponding to each drone based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone includes:

[0206] Drones with a first cumulative reward value greater than the corresponding second cumulative reward value are classified as Class 1 drones, and the remaining drones are classified as Class 2 drones. The mirror Actor network and mirror Critic network of Class 1 drones are updated to the corresponding first Actor network and first Critic network according to the preset first probability. The first Actor network and first Critic network of Class 2 drones are updated to the corresponding mirror Actor network and mirror Critic network according to the preset second probability.

[0207] Specifically, the first probability and the second probability can be flexibly set according to the actual situation. The statement that m times is executed is only an example, and the specific number of times can be flexibly set according to the actual situation.

[0208] Step S6: After each drone executes the first flight strategy, update the parameters of the corresponding mirror Actor network and mirror Critic network based on the mirror experience replay cache.

[0209] In a preferred embodiment, updating the parameters of the corresponding mirror Actor network and mirror Critic network for each mirror agent based on the mirror experience replay cache includes:

[0210] For each mirror agent, calculate the temporal error and sampling probability of each experience in the corresponding mirror experience replay cache; extract K second experiences from the corresponding mirror experience replay cache according to the probability; calculate the policy gradient of the second experience, and use gradient descent to update the network parameters of the corresponding mirror Actor network and the corresponding mirror Critic network.

[0211] The network parameters of the mirror Actor network are updated in the following manner:

[0212]

[0213] δ t =r t +γv ω (o t+1 )-v ω (o t )

[0214]

[0215] p t =|δ t |+∈

[0216] Where θ represents the network parameters of the mirror Actor network; η a Δ is the learning rate; a The gradient change of the mirror Actor network; K is the sampling size; w t The importance sampling weight; δ t This represents the time-series difference value; The gradient of the mirror Actor network; r t For instant rewards; γ is the discount factor; v ω (o t+1 ) is the value function for the next state; v ω (o t ) represents the value function of the current state; N represents the mirror experience replay cache. The capacity of P(b) t ) for experience b t The sampling probability; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t Priority;

[0217] Update the network parameters of the mirrored Critic network as follows:

[0218]

[0219] Where ω represents the network parameters of the mirrored Critic network; Δ c This represents the gradient update amount for the mirrored Critic network. This represents the gradient of the mirrored Critic network.

[0220] Specifically, in a preferred embodiment, the steps for acquiring and online updating the UAV swarm collaborative target tracking strategy are as follows: Figure 4 As shown. The number of times n is only for illustrative purposes; the specific number of times can be flexibly set according to the actual situation.

[0221] It should be noted that the online updating of the UAV swarm collaborative target tracking strategy in this invention is for the purpose of online fine-tuning and correction of the strategy. This method is also applicable to the target search domain, not just the target tracking domain. During the strategy fine-tuning and correction process, each UAV loads an executable strategy and generates a mirror agent. During the execution of the target tracking task, the current execution strategy is continuously updated by comparing the cumulative reward value of the currently executed strategy with the cumulative reward value obtained by the mirror agent. This method enables autonomous collaborative decision-making and control of the UAV swarm in target tracking scenarios and has good robustness.

[0222] The Advantage Actor-Critic algorithm based on probability-based experience extraction, as described in this invention, can be termed the Distributed Advantage Actor-Critic algorithm based on a priority experience replay mechanism. This distributed algorithm accelerates the training process and improves learning efficiency by prioritizing valuable experiences. Furthermore, the distributed architecture allows multiple agents to train in parallel, further enhancing the speed and generalization ability of policy learning.

[0223] In simulation training, this invention uses parameter sharing to train the agent, thereby obtaining a strategy with certain universality and accelerating the training speed of the agent.

[0224] During strategy fine-tuning, each UAV has an independent mirror agent, which has its own mirror Actor network and mirror Critic network. By comparing the cumulative reward obtained by the actual strategy network with the cumulative reward obtained by the virtual strategy network, the execution strategy is corrected and updated to obtain the optimal target search and tracking strategy.

[0225] The training process requires no human pilot assistance. Each drone can adaptively fine-tune and update its strategy based on data from the actual scenario, enhancing the diversity of drone swarm target search and tracking strategies and improving the strategy's generalization ability to the environment.

[0226] Based on the above method embodiments, the present invention provides corresponding apparatus embodiments.

[0227] like Figure 5 As shown, an embodiment of the present invention provides a device for acquiring and updating a collaborative target tracking strategy for a drone swarm, comprising: an initial network acquisition module, a network deployment module, a real-time observation information acquisition module, a cumulative reward value acquisition module, and a network update module;

[0228] The initial network acquisition module is used to acquire the first Actor network and the first Critic network generated through training in a preset target tracking simulation environment.

[0229] The network deployment module is used to deploy the first Actor network and the first Critic network to each UAV flight control system, and to create a mirror agent for each UAV. The mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache.

[0230] The real-time observation information acquisition module is used to acquire the observation information of each UAV in real time.

[0231] The cumulative reward value acquisition module is used to input the observation information of each UAV into the first Actor network and the mirror Actor network corresponding to each UAV after acquiring the observation information of each UAV, and generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; to acquire the first cumulative reward value of each UAV after driving each UAV to perform flight actions according to the corresponding first flight strategy; and to estimate the reward value and acquire the second cumulative reward value after each mirror agent performs virtual flight actions according to the mirror flight strategy.

[0232] The network update module is used to update the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each UAV based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each UAV after each UAV executes the first flight strategy m times; and to update the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent based on the mirror experience replay cache after each UAV executes the first flight strategy once.

[0233] It should be noted that the embodiments of the device described above correspond to the embodiments of the present invention described above, and can realize the online update method of the UAV swarm cooperative target tracking strategy network described in any one of the above embodiments of the present invention. Furthermore, the embodiments of the device described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without creative effort.

[0234] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0235] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for acquiring and online updating a cooperative target tracking strategy for unmanned aerial vehicle (UAV) swarms, characterized in that, include: Obtain an initial network generated through training in a preset target tracking simulation environment; wherein the initial network includes a first Actor network and a first Critic network; The initial network is deployed to each UAV flight control system, and a mirror agent is created for each UAV; the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache; Real-time acquisition of observation information from each drone; After acquiring the observation information of each UAV, the observation information is input into the first Actor network and the mirror Actor network corresponding to each UAV to generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; after driving each UAV to perform flight actions according to the corresponding first flight strategy, the first cumulative reward value of each UAV is obtained; after each mirror agent performs virtual flight actions according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value. After each drone executes the first flight strategy m times, the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network corresponding to each drone are updated based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone. After each drone executes its first flight strategy, the parameters of the corresponding mirror Actor network and mirror Critic network are updated based on the mirror experience replay cache.

2. The method for acquiring and online updating a UAV swarm cooperative target tracking strategy as described in claim 1, characterized in that, The process of obtaining the initial network generated through training in a preset target tracking simulation environment includes: Acquire the hardware parameters of the drone, the motion parameters of the target to be tracked, and the location information of the mission area; Based on the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the tracking area, a target tracking simulation environment is constructed; wherein, the target tracking simulation environment includes an environmental constraint model, a target model, and a UAV model; In the target tracking simulation environment, the state space, action space, reward function, and training objective of the agent are defined based on the Markov decision process framework; an Actor network and a Critic network are constructed; and an experience replay cache is built. Based on the Advantage Actor-Critic algorithm, the UAV is treated as an intelligent agent. The agent interacts with the target tracking simulation environment to train until a preset number of training iterations are reached, generating a trained policy network (Actor network) and an evaluation network (Critic network). During training, the agent generates state transition experience samples by interacting with the simulation environment and stores them in an experience replay cache. The network parameters of the Actor network and Critic network are optimized by sampling the experience in the experience replay cache. All agents share the same set of Actor and Critic networks during training. The trained policy network (Actor network) is denoted as the first Actor network; the trained evaluation network (Critic network) is denoted as the first Critic network; the first Actor network and the first Critic network are denoted as the initial network.

3. The method for acquiring and online updating the UAV swarm cooperative target tracking strategy as described in claim 2, characterized in that, The target model includes the target's motion pattern; The motion pattern of the target is specifically as follows: in, For target T i The position coordinates at time t; ground point Height; C cone The number of equivalent cones within the task area; Let J be the height of the j-th cone; The radius of the base of the j-th cone; The coordinates of the center of the base of the j-th cone; For target T i Maximum speed on the X-axis; For target T i Maximum speed on the Y-axis; It is a uniform distribution; It follows a normal distribution; D X D represents the lateral extent of the tracking area. Y The longitudinal range of the tracking area.

4. The method for acquiring and online updating the UAV swarm cooperative target tracking strategy as described in claim 3, characterized in that, The drone model includes the drone's motion mode, the drone's target tracking and locking capability range, the drone's maximum scanning and detection capability range, the drone's effective communication range, and the drone's maximum communication range. The movement mode of the drone is as follows: D(a,b)=sgn(a)-sgn(b) in, For U drones i The position coordinates at time t; s U sgn represents the unit flight step size of the UAV; sgn is the sign function. This is the command for the drone to move forward along the X-axis. This is the command for the drone to move backward along the X-axis. This is the command for the drone to move forward along the Y-axis. The command for the drone to move backward along the Y-axis; This is the command for the drone to move forward along the Z-axis. This is the command for the drone to move backward along the Z-axis. For point ground height; h min h is the minimum flight altitude of the drone relative to the ground. max This refers to the maximum flight altitude of the drone relative to the ground. The target tracking and locking capability range of the UAV is specifically as follows: in, For U drones i Target locking range; P max β is the radius of the drone target locking range; β is the maximum angle of the drone target locking range. The maximum scanning and detection capability range of the UAV is as follows: in, For U drones i Maximum detection range; P dec α is the radius of the drone's maximum detection range; α is the maximum angle of the drone's maximum detection range. The effective communication range of the UAV is specifically as follows: in, For U drones i Effective communication range; C e The effective communication radius of the drone; The maximum communication range of the drone is as follows: in, For U drones i Maximum communication range; C max This represents the maximum communication radius of the drone.

5. The method for acquiring and online updating a drone swarm cooperative target tracking strategy as described in claim 4, characterized in that, The state space includes local information, target information within the target tracking and locking capability range, and friendly machine information within the effective communication range; The local information is specifically as follows: in, For U drones i Local information at time t; The target information within the target tracking and locking capability range specifically includes: in, For U drones i Regarding target T within the target tracking and locking capability range k Target information at time t; For U drones i At time t relative to target T k Relative position on the X-axis; For U drones i At time t relative to target T k Relative position on the Y-axis; For U drones i At time t relative to target T k Relative position on the Z-axis; For U drones i With target T k The distance between times t; For target T k The velocity on the X-axis at time t; For target T k The velocity on the Y-axis at time t; The information on friendly devices within the effective communication range specifically includes: in, For U drones i Friendly machine U within the relatively effective communication range j Information about the friendly machine at time t; For U drones i Friendly machine U within the relatively effective communication range j The relative position on the X-axis at time t; UAV i Friendly machine U within the relatively effective communication range j The relative position on the Y-axis at time t; UAV i Friendly machine U within the relatively effective communication range j The relative position on the Z-axis at time t; For U drones i With friendly machine U j The distance at time t.

6. The method for acquiring and online updating a UAV swarm cooperative target tracking strategy as described in claim 5, characterized in that, The reward function includes environmental rewards, target rewards, and friendly machine rewards; The environmental rewards are specifically: in, For U drones i Environmental reward at time t; The target reward is specifically as follows: in, For U drones i The target reward at time t; For U drones i At time t relative to target T k Target rewards; The aforementioned friendly aircraft reward is as follows: in, For U drones i Rewards for friendly aircraft at time t For U drones i Friendly machine U within the relatively effective communication range j Rewards for friendly aircraft at time t.

7. The method for acquiring and online updating a UAV swarm cooperative target tracking strategy as described in claim 6, characterized in that, The Advantage Actor-Critic algorithm treats the UAV as an intelligent agent, which interacts with the target tracking simulation environment to train the agent until a preset number of training iterations are reached. This generates a trained policy network (Actor network) and an evaluation network (Critic network), including: Initialize the experience replay cache and the network parameters of the policy network (Actor network) and the evaluation network (Critic network); Repeat the parameter update operation until the training count is reached, and generate the trained policy network (Actor network) and evaluation network (Critic network). The parameter update operation includes: For each agent, current observation information is acquired, and corresponding state transition experience samples are generated based on the interaction process between the agent and the target tracking simulation environment. The observation information includes local information defined in the state space, target information within the target tracking locking capability range, and friendly agent information within the effective communication range. The state transition experience samples include the current observation information, the executed action, the environmental feedback reward, and the observation information at the next moment. Store the state transition experience samples into the experience replay cache and update the current experience replay cache; Calculate the timing error and sampling probability of each experience in the current experience replay buffer; For each agent, K experience samples are drawn from the current experience replay buffer according to probability; the policy gradient is calculated, and the network parameters of the policy network (Actor network) are updated using the gradient descent method; the network parameters of the evaluation network (Critic network) are updated according to the calculated time-series error value. Given the number of training iterations, the current policy network (Actor network) and evaluation network (Critic network) are denoted as the trained policy network and evaluation network.

8. The method for acquiring and online updating a drone swarm cooperative target tracking strategy as described in claim 7, characterized in that, The step of updating the parameters of the first Actor network, first Critic network, mirror Actor network, and mirror Critic network for each drone based on the difference between the first cumulative reward value and the corresponding second cumulative reward value includes: Drones with a first cumulative reward value greater than the corresponding second cumulative reward value are classified as Class 1 drones, and the remaining drones are classified as Class 2 drones. The mirror Actor network and mirror Critic network of Class 1 drones are updated to the corresponding first Actor network and first Critic network according to the preset first probability. The first Actor network and first Critic network of Class 2 drones are updated to the corresponding mirror Actor network and mirror Critic network according to the preset second probability.

9. The method for acquiring and online updating a UAV swarm cooperative target tracking strategy as described in claim 8, characterized in that, The step of updating the parameters of the corresponding mirror Actor network and mirror Critic network for each mirror agent based on the mirror experience replay cache includes: For each mirror agent, calculate the temporal error and sampling probability of each experience in the corresponding mirror experience replay cache; extract K second experiences from the corresponding mirror experience replay cache according to the probability; calculate the policy gradient, and use the gradient descent method to update the network parameters of the corresponding mirror Actor network and the corresponding mirror Critic network. The network parameters of the mirror Actor network are updated in the following manner: δ t =r t +γv ω (or t+1 )-v ω (or t ) p t =|δ t |+∈ Where θ represents the network parameters of the mirror Actor network; η a Δ is the learning rate; a The gradient change of the mirror Actor network; K is the sampling size; w t The importance sampling weight; δ t This represents the time-series difference value; The gradient of the mirror Actor network; r t For instant rewards; γ is the discount factor; v ω (o t+1 ) is the value function for the next state; v ω (o t P(b) is the value function of the current state; N is the capacity of the mirror experience replay buffer B; P(b) is the value function of the current state. t ) for experience b t The sampling probability; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t Priority; Update the network parameters of the mirrored Critic network as follows: Where ω represents the network parameters of the mirrored Critic network; Δ c This represents the gradient update amount for the mirrored Critic network. This represents the gradient of the mirrored Critic network.

10. A device for acquiring and updating a collaborative target tracking strategy for unmanned aerial vehicle (UAV) swarms, characterized in that: include: The module includes an initial network acquisition module, a network deployment module, a real-time observation information acquisition module, a cumulative reward value acquisition module, and a network update module. The initial network acquisition module is used to acquire the first Actor network and the first Critic network generated through training in a preset target tracking simulation environment. The network deployment module is used to deploy the first Actor network and the first Critic network to each UAV flight control system, and to create a mirror agent for each UAV. The mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay cache. The real-time observation information acquisition module is used to acquire the observation information of each UAV in real time. The cumulative reward value acquisition module is used to input the observation information of each UAV into the first Actor network and the mirror Actor network corresponding to each UAV after acquiring the observation information of each UAV, and generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; to acquire the first cumulative reward value of each UAV after driving each UAV to perform flight actions according to the corresponding first flight strategy; and to estimate the reward value and acquire the second cumulative reward value after each mirror agent performs virtual flight actions according to the mirror flight strategy. The network update module is used to update the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each UAV based on the difference between the first cumulative reward value and the corresponding second cumulative reward value of each UAV after each UAV executes the first flight strategy m times; and to update the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent based on the mirror experience replay cache after each UAV executes the first flight strategy once.

Images