Method and device for acquiring and online updating collaborative target tracking strategy of unmanned aerial vehicle cluster

Abstract

The invention discloses an unmanned aerial vehicle cluster collaborative target tracking strategy obtaining and online updating method and device, and the method comprises the steps: deploying a strategy network Actor network and an evaluation network Critic network to each unmanned aerial vehicle flight control system, and creating a mirror image agent for each unmanned aerial vehicle; observation information of each unmanned aerial vehicle is obtained in real time and input to a strategy Actor network and a mirror image Actor network of each unmanned aerial vehicle, and a corresponding flight strategy is generated. After the unmanned aerial vehicle is driven to execute a flight action according to the flight strategy, a first accumulated reward value of the unmanned aerial vehicle is obtained, and after the mirror image agent executes a virtual flight action according to the mirror image flight strategy, a reward value is estimated and a second accumulated reward value is obtained. And after the unmanned aerial vehicle is driven to execute a preset number of flight strategies each time, updating the parameters of the strategy network Actor network, the evaluation network Critic network and the mirror image network of the unmanned aerial vehicle. According to the invention, the accuracy of the tracking strategy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous cooperative decision-making and control of unmanned aerial vehicle (UAV) swarms, and particularly to a method and device for obtaining and online updating a cooperative target tracking strategy for UAV swarms. Background Art

[0002] In the task of cooperative target search and tracking for UAV swarms, UAVs must adjust their positions based on their own perception information so that the target can always be within the perception and control range of the UAV swarm. However, since the number and initial positions of the targets are usually unknown, and at the same time the motion laws and intentions of the targets are not clear, the UAV swarm faces great challenges when performing the target tracking task. To adapt to these uncertainties, the UAV swarm must adjust its position in real time according to its local observation information in order to track multiple dynamic targets simultaneously. This process requires efficient swarm cooperation, precise decision-making and control capabilities. In recent years, the method of UAV swarm target tracking based on reinforcement learning has been gradually widely studied. Through reinforcement learning, the agent can continuously optimize its strategy during the interaction with the environment, thereby improving the task execution efficiency and tracking accuracy of the swarm.

[0003] Existing target tracking methods based on reinforcement learning usually rely on simplified modeling of the actual environment to construct the training environment, while the actual environment is often more complex than the training environment. Directly applying the strategy obtained by the agent in the training environment to the actual control system may lead to a decline in the control effect and even produce sub-optimal decisions. Summary of the Invention

[0004] Embodiments of the present invention provide a method and device for obtaining and online updating a cooperative target tracking strategy for UAV swarms. By implementing the present invention, the generalization ability of the strategy can be improved, and the problem of sub-optimal decisions caused by the difference between the training environment and the actual environment can be avoided.

[0005] An embodiment of the present invention provides a method for obtaining and online updating a cooperative target tracking strategy for UAV swarms, including:

[0006] Obtaining an initial network generated by training through a preset target tracking simulation environment; wherein, the initial network includes a first Actor network and a first Critic network;

[0007] Deploying the initial network to each UAV flight control system, and creating a mirror agent for each UAV; the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay buffer;

[0008] Real-time obtaining the observation information of each UAV;

[0009] After obtaining the observation information of each UAV, input the observation information into the first Actor network and the mirrored Actor network corresponding to each UAV, and generate the first flight strategy and the corresponding mirrored flight strategy for each UAV respectively; after obtaining the first cumulative reward value of each UAV after driving each UAV to perform flight actions according to the corresponding first flight strategy; after each mirrored agent performs virtual flight actions according to the mirrored flight strategy, each mirrored agent estimates the reward value and obtains the second cumulative reward value;

[0010] After driving each UAV to execute the first flight strategy m times, update the parameters of the first Actor network, the first Critic network, the mirrored Actor network, and the mirrored Critic network corresponding to each UAV according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each UAV;

[0011] After driving each UAV to execute the first flight strategy once, update the parameters of the mirrored Actor network and the mirrored Critic network corresponding to each mirrored agent according to the mirrored experience replay buffer.

[0012] Furthermore, the obtaining of the initial network trained through a preset target tracking simulation environment includes:

[0013] Obtain the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the task area;

[0014] Construct a target tracking simulation environment according to the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the tracking area; wherein, the target tracking simulation environment includes an environment constraint model, a target model, and a UAV model;

[0015] In the target tracking simulation environment, define the state space, action space, reward function, and training objective of the agent based on the Markov decision process framework; construct an Actor network and a Critic network; construct an experience replay buffer;

[0016] Based on the Advantage Actor-Critic algorithm, regard the UAV as an agent, and interact with the target tracking simulation environment through the agent to train the agent until a preset number of training times is reached, generating a trained policy network Actor network and an evaluation network Critic network; wherein, during the training process, the agent generates state transition experience samples through interaction with the simulation environment and stores them in the experience replay buffer, and optimizes the network parameters of the Actor network and the Critic network by sampling the experiences in the experience replay buffer; all agents share the same set of Actor network and Critic network during the training process;

[0017] The trained policy network, the Actor network, is denoted as the first Actor network; the trained evaluation network, the Critic network, is denoted as the first Critic network; the first Actor network and the first Critic network are denoted as the initial network.

[0018] Further, the target model includes the motion pattern of the target.

[0019] The motion pattern of the target is specifically:

[0020]

[0021] where is the position coordinate of the target T i at time t; is the height of the ground point ; C cone is the number of equivalent cones in the mission area; is the height of the j-th cone; is the bottom radius of the j-th cone; is the coordinate of the center of the bottom surface of the j-th cone; is the target T i is the maximum speed on the X-axis; is the target T i is the maximum speed on the Y-axis; is the uniform distribution; is the normal distribution; D X is the lateral range of the tracking area; D Y is the longitudinal range of the tracking area.

[0022] Further, the UAV model includes the motion pattern of the UAV, the target tracking and locking ability range of the UAV, the maximum scanning and detection ability range of the UAV, the effective communication range of the UAV, and the maximum communication range of the UAV.

[0023] The motion pattern of the UAV is specifically:

[0024]

[0025] D(a,b) = sgn(a) - sgn(b)

[0026]

[0027] where is the position coordinate of the UAV U i at time t; s U is the unit flight step of the UAV; sgn is the sign function; is the forward movement instruction of the UAV in the X-axis direction; It is the instruction for the drone to move backward in the X-axis direction; It is the instruction for the drone to move forward in the Y-axis direction; It is the instruction for the drone to move backward in the Y-axis direction; It is the instruction for the drone to move forward in the Z-axis direction; It is the instruction for the drone to move backward in the Z-axis direction; It is the point The ground height of; h min It is the minimum flight height of the drone relative to the ground; h max It is the maximum flight height of the drone relative to the ground;

[0028] The target tracking and locking ability range of the drone is specifically:

[0029]

[0030] Among them, It is the target locking range of drone U i ; P max It is the radius of the drone's target locking range; β is the maximum angle of the drone's target locking range;

[0031] The maximum scanning and detection ability range of the drone is specifically:

[0032]

[0033] Among them, It is the maximum detection range of drone U i ; P dec It is the radius of the drone's maximum detection range; α is the maximum angle of the drone's maximum detection range;

[0034] The effective communication range of the drone is specifically:

[0035]

[0036] Among them, It is the effective communication range of drone U i ; C e It is the effective communication radius of the drone;

[0037] The maximum communication range of the drone is specifically:

[0038]

[0039] Among them, It is the maximum communication range of drone U i ; C max It is the maximum communication radius of the drone.

[0040] Furthermore, the state space includes the information of the own aircraft, the information of the targets within the target tracking and locking capability range, and the information of friendly aircraft within the effective communication range;

[0041] The information of the own aircraft is specifically:

[0042]

[0043] Among them, is the information of the own aircraft of the unmanned aircraft U i at time t;

[0044] The information of the targets within the target tracking and locking capability range is specifically:

[0045]

[0046] Among them, is the information of the target T within the target tracking and locking capability range of the unmanned aircraft U i at time t; k is the relative position of the unmanned aircraft U at time t relative to the target T i on the X-axis; k is the relative position of the unmanned aircraft U at time t relative to the target T i on the Y-axis; k is the relative position of the unmanned aircraft U at time t relative to the target T i on the Z-axis; k is the distance between the unmanned aircraft U and the target T i at time t; k is the speed of the target T at time t on the X-axis; k is the speed of the target T at time t on the Y-axis; k

[0047] The information of the friendly aircraft within the effective communication range is specifically:

[0048]

[0049] Among them, is the information of the friendly aircraft of the unmanned aircraft U i relative to the friendly aircraft U within the effective communication range j at time t; is the relative position of the unmanned aircraft U i relative to the friendly aircraft U within the effective communication range j on the X-axis at time t; Drone U i Friendly drone U within the relative effective communication range j Relative position on the Y-axis at time t; Drone U i Friendly drone U within the relative effective communication range j Relative position on the Z-axis at time t; For drone U i With friendly drone U j Distance at time t.

[0050] Furthermore, the reward function includes an environmental reward, a target reward, and a friendly-drone reward;

[0051] The environmental reward is specifically:

[0052]

[0053] Wherein, For drone U i Environmental reward at time t;

[0054] The target reward is specifically:

[0055]

[0056] Wherein, For drone U i Target reward at time t; For drone U i Relative to target T at time t k Target reward;

[0057] The friendly-drone reward is specifically:

[0058]

[0059] Wherein, For drone U i Friendly-drone reward at time t, For drone U i Friendly drone U within the relative effective communication range j Friendly-drone reward at time t.

[0060] Furthermore, in the Advantage Actor-Critic algorithm, the drone is regarded as an agent, and the agent interacts with the target-tracking simulation environment, and the agent is trained until a preset number of training times is reached, generating a trained policy network Actor network and an evaluation network Critic network, including:

[0061] Initialize the experience replay buffer and the network parameters of the policy network (Actor network) and the evaluation network (Critic network);

[0062] Repeat the parameter update operation until the number of training times is reached, and generate the trained policy network (Actor network) and evaluation network (Critic network);

[0063] The parameter update operation includes:

[0064] For each agent, obtain the current observation information, and generate corresponding state transition experience samples according to the interaction process between the agent and the target tracking simulation environment; wherein, the observation information includes the local information defined in the state space, the target information within the target tracking and locking ability range, and the information of friendly aircraft within the effective communication range; the state transition experience samples include the observation information at the current moment, the executed action, the environmental feedback reward, and the observation information at the next moment;

[0065] Store the state transition experience samples into the experience replay buffer and update the current experience replay buffer;

[0066] Calculate the temporal error and sampling probability of each experience in the current experience replay buffer;

[0067] For each agent, randomly sample K experience samples from the current experience replay buffer according to the probability; calculate the policy gradient, and use the gradient descent method to update the network parameters of the policy network (Actor network); update the network parameters of the evaluation network (Critic network) according to the calculated temporal error value;

[0068] Under the condition that the number of training times is reached, record the current policy network (Actor network) and evaluation network (Critic network) as the trained policy network and evaluation network.

[0069] Further, the updating of the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each unmanned aerial vehicle according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each unmanned aerial vehicle includes:

[0070] Record the unmanned aerial vehicles with the first cumulative reward value greater than the corresponding second cumulative reward value as the first type of unmanned aerial vehicles, and the remaining unmanned aerial vehicles as the second type of unmanned aerial vehicles; update the mirror Actor network and the mirror Critic network of the first type of unmanned aerial vehicles to the corresponding first Actor network and the first Critic network according to a preset first probability; update the first Actor network and the first Critic network of the second type of unmanned aerial vehicles to the corresponding mirror Actor network and the mirror Critic network according to a preset second probability.

[0071] Furthermore, updating the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent according to the mirror experience replay buffer includes:

[0072] For each mirror agent, calculate the temporal error and sampling probability of each experience in the corresponding mirror experience replay buffer; draw K second experiences according to the probability from the corresponding mirror experience replay buffer; calculate the policy gradient, and use the gradient descent method to update the network parameters of the corresponding mirror Actor network and the network parameters of the corresponding mirror Critic network;

[0073] Among them, the network parameters of the mirror Actor network are updated according to the following method:

[0074]

[0075] δ t =r t +γv ω (o t+1 )-v ω (o t )

[0076]

[0077] p t =|δ t |+∈

[0078] Among them, θ is the network parameter of the mirror Actor network; η a is the learning rate; Δ a is the gradient change amount of the mirror Actor network; K is the sampling size; w t is the importance sampling weight; δ t is the temporal difference value; is the gradient of the mirror Actor network; r t is the immediate reward; γ is the discount factor; v ω (o t+1 ) is the value function of the next state; v ω (o t ) is the value function of the current state; N is the capacity of the mirror experience replay buffer ; P(b t ) is the sampling probability of experience b t ; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t is the priority;

[0079] The network parameters of the mirror Critic network are updated according to the following method:

[0080]

[0081] Among them, ω is the network parameter of the mirror Critic network; Δ c is the gradient update amount of the mirror Critic network; is the gradient of the mirror Critic network.

[0082] Based on the above method item embodiments, the present invention correspondingly provides apparatus item embodiments.

[0083] An embodiment of the present invention provides an apparatus for obtaining and online updating a cooperative target tracking strategy for a drone swarm, including: an initial network obtaining module, a network deployment module, a real-time observation information obtaining module, an accumulated reward value obtaining module, and a network updating module;

[0084] The initial network obtaining module is configured to obtain a first Actor network and a first Critic network generated by training through a preset target tracking simulation environment;

[0085] The network deployment module is configured to deploy the first Actor network and the first Critic network to each drone flight control system, and create a mirror agent for each drone, where the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay buffer;

[0086] The real-time observation information obtaining module is configured to obtain the observation information of each drone in real time;

[0087] The accumulated reward value obtaining module is configured to, after obtaining the observation information of each drone, input the observation information into the first Actor network and the mirror Actor network corresponding to each drone, respectively generate the first flight strategy and the corresponding mirror flight strategy corresponding to each drone; obtain the first accumulated reward value of each drone after driving each drone to execute flight actions according to the corresponding first flight strategy; after each mirror agent executes virtual flight actions according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second accumulated reward value;

[0088] The network updating module is configured to, after driving each drone to execute the first flight strategy m times, update the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each drone according to the difference between the first accumulated reward value and the corresponding second accumulated reward value of each drone; after driving each drone to execute the first flight strategy once, update the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent according to the mirror experience replay buffer.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] The embodiment of the present invention provides a method and device for obtaining and online updating a cooperative target tracking strategy for an unmanned aerial vehicle (UAV) cluster. The method includes obtaining an initial network generated by training in a target tracking simulation environment, including a first Actor network and a first Critic network, and deploying it to the UAV flight control system. A mirror agent is created for each UAV, and the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay buffer. During the execution of the task, the observation information of each UAV is obtained in real time, and by inputting the observation information into the first Actor network and the mirror Actor network of the UAV, corresponding flight strategies are generated. By executing these flight strategies, the cumulative reward value is calculated and differentially updated to optimize the network parameters of the agent.

[0091] By deploying the initial strategy to the UAV flight control system and introducing a mirror agent for online correction and real-time adjustment of the strategy, the agent can better adapt to the complex actual environment during the execution of the task, improve the generalization ability of the strategy, and avoid the sub-optimal decision-making problem caused by the difference between the training environment and the actual environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 FIG. is a schematic flow chart of a method for obtaining and online updating a cooperative target tracking strategy for a UAV cluster provided by an embodiment of the present invention.

[0093] Figure 2 FIG. is a schematic diagram of the communication and sensing capabilities of a UAV provided by an embodiment of the present invention.

[0094] Figure 3 FIG. is a schematic diagram of the deployment of a policy network provided by an embodiment of the present invention.

[0095] Figure 4 FIG. is a schematic flow chart of a method for obtaining and online updating a cooperative target tracking strategy for a UAV cluster provided by another embodiment of the present invention.

[0096] Figure 5 FIG. is a schematic structural diagram of a device for obtaining and online updating a cooperative target tracking strategy for a UAV cluster provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0097] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0098] As Figure 1 shown, an embodiment of the present invention provides a method for obtaining and online updating a collaborative target tracking strategy for an unmanned aerial vehicle (UAV) cluster, which at least includes the following steps:

[0099] Step S1, obtain an initial network generated by training through a preset target tracking simulation environment.

[0100] Specifically, the initial network includes a first Actor network and a first Critic network;

[0101] In a preferred embodiment, the obtaining of the initial network generated by training through a preset target tracking simulation environment includes:

[0102] Obtain the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the mission area;

[0103] According to the hardware parameters of the UAV, the motion parameters of the target to be tracked, and the location information of the tracking area, construct a target tracking simulation environment; wherein, the target tracking simulation environment includes an environmental constraint model, a target model, and a UAV model;

[0104] In the target tracking simulation environment, define the state space, action space, reward function, and training objective of the agent based on the Markov decision process framework; construct an Actor network and a Critic network; construct an experience replay buffer;

[0105] Based on the Advantage Actor-Critic algorithm, regard the UAV as an agent, and through the interaction between the agent and the target tracking simulation environment, train the agent until a preset number of training times is reached, generating a trained policy network Actor network and an evaluation network Critic network; wherein, during the training process, the agent generates state transition experience samples through interaction with the simulation environment and stores them in the experience replay buffer, and optimizes the network parameters of the Actor network and the Critic network by sampling the experiences in the experience replay buffer; all agents share the same set of Actor network and Critic network during the training process;

[0106] Record the trained policy network Actor network as the first Actor network; record the trained evaluation network Critic network as the first Critic network; record the first Actor network and the first Critic network as the initial network.

[0107] Specifically, obtain the hardware parameters of the drone, the motion parameters of the target to be tracked, and the location information of the mission area. These hardware parameters include the flight ability, sensor performance, communication ability, etc. of the drone; the motion parameters of the target include the speed, acceleration, motion trajectory, and prediction model of the target, etc.; the location information of the mission area involves the size, shape, geographical features, obstacle information, etc. of the tracking area.

[0108] Based on the information obtained, construct a target tracking simulation environment. The construction of the simulation environment includes three main parts: an environmental constraint model, a target model, and a drone model. The environmental constraint model simulates the limiting conditions within the mission area; the target model describes the motion laws, behavior patterns of the target to be tracked, and its interaction with the environment, ensuring that the behavior and trajectory of the target conform to the actual situation; the drone model includes the dynamic characteristics, control algorithms, and communication systems of the drone, etc., simulating the flight behavior and collaborative working mode of the drone when performing tasks. These models work together to form a complete target tracking simulation environment that can truly reflect the performance of the drone in actual tasks.

[0109] The problem of cooperative target tracking of a drone swarm with bounded communication range and observation range can be described as follows: The target swarm contains n targets that can move randomly within the mission area, and the drone swarm is difficult to directly obtain all target information such as the number, position distribution, and speed. The drone can continuously maintain a communication connection with the drones within the effective communication range and share its own position information and action information with each other. The drone makes action decisions based on its own information, the information of the targets within the tracking and locking range, and the information of the friendly drones within the effective communication range, so that the targets can continuously be within the tracking and locking range of the drone swarm. At the same time, the drone swarm does not know how many targets need to be continuously observed to complete the task and can only consider that the more targets observed simultaneously, the better.

[0110] In a preferred embodiment, the environmental constraint model includes the tracking area boundary and geomorphic model factors;

[0111] The target model includes the motion range of the target and the motion mode of the target;

[0112] The drone model includes the motion mode of the drone, the target tracking and locking ability range of the drone, the maximum scanning and detection ability range of the drone, the effective communication range of the drone, and the maximum communication range of the drone;

[0113] The state space includes the information of the local aircraft, the information of the targets within the target tracking and locking ability range, and the information of the friendly drones within the effective communication range;

[0114] The action space includes the forward and backward movements of the UAV along the X-axis, Y-axis, and Z-axis.

[0115] The reward function includes environmental reward, target reward, and friendly aircraft reward.

[0116] The tracking area boundary is specifically:

[0117] D X ×D Y ×D Z

[0118] where D X is the lateral range of the tracking area; D Y is the longitudinal range of the tracking area; D Z is the height limit of the tracking area.

[0119] Optionally, the tracking area boundary can be set as a three-dimensional space of 2000×2000×1000. The landforms in the mission area are non-traversable areas for the UAV, but the target can move on the surface of the landforms.

[0120] The landform model factor is specifically:

[0121]

[0122] where z is the height coordinate of the bottom point (x, y) of the cone; (x 0 , y 0 ) is the coordinate of the center of the bottom circle of the cone; r is the radius of the bottom circle of the cone; h cone is the height of the cone, that is, the vertical distance from the vertex to the center of the bottom; here, the landform objects in the environment are modeled as cones.

[0123] The movement range of the target is the bottom surface of the cone.

[0124] The movement mode of the target is specifically:

[0125]

[0126] where is the position coordinate of the target T i at time t; is the height of the ground point ; C cone is the number of equivalent cones in the mission area; is the height of the j-th cone; is the bottom radius of the j-th cone; is the bottom center coordinate of the j-th cone; is the target T iThe maximum speed on the X-axis; is the target T i The maximum speed on the Y-axis; is a uniform distribution; is a normal distribution; D X is the lateral range of the tracking area; D Y is the longitudinal range of the tracking area;

[0127] Specifically, if the target is within the target locking range of any one of the drones in the drone swarm at time t, it is considered that the target is tracked by the drone swarm at time t.

[0128] The motion mode of the drone is specifically:

[0129]

[0130] D(a,b) = sgn(a) - sgn(b)

[0131]

[0132] Among them, is the drone U i The position coordinate at time t; s U is the unit flight step of the drone; sgn is the sign function; is the forward movement instruction of the drone in the X-axis direction; is the backward movement instruction of the drone in the X-axis direction; is the forward movement instruction of the drone in the Y-axis direction; The backward movement instruction of the drone in the Y-axis direction; is the forward movement instruction of the drone in the Z-axis direction; is the backward movement instruction of the drone in the Z-axis direction; is the point The ground height of; h min is the minimum flight height of the drone relative to the ground; h max is the maximum flight height of the drone relative to the ground;

[0133] The target tracking and locking ability range of the drone is specifically:

[0134]

[0135] Among them, is the drone U i The target locking range of; P max is the radius of the drone target locking range; β is the maximum angle of the drone target locking range;

[0136] The maximum scanning and detection capability range of the drone is specifically:

[0137]

[0138] Among them, is the maximum detection range of drone U i ; P dec is the radius of the maximum detection range of the drone; α is the maximum angle of the maximum detection range of the drone.

[0139] The effective communication range of the drone is specifically:

[0140]

[0141] Among them, is the effective communication range of drone U i ; C e is the effective communication radius of the drone.

[0142] The maximum communication range of the drone is specifically:

[0143]

[0144] Among them, is the maximum communication range of drone U i ; C max is the maximum communication radius of the drone.

[0145] It should be explained here that if the spatial distance d i between drone U j and drone U i,j is not greater than the effective communication radius C e , then drone U i can communicate and share position and action information with drone U j ; if the spatial distance d i between drone U j and drone U i,j is greater than the effective communication radius C e , but less than the maximum communication radius C max , then drone U i can only communicate and share position information with drone U j ; if the spatial distance d i between drone U j and drone U i,j is greater than the maximum communication radius C max , then drone U i cannot communicate with drone U j . Specifically, the schematic diagram of the drone's communication and sensing capabilities is asFigure 2 as shown

[0146] The local information mentioned above is specifically:

[0147]

[0148] Among them, is the local information of the unmanned aerial vehicle U i at time t;

[0149] The target information within the target tracking and locking capability range is specifically:

[0150]

[0151] Among them, is the target information of the unmanned aerial vehicle U i regarding the target T within the target tracking and locking capability range k at time t; is the relative position of the unmanned aerial vehicle U i relative to the target T k on the X-axis at time t; is the relative position of the unmanned aerial vehicle U i relative to the target T k on the Y-axis at time t; is the relative position of the unmanned aerial vehicle U i relative to the target T k on the Z-axis at time t; is the distance between the unmanned aerial vehicle U i and the target T k at time t; is the speed of the target T k on the X-axis at time t; is the speed of the target T k on the Y-axis at time t;

[0152] The information of friendly aircraft within the effective communication range is specifically:

[0153]

[0154] Among them, is the information of the friendly aircraft of the unmanned aerial vehicle U i relative to the friendly aircraft U within the effective communication range j at time t; is the relative position of the unmanned aerial vehicle U i relative to the friendly aircraft U within the effective communication range j on the X-axis at time t; The unmanned aerial vehicle U i relative to the friendly aircraft U within the effective communication range jRelative position on the Y-axis at time t; Drone U i Friendly drone U within the relative effective communication range j Relative position on the Z-axis at time t; For drone U i With friendly drone U j Distance at time t.

[0155] The environmental reward, specifically:

[0156]

[0157] Wherein, For drone U i Environmental reward at time t;

[0158] The target reward, specifically:

[0159]

[0160] Wherein, For drone U i Target reward at time t; For drone U i At time t relative to target T k Target reward;

[0161] The friendly drone reward, specifically:

[0162]

[0163]

[0164] Wherein, For drone U i Friendly drone reward at time t, For drone U i Relative to friendly drone U within the effective communication range j Friendly drone reward at time t.

[0165] It should be noted here that the swarm reward of the drone swarm is the sum of the rewards obtained by all drones, and the drone reward includes the environmental reward, target reward, and friendly drone reward in the reward function.

[0166] The training objective, specifically:

[0167]

[0168] s.t.

[0169]

[0170] Wherein, is the target T i at the position at time t; is the UAV swarm; T is the time sequence length.

[0171] It should be explained here that in the constraints of the above training objectives, the first and second items are to constrain the movement range of the target, and the third item represents the performance constraints of the UAVs.

[0172] In a preferred embodiment, based on the Advantage Actor-Critic algorithm, the UAVs are regarded as agents, and the agents interact with the target tracking simulation environment to train the agents until the preset number of training times is reached, generating the trained policy network Actor network and evaluation network Critic network, including:

[0173] Initialize the experience replay buffer and the network parameters of the policy network Actor network and evaluation network Critic network;

[0174] Repeat the parameter update operation until the number of training times is reached, generating the trained policy network Actor network and evaluation network Critic network;

[0175] The parameter update operation includes:

[0176] For each agent, obtain the current observation information, and generate corresponding state transition experience samples according to the interaction process between the agent and the target tracking simulation environment; wherein, the observation information includes the local information defined in the state space, the target information within the target tracking and locking ability range, and the information of friendly UAVs within the effective communication range; the state transition experience samples include the observation information at the current moment, the executed actions, the environmental feedback rewards, and the observation information at the next moment;

[0177] Store the state transition experience samples into the experience replay buffer and update the current experience replay buffer;

[0178] Calculate the temporal error and sampling probability of each experience in the current experience replay buffer;

[0179] For each agent, randomly draw K experience samples from the current experience replay buffer according to the probability; calculate the policy gradient, and update the network parameters of the policy network Actor network using the gradient descent method; update the network parameters of the evaluation network Critic network according to the calculated temporal error value;

[0180] Under the condition of reaching the number of training times, record the current policy network Actor network and evaluation network Critic network as the trained policy network and evaluation network.

[0181] Specifically, the network is trained using parameter sharing. After the training is completed, a target tracking strategy with a certain degree of generality is obtained.

[0182] Optionally, calculating the sampling probability of each piece of experience in the current experience replay buffer includes:

[0183] Calculating the temporal difference value of each piece of experience in the current experience replay buffer to generate the temporal difference values of each experience;

[0184] Generating the priority of each experience according to the temporal difference values of each experience and a preset constant;

[0185] Accumulating the priorities of each experience to generate the cumulative priority;

[0186] Determining the sampling probability of each experience according to the ratio of the priority of each experience to the cumulative priority.

[0187] Among them, the temporal difference value is calculated by the following formula:

[0188] δ t =r t +γv ω (o t+1 )-v ω (o t )

[0189] In the formula, δ t is the temporal difference value; r t is the immediate reward; γ is the discount factor; v ω (o t+1 ) is the value function of the next state; v ω (o t ) is the value function of the current state.

[0190] The priority is calculated by the following formula:

[0191] p t =|δ t |+∈

[0192] In the formula, p t is the priority; ∈ is a constant;

[0193] The sampling probability is determined by the following formula:

[0194]

[0195] In the formula, P(b t ) is the sampling probability of experience b t ; l is the compensation parameter;

[0196] Step S2: Deploy the initial network to each UAV flight control system and create a mirror agent for each UAV; the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay buffer. The deployment relationship is as shown in Figure 3 shown.

[0197] Here, it should be explained that each UAV is equipped with a microcomputer, and the mirror agent can be deployed in the microcomputer.

[0198] Step S3: Obtain the observation information of each UAV in real time.

[0199] Step S4: After obtaining the observation information of each UAV, input the observation information into the first Actor network and the mirror Actor network corresponding to each UAV, and generate the first flight strategy and the corresponding mirror flight strategy for each UAV respectively; after obtaining the first cumulative reward value of each UAV after driving each UAV to execute the flight action according to the corresponding first flight strategy; after each mirror agent executes the virtual flight action according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value.

[0200] Specifically, after obtaining the observation information of each UAV each time, first input the observation information into the first Actor network and the mirror Actor network corresponding to each UAV. Through the calculations of these two networks, the first flight strategy and the corresponding mirror flight strategy for each UAV are generated respectively. Among them, the first flight strategy represents the action plan selected by the UAV in the current state, and the mirror flight strategy is a similar strategy generated for the corresponding mirror agent, aiming to enhance the generalization ability of the system through the simulation training of the mirror agent.

[0201] Subsequently, by driving each UAV to execute the corresponding first flight strategy, during the actual execution process, the UAV will adjust its own behavior according to the environmental feedback. After execution, each UAV will obtain a first cumulative reward value according to the deviation from the target and environmental conditions. This reward value reflects the performance of the UAV in a specific task and is evaluated based on the reward function.

[0202] At the same time, when the mirror agent interacts with the actual environment, it will also execute the corresponding mirror flight strategy. Through this training simulation in the virtual environment, the mirror agent will obtain a second cumulative reward value, which indicates the effect of the mirror agent executing the mirror flight strategy in the actual environment and serves as a reference and assistance for the execution effect of the real UAV flight strategy.

[0203] These reward function values are used to update the networks of the UAV and the mirror agent, ensuring that each agent gradually improves the accuracy and effectiveness of its strategy during the continuous feedback and learning process.

[0204] Step S5. After each time of driving each drone to execute the first flight strategy m times, update the parameters of the first Actor network, the first Critic network, the mirrored Actor network, and the mirrored Critic network corresponding to each drone according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone.

[0205] In a preferred embodiment, the updating the parameters of the first Actor network, the first Critic network, the mirrored Actor network, and the mirrored Critic network corresponding to each drone according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone includes:

[0206] Mark the drones with the first cumulative reward value greater than the corresponding second cumulative reward value as the first type of drones, and mark the remaining drones as the second type of drones; update the mirrored Actor network and the mirrored Critic network of the first type of drones to the corresponding first Actor network and the first Critic network respectively according to a preset first probability; update the first Actor network and the first Critic network of the second type of drones to the corresponding mirrored Actor network and the mirrored Critic network respectively according to a preset second probability.

[0207] Specifically, the first probability and the second probability can be flexibly set according to the actual situation. The execution of m times is only for illustration, and the specific number of times can be flexibly set according to the actual situation.

[0208] Step S6. After each time of driving each drone to execute the first flight strategy once, update the parameters of the mirrored Actor network and the mirrored Critic network corresponding to each mirrored agent according to the mirrored experience replay buffer.

[0209] In a preferred embodiment, the updating the parameters of the mirrored Actor network and the mirrored Critic network corresponding to each mirrored agent according to the mirrored experience replay buffer includes:

[0210] For each mirrored agent, calculate the temporal error and the sampling probability of each piece of experience in the corresponding mirrored experience replay buffer; extract K second experiences from the corresponding mirrored experience replay buffer according to the probability; calculate the policy gradient of the second experiences, and use the gradient descent method to update the network parameters of the corresponding mirrored Actor network and the network parameters of the corresponding mirrored Critic network;

[0211] Among them, the network parameters of the mirrored Actor network are updated according to the following method:

[0212]

[0213] δ t = r t + γv ω (o t+1 ) - v ω (o t )

[0214]

[0215] p t = |δ t | + ∈

[0216] where θ is the network parameter of the mirror Actor network; η a is the learning rate; Δ a is the gradient change of the mirror Actor network; K is the sampling size; w t is the importance sampling weight; δ t is the time difference value; is the gradient of the mirror Actor network; r t is the immediate reward; γ is the discount factor; v ω (o t+1 ) is the value function of the next state; v ω (o t ) is the value function of the current state; N is the capacity of the mirror experience replay buffer P(b t ) is the sampling probability of the experience b t ; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t is the priority;

[0217] The network parameters of the mirror Critic network are updated in the following way:

[0218]

[0219] where ω is the network parameter of the mirror Critic network; Δ c is the gradient update of the mirror Critic network; is the gradient of the mirror Critic network.

[0220] Specifically, in a preferred embodiment, the steps for obtaining and online updating the UAV swarm cooperative target tracking strategy are as Figure 4 shown. The execution of n times is only for illustrative purposes, and the specific number of times can be flexibly set according to the actual situation.

[0221] It should be noted that the online update of the UAV swarm cooperative target tracking strategy in the present invention is to achieve online fine-tuning and correction of the UAV swarm cooperative target tracking strategy. The method is also applicable to the field of target search, not just limited to the field of target tracking. During the process of strategy fine-tuning and correction, each UAV loads an executable strategy and generates a mirror agent. During the execution of the target tracking task, by comparing the cumulative reward value of the currently executed strategy with the cumulative reward value obtained by the mirror agent, the currently executed strategy is continuously updated; this method can achieve autonomous cooperative decision-making and control of UAV swarms in target tracking scenarios and has good robustness.

[0222] The Advantage Actor-Critic algorithm based on probability sampling of experiences in the present invention can be called a distributed Advantage Actor-Critic algorithm based on the prioritized experience replay mechanism. The distributed Advantage Actor-Critic algorithm based on the prioritized experience replay mechanism can accelerate the training process and improve the learning efficiency by preferentially selecting valuable experiences. At the same time, the distributed architecture enables multiple agents to be trained in parallel, further improving the speed and generalization ability of policy learning.

[0223] When the present invention is in simulation training, the agents are trained in a parameter-sharing manner to obtain a strategy with a certain degree of generality, which speeds up the training speed of the agents;

[0224] During strategy fine-tuning, each UAV has an independent mirror agent, which has its own mirror Actor network and mirror Critic network. By comparing the cumulative reward obtained by the actual policy network with the cumulative reward obtained by the virtual policy network, the correction and update of the executed policy are realized, and the optimal target search and tracking strategy is obtained;

[0225] The training process does not require the assistance of human drivers. Each UAV can adaptively fine-tune, correct, and update the strategy according to the data in the actual scenario, enhancing the diversity of the UAV swarm target search and tracking strategy and improving the generalization ability of the strategy to the environment.

[0226] Based on the above method item embodiments, the present invention correspondingly provides device item embodiments.

[0227] As Figure 5 shown, an embodiment of the present invention provides a device for obtaining and online updating the UAV swarm cooperative target tracking strategy, including: an initial network acquisition module, a network deployment module, a real-time observation information acquisition module, a cumulative reward value acquisition module, and a network update module;

[0228] The initial network acquisition module is used to acquire a first Actor network and a first Critic network generated by training through a preset target tracking simulation environment;

[0229] The network deployment module is used to deploy the first Actor network and the first Critic network to each UAV flight control system, and create a mirror agent for each UAV. The mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience replay buffer;

[0230] The observation information real-time acquisition module is used to acquire the observation information of each UAV in real time;

[0231] The cumulative reward value acquisition module is used to, after acquiring the observation information of each UAV, input the observation information into the first Actor network and the mirror Actor network corresponding to each UAV, and generate the first flight strategy and the corresponding mirror flight strategy corresponding to each UAV respectively; after obtaining the first cumulative reward value of each UAV after driving each UAV to execute flight actions according to the corresponding first flight strategy; after each mirror agent executes virtual flight actions according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value;

[0232] The network update module is used to, after driving each UAV to execute the first flight strategy m times, update the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each UAV according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each UAV; after driving each UAV to execute the first flight strategy once, update the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent according to the mirror experience replay buffer.

[0233] It should be noted that the embodiments of the devices described above correspond to the above embodiments of the present invention, and can implement the online update method of the UAV cluster cooperative target tracking strategy network described in any one of the above of the present invention. In addition, the embodiments of the above devices are only illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement without creative efforts.

[0234] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0235] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A method for acquiring and online updating a UAV cluster cooperative target tracking strategy, characterized in that: include: Acquire an initial network generated by training in a preset target tracking simulation environment; wherein the initial network includes a first Actor network and a first Critic network; Deploy the initial network to each drone flight control system and create a mirror agent for each drone; the mirror agent includes a mirror actor network, a mirror critic network and a mirror experience playback cache; Obtain observation information of each drone in real time; After each observation information of each drone is obtained, the observation information is input into the first Actor network and the mirror Actor network corresponding to each drone, and a first flight strategy and a corresponding mirror flight strategy corresponding to each drone are generated respectively; after each drone is driven to perform a flight action according to the corresponding first flight strategy, a first cumulative reward value of each drone is obtained; after each mirror agent performs a virtual flight action according to the mirror flight strategy, each mirror agent estimates a reward value and obtains a second cumulative reward value; After each drone is driven to execute the first flight strategy m times, the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each drone are updated according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone; After each drone is driven to execute the first flight strategy once, the cache is replayed based on the mirror experience, and the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent are updated.

2. The method for acquiring and online updating the UAV cluster collaborative target tracking strategy according to claim 1, characterized in that: The obtaining of an initial network generated by training in a preset target tracking simulation environment comprises: Obtain the hardware parameters of the drone, the motion parameters of the target to be tracked, and the location information of the mission area; According to the hardware parameters of the UAV, the motion parameters of the target to be tracked and the position information of the tracking area, a target tracking simulation environment is constructed; wherein the target tracking simulation environment includes an environmental constraint model, a target model and a UAV model; In the target tracking simulation environment, the state space, action space, reward function and training target of the intelligent agent are defined based on the Markov decision process framework; the Actor network and the Critic network are constructed; and the experience replay cache is constructed; Based on the Advantage Actor-Critic algorithm, the UAV is regarded as an intelligent agent, and the intelligent agent interacts with the target tracking simulation environment, and the intelligent agent is trained until a preset number of training times is reached, and a trained policy network Actor network and an evaluation network Critic network are generated; wherein, during the training process, the intelligent agent generates state transition experience samples by interacting with the simulation environment and stores them in the experience replay cache, and optimizes the network parameters of the Actor network and the Critic network by sampling the experience in the experience replay cache; during the training process, all intelligent agents share the same set of Actor networks and Critic networks; The trained policy network Actor network is recorded as the first Actor network; the trained evaluation network Critic network is recorded as the first Critic network; the first Actor network and the first Critic network are recorded as the initial network.

3. The method for acquiring and online updating the UAV cluster collaborative target tracking strategy as claimed in claim 2, characterized in that: The target model includes a motion pattern of the target; The movement mode of the target is specifically: in, Target T i Position coordinates at time t; For ground point Height; C cone is the number of equivalent cones in the task area; is the height of the jth cone; The base radius of the jth cone; The coordinates of the center of the base of the j-th cone; Target T i Maximum speed on the X axis; Target T i Maximum speed on the Y axis; For uniform distribution; is a normal distribution; D X is the lateral extent of the tracking area; D Y is the vertical extent of the tracking area.

4. The method for acquiring and online updating the UAV cluster cooperative target tracking strategy as claimed in claim 3, characterized in that: The drone model includes the drone's motion mode, the drone's target tracking and locking capability range, the drone's maximum scanning and detection capability range, the drone's effective communication range, and the drone's maximum communication range; The motion mode of the UAV is specifically: D(a,b)=sgn(a)-sgn(b) in, For UAV i Position coordinates at time t; s U is the unit flight step of the UAV; sgn is the sign function; Instructs the drone to move forward along the X-axis. Instructs the drone to move backward in the X-axis direction; Instructs the drone to move forward along the Y axis. The drone moves backward in the Y-axis direction; Instructs the drone to move forward in the Z-axis direction; Instructs the drone to move backward in the Z-axis direction; For point Height from the ground; h min h is the minimum flight altitude of the UAV relative to the ground; max is the maximum flight altitude of the drone relative to the ground; The target tracking and locking capability range of the UAV is specifically as follows: in, For UAV i Target lock range; P max is the radius of the target locking range of the UAV; β is the maximum angle of the target locking range of the UAV; The maximum scanning detection capability range of the UAV is specifically: in, For UAV i The maximum detection range of P dec is the radius of the maximum detection range of the UAV; α is the maximum angle of the maximum detection range of the UAV; The effective communication range of the UAV is specifically: in, For UAV i The effective communication range of C e is the effective communication radius of the UAV; The maximum communication range of the UAV is specifically: in, For UAV i The maximum communication range of C max is the maximum communication radius of the UAV.

5. The method for acquiring and online updating the UAV cluster collaborative target tracking strategy as claimed in claim 4, characterized in that: The state space includes the aircraft information, target information within the target tracking and locking capability range, and friendly aircraft information within the effective communication range; The local machine information is specifically: in, For UAV i The local information at time t; The target information within the target tracking and locking capability range is specifically: in, For UAV i About the target tracking and locking capability within the target T k Target information at time t; For UAV i At time t relative to the target T k Relative position on the X axis; For UAV i At time t relative to the target T k Relative position on the Y axis; For UAV i At time t relative to the target T k Relative position on the Z axis; For UAV i With target T k The distance between the times t; Target T k The velocity on the X axis at time t; Target T k The velocity on the Y axis at time t; The friendly machine information within the effective communication range is specifically: in, For UAV i Friendly aircraft U within the effective communication range j Friendly aircraft information at time t; For UAV i Friendly aircraft U within the effective communication range j The relative position on the X-axis at time t; UAV i Friendly aircraft U within the effective communication range j The relative position on the Y axis at time t; UAV i Friendly aircraft U within the effective communication range j The relative position on the Z axis at time t; For UAV i With friends j The distance at time t.

6. The method for acquiring and online updating the UAV cluster cooperative target tracking strategy as claimed in claim 5, characterized in that: The reward function includes an environment reward, a target reward, and a friendly machine reward; The environmental rewards are specifically: in, For UAV i Environmental rewards at time t; The target rewards are specifically: in, For UAV i Target reward at time t; For UAV i At time t relative to the target T k Target rewards; The friendly aircraft rewards are specifically: in, For UAV i The friendly machine reward at time t, For UAV i Friendly aircraft U within the effective communication range j Friendly machine reward at time t.

7. The method for acquiring and online updating the UAV cluster cooperative target tracking strategy as claimed in claim 6, characterized in that: The Advantage Actor-Critic algorithm is based on treating the UAV as an intelligent agent, and the intelligent agent interacts with the target tracking simulation environment, and the intelligent agent is trained until a preset number of training times is reached, and a trained policy network Actor network and an evaluation network Critic network are generated, including: Initialize the experience replay cache and the network parameters of the policy network Actor network and the evaluation network Critic network; Repeat the parameter update operation until the training times are reached, and generate the trained policy network Actor network and evaluation network Critic network; The parameter updating operation includes: For each intelligent agent, current observation information is obtained, and corresponding state transition experience samples are generated according to the interaction process between the intelligent agent and the target tracking simulation environment; wherein the observation information includes the local information defined in the state space, the target information within the target tracking locking capability range, and the friendly machine information within the effective communication range; the state transition experience samples include the observation information at the current moment, the execution action, the environmental feedback reward, and the observation information at the next moment; Storing the state transition experience sample in the experience replay cache, and updating the current experience replay cache; Calculate the timing error and sampling probability of each experience in the current experience playback buffer; For each agent, K experience samples are extracted from the current experience playback cache according to probability; the policy gradient is calculated, and the network parameters of the policy network Actor network are updated using the gradient descent method; the network parameters of the evaluation network Critic network are updated according to the calculated timing error value; Under the condition that the number of training times is reached, the current policy network Actor network and evaluation network Critic network are recorded as the trained policy network and evaluation network.

8. The method for acquiring and online updating the UAV cluster cooperative target tracking strategy as claimed in claim 7, characterized in that: The updating of the parameters of the first Actor network, the first Critic network, the mirror Actor network, and the mirror Critic network corresponding to each drone according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone includes: The drones whose first cumulative reward value is greater than the corresponding second cumulative reward value are recorded as first-category drones, and the remaining drones are recorded as second-category drones; according to the preset first probability, the mirror actor network and the mirror critic network of the first-category drones are updated to the corresponding first actor network and first critic network respectively; according to the preset second probability, the first actor network and the first critic network of the second-category drones are updated to the corresponding mirror actor network and mirror critic network respectively.

9. The method for acquiring and online updating the UAV cluster cooperative target tracking strategy as claimed in claim 8, characterized in that: The method of replaying the cache based on the mirror experience and updating the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent includes: For each mirror agent, calculate the timing error and sampling probability of each experience in the corresponding mirror experience replay cache; extract K second experiences from the corresponding mirror experience replay cache according to probability; calculate the policy gradient, and use the gradient descent method to update the network parameters of the corresponding mirror actor network and the corresponding mirror critic network; Among them, the network parameters of the mirrored Actor network are updated according to the following method: δ t =r t +γv ω (or t+1 )-v ω (or t ) p t =|δ t |+∈ Among them, θ is the network parameter of the mirror actor network; η a is the learning rate; Δ a is the gradient change of the mirror Actor network; K is the sampling size; w t is the importance sampling weight; δ t is the time difference value; is the gradient of the mirror Actor network; r t is the immediate reward; γ is the discount factor; v ω (o t+1 ) is the value function of the next state; v ω (o t ) is the value function of the current state; N is the capacity of the mirror experience replay buffer B; P(b t ) is the experience b t The sampling probability; η is the priority parameter; ι is the compensation parameter; ∈ is a constant; p t For priority; Update the network parameters of the mirrored critic network in the following way: Among them, ω is the network parameter of the mirror critic network; Δ c is the gradient update amount of the mirror critic network; is the gradient of the mirrored Critic network.

10. A device for acquiring and online updating a UAV cluster cooperative target tracking strategy, characterized in that: include: Initial network acquisition module, network deployment module, observation information real-time acquisition module, cumulative reward value acquisition module and network update module; The initial network acquisition module is used to acquire a first Actor network and a first Critic network generated by training in a preset target tracking simulation environment; The network deployment module is used to deploy the first Actor network and the first Critic network to each UAV flight control system, and create a mirror agent for each UAV, wherein the mirror agent includes a mirror Actor network, a mirror Critic network, and a mirror experience playback cache; The observation information real-time acquisition module is used to acquire the observation information of each drone in real time; The cumulative reward value acquisition module is used to input the observation information of each drone into the first Actor network and the mirror Actor network corresponding to each drone after obtaining the observation information of each drone, and respectively generate the first flight strategy and the corresponding mirror flight strategy corresponding to each drone; obtain the first cumulative reward value of each drone after driving each drone to perform the flight action according to the corresponding first flight strategy; after each mirror agent performs the virtual flight action according to the mirror flight strategy, each mirror agent estimates the reward value and obtains the second cumulative reward value; The network update module is used to update the parameters of the first Actor network, the first Critic network, the mirror Actor network and the mirror Critic network corresponding to each drone according to the difference between the first cumulative reward value and the corresponding second cumulative reward value of each drone after each drone is driven to execute the first flight strategy m times; after each drone is driven to execute the first flight strategy once, the parameters of the mirror Actor network and the mirror Critic network corresponding to each mirror agent are updated according to the mirror experience playback cache.

Images