Multi-UAV formation control method based on offline sample-corrected reinforcement learning

Abstract

The present invention relates to a multi-UAV formation control method based on offline sample correction reinforcement learning, comprising: constructing a deep deterministic policy gradient network for offline training; the wingman interacts with the environment with the current strategy to generate the historical trajectory information of the wingman and obtain offline samples; randomly extracting offline samples, and calculating the importance weights for correcting offline samples according to the difference between the current strategy and the previous strategy; offline training the deep deterministic policy gradient network based on the importance weights to update the network parameters of the deep deterministic policy gradient network; constructing a deep deterministic policy gradient network for online execution, and loading the offline trained network parameters; taking the real-time interaction information between the wingman and the environment as input, generating an optimal action output for the wingman. The present invention can effectively improve the efficiency of learning formation strategies of multi-UAV systems, and has high application value.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle systems, and in particular to a multi-unmanned aerial vehicle formation control method based on offline sample correction reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence technology and wireless communication technology, multi-UAV systems are becoming increasingly intelligent and practical, and are widely used in military and civilian fields. Formation control is one of the core technologies for the coordinated combat of multi-UAV systems. In recent years, multi-UAV formation control methods based on deep reinforcement learning have received extensive attention and research. Deep reinforcement learning shows great potential in multi-UAV formation control by combining the feature extraction advantages of deep learning with the decision optimization ability of reinforcement learning. However, due to the unknown nature of the task and the limited combat time, UAV formations face the problem of training timeliness when performing tasks. The deep reinforcement learning algorithm continuously interacts with the environment to collect training sample data to optimize the formation control strategy. The quality of training samples greatly affects the effect of strategy learning. However, in the training process, the traditional deep reinforcement learning algorithm extracts training samples through a random sampling paradigm to learn strategies. This sampling paradigm is difficult to ensure the quality of samples, resulting in low training efficiency. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a multi-UAV formation control method based on offline sample correction reinforcement learning.

[0004] To achieve the above-mentioned object of the invention, the present invention provides a multi-UAV formation control method based on offline sample correction reinforcement learning, comprising:

[0005] S1. Offline sample collection phase:

[0006] Build a deep deterministic policy gradient network for offline training;

[0007] The wingman in the multi-UAV formation interacts with the environment with the current strategy of the deep deterministic policy gradient network to generate the historical trajectory information of the wingman, and obtains the offline sample based on the historical trajectory information and stores it in the experience replay pool;

[0008] S2. Offline sample correction stage:

[0009] Randomly sampling from the experience replay pool, and calculating the importance weight for correcting the offline sample according to the difference between the current policy of the deep deterministic policy gradient network and the previous policy corresponding to the offline sample; wherein, including:

[0010] S21. extracting training samples from the experience replay pool by random sampling paradigm;

[0011] S22. quantifying the difference between the previous strategy and the current strategy according to Jensen-Shannon divergence, and mapping the difference between the two strategies into importance weights of training samples through nonlinear transformation;

[0012] S3. Offline training phase:

[0013] Performing offline training on the deep deterministic policy gradient network based on the importance weights to update network parameters of the deep deterministic policy gradient network;

[0014] S4. Online execution phase:

[0015] Construct a deep deterministic policy gradient network for online execution and load the network parameters trained offline;

[0016] A deep deterministic policy gradient network executed online generates an optimal action output for the wingman based on real-time interaction information between the wingman and the environment;

[0017] The wingman executes the output optimal action to maintain the formation configuration.

[0018] According to one aspect of the present invention, in step S1, in the step of constructing a deep deterministic policy gradient network for offline training, a truncated dual Learning technology and updating the network parameters of the deep deterministic policy gradient network using stochastic gradient descent;

[0019] The deep deterministic policy gradient network includes: an action strategy network, a first value network, a second value network, a target action strategy network, a first target value network, and a second target value network;

[0020] The action strategy network is used to receive interaction information between the wingman and the environment and output actions for the wingman to perform;

[0021] The first value network is used to learn the first value of the action strategy network execution process. value;

[0022] The second value network is used to learn the second value of the action strategy network execution process. value;

[0023] The target action strategy network serves as a target network of the action strategy network to provide a stable strategy update target;

[0024] The first target value network is used as the target network of the first value network to provide a stable first Value updates the target;

[0025] The second target value network is used as the target network of the second value network to provide a stable second The value updates the target.

[0026] According to one aspect of the present invention, in step S1, the step in which the wingman in the multi-UAV formation interacts with the environment using the current strategy of the deep deterministic policy gradient network includes:

[0027] The wingman interacts with the environment to generate interaction information; wherein the interaction information is status information between the wingman and the leader;

[0028] Based on the state information, construct a system joint state vector at the current time step;

[0029] The deep deterministic policy gradient network takes the joint state vector as input, generates an action for the wingman, and the wingman executes the output action and obtains the current reward / penalty and the system joint state at the next time step.

[0030] According to one aspect of the present invention, in the step of constructing a system joint state vector at the current time step based on the state information, the system joint state vector has 6 elements, which are: the speed and heading angle information of the leader and the wingman in the global coordinate system, and the horizontal and vertical deviations of the desired target position in the wingman's body coordinate system.

[0031] According to one aspect of the present invention, in step S1, in the step of obtaining the offline sample based on the historical trajectory information and storing it in the experience replay pool, the obtained historical trajectory information is expressed in the form of a quadruple, and it is expressed as:

[0032] ;

[0033] in, is the current system joint state, is the action performed by the wingman, i.e. the speed and angular velocity of the wingman, It is the wingman who performs the action Rewards / penalties received, It is the wingman who performs the action Then transfer to the joint state of the system at the next time step in the environment.

[0034] According to one aspect of the present invention, in step S21, in the step of extracting training samples from the experience replay pool by random sampling, the extracted training samples are represented as:

[0035] ;

[0036] in, represents the state information matrix in the historical trajectory, represents the action matrix in the historical trajectory, represents the reward / penalty matrix in the historical trajectory, Represents the state information matrix of the next moment in the historical trajectory, represents the feature number of the action matrix, represents the eigenvalue of the state information matrix, is the number of batches of training samples extracted;

[0037] In step S22, the step of quantifying the difference between the previous strategy and the current strategy according to the Jensen-Shannon divergence and mapping the difference between the two strategies into the importance weight of the training sample through nonlinear transformation includes:

[0038] The difference in action selection between the current strategy and the previous strategy is calculated and expressed as:

[0039] ;

[0040] in, represents the difference in action selection between the current policy and the previous policy, represents the action selected by the previous policy, represents the action selected by the current policy;

[0041] A multivariate Gaussian distribution is constructed, and the difference between the previous strategy and the current strategy is calculated using Jensen-Shannon divergence, wherein the multivariate Gaussian distribution is expressed as:

[0042] ;

[0043] ;

[0044] ;

[0045] in, represents the mean, represents the covariance matrix, subscript t Indicates the sample serial number;

[0046] The difference is expressed as:

[0047] ;

[0048] in, express Divergence, represents a normal distribution, represents the standard deviation of the exploration noise, represents the zero matrix, represents the identity matrix;

[0049] The difference between the two strategies is mapped to to generate the importance weight, wherein the importance weight is expressed as:

[0050] ;

[0051] in, The two strategies are completely consistent. This means that the two strategies are completely inconsistent.

[0052] According to one aspect of the present invention, in step S3, the deep deterministic policy gradient network is offline trained based on the importance weight to update the network parameters of the deep deterministic policy gradient network, and the network parameters of the deep deterministic policy gradient network are updated by stochastic gradient descent, wherein the network parameters of the action policy network are batch updated by the Adam optimization algorithm, and the network parameters of the target action policy network are soft updated every preset number of time steps.

[0053] According to one aspect of the present invention, in step S3, the step of performing offline training on the deep deterministic policy gradient network based on the importance weight to update the network parameters of the deep deterministic policy gradient network includes:

[0054] S31. Calculate the TD target based on the constructed deep deterministic policy gradient network, wherein the TD target is expressed as:

[0055] ;

[0056] in, Indicates reward / punishment, Indicates the sample number, Indicates the sample number action sample, by the target action strategy network The action selected; represents the deduction factor, Represents the first target value network output The value updates the target, Indicates the sample number A sample of the system joint state;

[0057] S32. Update the parameters of the first value network and the second value network by minimizing the TD target error, and it is expressed as:

[0058] ;

[0059] in, is the value network parameter, is the gradient operator of the value network parameters, is a value network, if is the first value network, if For the second value network, Indicates the sample number A sample of the system joint state, Indicates the sample number Action samples:

[0060] S33. Iteratively update the network parameters of the action policy network of the deep deterministic policy gradient network in a gradient descent manner based on the importance weight, and it is expressed as:

[0061] ;

[0062] in, is the gradient operator of the action policy network parameters, is the gradient operator of the action, As the first value network, is the action policy network, are the action strategy network parameters;

[0063] S34. Every preset number of time steps, soft update the network parameters of the target action strategy network, which is expressed as:

[0064] ;

[0065] in, are the target action strategy network parameters, is the soft update coefficient, is the target value network parameter;

[0066] S35. When the iterative optimization step of the training process reaches a preset upper limit, the updating process is terminated and the network parameters of the deep deterministic policy gradient network are saved.

[0067] According to one aspect of the present invention, in step S4, in the step of generating an optimal action output for the wingman using the real-time interaction information between the wingman and the environment by the deep deterministic policy gradient network executed online, the system joint state vector of the wingman at the current time step is obtained based on the real-time interaction information between the wingman and the environment, and the system joint state vector is used as the input of the deep deterministic policy gradient network executed online to generate the optimal action of the wingman, wherein the optimal action includes: an optimal expected speed and an optimal angular speed.

[0068] According to a solution of the present invention, the present invention can effectively improve the efficiency of multi-UAV system learning formation strategy and has high application value.

[0069] According to a solution of the present invention, the present invention effectively solves the problem of low training efficiency of traditional deep reinforcement learning. Specifically, in the process of strategy learning, the present invention quantifies the difference between the previous strategy and the current strategy according to the Jensen-Shannon divergence, and maps the difference between the two strategies to the importance weight of the training samples through nonlinear transformation, so that more reliable training samples can be generated using limited experience samples, effectively improving the training efficiency and learning effect.

[0070] According to a solution of the present invention, the present invention does not rely on the environmental model and realizes a system joint state-action end-to-end formation control. Through the deep learning network architecture, features are extracted from the state information and the optimal action decision is learned, realizing the direct mapping from state input to action output, which is suitable for sequential decision-making in unknown environments, effectively improving the adaptability of multi-UAV formations in unknown environments, and improving the performance of overall formation control. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A flowchart of a multi-UAV formation control method based on offline sample correction reinforcement learning according to an embodiment of the present invention;

[0072] Figure 2 A control system block diagram of a multi-UAV formation control method based on offline sample correction reinforcement learning according to an embodiment of the present invention;

[0073] Figure 3 A structural diagram of a deep deterministic policy gradient network according to an embodiment of the present invention;

[0074] Figure 4 The present invention is a flight trajectory diagram of a UAV formation in a simulation environment according to an embodiment of the present invention, wherein (a) represents a trajectory diagram of a UAV formation flying along a straight trajectory, (b) represents a trajectory diagram of a UAV formation flying along an "S" trajectory, and (c) represents a trajectory diagram of a UAV formation flying along a "Z" trajectory. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0076] Combination Figure 1 , Figure 2 and Figure 3As shown, according to one embodiment of the present invention, a multi-UAV formation control method based on offline sample correction reinforcement learning of the present invention includes:

[0077] S1. Offline sample collection phase:

[0078] Construct a deep deterministic policy gradient network for offline training; in this embodiment, a truncated dual Learning technology, and using stochastic gradient descent to update the network parameters of the deep deterministic policy gradient network; the deep deterministic policy gradient network includes: action policy network , the first value network , the second value network , target action strategy network , First target value network and the second target value network In this embodiment, the action strategy network Used to receive the interaction information between the wingman and the environment and output actions for the wingman to perform; the first value network For learning action policy network The first step in the execution process Value; Second Value Network For learning action policy network The second step of execution value; further, the target action strategy network As an action policy network The target network is used to provide a stable strategy update target; the first target value network As the first value network The target network is used to provide a stable first Value update target; second target value network As the second value network The target network is used to provide a stable second In this embodiment, the action strategy network , the first value network , the second value network , target action strategy network , First target value network and the second target value network They are composed of multi-layer perceptrons, among which, see Figure 3 As shown, the action strategy network and target action policy network The number of hidden nodes contained is 256 and 128 respectively, and the node output is the ReLU activation function. , the second value network , First target value network and the second target value network The number of hidden nodes is 256, 128, and 64 respectively, and the node output is the ReLU activation function. Based on the above settings, the action policy network During the training process, the first value network supply Value estimation to guide the action policy network The parameters are gradient-dropped. During the value network training process (including the first value network , the second value network ), according to the corresponding target value network (including the first target value network and the second target value network ) Calculation target value, and calculate the TD target by taking a small paradigm, and then update the parameters of the value network.

[0079] Furthermore, the wingman in the multi-UAV formation interacts with the environment with the current strategy of the deep deterministic policy gradient network to generate historical trajectory information of the wingman, and obtains offline samples based on the historical trajectory information and stores them in the experience replay pool; in this embodiment, the step of the wingman in the multi-UAV formation interacting with the environment with the current strategy of the deep deterministic policy gradient network includes:

[0080] The wingman interacts with the environment to generate interaction information; wherein the interaction information is the status information between the wingman and the leader;

[0081] Based on the state information, the system joint state vector at the current time step is constructed; the system joint state vector has 6 elements, which are: the speed and heading angle information of the leader and wingman in the global coordinate system, and the horizontal and vertical deviations of the desired target position in the wingman's body coordinate system.

[0082] The deep deterministic policy gradient network takes the joint state vector as input and generates an action for the wingman, which executes the output action and obtains the current reward / penalty and the joint state of the system at the next time step.

[0083] Combination Figure 1 , Figure 2 and Figure 3As shown, according to an embodiment of the present invention, in step S1, in the step of obtaining offline samples based on historical trajectory information and storing them in the experience replay pool, the obtained historical trajectory information is expressed in the form of a four-tuple, and it is expressed as:

[0084] ;

[0085] in, is the current system joint state (i.e., system joint state vector), is the action performed by the wingman, i.e. the speed and angular velocity of the wingman, It is the wingman who performs the action Rewards / penalties received, It is the wingman who performs the action Then it is transferred to the system joint state (i.e., the system joint state vector) of the next time step in the environment.

[0086] S2. Offline sample correction stage:

[0087] Randomly sample from the experience replay pool, and calculate the importance weights for correcting offline samples based on the difference between the current policy of the deep deterministic policy gradient network and the previous policy corresponding to the offline sample; it includes:

[0088] S21. Training samples are extracted from the experience replay pool through a random sampling paradigm; the historical trajectory obtained by the interaction between the wingman and the environment (The four-tuple includes state, action, reward, and next-moment state) will be stored in the experience replay pool, and then randomly sampled from the experience replay pool during training The historical trajectories constitute the training samples. In this embodiment, the extracted training samples are expressed as:

[0089] ;

[0090] in, Represents the state information matrix in the historical trajectory, which is composed of the current system joint state in the historical trajectory Constructed, Represents the action matrix in the historical trajectory, the actions performed by the wingman in the historical trajectory Constructed, Represents the reward / penalty matrix in the historical trajectory, where the wingman performs actions in the historical trajectory Rewards / penalties received Constructed, Represents the state information matrix of the next moment in the historical trajectory, and the action is performed by the wingman in the historical trajectory Then transfer to the joint state of the system at the next time step in the environment Constructed, represents the feature number of the action matrix, represents the eigenvalue of the state information matrix, is the number of batches of training samples extracted; in this embodiment, , , .

[0091] S22. Quantify the difference between the previous strategy and the current strategy according to the Jensen-Shannon divergence, and map the difference between the two strategies to the importance weights of the training samples through nonlinear transformation, which includes:

[0092] The difference in action selection between the current policy and the previous policy is calculated and expressed as:

[0093] ;

[0094] in, represents the difference in action selection between the current policy and the previous policy, represents the action selected by the previous policy (i.e., the offline sample stored by the experience replay pool), represents the action selected by the current policy;

[0095] Construct a multivariate Gaussian distribution and use Jensen-Shannon divergence to calculate the difference between the previous strategy and the current strategy, where the multivariate Gaussian distribution is expressed as:

[0096] ;

[0097] ;

[0098] ;

[0099] in, represents the mean, represents the covariance matrix, subscript t Indicates the sample serial number;

[0100] Furthermore, the difference is expressed as:

[0101] ;

[0102] in, express , represents a normal distribution, represents the standard deviation of the exploration noise, represents the zero matrix, represents the identity matrix; in this embodiment, the standard deviation of the noise is explored It decays from 0.5 to 0.05 as the number of episodes (i.e. the number of training rounds) increases.

[0103] The difference between the two strategies is mapped to , to generate importance weights, where the importance weights are expressed as:

[0104] ;

[0105] in, The two strategies are completely consistent. This means that the two strategies are completely inconsistent.

[0106] S3. Offline training phase:

[0107] The deep deterministic policy gradient network is trained offline based on the importance weights to update the network parameters of the deep deterministic policy gradient network; wherein the network parameters of the deep deterministic policy gradient network are updated by a stochastic gradient descent method, wherein the network parameters of the action policy network are updated in batches by an Adam optimization algorithm. In this embodiment, the action network (i.e., the action policy network and target action policy network ) and the value network (i.e. the first value network , the second value network , First target value network and the second target value network ) are all set to have a learning rate of 0.001. And, every preset number (e.g., 4) of time steps, the network parameters of the target action policy network are soft-updated to make them close to the main network parameters.

[0108] Furthermore, in step S3, the step of performing offline training on the deep deterministic policy gradient network based on the importance weights to update the network parameters of the deep deterministic policy gradient network includes:

[0109] S31. Calculate the TD target based on the constructed deep deterministic policy gradient network, where the TD target is expressed as:

[0110] ;

[0111] in, Indicates reward / punishment, Indicates the sample number, Indicates the sample number action sample, by the target action strategy network The action selected; represents the deduction factor, Represents the first target value network output The value updates the target, Indicates the sample number The system joint state sample; in this embodiment, the deduction factor can be set to: ;

[0112] S32. Update the parameters of the first value network and the second value network by minimizing the TD target error, and it is expressed as:

[0113] ;

[0114] in, is the value network parameter, is the gradient operator of the value network parameters, is a value network, if is the first value network, if For the second value network, Indicates the sample number A sample of the system joint state, Indicates the sample number action sample; in this embodiment, ;

[0115] S33. Iteratively update the network parameters of the action policy network of the deep deterministic policy gradient network in a gradient descent manner based on the importance weights, and it is expressed as:

[0116] ;

[0117] in, is the gradient operator of the action policy network parameters, is the gradient operator of the action, As the first value network, is the action policy network, are the parameters of the action strategy network;

[0118] S34. Every preset number of time steps, soft update the network parameters of the target action strategy network, which is expressed as:

[0119] ;

[0120] in, is the target value network parameter, is the soft update coefficient, is the target action strategy network parameter; in this embodiment, ;

[0121] S35. When the iterative optimization step of the training process reaches the preset upper limit, the update process is terminated and the network parameters of the deep deterministic policy gradient network are saved; in this embodiment, the entire update process is performed step by step in units of each episode. In each episode, the initial states of the agents (at the initial moment, the initial states of the leader and the wingman are randomly given, and are collectively referred to as agents for this purpose) are randomly given. When the training time step reaches the preset upper limit, the episode training is terminated and the next episode training is performed; in this embodiment, the preset upper limit of the iterative optimization step of the training process is 150 time steps.

[0122] S4. Online execution phase:

[0123] A deep deterministic policy gradient network for online execution is constructed, and the network parameters trained offline are loaded; in this embodiment, the constructed deep deterministic policy gradient network is consistent with the aforementioned network, and will not be described in detail here. Therefore, the network parameters obtained based on the aforementioned training process can be directly loaded into the newly constructed deep deterministic policy gradient network, thereby achieving the effect of its online execution.

[0124] The deep deterministic policy gradient network executed online generates an optimal action output for the wingman based on the real-time interaction information between the wingman and the environment; in this embodiment, the system joint state vector of the current time step of the wingman is obtained based on the real-time interaction information between the wingman and the environment, and the system joint state vector is used as the input of the deep deterministic policy gradient network executed online to generate the optimal action of the wingman, wherein the optimal action includes: the optimal expected speed and the optimal angular speed. In this embodiment, the system joint state vector is consistent with the aforementioned setting method and will not be repeated here.

[0125] The wingman executes the outputted optimal action to maintain the formation configuration; in this embodiment, after the wingman completes the currently received optimal action, the system joint state information of the next time step can be constructed;

[0126] Furthermore, the above process of executing the optimal action based on real-time interactive information is repeated to complete the formation control flight between the leader and the wingman; in this embodiment, the flight trajectory of the leader can be added based on a preset method, so that the leader can maintain accurate control of the wingman's flight while flying along the flight trajectory. In this embodiment, the flight trajectory of the multi-UAV formation in the simulation environment is as follows: Figure 4 As shown; among them, Figure 4 (a) and (b) show the trajectories of the lead aircraft along a straight and S-shaped curve path. As the flight progresses, the formation errors of the wingmen in both scenarios gradually decrease, indicating that they are able to follow the lead aircraft and adjust their positions to form a coordinated formation. This verifies the effectiveness of the proposed formation algorithm in straight and curved trajectories. Figure 4(c) shows the trajectory of the Z-shaped curve navigated by the lead aircraft, in which the wingman initially lags behind at two turns, but then gradually synchronizes with the lead aircraft to maintain the formation configuration. This highlights the robustness of the proposed formation algorithm, which can adapt and recover in complex trajectories to ensure stable formation flight.

[0127] The above contents are merely examples of specific solutions of the present invention. For devices and structures not described in detail therein, it should be understood that they can be implemented by adopting general devices and general methods available in the art.

[0128] The above is only one solution of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-UAV formation control method based on offline sample correction reinforcement learning, characterized in that: include: S1. Offline sample collection phase: Build a deep deterministic policy gradient network for offline training; The wingman in the multi-UAV formation interacts with the environment with the current strategy of the deep deterministic policy gradient network to generate the historical trajectory information of the wingman, and obtains the offline sample based on the historical trajectory information and stores it in the experience replay pool; S2. Offline sample correction stage: Randomly sampling from the experience replay pool, and calculating the importance weight for correcting the offline sample according to the difference between the current policy of the deep deterministic policy gradient network and the previous policy corresponding to the offline sample; wherein, including: S21. extracting training samples from the experience replay pool by random sampling paradigm; S22. quantifying the difference between the previous strategy and the current strategy according to Jensen-Shannon divergence, and mapping the difference between the two strategies into importance weights of training samples through nonlinear transformation; S3. Offline training phase: Performing offline training on the deep deterministic policy gradient network based on the importance weights to update network parameters of the deep deterministic policy gradient network; S4. Online execution phase: Construct a deep deterministic policy gradient network for online execution and load the network parameters trained offline; A deep deterministic policy gradient network executed online generates an optimal action output for the wingman based on real-time interaction information between the wingman and the environment; The wingman executes the output optimal action to maintain the formation configuration.

2. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 1 is characterized in that: In step S1, in the step of constructing a deep deterministic policy gradient network for offline training, a truncated dual policy gradient network is added to the deep deterministic policy gradient network. Learning technology and updating the network parameters of the deep deterministic policy gradient network using stochastic gradient descent; The deep deterministic policy gradient network includes: an action strategy network, a first value network, a second value network, a target action strategy network, a first target value network, and a second target value network; The action strategy network is used to receive interaction information between the wingman and the environment and output actions for the wingman to perform; The first value network is used to learn the first value of the action strategy network execution process. value; The second value network is used to learn the second value of the action strategy network execution process. value; The target action strategy network serves as a target network of the action strategy network to provide a stable strategy update target; The first target value network is used as the target network of the first value network to provide a stable first Value updates the target; The second target value network is used as the target network of the second value network to provide a stable second The value updates the target.

3. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 2 is characterized in that: In step S1, the step of the wingman in the multi-UAV formation interacting with the environment using the current strategy of the deep deterministic policy gradient network includes: The wingman interacts with the environment to generate interaction information; wherein the interaction information is status information between the wingman and the leader; Based on the state information, construct a system joint state vector at the current time step; The deep deterministic policy gradient network takes the joint state vector as input, generates an action for the wingman, and the wingman executes the output action and obtains the current reward / penalty and the system joint state at the next time step.

4. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 3 is characterized in that: In the step of constructing a system joint state vector at the current time step based on the state information, the system joint state vector has 6 elements, which are: the speed and heading angle information of the leader and the wingman in the global coordinate system, and the horizontal and vertical deviations of the desired target position in the wingman's body coordinate system.

5. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 4 is characterized in that: In step S1, in the step of obtaining the offline sample based on the historical trajectory information and storing it in the experience replay pool, the obtained historical trajectory information is expressed in the form of a quaternary, and is represented as: in, is the current system joint state, is the action performed by the wingman, i.e. the speed and angular velocity of the wingman, It is the wingman who performs the action Rewards / penalties received, It is the wingman who performs the action Then transfer to the joint state of the system at the next time step in the environment.

6. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 5 is characterized in that: In step S21, in the step of extracting training samples from the experience replay pool by random sampling, the extracted training samples are expressed as: in, represents the state information matrix in the historical trajectory, represents the action matrix in the historical trajectory, represents the reward / penalty matrix in the historical trajectory, Represents the state information matrix of the next moment in the historical trajectory, represents the feature number of the action matrix, represents the eigenvalue of the state information matrix, is the number of batches of training samples extracted; In step S22, the step of quantifying the difference between the previous strategy and the current strategy according to the Jensen-Shannon divergence and mapping the difference between the two strategies into the importance weight of the training sample through nonlinear transformation includes: The difference in action selection between the current strategy and the previous strategy is calculated and expressed as: in, represents the difference in action selection between the current policy and the previous policy, represents the action selected by the previous policy, represents the action selected by the current policy; A multivariate Gaussian distribution is constructed, and the difference between the previous strategy and the current strategy is calculated using Jensen-Shannon divergence, wherein the multivariate Gaussian distribution is expressed as: ； in, represents the mean, represents the covariance matrix, subscript t Indicates the sample serial number; The difference is expressed as: in, express Divergence, represents a normal distribution, represents the standard deviation of the exploration noise, represents the zero matrix, represents the identity matrix; The difference between the two strategies is mapped to to generate the importance weight, wherein the importance weight is expressed as: in, The two strategies are completely consistent. This means that the two strategies are completely inconsistent.

7. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 6 is characterized in that: In step S3, the deep deterministic policy gradient network is offline trained based on the importance weight to update the network parameters of the deep deterministic policy gradient network, and the network parameters of the deep deterministic policy gradient network are updated by stochastic gradient descent, wherein the network parameters of the action policy network are batch updated by the Adam optimization algorithm, and the network parameters of the target action policy network are soft updated every preset number of time steps.

8. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 7 is characterized in that: In step S3, the step of performing offline training on the deep deterministic policy gradient network based on the importance weight to update the network parameters of the deep deterministic policy gradient network includes: S31. Calculate the TD target based on the constructed deep deterministic policy gradient network, wherein the TD target is expressed as: in, Indicates reward / punishment, Indicates the sample number, Indicates the sample number action sample, by the target action strategy network The action selected; represents the deduction factor, Represents the first target value network output The value updates the target, Indicates the sample number A sample of the system joint state; S32. Update the parameters of the first value network and the second value network by minimizing the TD target error, and it is expressed as: in, is the value network parameter, is the gradient operator of the value network parameters, is a value network, if is the first value network, if For the second value network, Indicates the sample number A sample of the system joint state, Indicates the sample number Action samples: S33. Iteratively update the network parameters of the action policy network of the deep deterministic policy gradient network in a gradient descent manner based on the importance weight, and it is expressed as: in, is the gradient operator of the action policy network parameters, is the gradient operator of the action, As the first value network, is the action policy network, are the action strategy network parameters; S34. Every preset number of time steps, soft update the network parameters of the target action strategy network, which is expressed as: in, are the target action strategy network parameters, is the soft update coefficient, is the target value network parameter; S35. When the iterative optimization step of the training process reaches a preset upper limit, the updating process is terminated and the network parameters of the deep deterministic policy gradient network are saved.

9. The multi-UAV formation control method based on offline sample correction reinforcement learning according to claim 8 is characterized in that: In step S4, in the step of generating an optimal action output for the wingman using the real-time interaction information between the wingman and the environment by the deep deterministic policy gradient network executed online, the system joint state vector of the wingman at the current time step is obtained based on the real-time interaction information between the wingman and the environment, and the system joint state vector is used as the input of the deep deterministic policy gradient network executed online to generate the optimal action of the wingman, wherein the optimal action includes: an optimal expected speed and an optimal angular speed.

Images