Multi-unmanned aerial vehicle formation and obstacle avoidance control method based on safety reinforcement learning in low-altitude environment

Abstract

The invention discloses a multi-unmanned aerial vehicle formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment, and the method is characterized in that a safety filter is designed based on a control obstacle function, and is used for filtering potential unsafe or unreasonable actions of unmanned aerial vehicles in a flight process; meanwhile, a maximum pooling multi-agent depth deterministic strategy gradient network is constructed in combination with a parameter sharing technology and an observation embedded layer network, and a nominal strategy of formation control is trained. In the online execution stage, key features of unmanned aerial vehicle local observation information are extracted through the maximum pooling multi-agent depth deterministic strategy gradient network, feature vectors serve as network input, nominal actions are output, and the safety filter obtains the unmanned aerial vehicle local observation information through a quadratic programming solution mode according to the nominal actions and locally observed obstacle information. And generating a control instruction complying with the safety constraint for the unmanned aerial vehicle. According to the method, the autonomous formation obstacle avoidance capability of the multi-unmanned-aerial-vehicle system in an unknown environment is effectively improved, and autonomous cooperative formation obstacle avoidance flight of the multi-unmanned-aerial-vehicle system is realized.

Description

Technical Field

[0001] The present invention relates to a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment, and belongs to the technical field of UAV formation control. Background Art

[0002] As an emerging industry, low-altitude economy is increasingly becoming a new driving force for economic and social development. Urban air traffic, as a core component of the low-altitude economy, mainly focuses on cities and their surrounding areas, providing diversified low-altitude traffic services including air patrols and logistics transportation. Multi-UAV formations have shown great application potential and research value in the field of urban air traffic due to their robustness, fault tolerance and economic benefits.

[0003] In recent years, with the rapid development of artificial intelligence, especially the significant progress of deep reinforcement learning technology, a new technical approach has been provided for the design of multi-UAV formation control strategies. Unlike traditional control methods, deep reinforcement learning does not require accurate modeling of the environment model. It is an end-to-end learning method that directly maps the original environment state input to the action decision output. Deep reinforcement learning combines deep neural networks with reinforcement learning, enabling the agent to perceive, abstract the environment and make decisions, and to maximize rewards by exploring and learning the optimal strategy. At present, many studies have preliminarily verified the effectiveness of deep reinforcement learning methods in obstacle avoidance control of multi-robot collaborative formations. However, due to the uncertainty and complexity of the environment, and the need to explore unknown areas during strategy learning, the agent may take unsafe or unreasonable actions, which may lead to problems such as learning unsafe strategies. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment, which can effectively solve the problem of autonomous formation obstacle avoidance control of multi-UAV systems in partially observable environments and effectively improve the safety of the formation strategy.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] The multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment includes the following steps:

[0007] Step 1: With one of the drones as the leader and the remaining drones as wingmen, the formation configuration of multiple drones is pre-set, and the target position of the leader’s flight is determined; a control obstacle function is defined, and a safety filter is constructed based on the control obstacle function to filter out potential unsafe or unreasonable actions of drones during flight;

[0008] Step 2: construct a maximum pooling multi-agent deep deterministic policy gradient network for each wingman, obtain the local observation state information of the wingman at the current time step, use the local observation state information as the input of the maximum pooling multi-agent deep deterministic policy gradient network, and the maximum pooling multi-agent deep deterministic policy gradient network generates a nominal action for the wingman; the safety filter generates actual action instructions for the wingman based on the nominal action and the obstacle information observed by the wingman; the wingman executes the actual action instruction to obtain the reward of the current time step and the local observation state information of the next time step; according to the above process, the maximum pooling multi-agent deep deterministic policy gradient network is trained to obtain the trained network parameters;

[0009] In step 3, the trained network parameters are loaded into the maximum pooling multi-agent deep deterministic policy gradient network constructed in step 2, and the trained maximum pooling multi-agent deep deterministic policy gradient network and safety filter are used to generate actual action instructions for the wingman. The wingman executes the actual action instructions to realize autonomous collaborative formation obstacle avoidance flight of multiple UAVs.

[0010] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0011] 1. The present invention constructs an end-to-end autonomous formation obstacle avoidance framework that directly maps local observation information input to action decision output. The framework can be used when the UAV perception is limited, effectively improving the formation flight performance of multi-UAV systems in unknown environments, ensuring that UAVs can avoid obstacles and achieve autonomous formation flight.

[0012] 2. The present invention designs a maximum pooling multi-agent deep deterministic policy gradient network as the nominal strategy for formation control. The network can more effectively process the wingman's observation information, especially its observation information of its neighbor wingmen, and improve the overall formation control performance.

[0013] 3. The present invention constructs a safety filter based on the control obstacle function to correct the unsafe or unreasonable actions output by the nominal strategy to ensure the safety of the formation flight process, thereby significantly improving the safety performance of the formation flight. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural block diagram of a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment of the present invention;

[0015] Figure 2 It is a flight trajectory diagram of a UAV formation in a simulation environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0017] like Figure 1 As shown, the present invention proposes a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment, and the specific steps are as follows:

[0018] Step 1: Build a safety filter based on the control obstacle function to filter out potentially unsafe or unreasonable actions of the drone during flight.

[0019] Consider the affine nonlinear state equation of the control system:

[0020]

[0021] Where x is the state of the system; f(x) is the drift vector field of the system, and g(x) is the control vector field of the system, both of which satisfy the local Lipschitz condition; is the input of the system. Definition is a safe set. If the state x of the drone is in the set The goal of controlling the obstacle function is to design a controller that ensures that the state of the drone always remains in the forward invariant set. In, that is, when have In form, the security set Defined as some obstacle function h: The super level set of:

[0022]

[0023] Where h(x) is the control barrier function. When the derivative of the control barrier function remains non-negative, the system state can be prevented from entering the dangerous area, thereby ensuring that the system state always remains in the safe set:

[0024]

[0025] Among them, L f h(x)+L g h(x)a is the first-order Lie derivative of the control system; κ is a local Lipschitz expansion function. By adjusting the angular velocity of the drone to achieve obstacle avoidance, the obstacle control function can be defined as:

[0026] h(p,ψ)=||pp o || 2 -α o cos 2(ψ-ψ o )-(R o +ρ o ) 2

[0027] Among them, p and p o Respectively represent the position of the drone and the center of the obstacle circle closest to the drone; ψ is the current heading angle of the drone; ψ o is the direction difference between the drone and the obstacle center; is a constant; R o is the radius of the obstacle; ρ o is the safety threshold. The derivative constraint function is defined as:

[0028]

[0029] Step 2: Take one of the drones as the leader and the other drones as wingmen, and build a maximum pooling multi-agent deep deterministic policy gradient network for each wingman, including a main network and a target network. The wingman obtains the state information of the leader and its own local observation state information, and obtains an observation feature vector of fixed length through nonlinear transformation. The main network outputs the nominal action according to the feature vector and inputs it into the safety filter to solve the actual control action. Evaluate the reward or penalty of the wingman at the current time step based on the updated system state information. Train the network according to the above process to obtain the trained network parameters.

[0030] Step 2.1, construct the local observation state information of the wingman. The state information has four elements, including the observation information of its own state information, the observation information of the configuration target point state, the observation information of the neighbor wingman within the perception radius, and the observation information of the nearest obstacle within the perception radius;

[0031] Step 2.2, combine parameter sharing technology and observation embedding layer network to construct the main network and target network of the maximum pooling multi-agent deep deterministic policy gradient network. The maximum pooling multi-agent deep deterministic policy network takes local observation state information as input, converts the local observation state information into an observation feature vector with a fixed length through a nonlinear function, and generates a nominal action for the wingman output based on the observation feature vector. The safety filter uses a quadratic programming solution method to generate actual action instructions for the wingman based on the nominal action and the obstacle information locally observed by the wingman. The wingman executes the action and obtains the current reward and the local observation state information of the next time step;

[0032] Step 2.2.1: Introduce parameter sharing technology, that is, all wingmen share a strategy and build an action network μ and value network Q;

[0033] Step 2.2.2: Use the observation embedding layer to extract key features from local observation information:

[0034]

[0035] Among them, FC(·) is a fully connected layer, and They represent the observation information of the i-th wingman on its own state information at the current moment, the observation information of the configuration target point state, the observation information of the nearest obstacle within the perception radius, and the observation information of the neighbor wingman within the perception radius. In order to better refine the observation information of the neighbor wingman, a maximum pooling layer MP(·) is further used to process the neighbor observation feature vector:

[0036]

[0037] in, is the processed neighbor observation feature vector. The processed neighbor observation feature vector is concatenated with other observation embedding vectors, and the concatenated vector is input into a fully connected layer to obtain a fixed-length observation feature vector through nonlinear transformation:

[0038]

[0039] Among them, f t i is the observed feature vector of the i-th wingman at the current moment; || represents the concatenation operation. In this embodiment, the observation embedding layer consists of 4 fully connected layers, which respectively process the wingman's observation of its own information, the observation of the target point state, the observation of obstacles, and the observation of neighbor wingmen. The number of nodes is set to 32, 32, 32, and 64, respectively. The action network and the value network are both composed of multi-layer perceptrons, with 224 and 64 hidden nodes. Among them, the node output of the action network is activated using the tanh function, while the node output of the value network is activated using the linear function.

[0040] Step 2.2.3: Select the nominal action based on the current strategy and exploration noise:

[0041]

[0042] in, is the nominal action of the wingman at the current moment, is the number of wingmen, is the nominal acceleration of the wingman at the current moment, is the nominal angular velocity of the wingman at the current moment; ε e To explore the noise, a normal distribution is satisfied σ e To explore the noise standard deviation;

[0043] Step 2.2.4: Determine whether the wingman has observed an obstacle. If the wingman has not observed an obstacle, it is determined that the wingman is currently in a safe state and the safety filter will not be triggered. On the contrary, when the wingman observes an obstacle, there is a risk of collision, and the safety filter will be executed to ensure safety. The safety filter adjusts the angular velocity of the drone to achieve obstacle avoidance. It uses quadratic programming to solve a control action instruction that is closest to the nominal action and meets the safety constraints:

[0044]

[0045] in, and To control the barrier function constraints:

[0046]

[0047] Among them, α o is a constant; is the yaw angle of the i-th wingman at the current moment; is the relative yaw angle of the nearest obstacle observed by the i-th wingman at the current moment; is the forward speed of the i-th wingman at the current moment; is the relative distance of the nearest obstacle observed by the i-th wingman at the current moment;

[0048] Step 2.2.5: The wingman performs the action and obtains the current reward and the local observation state information of the next time step.

[0049] Step 2.3, store the generated historical trajectory in the form of a four-tuple (o, a, r, o') in the experience replay pool. Among them, o is the local observation state information of the wingman, a is the action performed by the wingman, that is, the speed and angular velocity of the wingman, r is the reward or penalty obtained by the wingman for performing action a, and o' is the local observation state information of the environment transferred to the next time step after the wingman performs action a;

[0050] In this embodiment, the reward function is defined as:

[0051]

[0052] in, Rewards for maintaining the team; and They represent the obstacle avoidance reward between the wingman and the obstacle and the collision avoidance reward between the wingman and the neighboring wingman. The formation maintenance reward function is designed as:

[0053]

[0054] Where k1 and k2 are constants. Design obstacle avoidance reward function for:

[0055]

[0056] Among them, P1 is the obstacle avoidance penalty constant; R s is the safety radius of the wingman. Design collision avoidance reward function for:

[0057]

[0058] Among them, P2 is the collision avoidance penalty constant.

[0059] Step 2.4: When the experience replay pool reaches the preset number, randomly extract small batch samples from it and use the Adam optimization algorithm to batch update the main network parameters of the maximum pooled multi-agent deep deterministic policy network. At the same time, soft update the target network parameters to make them partially close to the main network parameters. In this embodiment, the learning rates of the action network and the value network are 0.0001 and 0.001 respectively.

[0060] Step 2.4.1: Use the centralized training-distributed execution training method to utilize the experience data (x t ,a t ,r t ,x t+1 ) trains a shared strategy, and (The superscript indicates the number of the wingman, represents the number of wingmen) respectively represents the vector formed by the concatenation of observations, actions, rewards and next moment observations of all wingmen;

[0061] Step 2.4.2, the parameter update of the action network is expressed as:

[0062]

[0063] Among them, μ is the action network; θ μ is the action network parameter; is the experience replay pool; Q is the value network;

[0064] Step 2.4.3. Parameters of the value network ω Q Update by minimizing the TD error:

[0065]

[0066] in, Indicates the target Q value:

[0067]

[0068] Wherein, γ is the discount factor; Q' is the value network in the target network; μ' is the action network in the target network. In this embodiment, γ = 0.95.

[0069] Step 2.4.4: Soft update the target network parameters:

[0070]

[0071] Among them, θ μ′ and ω Q′ are the parameters of the action network and the value network in the target network, respectively, and τ is the soft update coefficient. In this embodiment, τ=0.01.

[0072] Step 2.5: When the number of training rounds reaches the preset upper limit, the update process is terminated and the parameters of the maximum pooling multi-agent deep deterministic policy gradient network model are saved; the entire update process is performed step by step in units of each round. In each round, the initial state of the wingman is randomly given. When the training time step reaches the preset upper limit and the lead aircraft flies to its target position, the training of this round is terminated and the next round of training is carried out.

[0073] Step 3: Load the trained network parameters. The main network outputs a nominal action based on the observed feature vector of the wingman. The safety filter generates the actual control action for the wingman based on the action and the obstacle information observed by the wingman. The wingman executes the action to autonomously avoid obstacles and maintain the formation configuration, completing the formation control task.

[0074] The main network transforms the local observation state vector of the wingman into an observation feature vector with a fixed length through a nonlinear function, and generates a nominal action for the wingman output;

[0075] Determine whether the wingman has observed an obstacle. If an obstacle is observed, a safety filter is used to combine the nominal action and the observed obstacle information to solve the corrected action command. If no obstacle is observed, the actual control command of the wingman is the nominal action;

[0076] The wingman performs the action and obtains the local observation state information of the next time step;

[0077] The above process is repeated so that the UAV formation has no collision during the entire flight process and maintains the formation configuration until the lead aircraft flies to the target area and completes the autonomous formation obstacle avoidance task. Figure 2 shown.

[0078] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment are implemented.

[0079] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the aforementioned multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment, characterized by: The steps include: Step 1: With one of the drones as the leader and the remaining drones as wingmen, the formation configuration of multiple drones is pre-set, and the target position of the leader’s flight is determined; a control obstacle function is defined, and a safety filter is constructed based on the control obstacle function to filter out potential unsafe or unreasonable actions of drones during flight; Step 2: construct a maximum pooling multi-agent deep deterministic policy gradient network for each wingman, obtain the local observation state information of the wingman at the current time step, use the local observation state information as the input of the maximum pooling multi-agent deep deterministic policy gradient network, and the maximum pooling multi-agent deep deterministic policy gradient network generates a nominal action for the wingman; The safety filter generates actual action instructions for the wingman based on the nominal action and the obstacle information observed by the wingman; the wingman executes the actual action instructions to obtain the reward of the current time step and the local observation state information of the next time step; according to the above process, the maximum pooling multi-agent deep deterministic policy gradient network is trained to obtain the trained network parameters; In step 3, the trained network parameters are loaded into the maximum pooling multi-agent deep deterministic policy gradient network constructed in step 2, and the trained maximum pooling multi-agent deep deterministic policy gradient network and safety filter are used to generate actual action instructions for the wingman. The wingman executes the actual action instructions to realize autonomous collaborative formation obstacle avoidance flight of multiple UAVs.

2. The multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment according to claim 1 is characterized in that: In step 1, the definition process of the control barrier function is as follows: According to the affine nonlinear state equation of the multi-UAV control system, the safety set is defined If the current state x of the drone is in the safety set In the middle, it is believed that the current state of the drone is safe; When the derivative of the control obstacle function always remains non-negative, it ensures that the state of the drone always remains in the safe set. In Chinese, that is: Among them, L f h(x)+L g h(x)a is the first-order Lie derivative of the control system, L f is the Lie derivative of h(x) along the vector field f(x), L g is the Lie derivative of h(x) along the vector field g(x), h(x) is the control obstacle function, a∈A is the input of the control system, and κ is the local Lipschitz expansion function; The control barrier function is defined as: h(p,ψ)=||pp o || 2 -a o cos 2 (ψ-ψ o )-(R o +r o ) 2 Among them, p and p o They represent the positions of the drone and the center of the obstacle closest to the drone, ψ is the heading angle of the drone, and ψ o is the orientation difference between the drone and the obstacle center, α o is a constant, R o is the obstacle radius, ρ o is the safety threshold; The derivative constraint function S(p,ψ) of the control barrier function is defined as: Among them, ω is the yaw angular velocity of the UAV, is the derivative of h(p,ψ).

3. The multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment according to claim 1 is characterized in that: The specific process of step 2 is as follows: Step 2.1, obtaining the local observation state information of the wingman at the current time step, including: observation information of its own state information, observation information of the state of the preset formation configuration target point, observation information of neighboring wingmen within the perception radius, and observation information of the nearest obstacle within the perception radius; Step 2.2, build a maximum pooling multi-agent deep deterministic policy gradient network for each wingman, including a main network and a target network, and the structures of the main network and the target network are the same; use the local observation state information of the wingman at the current time step as the input of the main network, output the nominal action for the wingman, and the safety filter generates actual action instructions for the wingman based on the nominal action and the obstacle information observed by the wingman; the wingman executes the actual action instruction and obtains the reward or penalty of the current time step and the local observation state information of the next time step; Step 2.3: In each training round, the local observation state information o of the wingman at the current time step, the action a executed, the reward or penalty r obtained by executing action a, and the local observation state information o′ of the next time step obtained by executing action a are stored in the experience replay pool in the form of a four-tuple (o, a, r, o′); Step 2.4, when the number of quadruplets in the experience replay pool reaches the preset number, randomly extract batch samples from it, and use the Adam optimization algorithm to update the main network parameters of the maximum pooled multi-agent deep deterministic policy network; at the same time, soft update the target network parameters; Step 2.5, when the training round reaches the preset upper limit, the update process is terminated and the parameters of the maximum pooling multi-agent deep deterministic policy gradient network are saved; in each training round, the initial state of the wingman is randomly given, and when the time step reaches the preset upper limit or the lead aircraft flies to the target position, the training round ends and the next training round is carried out.

4. The multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment according to claim 3 is characterized in that: The specific process of step 2.2 is as follows: Step 2.2.1, introduce parameter sharing technology, that is, all wingmen share a maximum pooling multi-agent deep deterministic policy gradient network and a shared experience replay pool; build the action network μ and the value network Q; Step 2.2.2, for the main network, use the observation embedding layer to extract the observation feature vector in the local observation state information. The observation embedding layer includes four fully connected layers: Among them, FC(·) is a fully connected layer, and They respectively represent the observation information of the i-th wingman’s own state information at time step t, the observation information of the preset formation configuration target point state, the observation information of the nearest obstacle within the perception radius, and the observation information of the neighbor wingman within the perception radius; and Respectively and The corresponding observed feature vector; Use the maximum pooling layer To process: Among them, MP(·) is the maximum pooling layer, is the observed feature vector of the neighbor wingman after processing; Using the fully connected layer and and Perform splicing and obtain the observed feature vector through nonlinear transformation: Among them, f t i is the observed feature vector of the i-th wingman at time step t, || represents the splicing operation; Step 2.2.3, based on the observed feature vector f of the wingman at time step t t and exploration noise to generate nominal actions: in, is the nominal action of the wingman at time step t, is the number of wingmen, is the nominal acceleration of the wingman at time step t, is the nominal angular velocity of the wingman at time step t; μ is the action network, ε e To explore the noise, the normal distribution is satisfied σ e To explore the noise standard deviation; Step 2.2.4, determine whether the wingman observes an obstacle. If the wingman does not observe an obstacle, it is determined that the wingman is currently in a safe state and the safety filter is not triggered. That is, the nominal action generated in step 2.2.3 is used as the actual action instruction of the wingman. If the wingman observes an obstacle, it is determined that there is a risk of collision with the wingman. The safety filter is used to adjust the angular velocity of the wingman to avoid the obstacle. The quadratic programming method is used to solve an action instruction that is closest to the nominal action and meets the safety constraints: and To control the barrier function constraints: in, is the safety-filtered yaw rate, is the yaw velocity of the i-th wingman at time step t, is the nominal angular velocity of the i-th wingman at time step t, ω min ,ω max are the minimum and maximum yaw angular velocity of the UAV, α o is a constant, is the yaw angle of the i-th wingman at time step t, is the relative yaw angle of the nearest obstacle observed by the i-th wingman at time step t, is the forward velocity of the i-th wingman at time step t; is the relative distance of the nearest obstacle observed by the i-th wingman at time step t, is the position of the i-th wingman at time step t, is the position of the nearest obstacle observed by the i-th frame at time step t; In step 2.2.5, the wingman executes the actual action command and obtains the reward or penalty of the current time step and the local observation state information of the next time step.

5. The multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment according to claim 3 is characterized in that: The specific process of step 2.4 is as follows: Step 2.4.1, adopt the centralized training-distributed execution training method, using the four-tuple data (x t ,a t ,r t ,x t+1 ) train a shared policy, and Representing the 1st to the The vector is the concatenation of the observed state information, action, reward or penalty of the wingman at time step t and the observed state information at time step t+1. The superscript indicates the number of the wingman. is the number of wingmen; Step 2.4.2, the parameter update of the action network in the main network is expressed as: in, is the gradient operator of the action network parameters, J(μ) is the target update function of the action network, μ is the action network, θ μ are the action network parameters, For the experience replay pool, The action network μ is based on the observed feature vector f of the i-th wingman at time step t t i Output Action for The gradient operator, Q is the value network, is the observed feature vector of the i-th wingman at time step t, is the action performed by the i-th wingman at time step t, Step 2.4.3, Parameters ω of the value network in the main network Q Update by minimizing the time difference error: in, is the target update function of the value network, is the target Q value, r t i is the reward or penalty of the i-th wingman at time step t, γ is the discount factor, Q′ is the value network in the target network, and μ′ is the action network in the target network; Step 2.4.4, soft update the parameters of the target network: Among them, θ μ′ and ω Q′ are the parameters of the action network and value network in the target network respectively, and τ is the soft update coefficient.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment are implemented as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, the steps of the multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in a low-altitude environment are implemented as described in any one of claims 1 to 5.

Images