A Multi-UAV Formation and Obstacle Avoidance Control Method Based on Safety Reinforcement Learning in Low-Altitude Environments

Abstract

This invention discloses a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in low-altitude environments. This method designs a safety filter based on a control obstacle function to filter potentially unsafe or unreasonable actions of the UAVs during flight. Simultaneously, it combines parameter sharing technology and an observation embedding layer network to construct a max-pooling multi-agent deep deterministic policy gradient network to train the nominal policy for formation control. During the online execution phase, the max-pooling multi-agent deep deterministic policy gradient network extracts key features from the UAV's local observation information and uses the feature vector as network input, outputting a nominal action. The safety filter, based on this nominal action and the locally observed obstacle information, generates control commands for the UAVs that comply with safety constraints through a quadratic programming solution. This invention effectively improves the autonomous formation and obstacle avoidance capabilities of multi-UAV systems in unknown environments, enabling autonomous cooperative formation and obstacle avoidance flight of multi-UAV systems.

Description

Technical Field

[0001] This invention relates to a method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments, belonging to the field of UAV formation control technology. Background Technology

[0002] As an emerging industry, the low-altitude economy is increasingly becoming a new driving force for economic and social development. Urban air traffic, as a core component of the low-altitude economy, mainly focuses on cities and their surrounding areas, providing diversified low-altitude traffic services including aerial patrol and logistics transportation. Multi-UAV swarms, with their robustness, fault tolerance, and economic benefits, demonstrate enormous application potential and research value in the field of urban air traffic.

[0003] In recent years, the rapid development of artificial intelligence, especially the significant advancements in deep reinforcement learning technology, has provided a novel technical approach to the design of control strategies for multi-drone formations. Unlike traditional control methods, deep reinforcement learning does not require precise modeling of the environment; it is an end-to-end learning method that directly maps the original environmental state input to action decision output. Deep reinforcement learning combines deep neural networks with reinforcement learning, enabling agents to perceive, abstract the environment, and make decisions, maximizing rewards through exploration and learning of optimal strategies. Currently, several studies have preliminarily verified the effectiveness of deep reinforcement learning methods in obstacle avoidance control for multi-robot cooperative formations. However, due to the uncertainty and complexity of the environment, and the need to explore unknown domains during policy learning, agents may take unsafe or unreasonable actions, potentially leading to the learning of unsafe strategies. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments, which can effectively solve the problem of autonomous formation and obstacle avoidance control of multi-UAV systems in some observable environments and effectively improve the safety of formation strategies.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments includes the following steps:

[0007] Step 1: Using one drone as the lead drone and the remaining drones as wingmen, pre-set the formation configuration of the multiple drones and determine the target position of the lead drone; define a control obstacle function, and build a safety filter based on the control obstacle function to filter out potential unsafe or unreasonable actions of the drones during flight;

[0008] Step 2: Construct a max-pooling multi-agent deep deterministic policy gradient network for each wingman, obtain the local observation state information of the wingman at the current time step, and use the local observation state information as the input of the max-pooling multi-agent deep deterministic policy gradient network. The max-pooling multi-agent deep deterministic policy gradient network generates nominal actions for the wingman. The safety filter generates actual action commands for the wingman based on the nominal actions and the obstacle information observed by the wingman. The wingman executes the actual action commands and obtains the reward for the current time step and the local observation state information for the next time step. Train the max-pooling multi-agent deep deterministic policy gradient network according to the above process to obtain the trained network parameters.

[0009] Step 3: Load the trained network parameters into the max pooling multi-agent deep deterministic policy gradient network constructed in Step 2. Use the trained max pooling multi-agent deep deterministic policy gradient network and safety filter to generate actual action commands for the wingman. The wingman executes the actual action commands to achieve autonomous cooperative formation obstacle avoidance flight of multiple UAVs.

[0010] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0011] 1. This invention constructs an end-to-end autonomous formation obstacle avoidance framework that directly maps local observation information input to action decision output. This framework can be used when UAV perception is limited, effectively improving the formation flight performance of multi-UAV systems in unknown environments, ensuring that UAVs can avoid obstacles and achieve autonomous formation flight.

[0012] 2. This invention designs a maximum pooling multi-agent deep deterministic policy gradient network as the nominal policy for formation control. This network can more effectively process the observation information of wingmen, especially their observation information of neighboring wingmen, and improve the overall formation control performance.

[0013] 3. This invention constructs a safety filter based on the control obstacle function to correct unsafe or unreasonable actions in the nominal strategy output, thereby ensuring the safety of the formation flight process and significantly improving the safety performance of formation flight. Attached Figure Description

[0014] Figure 1 This is a structural block diagram of the multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in low-altitude environments according to the present invention.

[0015] Figure 2 This is a flight trajectory diagram of the drone formation in a simulated environment in an embodiment of the present invention. Detailed Implementation

[0016] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0017] like Figure 1 As shown, this invention proposes a multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in low-altitude environments. The specific steps are as follows:

[0018] Step 1: Construct a safety filter based on the control obstacle function to filter out potentially unsafe or unreasonable actions of the drone during flight.

[0019] Consider the affine nonlinear state equations of the control system:

[0020]

[0021] Where x is the state of the system; f(x) is the drift vector field of the system; and g(x) is the control vector field of the system, both of which satisfy the local Lipschitz condition. For system input. Definition For a safe set, if the state x of the drone is in the set... In this context, the drone's current state is considered safe. The goal of the obstacle control function is to design a controller that guarantees the drone's state always remains within the forward-invariant set. In the middle, that is, when have In terms of form, security sets can be Defined as a certain barrier function h: Superlevel set:

[0022]

[0023] Here, h(x) is the control barrier function. When the derivative of the control barrier function remains non-negative, it prevents the system state from entering the danger zone, thus ensuring that the system state always remains within a safe concentration.

[0024]

[0025] Among them, L f h(x)+L g h(x)a is the first-order Lie derivative of the control system; κ is a local Lipschitz spread function. Obstacle avoidance is achieved by adjusting the angular velocity of the UAV; the obstacle control function can be defined as:

[0026] h(p,ψ)=||pp o || 2 -α o cos 2(ψ-ψ o )-(R o +ρ o ) 2

[0027] Where p and p o These represent the positions of the drone and the center of the nearest obstacle, respectively; ψ is the drone's current heading angle; ψ o The difference in orientation between the drone and the center of the obstacle; R is a constant; o ρ is the radius of the obstacle; o This is the safety threshold. The derivative constraint function is defined as:

[0028]

[0029] Step 2: Using one drone as the lead drone and the others as wingmen, construct a max-pooling multi-agent deep deterministic policy gradient network for each wingman, including a main network and a target network. The wingmen acquire the lead drone's state information and their own local observation state information, and obtain a fixed-length observation feature vector through nonlinear transformation. The main network outputs a nominal action based on this feature vector, which is then input into a safety filter to calculate the actual control action. The reward or penalty for the wingmen at the current time step is evaluated based on the updated system state information. The network is trained according to the above process to obtain the trained network parameters.

[0030] Step 2.1: Construct local observation status information for wingmen. The status information has four elements, including observation information of its own status, observation information of the status of the configuration target point, observation information of neighboring wingmen within the perception radius, and observation information of the nearest obstacle within the perception radius.

[0031] Step 2.2: Construct the main network and target network of the max-pooling multi-agent deep deterministic policy gradient network by combining parameter sharing technology and the observation embedding layer network. The max-pooling multi-agent deep deterministic policy network takes local observation state information as input, transforms it into a fixed-length observation feature vector through a nonlinear function, and generates a nominal action for the wingman based on the observation feature vector. The safety filter, based on this nominal action and the obstacle information locally observed by the wingman, uses a quadratic programming approach to generate the actual action command for the wingman. The wingman executes the action and obtains the current reward and the local observation state information for the next time step.

[0032] Step 2.2.1: Introduce parameter sharing technology, that is, all wingmen share a strategy, and construct an action network μ and a value network Q;

[0033] Step 2.2.2: Use the observation embedding layer to extract key features from local observation information:

[0034]

[0035] Where FC(·) is a fully connected layer, and Let represent the observation information of the i-th wingman at the current moment regarding its own state, the state of the target point, the nearest obstacle within its perception radius, and the neighboring wingmen within its perception radius, respectively. To better extract the observation information of the neighboring wingmen, a max-pooling layer MP(·) is further used to process the neighboring observation feature vector:

[0036]

[0037] in, This is the processed neighbor observation feature vector. The processed neighbor observation feature vector is concatenated with other observation embedding vectors, and the concatenated vector is input into a fully connected layer. Through nonlinear transformation, a fixed-length observation feature vector is obtained.

[0038]

[0039] Among them, f t i Let be the observation feature vector of the i-th wingman at the current moment; || denotes the concatenation operation. In this embodiment, the observation embedding layer consists of four fully connected layers, which respectively handle the wingman's observations of its own information, the state of the target point, the obstacles, and the neighboring wingmen, with the number of nodes set to 32, 32, 32, and 64, respectively. Both the action network and the value network are composed of multilayer perceptrons, with 224 and 64 hidden nodes, respectively. The node outputs of the action network are activated using the tanh function, while the node outputs of the value network are activated using a linear function.

[0040] Step 2.2.3: Select the nominal action based on the current strategy and the noise level.

[0041]

[0042] in, The wingman's current action. For the number of wingmen, This represents the nominal acceleration of the wingman at the current moment. ε is the nominal angular velocity of the wingman at the current moment; e To explore noise, it must satisfy a normal distribution. σ e To explore the standard deviation of noise;

[0043] Step 2.2.4: Determine if the wingman has observed an obstacle. If the wingman has not observed an obstacle, it is determined that the wingman is currently in a safe state and the safety filter will not be triggered. Conversely, if the wingman observes an obstacle, there is a risk of collision, and the safety filter will be executed to ensure safety. The safety filter achieves obstacle avoidance by adjusting the angular velocity of the UAV. It uses quadratic programming to solve for a control action command that is closest to the nominal action and satisfies the safety constraints.

[0044]

[0045] in, and To control the obstacle function constraints:

[0046]

[0047] Where, α o It is a constant; Let yaw angle be the current angle of the i-th wingman; Let be the relative yaw angle of the i-th wingman to the nearest obstacle observed at the current moment; Let i be the forward velocity of the i-th wingman at the current moment; Let be the relative distance between the i-th wingman and the nearest obstacle observed at the current moment;

[0048] Step 2.2.5: The wingman performs this action and obtains the current reward and the local observation status information for the next time step.

[0049] Step 2.3: Store the generated historical trajectory in the experience replay pool as a quadruple (o, a, r, o′). Here, o represents the local observation state information of the wingman, a represents the action performed by the wingman (i.e., the speed and angular velocity), r represents the reward or penalty obtained by the wingman for performing action a, and o′ represents the local observation state information of the environment transitioning to the next time step after the wingman performs action a.

[0050] In this embodiment, the reward function is defined as:

[0051]

[0052] in, The representative formation will maintain the reward; and Let represent the obstacle avoidance reward between a wingman and an obstacle, and the collision avoidance reward between a wingman and a neighboring wingman, respectively. The formation maintenance reward function is designed as follows:

[0053]

[0054] Where k1 and k2 are constants. Design the obstacle avoidance reward function. for:

[0055]

[0056] Where P1 is the obstacle avoidance penalty constant; R s Define the safe radius for the wingman. Design a collision avoidance reward function. for:

[0057]

[0058] Where P2 is the collision avoidance penalty constant.

[0059] Step 2.4: Once the experience replay pool reaches a preset number, a small batch of samples is randomly selected from it, and the main network parameters of the max-pooled multi-agent deep deterministic policy network are updated in batches using the Adam optimization algorithm. Simultaneously, the target network parameters are softly updated to partially approximate the main network parameters. In this embodiment, the learning rates of the action network and the value network are 0.0001 and 0.001, respectively.

[0060] Step 2.4.1: Adopt a centralized training-distributed execution training method, utilizing the experience data (x) collected by all wingmen. t ,a t ,r t ,x t+1 Train a shared policy. and (The superscript indicates the wingman's number) (representing the number of wingmen) represents a vector composed of the concatenation of the observations, actions, rewards, and next-moment observations of all wingmen;

[0061] Step 2.4.2, the parameter update of the action network is represented as follows:

[0062]

[0063] Where μ is the action network; θ μ These are the parameters for the action network; Q represents the experience replay pool; Q represents the value network.

[0064] Step 2.4.3, Parameters ω of the value network Q Update by minimizing the TD error:

[0065]

[0066] in, Indicates the target Q value:

[0067]

[0068] Where γ is the discount factor; Q′ is the value network in the target network; and μ′ is the action network in the target network. In this embodiment, γ = 0.95.

[0069] Step 2.4.4: Soft update the parameters of the target network:

[0070]

[0071] Where, θ μ′ and ω Q′ τ represents the parameters of the action network and value network in the target network, respectively, and τ is the soft update coefficient. In this embodiment, τ = 0.01.

[0072] Step 2.5: When the number of training rounds reaches the preset limit, terminate the update process and save the parameters of the max-pooling multi-agent deep deterministic policy gradient network model. The entire update process is carried out step by step on a round-by-round basis. In each round, the initial state of the wingman is randomly given. When the training time step reaches the preset limit and the lead aircraft flies to its target position, the training for that round ends and the next round begins.

[0073] Step 3: Load the trained network parameters. The main network outputs a nominal action based on the wingman's observed feature vector. The safety filter generates the actual control action for the wingman based on this action and the obstacle information observed by the wingman. The wingman executes this action, thereby autonomously avoiding obstacles and maintaining formation configuration, completing the formation control task.

[0074] The main network transforms the wingman's local observation state vector into a fixed-length observation feature vector through a nonlinear function, generating a nominal action for the wingman's output.

[0075] Determine whether the wingman has detected an obstacle. If an obstacle is detected, use a safety filter to combine the nominal action and the detected obstacle information to calculate a corrected action command. If no obstacle is detected, the wingman's actual control command is the nominal action.

[0076] The wingman executes the action and obtains local observation status information for the next time step;

[0077] Repeating the above process ensures that the drone formation flies without collisions and maintains its formation configuration until the lead drone reaches the target area, completing the autonomous formation obstacle avoidance mission. In this embodiment, the flight trajectory of the multi-drone formation in the simulation environment is as follows: Figure 2 As shown.

[0078] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments.

[0079] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments.

[0080] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments, characterized in that, Includes the following steps: Step 1: Using one drone as the lead drone and the remaining drones as wingmen, pre-set the formation configuration of the multiple drones and determine the target position of the lead drone; define a control obstacle function, and build a safety filter based on the control obstacle function to filter out potential unsafe or unreasonable actions of the drones during flight; In step 1, the definition process of the control barrier function is as follows: Based on the affine nonlinear state equations of a multi-UAV control system, a safety set is defined. If the current state of the drone In security set The Chinese side considers the drone to be safe in its current state. When the derivative of the control barrier function remains non-negative, the state of the UAV is always kept within a safe set. In, that is: ， in, For the first-order Lie derivative of the control system, for Along the vector field Li Daoshu, for Along the vector field Li Daoshu, For the control barrier function, For the input of the control system, For local Lipschitz spread functions; The control barrier function is defined as follows: ， in, and These represent the positions of the drone and the center of the nearest obstacle, respectively. The heading angle of the drone. The difference in orientation between the center of the drone and the center of the obstacle. It is a constant. Let the radius be the obstacle. This is a safety threshold; Derivative constraint function of control barrier function Defined as: ， in, Let be the yaw rate of the drone. for The derivative; Step 2: Construct a max-pooling multi-agent deep deterministic policy gradient network for each wingman, obtain the local observation state information of the wingman at the current time step, and use the local observation state information as the input of the max-pooling multi-agent deep deterministic policy gradient network. The max-pooling multi-agent deep deterministic policy gradient network generates nominal actions for the wingman. The safety filter generates actual action commands for the wingman based on the nominal actions and the obstacle information observed by the wingman. The wingman executes the actual action commands and obtains the reward for the current time step and the local observation state information for the next time step. Train the max-pooling multi-agent deep deterministic policy gradient network according to the above process to obtain the trained network parameters. Step 3: Load the trained network parameters into the max pooling multi-agent deep deterministic policy gradient network constructed in Step 2. Use the trained max pooling multi-agent deep deterministic policy gradient network and safety filter to generate actual action commands for the wingman. The wingman executes the actual action commands to achieve autonomous cooperative formation obstacle avoidance flight of multiple UAVs.

2. The method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments according to claim 1, characterized in that, The specific process of step 2 is as follows: Step 2.1: Obtain the local observation status information of the wingman at the current time step, including: observation information of its own status, observation information of the target point status of the preset formation configuration, observation information of neighboring wingmen within the perception radius, and observation information of the nearest obstacle within the perception radius. Step 2.2: Construct a max-pooling multi-agent deep deterministic policy gradient network for each wingman, including a main network and a target network, with identical structures. Use the local observation state information of the wingman at the current time step as input to the main network to output a nominal action. The safety filter generates actual action commands for the wingman based on the nominal action and the obstacle information observed by the wingman. The wingman executes the actual action commands and obtains the reward or penalty at the current time step and the local observation state information for the next time step. Step 2.3: In each training round, the local observation state information of the wingman at the current time step is recorded. Actions performed Execution of actions Rewards or punishments received and the execution of actions The obtained local observation state information at the next time step , with quadruple The data is stored in the experience replay pool in the form of [database name]. Step 2.4: When the number of quadruplets in the experience replay pool reaches the preset number, a batch of samples is randomly selected from them, and the main network parameters of the max pooling multi-agent deep deterministic policy network are updated using the Adam optimization algorithm; at the same time, the target network parameters are softly updated. Step 2.5: When the training rounds reach the preset limit, terminate the update process and save the parameters of the max pooling multi-agent deep deterministic policy gradient network. In each training round, the initial state of the wingman is randomly given. When the time step reaches the preset limit or the lead aircraft flies to the target position, the training round ends and the next training round begins.

3. The method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments according to claim 2, characterized in that, The specific process of step 2.2 is as follows: Step 2.2.1 introduces parameter sharing technology, whereby all wingmen share a max-pooled multi-agent deep deterministic policy gradient network and a shared experience replay pool; and constructs an action network. and value network ; Step 2.2.2: For the main network, the observation embedding layer is used to extract the observation feature vector from the local observation state information. The observation embedding layer consists of four fully connected layers: ， in, It is a fully connected layer. and They represent the first Wingman in time step The observation information includes the self-state information, the target point state of the preset formation configuration, the nearest obstacle within the perception radius, and the neighboring wingman within the perception radius. and They represent and The corresponding observed feature vector; Using maximum pooling layers Processing: ， in, For max pooling layer, This represents the processed observation feature vector of the neighboring wingman; Using fully connected layers and and The observed feature vectors are obtained by concatenating the features and performing a nonlinear transformation: ， in, For the first Wingman in time step The observed feature vector, Indicates a splicing operation; Step 2.2.3, based on the wingman's timing... Observation feature vector And explore the nominal action of noise generation: ， in, For the wingman in time step The nominal action, For the number of wingmen, For the wingman in time step The nominal acceleration, For the wingman in time step The nominal angular velocity; For action networks, To explore noise that follows a normal distribution , To explore the standard deviation of noise; Step 2.2.4: Determine if the wingman has observed an obstacle. If the wingman has not observed an obstacle, it is determined that the wingman is currently in a safe state and the safety filter is not triggered. That is, the nominal action generated in step 2.2.3 is used as the actual action command for the wingman. If the wingman observes an obstacle, it is determined that the wingman is at risk of collision. The safety filter is used to adjust the wingman's angular velocity to avoid the obstacle. A quadratic programming approach is used to solve for an action command that is closest to the nominal action and satisfies the safety constraints. ， and To control the obstacle function constraints: ， in, The yaw rate is the safety-filtered value. For the first Wingman in time step yaw rate, For the first Wingman in time step The nominal angular velocity, , These are the minimum and maximum yaw angular velocities of the drone, respectively. It is a constant. For the first Wingman in time step Yaw angle, For the first Wingman in time step The relative yaw angle of the nearest obstacle was observed. For the first Wingman in time step Forward velocity; For the first Wingman in time step The relative distance to the nearest obstacle was observed. For the first Wingman in time step Location, For the first Built on time step The location of the nearest obstacle was observed; Step 2.2.5: The wingman executes the actual action command and obtains the reward or penalty for the current time step and the local observation status information for the next time step.

4. The method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments according to claim 2, characterized in that, The specific process of step 2.4 is as follows: Step 2.4.1 employs a centralized training-distributed execution training approach, utilizing the quadruple data collected by all wingmen. Train a shared policy, , , and Representing the 1st to the 2nd Wingman in time step The observed state information, actions, rewards or punishments, and time steps The vector is formed by concatenating the observation status information, with the superscript indicating the wingman's number. The number of wingmen; Step 2.4.2, the parameter update of the action network in the main network is represented as: ， in, The gradient operator for the action network parameters. For the target update function of the action network, For action networks, For action network parameters, For experience replay pool, For Action Networks According to the Wingman in time step Observation feature vector Output action , for gradient operator, For value networks, For the first Wingman in time step The observed feature vector, For the first Wingman in time step The action performed ; Step 2.4.3, Parameters of the value network in the main network Update by minimizing the time difference error: ， in, The target update function for the value network. For the target Q value, For the first Wingman in time step Rewards or punishments As a discount factor, For the value network in the target network, For the action network in the target network; Step 2.4.4, Soft update the parameters of the target network: ， in, and These are the parameters of the action network and the value network in the target network, respectively. This is the soft update coefficient.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multi-UAV formation and obstacle avoidance control method based on safety reinforcement learning in low-altitude environments as described in any one of claims 1 to 4.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for multi-UAV formation and obstacle avoidance control based on safety reinforcement learning in low-altitude environments as described in any one of claims 1 to 4.

Images