Method and device for formation keeping and cooperative obstacle avoidance control of aircraft cluster

Abstract

The application discloses a kind of aircraft cluster formation keeping and cooperative obstacle avoidance control method and electronic equipment.Method includes: according to the original state observation information of aircraft cluster, based on the preset safe soft actor-critic algorithm, aircraft cluster control model is constructed;Based on aircraft cluster control model, according to the relative observation state of any aircraft in aircraft cluster, the control instruction for any aircraft in aircraft cluster is determined.The application utilizes safe soft actor-critic algorithm, on the one hand, by mapping position and velocity coordination error into virtual elastic potential energy and damping dissipation energy to deduce continuous coordination reward function, solve the problem of reward tampering of reward function, on the other hand, by introducing Lagrange multiplier to the constrained Markov decision process framework, the priority of experience replay strategy is adaptively adjusted to ensure the stable formation configuration of aircraft cluster formation inside anti-collision and the cooperative stable fly-around outside formation to obstacles.

Description

Technical Field

[0001] This application relates to a method and electronic equipment for maintaining and cooperating obstacle avoidance in aircraft swarm formation, belonging to the field of aircraft swarm control technology. Background Technology

[0002] Multi-vehicle swarm systems have broad application prospects in patrol and reconnaissance, disaster search and rescue, area surveillance, and autonomous collaborative operations. During the execution of missions by swarm systems, the following requirements must usually be met simultaneously: maintaining the predetermined formation structure; avoiding collisions between aircraft; avoiding obstacles and no-fly zones in the environment; and ensuring the successful completion of the overall mission objectives.

[0003] Among current technologies, deep reinforcement learning can autonomously learn control strategies through interaction with the environment, exhibiting good environmental adaptability. However, current deep reinforcement learning methods mainly rely on reward functions to optimize control objectives. When the spatial environment is complex and involves multiple constraints and environmental uncertainties, the lack of hard safety constraints often leads to reward tampering. Furthermore, it faces limitations in utilizing sparse collision samples efficiently, failing to simultaneously ensure the stable maintenance of the formation configuration and real-time obstacle avoidance safety in complex obstacle-filled spaces. Summary of the Invention

[0004] This application discloses a method and electronic equipment for maintaining formation and cooperative obstacle avoidance control of aircraft swarms.

[0005] The aircraft swarm formation maintenance and cooperative obstacle avoidance control method in this application includes the following steps: Based on the original state observation information of the aircraft cluster, a control model for the aircraft cluster is constructed using a pre-defined safety soft actor-commentator algorithm. Based on the aforementioned aircraft cluster control model, and according to the relative observation state of any aircraft in the aircraft cluster, control commands are determined for any aircraft in the aircraft cluster to control the aircraft cluster to perform formation maintenance or cooperative obstacle avoidance.

[0006] In some implementations, the step of constructing an aircraft swarm control model based on the raw state observation information of the aircraft swarm and a preset safety soft actor-commentator algorithm includes: The original state observation information of the aircraft cluster is obtained to determine the state observation vector of any aircraft in the aircraft cluster, wherein the original state observation information includes the original target observation information, original neighbor observation information, original obstacle observation information, and original self-flight state observation information of any aircraft in the aircraft cluster. Based on the state observation vector and the attribute parameters of the aircraft cluster, a mission reward function and a safety cost signal for the constrained Markov decision process are constructed. Based on the continuous motion control commands of the aircraft cluster, the state observation vector, the mission reward function, and the safety cost signal, a control model for the aircraft cluster is constructed using a preset safety soft actor-commentator algorithm. The continuous motion control commands are pre-constructed based on the kinematic and dynamic parameters of any aircraft in the aircraft cluster.

[0007] In some implementations, constructing the mission reward function and the safety cost signal of the constrained Markov decision process based on the state observation vector and the attribute parameters of the aircraft cluster includes: Based on the original target observation information, the preset first normalization factor and the preset first weight coefficient, a target assault guidance reward function is constructed, wherein the attribute parameters include the first normalization factor and the first weight coefficient. Based on the original target observation information, the preset second normalization factor, and the preset second weighting coefficient, a formation cooperative maintenance reward function is constructed based on the strict Lyapunov stability theory, wherein the attribute parameters include the second normalization factor and the second weighting coefficient. Based on the original target observation information, the preset warning boundary parameters, and the preset reward coefficient, a dense obstacle avoidance guidance reward function is constructed, wherein the attribute parameters include the warning boundary parameters and the reward coefficient; The mission reward function is constructed based on the target assault guidance reward function, the formation coordination maintenance reward function, and the dense obstacle avoidance guidance reward function.

[0008] In some implementations, constructing a target assault guidance reward function based on the original target observation information, a preset first normalization factor, and a preset first weighting coefficient includes: Based on the relative distance of any aircraft in the aircraft cluster from the target, a preset relative distance normalization factor, and a preset relative distance weighting coefficient, a position progress reward subfunction is constructed, wherein the relative distance of any aircraft from the target is determined based on the original target observation information, the first normalization factor includes the relative distance normalization factor, and the first weighting coefficient includes the relative distance weighting coefficient. Based on the projection of the velocity of any aircraft in the aircraft cluster onto the target line of sight, a preset velocity normalization factor, and a preset velocity weighting coefficient, a velocity alignment reward subfunction is constructed. The projection of the velocity of any aircraft onto the target line of sight is determined based on the original target observation information. The first normalization factor includes the velocity normalization factor, and the first weighting coefficient includes the velocity weighting coefficient. Based on the positional relationship between any aircraft in the aircraft cluster and the target, a sparse reward subfunction for task completion is constructed, wherein the positional relationship is determined based on the original target observation information. The value of the sparse reward subfunction for task completion is 0 or a preset function value. The preset function value is greater than the maximum function value of the position progress reward subfunction and the velocity alignment reward subfunction. The preset function value is configured to represent a large positive incentive. The target assault guidance reward function is constructed based on the position progress reward subfunction, the speed alignment reward subfunction, and the task completion sparse reward subfunction.

[0009] In some implementations, the formula for the formation cooperative maintenance reward function is:

[0010] in To maintain the reward function for the formation coordination, This represents the position error potential energy based on the virtual spring model. This represents the energy dissipation caused by negative work done in a virtual damping model. This represents the position coordination error of the aircraft cluster. This indicates the speed coordination error of the aircraft cluster. This is the stiffness weighting coefficient. This is the damping weighting coefficient. For position normalization factor, This is the velocity normalization factor; The position coordination error and the velocity coordination error are determined based on the original target observation information. The second normalization factor includes the position normalization factor and the velocity normalization factor. The second weighting coefficient includes the stiffness weighting coefficient and the damping weighting coefficient.

[0011] In some implementations, the formula for the dense obstacle avoidance guidance reward function is:

[0012] in This refers to the dense obstacle avoidance guided reward function. The reward coefficient is... This is the minimum distance between the aircraft cluster and the center of the obstacle. The equivalent danger radius of the obstacle. Indicates the warning boundary parameters; The minimum distance and the equivalent danger radius are determined based on the original target observation information.

[0013] In some implementations, the step of constructing the mission reward function and the safety cost signal of the constrained Markov decision process based on the state observation vector and the attribute parameters of the aircraft cluster further includes: The safety cost signal is constructed based on the minimum distance between the aircraft cluster and the center of the obstacle, the equivalent danger radius of the obstacle, and the preset safety buffer distance, wherein the value of the safety cost signal is 0 or 1.

[0014] In some implementations, constructing the aircraft cluster control model based on a preset safety soft actor-critic algorithm, according to the continuous action control commands of the aircraft cluster, the state observation vector, the mission reward function, and the safety cost signal, includes: Based on the state observation vector and the continuous action control command, a policy network, a dual-path task evaluation network, and a cost evaluation network are constructed using the safe soft actor-critic algorithm. Multiple loss functions are constructed based on the preset Lagrange multipliers, the policy network, the dual-path task evaluation network, and the cost evaluation network; Based on the state observation vector, continuous action control command, mission reward function and safety cost signal corresponding to any aircraft in the aircraft cluster, a comprehensive priority parameter corresponding to any aircraft is constructed. The loss function is updated based on the comprehensive priority parameters to construct the aircraft cluster control model.

[0015] In some implementations, determining control commands for any aircraft in the aircraft cluster based on the relative observation state of any aircraft in the cluster, according to the aircraft cluster control model, to control the aircraft cluster to perform formation keeping or cooperative obstacle avoidance, includes: Initialize the initial physical state of the aircraft cluster and initialize the aircraft cluster control model; Obtain the relative observation state of any aircraft in the aircraft cluster; The relative observation state is input into the policy network of the aircraft cluster control model to determine the continuous action commands for any of the aircraft. The continuous action commands are subjected to hyperbolic tangent activation limiting and linear inverse mapping to ensure that the arbitrary aircraft meets the corresponding maximum available overload physical boundary constraints, thereby determining the control commands.

[0016] The electronic device in this application includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the aircraft swarm formation maintenance and cooperative obstacle avoidance control method in the above-described embodiments is implemented.

[0017] The beneficial effects of this application are as follows: This application utilizes the safe soft actor-critic algorithm. On the one hand, by mapping the position and velocity cooperative error to virtual elastic potential energy and damping dissipation energy to derive a continuous cooperative reward function, it can fill the gap in hard safety constraints and thus solve the problem of reward function tampering. On the other hand, by introducing Lagrange multipliers into the constrained Markov decision process framework that decouples dense task rewards and sparse safety costs, it adaptively adjusts the priority of the experience replay strategy to ensure stable formation configuration for collision avoidance within the aircraft swarm and cooperative stable flight around obstacles outside the formation. Attached Figure Description

[0018] Figure 1 This is one of the flowcharts illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 2 This is the second flowchart illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 3 This is the third flowchart illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 4 This is the fourth flowchart illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 5 This is the fifth flowchart illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 6 This is the sixth flowchart illustrating the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 7 This is one of the application scenario diagrams of the aircraft swarm formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 8 This is the second schematic diagram of the application scenario of the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 9 This is the third schematic diagram of the application scenario of the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 10 This is the fourth schematic diagram of the application scenario of the aircraft cluster formation maintenance and cooperative obstacle avoidance control method in the embodiments of this application; Figure 11 This is a schematic diagram comparing the normalized performance indicators of the aircraft cluster formation maintenance and cooperative obstacle avoidance control methods in the embodiments of this application. Detailed Implementation

[0019] Please see Figure 1 The aircraft swarm formation maintenance and cooperative obstacle avoidance control method in this application includes the following steps: Step 01: Based on the original state observation information of the aircraft cluster, construct an aircraft cluster control model based on the preset safety soft actor-commentator algorithm; Step 02: Based on the aircraft swarm control model, determine the control commands for any aircraft in the aircraft swarm according to the relative observation state of any aircraft in the aircraft swarm, so as to control the aircraft swarm to perform formation keeping or cooperative obstacle avoidance.

[0020] Specifically, firstly, the influencing factors for the overall formation maintenance and cooperative obstacle avoidance control of an aircraft swarm include both the real-time operational status of each aircraft within the swarm and the shape, distribution, and distance of obstacles in the surrounding environment. Considering the complexity of these factors, the aircraft swarm formation maintenance and cooperative obstacle avoidance control method in this application first constructs a corresponding aircraft swarm control model based on the swarm. Then, based on the constructed model, it determines corresponding control commands for each aircraft within the swarm, thereby enabling each aircraft in the entire swarm to maintain its formation position and achieve a cooperative effect in position changes during obstacle avoidance.

[0021] For the construction of aircraft swarm control models, the Safe Soft Actor-Critic algorithm (SSAC-PER algorithm) with priority experience replay under explicit safety constraints is generally adopted. The SSAC-PER algorithm can quickly get out of the low-reward region in the early stage of model construction and training, and converge stably after a short training period and obtain a larger average reward value. In contrast, the soft actor algorithm (SAC algorithm) or the algorithm that only introduces priority experience replay in the current related technologies lacks effective hard safety constraints. The reward value may fluctuate significantly during the training process, and the final reward value is difficult to converge to the expected level.

[0022] In some implementations, please refer to Figure 2 Step 01 includes: Step 011: Obtain the raw state observation information of the aircraft cluster to determine the state observation vector of any aircraft in the cluster. The original state observation information includes the original target observation information, original neighbor observation information, original obstacle observation information, and original flight state observation information of any aircraft in the aircraft cluster; Step 012: Based on the state observation vector and the attribute parameters of the aircraft cluster, construct the mission reward function and the safety cost signal of the constrained Markov decision process; Step 013: Based on the continuous action control commands, state observation vectors, mission reward function, and safety cost signals of the aircraft swarm, construct an aircraft swarm control model using a pre-defined safety soft actor-critic algorithm. The continuous motion control commands are pre-constructed based on the kinematic and dynamic parameters of any aircraft in the aircraft cluster.

[0023] Specifically, the following section will explain the construction process of the aircraft cluster control model.

[0024] First, for any aircraft in the aircraft cluster, obtain its corresponding raw state observation information. The raw state observation information generally includes the aircraft's raw target observation information, raw neighbor observation information, raw obstacle observation information, and raw self-flight state observation information.

[0025] The original target observation information includes the unit line-of-sight vector of the aircraft pointing to the target point and the relative distance information, which is used to guide the aircraft to attack the mission target point; The original neighbor observation information includes the position consistency deviation vector and velocity consistency deviation vector between the nearest neighboring aircraft selected by physical distance within the current aircraft's perception range. It is mainly used to maintain the ensemble formation configuration of the aircraft cluster to avoid collisions between aircraft within the aircraft cluster. The raw obstacle observation information includes the surface distances of multiple obstacles closest to the current aircraft, as well as the repulsion guidance vector calculated inspired by the principle of artificial potential field, which is mainly used to characterize the collision threat in the environment; The original flight status observation information includes the equivalent gravity component of the current aircraft in the velocity coordinate system, the current aircraft speed, and the current aircraft absolute flight altitude, which are mainly used to represent the current attitude and energy state of each aircraft.

[0026] All the relative kinematic positions and velocity vectors in the above information are transformed into the aircraft's own velocity coordinate system using a cosine matrix and normalized to ensure that the data have relative spatial consistency.

[0027] Based on the above-mentioned original state observation information, the observed state of any aircraft in the aircraft cluster can be expressed as the state observation vector form shown in Formula 1.

[0028] ... Formula 1 in Indicates the first in the aircraft cluster The state observation vector of each aircraft Indicates the first Original target observation information of each aircraft Indicates the first Original neighbor observation information of each aircraft Indicates the first Original obstacle observation information for each aircraft Indicates the first The original flight status observation information of each aircraft.

[0029] Next, using the aforementioned state observation vectors and the current multiple attribute parameters of the aircraft cluster, we will construct the task reward function and the safety cost signal of the constrained Markov decision process, respectively, to provide optimization objectives for the subsequent construction of the aircraft cluster control model using the SSAC-PER algorithm.

[0030] In some implementations, please refer to Figure 3 Step 012 specifically includes: Step 0121: Based on the original target observation information, the preset first normalization factor, and the preset first weighting coefficient, construct the target assault guidance reward function. The attribute parameters include the first normalization factor and the first weighting coefficient; Step 0122: Based on the original target observation information, the preset second normalization factor, and the preset second weighting coefficient, construct the formation cooperative maintenance reward function based on the rigorous Lyapunov stability theory. The attribute parameters include the second normalization factor and the second weighting coefficient; Step 0123: Based on the original target observation information, preset warning boundary parameters, and preset reward coefficients, construct a dense obstacle avoidance guidance reward function. The attribute parameters include the warning boundary parameters and the reward coefficient; Step 0124: Construct the mission reward function based on the target assault guidance reward function, the formation coordination maintenance reward function, and the dense obstacle avoidance guidance reward function.

[0031] It should be noted that there is no fixed order between steps 0121, 0122, and 0123. They only need to be executed before step 0124. The order of these three steps is arbitrary, and they can also be executed simultaneously. Figure 3 The diagram shown is merely a flowchart and should not be construed as limiting the execution order of the three steps described above.

[0032] The mission reward function consists of three components: the target assault guidance reward function, the formation coordination maintenance reward function, and the dense obstacle avoidance guidance reward function, as shown in Formula 2.

[0033] ... Formula 2 in This represents the task reward function. This represents the reward function for target assault guidance. This indicates that the formation maintains a reward function in coordination. This represents the dense obstacle avoidance guided reward function.

[0034] First, for the target assault guided reward function Its main function is to enable any aircraft in the aircraft swarm to move quickly and stably toward the target. It mainly consists of three parts: a position progress reward sub-function, a velocity alignment reward sub-function, and a mission completion sparsity reward sub-function. When constructing the target assault guidance reward function, it is necessary to calculate multiple parameters related to the state observation based on the state observation vector described in the above implementation, as the data basis for the above three sub-functions. Simultaneously, one or more first normalization factors and first weight coefficients are selected from the attribute parameters exhibited based on the overall situation of the aircraft swarm, combined with the aforementioned multiple parameters related to the state observation to form the above three sub-functions. For example, please refer to... Figure 4 In some embodiments, step 0121 further includes: Step 01211: Based on the relative distance of any aircraft in the aircraft cluster to the target, a preset relative distance normalization factor, and a preset relative distance weighting coefficient, construct a position progress reward sub-function. The relative distance between any aircraft and the target is determined based on the original target observation information. The first normalization factor includes the relative distance normalization factor, and the first weighting coefficient includes the relative distance weighting coefficient. Step 01212: Based on the projection of the velocity of any aircraft in the aircraft cluster onto the target's line-of-sight direction, the preset velocity normalization factor, and the preset velocity weighting coefficient, construct a velocity alignment reward sub-function. The projection of the velocity of any aircraft in the direction of the target's line of sight is determined based on the original target observation information. The first normalization factor includes the velocity normalization factor, and the first weighting coefficient includes the velocity weighting coefficient. Step 01213: Based on the positional relationship between any aircraft in the aircraft swarm and the target, construct a sparse reward sub-function for task completion. The positional relationship is determined based on the original target observation information. The value of the sparse reward sub-function for task completion is 0 or a preset function value. The preset function value is greater than the maximum function value of the position progress reward sub-function and the velocity alignment reward sub-function. The preset function value is configured to represent a huge positive incentive. Step 01214: Construct the target assault guidance reward function based on the position progress reward sub-function, the speed alignment reward sub-function, and the task completion sparse reward sub-function.

[0035] It should be noted that there is no fixed order between steps 01211, 01212, and 01213. They only need to be executed before step 01214. The three steps can be executed in any order, or simultaneously. Figure 4 The diagram shown is merely a flowchart and should not be construed as limiting the execution order of the three steps described above.

[0036] Specifically, the construction process of the above three sub-functions will be explained in detail below.

[0037] First, for the position progress reward sub-function, the data determined based on the above-mentioned original target observation information includes the relative distance between a certain aircraft in the aircraft cluster and the target at the previous moment and the relative distance between the aircraft and the target at the current moment. The first normalization factor used includes the relative distance normalization factor, and the first weighting coefficient used includes the relative distance weighting coefficient. The specific calculation method is shown in Formula 3.

[0038] ... Formula 3 in The above refers to the position progress reward sub-function. The relative distance weighting coefficient, This represents the relative distance between a particular aircraft in the aircraft swarm and the target at the previous moment. This represents the relative distance between a particular aircraft in the aircraft swarm and the target at the current moment. This is the relative distance normalization factor mentioned above.

[0039] Secondly, the velocity alignment reward subfunction primarily serves to maintain the structural stability within the cluster, ensuring a stable relationship between the velocities of each aircraft relative to the target and preventing collisions or positional interference within the cluster as aircraft fly towards the target. Therefore, when constructing the velocity alignment reward subfunction, the data determined based on the aforementioned original target observation information is the projection of the velocity of a specific aircraft within the cluster onto the target's line-of-sight direction; that is, the projection of the aircraft's velocity onto the ray pointing from the aircraft as the origin towards the target. The first normalization factor also includes a velocity normalization factor, and the first weighting coefficient also includes a velocity weighting coefficient. The specific calculation method is shown in Formula 4.

[0040] ... Formula 4 in For the speed-aligned reward subfunction mentioned above, For speed weighting coefficients, This is the projection of the aircraft's own velocity onto the target's line-of-sight direction. This is the velocity normalization factor.

[0041] Secondly, the main function of the sparse reward subfunction for task completion is to provide clear positive guidance when an aircraft arrives near the target. When constructing the sparse reward subfunction for task completion, the data determined based on the aforementioned original target observation information is the positional relationship between any aircraft in the aircraft swarm and the target. Specifically, this relationship is expressed as the relationship between the current distance between the aircraft and the target and the target's own destruction radius, without needing to use the first normalization factor or the first weighting coefficient. Specifically, when the current distance between the aircraft and the target is greater than the target's own destruction radius, the function value of the sparse reward subfunction for task completion is set to 0. Conversely, when the current distance between the aircraft and the target is less than or equal to the target's own destruction radius, the function value of the sparse reward subfunction for task completion is set to a preset value much larger than the maximum value of the aforementioned position progress reward subfunction and velocity alignment reward subfunction, thus providing a significant positive incentive for aircraft arriving near the target.

[0042] Based on the above three sub-functions, the three are combined to form the target assault guidance reward function, as shown in Formula 5.

[0043] ... Formula 5 in This indicates the sparse reward subfunction for completing the above task.

[0044] In addition to the target assault guidance reward function, the mission reward function also includes the formation coordination maintenance reward function and the dense obstacle avoidance guidance reward function. The calculation and derivation process for the formation coordination maintenance reward function is based on rigorous Lyapunov stability theory, ensuring asymptotic stability from a control principle perspective. This addresses the issue of position divergence or oscillation that often occurs in aircraft swarms employing rigid position formations in current related technologies.

[0045] Based on the rigorous Lyapunov stability theory, the data determined from the aforementioned original target observation information includes the position coordination error and velocity coordination error of any aircraft in the aircraft swarm. It also includes the potential energy of the position error of any aircraft in the swarm under the condition of combining a virtual spring model, and the energy dissipation of any aircraft due to negative work under the condition of combining a virtual damping model. The second normalization factor used includes the position normalization factor and the velocity normalization factor, and the second weighting coefficients include the stiffness weighting coefficient and the damping weighting coefficient, respectively. The specific calculation of the formation coordination maintenance reward function is shown in Equation 6.

[0046] ... Formula 6 in To maintain the reward function for formation coordination, Indicates the first in the aircraft cluster Position error potential energy of an aircraft based on a virtual spring model Indicates the first Energy dissipation of an aircraft due to doing negative work, based on a virtual damping model. Indicates the first in the aircraft cluster Position coordination error of individual aircraft Indicates the first in the aircraft cluster Speed ​​coordination error of individual aircraft For the first Stiffness weighting coefficients for individual aircraft For the first Damping weighting coefficient for an aircraft For position normalization factor, This is the velocity normalization factor.

[0047] The main function of the dense obstacle avoidance guidance reward function is to set a warning buffer distance outside the radius of the physical obstacle, thereby forming a repulsive artificial potential field guidance at the linear level. Based on the aforementioned original target observation information, the data determined are the minimum distance between the aircraft cluster and the obstacle center, and the equivalent danger radius of the obstacle itself. Furthermore, the attribute parameters of the aircraft cluster also need to further select preset obstacle avoidance warning buffer distance boundaries (corresponding to warning boundary parameters) and flexible guidance reward coefficients (corresponding to reward coefficients). The specific calculation of the dense obstacle avoidance guidance reward function is shown in Formula 7.

[0048] ... Formula 7 in A reward function for dense obstacle avoidance. As the reward coefficient, The minimum distance between the aircraft cluster and the center of the obstacle. The equivalent danger radius of the obstacle. This indicates the warning boundary parameters.

[0049] Finally, the target assault guidance reward function, formation coordination maintenance reward function, and dense obstacle avoidance guidance reward function are combined according to Formula 2 to construct the mission reward function. .

[0050] In some implementations, step 02 further includes: A safety cost signal is constructed based on the minimum distance between the aircraft cluster and the center of the obstacle, the equivalent danger radius of the obstacle, and the preset safety buffer distance. The value of the safety cost signal is either 0 or 1.

[0051] Specifically, in addition to the task reward function mentioned above, the optimization objective of the subsequent process of constructing the aircraft swarm control model using the SSAC-PER algorithm also includes a safety cost signal for the constrained Markov decision process (CMDP). This safety cost signal is specifically used to trigger the hard constraints included in the CMDP. For example, this safety signal takes the form of a sparse indicator function, as shown in Equation 8.

[0052] ... Formula 8 in Signals of security costs The minimum distance between the aircraft cluster and the center of the obstacle. The equivalent danger radius of the obstacle. This is the safety buffer distance above the preset physical limit radius (corresponding to the safety buffer distance). At this point, the mission reward function and safety cost signal are complete. Next, the SSAC-PER algorithm can be used to construct the aircraft swarm control model based on the information determined in the above implementation.

[0053] In some implementations, please refer to Figure 5 Step 013 further includes: Step 0131: Based on the state observation vector and continuous action control instructions, construct a policy network, a dual-path task evaluation network, and a cost evaluation network using the safe soft actor-critic algorithm; Step 0132: Construct multiple loss functions based on the preset Lagrange multipliers, policy network, dual-path task evaluation network, and cost evaluation network; Step 0133: Construct a comprehensive priority parameter corresponding to any aircraft based on the state observation vector, continuous action control command, mission reward function, and safety cost signal corresponding to any aircraft in the aircraft cluster; Step 0134: Update the loss function according to the comprehensive priority parameter until all loss functions meet the preset convergence condition to construct the aircraft cluster control model.

[0054] Specifically, the next step is to construct an aircraft swarm control model based on the SSAC-PER algorithm. Before constructing the aircraft swarm control model, it is also necessary to pre-construct continuous motion control commands for any aircraft in the swarm based on the kinematic and dynamic parameters of any aircraft. For example, the continuous motion control commands for aircraft are defined as continuous motion space vectors, as shown in Equation 9.

[0055] ... Formula 9 in For continuous action space vectors, This indicates the normal overload component of the command, used to control the pitch maneuver of the aircraft. This indicates the lateral overload component of the command, used to control the yaw maneuver of the aircraft.

[0056] Furthermore, a hyperbolic tangent activation function can be used to limit the output motion of the aircraft to the range of [-1,1] based on the continuous motion space vector mentioned above, and a linear inverse mapping can be performed before interacting with the physical environment to strictly match the physical boundary constraints of the aircraft's maximum available overload.

[0057] Based on the continuous motion control commands defined in the above example, the next step is to construct an aircraft swarm control model. Based on the SSAC-PER algorithm, the model includes three types of neural networks: policy networks... Dual-path task evaluation network and cost evaluation network The three types of value networks mentioned above are independent of each other, and their input data are all continuous action space vectors as described in the above embodiments. and the state parameter vectors of each aircraft The aforementioned independent neural networks collectively form a deep neural network system, where the parameters of the policy network are... Its output is a continuous Gaussian distribution of motion for any aircraft (including the mean and variance of normal and lateral overload components). The parameters of the dual-path task evaluation network are: ,in Choosing either 1 or 2, its output consists of two estimates of the cumulative reward for long-term tasks. The parameters of the cost evaluation network are... Its output is the cumulative valuation of long-term security costs.

[0058] Next, based on the three sets of value networks mentioned above, multiple loss functions are constructed, and non-negative Lagrange multipliers are introduced. As a dual regulating valve, based on the task reward function and security cost signal in the above embodiments, the original CMDP constrained optimization objective is to maximize the expected task reward while constraining the long-term security cost, ensuring that the long-term security cost does not exceed a preset expected threshold. Then, according to the introduced Lagrange multipliers... A Lagrangian objective is constructed, and then Lagrangian dual relaxation is used to transform the hard constraints of CMDP into a penalty term with adaptive weights, thus transforming the constrained optimization problem into an unconstrained dual optimization problem. Before constructing the loss function, the alternating gradient update process implemented during model training is as follows: First, fix the Lagrange multipliers and update the policy network, dual-path task evaluation network, and cost evaluation network to maximize the constructed Lagrange objective. Then, fix the policy network and update the Lagrange multipliers on the dual gradient. If the obtained long-term security cost is greater than the preset long-term security cost expectation threshold, the Lagrange multipliers are further increased to amplify the penalty. If the long-term security cost is less than the preset long-term security cost expectation threshold, the Lagrange multipliers are decreased to weaken the security constraint. This process is repeated.

[0059] In the above update process, the construction and calculation of the above loss functions are as shown in Equation 10: ...Formula 10 in Let the value network loss function be... For the cost network loss function, The temperature coefficient loss function The loss function updated on the dual gradient of the Lagrange multipliers. This indicates a priority for the experience replay pool. It is the minimum value of the target dual-task evaluation network corresponding to the dual-task evaluation network. It is the target cost evaluation network corresponding to the cost evaluation network. It is a discount factor. The preset reward value, The value is the security cost signal value in the above implementation method. It is an adaptively adjustable temperature coefficient. For information entropy, The aforementioned expected threshold for long-term security costs, For Lagrange operators, Represents the state observation vector of the aircraft. The space vector representing the continuous motion of the aircraft.

[0060] Based on this, for policy networks In terms of its parameters Its role is to reduce the loss by updating the objective function, thereby maximizing the expected value including entropy in the computation. The loss function corresponding to the policy network is shown in Equation 11.

[0061] ... Formula 11 Next, based on the loss function structure described above, for each aircraft in each aircraft cluster, a binary tree is used to store the interaction tuple corresponding to that aircraft. This tuple generally includes at least the state observation vector. The continuous motion space vector of the aircraft The reward value corresponding to the aircraft The corresponding value of the aircraft The sparse reward sub-function value for mission completion corresponding to the aircraft In addition to the data, for each aircraft, two types of error information are calculated and stored in real time: the TD error of the reward evaluator and the TD error of the cost evaluator. Then, by adopting a priority sampling mechanism that fuses hybrid temporal differential TD errors, the comprehensive priority parameter corresponding to each interaction tuple (i.e., each aircraft) is calculated using the method shown in Formula 12: ... Formula 12 in For the first The overall priority parameters of each aircraft For the first The TD error of the reward judge for each aircraft For the first The TD error of the cost evaluator corresponding to each aircraft This is a preset error amplification factor used to adjust data sensitivity. It is a preset integer used to ensure that the probability of being selected is non-zero.

[0062] Therefore, in the above model training process, the experience replay pool is prioritized. As the network is continuously updated, based on the comprehensive priority parameters mentioned above, and according to the probability distribution shown in Formula 13, supplemented by the importance sampling weight shown in Formula 14, samples containing high-risk collisions or key state transitions are given high-level attention in the network update. This improves the sample utilization, convergence speed, and stability of the generated aircraft cluster control model in a strongly constrained high-dimensional continuous action space.

[0063] ... Formula 13 in For the first The probability distribution corresponding to the comprehensive priority parameters of each aircraft It is a priority index used to control the degree of priority tilt.

[0064] ... Formula 14 in For the first The importance sampling weights for each aircraft For the number of samples, The importance sampling index is used to correct for gradient distribution shifts caused by non-uniform sampling. Its value is iteratively adjusted with increasing iterations and eventually approaches 1. During model training, the value network loss function is calculated. and the cost network loss function At that time, the importance sampling weights mentioned above can be multiplied. To eliminate PER sampling error.

[0065] Based on this, after each batch of iterations and calculation of the loss function, the two types of errors mentioned above are recalculated, and the comprehensive priority parameters corresponding to each aircraft are iteratively updated. This process is repeated multiple times until a usable aircraft cluster control model is obtained.

[0066] In some implementations, please refer to Figure 6 Step 02 further includes: Step 021: Initialize the initial physical state of the aircraft cluster and initialize the aircraft cluster control model; Step 022: Obtain the relative observation status of any aircraft in the aircraft cluster; Step 023: Input the relative observation state into the policy network of the aircraft cluster control model to determine the continuous action commands for any aircraft; Step 024: Perform hyperbolic tangent activation limiting and linear inverse mapping on continuous action commands to make any aircraft meet the corresponding maximum available overload physical boundary constraints, so as to determine the control commands.

[0067] Specifically, based on the above implementation method and after training the aircraft cluster control model, the logic of constructing the aircraft cluster control model based on the above implementation method is used. The relative observation state of any aircraft in the aircraft cluster is used as input, and the aircraft cluster control model outputs continuous action commands for any aircraft. By repeating the above process, continuous action commands can be generated for all aircraft in the aircraft cluster, thereby realizing action control for the entire aircraft cluster. This enables flexible formation reconfiguration and safe flight around the aircraft cluster in a dynamic environment, until the aircraft cluster safely avoids all physical obstacles and no-fly zones and successfully reaches the mission target point.

[0068] Specifically, in some examples, an aircraft swarm control model is applied. First, the flight mission environment, the spatial topology of the aircraft swarm, and the initial physical state are initialized. At the same time, the initial parameters of each neural network (policy network, dual-path task evaluation network, cost evaluation network) are loaded for the already trained aircraft swarm control model.

[0069] Next, iterate through the entire aircraft cluster, targeting the first... One aircraft, within the control cycle Its relative observation state is acquired in real time, which is the state observation vector in Formula 1. .

[0070] Then, the state observation vector corresponding to each aircraft is... The policy network input to the above-mentioned aircraft cluster control model In the middle, policy network The deterministic continuous maneuver control command is output through forward propagation, which is the continuous motion space vector in Equation 9. .

[0071] Finally, for each aircraft, the corresponding continuous motion space vector The hyperbolic tangent activation function is used to transform the continuous action space vector. The corresponding output action is limited to the range [-1, 1] and undergoes a linear inverse mapping before interacting with the physical environment to strictly match the physical boundary constraints of the aircraft's maximum available overload. Based on the aforementioned limiting and linear inverse mapping, i.e., according to the continuous action space vector... Once the control commands for each aircraft are determined, these commands are distributed to the corresponding aircraft, which in turn drives each aircraft to perform pitch and yaw maneuvers, thereby controlling all aircraft in [the desired state]. The flight maneuvers that need to be executed at any given moment. Therefore, for any point in the entire control cycle, control commands for all aircraft can be generated by repeating the above process. This allows for maneuver control of all aircraft at any time, enabling flexible formation reconfiguration and safe flyaround of the aircraft swarm in a dynamic environment, until the aircraft swarm safely avoids all physical obstacles and no-fly zones and successfully reaches the mission objective.

[0072] Please see Figure 7 , Figure 7 This is a three-dimensional trajectory of an aircraft swarm controlled by the aircraft swarm formation maintenance and cooperative obstacle avoidance control method described in the above embodiments, in some implementations. The cylinders in the figure represent simulated physical obstacles or no-fly zones, and each curve corresponds to the flight trajectory of different aircraft within the swarm. Figure 7 It can be seen that the trained aircraft swarm control model can control an aircraft swarm consisting of multiple aircraft, and when encountering cylindrical obstacles or no-fly zones, it can adaptively and flexibly maneuver around them and safely reach the target area. Please continue reading. Figure 8 , Figure 8 for Figure 7 The planar projection of the three-dimensional trajectory shown is based on Figure 8 It can be seen that when the aircraft formation traversed the obstacle area, there were no obvious intersections or disorder between local flight paths. Each aircraft was able to complete coordinated obstacle avoidance maneuvers according to the spatial distribution of obstacles or no-fly zones, and quickly restore its configuration through a virtual spring-damping mechanism after passing the obstacle, maintaining the overall trend of advancing towards the target area. This achieved effective coordination between local safe obstacle avoidance and overall mission advancement. Please continue reading. Figure 9 as well as Figure 10 , Figure 9 yes Figure 7 The flight altitude change curves corresponding to each three-dimensional trajectory are shown below. Figure 10 yes Figure 7 The flight speed change curves corresponding to each three-dimensional trajectory are shown. According to... Figure 9 As shown in section 10, although the individual aircraft in the aircraft swarm may exhibit slight differences in maneuverability due to local obstacle avoidance requirements, the overall evolution trend of altitude and speed is smooth and continuous, without any violent oscillations or unreasonable abrupt changes. This indicates that the aircraft swarm control model trained in the above implementation method can not only successfully achieve the objectives of obstacle avoidance and formation configuration maintenance, but also that the output control commands meet the basic physical and dynamic continuity requirements during aircraft maneuvers.

[0073] Please see Figure 11 , Figure 11 This paper compares the aircraft swarm formation maintenance and cooperative obstacle avoidance control method using the SSAC-PER algorithm in the above embodiments with Monte Carlo simulation results of similar control methods using the SAC-PER algorithm and the original SAC algorithm under the same control conditions. The comparison results show that, under conditions of multiple obstacles and high-risk coupling constraints, the aircraft swarm formation maintenance and cooperative obstacle avoidance control method of this application can significantly reduce the collision risk between aircraft in the swarm and maintain a better formation configuration compared to the SAC-PER algorithm and the original SAC algorithm in current related technologies, ultimately improving the ability to complete the intended task.

[0074] The electronic device in this application includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the aircraft swarm formation maintenance and cooperative obstacle avoidance control method in the above-described embodiments is implemented.

[0075] Thus, this application utilizes a safe soft actor-critic algorithm. On the one hand, by mapping the position and velocity coordination error to virtual elastic potential energy and damping dissipation energy to derive a continuous cooperative reward function, it can fill the gap in hard safety constraints and solve the problem of reward function tampering. On the other hand, by introducing Lagrange multipliers into a constrained Markov decision process framework that decouples dense task rewards from sparse safety costs, it adaptively adjusts the priority of the experience replay strategy to ensure stable formation configuration for collision avoidance within the aircraft swarm and cooperative stable flight around obstacles outside the formation.

[0076] The above description is merely a preferred embodiment of this application and is not intended to limit this application in any way. Although this application has disclosed the preferred embodiment as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the technical solution of this application, based on the technical essence of this application and within the spirit and principles of this application, shall still fall within the protection scope of the technical solution of this application.

Claims

1. A method for maintaining and coordinating obstacle avoidance control in aircraft swarms, characterized in that, The method includes: Based on the original state observation information of the aircraft cluster, a control model for the aircraft cluster is constructed using a pre-defined safety soft actor-commentator algorithm. Based on the aforementioned aircraft cluster control model, and according to the relative observation state of any aircraft in the aircraft cluster, control commands are determined for any aircraft in the aircraft cluster to control the aircraft cluster to perform formation maintenance or cooperative obstacle avoidance.

2. The method according to claim 1, characterized in that, The process involves constructing a flight cluster control model based on the initial state observation information of the flight cluster and a pre-defined safe soft actor-commentator algorithm, including: The original state observation information of the aircraft cluster is obtained to determine the state observation vector of any aircraft in the aircraft cluster, wherein the original state observation information includes the original target observation information, original neighbor observation information, original obstacle observation information, and original self-flight state observation information of any aircraft in the aircraft cluster. Based on the state observation vector and the attribute parameters of the aircraft cluster, a mission reward function and a safety cost signal for the constrained Markov decision process are constructed. Based on the continuous motion control commands of the aircraft cluster, the state observation vector, the mission reward function, and the safety cost signal, a control model for the aircraft cluster is constructed using a preset safety soft actor-commentator algorithm. The continuous motion control commands are pre-constructed based on the kinematic and dynamic parameters of any aircraft in the aircraft cluster.

3. The method according to claim 2, characterized in that, The step of constructing the mission reward function and the safety cost signal of the constrained Markov decision process based on the state observation vector and the attribute parameters of the aircraft cluster includes: Based on the original target observation information, the preset first normalization factor and the preset first weight coefficient, a target assault guidance reward function is constructed, wherein the attribute parameters include the first normalization factor and the first weight coefficient. Based on the original target observation information, the preset second normalization factor, and the preset second weighting coefficient, a formation cooperative maintenance reward function is constructed based on the strict Lyapunov stability theory, wherein the attribute parameters include the second normalization factor and the second weighting coefficient. Based on the original target observation information, the preset warning boundary parameters, and the preset reward coefficient, a dense obstacle avoidance guidance reward function is constructed, wherein the attribute parameters include the warning boundary parameters and the reward coefficient; The mission reward function is constructed based on the target assault guidance reward function, the formation coordination maintenance reward function, and the dense obstacle avoidance guidance reward function.

4. The method according to claim 3, characterized in that, The step of constructing a target assault guidance reward function based on the original target observation information, a preset first normalization factor, and a preset first weighting coefficient includes: Based on the relative distance of any aircraft in the aircraft cluster from the target, a preset relative distance normalization factor, and a preset relative distance weighting coefficient, a position progress reward subfunction is constructed, wherein the relative distance of any aircraft from the target is determined based on the original target observation information, the first normalization factor includes the relative distance normalization factor, and the first weighting coefficient includes the relative distance weighting coefficient. Based on the projection of the velocity of any aircraft in the aircraft cluster onto the target line of sight, a preset velocity normalization factor, and a preset velocity weighting coefficient, a velocity alignment reward subfunction is constructed. The projection of the velocity of any aircraft onto the target line of sight is determined based on the original target observation information. The first normalization factor includes the velocity normalization factor, and the first weighting coefficient includes the velocity weighting coefficient. Based on the positional relationship between any aircraft in the aircraft cluster and the target, a sparse reward subfunction for task completion is constructed, wherein the positional relationship is determined based on the original target observation information. The value of the sparse reward subfunction for task completion is 0 or a preset function value. The preset function value is greater than the maximum function value of the position progress reward subfunction and the velocity alignment reward subfunction. The preset function value is configured to represent a large positive incentive. The target assault guidance reward function is constructed based on the position progress reward subfunction, the speed alignment reward subfunction, and the task completion sparse reward subfunction.

5. The method according to claim 3, characterized in that, The formula for the formation cooperative maintenance reward function is: in To maintain the reward function for the formation coordination, Indicates the first in the aircraft cluster Position error potential energy of an aircraft based on a virtual spring model Indicates the first in the aircraft cluster Energy dissipation of an aircraft due to doing negative work, based on a virtual damping model. Indicates the first in the aircraft cluster Position coordination error of individual aircraft Indicates the first in the aircraft cluster Speed ​​coordination error of individual aircraft The first in the aircraft cluster Stiffness weighting coefficients for individual aircraft The first in the aircraft cluster Damping weighting coefficient for an aircraft For position normalization factor, This is the velocity normalization factor; The position coordination error and the velocity coordination error are determined based on the original target observation information. The second normalization factor includes the position normalization factor and the velocity normalization factor. The second weighting coefficient includes the stiffness weighting coefficient and the damping weighting coefficient.

6. The method according to claim 3, characterized in that, The formula for the dense obstacle avoidance guidance reward function is: in This refers to the dense obstacle avoidance guided reward function. The reward coefficient is... This is the minimum distance between the aircraft cluster and the center of the obstacle. The equivalent danger radius of the obstacle. Indicates the warning boundary parameters; The minimum distance and the equivalent danger radius are determined based on the original target observation information.

7. The method according to claim 3, characterized in that, The step of constructing the mission reward function and the safety cost signal of the constrained Markov decision process based on the state observation vector and the attribute parameters of the aircraft cluster further includes: The safety cost signal is constructed based on the minimum distance between the aircraft cluster and the center of the obstacle, the equivalent danger radius of the obstacle, and the preset safety buffer distance, wherein the value of the safety cost signal is 0 or 1.

8. The method according to claim 2, characterized in that, The process of constructing a control model for the aircraft swarm based on the continuous action control commands of the aircraft swarm, the state observation vector, the mission reward function, and the safety cost signal, using a preset safety soft actor-commentator algorithm, includes: Based on the state observation vector and the continuous action control command, a policy network, a dual-path task evaluation network, and a cost evaluation network are constructed using the safe soft actor-critic algorithm. Multiple loss functions are constructed based on the preset Lagrange multipliers, the policy network, the dual-path task evaluation network, and the cost evaluation network; Based on the state observation vector, continuous action control command, mission reward function and safety cost signal corresponding to any aircraft in the aircraft cluster, a comprehensive priority parameter corresponding to any aircraft is constructed. The loss function is updated based on the comprehensive priority parameters to construct the aircraft cluster control model.

9. The method according to claim 1, characterized in that, The method based on the aircraft swarm control model, determining control commands for any aircraft in the swarm according to the relative observation state of any aircraft in the swarm, to control the aircraft swarm to perform formation keeping or cooperative obstacle avoidance, includes: Initialize the initial physical state of the aircraft cluster and initialize the aircraft cluster control model; Obtain the relative observation state of any aircraft in the aircraft cluster; The relative observation state is input into the policy network of the aircraft cluster control model to determine the continuous action commands for any of the aircraft. The continuous action commands are subjected to hyperbolic tangent activation limiting and linear inverse mapping to ensure that the arbitrary aircraft meets the corresponding maximum available overload physical boundary constraints, thereby determining the control commands.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the aircraft swarm formation maintenance and cooperative obstacle avoidance control method as described in any one of claims 1-9.

Images