High-speed aircraft cluster collaboration method and system based on safety reinforcement learning

Abstract

The invention provides a high-speed aircraft cluster collaboration method and system based on safety reinforcement learning, and belongs to the field of aircraft cluster collaboration. The objective of the invention is to solve the security problem caused by uninterpretability when an existing reinforcement learning algorithm is applied to a high-speed aircraft cluster collaborative task. According to the method, a security reinforcement learning theory is introduced into an aircraft cluster intelligent decision-making problem, an improved soft actor-commentator algorithm is proposed based on a Lyapunov stability theory, a dynamic model taking normal overload as a control quantity is established, tasks are divided into target strike, formation maintenance and threat area avoidance, and an aircraft guide reward function is designed. Through comparison with an actor-commentator algorithm, a conclusion that the method can enable the agent to keep low safety cost in the whole training process is obtained. According to the method, the high-speed aircraft cluster collaborative intelligent strategy with partial security interpretability can be obtained, so that the situation that a threat area evasion task fails due to improper design of a reward function is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft cluster collaboration, and in particular to a high-speed aircraft cluster collaboration method and system based on safety reinforcement learning. Background Art

[0002] As a multi-domain, high-efficiency mission carrier, high-speed aircraft has become a hot topic for research in countries around the world. Due to the small field of view and single mission type of a single aircraft, the development of high-speed aircraft swarm technology has brought new requirements and applications in the military and civilian fields. Dynamic uncertain environments and complex tasks determine that high-speed aircraft swarm systems are bound to develop in the direction of autonomy and intelligence. In order to achieve this goal, how to combine swarm intelligence algorithms with the complex behavior of swarms has become a key issue.

[0003] Reinforcement learning algorithms have been widely used in many fields, such as video games, communication networks, and transportation scheduling. However, since the reinforcement learning training process completely relies on the traction of the reward function, the intelligent agent is prone to speculative behavior in the optimization process, which makes it difficult to ensure the safety of its training results in the actual application of aircraft clusters. For example, if the threat area is regarded as an unsafe area in reinforcement learning, when the combat mission scenario changes greatly, the aircraft may slightly enter the threat area in order to speed up the completion of the mission. So, how to ensure that the aircraft cluster completely avoids the threat area and always stays in the safe area in any combat scenario is a key issue that needs to be explored in depth.

[0004] This problem essentially comes from the unexplainability of neural networks when fitting the functional relationship between input and output. Therefore, the present invention improves the soft actor-critic algorithm in combination with the Lyapunov stability constraint, and designs a learning framework for high-speed aircraft cluster collaborative tasks based on this algorithm, and proposes a high-speed aircraft cluster collaborative method based on safety reinforcement learning. This method can ensure that the aircraft cluster always stays in the safe area, thereby effectively avoiding the failure of the threat zone avoidance task due to improper reward function design. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In order to solve the safety problems caused by the unexplainability of existing reinforcement learning algorithms when applied to high-speed aircraft cluster collaborative tasks.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The present invention provides a high-speed aircraft cluster collaboration method based on safety reinforcement learning, comprising the following steps:

[0009] S100, mathematically modeling the cluster cooperative control problem of high-speed aircraft using the three-degree-of-freedom model of the aircraft;

[0010] S200, according to the requirements of the high-speed aircraft cluster coordination task, the task is divided into target strike, formation maintenance and threat zone avoidance, thereby designing the state space and action space in the reinforcement learning training process;

[0011] S300, designing a reward function in the reinforcement learning training process according to the relationship between the task requirements and the designed state quantity; corresponding to the design of the state space, the design of the reward function includes formation maintenance, target strike and threat zone avoidance, wherein the formation maintenance reward includes a relative position reward and a relative speed reward;

[0012] S400, using the Lyapunov method, improves the SAC algorithm, constructs the Lyapunov network, and designs a high-speed aircraft cluster collaborative reinforcement learning training framework with safety assurance, including a policy network, a value network, and a Lyapunov network; among them, the policy network observes the state of the high-speed aircraft cluster to generate actions, the value network evaluates the value of executing the action in the current state, and the Lyapunov network evaluates the safety cost of the current action; according to the Lyapunov theory, the safety cost eventually converges to a specific value, which is artificially adjustable.

[0013] Further, in step S100, the control amount of the high-speed aircraft includes overloads in three directions [n x n y n z ] T , its state quantity is [Vθψ V xyz] T , the kinematic model of the high-speed aircraft can be expressed as follows:

[0014]

[0015] Among them, V is the speed of the high-speed aircraft, θ is the ballistic inclination angle, ψ V is the trajectory deviation angle, and [x, y, z] is the position coordinate of the high-speed aircraft in the ground coordinate system.

[0016] Furthermore, in step S200, it specifically includes:

[0017] S210, design the formation to maintain the observation vector, including,

[0018] S211. Observe the relative position information. The relative position deviation between the i-th adjacent aircraft and the current aircraft is:

[0019]

[0020] Among them, R i is the relative position between the neighboring aircraft i and the current aircraft, is the relative position between the neighboring aircraft i and the current aircraft at the initial moment;

[0021] The observation vector of the relative position of the aircraft is composed of the relative position deviation vectors ΔR of all neighboring aircraft i Stitched together:

[0022]

[0023] Among them, R form is the distance scaling factor, which makes all dimensions in the observation vector at the same order of magnitude;

[0024] S212. Observe the relative speed information and calculate the average movement speed of the neighboring aircraft except the current aircraft:

[0025]

[0026] Among them, V i is the velocity vector of the i-th adjacent aircraft, w i is the weight of the speed of the i-th aircraft, calculated as follows:

[0027]

[0028] Among them, k i =||R i || / R l , R l is the scaling factor;

[0029] Calculate speed deviation The observation vector form of relative velocity is:

[0030]

[0031] Among them, V form is the speed scaling factor;

[0032] Get the observation vector maintained by the formation:

[0033] s form =[s pos ,s vel ].

[0034] Furthermore, in step S300, it specifically includes:

[0035] S310, design formation to maintain rewards,

[0036] Including relative position rewards, by weighting the positions of all aircraft in the cluster, the position of the virtual leader (x c ,y c ,z c ) as a reference; the relative position deviation between the current aircraft and the leading aircraft is:

[0037] ΔR c =R c -R c0

[0038] Among them, R c is the relative position between the current aircraft and the leading aircraft, R c0 is the relative position between the i-th aircraft and the leading aircraft at the initial moment;

[0039] Add formation deviation from neighboring aircraft:

[0040]

[0041] Among them, i min is the number of the nearest neighboring aircraft to the current aircraft. and are the current time and the initial time respectively, the current aircraft relative to the neighboring aircraft i min The relative position information of

[0042] The relative position reward is:

[0043]

[0044] Among them, λ is a negative coefficient;

[0045] It also includes relative speed bonus. The relative speed bonus between individual aircraft and the cluster is:

[0046]

[0047] Among them, μ is a negative coefficient;

[0048] S320, Design target strike reward,

[0049] The high-speed aircraft cluster needs to hit the target at a specified time. If it hits the target, it will be rewarded. If it misses the target, it will not be rewarded. In order to assist the training convergence, speed rewards, position rewards and distance change rewards are designed.

[0050] The speed reward function is:

[0051]

[0052] Among them, V taris the vector product of the aircraft velocity vector and the target point relative position vector, ρ is a positive coefficient, V tar is the scaling factor;

[0053] When the distance between the aircraft and the target point is R tar Get additional position bonus when:

[0054]

[0055] Among them, κ is a positive reward coefficient, R tar is the approach distance. When the distance between the aircraft and the target point is less than R tar When , the aircraft is considered to be close to the target point, and according to the artificial potential field method, a reward inversely proportional to the relative distance is applied; R done is the hitting distance. When the distance between the aircraft and the target point is less than R done When the target is hit, a larger reward is obtained;

[0056] The distance change reward is related to the change in distance between the vehicle and the target point at each simulation step:

[0057]

[0058] Among them, ||T|| t and ||T|| t+1 are the distances between the aircraft and the target at the t-th step and the t+1-th step, ζ is a positive coefficient, T tar is the step size scaling factor;

[0059] S330, Design threat zone avoidance reward,

[0060] The reward function for threat zone avoidance is designed as follows:

[0061]

[0062] Among them, η and σ are negative reward coefficients;

[0063] The overall reward function is:

[0064]

[0065] Furthermore, in step S400, it specifically includes:

[0066] The SAC algorithm is improved by using the Lyapunov method; for a given safety constraint If there exists a function L(s):S→R + and the positive constant α 1 , α 2 , α 3 , ηs ,e, the system is uniformly bounded under the average security cost and the final bound is

[0067]

[0068] in, is the average probability that s is in the edge set Δ under strategy π;

[0069] The Lyapunov method includes:

[0070] (1) Input hyperparameters, learning rate α θ , UUB theory related parameters e, η s , α 3 ;

[0071] (2) Randomly initialize a critic network Q(s,a) and a Lyapunov critic network L c (s,a), and a parameter φ Q , The actor network π(a|s) of θ and the Lagrange multipliers β,λ s ,ν s ;

[0072] (3) Repeat steps (4) to (11) until the total number of rounds reaches a set value;

[0073] (4) According to ρ s Sampling to get the initial state s 0 ;

[0074] (5) Repeat steps (6) to (8) until the current round ends;

[0075] (6) Sample action a from the π(s) distribution t ;

[0076] (7) Observe the next moment s t+1 , r t , c t , and (s t ,a t ,r t ,c t ,s t+1 ) is stored in the experience pool D; where s t+1 represents the state at the next moment, r t represents the current moment reward, c t Represents the current moment price;

[0077] (8) If s t ∈Δ, (s t,a t ,r t ,c t ,s t+1 ) Stored in experience pool D Δ middle;

[0078] (9) When the current time step is the time step that needs to update the network, run steps (10) to (11);

[0079] (10) From the experience pools D and D Δ Sampling and updating Q, L according to the gradient c ,π and Lagrange multipliers;

[0080] (11) Update the target network based on the soft substitution.

[0081] The method further includes designing a high-speed aircraft cluster training scenario, training through the training framework obtained in step S400 to obtain a high-speed aircraft cluster flight strategy network; then designing a high-speed aircraft cluster test scenario, and evaluating the training strategy through an overload disturbance test.

[0082] The present invention provides a high-speed aircraft cluster collaboration system based on safety reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned high-speed aircraft cluster collaboration method based on safety reinforcement learning during operation.

[0083] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a high-speed aircraft cluster collaboration method based on safety reinforcement learning when called by a processor.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] The present invention uses the Lyapunov method to constrain unsafe behaviors in the process of learning the high-speed aircraft cluster collaborative strategy based on reinforcement learning, and can achieve the acquisition of a partially safety-explainable high-speed aircraft cluster collaborative intelligent strategy. Compared with directly using the traditional reinforcement learning framework, the safe reinforcement learning method has the following advantages: 1) Using the Lyapunov network to replace the penalty term representing the risk behavior in the reward function can decouple the multi-objective complex reward function; 2) Using the Lyapunov stability theory, it can ensure that the risk behavior of the aircraft cluster converges to an adjustable value after training;

[0086] The present invention provides a safer, controllable and efficient solution for the coordinated combat of high-speed aircraft clusters by combining reinforcement learning methods with Lyapunov theory, which can greatly improve the cluster's execution capability in complex tasks while effectively avoiding potential risks and threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a flow chart of a high-speed aircraft cluster collaboration method based on safety reinforcement learning in an embodiment of the present invention;

[0088] Figure 2 is a definition diagram of a set in a state space in an embodiment of the present invention;

[0089] Figure 3 This is a schematic diagram of a value function network architecture in an embodiment of the present invention;

[0090] Figure 4 This is a schematic diagram of a policy network architecture in an embodiment of the present invention;

[0091] Figure 5 A schematic diagram of a simulation scene in an embodiment of the present invention;

[0092] Figure 6 It is a comparison curve of the cumulative reward mean of the LSAC algorithm of the present invention and the existing SAC algorithm in the embodiment of the present invention;

[0093] Figure 7 Schematic diagram of flight trajectory comparison between the LSAC algorithm of the present invention and the existing SAC algorithm in an embodiment of the present invention, wherein (a) is a flight trajectory diagram of an aircraft cluster based on the LSAC algorithm, and (b) is a flight trajectory diagram of an aircraft cluster based on the SAC algorithm. DETAILED DESCRIPTION

[0094] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0095] Specific implementation plan 1: Combine Figures 1 to 3 As shown, the present invention provides a high-speed aircraft cluster collaboration method based on safety reinforcement learning, comprising the following steps:

[0096] S100, mathematically modeling the cluster cooperative control problem of high-speed aircraft using the three-degree-of-freedom model of the aircraft, specifically including:

[0097] The control quantity of high-speed aircraft includes overload in three directions [n x n y n z ] T , its state quantity is [Vθψ V xyz] T , the kinematic model of the high-speed aircraft can be expressed as follows:

[0098]

[0099] Among them, V is the speed of the high-speed aircraft, θ is the ballistic inclination angle, ψ V is the trajectory deviation angle, [x, y, z] is the position coordinate of the high-speed aircraft in the ground coordinate system;

[0100] S200, according to the requirements of the high-speed aircraft cluster coordination task, the task is divided into target attack, formation maintenance and threat zone avoidance, and the state space and action space in the reinforcement learning training process are designed, including:

[0101] S210, design the formation to maintain the observation vector, including,

[0102] S211. Observe the relative position information. The relative position deviation between the i-th adjacent aircraft and the current aircraft is:

[0103]

[0104] Among them, R i is the relative position between the neighboring aircraft i and the current aircraft, is the relative position between the neighboring aircraft i and the current aircraft at the initial moment;

[0105] The observation vector of the relative position of the aircraft is composed of the relative position deviation vectors ΔR of all neighboring aircraft i Stitched together:

[0106]

[0107] Among them, R form is the distance scaling factor, which makes all dimensions in the observation vector at the same order of magnitude;

[0108] S212. Observe the relative speed information and first calculate the average movement speed of the neighboring aircraft except the current aircraft:

[0109]

[0110] Among them, V i is the velocity vector of the i-th adjacent aircraft, w i is the weight of the speed of the i-th aircraft, calculated as follows:

[0111]

[0112] Among them, k i =||R i ‖ / R l , R l is the scaling factor;

[0113] The speed deviation can then be calculated Therefore, the observation vector of relative velocity is in the form of:

[0114]

[0115] Among them, V form is the speed scaling factor;

[0116] In summary, the observation vector maintained by the formation is obtained:

[0117] s form =[s pos ,s vel ]

[0118] S300, designing a reward function in the reinforcement learning training process according to the relationship between the task requirements and the designed state quantity; corresponding to the design of the state space, the design of the reward function also includes three parts: formation maintenance, target strike and threat zone avoidance, among which the formation maintenance reward includes relative position reward and relative speed reward, specifically including:

[0119] S310, Formation Maintenance Reward

[0120] The first is the relative position reward. By weighting the positions of all aircraft in the cluster, the position of a virtual “leader” (x c ,y c ,z c ) as a reference; the relative position deviation between the current aircraft and the "leader" is:

[0121] ΔR c =R c -R c0

[0122] Among them, R c is the relative position between the current aircraft and the “leader”, R c0 is the relative position between the i-th aircraft and the “leader” at the initial moment;

[0123] On this basis, the formation deviation from the adjacent aircraft is added to supplement it:

[0124]

[0125] Among them, i min is the ID of the nearest neighboring aircraft to the current aircraft. and are the current time and the initial time respectively, and the current aircraft is relative to the neighboring aircraft i min The relative position information of

[0126] In summary, the relative position reward is:

[0127]

[0128] Among them, λ is a negative coefficient;

[0129] The second is the relative speed reward. The relative speed reward between individual aircraft and the cluster is:

[0130]

[0131] Among them, μ is a negative coefficient;

[0132] S320, Target Attack Reward

[0133] The high-speed aircraft swarm needs to hit the target at a specified time. If it hits the target, it will be rewarded. If it misses the target, it will not be rewarded. Since the rewards are too sparse at this time, speed rewards, position rewards, and distance change rewards are designed to assist training convergence.

[0134] The speed reward function is:

[0135]

[0136] Among them, V tar is the vector product of the aircraft velocity vector and the target point relative position vector, ρ is a positive coefficient, V tar is the scaling factor;

[0137] When the distance between the aircraft and the target point is R tar Additional position rewards can be obtained:

[0138]

[0139] Among them, κ is a positive reward coefficient, R tar is the approach distance. When the distance between the aircraft and the target point is less than R tar When , the aircraft is considered to be close to the target point, and according to the artificial potential field method, a reward inversely proportional to the relative distance is applied; R done is the hitting distance. When the distance between the aircraft and the target point is less than R done When the target is hit, a larger reward is obtained. Without this item, the training results sometimes tend to circle around the target.

[0140] The distance change reward is related to the change in distance between the vehicle and the target point at each simulation step:

[0141]

[0142] Among them, ‖T|| t and ||T|| t+1 are the distances between the aircraft and the target at the t-th step and the t+1-th step, ζ is a positive coefficient, Ttar is the step size scaling factor;

[0143] S330, Threat Zone Avoidance Reward

[0144] The reward function for threat zone avoidance is designed as follows:

[0145]

[0146] Among them, η and σ are negative reward coefficients;

[0147] In summary, the total reward function is:

[0148]

[0149] S400, using the Lyapunov method, improve the SAC algorithm, build a Lyapunov network, and design a high-speed aircraft cluster collaborative reinforcement learning training framework with safety assurance, including a policy network, a value network, and a Lyapunov network; among them, the policy network observes the state of the high-speed aircraft cluster to generate actions, the value network evaluates the value of executing the action in the current state, and the Lyapunov network evaluates the safety cost of the current action; according to the Lyapunov theory, the safety cost will eventually converge to a specific value, which is artificially adjustable, including:

[0150] The SAC algorithm is improved by using the Lyapunov method;

[0151] Constrained Markov decision process (CMDP) is a basic framework for safety reinforcement learning, which can achieve the goal of satisfying safety constraints while maximizing rewards. s ) represents; where c(s,a) needs to be kept below a certain safety threshold The constraint function is s, s represents the state, and a represents the action; the security cost function under the strategy π is defined as:

[0152]

[0153] like Figure 2 As shown, in the state space is the starting set, the initial state of the agent is selected from the starting set, and the cost function of the agent in this state is less than the constraint b; is a safe set, the cost function of the agent in this state is less than the constraint It is a subset of security costs; is the inner set, which is the state space where the cost function is close to 0, and is also the optimization goal of this paper; the opposite of the inner set B is the edge set is the set of states in or close to the unsafe area, B∪Δ=S;

[0154] In addition, t 0 represents the moment when the trajectory enters the edge set Δ, P(t 0 |ρ,π) represents t 0 Distribution under strategy π; for a whole Markov decision sequence, the trajectory may stay in the edge set Δ or enter the inner set B. T represents the residence time of the trajectory in the edge set Δ, and the maximum residence time of all trajectories is recorded as N;

[0155] It can be proved that when the following equation holds, for a given safety constraint If there is a function and the positive constant α 1 , α 2 , α 3 , η s , e, where S represents the state set; the system is uniformly bounded under the average safety cost and the final bound is

[0156]

[0157] in, is the average probability that s is in the edge set Δ under strategy π;

[0158] The specific steps include:

[0159] (1) Input hyperparameters: learning rate α θ , UUB (uniformly ultimately bounded) theory related parameters e, η s , α 3 ;

[0160] (2) Randomly initialize a critic network Q(s,a) and a Lyapunov critic network L c (s,a), and a parameter φ Q , The actor network π(a|s) of θ and the Lagrange multipliers β,λ s ,ν s ;

[0161] (3) Repeating steps (4) to (11) until the total number of rounds is set, wherein each repetition is considered one round;

[0162] (4) According to ρ s Sampling to get the initial state s 0 ;

[0163] (5) Repeat steps (6) to (8) until the current round ends; each repetition represents one time step;

[0164] (6) Sample action a from the π(s) distribution t ;

[0165] (7) Observe the next moment s t+1 , r t , c t , and (s t ,a t ,r t ,c t ,s t+1 ) is stored in the experience pool D; where s t+1 represents the state at the next moment, r t represents the current moment reward, c t Represents the current moment price;

[0166] (8) If s t ∈Δ, (s t ,a t ,r t ,c t ,s t+1 ) Stored in experience pool D Δ middle;

[0167] (9) When the current time step is the time step that needs to update the network, run steps (10) to (11);

[0168] (10) From the experience pools D and D Δ Sampling and updating Q, L according to the gradient c ,π and Lagrange multipliers;

[0169] (11) updating the target network according to the soft substitution;

[0170] The value function network is as follows Figure 3 As shown in the figure, its input includes not only cluster formation, target attack and threat zone avoidance information, but also control instructions for the intelligent agent; the number of nodes in the fully connected layer is set to 128. Since a single hidden layer is difficult to fit complex nonlinear relationships, a total of two hidden layers are set; the activation function uses ReLU, and finally the corresponding value function is output, that is, Figure 3 The Q function in ;

[0171] Strategy network Figure 4 As shown in the figure, the input layer is first an observation vector containing 47 elements, then passes through a 2-layer hidden layer with 128 nodes, and the activation function is ReLU; finally, the mean and variance of the overload instruction are output through a 2-node fully connected layer, and the activation functions are Linear and Tanh respectively;

[0172] S500, designing a high-speed aircraft cluster training scenario, training through the training framework obtained in step S400, and obtaining a high-speed aircraft cluster flight strategy network, specifically including:

[0173] The aircraft cluster can be composed of 10 identical high-speed aircraft. The initial formation configuration is an equilateral triangle, and the initial distance between the aircraft is 200m. The initial speed of the aircraft cluster is 238m / s, the initial flight altitude is 1500m, and the attack target is a random position on the ground 30km away from the aircraft. The threat zone is set as a hemispherical area with a radius of 2000m on the ground, evenly distributed on a circle 15km away from the initial position of the cluster, with a total of 8 threat zones, such as Figure 5 As shown;

[0174] At the initial moment, the trajectory inclination of the aircraft is 0. In order to improve the convergence speed of training, the flight speed direction points to the target, and the maximum overload of the aircraft is 2. The maximum number of rounds in the training process is still set to 2000 times. The curve of the cumulative reward of the high-speed aircraft cluster system in an obstacle environment changing with the total number of training rounds is as follows: Figure 6 As shown;

[0175] from Figure 6 It can be found that the agent's training reward value basically remains above 0 after about 500 rounds, indicating that it has learned a strategy that can successfully hit the target; compared with the curve of the cumulative reward based on the SAC algorithm changing with the number of rounds, the LSAC algorithm improves the efficiency of training. The reason for this is that the constraints of the Lyapunov function are stronger than the constraints of the threat zone avoidance reward function, which reduces unnecessary exploration during training;

[0176] S600, design high-speed aircraft cluster test scenarios, evaluate training strategies through overload disturbance tests,

[0177] Figure 7 (a) is the flight trajectory of the aircraft cluster based on the LSAC algorithm, Figure 7 (a) It can be seen that the flight trajectory completely bypasses the threat zone and arrives at the target, verifying that the LSAC algorithm can achieve the characteristic of the intelligent agent staying in the safe area. Compared with the situation based on the SAC algorithm where the flight trajectory partially enters the threat zone and requires manual adjustment of the reward function, the LSAC algorithm improves the intelligence of the training environment construction process.

[0178] Due to the influence of aerodynamic factors in the actual flight process, the aircraft cannot obtain the predetermined overload. The following test is conducted to see whether the trajectory trained under the condition of overload disturbance can complete the task. The test results are shown in Tables 1 and 2. The arrival time is the proportion of the flight time within the given time of 300s.

[0179] Table 1

[0180] Overload disturbance amplitude award Arrival time Whether the target is hit Number of entry into threat zones -30％ -19.08 1.0 no 9 -29％ -18.67 1.0 no 4 -28％ -17.86 1.0 no 0 -27％ 117.3 0.43 yes 3 -26％ 118.0 0.43 yes 0 -25％ 117.2 0.4267 yes 0 -10％ 117.5 0.42 yes 0 0 118.8 0.42 yes 0 +3％ 120.2 0.4217 yes 0 +4％ -13.2 1.0 no 0 +5％ -13.48 1.0 no 0 +10％ -14.88 1.0 no 0

[0181] As shown in Table 1, when the overload disturbance is between -26% and +3%, the learned strategy can hit the target while avoiding the threat zone. When the overload disturbance amplitude is -27%, the learned strategy can hit the target but fails to completely avoid the threat zone. The reason for this is that the overload is too small, resulting in an insufficient turning radius of the aircraft, which cannot completely bypass the threat zone.

[0182] Table 2

[0183]

[0184]

[0185] Table 2 shows the test results of the LSAC algorithm under overload disturbance. It can be seen from Table 2 that when the overload disturbance is between -30% and +10%, the learned strategy can achieve the target strike. At this time, as the disturbance amplitude increases, the reward tends to increase slightly. The reason for this may be that the increase in flight time causes the aircraft to obtain more distance change rewards. In all test results, the aircraft cluster successfully avoided obstacles, indicating that the learned strategy is highly secure and the LSAC algorithm can well ensure the safety of the aircraft cluster.

[0186] Specific implementation scheme 2: The present invention provides a high-speed aircraft cluster collaboration system based on safety reinforcement learning. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned high-speed aircraft cluster collaboration method based on safety reinforcement learning during operation.

[0187] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0188] Specific implementation scheme three: The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a high-speed aircraft cluster collaboration method based on safety reinforcement learning when called by a processor.

[0189] The other combinations and connection relationships of this embodiment are the same as those of the first embodiment.

[0190] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A high-speed aircraft cluster collaboration method based on safety reinforcement learning, characterized in that: The following steps are involved: S100, mathematically modeling the cluster cooperative control problem of high-speed aircraft using the three-degree-of-freedom model of the aircraft; S200, according to the requirements of the high-speed aircraft cluster coordination task, the task is divided into target strike, formation maintenance and threat zone avoidance, thereby designing the state space and action space in the reinforcement learning training process; S300, designing a reward function in the reinforcement learning training process according to the relationship between the task requirements and the designed state quantity; corresponding to the design of the state space, the design of the reward function includes formation maintenance, target strike and threat zone avoidance, wherein the formation maintenance reward includes a relative position reward and a relative speed reward; S400, using the Lyapunov method, improves the SAC algorithm, constructs the Lyapunov network, and designs a high-speed aircraft cluster collaborative reinforcement learning training framework with safety assurance, including a policy network, a value network, and a Lyapunov network; among them, the policy network observes the state of the high-speed aircraft cluster to generate actions, the value network evaluates the value of executing the action in the current state, and the Lyapunov network evaluates the safety cost of the current action; according to the Lyapunov theory, the safety cost eventually converges to a specific value, which is artificially adjustable.

2. The high-speed aircraft cluster collaboration method based on safety reinforcement learning according to claim 1 is characterized in that: In step S100, specifically, the control amount of the high-speed aircraft includes overload in three directions [n x n y n z ] T , its state quantity is [Vθψ V xyz] T , the kinematic model of the high-speed aircraft can be expressed as follows: Among them, V is the speed of the high-speed aircraft, θ is the ballistic inclination angle, ψ V is the trajectory deviation angle, and [x, y, z] is the position coordinate of the high-speed aircraft in the ground coordinate system.

3. The high-speed aircraft cluster collaboration method based on safety reinforcement learning according to claim 2 is characterized in that: In step S200, it specifically includes: S210, design the formation to maintain the observation vector, including, S211. Observe the relative position information. The relative position deviation between the i-th adjacent aircraft and the current aircraft is: Among them, R i is the relative position between the neighboring aircraft i and the current aircraft, is the relative position between the neighboring aircraft i and the current aircraft at the initial moment; The observation vector of the relative position of the aircraft is composed of the relative position deviation vectors ΔR of all neighboring aircraft i Stitched together: Among them, R form is the distance scaling factor, which makes all dimensions in the observation vector at the same order of magnitude; S212. Observe the relative speed information and calculate the average movement speed of the neighboring aircraft except the current aircraft: Among them, V i is the velocity vector of the i-th adjacent aircraft, w i is the weight of the speed of the i-th aircraft, calculated as follows: Among them, k i =‖R i ‖ / R l , R l is the scaling factor; Calculate speed deviation The observation vector form of relative velocity is: Among them, V form is the speed scaling factor; Get the observation vector maintained by the formation: s form =[s pos ,s vel ]。 4. The high-speed aircraft cluster collaboration method based on safety reinforcement learning according to claim 3 is characterized in that: In step S300, it specifically includes: S310, design formation to maintain rewards, Including relative position rewards, by weighting the positions of all aircraft in the cluster, the position of the virtual leader (x c ,y c ,z c ) as a reference; the relative position deviation between the current aircraft and the leading aircraft is: ΔR c =R c -R c0 Among them, R c is the relative position between the current aircraft and the leading aircraft, R c0 is the relative position between the i-th aircraft and the leading aircraft at the initial moment; Add formation deviation from neighboring aircraft: Among them, i min is the ID of the nearest neighboring aircraft to the current aircraft. and are the current time and the initial time respectively, and the current aircraft is relative to the neighboring aircraft i min The relative position information of The relative position reward is: Among them, λ is a negative coefficient; It also includes relative speed bonus. The relative speed bonus between individual aircraft and the cluster is: Among them, μ is a negative coefficient; S320, design target strike reward, The high-speed aircraft cluster needs to hit the target at a specified time. If it hits the target, it will be rewarded. If it misses the target, it will not be rewarded. In order to assist the training convergence, speed rewards, position rewards and distance change rewards are designed. The speed reward function is: Among them, V tar is the vector product of the aircraft velocity vector and the target point relative position vector, ρ is a positive coefficient, V tar is the scaling factor; When the distance between the aircraft and the target point is R tar Get additional position bonus when: Among them, κ is a positive reward coefficient, R tar is the approach distance. When the distance between the aircraft and the target point is less than R tar When , the aircraft is considered to be close to the target point, and according to the artificial potential field method, a reward inversely proportional to the relative distance is applied; R done is the hitting distance. When the distance between the aircraft and the target point is less than R done When the target is hit, a larger reward is obtained; The distance change reward is related to the change in distance between the vehicle and the target point at each simulation step: Among them, ||T|| t and ||T|| t+1 are the distances between the aircraft and the target at the t-th step and the t+1-th step, ζ is a positive coefficient, T tar is the step size scaling factor; S330, Design threat zone avoidance reward, The reward function for threat zone avoidance is designed as follows: Among them, η and σ are negative reward coefficients; The overall reward function is:

5. The high-speed aircraft cluster collaboration method based on safety reinforcement learning according to claim 4 is characterized in that: In step S400, it specifically includes: The SAC algorithm is improved by using the Lyapunov method; for a given safety constraint If there exists a function L(s):S→R + and the positive constants α1, α2, α3, η s ,e, the system is uniformly bounded under the average security cost and the final bound is in, is the average probability that s is in the edge set Δ under strategy π; The Lyapunov method includes: (1) Input hyperparameters, learning rate α θ , UUB theory related parameters e, η s , α3; (2) Randomly initialize a critic network Q(s,a) and a Lyapunov critic network L c (s,a), and a parameter φ Q , The actor network π(a|s) of θ and the Lagrange multipliers β,λ s ,ν s ; (3) Repeat steps (4) to (11) until the total number of rounds reaches a set value; (4) According to ρ s Sampling to obtain the initial state s0; (5) Repeat steps (6) to (8) until the current round ends; (6) Sample action a from the π(s) distribution t ; (7) Observe the next moment s t+1 , r t , c t , and (s t ,a t ,r t ,c t ,s t+1 ) is stored in the experience pool D; where s t+1 represents the state at the next moment, r t represents the current moment reward, c t Represents the current moment price; (8) If s t ∈Δ, (s t ,a t ,r t ,c t ,s t+1 ) Stored in experience pool D Δ middle; (9) When the current time step is the time step that needs to update the network, run steps (10) to (11); (10) From the experience pools D and D Δ Sampling and updating Q, L according to the gradient c ,π and Lagrange multipliers; (11) Update the target network based on the soft substitution.

6. The high-speed aircraft cluster collaboration method based on safety reinforcement learning according to claim 5 is characterized by: The method also includes designing a high-speed aircraft cluster training scenario, training through the training framework obtained in step S400 to obtain a high-speed aircraft cluster flight strategy network; then designing a high-speed aircraft cluster test scenario, and evaluating the training strategy through an overload disturbance test.

7. A high-speed aircraft cluster coordination system based on safety reinforcement learning, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 6 above, and executes the steps in the above-mentioned high-speed aircraft cluster collaboration method based on safety reinforcement learning when running.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the high-speed aircraft cluster collaboration method based on safety reinforcement learning according to any one of claims 1 to 6 when called by a processor.

Images