A Deep Reinforcement Learning-Based Low-Level Flight Control Method for Fighter Jets

Abstract

This invention relates to a fighter jet low-level flight control method based on deep reinforcement learning. The method includes the following steps: S1, designing the target update method and termination condition function of the flight control system according to the simulation environment; S2, constructing the state space, action space, and reward function of the fighter jet control system according to the simulation environment; S3, building a deep neural network model based on the PPO algorithm; S4, training the fighter jet low-level flight control model based on the PPO algorithm. The control method of this invention allows the agent to focus only on high-level mission planning when making high-level mission decisions such as combat and tracking, thereby reducing the training on basic low-level aircraft control decisions, making the algorithm more likely to converge, and reducing learning costs.

Description

Technical Field

[0001] This invention relates to a control method, specifically a fighter jet flight control method based on deep reinforcement learning, belonging to the field of fighter jet flight control technology. Background Technology

[0002] As a complex electromechanical system, fighter jets rely on a flight control system (Fly Control System) to dynamically convert low-level control commands (such as ailerons, elevators, rudder, and throttle) into flight trajectories, ensuring stability and controllability in the dynamic and high-load air combat environment. Because the response of a fighter jet's flight control system to input commands is limited by multiple factors, including flight dynamics, aerodynamic layout, and pilot physiological tolerance, its operation is extremely difficult, requiring extensive training for pilots to master. This has led to a high degree of reliance on manual operation in traditional fighter jet control, posing a severe challenge to human-machine collaboration efficiency in within-visual-range (WVR) or beyond-visual-range (BVR) air combat missions.

[0003] With the rapid development of artificial intelligence (AI) and deep learning technologies, the application potential of intelligent agents in air combat has been widely studied, especially in unmanned aerial vehicle (UAV) and manned-unmanned cooperative combat scenarios. The application of AI intelligent agents can significantly reduce the burden on pilots and improve combat efficiency. In practical applications, intelligent agents can take over the low-level flight control of fighter jets, allowing pilots to focus on higher-level tactical decisions; or perform multi-aircraft cooperative missions in swarm flights, such as reconnaissance, escort, and decoy missions. These potential applications have greatly promoted the research progress of intelligent agents in air combat. However, the high-precision simulations in air combat simulation environments (including the flight dynamics characteristics of fighter jets and flight control system response models) consume a large amount of computing resources, making it difficult to rapidly iterate the deep reinforcement learning training of intelligent agents, thus limiting the algorithm's convergence speed.

[0004] Current training methods for air combat agents typically divide the agent's tasks into two modules: high-level decision-making and low-level control, to simplify the training process and improve learning efficiency. In this framework, the high-level module focuses on air combat strategies and tactical choices, while the low-level module is responsible for basic flight control. However, this modular approach faces numerous technical challenges in implementation. In particular, due to the highly dynamic flight characteristics of fighter jets (such as high speed, instantaneous acceleration, and maneuvering), agents are prone to problems like multiple stalls and jitter in low-level flight control, severely impacting flight stability and making it difficult for the high-level decision-making module to focus on advanced tasks. Furthermore, traditional deep reinforcement learning algorithms struggle to achieve rapid convergence when faced with large-scale distributed interactive sampling requirements, often resulting in performance that fails to meet the demands of air combat applications.

[0005] To address this issue, existing research typically optimizes flight control through two technical measures: first, optimizing the agent's reward mechanism to provide rapid and effective feedback after completing specific actions, thereby accelerating the learning process; and second, dividing the agent's learning process into multiple stages, first completing basic actions such as takeoff and cruise, and then gradually learning more complex air combat maneuvers. However, this staged learning approach still faces convergence problems when encountering dynamic and challenging flight missions (such as high-maneuverability maneuvers of fighter jets under high-speed and high-G conditions), resulting in poor control stability of the agent in complex environments. Therefore, a new solution is urgently needed to address this technical problem. Summary of the Invention

[0006] This invention addresses the technical problems existing in the prior art by providing a fighter jet low-level flight control method based on deep reinforcement learning. This solution addresses the issues of multiple stalls and jitters that easily occur in air combat agents in highly dynamic environments. The method uses a deep reinforcement learning model with speed and angle as targets to perform refined control of the fighter jet's low-level control system, thereby reducing stall and jitter problems caused by complex environmental changes in low-level control. By optimizing the low-level flight control module, the control method of this invention allows the agent to focus solely on high-level mission planning when making high-level mission decisions such as combat and tracking, thus reducing the training on basic low-level aircraft control decisions, making the algorithm more convergent, and reducing learning costs.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows: a fighter jet low-level flight control method based on deep reinforcement learning, the method comprising the following steps:

[0008] S1. Design the target update method and termination condition function of the flight control system based on the simulation environment;

[0009] S2. Construct the state space, action space, and reward function for fighter jet control based on the simulation environment;

[0010] S3. Build a deep neural network model based on the PPO algorithm;

[0011] S4. Train the fighter jet's underlying flight control model based on the PPO algorithm.

[0012] In step S1, the target update method and termination condition function of the flight control are designed according to the simulation environment. The specific steps are as follows:

[0013] S11. Define the origin of the aircraft's coordinate system as the center of mass of the aircraft, and construct the kinematic and dynamic equations of the fighter jet's trajectory:

[0014]

[0015] Where X, Y, and Z represent the aircraft's position in the ground coordinate system, u, v, and w represent the components of the aircraft's airspeed in the body coordinate system, and θ, φ represents the pitch angle, yaw angle, and roll angle of the aircraft.

[0016]

[0017] Where p, q, and r are the roll rate, pitch rate, and yaw rate of the aircraft, respectively, and F x F y F z Let g be the component of the resultant force in the aircraft's coordinate system, g be the acceleration due to gravity, and m be the mass of the aircraft.

[0018] S12. Design a target update method for flight control training.

[0019] This method is used to initialize the target-related state parameters and set a new flight target state when resetting the flight target in the flight control system, so as to ensure that the new target setting is not affected by the previous state. The specific steps are as follows:

[0020] (1) Reset the state flag variables related to the flight target and randomly generate a three-dimensional deviation vector for velocity, pitch angle and yaw angle, with each component ranging from [-1, 1].

[0021] (2) Receive the current state information of the fighter jet, store it in the current state vector, decode the generated deviation vector, and convert it into the actual target change range.

[0022] (3) Add the decoded deviation value to the current state value to generate a new target state, and impose upper and lower limits on the target value to ensure that the set target value is within a controllable range.

[0023] (4) Calculate the difference between the newly set target state and the current state to obtain the deviation state, which will eventually be part of the state space.

[0024] S13. Design the termination condition function for flight control training.

[0025] By setting the termination conditions of this invention, the system can effectively monitor the achieved state during flight control training and provide timely reward or penalty signals, as follows:

[0026] (1) Achievement condition check: Based on the difference between the current state and the target, determine whether the set speed and angle targets have been achieved. Compare the difference between the current state and the target state with a threshold. If the difference is less than the target threshold, mark the target as achieved. If all speed and angle targets are achieved, mark the unit as "all achieved".

[0027] (2) Achievement Marking and Recording: Based on the achievement status of each unit, update the internal marker variables and counters to record the number of steps achieved each time, the number of times the goal is achieved, and the number of consecutive steps achieved. If the goal has been achieved consecutively, update the counter; if the goal has not been achieved consecutively, record the number of times the goal was not achieved.

[0028] (3) Termination condition judgment: If the objective is achieved 5 times consecutively, the task is marked as completed and the reward is triggered.

[0029] If the current flight control steps reach the preset step limit, a "timeout" state is triggered; if the current unit does not meet the target state (such as too low speed, too low or too high flight altitude, etc.), the mission is judged to have failed. Regardless of whether it is timeout, failure or success, the termination flag is always "not terminated".

[0030] In step S2, the state space, action space, and reward function of the aircraft control are constructed based on the simulation environment. The specific steps are as follows:

[0031] S21. Define the fighter jet flight control state space.

[0032] When training flight control in a simulation environment, the flight status of a fighter jet can be described by its own attitude, its situational relationship with the target, and its velocity relationship, as shown below:

[0033]

[0034] Among them, V x,y,z The three-dimensional velocity of the fighter jet in the northeast coordinate system; For the fighter jet's pitch, yaw, and roll angles; ω p,q,r Here, H represents the fighter jet's roll rate, pitch rate, and yaw rate; H represents the fighter jet's altitude. The difference between the target velocity and the current velocity scalar; The difference between the pitch angle and yaw angle at the target and the current speed.

[0035] S22, Define the fighter jet flight control action space:

[0036] A = [W] aileron W elevator W rudder ,δ t ]

[0037] Among them, W aileron W is used to control the aircraft's roll on the fighter jet's control stick. elevator For fighter jets, the control stick controls the aircraft's pitch. rudder For fighter jets, the control stick is used to control the aircraft's yaw, δ T This refers to the displacement of the fighter jet's throttle lever.

[0038] S23. Define the fighter jet flight control reward function:

[0039] The total reward function is composed of a weighted average of the height reward function, roll reward function, angle reward function, speed reward function, time consumption reward function, and safety reward function: r = ∑ω j r j , where r j For all the above reward items, ω j The weighting coefficient for the response reward item.

[0040] The reward functions for height, safety, and time consumption are all event-based rewards, and the main reward functions are shown below:

[0041] The speed reward function measures the degree of match between the current flight speed and the target speed, and the specific formula is as follows:

[0042] R v =exp(-w v Δv)

[0043] Where Δv is the weighted sum of velocity differences, comprising three parts: the difference between the current velocity scalar and the target velocity scalar; the difference between the current velocity tilt angle and the target velocity tilt angle; and the difference between the current velocity yaw angle and the target velocity yaw angle. v This is a speed reward weighting coefficient used to control the severity of penalties.

[0044] This reward function comprehensively evaluates the degree of matching between the fighter jet's current speed and the target speed, including differences in speed magnitude, bank angle, and yaw angle. Compared to existing technologies that only consider speed as a single indicator, this method significantly improves the accuracy of speed control, enabling the agent to more efficiently approach the target flight state under complex flight conditions. This reward function significantly enhances the fighter jet's ability to precisely control its speed, bringing it closer to the target flight trajectory.

[0045] The core objective of angle reward is to detect abnormal flight attitude and assign penalties or rewards based on the degree of abnormality. The specific formula is shown below:

[0046]

[0047] Among them, w θ The angle reward weighting coefficient is defined as follows: When the absolute value of the roll angle θ is greater than 90 degrees, it indicates an abnormal aircraft attitude, requiring a penalty; otherwise, it is considered normal, and the reward is 0. The angle reward system detects whether the flight attitude exceeds a safety threshold, enabling real-time identification and intervention of potentially dangerous states. This penalty mechanism further enhances flight safety, comprehensively preventing dangerous flight states compared to existing technologies and ensuring the stability of the control model in highly dynamic environments.

[0048] The roll bonus is designed to suppress excessive changes in roll angular velocity, thereby reducing flight instability caused by excessive maneuvers. Its formula is shown below:

[0049]

[0050] Among them, w roll This is the roll reward weighting coefficient. If the roll angular velocity σ roll Compared with the previous moment's roll angular velocity σ' roll A discrepancy between the positive and negative signs indicates abnormal shaking in the aircraft's attitude. In this case, a penalty is imposed by imposing a drastic change in the roll rate, which can effectively prevent the shaking from interfering with the flight attitude.

[0051] Real-time detection of attitude jitter is incorporated into the roll reward mechanism. By determining the trend of roll angular velocity changes, abnormal jitter is penalized promptly. This design effectively reduces attitude jitter during flight, significantly improving control stability and execution accuracy. Compared to traditional methods that ignore the details of roll dynamics, this invention focuses more on the dynamic change process, enhancing the model's sensitivity to and ability to correct attitude anomalies.

[0052] This invention upgrades the reward function design from sparse rewards to multidimensional, real-time rewards, significantly improving the model's training efficiency and convergence speed. Through the penalty design of roll and angle reward functions, the model can detect and correct flight attitude anomalies in real time, thereby enhancing control stability and safety. Furthermore, by comprehensively considering speed, attitude, and safety, the technical solution demonstrates significantly better robustness and adaptability in complex air combat environments than traditional methods, fully reflecting its innovation and practical value.

[0053] In step S3, the policy network of the PPO algorithm is used to select actions, and the policy is optimized through reward feedback on action-state pairs. The steps are as follows:

[0054] The PPO algorithm model consists of an experience pool and two neural networks, including a policy Actor network and a value Critic network, with parameters θ and φ, respectively.

[0055] S31. The experience pool is used to store experience samples generated by the interaction between the UAV and the environment during the training process. These experience samples are stored in the buffer for subsequent training, which can improve the independence of training, sample efficiency, generalization performance and stability.

[0056] S32, The input to the policy Actor network is state S t The output is the action probability distribution π(a) t |s tThe optimization objective is that the more the action probability distribution output by the policy Actor network increases the value of the policy objective, the better the policy performance. The PPO algorithm introduces a pruning mechanism to limit the update amplitude to ensure training stability. The objective function of policy optimization using the pruning loss function is defined as follows:

[0057]

[0058] in The strategy probability ratio measures the proportion of change between the old and new strategies. ε is the advantage function, used to evaluate the relative merit of the current action; ε is the pruning hyperparameter, controlling the magnitude of policy updates to avoid drastic policy changes.

[0059] S33. The input to the Critic network is the state, and the output is the estimated state value V(s). The Critic network is optimized by minimizing the value error, enabling the model to more accurately evaluate the value of the current state. The mean squared error loss function is used to define the loss function of the Critic network.

[0060]

[0061] Among them, V φ (s t ) is the Critic network for state S t The estimated value of R; t This is the actual return value.

[0062] S34. The dominance function is calculated using the generalized dominance estimation (GAE) method. To evaluate the merits of each action relative to the current strategy, the GAE expression is:

[0063]

[0064] Where, δ t =r t +γV(s t+1 )-V(s t ) represents the TD error, γ is the discount factor, and λ is the attenuation factor.

[0065] The GAE method, by introducing discount and decay factors, balances the relationship between long-term rewards and immediate feedback, effectively reducing the volatility and instability of advantage estimation in traditional methods. This smooth estimation mechanism makes the model more robust during training and enables it to adapt more quickly to dynamic changes in complex environments.

[0066] By flexibly adjusting the time difference error, the importance of short-term and long-term rewards can be dynamically weighed in different scenarios. This feature is particularly suitable for high-dynamic scenarios in fighter jet flight control, such as high-maneuverability turns or complex flight missions, and significantly improves the model's adaptability and performance in multi-mission switching compared to traditional advantage estimation methods.

[0067] Furthermore, GAE can effectively alleviate the problem of insufficient sample utilization. Existing technologies typically rely heavily on sample size, while GAE improves sample utilization efficiency through more efficient advantage evaluation, thereby reducing the need for large amounts of computational resources during training.

[0068] In step S4, the fighter jet's underlying flight control model based on the PPO algorithm is trained, and the steps are as follows:

[0069] The specific steps for training the network are as follows:

[0070] S41. Set network training parameters: Randomly initialize the parameters of the policy Actor network and the value Critic network, and initialize the experience cache pool to store experience samples generated during training.

[0071] S42. Initialize the fighter jet status, initialize the target-related status parameters, and set the new flight target status.

[0072] S43. Input the current state into the policy Actor network to generate the corresponding action; execute the action in the current state to obtain the new state and reward, record the relevant training data, and store it in the experience cache pool.

[0073] S44. Check the amount of data in the experience buffer: When the amount of experience data exceeds the set mini-batch size, begin training. Randomly sample a batch of samples from the experience buffer to update the policy Actor network and Critic network.

[0074] S45. Optimize the parameters of the Actor network using the pruning loss function defined by PPO; optimize the parameters of the Critic network using the mean squared error loss function; use the Adam optimizer for gradient updates.

[0075] S46. Policy Update: Check the policy update conditions. If the conditions are met, perform a policy pruning update; otherwise, keep the current policy unchanged.

[0076] S47. Check if the round end condition has been met: If it has been met, proceed to step (47); otherwise, return to step (43) to continue a new round of state-action interaction.

[0077] S48. Training termination condition: Determine whether the algorithm model has reached the preset maximum training rounds. If it has, end the training and save the main strategy model after the training is completed. If it has not, return to step (42) to continue training.

[0078] First, in the training initialization phase (S41, S42), this invention significantly improves the diversity and utilization efficiency of training samples by randomly initializing network parameters and designing an efficient experience cache pool. Compared to existing technologies that rely on single initialization, this mechanism effectively enhances the model's generalization ability, enabling it to adapt to more complex flight environments.

[0079] Secondly, in the state and action interaction phases (S43, S44), this invention enhances the use of the experience cache pool, achieving efficient sample utilization by dynamically sampling small batches of data and optimizing the network training process. Simultaneously, the adoption of a step-by-step optimized policy network and value network (Actor and Critic networks) ensures that the model can be rapidly updated in each iteration, significantly improving the convergence speed.

[0080] Furthermore, by combining the PPO pruning loss function (S45) with the Adam optimizer, the stability and efficiency of policy optimization are improved. Compared to the problems of policy updates being too fast or too slow that are prone to occur in traditional methods, this invention avoids drastic fluctuations in the policy network by introducing a pruning mechanism, ensuring a more robust training process.

[0081] In the policy update and round checking stages (S46, S47), by flexibly setting update conditions and round end conditions, this invention can dynamically adjust the training process, avoiding resource waste due to overtraining or performance degradation due to insufficient training. Simultaneously, the added multi-round loop mechanism provides a guarantee for long-term model optimization, enabling it to continuously improve performance in complex task scenarios.

[0082] Finally, in setting the termination condition (S48), this invention employs explicit training round limits and a model saving mechanism. This design avoids invalid training while ensuring that the optimal policy model can be output at the end of training. Compared with the overly simple or unoptimized termination conditions of existing technologies, it exhibits higher efficiency and reliability.

[0083] In summary, this invention significantly improves the convergence speed, adaptability, and robustness of the PPO algorithm in fighter jet low-level flight control models by refining the training steps, enabling it to exhibit superior control capabilities in highly dynamic and complex environments.

[0084] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the deep reinforcement learning-based fighter jet low-level flight control method as described above.

[0085] A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned deep reinforcement learning-based fighter jet low-level flight control method.

[0086] The proposed deep reinforcement learning-based low-level flight control method for fighter jets significantly improves the flight control stability and convergence speed of the agent in high-dynamic environments. Through refined control of the fighter jet's low-level flight control system, especially in high-dynamic and high-load air combat environments, this method exhibits numerous advantages:

[0087] (1) Improve flight control stability,

[0088] In traditional methods, intelligent agents often experience stalls and vibrations due to environmental complexity when performing flight missions under high-speed and high-G conditions. This invention addresses these issues by using speed and angle as control targets, combined with an improved reward mechanism, enabling the intelligent agent to more precisely control the fighter jet's attitude and speed. This method effectively reduces multiple stalls and vibrations caused by sudden attitude changes, significantly improving flight control stability and ensuring the agent maintains smooth operation under high-speed turns and dynamic changes.

[0089] (2) Optimize the training efficiency of the agent.

[0090] Traditional deep reinforcement learning methods are ill-suited for large-scale distributed interactive environments due to their slow convergence speed, making them unsuitable for air combat applications. This invention employs the PPO algorithm, optimizing the training process of the policy network through an experience pool mechanism and a pruned loss function. This improves sample utilization during training while ensuring training independence and stability. Simultaneously, the Generalized Advantage Estimation (GAE) method accelerates the calculation of the advantage function, enabling the agent to obtain training feedback more quickly, ultimately achieving more efficient learning convergence and reducing training costs.

[0091] (3) Improve the fault tolerance of the control system.

[0092] This invention enhances the robustness of the control system by setting target update methods and termination condition functions. For example, the system can monitor the difference between the fighter jet's flight state and the target state in real time and promptly trigger corresponding reward or penalty signals to prompt the agent to correct deviations as quickly as possible. This function effectively improves the agent's fault tolerance in complex and dynamic environments, enabling it to adapt to changing air combat environments and ensuring safety during the control process.

[0093] (4) Modular design improves system adaptability.

[0094] The underlying control method of this invention can operate independently of the higher-level tactical decision-making module. The two are designed separately to achieve flight control and tactical selection. This ensures that even with changing tactical requirements, the underlying flight control module maintains its operational stability and accuracy. Furthermore, this method can be reused in different fighter jet simulation environments, demonstrating good adaptability and versatility. Attached Figure Description

[0095] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0096] Figure 2 This is a schematic diagram illustrating the specific process of training the network. Detailed Implementation

[0097] To enhance understanding of the present invention, the embodiments will be described in detail below with reference to the accompanying drawings.

[0098] Example 1: See Figure 1 , Figure 2 A method for low-level flight control of a fighter jet based on deep reinforcement learning, the method comprising the following steps:

[0099] S1. Design the target update method and termination condition function of the flight control system based on the simulation environment;

[0100] S2. Construct the state space, action space, and reward function for fighter jet control based on the simulation environment;

[0101] S3. Build a deep neural network model based on the PPO algorithm;

[0102] S4. Train the fighter jet's underlying flight control model based on the PPO algorithm.

[0103] In step S1, the target update method and termination condition function of the flight control are designed according to the simulation environment. The specific steps are as follows:

[0104] S11. Define the origin of the aircraft's coordinate system as the center of mass of the aircraft, and construct the kinematic and dynamic equations of the fighter jet's trajectory:

[0105]

[0106] Where X, Y, and Z represent the aircraft's position in the ground coordinate system, u, v, and w represent the components of the aircraft's airspeed in the body coordinate system, and θ, φ represents the pitch angle, yaw angle, and roll angle of the aircraft.

[0107]

[0108] Where p, q, and r are the roll rate, pitch rate, and yaw rate of the aircraft, respectively, and F x F y F z Let g be the component of the resultant force in the aircraft's coordinate system, g be the acceleration due to gravity, and m be the mass of the aircraft.

[0109] S12. Design a target update method for flight control training.

[0110] This method is used to initialize the target-related state parameters and set a new flight target state when resetting the flight target in the flight control system, so as to ensure that the new target setting is not affected by the previous state. The specific steps are as follows:

[0111] (1) Reset the state flag variables related to the flight target and randomly generate a three-dimensional deviation vector for velocity, pitch angle and yaw angle, with each component ranging from [-1, 1].

[0112] (2) Receive the current state information of the fighter jet, store it in the current state vector, decode the generated deviation vector, and convert it into the actual target change range.

[0113] (3) Add the decoded deviation value to the current state value to generate a new target state, and impose upper and lower limits on the target value to ensure that the set target value is within a controllable range.

[0114] (4) Calculate the difference between the newly set target state and the current state to obtain the deviation state, which will eventually be part of the state space.

[0115] S13. Design the termination condition function for flight control training.

[0116] By setting the termination conditions of this invention, the system can effectively monitor the achieved state during flight control training and provide timely reward or penalty signals, as follows:

[0117] (1) Achievement condition check: Based on the difference between the current state and the target, determine whether the set speed and angle targets have been achieved. Compare the difference between the current state and the target state with a threshold. If the difference is less than the target threshold, mark the target as achieved. If all speed and angle targets are achieved, mark the unit as "all achieved".

[0118] (2) Achievement Marking and Recording: Based on the achievement status of each unit, update the internal marker variables and counters to record the number of steps achieved each time, the number of times the goal is achieved, and the number of consecutive steps achieved. If the goal has been achieved consecutively, update the counter; if the goal has not been achieved consecutively, record the number of times the goal was not achieved.

[0119] (3) Termination condition judgment: If the objective is achieved 5 times consecutively, the task is marked as completed and the reward is triggered.

[0120] If the current flight control steps reach the preset step limit, a "timeout" state is triggered; if the current unit does not meet the target state (such as too low speed, too low or too high flight altitude, etc.), the mission is judged to have failed. Regardless of whether it is timeout, failure or success, the termination flag is always "not terminated".

[0121] In step S2, the state space, action space, and reward function of the aircraft control are constructed based on the simulation environment. The specific steps are as follows:

[0122] S21. Define the fighter jet flight control state space.

[0123] When training flight control in a simulation environment, the flight status of a fighter jet can be described by its own attitude, its situational relationship with the target, and its velocity relationship, as shown below:

[0124]

[0125] Among them, V x,y,z The three-dimensional velocity of the fighter jet in the northeast coordinate system; For the fighter jet's pitch, yaw, and roll angles; ω p,q,r Here, H represents the fighter jet's roll rate, pitch rate, and yaw rate; H represents the fighter jet's altitude. The difference between the target velocity and the current velocity scalar; The difference between the pitch angle and yaw angle at the target and the current speed.

[0126] S22, Define the fighter jet flight control action space:

[0127] A = [W] aileron W elevator W rudder ,δ t ]

[0128] Among them, W aileron W is used to control the aircraft's roll on the fighter jet's control stick. elevator For fighter jets, the control stick controls the aircraft's pitch. rudder For fighter jets, the control stick is used to control the aircraft's yaw, δ T This refers to the displacement of the fighter jet's throttle lever.

[0129] S23. Define the fighter jet flight control reward function:

[0130] The total reward function is composed of a weighted average of the height reward function, roll reward function, angle reward function, speed reward function, time consumption reward function, and safety reward function: r = ∑ω j rj , where r j For all the above reward items, ω j The weighting coefficient for the response reward item.

[0131] In step S3, the policy network of the PPO algorithm is used to select actions, and the policy is optimized through reward feedback on action-state pairs. The steps are as follows:

[0132] The PPO algorithm model consists of an experience pool and two neural networks, including a policy Actor network and a value Critic network, with parameters θ and φ, respectively.

[0133] S31. The experience pool is used to store experience samples generated by the interaction between the UAV and the environment during the training process. These experience samples are stored in the buffer for subsequent training, which can improve the independence of training, sample efficiency, generalization performance and stability.

[0134] S32, The input to the policy Actor network is state S t The output is the action probability distribution π(a) t |s t The optimization objective is that the more the action probability distribution output by the policy Actor network increases the value of the policy objective, the better the policy performance. The PPO algorithm introduces a pruning mechanism to limit the update amplitude to ensure training stability. The objective function of policy optimization using the pruning loss function is defined as follows:

[0135]

[0136] in The strategy probability ratio measures the proportion of change between the old and new strategies. ε is the advantage function, used to evaluate the relative merit of the current action; ε is the pruning hyperparameter, controlling the magnitude of policy updates to avoid drastic policy changes.

[0137] S33. The input to the Critic network is the state, and the output is the estimated state value V(s). The Critic network is optimized by minimizing the value error, enabling the model to more accurately evaluate the value of the current state. The mean squared error loss function is used to define the loss function of the Critic network.

[0138]

[0139] Among them, V φ (s t ) is the Critic network for state S t The estimated value of R; t This is the actual return value.

[0140] S34. The dominance function is calculated using the generalized dominance estimation (GAE) method. To evaluate the merits of each action relative to the current strategy, the GAE expression is:

[0141]

[0142] Where, δ t =r t +γV(s t+1 )-V(s t ) represents the TD error, γ is the discount factor, and λ is the attenuation factor.

[0143] In step S4, the fighter jet's underlying flight control model based on the PPO algorithm is trained, and the steps are as follows:

[0144] See Figure 2 The specific steps for training the network are as follows:

[0145] S41. Set network training parameters: Randomly initialize the parameters of the policy Actor network and the value Critic network, and initialize the experience cache pool to store experience samples generated during training.

[0146] S42. Initialize the fighter jet status, initialize the target-related status parameters, and set the new flight target status.

[0147] S43. Input the current state into the policy Actor network to generate the corresponding action; execute the action in the current state to obtain the new state and reward, record the relevant training data, and store it in the experience cache pool.

[0148] S44. Check the amount of data in the experience buffer: When the amount of experience data exceeds the set mini-batch size, begin training. Randomly sample a batch of samples from the experience buffer to update the policy Actor network and Critic network.

[0149] S45. Optimize the parameters of the Actor network using the pruning loss function defined by PPO; optimize the parameters of the Critic network using the mean squared error loss function; use the Adam optimizer for gradient updates.

[0150] S46. Policy Update: Check the policy update conditions. If the conditions are met, perform a policy pruning update; otherwise, keep the current policy unchanged.

[0151] S47. Check if the round end condition has been met: If it has been met, proceed to step (47); otherwise, return to step (43) to continue a new round of state-action interaction.

[0152] S48. Training termination condition: Determine whether the algorithm model has reached the preset maximum training rounds. If it has, end the training and save the main strategy model after the training is completed. If it has not, return to step (42) to continue training.

[0153] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A fighter jet low-level flight control method based on deep reinforcement learning, characterized in that, The method includes the following steps: S1. Design the target update method and termination condition function of the flight control system based on the simulation environment; S2. Construct the state space, action space, and reward function for fighter jet control based on the simulation environment; S3. Build a deep neural network model based on the PPO algorithm; S4. Train the fighter jet's underlying flight control model based on the PPO algorithm; In step S4, the fighter jet's underlying flight control model based on the PPO algorithm is trained, and the steps are as follows: The specific steps for training the network are as follows: S41. Set network training parameters: Randomly initialize the parameters of the policy Actor network and the value Critic network, initialize the experience cache pool to store experience samples generated during training, and set the optimizer's learning rate, adaptive gradient pruning threshold, and gradient penalty coefficient according to the task scenario to improve training stability. S42. Initialize the fighter jet state, initialize the target-related state parameters, and set a new flight target state. Simultaneously, design a dynamic target adjustment strategy to enable the target state to be dynamically adjusted according to environmental complexity during training, thereby improving the model's generalization ability. S43. Input the current state into the policy Actor network to generate the corresponding action; execute the action in the current state to obtain the new state and reward, record the relevant training data, and store it in the experience cache pool. At the same time, add state transition constraint rules to ensure the physical rationality of the state transition. S44. Check the amount of data in the experience buffer: When the amount of experience data exceeds the set mini-batch size, training begins. A batch of samples is randomly sampled from the experience buffer to update the policy Actor network and Critic network. At the same time, a priority experience replay mechanism is adopted, allocating higher sampling probabilities to samples with higher rewards or those related to state scarcity. S45. Optimize the parameters of the Actor network using the pruning loss function defined by PPO; optimize the parameters of the Critic network using the mean squared error loss function; use the Adam optimizer for gradient updates and add a gradient entropy regularization term to prevent the model from getting trapped in local optima. S46. Policy Update: Check the policy update conditions. If the conditions are met, perform a policy pruning update; otherwise, keep the current policy unchanged. Simultaneously, introduce policy diversity constraints to avoid the problem of monotonous behavior patterns during policy updates. S47. Check if the round end condition has been met: If it has, proceed to the round end processing; otherwise, return to step S43 to continue a new round of state-action interaction. In this step, add a flight behavior diversity reward to encourage the fighter jet to explore more effective strategies. S48. Training Termination Condition: Determine whether the algorithm model has reached the preset maximum number of training rounds. If it has, end the training and save the main strategy model after the training is completed; if it has not, return to step S42 to continue training.

2. The fighter jet low-level flight control method based on deep reinforcement learning according to claim 1, characterized in that, In step S1, the target update method and termination condition function of the flight control are designed according to the simulation environment. The specific steps are as follows: S11. Define the origin of the aircraft coordinate system as the center of mass of the fighter jet, and construct the kinematic and dynamic equations of the fighter jet's trajectory motion: in, , , This represents the fighter jet's position in the ground coordinate system. , , Let V be the velocity component of the fighter jet's airspeed in the airframe coordinate system. , , For the pitch angle, yaw angle, and roll angle of a fighter jet, in, , , This refers to the acceleration components of the fighter jet in the body coordinate system. , , For the fighter jet's roll rate, pitch rate, and yaw rate, , , The components of the resultant force in the body coordinate system. It is the acceleration due to gravity. For fighter jet quality, S12. Design a target update method for flight control training. The specific steps are as follows: (1) Reset the state flag variables related to the flight target and randomly generate a three-dimensional deviation vector to generate the deviations of the target speed, target pitch angle and target yaw angle. (2) Receive the current state information of the fighter jet, store it in the current state vector, decode the generated deviation vector, and convert it into the actual target change range. (3) Add the decoded deviation value to the current state value to generate a new target state, and impose upper and lower limits on the target value to ensure that the set target value is within a controllable range. (4) Calculate the difference between the newly set target state and the current state to obtain the deviation state, which will eventually be part of the state space. S13. Design the termination condition function for flight control training. Specifically as follows: (1) Achievement condition check: Based on the difference between the current state and the target, determine whether the set speed and angle targets have been achieved. Compare the difference between the current state and the target state with a threshold. If the difference is less than the target threshold, mark the target as achieved. If all speed and angle targets are achieved, mark the unit as "all achieved". (2) Achievement Marking and Recording: Based on the achievement status of each unit, update the internal marker variables and counters to record the number of steps achieved each time, the number of times the goal is achieved, and the number of consecutive steps achieved. If the goal has been achieved consecutively, update the counter; if the goal has not been achieved consecutively, record the number of times the goal was not achieved. (3) Termination condition judgment: If the objective is achieved 5 times consecutively, the task is marked as completed and the reward is triggered. If the current flight control steps reach the preset step limit, a "timeout" state is triggered; if the current unit does not meet the target state, including too low speed, too low or too high flight altitude, the mission is judged to have failed.

3. The fighter jet low-level flight control method based on deep reinforcement learning according to claim 2, characterized in that, In step S2, the state space, action space, and reward function of the fighter jet control are constructed based on the simulation environment. The specific steps are as follows: S21. Define the fighter jet flight control state space. When training flight control in a simulation environment, the flight status of a fighter jet can be described by its own attitude, its situational relationship with the target, and its velocity relationship, as shown below: in, The three-dimensional velocity of the fighter jet in the northeast coordinate system; These refer to the pitch angle, yaw angle, and roll angle of a fighter jet. For the fighter jet's roll rate, pitch rate, and yaw rate; This refers to the altitude of the fighter jet. The difference between the target velocity and the current velocity scalar; The difference between the pitch and yaw angles in the target velocity direction and the pitch and yaw angles in the current velocity direction; S22, Define the fighter jet flight control action space: in, The control stick is used to control the aircraft's roll. The control stick is used to control the pitch of a fighter jet. The control stick is used to control the yaw of a fighter jet. This refers to the displacement of the fighter jet's throttle lever. S23. Define the fighter jet flight control reward function: The total reward function is composed of a weighted average of the height reward function, roll reward function, angle reward function, speed reward function, time consumption reward function, and safety reward function. ,in, For all of the above rewards, The weighting coefficient for the response reward item.

4. The fighter jet low-level flight control method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the policy network of the PPO algorithm is used to select actions, and the policy is optimized through reward feedback on action-state pairs. The steps are as follows: The PPO algorithm model consists of an experience pool and two neural networks, including a policy Actor network and a value Critic network. S31. The experience pool is used to store experience samples generated by the interaction between the fighter jet and the environment during training. These experience samples are stored in a buffer for subsequent training. S32, The input to the policy Actor network is the state. The output is the action probability distribution. The optimization objective is that the more the action probability distribution output by the policy Actor network increases the value of the policy objective, the better the policy performance. The PPO algorithm introduces a pruning mechanism to limit the update amplitude to ensure training stability. The objective function of policy optimization using the pruning loss function is defined as follows: in The strategy probability ratio measures the proportion of change between the old and new strategies. This is the advantage function, used to evaluate the relative merits of the current action; To prune hyperparameters and control the policy update magnitude to avoid drastic policy changes, S33. The input to the value Critic network is state S. t The output is the estimated state value V(s) t The Critic network optimizes by minimizing the value error, enabling the model to more accurately assess the value of the current state. The mean squared error loss function is used to define the loss function of the Critic network. in, It is a Critic network for state The estimated value; It is the actual return value. S34. The dominance function is calculated using the Generalized Dominance Estimation (GAE) method. To evaluate the merits of each action relative to the current strategy, the GAE expression is: in, Indicates TD error, As a discount factor, This is the attenuation factor.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the fighter jet low-level flight control method based on deep reinforcement learning as described in claim 1.

6. A computer-readable storage medium storing computer instructions thereon, characterized in that: When the computer instructions are executed by the processor, they implement the fighter jet low-level flight control method based on deep reinforcement learning as described in claim 1.

Images