Aircraft online route planning method, device, equipment and storage medium

Abstract

The application discloses an aircraft online route planning method and device, equipment and a storage medium, the method obtains flight data of the aircraft, and a Markov model corresponding to online route planning of the aircraft is constructed according to the flight data; a deep deterministic policy gradient (DDPG) neural network of the aircraft is constructed, and a curriculum learning (CL) model of the aircraft is constructed according to the DDPG neural network; when a curriculum learning task is received, the online route planning model of the aircraft in the curriculum learning task is trained according to the Markov model and the CL model, and a trained final route planning is obtained, the exploration ability of the network to unknown environments and strategies can be improved, the model training rate is improved, the target region is avoided, the online route planning task is effectively completed, the real task scene demand can be met, the aircraft online route planning speed and efficiency are improved, and good general performance is achieved.

Claims

1. An online route planning method for aircraft, characterized in that, The online flight path planning method for the aircraft includes: Acquire flight data of the aircraft, and construct a Markov model corresponding to the online route planning of the aircraft based on the flight data; Construct the Depth Deterministic Policy Gradient Algorithm (DDPG) neural network for the aircraft, and construct the course learning CL model for the aircraft based on the DDPG neural network. Upon receiving a course learning task, the online route planning model of the aircraft is trained in the course learning task based on the Markov model and the course learning CL model to obtain the trained final route plan. The construction of the Depth Deterministic Policy Gradient Algorithm (DDPG) neural network for the aircraft, and the construction of the course learning (CL) model for the aircraft based on the DDPG neural network, include: The state behavior values ​​of the aircraft at different times and under different aircraft states in the evaluation network are obtained by the following formula: in, For state behavior values, The state of the aircraft at time t in the evaluation network. This is the action output value of the aircraft at time t in the evaluation network. To evaluate the network; Based on the state behavior values, the action network and Critic evaluation network of the Deep Deterministic Policy Gradient Algorithm (DDPG) neural network are copied to generate an action target network and an evaluation target network. The DDPG neural network of the aircraft is then constructed based on the action target network and the evaluation target network. A course learning CL model for the aircraft is constructed based on the DDPG neural network. Optionally, constructing the course learning CL model for the aircraft based on the DDPG neural network includes: Obtain the initial action output value and the action value after Gaussian noise processing of the DDPG neural network, and update the action output value of the corresponding path planning subtask of the aircraft according to the initial action output value and the action value using the following formula: in, This is the action output value of the aircraft at time t in the evaluation network. Output initial values ​​for the action. For the Gaussian noise that corresponds to the target of path planning subtask 1, For the Gaussian noise corresponding to threat avoidance in the path planning subtask 2, The Gaussian noise corresponding to the route optimization in the path planning subtask 3 is... The action value is the result of processing with Gaussian noise. Construct a course learning CL model for the aircraft based on the action output values ​​and the preset training course; Optionally, upon receiving a course learning task, the online route planning model of the aircraft is trained using the Markov model and the course learning CL model in the course learning task to obtain a trained final route plan, including: Upon receiving a course learning task, the training scenario, number of training rounds, experience replay queue capacity, sampling size, maximum step size per round, action network learning rate, evaluation network learning rate, and soft update coefficient are determined based on the Markov model and the course learning CL model. The aircraft's action network and evaluation network are updated based on the training scenario, the number of training rounds, the capacity of the experience replay queue, the sampling size, the maximum step size per round, the learning rate of the action network, the learning rate of the evaluation network, and the soft update coefficient. The online route planning model of the aircraft is then trained to obtain the final route plan.

2. The online flight path planning method for aircraft as described in claim 1, characterized in that, The process of acquiring flight data of the aircraft and constructing a Markov model corresponding to the online route planning of the aircraft based on the flight data includes: Acquire flight data of the aircraft, and obtain the aircraft state space, aircraft action space, preset reward function, and preset reward function discount factor based on the flight data; Based on the aircraft state space, the aircraft action space, the preset reward function, and the preset reward function discount factor, a Markov model is constructed using the following formula: in, This is the state space of the aircraft. For the operational space of the aircraft, For the preset reward function, The preset reward function discount factor.

3. The online flight path planning method for aircraft as described in claim 2, characterized in that, The acquisition of flight data of the aircraft, and the determination of the aircraft state space, aircraft action space, preset reward function, and preset reward function discount factor based on the flight data, include: The flight data of the aircraft is acquired, and based on the flight data, the aircraft's own state information, target point-related state information, and aircraft detection state information are obtained. The aircraft's state space is then determined using the following formula based on the aircraft's own state information, the target point-related state information, and the aircraft detection state information: in, This refers to the aircraft's own state information. The target point's related status information, Detect the status information of the aircraft; Accordingly, in, Here are the coordinates of the spacecraft in the due north direction in the northeast-northeast coordinate system. Let be the coordinates of the spacecraft in the due east direction in the northeast-northeast coordinate system. For the speed of the aircraft, Let be the angle between the aircraft's flight velocity and the positive y-axis. , For flight time, This is the maximum speed of the aircraft; For scene width, The scene length; Accordingly, in, and The coordinates of the target point, The azimuth angle of the target point, with the positive y-axis as the positive direction. ; Accordingly, in, This refers to the maximum detection range of the aircraft detection equipment. The detection readings are for the detection area in front of the aircraft. The aircraft's velocity change rate and angular velocity are obtained from the flight data. Based on these data, the aircraft's motion space is determined using the following formula: in, For the aircraft's operational space, The velocity change rate of the aircraft is... The angular velocity of the aircraft; Obtain preset aircraft distance reward, preset aircraft safe flight reward, preset aircraft velocity vector reward, and preset aircraft flight mission reward from the flight data. Determine a preset reward function based on the preset aircraft distance reward, the preset aircraft safe flight reward, the preset aircraft velocity vector reward, and the preset aircraft flight mission reward, and obtain the preset reward function discount factor corresponding to the preset reward function.

4. The online flight path planning method for aircraft as described in claim 3, characterized in that, The process of obtaining preset aircraft distance rewards, preset aircraft safe flight rewards, preset aircraft velocity vector rewards, and preset aircraft flight mission rewards from the flight data; determining a preset reward function based on the preset aircraft distance rewards, preset aircraft safe flight rewards, preset aircraft velocity vector rewards, and preset aircraft flight mission rewards; and obtaining a preset reward function discount factor corresponding to the preset reward function includes: The real-time distance between the aircraft and the target point is obtained from the flight data, and the aircraft distance reward is determined based on the real-time distance using the following formula: in, As a distance bonus for the aircraft, This represents the real-time distance between the spacecraft and the target point at the previous moment. This represents the real-time distance between the aircraft and the target point at the current moment. The aircraft's detection readings and detection distance are obtained from the flight data. Based on these readings and distance, the aircraft's safe flight reward is determined using the following formula: in, Rewards for safe flight of aircraft The distance from the aircraft to the threat boundary. The detection readings are set within a preset angle range for the left and right front sides of the aircraft. For detection distance; The preset aircraft velocity vector reward is obtained using the following formula: in, As a preset aircraft velocity vector reward, For the speed of the aircraft, Let be the angle between the aircraft's flight velocity and the positive y-axis. The azimuth of the target point; The reward for a pre-defined aircraft flight mission is determined using the following formula: in, The reward for the preset flight mission is t, where t is the flight time of the aircraft. The preset reward function is determined by the following formula: in, For the preset reward function, These are the four sub-reward coefficients; Obtain the preset reward function discount factor corresponding to the preset reward function.

5. An online flight path planning device for aircraft, characterized in that, The online flight path planning device for the aircraft includes: The Markov model building module is used to acquire the flight data of the aircraft and build a Markov model corresponding to the online route planning of the aircraft based on the flight data. The course learning model construction module is used to construct the Deep Deterministic Policy Gradient Algorithm (DDPG) neural network of the aircraft, and to construct the course learning CL model of the aircraft based on the DDPG neural network. The training module is used to train the online route planning model of the aircraft in the course learning task based on the Markov model and the course learning CL model when a course learning task is received, so as to obtain the trained final route plan. The training module is also used to obtain the state behavior values ​​of the aircraft at different times and under different aircraft states in the evaluation network using the following formula: in, For state behavior values, The state of the aircraft at time t in the evaluation network. This is the action output value of the aircraft at time t in the evaluation network. To evaluate the network; Based on the state behavior values, the action network and Critic evaluation network of the Deep Deterministic Policy Gradient Algorithm (DDPG) neural network are copied to generate an action target network and an evaluation target network. The DDPG neural network of the aircraft is then constructed based on the action target network and the evaluation target network. A course learning CL model for the aircraft is constructed based on the DDPG neural network. The training module is further configured to obtain the initial action output value and the action value after Gaussian noise processing of the DDPG neural network, and update the action output value of the corresponding path planning subtask of the aircraft according to the initial action output value and the action value using the following formula: in, This is the action output value of the aircraft at time t in the evaluation network. Output initial values ​​for the action. For the Gaussian noise that corresponds to the target of path planning subtask 1, For the Gaussian noise corresponding to threat avoidance in the path planning subtask 2, The Gaussian noise corresponding to the route optimization in the path planning subtask 3 is... The action value is the result of processing with Gaussian noise. Construct a course learning CL model for the aircraft based on the action output values ​​and the preset training course; The training module is further configured to, upon receiving a course learning task, determine the training scenario, number of training rounds, experience replay queue capacity, sampling size, maximum step size per round, action network learning rate, evaluation network learning rate, and soft update coefficient based on the Markov model and the course learning CL model; update the aircraft's action network and evaluation network based on the training scenario, number of training rounds, experience replay queue capacity, sampling size, maximum step size per round, action network learning rate, evaluation network learning rate, and soft update coefficient; and train the aircraft's online route planning model to obtain the trained final route plan.

6. An online flight path planning device for aircraft, characterized in that, The online flight path planning device includes: a memory, a processor, and an online flight path planning program stored in the memory and executable on the processor, the online flight path planning program being configured to implement the steps of the online flight path planning method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores an online flight path planning program for an aircraft, which, when executed by a processor, implements the steps of the online flight path planning method for an aircraft as described in any one of claims 1 to 4.

Images