A hypersonic vehicle trajectory planning method based on reinforcement learning
Through a method based on reinforcement learning, DQN network is constructed to carry out trajectory planning of hypersonic vehicles, solving the problem of trajectory planning accuracy and speed of the aircraft in the no-fly zone evasion, achieving higher safety and efficiency.
Patent Information
- Application Number
- CN202211400265.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-09
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2042-11-09
AI Technical Summary
It is difficult for hypersonic vehicles to quickly and accurately avoid the no-fly zone during flight, and traditional trajectory planning methods are difficult to take into account maneuverability constraints, trajectory planning accuracy and speed.
Using a reinforcement learning-based method, by constructing a DQN network, input the aircraft's flight status and no-fly zone constraints, output the evaluation value of the action space, and filter and modify the evaluation value of the infeasible action, thereby achieving fast and accurate trajectory planning.
It improves the accuracy and speed of trajectory planning of hypersonic vehicles in the evasion of no-fly zones, and enhances the safety and efficiency of the aircraft.
Smart Images
Figure CN115903887B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hypersonic aircraft trajectory planning, and in particular to a hypersonic aircraft trajectory planning method based on reinforcement learning. Background Art
[0002] During the flight of a hypersonic aircraft, due to political and geographical factors, air defense and anti-missile strategic deployment and other issues, these locations become no-fly zones, and these no-fly zones must be maneuvered to avoid them. However, hypersonic aircraft have weak maneuverability and are difficult to avoid once a new no-fly zone appears temporarily. Therefore, fast and accurate trajectory planning can improve the avoidance effect of the aircraft.
[0003] Hypersonic vehicle trajectory planning is usually done through path planning and trajectory optimization. Hypersonic vehicles have long flight distances, high flight speeds, and weak maneuverability. In this context, traditional methods are difficult to take into account maneuverability constraints, trajectory planning accuracy, and trajectory planning speed at the same time.
[0004] Therefore, how to provide a hypersonic aircraft trajectory planning method that improves the accuracy and speed of hypersonic aircraft trajectory planning while meeting the maneuverability constraints has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The purpose of the present invention is to provide a hypersonic aircraft trajectory planning method based on reinforcement learning to solve the above problems.
[0006] The present invention solves the technical problem by adopting the following technical solution:
[0007] A hypersonic vehicle trajectory planning method based on reinforcement learning, the method comprising:
[0008] Step S1: constructing a trajectory planning scenario, wherein the trajectory planning scenario is planned based on the flight status of the aircraft and the no-fly zone constraints;
[0009] Step S2: construct the state space, design the space action set and reward form;
[0010] Step S3: Establish a DQN network, wherein the input variable of the DQN network is the state space constructed in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2;
[0011] Step S4: Determine the element evaluation value, input the current state space into the DQN network, and output the evaluation value of each element in the action space set under the current state;
[0012] Step S5: updating the element evaluation value, based on the trajectory prediction result, filtering the inactive actions in the current state, and modifying the evaluation value of the inactive action element;
[0013] Step S6: Input trajectory planning control parameters, select the action element with the highest evaluation value as the input of the aircraft kinematic equation, and update the position, speed, and state based on the aircraft kinematics.
[0014] Step S7: Task determination: The task determination is based on the latest position of step S6 to determine whether the trajectory planning task is completed. If the task completion condition is met, the trajectory is saved and introduced into the training data set. Otherwise, steps S4, S5, S6, and S7 are iteratively executed.
[0015] A hypersonic vehicle trajectory planning system based on reinforcement learning, the system comprising the following units;
[0016] Trajectory planning scenario construction unit: used to construct trajectory planning scenarios, wherein the trajectory planning scenarios are planned based on the flight status of the aircraft and the no-fly zone constraints;
[0017] Learning environment building unit: used to build state space, design space action set and reward form;
[0018] DQN network construction unit: used to establish a DQN network, wherein the input variable of the DQN network is the state space constructed in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2;
[0019] The first execution unit is used to determine the element evaluation value, input the current state space into the DQN network, and output the evaluation value of each element in the action space set under the current state;
[0020] The second execution unit is used to update the element evaluation value, based on the trajectory prediction result, filter the inactive actions in the current state, and modify the evaluation value of the inactive action element;
[0021] The third execution unit is used for trajectory planning control parameter input, selecting the action element with the highest evaluation value as the input of the aircraft kinematic equation, and updating the position, speed, and state based on the aircraft kinematics.
[0022] Determination iteration unit: used for task determination. The task determination is based on the latest position of the aircraft to determine whether the trajectory planning task is completed. If the task end condition is met, the trajectory is saved and introduced into the training data set. Otherwise, the execution steps from the first execution unit to the third execution unit are iteratively executed.
[0023] An information storage device stores a hypersonic aircraft trajectory planning method based on reinforcement learning.
[0024] Beneficial effects:
[0025] The present invention discloses a hypersonic aircraft trajectory planning method based on reinforcement learning. Based on a DQN network, the value of elements in an action set is quickly evaluated. By modifying the evaluation value of an inactionable action, actions that may enter a no-fly zone are excluded, thereby realizing rapid trajectory planning while avoiding the no-fly zone. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 A flow chart of a hypersonic vehicle trajectory planning method based on reinforcement learning provided by the present invention.
[0027] Figure 2 This is the scene state diagram of the present invention.
[0028] Figure 3 Schematic diagram of the scene state of the present invention.
[0029] Figure 4 This is a DQN network structure diagram of the present invention.
[0030] Figure 5 This is a schematic diagram of invalid action screening of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] Example 1
[0033] The present invention discloses a hypersonic vehicle trajectory planning method based on reinforcement learning, comprising the following steps:
[0034] Step S1: construct a trajectory planning scenario based on the aircraft flight status and no-fly zone constraints;
[0035] Step S2, designing action sets, state spaces, and reward forms, extracting images and the initial position of the aircraft and the position information of the target point from the trajectory planning scene to construct the state space;
[0036] Step S3, establish a DQN network, the input variable is the state space obtained in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2;
[0037] Step S4: Input the current state to the DQN network and output the evaluation value of each element in the current state;
[0038] Step S5: based on the trajectory prediction result, filter the inactive actions in the current state and modify the evaluation value of the inactive action element;
[0039] Step S6: Select the action element with the highest evaluation value as the input of the aircraft kinematics equation, and update the position, speed and state based on the aircraft kinematics;
[0040] Step S7: According to the latest position of step S6, determine whether the trajectory planning task is completed. If the task completion condition is met, save the trajectory and introduce the trajectory into the training data set. Otherwise, iterate and execute steps S4, S5, S6, and S7.
[0041] In this embodiment, the step S2 specifically includes:
[0042] Design action set, state space, and reward form:
[0043] Take a screenshot of the combat scene and binarize the safe area and the no-fly zone. This binary image is part of the state; the other part of the state is the relative position of the aircraft and the target.
[0044] The action space is to uniformly take a finite number of acceleration values within the range of lateral acceleration within the maneuverability of the aircraft;
[0045] The reward form is positive reward for entering the target area, negative reward for entering the no-fly zone, and negative reward for flight time and overload changes.
[0046] According to the above steps, a learning environment for reinforcement learning interaction is obtained.
[0047] In this embodiment, step S3 specifically includes:
[0048] The DQN network structure is sufficient for processing state input;
[0049] The input of the network is a three-dimensional tensor of a fixed-size image and a vector of the relative position of the target and the aircraft. The network structure can extract and reduce the dimension of the line of sight input features;
[0050] The network consists of convolutional layers, pooling layers and fully connected layers.
[0051] In this embodiment, step S5 specifically includes:
[0052] Calculate the disabled actions in the current state;
[0053] Classify each no-fly zone according to its distance and direction from the aircraft to determine whether a normal flight will enter a no-fly zone;
[0054] If normal flight will enter a no-fly zone, the most likely no-fly zone and the no-fly zones bordering it are screened out.
[0055] Calculate the acceleration range that can enter the screened no-fly zone under constant acceleration.
[0056] Compare the above acceleration range with the elements in the action set, filter out the actions within the range, and reduce their evaluation values.
[0057] In this embodiment, the step S7 specifically includes the following steps:
[0058] If the latest position in step S6 reaches the target area or enters the no-fly zone, the state, action and reward result in the trajectory planning process are added to the training data.
[0059] If the latest position in step S6 is in the safe area outside the target area, then continue to iterate and execute steps S4, S5, S6, and S7.
[0060] Example 2
[0061] like Figure 1 As shown, the present invention provides a hypersonic aircraft trajectory planning method based on reinforcement learning, comprising the following steps:
[0062] Step 1: Based on the flight status of the aircraft and the no-fly zone constraints, construct a trajectory planning scenario. Obtain the no-fly zone location, no-fly zone radius, starting point location, and end point location, and draw the no-fly zone in the form of a circle in the scene graph.
[0063] Step 2: Design the action set, state space, and reward form, and extract the image and the initial position of the aircraft and the target point position information from the trajectory planning scene to construct the state space.
[0064] Specifically, the steps of action set, state space, and reward form design are described in detail as follows:
[0065] Step 2.1, design action space:
[0066] The maximum maneuvering acceleration of a hypersonic vehicle is 20m / s 2 , take 9 values evenly in this space, specifically -20m / s 2 、-15m / s 2 、-10m / s 2 、-5m / s 2 、0m / s 2 , 5m / s 2 、10m / s 2 、15m / s 2 、20m / s 2.
[0067] Step 2.2: Design state space
[0068] The state space is a binary image and a vector. The image is a thumbnail of the combat scene, such as Figure 2 As shown, the image size is a 2:1 rectangle, with the long side being twice the aircraft's maximum flight distance and the short side being the aircraft's maximum flight distance. The aircraft's position should correspond to the midpoint of the bottom long side, and the aircraft's heading should point to the center of the image. Figure 2 The meaning of the inner elements is shown in Figure 3 The vector state is the direction and distance of the target relative to the aircraft.
[0069] Step 2.3, set the reward value, +1000 for reaching the target point, -10000 for entering the no-fly zone, -1 for every 1s of track flight time, and each time an action is selected, the reward is the opposite of the absolute value of the difference between the selected action and the last selected action.
[0070] Step 3: Based on the state space, action space, and reward value obtained in step 2, a DQN evaluation network is established to output the reward value of each action.
[0071] Specifically, Figure 4 As shown, the detailed description of the established DQN network prediction model structure is as follows:
[0072] Step 3.1: The network input is divided into two parts, one is the picture part and the other is the vector part.
[0073] Step 3.2: The image input size is m×n, the vector length is 2, and it is input into the DQN network.
[0074] Step 3.3: The image is first input into the convolutional pooling layer for feature extraction, and the features are used instead of the image pixels to input into the fully connected network. The m×n-dimensional image is input into the convolutional pooling layer, and k features are extracted; the k features and the two elements of the state vector are combined into a vector of length k+2; the vector of length k+2 is input into the fully connected network to output 9 values.
[0075] Step 4: Input the current state into the DQN network and output the evaluation value of each element in the current state.
[0076] Step 5: Based on the trajectory prediction results, filter the inactive actions in the current state and modify the evaluation values of the inactive action elements.
[0077] Specifically, the action screening method is as follows:
[0078] Step 5.1: First check whether there is a no-fly zone within the 800km range. If there is no no-fly zone, do not filter. Otherwise, go to step 5.2.
[0079] Step 5.2, calculate the circular motion trajectory tangent to the no-fly zone, such as Figure 5 As shown, the inoperable action range is obtained:
[0080]
[0081] V represents the flight speed of the aircraft, R d Represents the radius of the no-fly zone; r d Represents the distance of the no-fly zone relative to the aircraft; η d Indicates the angle between the aircraft's speed direction and the no-fly zone.
[0082] Step 5.3, change the action evaluation within this range to -∞;
[0083] Step 6: Select the action element with the highest evaluation value as the input of the aircraft kinematic equation, and update the position, velocity, and state based on the aircraft kinematics.
[0084]
[0085] Said Indicates the speed of the aircraft in the X-axis direction. Indicates the speed of the aircraft in the Y direction, represents the movement angle of the aircraft, and V represents the flight speed of the aircraft;
[0086] Step 7: According to the latest position in step 6, determine whether the trajectory planning task is completed. If the task end condition is met, save the trajectory and introduce the trajectory into the training data set. Otherwise, iterate and execute steps 4, 5, 6, and 7.
[0087] Embodiment 3
[0088] A hypersonic vehicle trajectory planning system based on reinforcement learning, the system comprising the following units;
[0089] Trajectory planning scenario construction unit: used to construct trajectory planning scenarios, wherein the trajectory planning scenarios are planned based on the flight status of the aircraft and the no-fly zone constraints;
[0090] Learning environment building unit: used to build state space, design space action set and reward form;
[0091] DQN network construction unit: used to establish a DQN network, wherein the input variable of the DQN network is the state space constructed in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2;
[0092] The first execution unit is used to determine the element evaluation value, input the current state space into the DQN network, and output the evaluation value of each element in the action space set under the current state;
[0093] The second execution unit is used to update the element evaluation value, based on the trajectory prediction result, filter the inactive actions in the current state, and modify the evaluation value of the inactive action element;
[0094] The third execution unit is used for trajectory planning control parameter input, selecting the action element with the highest evaluation value as the input of the aircraft kinematic equation, and updating the position, speed, and state based on the aircraft kinematics.
[0095] Determination iteration unit: used for task determination. The task determination is based on the latest position of the aircraft to determine whether the trajectory planning task is completed. If the task end condition is met, the trajectory is saved and introduced into the training data set. Otherwise, the execution steps from the first execution unit to the third execution unit are iteratively executed.
[0096] The present invention discloses a hypersonic aircraft trajectory planning method based on reinforcement learning. Based on the DQN network, the value of elements in the action set is quickly evaluated; by modifying the evaluation value of the inaction action, the action that will enter the no-fly zone is excluded, and the trajectory planning is realized quickly while avoiding the no-fly zone, which provides favorable support for the safe flight of hypersonic aircraft.
[0097] In view of the shortcomings of existing methods, the present invention proposes a hypersonic vehicle trajectory planning method based on reinforcement learning. On the basis of trajectory planning scenario modeling, the current action value is evaluated by alternating convolutional pooling layers and fully connected layers; based on action screening and inaction action value modification, the safety of trajectory planning is improved, the efficiency of trajectory planning is improved, and real-time trajectory planning is realized.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A hypersonic vehicle trajectory planning method based on reinforcement learning, characterized in that: The method comprises: Step S1: constructing a trajectory planning scenario, wherein the trajectory planning scenario is planned based on the flight status of the aircraft and the no-fly zone constraints; Step S2: construct the state space, design the space action set and reward form; Step S3: Establish a DQN network, wherein the input variable of the DQN network is the state space constructed in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2; Step S4: Determine the element evaluation value, input the current state space into the DQN network, and output the evaluation value of each element in the action space set under the current state; Step S5: updating the element evaluation value, based on the trajectory prediction result, filtering the inactive actions in the current state, and modifying the evaluation value of the inactive action element; Step S6: trajectory planning control parameter input, select the action element with the highest evaluation value as the aircraft kinematic equation input, and update the position, speed, and state based on the aircraft kinematics; Step S7: Task determination, the task determination is based on the latest position of step S6 to determine whether the trajectory planning task is completed. If the task end condition is met, the trajectory is saved and introduced into the training data set, otherwise steps S4, S5, S6, and S7 are iteratively executed; The specific steps of constructing the state space are as follows: taking a screenshot of the combat scene, binarizing the safe area and the no-fly zone, and the binarized image is part of the state; the other part of the state is the relative position of the aircraft and the target; The spatial action set is a collection of finite acceleration values uniformly taken within the range of lateral acceleration within the maneuverability of the aircraft; The reward form is determined as a positive reward for entering the target area, a negative reward for entering the no-fly zone, and changes in flight time and overload produce a penalty effect in the form of negative rewards.
2. The hypersonic vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The aircraft flight state includes a starting point position and an ending point position, and the no-fly zone constraint includes a no-fly zone position and a no-fly zone radius.
3. The hypersonic vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The DQN network consists of a convolutional layer, a pooling layer and a fully connected layer. The input of the DQN network is a three-dimensional tensor of a fixed-size image and a vector of the relative position of the target and the aircraft. The network structure can extract and reduce the dimension of the line of sight input features.
4. The hypersonic vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The inoperable actions are classified according to the distance and direction of each no-fly zone relative to the aircraft to determine whether normal flight will enter the no-fly zone; If normal flight will enter the no-fly zone, the most likely no-fly zone and the no-fly zones adjacent to it are screened out; the acceleration range that can enter the screened no-fly zone under constant acceleration is calculated; the above acceleration range is compared with the elements in the spatial action set, and the actions within the range are screened out, and their evaluation values are reduced.
5. The hypersonic vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The evaluation value of the inactive action is infinitely small.
6. A hypersonic vehicle trajectory planning system based on reinforcement learning, characterized in that: The system comprises the following units: Trajectory planning scenario construction unit: used to construct trajectory planning scenarios, wherein the trajectory planning scenarios are planned based on the flight status of the aircraft and the no-fly zone constraints; Learning environment building unit: used to build state space, design space action set and reward form, The specific steps of constructing the state space are as follows: taking a screenshot of the combat scene, binarizing the safe area and the no-fly zone, and the binarized image is part of the state; the other part of the state is the relative position of the aircraft and the target; The spatial action set is a collection of finite acceleration values uniformly taken within the range of lateral acceleration within the maneuverability of the aircraft; The reward form is determined as follows: entering the target area will get a positive reward, entering the no-fly zone will get a negative reward, and the change in flight time and overload will produce a penalty effect in the form of a negative reward; DQN network construction unit: used to establish a DQN network, wherein the input variable of the DQN network is the state space constructed in step S2, the output variable is the action space obtained in step S2, and the prediction result is the evaluation value of each element in the action space set in step S2; The first execution unit is used to determine the element evaluation value, input the current state space into the DQN network, and output the evaluation value of each element in the action space set under the current state; The second execution unit is used to update the element evaluation value, based on the trajectory prediction result, filter the inactive actions in the current state, and modify the evaluation value of the inactive action element; The third execution unit is used for trajectory planning control parameter input, selecting the action element with the highest evaluation value as the aircraft kinematic equation input, and updating the position, speed, and state based on the aircraft kinematics; Determination iteration unit: used for task determination. The task determination is based on the latest position of the aircraft to determine whether the trajectory planning task is completed. If the task end condition is met, the trajectory is saved and introduced into the training data set. Otherwise, the execution steps from the first execution unit to the third execution unit are iteratively executed.
7. An information storage device, characterized in that: The storage device stores a hypersonic aircraft trajectory planning method based on reinforcement learning as described in one of claims 1-5.