Reentry vehicle formation control method and system based on hybrid learning method, and storage medium
By combining hybrid learning methods with supervised learning and deep reinforcement learning, an agent decision network and a multi-agent collaborative control network are established, and a coupled reward mechanism is used for training, which solves the problem that the reentry aircraft formation control method in the existing technology is difficult to explore performance boundaries and cannot ensure safety, and realizes safe formation control of the reentry aircraft cluster.
Patent Information
- Application Number
- CN202411957434.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
The existing control methods based on reinforcement learning are difficult to explore performance boundaries, and the control methods cannot ensure safety.
The reentry aircraft formation control method based on a hybrid learning method is adopted, and through a combination of supervised learning and deep reinforcement learning, an agent decision network and a multi-agent collaborative control network are established, and a coupled reward mechanism is used for training to ensure that the agent is safe during the formation control process.
It provides safety guarantees for formation flight control of reentered aircraft clusters, ensuring that the agent has safety during the exploration process and can effectively explore performance boundaries.
Smart Images
Figure CN119847199A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of reentry vehicle control technology, and in particular to a reentry vehicle formation control method, system and storage medium based on a hybrid learning method. Background Art
[0002] How to achieve safe formation control in a high-speed environment during the coordinated flight of reentry vehicles is a key issue. Traditional formation control methods are difficult to achieve effective formation control. For example, the existing leader-follower control method, virtual structure control method, behavior control method, and consistency formation control method are all based on models and error elimination methods to achieve formation coordination. In addition, intelligent formation methods are represented by reinforcement learning. The aircraft continuously interact and learn in a virtual environment to obtain an autonomous formation control strategy. However, in the reinforcement learning process, reentry vehicles have problems with exploring dangerous areas and unsafe actions, which brings certain risks to formation control. In addition, due to the influence of high dynamic environments and strong time-varying parameters, the above traditional control methods are difficult to explore performance boundaries, and the above intelligent control methods cannot ensure safety. Summary of the invention
[0003] The technical problems to be solved by the present invention are:
[0004] Existing control methods based on reinforcement learning have difficulty in exploring performance boundaries and cannot ensure safety.
[0005] The present invention adopts the following technical solutions to solve the above technical problems:
[0006] The present invention provides a reentry vehicle formation control method based on a hybrid learning method, comprising the following steps:
[0007] Step 1: Obtain the flight trajectory data of a single aircraft, including aircraft attitude data, control command data and position data;
[0008] Step 2: Take the aircraft as the intelligent agent, establish an intelligent agent decision network, train the intelligent agent decision model through supervised learning, and obtain the intelligent agent autonomous control model;
[0009] Step 3: Based on the deep reinforcement learning theory, a multi-agent collaborative control network is established. The agent autonomous control model of step 2 is applied to the multi-agent collaborative control network. A coupled reward mechanism including single-vessel reward and group reward is adopted to conduct collaborative control training on the agent autonomous control model to obtain a multi-agent collaborative control model.
[0010] Step 4: Obtain the re-entry vehicle attitude data, position data and relative position error in the flight environment, obtain the formation control instructions of the re-entry vehicle cluster through the multi-agent collaborative control model, and perform collaborative control of the re-entry vehicle formation.
[0011] Furthermore, step one includes the following steps:
[0012] Step 1: Initialize the attitude of a single aircraft;
[0013] Step 1 and 2: Reenter the aircraft and obtain the aircraft attitude data and position data under different control instructions;
[0014] Step 13: Repeat step 11 to step 12 to obtain multiple aircraft trajectory data.
[0015] Furthermore, the aircraft attitude data in steps one and two include: altitude, track inclination angle, track yaw angle and speed data, and the aircraft control instructions include angle of attack and roll angle.
[0016] Furthermore, step 2 includes the following steps:
[0017] Step 21: Build an agent decision network, including:
[0018] Input layer: The number of input layer nodes is 5;
[0019] Hidden layer: The hidden layer consists of 2 layers;
[0020] Output layer: The number of nodes in the output layer is 2, and the ReLU activation function is used;
[0021] Step 22: Use the behavior cloning learning method to train the agent decision network and obtain the agent autonomous control model.
[0022] Furthermore, step three includes the following process:
[0023] Step 31: Establish a multi-agent collaborative control network based on deep reinforcement learning theory, and apply the agent autonomous control model to the multi-agent collaborative control network;
[0024] Step 32: Construct a reward function that includes individual aircraft rewards and group rewards. Individual rewards include flight altitude maintenance rewards and flight direction maintenance rewards, and group rewards include error elimination rewards between aircraft. Construct a group reward discount factor. When the individual reward is small, provide a larger discount factor and reduce the group reward at the same time.
[0025] Step 33: Obtain the aircraft attitude data, position data, control instructions and relative position error during the flight of the aircraft, use reinforcement learning to train the intelligent agent autonomous control model, extract batch samples from the experience pool, update the value function network, minimize the loss function, optimize the control strategy network parameters of the aircraft, and finally obtain a multi-agent collaborative control model.
[0026] Furthermore, the optimization process of the aircraft control strategy in step 3 is:
[0027]
[0028] In the formula, π is the strategy, π * is the optimal strategy, arg is the optimization target state, is the expectation, t is the current step number, ρ π represents the state transition distribution under strategy π, Represents the observation quantity of agent i Next action Rewards received, For strategy The entropy of h is the entropy weight coefficient.
[0029] Furthermore, in step 33, the angle of attack in the sample data is subtracted from the angle of attack obtained by the model decision, and the roll angle in the sample data is subtracted from the roll angle obtained by the model decision. The two differences are weighted and combined to obtain the loss function, that is:
[0030] Loss = k1(α output -α true )+k2(σ output -σ true )
[0031] In the formula, Loss is the loss function, α output is the output angle of attack, σ output is the output roll angle value, α true is the angle of attack value in the sample, σ true is the roll angle value in the sample, k1 and k2 are weight coefficients.
[0032] Furthermore, in step 4, the communication topology is dynamically adjusted according to the distance between the aircraft, the state average value of a certain aircraft or several aircraft is used as a reference, and the error amount is calculated as the relative position error of the aircraft.
[0033] The present invention provides a re-entry vehicle formation control system based on a hybrid learning method. The system has a program module corresponding to the steps of the method described in any one of the above technical solutions, and executes the steps in the above-mentioned re-entry vehicle formation control method based on the hybrid learning method during operation.
[0034] The present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to implement the steps of the reentry vehicle formation control method based on a hybrid learning method described in any one of the above-mentioned technical solutions when called by a processor.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The present invention combines supervised learning and reinforcement learning to provide safety for the formation flight control of a cluster of reentry vehicles. In the process of supervised learning, a behavioral cloning method is used to learn a general single-body control method using a single reentry vehicle, ensuring the safety of the aircraft itself when the intelligent agent makes decisions. In the learning of the collaborative formation control strategy, the strategy obtained by supervised learning is used as the initial intelligent agent strategy, and the reinforcement learning method is used for training. A reward function that integrates single-body rewards and group rewards is designed, and coupled rewards are used to ensure that the intelligent agent can ensure its own safety when learning the formation control strategy.
[0037] The purpose of the present invention is to provide a method for intelligent formation control of reentry vehicles, avoid the random exploration of dangerous areas by intelligent agents, provide an initial strategy for reinforcement learning through supervised training of behavior cloning, and improve the safety of the intelligent agent's exploration process. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a framework diagram of a reentry vehicle formation control method based on a hybrid learning method in an embodiment of the present invention;
[0039] Figure 2 is a schematic diagram of a deep neural network in an embodiment of the present invention;
[0040] Figure 3 A diagram showing the interaction between a reentry vehicle and an environment in an embodiment of the present invention;
[0041] Figure 4 This is a schematic diagram of a horizontal plane projection result of performing formation control on two aircraft in an embodiment of the present invention;
[0042] Figure 5 It is a schematic diagram of the longitudinal screen projection result of formation control of two aircraft in an embodiment of the present invention. DETAILED DESCRIPTION
[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.
[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0045] Specific implementation plan 1: Combine Figures 1 to 4 As shown, the present invention provides a reentry vehicle formation control method based on a hybrid learning method, comprising the following steps:
[0046] Step 1: Obtain the flight trajectory data of a single aircraft under the traditional control method, including aircraft attitude data, control command data and position data;
[0047] Step 2: Take the aircraft as the intelligent agent, establish an intelligent agent decision network, train the intelligent agent decision model through supervised learning, and obtain the intelligent agent autonomous control model;
[0048] Step 3: Based on the deep reinforcement learning theory, a multi-agent collaborative control network is established. The agent autonomous control model of step 2 is applied to the multi-agent collaborative control network. A coupled reward mechanism including single-vessel reward and group reward is adopted to conduct collaborative control training on the agent autonomous control model to obtain a multi-agent collaborative control model.
[0049] Step 4: Obtain the re-entry vehicle attitude data, position data and relative position error in the actual flight environment, obtain the formation control instructions of the re-entry vehicle cluster through the multi-agent collaborative control model, and perform collaborative control of the re-entry vehicle formation.
[0050] Specific implementation plan 2: Step 1 includes the following steps:
[0051] Step 1: Initialize the attitude of a single aircraft and select different target positions within a certain range as a new simulation scene;
[0052] Step 1 and 2: The reentry vehicle flies according to the quasi-balanced gliding guidance method to obtain the vehicle attitude data and position data under different control instructions;
[0053] Step 13: Perform dynamic integration until the aircraft reaches the target position;
[0054] Step 13: Repeat steps 11 to 13 to obtain multiple aircraft trajectory data. The data at each moment in a trajectory is a sample.
[0055] Specific implementation plan three: The aircraft attitude data in steps one and two include: altitude, track inclination angle, track yaw angle and speed data, aircraft control command data, and control command data calculated according to the flight status, including angle of attack and roll angle.
[0056] Specific implementation scheme 4: Step 2 includes the following steps:
[0057] Step 21: Build an agent decision network, including:
[0058] Input layer: The number of input layer nodes is 5, and the input includes the aircraft's altitude, track inclination angle, track yaw angle, speed, and relative position error. For a single aircraft, multi-agent collaboration is not considered. Since this decision network needs to be aligned with the network input dimension in subsequent reinforcement learning, 5 inputs are used.
[0059] Hidden layer: The hidden layer consists of 2 layers;
[0060] Output layer: The number of nodes in the output layer is 2. The output includes the angle of attack and the roll angle, and the ReLU activation function is used.
[0061] Step 22: Use the behavior cloning learning method to train the intelligent agent decision network so that the network can completely clone the aircraft behavior in the sample data and obtain the intelligent agent autonomous control model.
[0062] Specific implementation plan five: Step three includes the following process:
[0063] Step 31: Establish a multi-agent collaborative control network based on deep reinforcement learning theory, and apply the agent autonomous control model to the multi-agent collaborative control network; make the initial network of the agent a network trained by supervised learning, so that it has the ability to fly autonomously and safely from the beginning, and further learn how to collaborate with other agents in subsequent reinforcement learning training.
[0064] Step 32: Construct a reward function that includes individual rewards for aircraft and group rewards. Individual rewards include rewards for maintaining flight altitude and maintaining flight direction, and group rewards include rewards for eliminating errors between aircraft. Construct a group reward discount factor. When the individual reward is small, provide a larger discount factor while reducing the group reward. Ensure that the agent maintains its own safety before completing the collaborative task.
[0065] Step 33: Initialize the state of the simulation environment. The aircraft, as an intelligent agent, continuously interacts with the simulation environment to obtain the current observations of the aircraft during flight (i.e., aircraft altitude, track azimuth, track inclination, speed, and relative position error). The action (angle of attack, angle of approach) obtained based on the current observations, the reward obtained, and the observation at the next moment are stored as a set of experience in the experience pool. The agent autonomous control model is trained using a method based on soft actor-critic reinforcement learning. Batch samples are extracted from the experience pool, the value function network is updated, the loss function is minimized, and the control strategy network parameters of the aircraft are optimized to finally obtain a multi-agent collaborative control model.
[0066] Specific implementation plan six: The optimization process of the aircraft control strategy in step three is:
[0067]
[0068] In the formula, π is the strategy, π* is the optimal strategy, and arg is the state that needs to be achieved by the optimization goal, that is, the goal is to maximize the expected value of the cumulative reward under strategy π. Find the expected value of the mathematical symbol, t is the current step number, ρ π represents the state transition distribution under strategy π, Represents the observation quantity of agent i Next action Rewards received, For strategy The entropy of h is the entropy weight coefficient.
[0069] Specific implementation scheme seven: In step 33, the angle of attack in the sample data is subtracted from the angle of attack obtained by the model decision, and the roll angle in the sample data is subtracted from the roll angle obtained by the model decision, and the two differences are weightedly combined to obtain the loss function, that is:
[0070] Loss = k1(α output -α true )+k2(σ output -σ true )
[0071] In the formula, Loss is the loss function, α output is the output angle of attack, σ output is the output roll angle value, α true is the angle of attack value in the sample, σ trueis the roll angle value in the sample, k1 and k2 are weight coefficients. The part with a larger weight coefficient will occupy a larger proportion in the learning process. The weight coefficient can be selected according to the variation range of the two parameters. The variation range of the attack angle is generally 0°~20°, and the variation range of the roll angle is generally -30°~30°. In order to make the influence of each part basically the same, in the actual process, k1=1 / 20, k2=1 / 60.
[0072] Specific implementation scheme eight: In step four, the communication topology is dynamically adjusted according to the distance between the aircraft, the state average value of a certain aircraft or several aircraft is used as a reference, and the error amount is calculated as the relative position error of the aircraft.
[0073] The reentry vehicle formation control method (algorithm) based on a hybrid learning method proposed in the present invention is the underlying technical core of the present invention, and various products can be derived based on the algorithm.
[0074] Based on the method proposed in the present invention, a re-entry vehicle formation control system based on a hybrid learning method is developed using a programming language. The system has program modules corresponding to the steps of the above-mentioned technical solution, and executes the steps in the above-mentioned re-entry vehicle formation control method based on the hybrid learning method during runtime.
[0075] The computer program of the developed system (software) is stored on a computer-readable storage medium, and the computer program is configured to implement the steps of the above-mentioned reentry vehicle formation control method based on the hybrid learning method when called by a processor, that is, the present invention is materialized on a carrier to become a computer program product.
[0076] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0077] The computer programs (also referred to as programs, software, software applications, or codes) of the present invention include machine instructions for programmable processors, and these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or device (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0078] Example 1
[0079] This embodiment experiment uses a digital model of a typical reentry vehicle for simulation verification:
[0080] The experiment conducted formation control on two reentry vehicles. The initial altitude of the two vehicles was 80 km, the initial speed was 3000 m / s, and the initial coordinates on the horizontal plane were (0 m, 50 km) 、 (0m, -50km). A fixed direction is selected as the desired formation flight direction. The simulation results are as follows: Figure 4 and Figure 5 As shown, Figure 4 This is the projection of the flight paths of the two aircraft on the horizontal plane. It can be seen that the two aircraft can adjust their flight directions synchronously and keep flying in the same direction. Figure 5 The projection of the flight trajectories of the two aircraft on the longitudinal plane shows that the aircraft can basically maintain the same flight altitude and fly in the same direction. The results show that the hybrid learning method of the present invention can enable the reentry vehicle to maintain its own stable gliding while maintaining the formation, which has good safety in the initial exploration stage, learning process and actual application stage of the aircraft.
[0081] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.
Claims
1. A reentry vehicle formation control method based on a hybrid learning method, characterized in that: The following steps are involved: Step 1: Obtain the flight trajectory data of a single aircraft, including aircraft attitude data, control command data and position data; Step 2: Take the aircraft as the intelligent agent, establish an intelligent agent decision network, train the intelligent agent decision model through supervised learning, and obtain the intelligent agent autonomous control model; Step 3: Based on the deep reinforcement learning theory, a multi-agent collaborative control network is established. The agent autonomous control model of step 2 is applied to the multi-agent collaborative control network. A coupled reward mechanism including single-vessel reward and group reward is adopted to conduct collaborative control training on the agent autonomous control model to obtain a multi-agent collaborative control model. Step 4: Obtain the re-entry vehicle attitude data, position data and relative position error in the flight environment, obtain the formation control instructions of the re-entry vehicle cluster through the multi-agent collaborative control model, and perform collaborative control of the re-entry vehicle formation.
2. The reentry vehicle formation control method based on hybrid learning method according to claim 1, characterized in that: Step 1 includes the following steps: Step 1: Initialize the attitude of a single aircraft; Step 1 and 2: Reenter the aircraft and obtain the aircraft attitude data and position data under different control instructions; Step 13: Repeat step 11 to step 12 to obtain multiple aircraft trajectory data.
3. The reentry vehicle formation control method based on hybrid learning method according to claim 2, characterized in that: The aircraft attitude data in steps one and two include: altitude, track inclination angle, track yaw angle and speed data, and the aircraft control instructions include angle of attack and roll angle.
4. The reentry vehicle formation control method based on hybrid learning method according to claim 1, characterized in that: Step 2 includes the following steps: Step 21: Build an agent decision network, including: Input layer: The number of input layer nodes is 5; Hidden layer: The hidden layer consists of 2 layers; Output layer: The number of nodes in the output layer is 2, and the ReLU activation function is used; Step 22: Use the behavior cloning learning method to train the agent decision network and obtain the agent autonomous control model.
5. The reentry vehicle formation control method based on hybrid learning method according to claim 4, characterized in that: Step 3 includes the following process: Step 31: Establish a multi-agent collaborative control network based on deep reinforcement learning theory, and apply the agent autonomous control model to the multi-agent collaborative control network; Step 32: Construct a reward function that includes individual aircraft rewards and group rewards. Individual rewards include flight altitude maintenance rewards and flight direction maintenance rewards, and group rewards include error elimination rewards between aircraft. Construct a group reward discount factor. When the individual reward is small, provide a larger discount factor and reduce the group reward at the same time. Step 33: Obtain the aircraft attitude data, position data, control instructions and relative position error during the flight of the aircraft, use reinforcement learning to train the intelligent agent autonomous control model, extract batch samples from the experience pool, update the value function network, minimize the loss function, optimize the control strategy network parameters of the aircraft, and finally obtain a multi-agent collaborative control model.
6. The reentry vehicle formation control method based on hybrid learning method according to claim 5, characterized in that: The optimization process of the aircraft control strategy in step 3 is: In the formula, π is the strategy, π* is the optimal strategy, arg is the optimized target state, is the expectation, t is the current step number, ρ π represents the state transition distribution under strategy π, Represents the observation quantity of agent i Next action Rewards received, For strategy The entropy of , αh is the entropy weight coefficient.
7. The reentry vehicle formation control method based on hybrid learning method according to claim 6, characterized in that: In step 3, the angle of attack in the sample data is subtracted from the angle of attack obtained by the model decision, and the roll angle in the sample data is subtracted from the roll angle obtained by the model decision. The two differences are weighted and combined to obtain the loss function, that is: Loss=k1(αoutput-αtrue)+k2(σoutput-σtrue) Where Loss is the loss function, αoutput is the output angle of attack value, σoutput is the output roll angle value, αtrue is the angle of attack value in the sample, σtrue is the roll angle value in the sample, and k1 and k2 are weight coefficients.
8. The reentry vehicle formation control method based on hybrid learning method according to claim 7, characterized in that: In step 4, the communication topology is dynamically adjusted according to the distance between the aircraft, the state average of a certain aircraft or several aircraft is used as a reference, and the error amount is calculated as the relative position error of the aircraft.
9. A reentry vehicle formation control system based on a hybrid learning method, characterized in that: The system has a program module corresponding to the steps of the method described in any one of claims 1 to 8, and executes the steps of the reentry vehicle formation control method based on the hybrid learning method when running.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps in the reentry vehicle formation control method based on a hybrid learning method according to any one of claims 1 to 8 when called by a processor.