Helicopter visual simulation platform building method based on reinforcement learning

Abstract

The invention discloses a helicopter visual simulation platform building method based on reinforcement learning. The helicopter visual simulation platform building method comprises the following steps: S1, packaging a helicopter dynamics model; s2, constructing an initialized agent through combination of a reinforcement learning method and a helicopter dynamics system; s3, interacting with the environment through an operation instruction generated by the intelligent agent, and training the intelligent agent; s4, testing the training result of the intelligent agent, and explicitly displaying the flight effect of the testing process; the invention provides a helicopter flight control method based on reinforcement learning, and aims to improve the performance of a flight control and decision-making system through adaptive model adjustment and self-supervision auxiliary tasks. The operation and the environment are effectively combined through a reinforcement learning method, so that the helicopter can realize flight control under complex and uncertain conditions, and a specified task is completed.

Description

Technical Field

[0001] The invention relates to the technical field of alloy materials, and in particular to a method for building a helicopter visualization simulation platform based on reinforcement learning. Background Art

[0002] As a unique aircraft, helicopters have multiple functions such as vertical take-off and landing, hovering, forward, backward and lateral flight, which makes them play an irreplaceable role in many fields. These fields include but are not limited to medical emergency, search and rescue, military operations, traffic management, environmental monitoring, forest fire prevention, oil and gas exploration and news gathering. Its unique flight capabilities give it a unique advantage in difficult terrain and urban environments. Its flexibility and agility are unmatched by fixed-wing aircraft. This capability is particularly important in emergency rescue, disaster response and transportation in complex terrain.

[0003] However, the flight operation of a helicopter is complex, requiring the pilot to simultaneously control multiple control surfaces (such as rotors, tail rotors, etc.) and power systems. Due to the relatively low stability of helicopters in the air, operational errors may lead to serious flight accidents. Pilots need to face a variety of complex flight environments and emergency situations in actual operations, and every mistake may have catastrophic consequences. Therefore, pilots need to undergo a lot of training to master various flight skills and emergency handling capabilities. At the same time, the mechanical system of a helicopter includes multiple key components such as rotors, transmission systems, and engines, and the failure of these components may lead to flight accidents. Especially under high load or harsh environments, the probability of mechanical failure increases. The actual flight training and testing of helicopters requires a lot of fuel, maintenance resources and manpower costs. Every flight involves high costs, especially when conducting complex flight missions or emergency drills. In addition, the loss and failure maintenance of helicopters also increase the economic burden of actual flights.

[0004] Building a helicopter simulation platform can provide a safe, controllable and efficient training environment. By controlling the variables in the environment to simulate the external environment and emergency situations during flight, it can reduce resource consumption caused by helicopter flight and reduce safety hazards. At present, some helicopter control methods use PID controller methods. However, the performance of PID controllers depends on the precise adjustment of control parameters (proportional, integral, and differential gain). These parameters usually need to be manually adjusted through trial and error during the system design phase. Once the system environment changes, the PID controller may need to be readjusted. The PID controller performs well when processing linear systems, but has limited processing capabilities for nonlinear systems. In order to better fit the actual scene, the helicopter simulation platform will perform complex environmental settings, which makes the PID method ineffective. Therefore, our invention adopts a method based on reinforcement learning, which automatically adapts to changes in the environment through continuous interaction and learning, without manual intervention, and does not rely on the linear assumptions of the system. It can handle complex nonlinear relationships and work effectively in a dynamically changing environment, and can adapt to complex external environments.

[0005] Specifically, in the reinforcement learning method, the state space should fully describe the flight state of the helicopter, such as position, speed and attitude, etc.; the action space defines all the operations that the helicopter can perform, such as the adjustment of the pitch lever and pedals. The reward function is the core of reinforcement learning, and it is necessary to balance factors such as flight stability, task completion, energy consumption and safety. Select a suitable reinforcement learning algorithm, combine it with experience replay and target network and other technologies for training, and optimize the strategy to improve its performance in a real environment. Through the above steps, combined with the flight dynamics model and reinforcement learning, multiple iterative optimizations of helicopter flight control can be achieved, and the automatic driving task can be gradually realized under the setting of a complex flight environment, and a helicopter intelligent flight control method based on reinforcement learning can be constructed. At the same time, in order to show the effect of the aircraft controlled by reinforcement learning, the present invention adopts a visualization method to record the position, speed, attitude and other information of the aircraft during the test process, and display the flight trajectory of the helicopter through three-dimensional space coordinates, as well as the change of the attitude of the helicopter during the flight.

[0006] In summary, this paper proposes a helicopter flight control method based on reinforcement learning, which aims to improve the performance of flight control and decision-making systems through adaptive model adjustment and self-supervised auxiliary tasks. Through the reinforcement learning method, the operation and environment are effectively combined, so that the helicopter can achieve flight control and complete the specified task in complex and uncertain situations. Summary of the invention

[0007] The purpose of the present invention is to provide a method for building a helicopter visualization simulation platform based on reinforcement learning.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A method for building a helicopter visualization simulation platform based on reinforcement learning proposed in the present invention comprises the following steps:

[0010] S1, encapsulated helicopter dynamics model;

[0011] S2, constructing an initialization agent by combining reinforcement learning method with helicopter dynamics system;

[0012] S3, train the agent by interacting with the environment through the operation instructions generated by the agent;

[0013] S4. Test the training results of the intelligent agent and explicitly display the flight effect of the test process.

[0014] Preferably, in step S1, the helicopter dynamics model is a flight dynamics model of an AW109 helicopter, which is packaged and constructed by defining initial parameters of the helicopter and combining its dynamic characteristics.

[0015] Preferably, in step S2, the reinforcement learning method adopts an Actor-Critic algorithm.

[0016] Preferably, in step S2, the reinforcement learning method adopts a double-delayed-deterministic policy gradient algorithm (TD3).

[0017] Preferably, in step S2, a reward function is set to drive the agent to self-learn and update, and the reward function is set as follows (index is the current target point sequence number, n1, n2 are failure and success judgments):

[0018] reward position =-|Δx|-|Δy|-|Δz|

[0019]

[0020] Preferably, in step S3, a Replay Buffer mechanism is used to optimize the training process.

[0021] Preferably, step S4 includes flight path visualization and attitude visualization.

[0022] The flight path visualization uses Matplotlib to visualize the three-dimensional space;

[0023] The posture visualization adopts the OpenGL framework.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] In the reinforcement learning simulation platform provided by the present invention, at any time during the flight of the helicopter in the simulation environment, the intelligent agent can generate the operation instruction of the helicopter, and obtain the reward value of the operation instruction by interacting with the flight dynamics model, that is, the environment, and adjust the parameters of the intelligent agent according to the reward value. The intelligent agent can generate reliable operation instructions in real time by repeated attempts, iterating its own strategy, and updating network parameters, and finally the intelligent agent can control the helicopter to complete the specified task. By encapsulating the complex helicopter aerodynamic model into a black box model, in the interaction process, only four-dimensional operation quantities need to be provided, and the output 17-dimensional operation quantities need not be paid attention to, without paying attention to the dynamic details related to the aircraft, so that users can focus more on the development of the control method, incorporate the helicopter aerodynamic model into the environment part of the reinforcement learning, and define the interactive relationship between the intelligent agent and the environment, providing a complete reinforcement learning training process closed loop, the simulation platform can modify the reward function according to different task requirements to control the helicopter to complete a specific task, or modify the environment or flight dynamics model according to the task requirements to simulate various human scenes, with good applicability, and provide two visualization methods for observing the flight path of the helicopter in the test phase and the attitude changes during the flight process, so as to facilitate the evaluation of the control performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A block diagram of the present invention;

[0027] Figure 2 Schematic diagram of the training process of the reinforcement learning model.

[0028] Figure 3 A visual representation of the flight path.

[0029] Figure 4 This is the effect picture of the posture visualization window. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The present invention provides a method for building a helicopter visualization simulation platform based on reinforcement learning, comprising the following steps:

[0032] 1: Encapsulate the helicopter dynamics model;

[0033] 2: Construct a reinforcement learning method combined with the helicopter dynamics system;

[0034] 3: Train the agent by interacting with the environment through the operation instructions generated by the agent;

[0035] 4: Test the training results of the intelligent agent and explicitly display the flight effect of the test process.

[0036] The specific implementation is as follows:

[0037] 1: Encapsulate the flight dynamics model

[0038] An accurate flight dynamics model is the basis for building a simulation platform, which can describe in detail the physical behavior of a helicopter under various flight conditions. The flight dynamics model considered in the present invention includes six degrees of freedom (three spatial position coordinates and three rotation angles), as well as corresponding dynamic equations. The model must cover the aerodynamic characteristics of the helicopter, such as lift, drag, thrust and gravity, as well as the control system of the helicopter, such as rotor speed, attitude control and power distribution. Specifically, the flight dynamics model of the AW109 helicopter is adopted in the present invention. First, the initial parameters of the helicopter are defined, and a complete flight dynamics model is constructed in combination with its dynamic characteristics. Through this model, the flight behavior of the helicopter under various flight environments and conditions can be simulated by inputting operating instructions.

[0039] In this simulation platform, the flight dynamics model is abstracted into a black box model, which receives four-dimensional operation instructions as input: cyclic pitch lever lateral control, cyclic pitch lever longitudinal control, pedals, collective pitch, and outputs seventeen-dimensional helicopter state quantities. By inputting these operation instructions and outputting states, the simulation platform can accurately simulate the dynamic response of the helicopter under different flight conditions, thus providing a reliable tool for the development of flight control systems. The relevant input and output are as follows:

[0040] Flight dynamics model input:

[0041] Serial number Variable Name Technical meaning unit 0 Collective Total distance mm 1 Lon Cyclic pitch lever longitudinal control mm 2 Lat Cyclic pitch lever lateral control mm 3 Pedal Pedals mm

[0042] Flight dynamics model output:

[0043]

[0044]

[0045] In this simulation platform, the helicopter is abstracted into a black box model based on the above input and output, and the latest state of the helicopter after receiving the input control command is calculated through the dynamic formula.

[0046] 2: Initialize the agent

[0047] The agent is an integral part of the reinforcement learning process. The agent learns strategies to maximize rewards or achieve specific goals during the interaction with the environment. Specifically, reinforcement learning is a learning mechanism that learns how to map from state to behavior to maximize the reward. Such an agent needs to continuously experiment in the environment and continuously optimize the state-behavior correspondence through the feedback (reward) given by the environment. The agent controls the helicopter to perform various tasks by generating strategies, i.e., manipulation instructions. The strategy defines the action taken by the agent for a given state. In other words, it is a mapping from state to action, where the state includes the state of the environment and the state of the agent. The strategy is the core of the reinforcement learning system, because the action in each state can be determined by the strategy. The characteristics of the strategy can be summarized as follows: the reinforcement learning system defines the action of the agent by the strategy; the strategy is a mapping from state to action; the strategy itself can be a specific mapping or a random distribution.

[0048] Commonly used classic reinforcement learning algorithms can be divided into policy-based and value-based reinforcement learning algorithms according to the learning objectives. The policy-based method directly outputs the probability of the next action and selects the action based on the probability. However, the action with the highest probability will not necessarily be selected, and the overall consideration will still be made. However, the policy-based algorithm usually converges to the local optimum rather than the global optimum, and there are large differences between each trajectory. There is a high variance problem that leads to training problems and difficulty in policy convergence; the value-based algorithm has the problem that the value function-based method is difficult to converge due to discontinuous changes, and there will be high deviations, because the error between the estimated value function and the actual value function is difficult to eliminate. Taking into account the complexity of helicopter missions and the complexity of the environment, the present invention adopts the actor-critic method, aiming to construct an all-round intelligent agent that can not only directly output strategies, but also evaluate the quality of strategies in real time through value functions. The actor and critic are continuously updated in real time for complementary training. Specifically, Actor-Critic is a method that combines Value-Based and Policy-Based. The actor is the policy function, which is used to learn a strategy to obtain the highest possible reward; the critic is the value function, which is used to estimate the value function of the current strategy, that is, to evaluate the quality of the actor.

[0049] Specifically, the present invention adopts the double delayed-deterministic policy gradient algorithm (TD3), which uses two networks to estimate the action value function, and selects the smaller value as the estimated value during training. When implementing the TD3 algorithm, an Actor network and two Critic networks are first initialized. When calculating the target value, the smaller value between the two is used to estimate the state action value of the next state action pair (s, a), that is:

[0050]

[0051] This avoids overestimation caused by maximization problems. There is a problem with deterministic strategies: it overfits to shrink the peak in the value estimate. When updating the Critic network, the learning objective using a deterministic strategy is extremely susceptible to function approximation errors, resulting in large variance in the target estimate and inaccurate estimates. This induced variance can be reduced by regularization, so TD3 introduces a regularization strategy for deep value learning - target policy smoothing. In actual operation, the expectation of the action can be approximated by adding a small amount of random noise to the target action and averaging it in a small batch. In order to smooth the noise added early, noise is added when selecting the action value in each iteration:

[0052] a~π φ (s)+∈,∈~N(0,σ)

[0053] These noises follow a normal distribution, and the sampled noise is clipped to keep the target close to the original action. After the Critic network is updated, the Actor network adopts a delayed update, generally using the method of Critic update twice and Actor update once, updating the Actor network through gradient ascent and updating the Critic network through soft update.

[0054] 3: Set up the reward function

[0055] The reward function is the key information that drives the self-learning and updating of the agent. The reward defines the goal of the reinforcement learning problem. In each time step, the scalar value sent by the environment to the reinforcement learning is the reward. The reward can define the performance of the agent. The reward function is the main factor affecting the strategy. It can characterize the performance of the agent in a specific step. In the entire reinforcement learning system, the task of the agent is to find the best strategy to maximize the total reward value (return) within a period of time. In the helicopter control flight mission, a higher reward can be given when the aircraft is flying in a normal posture and at an appropriate speed or avoiding obstacles. A negative reward can be given when it loses control, has an abnormal posture, or collides with obstacles during flight, thereby training.

[0056] The goal of reinforcement learning is to maximize the total reward that individuals can obtain from the environment, that is, the goal is not to obtain the maximum reward after a short-term action, but to obtain more rewards in the long term. For example, in the helicopter flight control task, the reward should not only consider whether the aircraft is flying normally at a normal posture and speed at the current moment, but also whether it can maintain a reasonable posture and speed to complete the obstacle avoidance task after a period of time. In many common tasks, such as playing Go, the reward is often zero before a game is over, and only when the game is over will a reward value be generated based on the individual's win or loss; in other tasks, the rewards given by the environment may be distributed in almost every moment. For tasks like playing Go, which have a terminal state and all rewards will be settled in this terminal state and before, they are called episodic tasks; there is another type of task that does not have a terminal state, that is, in principle they can run forever, and the rewards of this type of task are scattered in this continuous series of moments, which is called a continuing task. Since the goal is to maximize the total reward, we want to quantitatively define this total reward, which is called return. For turn-based tasks, the return can be directly defined as:

[0057]

[0058] Where T is the time when the round ends. For continuous tasks, there is no such terminal state. Therefore, such a definition may diverge in continuous tasks, and the entire reward sum tends to infinity. Therefore, another way to calculate the return is introduced, called discounted return:

[0059]

[0060] The specific reward function setting is inseparable from the task requirements. In the simulation platform provided by the present invention, the reward function and the related penalty items can be set according to different action requirements to constrain the flight speed and flight attitude of the helicopter. Taking a simple flight mission to fly to the target point as an example, the action space is specified as the total distance, pedal, lateral periodic pitch change and longitudinal periodic pitch change. The action space value range is [-1,1], and the state space is composed of the state quantities provided by the helicopter sensor. The success judgment condition is that the helicopter successfully reaches n2 meters near the last target point, and the failure judgment condition is that the shortest distance between the helicopter and all target points is greater than the preset distance n1 meters. The reward function is set as follows (index is the serial number of the current target point):

[0061] reward position=-|Δx|-|Δy|-|Δz|

[0062]

[0063] 4: Iterative training

[0064] After defining the agent, environment, and reward function, the logical closed loop of reinforcement learning model training can be realized. The action space is specified to be consistent with the AW109 flight dynamics model, that is, the agent can generate a set of four-dimensional scalars, which represent four operation quantities, i.e., the strategy in the reinforcement learning process. Specifically, the strategy is obtained through the action probability distribution generated by the Actor inside the TD3 agent, and the strategy is used as the agent output to participate in the subsequent process.

[0065] The present invention discretizes the entire training process into different moments. At each moment, the environment and the agent will have corresponding interactions. The actions taken by the agent are applied to the environment. In the present invention, it is directly applied to the helicopter flight dynamics model. After the flight dynamics model interacts with the mission scenario, a new state quantity is output, and the current state of the environment and the reward generated by the previous action are fed back to the agent according to the reward function. The agent updates its own parameters according to the feedback reward, and generates a new strategy with the new state quantity and reward value as input, and then repeats the above process.

[0066] In order to reduce the overhead caused by a large number of search processes and effectively utilize successful training results, the reinforcement learning method adopted by the present invention adopts a Replay Buffer mechanism to optimize the training process.

[0067] Figure 2 It shows the role of Replay Buffer in the reinforcement learning training process. It collects historical state data to help train a more intelligent and adaptable helicopter control system, thereby improving the execution efficiency and safety of the helicopter in various tasks and environments. Replay Buffer is a mechanism used to store experience data collected during the interaction between the agent and the environment in reinforcement learning. By replaying and sampling these historical data, ReplayBuffer can significantly improve the efficiency and stability of training. Replay Buffer is also commonly called experience replay. It is a circular buffer used to store tuples of state-action-reward-next state experienced by the agent in the environment. Specifically, Replay Buffer plays an important role in the following aspects:

[0068] Storing experience: Replay Buffer stores the state, action, reward, and next state experience of the agent in the environment. These experiences are stored in the form of four-tuples, that is, the state-action-reward-next state tuple experienced by the agent in the environment.

[0069] Breaking time correlation: Replay Buffer can break the time correlation between consecutive experiences by randomly sampling these stored tuples for training at each step of training. The experience collected by the agent during its interaction with the environment is time-dependent. If these related experiences are directly used for training, it may lead to overfitting of the model and unstable learning. Replay Buffer breaks this time correlation through random sampling and improves the stability of training.

[0070] Improve sample utilization: Replay Buffer can improve sample utilization. In the traditional online learning process, each experience collected by the agent is used only once and then discarded. With Replay Buffer, the stored experience can be used multiple times for multiple trainings, thereby improving sample utilization. This is especially important for sparse reward environments, because in such environments, it is very difficult to obtain effective experience, and multiple uses of Replay Buffer can effectively alleviate this problem.

[0071] Smooth learning process: Replay Buffer makes the training process smoother by mixing and replaying a large number of experiences collected at different time points, reducing the oscillation and instability that may occur during the training process. This helps improve the training effect and convergence speed of the model.

[0072] 5: Testing and Visualization

[0073] The model trained by the above process can be used for testing. In the test scenario, the present invention provides Matplotlib to visualize the three-dimensional space (XYZ axis in the inertial coordinate system). Matplotlib is a drawing library for Python, which is widely used to generate various static, dynamic and interactive graphics. Its main uses include data visualization, scientific computing and engineering chart drawing. In the present invention, Matplotlib is mainly used to draw the flight path of the helicopter. Specifically, the motion trajectory of the helicopter in space is displayed in three-dimensional space, and different states or time points are represented by changes in color and marks. Through visualization, researchers can observe and evaluate the performance of reinforcement learning algorithms. Flight path visualization such as Figure 3 shown.

[0074] The red color is the expected trajectory, and the blue color is the actual flight trajectory of the helicopter controlled by the agent during the test phase.

[0075] The present invention adopts the OpenGL framework to provide visual information of the flight process of the helicopter model during the test.

[0076] OpenGL (Open Graphics Library) is an industrial standard in the field of graphics. It is a cross-programming language, cross-platform, professional graphics programming (software) interface. It is used for two-dimensional and three-dimensional images. It is a powerful, easy-to-call underlying graphics library. It is hardware-independent and can be ported between different platforms. In the present invention, OpenGL is used for modeling and demonstration of three-dimensional helicopters, showing the attitude maintenance of the helicopter. At the same time, a flight status window similar to a dashboard is provided in the upper left corner, which displays three-axis coordinates, flight speed, Euler angle value, Euler angular velocity and other flight-related status information in real time. The attitude visualization window in the test phase is as follows Figure 4 shown.

[0077] The helicopter simulation platform based on reinforcement learning algorithm can adjust the flight conditions, and the trained intelligent agent can automatically adjust the control strategy to improve the stability and responsiveness of the helicopter. Under this platform, users can train helicopters to perform complex tasks, such as hovering, precise landing, and route tracking, and simulate various mission scenarios through the simulation environment to improve the generalization ability of the algorithm. In addition, various system failures, such as engine failure and sensor failure, can be simulated in the simulation environment to train reinforcement learning algorithms to handle failures, as well as the correct operation strategy of the helicopter under failure conditions, to ensure that the flight mission can be safely completed or an emergency landing can be implemented in the event of a failure. The model trained in the simulation environment reduces resource consumption costs and can be migrated and deployed to physical helicopters in the future to assist helicopter pilots in completing a variety of complex tasks.

[0078] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0079] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for building a helicopter visualization simulation platform based on reinforcement learning, characterized in that: The following steps are involved: S1, encapsulated helicopter dynamics model; S2, constructing an initialization agent by combining reinforcement learning method with helicopter dynamics system; S3, train the agent by interacting with the environment through the operation instructions generated by the agent; S4. Test the training results of the intelligent agent and explicitly display the flight effect of the test process.

2. The method for building a helicopter visualization simulation platform based on reinforcement learning according to claim 1, characterized in that: In step S1, the helicopter dynamics model is a flight dynamics model of the AW109 helicopter, which is encapsulated and constructed by defining initial parameters of the helicopter and combining its dynamic characteristics.

3. The method for building a helicopter visualization simulation platform based on reinforcement learning according to claim 1, characterized in that: In step S2, the reinforcement learning method adopts the Actor-Critic algorithm.

4. The method for building a helicopter visualization simulation platform based on reinforcement learning according to any one of claims 1 or 3, characterized in that: In step S2, the reinforcement learning method adopts a double-delayed-deterministic policy gradient algorithm (TD3).

5. The method for building a helicopter visualization simulation platform based on reinforcement learning according to claim 1, characterized in that: In step S2, a reward function is set to drive the agent to self-learn and update. The reward function is set as follows (index is the current target point number, n1 and n2 are failure and success judgments): reward position =- | Δx | - | Δy | - | Δz| 6. The method for building a helicopter visualization simulation platform based on reinforcement learning according to claim 1, characterized in that: In step S3, the Replay Buffer mechanism is used to optimize the training process.

7. The method for building a helicopter visualization simulation platform based on reinforcement learning according to claim 1, characterized in that: The step S4 includes flight path visualization and attitude visualization. The flight path visualization uses Matplotlib to visualize the three-dimensional space; The posture visualization adopts the OpenGL framework.

Images