A method for training a reinforcement learning agent to provide inputs to an automated system on an aircraft

Abstract

A method for training reinforcement learning agent 103 to provide inputs to automated system 102 on an aircraft. The method comprises steps of: providing a machine learning environment including a simulated aircraft configured to perform a simulated flight; providing, as the reinforcement learning agent, a flight aid agent configured to provide control commands to adjust one or more flight parameters of the simulated aircraft, rewarding the flight aid agent based on a flight aid reward function, wherein the flight aid reward function is configured to provide a reward based on a difference between achieved flight conditions and target flight conditions; and providing, as an adversarial reinforcement learning agent configured to act in opposition to the flight aid agent, an opponent agent configured to manipulate the simulated aircraft.

Description

The present invention relates to a method for training a reinforcement learning agent to provide inputs to an automated system on an aircraft. The invention also relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method. The invention also relates to a system comprising a flight aid agent trained using the method. Automation systems are frequently implemented in aircraft to improve efficiency and safety. These improvements have led to a reduction in the workload of pilots, with the intention of relocating pilot attention to more complex tasks to mitigate accidents. This allowed for a reduction of crew in the cockpit, as well as using automation systems to fly the aircraft for large parts of the flight. With an increase in automation systems, the understanding of all systems, their interconnection, and dependence become more complex. This understanding is essential in abnormal and emergency situations when the level of automation may be degraded (or completely lost) - for example, the autopilot disconnects due to sensor or actuator failure - leading to a reduction of flight envelope protections and leaving the flight crew to deal with the situation alone. In this case, it becomes harder for the flight crew to identify the cause of the problem, determine which systems can still be relied on and which should be disconnected, and take corrective action in a timely manner. In the worst cases, the disconnect between the pilot and the automation can lead to an inability to recognise and recover from complex and dangerous situations. US2020202723 discloses systems and methods for managing the flight of an aircraft. In particular, the method includes the steps of receiving data from recordings of the flight of an aircraft; the data comprising data from sensors and / or data from the on-board avionics; determining the aircraft state at a point N on the basis of the received data; determining the state of the aircraft at the point N+l on the basis of the state of the aircraft at point N by applying a model learnt by means of machine learning. These methods may be used in various applications, including assisting in the piloting of an aircraft. The methods may employ machine learning techniques including reinforcement learning and / or an unsupervised learning algorithm, such as a generative adversarial neural network. Given the existing technology, there remains a problem of providing an automated system with the ability to not merely predict possible future aircraft states in different flight situations, but to determine actions that may be taken to maintain or return to safe and stable aircraft flight in unusual, and possibly unforeseen, flight conditions. In particular, there is a 1 problem of providing an automated system to interface with existing autopilot systems and to work in collaboration with pilots to overcome dangerous flight situations, for example by suggesting appropriate pilot actions. According to an aspect of the invention, there is provided a method for training a reinforcement learning agent to provide inputs to an automated system on an aircraft. The method comprises steps of: providing a machine learning environment including a simulated aircraft configured to perform a simulated flight; providing, as the reinforcement learning agent, a flight aid agent configured to provide control commands to adjust one or more flight parameters of the simulated aircraft; and rewarding the flight aid agent based on a flight aid reward function. The flight aid reward function is configured to provide a reward based on a difference between achieved flight conditions and target flight conditions. The method further comprises a step of providing, as an adversarial reinforcement learning agent configured to act in opposition to the flight aid agent, an opponent agent configured to manipulate the simulated aircraft. This arrangement may teach the flight aid agent to become robust and adaptable to new and unexpected flight situations. In particular, the flight aid agent may be improved by striving to achieve target flight conditions despite the opponent agent acting in opposition to the flight aid agent. In an embodiment, the flight aid agent is configured to provide control commands to adjust flight parameters of the simulated aircraft. The flight parameters optionally include one or more flight controls of the simulated aircraft. The flight controls optionally include one or more flight control surfaces. The flight controls optionally include one or more of: thrust, elevator, rudder, ailerons, and flaps. In this way, the flight aid agent may be given control over the aircraft in order to determine how the aircraft reacts to different control commands. As such, the flight aid agent may learn how best to control the aircraft in order to safely endure or overcome different flight scenarios. In an embodiment, the simulated aircraft comprises a plurality of flight condition sensors configured to measure flight condition data during the simulated flight. During the simulated flight, a state space of the flight aid agent may comprise data gathered by the plurality of flight condition sensors. Optionally, during the simulated flight, a state space of the opponent agent may comprise: data gathered by the plurality of flight condition sensors, and actual flight conditions of the simulated aircraft during the simulated flight. Optionally, the opponent agent may be configured to manipulate the simulated aircraft by controlling factors affecting the accuracy of one or more of the plurality of flight condition sensors. In this way, the opponent agent may be able to replicate realistic aircraft malfunctions such that 2 the flight aid agent has an opportunity to learn how to identify and / or overcome these malfunctions. In an embodiment, during the simulated flight, a state space of the opponent agent comprises: the control commands provided by the flight aid agent to adjust the one or more flight parameters of the simulated aircraft, and an actual control provided to the simulated aircraft during the simulated flight. Optionally, the opponent agent may be configured to manipulate the simulated aircraft by controlling factors affecting the response of the simulated aircraft to control commands provided by the flight aid agent. For example, the opponent agent may be configured to manipulate the simulated aircraft by controlling factors affecting the response of the flight parameters to control commands provided by the flight aid agent. In this way, the opponent may be provided with an amount of flight data and control to enable the opponent agent to act as an effective adversary to the flight aid agent. In this way, the flight aid agent may be forced to improve further in order to overcome the opponent agent. In an embodiment, the target flight conditions include one or more of the following flight conditions: true airspeed, angle of attack, heading, altitude, absolute vertical acceleration and absolute vertical speed of the simulated aircraft. In an embodiment, the flight aid reward function is configured to provide a lower reward if flight conditions exceed an upper threshold, than if the flight conditions do not exceed the upper threshold, for at least one of the following flight conditions: absolute vertical acceleration and absolute vertical speed of the simulated aircraft. In this way, the flight aid agent may be motivated to avoid steep dives and climbs, both of which can be dangerous to the aircraft. In particular, the flight aid agent may beneficially reduce self-induced oscillations and erratic usage of the controls. In an embodiment, the flight aid agent is configured to determine the control command and a clipped control command at each time step of the simulated flight. If a magnitude of the control command is below an upper limit of the flight control of the simulated aircraft, a magnitude of the clipped control command is equal to the magnitude of the control command applied at the time step. If the magnitude of the control command exceeds the upper limit of the flight control of the flight control of the simulated aircraft, the magnitude of the clipped control command corresponds to the exceeded upper limit. The flight aid reward function may be configured to provide the flight aid agent with a greater reward the smaller the difference between the magnitude of control command and the magnitude of clipped control command. In this way, the flight aid agent may be trained to operate each flight control within an operation range representative of practical limitations on the flight controls of the corresponding real aircraft. As such, when the trained flight aid agent is used in a real aircraft, the control commands suggested by the flight aid agent may be realistically applicable instead of merely theoretical. In an embodiment, the flight aid reward function is normalised. In an embodiment, the method further comprises rewarding the opponent agent based on an opponent reward function. Optionally, the opponent reward function may be configured to provide a reward based on a difference between the achieved flight conditions and the target flight conditions. The opponent agent may receive a greater reward the greater the difference between the achieved flight conditions and the target flight conditions. In this way, the opponent agent may be motivated to act as an effective adversary to the flight aid agent and to adapt to improvements in the flight aid agent during training. As such, the opponent agent may desirably challenge the flight aid agent in a way that forces improvement of the flight aid agent. For example, the flight aid agent may be subjected to more new and unanticipated malfunctions. This may prompt the flight aid agent to explore the machine learning space and find solutions to different flight scenarios. In an embodiment, the simulated flight is configured to be terminated if the flight conditions reach or exceed a termination condition. The termination condition may be a maximum deviation from one or more of the target flight conditions. Optionally, in response to a termination condition being reached the flight aid agent may receive a reduced reward or a negative reward and / or the opponent agent may receive a positive reward. In this way, the flight aid agent may more quickly learn to avoid the undesirable, and possibly critically dangerous, flight conditions defined as the termination conditions. The flight aid agent may therefore develop skills by performing more simulated flights while remaining within acceptable flight conditions, which are more applicable to practical flight scenarios in real life. In particular, this may desirably train the flight aid agent not to suggest control commands / actions / instructions which may further endanger the aircraft. As such, the usefulness of the flight aid agent in improving flight safety may be improved. In an embodiment, the training is repeated for a plurality of simulated flights. Each of the simulated flights may be considered and referred to as a training episode. Each of the plurality of simulated flights is initiated with the simulated aircraft in initial simulated flight conditions. The initial simulated flight conditions vary between different simulated flights. Optionally, the initial flight conditions configured to be varied between different simulated flights may include one or more of: angle of attack, true airspeed and altitude of the simulated aircraft. In an embodiment, each of the plurality of simulated flights is initiated with target flight conditions. The target flight conditions vary between different simulated flights. In this way, the flight aid agent may be trained to respond appropriately in different flight conditions, 4 and may desirably learn to respond appropriately even when a new, unexpected and potentially dangerous flight scenario is encountered. Furthermore, the flight aid agent may desirably leam to respond to changing objectives as a situation develops. In an embodiment, a proximal policy optimization algorithm is used to train the flight aid agent and / or the opponent agent. According to an aspect of the invention, there is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of the preceding aspects. According to an aspect of the invention, there is provided a system comprising a flight aid agent trained using the method of any of the preceding aspects. The system is configured to provide inputs to an automated system on an aircraft based on control commands from the flight aid agent. The automated system may be an autopilot. Alternatively, or additionally, the automated system may include other systems, such as a flight management system and / or a flight control system. The system is configured to provide audible and / or visual information to a pilot of the aircraft. This arrangement may mitigate errors caused by a disconnect that may occur between pilot and existing automation systems of aircraft. In particularly, this may reduce the likelihood of the aircraft crashing in a critical situation, such as a potential stall and / or a situation in which the aircraft has suffered a malfunction. The present invention will now be described with reference to exemplary embodiments and the accompanying figures, in which: Figure 1A-B illustrates example scenarios of interaction between cockpit resources, including pilot(s) interacting with existing automated systems and an additional system. Figure 2A illustrates several different flight parameters plotted against the episode step for a simulated training experiment in which a malfunction is introduced during a simulated flight. Figure 2B illustrates the zero-shot performance of the flight aid agent in simulations where a malfunction was introduced. Figure 2C illustrates an increase of reward following further training of the flight aid agent compared to the rewards achieved when malfunctions were initially introduced. Figure 3A-B illustrates learning curves for different flight aid agents with and without a pitot static system blockage type malfunction. Figure 4A-B illustrates convergence of the flight aid agent when a positive reward is given for not exceeding upper limits of vertical acceleration and vertical speed. Figure 5A-D illustrates the opponent goal reward against the number of training iterations for different malfunction-reward combinations. Figure 6 illustrates difference between the achieved and target altitude for flight aid agents during simulated flights performed with different termination strategies. Automation systems, such as autopilot systems, implemented in aircraft may be used to perform flight tasks and / or to alert the pilot of changes in the aircraft state or flight conditions. The increased use of automated systems has in some instances enabled a reduction of crew in the cockpit. Furthermore, in some instances, the automated systems are capable of managing and controlling the flight of the aircraft over flight regimes which may make up a substantial amount of total flying time. With an increase in automation systems, it can be more difficult for pilots to understand all systems, how the systems interact with or depend on each other and how these systems interact with the aircraft sensors and controls. A good understanding of these systems and their interaction can be a significant factor in the likelihood of successfully evading critical situations, which may be dangerous situations or situations in which the aircraft exceeds predetermined flight conditions. In some such situations, for example, the level of automation may be degraded (or completely lost) - for example, the autopilot disconnects due to sensor or actuator failure -leading to a reduction of flight envelope protections and leaving the flight crew to deal with the situation alone. In this case, it becomes harder for the flight crew to identify the cause of the problem, determine which systems can still be relied on and which should be disconnected / disengaged, and take corrective action in a timely manner. This can be a particularly demanding task in time-critical situations in which pilots may be preoccupied with the tasks of navigating towards safety and communication with each other and / or air traffic control about the situation. If the pilot is not able to determine the correct action to take in regards to their interaction with the automated systems of the aircraft, or if the pilot makes the wrong decision due to an incomplete understanding of the complicated system interactions, it is possible that the aircraft may not recover from the situation. [Additional system] An additional system may be provided with an aim of reducing the disconnection between the pilot(s) and automation systems in the aircraft cockpit. The additional system is configured to provide inputs to one or more automated systems on an aircraft. The one or more automated systems is preferably one or more existing automation systems, preferably the automated system is an autopilot system. Alternatively, or additionally, the automated system may include a flight management system and / or a flight control system. The additional system may be particularly configured for use in large transport category aircraft. The additional system is preferably a supplement to, rather than an adjustment of, existing automation systems. In this way, automation complexity in the aircraft may not be increased, as this could compound the existent human-machine interaction challenge. Instead, the level of existing automation is maintained at the same level and technology, and an additional new system is introduced. Figure 1 provides example illustrations outlining the resource structure of a cockpit comprising the additional system 103 in addition to pilot(s) 101 and the existing aircraft automation system 102. The additional system may be configured to act in parallel to the pilot(s), for example as illustrated in Figure 1 A, and / or the additional system 103 may be configured to as an intermediary between the pilot(s) 101 and the existing automation systems 102, for example as illustrated in Figure IB. This additional system 103 may be configured to act as if it were an additional crew member in the cockpit. In other words, the additional system may be configured to communicate audibly, preferably verbally, with the pilot(s) in a similar manner to the communication between multiple human pilots in an aircraft cockpit. The additional system is configured to handle, or control, the existing aircraft automation systems in addition to, or instead of, the pilot controlling the existing aircraft automation systems. In this way, the additional system may aid in managing the existing automated systems to reduce the likelihood of incorrect pilot action due to a lack of pilot understanding of the existing automated systems and how the existing automated systems interact with each other and / or react in unexpected, critical situations. Artificial intelligence and / or machine learning techniques may be applied to train the additional system. At least some, if not all, training of the additional system desirably takes place prior to installation in a flight-ready aircraft. The intelligent additional system may be better suited than human pilots to handle the existing automation systems, particularly during emergency situations likely to cause the human pilots stress and distraction. In this way, the additional system desirably may mitigate errors caused by a disconnect that may occur between pilot and existing automation systems of aircraft. The additional system is configured to provide audible and / or visual information to a pilot of the aircraft. The additional system is preferably configured to communicate with the pilot(s) by a voice communication interface and / or a touch screen communication interface, for example, as disclosed in US9710145B2. The additional system may be configured to select radio frequencies. In this way the additional system may further reduce human pilot workload, reduce the number of tasks to be performed directly by the human pilot during critical situations, and facilitate pilot communications. [Flight aid agent training - introduction] The additional system is preferably configured to provide inputs to an automated system on an aircraft based on commands from a reinforcement learning agent. The reinforcement learning agent is trained using reinforcement learning techniques. Instructions which, when executed by a computer, cause the computer to carry out the training may be provided on a computer-readable medium. The reinforcement learning agent is trained to provide inputs to an automated system, for example an autopilot, on an aircraft. The reinforcement learning agent is trained in a machine learning environment. The machine learning environment includes a simulated aircraft. The simulated aircraft is configured to perform a simulated flight. In this way, the reinforcement learning agent may be subjected to simulated flights including off-nominal flight conditions, such as flight outside of an acceptable flight envelope of the aircraft or flight during a critical situation. The reinforcement learning agent may learn to react appropriately to unexpected, and unanticipated, situations without risk to a real aircraft in a real flight scenario through use of the machine learning environment. In particular, the use of the machine learning environment may enable the reinforcement learning agent to be trained using a plurality of simulated flights. It may therefore be possible for the reinforcement learning agent to gain experience based on a large number of simulated flights. In particular, it is possible that the number of simulated flights used to train the reinforcement learning agent may exceed the number of training flights performed by human pilots, for example using simulators. In this way, once trained and implemented in an additional system on an aircraft, the reinforcement learning agent may be more capable of reacting appropriately to unexpected, off-nominal, flight conditions than the human pilots of the aircraft. The reinforcement learning agent is a flight aid agent. The flight aid agent is configured to provide control commands to adjust one or more flight parameters of a simulated aircraft. The flight parameters optionally include one or more flight controls of the simulated aircraft. The flight controls optionally include one or more flight control surfaces. The flight controls optionally include one or more of: thrust, elevator, rudder, ailerons, and flaps. In this way, the flight aid agent may learn the amount and type of response of the aircraft to different changes in flight parameters during different flight scenarios. In particular, the flight aid agent may learn to provide appropriate control to the aircraft during different flight scenarios. The flight parameters of the simulated aircraft may, additionally or alternatively, include control systems, buttons and switches. In other words, the flight aid agent may be configured to control any input mechanism available to the pilot(s) in the cockpit. The flight aid agent is rewarded based on a flight aid reward function. The flight aid reward function is configured to provide a reward based on a difference between achieved flight conditions and target flight conditions. The target flight conditions may be one or more flight conditions associated with safe and / or stable flight. The target flight conditions may, for example, include a target altitude and / or a target rate of change of altitude. The greater the deviation between the achieved flight conditions, achieved during or by the end of the simulated flight, and the target flight conditions, the lower the reward provided to the flight aid agent. In this way, the flight aid agent may be penalised by a low reward if the achieved flight conditions deviate greatly from the target flight conditions. The target flight conditions may optionally be changed during a simulated flight. In this way, the flight aid agent may learn to adapt to changing goals with different initial conditions, which in this case are the inflight conditions at the timepoint of the change of target flight conditions. With this approach, the flight aid agent has a reward-based incentive to achieve, or come close to achieving, the target flight conditions. The flight aid agent may therefore learn how to adjust flight parameters in order to get a higher reward by achieving, or getting within an acceptable range of, the target flight conditions. An adversarial reinforcement learning agent is also provided in the machine learning environment. The adversarial reinforcement learning agent is an opponent agent. The opponent agent is configured to act in opposition to the flight aid agent. In this way, the flight aid agent may be further improved by striving to achieve target flight conditions despite the opponent agent acting in opposition to the flight aid agent. [Architecture] The simulated aircraft may comprise a plurality of (virtual) flight condition sensors configured to measure flight condition data during the simulated flight. During the simulated flight a state space of the flight aid agent preferably comprises data gathered by the plurality of flight condition sensors. In this way, the flight aid agent may be provided with information representative of flight sensor data. The sensor data would typically be available during real flights to the pilot(s) and existing automation systems. It is therefore appropriate for the sensor data to be available to the flight aid agent during training, such that the flight aid agent might desirably learn to use this sensor data to gain information about the situation and what actions may be required to achieve the target flight conditions. Alternatively, the flight aid agent may be provided with actual flight conditions of the simulated aircraft during the simulated flight. In other words, flight conditions of the simulated aircraft within the machine learning environment, not as measured by (virtual) flight condition sensors. This may be useful, particularly in early stages of training the flight aid agent, for the flight aid agent to learn the actual response of the simulated aircraft to different conditions and input control commands. However, providing this information by means of measurement via (virtual) flight condition sensors has the additional benefit that the flight aid agent may learn to account for the real-life issues associated with sensor data, such as any inaccuracies (e.g., due to sensor noise, bias etc.) or malfunctions of the sensors. During the simulated flight a state space of the opponent agent may comprise data gathered by the plurality of flight condition sensors and / or actual flight conditions of the simulated aircraft during the simulated flight. The state space of the opponent agent preferably comprises both the data gathered by the plurality of flight condition sensors and actual flight conditions of the simulated aircraft during the simulated flight. In this way, the opponent agent may be provided with information representative of flight sensor data as well as actual flight conditions of the simulated aircraft. In this way, the opponent agent would have information regarding the accuracy of the data from the flight sensors. If the flight aid agent is provided with flight sensor data and not actual flight condition data, then the opponent agent has more information about the simulated aircraft during the simulated flight than the flight aid agent. In this way, the flight aid agent is put at a disadvantage compared to the opponent agent such that the flight aid agent may be better trained such that it may overcome disadvantages. During simulated flight, the state space of the opponent agent optionally comprises the control commands provided by the flight aid agent to control the one or more flight parameters of the simulated aircraft, or the actual control provided to the simulated aircraft during the simulated flight. The state space of the opponent agent preferably comprises both the control commands provided by the flight aid agent to the one or more flight parameters of the simulated aircraft, and the actual control provided to the simulated aircraft during the simulated flight. In this way, the opponent agent may be provided with information representative of flight control commands as well as actual flight control responses of the simulated aircraft. As such, the opponent agent may have information regarding whether the flight controls are operating as instructed. If the flight aid agent is provided the flight control commands, and not actual flight control responses, then the opponent agent has more information about the simulated aircraft during the simulated flight than the flight aid agent. In this way, the flight aid agent is put at a disadvantage compared to the opponent agent such that the flight aid agent may be better trained such that it may overcome disadvantages. The opponent agent may be configured to manipulate the simulated aircraft. For example, the opponent agent may be configured to manipulate systems and / or sub-systems of the simulated aircraft, for example by introducing malfunctions in the systems and / or subsystems. The opponent agent is preferably configured to manipulate the simulated aircraft to alter the flight conditions. In particular, the opponent agent may be configured to alter the flight conditions such that the achieved flight conditions deviate from the target flight conditions. In this way the opponent agent may act in opposition to the flight aid agent which may have a goal of maximising a reward by achieving the target flight conditions. The opponent agent is preferably configured to manipulate the simulated aircraft to control factors affecting the response of the flight parameters to control commands provided by the flight aid agent. In other words, the opponent agent may be configured to manipulate control factors affecting the flight controls of the simulated aircraft such that the simulated aircraft does not perform the control as instructed by the control commands provided by the flight aid agent. In particular, the opponent agent may be configured to replicate a malfunction of one or more flight controls such that the actual response of the simulated aircraft does not match the control command applied. For example, a control command to the rudder may be applied by the flight aid agent, and the opponent agent may have influenced the rudder control such that the rudder does not respond as instructed. In this example, the rudder may not respond at all, or may deflect by a greater or lesser magnitude than instructed, and / or may deflect in the opposite direction to the instructed direction. In this way, the opponent agent may inhibit the ability of the flight aid agent to achieve the target flight conditions. This may desirably prompt the flight aid agent to find alternative solutions to achieving the target flight conditions in spite of the control issues. This may have real-life benefits when the trained flight aid agent is used in an aircraft, for example in an additional system. In particular, the flight aid agent may be trained to identify and / or take appropriate actions to regain / maintain safe flight when there is a malfunction in the flight control systems. Additionally, or alternatively, the opponent agent may be configured to manipulate the simulated aircraft by controlling factors affecting the accuracy of one or more of the plurality of flight condition sensors. In this way, the information provided to the flight aid agent may be incorrect. For example, the opponent agent may manipulate a virtual airspeed sensor of the simulated aircraft such that airspeed measured by a virtual airspeed sensor, and provided to the flight aid agent, is incorrect. The training may replicate the scenario where there is a sensor malfunction. The flight aid agent may therefore be trained to identify a sensor malfunction and / or to take appropriate actions in order to achieve target flight conditions despite a sensor malfunction. The abilities and usefulness of the flight aid agent may therefore be enhanced by learning to overcome this type of interference by the opponent agent. It is noted that, in initial stages of training the flight aid agent, the failures may be introduced without control of the failures being provided by the opponent agent. The opponent agent may then be introduced at a later stage in training of the flight aid agent. This may enable the flight aid agent to begin learning to overcome failures / malfunctions before further learning in the more challenging situation where the opponent agent has control over the failures / malfunctions and is acting in opposition to the flight aid agent. [Example simulated training experiment - introduction of malfunctions] In an example simulated training experiment, malfunctions of different categories have been introduced to the simulated aircraft. The malfunctions that have been considered can be grouped into actual malfunctions and pitot static system malfunctions. The pitot static system of an aircraft is a pressure-sensitive instrument used to determine airspeed and possibly also altitude and / or vertical speed. The pitot static system therefore does not directly influence the flight conditions of the aircraft, but instead can lead the aircraft pilot(s) and systems to be provided with conflicting and / or incorrect airspeed data. The “actual” malfunctions are factors which directly affect the aircraft in different ways. In other words, unlike a malfunction of the pitot static system, the actual malfunctions may directly influence the flight conditions of the simulated aircraft. The actual malfunctions considered are: moving the centre of gravity position of the simulated aircraft, executing commands only partially (in other words, actuating a flight control surface such that it moves by a smaller magnitude than stated in the control command), and single or double engine failure. The moving centre of gravity is simulated by changing the position of two fuel tanks in the simulated aircraft. The pitot static malfunctions are blockages of the pitot tube, drain and / or static port. In the simulated training experiment, each simulated flight is performed over a plurality of episode steps, for a duration of over 1,000 episode steps. In each simulated flight, the malfunction is introduced at the 600th episode step. Example results of the simulated training experiment are provided in the graphs of Figure 2A. Figure 2A shows several different flight parameters plotted on the respective Y-axes against the episode step on 12 the X-axes. Each line on each of the graphs of Figure 2A represents a different type of actual malfunction, including change of centre of gravity, engine failure, and an actuator (for a flight control surface) being stuck. The flight parameters shown in Figure 2A for episode steps 0 to 600 correspond to the state of the simulated aircraft with no malfunction present. At episode step 600 the malfunction is introduced, and the resulting effect of the malfunction on the simulated aircraft is seen in episode steps 600 onwards. As can be seen in Figure 2A, the engine failure and change of centre of gravity appear to have a smaller effect on flight conditions than the stuck actuator. In the simulated training experiment, the flight aid agent has been previously trained in the machine learning environment to perform simulated flights without any malfunctions being introduced. To assess the ability of the flight aid agent to identify and / or adapt to a malfunction, the performance (sometimes referred to as zero-shot performance) of the flight aid agent when the malfunctions are directly applied is analysed. Figure 2B shows the zeroshot performance of the flight aid agent by giving, on the Y-axis, the reward received by the flight aid agent in simulations where a malfunction was introduced as a percentage of the reward received by the flight aid agent in corresponding simulations in which no malfunction was introduced. This has been repeated for a variety of different malfunction types as indicated on the X-axis of Figure 2B. In this way, the influence of each malfunction type on the flight aid agent’s ability to achieve the reward can be assessed. Furthermore, for each malfunction type, the simulations have been repeated with the artificial aircraft at different initial angle of attack, in a low range of less than 5 degrees, a mid-sized range of between 5 and 10 degrees, and a high range of greater than 10 degrees. Except for the moving centre of gravity, the malfunctions cause the overall reward achieved by the flight aid agent to decrease by at least 25% compared to the reward achieved in the corresponding no-malfunction simulations. The stuck actuator malfunction type generally has the largest influence on reward received by the flight aid agent. For the full pitot static blockage type malfunction, the angle of attack range associated with the lowest rewards is the low range. Moving the centre of gravity has the smallest effect on the reward achieved by the flight aid agent, with the reward even increasing with angles of attack in the high range. Nevertheless, the flight aid agent was not able to achieve the target, safe, flight conditions in the majority of the simulated flights used to generate the results of Figure 2B, and the majority of the malfunctions did negatively impact the ability of the flight aid agent to achieve rewards. The flight aid agent was then subjected to further training in an attempt to improve the ability of the flight aid agent to achieve rewards when these particular malfunctions are introduced. This further training has been performed by a continuation of training for only 100 iterations. In other words, the flight aid agent was trained on 100 simulated flights in which the corresponding malfunction was introduced. The results for the flight aid agents having undergone this further training are summarized in Figure 2C. In particular, Figure 2C shows the increase of reward as percentage compared to the rewards achieved when malfunctions were initially introduced (as seen in Figure 2B). For all failure types, the performance of the flight aid agent improved, although not for all angles of attack. For the majority of considered scenarios, there was an increase in reward of more than 50%. The largest increase in reward was for the pitot tube blockage type malfunction, where the percentage increase for mid-sized angles of attack increased by almost 200%. These results show that the performance can be improved significantly with only a small amount of training iterations, and thus the flight aid agent can be trained to address specific malfunctions by targeted further training on these malfunctions. This demonstrated that the flight aid agent can learn a strategy against these individual malfunctions when applied separately. The above-described experiments, used to obtain the results of Figure 2A-C have been generated without the use of an opponent agent. An opponent agent can be trained to try to defeat the flight aid agent, or an opponent agent can be trained with the goal to create the best flight aid agent. During training, the opponent agent and flight aid agent may be configured to compete. The flight aid agent may desirably adapt to overcome the opponent agent and vice versa. Further simulated training experiments have been conducted to investigate the effect that the opponent agent may have on the flight aid agent. The further simulated training experiments have been conducted by using a flight aid agent trained using an opponent agent. The flight aid agent is then subjected to new scenarios to investigate how well the flight aid agent has learned to adapt to new situations. In particular, the results shown in Figure 3 have been generated using a flight aid agent (labelled in Figure 3 as ‘Angel - Gremlin’) trained with an opponent agent having a reward which is the opposite of the flight aid agent reward, in addition to showing results based on different, comparison flight aid agents. The comparison flight aid agents have been created by training the flight aid agent without any opponent agent, and by training the flight aid agent to deal with the pitot static system malfunctions. The two comparison flight aid agents of Figure 3 trained without any opponent agent are: • A flight aid agent (labelled in Figure 3 as ‘Angel’) trained without an opponent agent; and • A flight aid agent (labelled in Figure 3 as ‘Angel - random’) trained without an opponent agent but instead with uniform randomly sampled blockages being introduced during training simulation flights. The flight aid agent trained with random malfunctions can then serves as a baseline to identify any advantages of training using an opponent agent as opposed to training with uniform sampled blockages. During the above-described training experiments (used to generate the results shown in Figure 2A-C), the lowest altitude used as an initial flight condition was 15,000 ft (approx. 4572 m). The further experiments (used to generate the results shown in Figure 3 A-B) have been conducted. In these further experiments, the flight aid agent is tested with lower starting altitudes; between 1,500 (approx. 457 m) and 15,000 ft (approx. 4572 m). The flight aid agents used to generate the results of Figure 3 A-B have therefore been subjected to new initial conditions, different to those it has previously trained. Figure 3 A and 3B show the learning curves from training by plotting the episode length on the Y-axis against the number of iterations on the X-axis for each flight aid agent during comparable simulated flights. Figure 3 A shows the results for low altitude flights without any simulated malfunction in the pitot static system, whereas Figure 3B shows the results for low altitude flights in which the pitot static system has a blockage malfunction. Each episode corresponds to a simulated flight, and the simulated flight is configured to be terminated if the flight conditions deviate by more than a predetermined amount from the target flight conditions. Thus, if the flight aid agent is not able to maintain safe flight conditions, the episode length will be short, whereas the longer safe flight is maintained the longer the episode length, up to a maximum of 1200 episode steps for a fully completed episode / simulated flight. As shown in Figure 3 A, within relatively few iterations, all three of the flight aid agents are able to learn to complete the full 1200 episode steps of safe flight at low altitude when no malfunction is introduced. Figure 3B shows the same graph but for simulations in which a fully blocked pitot tube malfunction type has been introduced. The shorter episode lengths of Figure 3B show that for the flight aid agent trained with uniform randomly sampled blockages, flying at low altitude becomes more difficult. The flight aid agent trained with random blockage malfunctions eventually is able to finish its episodes by reaching 1200 episode steps. The flight aid agent trained by an opponent agent does not show this decrease in performance during training. This analysis shows that the flight aid agent learns to fly at low altitude by interacting with other parts of the simulated environment, and when combined with an opponent agent can achieve a similar performance while experiencing pitot blockage malfunctions. [Reward function] The target flight conditions may include any condition associated with the simulated aircraft. The target flight conditions preferably include one or more of the following flight conditions: true airspeed, angle of attack, heading, altitude, absolute vertical acceleration and absolute vertical speed of the simulated aircraft. [Control clipping] The flight aid agent is optionally configured to determine the control command(s) at each time step of the simulated flight. Furthermore, the flight aid agent may also determine a clipped control command at each time step of the simulated flight. If a magnitude of the control command is below an upper limit of the flight control of the simulated aircraft, the magnitude of the clipped control command is equal to the magnitude of the control command applied at the time step. If the magnitude of the control command exceeds the upper limit of the flight control of the simulated aircraft, the magnitude of the clipped control command may be adapted to correspond to the exceeded upper limit. In this way, the simulation may force the simulated aircraft to operate within the operational limits of each aircraft control mechanism. The flight aid reward function may optionally be configured to provide the flight aid agent with a greater reward the smaller the difference between the magnitude of control command and the magnitude of clipped control command. In other words, if the control commands issued by the flight aid agent do not exceed an upper limit during a simulated flight, the flight aid agent may be positively rewarded. Alternatively, or additionally, if the control commands issued by the flight aid agent exceed an upper limit during a simulated flight, the flight aid agent may be negatively rewarded. In other words, the flight aid agent may receive a negative reward or a reduced magnitude of positive reward as a result of the control commands exceeding an upper limit In this way, the flight aid agent may desirably be trained to provide control commands falling within allowable limits. In other words, the flight aid agent may be provided with a positive reward if the flight aid agent provides control commands associated with control actions staying within a predetermined action range, which is preferably a range within which the associated control mechanism is capable of operating safely. The predefined action range may be based on operational recommendations and / or limits (e.g., physical limits) of the corresponding simulated aircraft controls, for example flight control surfaces, actuators, etc. For example, the maximum thrust of the engines and / or maximum possible or maximum recommended deflection of the elevator may be included in the predetermined limits defining the action range. [Reduced rewardfor high vertical speed / acceleration] The flight aid reward function is preferably configured to reduce the reward provided to the flight aid agent if the flight conditions exceed an upper threshold for at least one flight condition compared to the reward given if the upper threshold had not been exceeded. In other words, the flight aid agent may receive zero reward or a reduced magnitude of reward as a consequence of the control commands exceeding an upper limit. In particular, the flight aid reward function may be configured to provide the flight aid agent with a lower reward for achievement of flight conditions exceeding an upper threshold for at least one flight condition of the following flight conditions: absolute vertical acceleration and absolute vertical speed of the simulated aircraft. The lower reward may be equal to 0, whereas the corresponding higher reward, if the upper threshold had not been exceeded, may have been equal to 1. In many, if not most, flight scenarios, it is desirable that the aircraft does not ascend or descend too rapidly. Such rapid ascent and descent may be indicative of unstable flight or an uncontrolled dive. As such, it is desirable to train the flight aid agent to avoid rapid descent and / or ascent scenarios. One approach is to provide upper thresholds values for flight conditions such as absolute vertical acceleration and / or absolute vertical speed and to reduce the reward to the flight aid agent for exceeding the one or more upper thresholds. Further simulated training experiments have been conducted using the reward function which discourages high vertical acceleration and high vertical speed. This is intended to encourage the flight aid agent to meet operational limits and safety constraints, without limiting exploration of the simulation environment by the flight aid agent. This may have the further benefit of discouraging erratic and oscillating behavior as well as large changes in actions, which are undesirable. The results of these further simulated training experiments are shown in Figure 4A and 4B. Figure 4A shows the mean of the occurrences of high vertical speed and acceleration plotted on the Y-axis against the number of iterations on the X-axis. The flight aid agent is able to reduce the number of occurrences exceeding the target maximums early on in training after relatively few iterations. Figure 4B shows the convergence of the actions to an action range corresponding to the real-life range of control for each control parameter of the simulated aircraft. For example, the aircraft elevator may have an absolute limit on its range of motion defined by physical constraints of the aircraft, and / or may have a recommended operating range beyond which flight can become erratic due to sudden movements. Figure 4B shows a decrease of mean difference to the action ranges after relatively few iterations. This is indicative that the maximum vertical speed and maximum vertical acceleration based rewards may be sufficient for teaching the flight aid agent to maintain flight within the action range. [Action loss and additional prediction] The action loss is the mean squared error of the Euclidean distance between actions at different points in time. In the machine learning environment, sigmoid with further shifting and scaling is used to ensure action bounds. With this, reducing the action usage may become part of the optimization objective and allow the flight aid agent to focus only on the targets in the reward function, leading to better convergence. The action loss may beneficially reduce the overuse, erratic behaviour and self-induced oscillations of the control commands. The action loss may be provided as an alternative or in addition to the other rewards for the flight aid agent. For example, action loss may be implemented in addition to clipping of the magnitude of control commands and / or the flight aid reward being based at least partly on clipping, and / or the flight aid reward being at least partly based on vertical speed and / or vertical acceleration. Alternatively, or additionally to the action loss, an additional prediction head may be included in the neural network of the flight aid agent. The additional prediction head is preferably configured to estimate the current state of the simulated aircraft. This auxiliary task implementation affects the weights of the same neural network that also provides the action prediction of the flight aid agent. This is implemented as an auxiliary task as this estimation is not only related but also useful for the action prediction of the flight aid agent. This state estimation may desirably allow fast adaptation to malfunctions, particularly malfunctions in flight condition sensors. Additionally, this state estimation acts as an alerting system for the pilots, which increases situational awareness by providing information about possible differences between the current state of the aircraft as estimated by the flight aid agent, and what is shown on flight instruments, which may include incorrect / corrupted information based on malfunctioning sensors. The total objective function of the actor neural network of the flight aid agent is preferably a multi-objective loss function with a weighted combination of policy loss, the action loss, and auxiliary task loss. Where policy loss is loss from the optimisation algorithm, which is preferably a proximal policy optimisation algorithm. The auxiliary loss is loss from the 'reconstruction of the state' implementation. 18 [Opponent agent rewards] In addition to the flight aid agent being rewarded, the opponent agent is optionally also rewarded. The opponent agent may be rewarded based on an opponent reward function. The opponent reward function may be configured to provide a reward based on a difference between the achieved flight conditions and the target flight conditions. The opponent agent may receive a greater reward the greater the difference between the achieved flight conditions and the target flight conditions. In other words, the opponent agent may achieve a reward if the flight aid agent has not achieved its objectives. The greater the deviation between the achieved flight conditions and the target flight conditions, the greater the reward to the opponent agent. In this way, the opponent agent may be incentivised to hinder the flight aid agent from achieving its objectives. The opponent agent may learn and become more effective. As such, the flight aid agent is forced to also learn more in order to overcome these hindrances. The reward function of the flight aid agent may be goal oriented. Similarly, the reward function of the opponent agent may also be goal oriented. Such a goal-oriented opponent agent can be used in an effort to take advantage of a specific weakness of the flight aid agent or, for example, to discover whether there are ways to reach a specific state using a malfunction type. To demonstrate this, in another simulation training experiment, opponent agents have been trained for four different scenarios against a flight aid agent which has already been trained for 5000 iterations. The flight aid agent is not kept constant during the further simulated training experiment, in order to test the ability of the opponent agent to stay competitive even when the flight aid agent is learning. One opponent agent has been trained for each tested combination of malfunction type and goal reward. The tested combinations are: a) full pitot static system blockage type malfunction with reward towards zero g acceleration in the vertical direction; b) moving centre of gravity type malfunction with reward towards an angle of attack larger than 15 degrees; c) stuck actuator type malfunction with reward towards an angle of attack larger than 15 degrees; and d) engine failure type malfunction with reward towards zero g acceleration in the vertical direction. The high angle of attack reward can lead the opponent agent to induce an aerodynamic stall, which is a dangerous situation which can result in rapid descent or loss of 19 control of the aircraft due to a loss of lift over the main wing and / or control surfaces. One of the ways in which the opponent agent can induce an aerodynamic stall is by moving the centre of gravity towards the rear of the aircraft. Another way in which the opponent agent can increase the risk of an aerodynamic stall is by exploiting a stuck actuator malfunction where the elevator is stuck (jammed) in the up position, such that the aircraft has a high nose-up attitude. The zero g acceleration reward can lead the opponent agent to cause a dive, also meaning a rapid descent of the aircraft. Control over a full pitot static system malfunction can lead to both corruption of altitude and airspeed information. The introduction of malfunctions in the pitot static system can also be exploited by the opponent agent to mislead the flight aid agent into unnecessary behaviour in an attempt to keep increasing speed and / or lowering altitude, which can result in a dive. Engine failure malfunction control can be strategically engaged and disengaged by the opponent agent during descent manoeuvres, which may also induce a dive. This goal reward is a positive reward that is only rewarded to the opponent agent if the condition is true. Besides this goal reward, the opponent agent also has a control reward similar to the control reward of the flight aid agent, which rewards control actions staying within a predetermined action range, which are preferably based on operational recommendations and / or physical limits of the corresponding simulated aircraft controls. In these further simulated training experiments (used to generate the results shown in Figure 5A-D) no termination reward has been given. Termination is set to occur only in overspeed condition or below 500ft (approx 152m) altitude. The results are given in Figure 5A-D which show the opponent goal reward on the Y-axis plotted against the number of training iterations on the X-axis. Figure 5 A provides results for tested combination (a) as described above. Figure 5B provides results for tested combination (b) as described above. Figure 5C provides results for tested combination (c) as described above. Figure 5D provides results for tested combination (d) as described above. In addition to providing the results (Angel vs Gremlin) for simulations in which an opponent agent (Gremlin) acted in opposition to a corresponding flight agent (Angel), each of Figures 5A-D also includes results (Angel vs Random) for simulations in which the flight aid agent (Angel) was subjected to malfunctions being introduced randomly to the simulated flight (with the reward being based on the equivalent the opponent agent would have achieved given the resulting achieved flight conditions in the simulated flights). The difference between the opponent reward achieved using the opponent agent and using random malfunctions may be considered to represent differences in the level of difficulty provided for the flight aid agent. When the opponent agent is able to gain more reward than random malfunctions, it means that the opponent agent is a more difficult opponent for the flight aid agent. Increased difficulty, especially in later stages of training, may lead to a more capable and robust flight aid agent once the training has completed. For all examples given in Figure 5 A-D, the opponent agent eventually obtains a higher reward than the random malfunctions, although in different magnitudes for each tested malfunctionreward combination. As shown for example in Figures 5C and 5D, for stuck actuators and engine failure type malfunctions, the reward for the opponent agent suggests that the opponent agent does not show notable improvement, however, over the same number of iterations the reward for the random malfunctions shows a visible decrease. This decrease in reward for the random malfunctions is likely demonstrative of the flight aid agent having learned techniques to overcome the malfunctions being applied. This means that the flight aid agent learns, by interacting with the opponent agent, how to counter such random manipulations. This suggests that opponent agent did not stay constant but did evolve in order to achieve better reward results than the random malfunctions when the number of iterations is high enough. These observations suggest that the opponent agent is a useful opponent to aid in improving the flight aid agent. [Termination rewards] The simulated flight is optionally configured to be terminated if the flight conditions reach or exceed a termination condition. The termination condition may be a maximum deviation from one or more of the target flight conditions. In response to a termination condition being reached, the flight aid agent may receive a negative reward and / or the opponent agent may receive a positive reward. In this way, the opponent may be incentivised to deviate the aircraft from the target, or safe and stable, flight conditions and the flight aid agent may be incentivised to maintain stable flight. The flight aid agent may therefore learn to maintain safe flight even in difficult circumstances, as it is given training experience where the opponent agent acts to deviate the simulated aircraft away from the target flight conditions. The termination reward is a negative reward that is only given to the flight aid agent at the termination of a simulated flight. Terminating the simulated flight before the maximum duration time-out may desirably aid in teaching the flight aid agent to converge on solutions enabling the flight aid agent to achieve and maintain safe flight. Figure 6 shows the difference between the achieved and target altitude for flight aid agents during simulated flights performed with different termination strategies. In particular, results are shown for simulated flight experiments where: the simulated flight is not terminated regardless of the deviation between the target and achieved altitude; the simulated flight is terminated if the difference between the target and achieved altitude exceeds 100 feet (approx. 30 m); and the simulated flight is terminated if the difference between the target and achieved altitude exceeds 200 feet (approx. 61 m). The progress with increasing number of iterations shows that without any termination for altitude deviation, the flight aid agent is not able to converge on a desirable altitude. Termination was rewarded with a negative reward which resulted in convergence towards the goal within relatively few iterations. For training, in the examples studied in the simulated training experiment, the best combination of termination conditions was found to be 100 ft (approx. 30 m) deviation for altitude and 50 ft / s (approx. 15 m / s) for airspeed. Termination also occurs when in overspeed or below 500ft (approx. 152 m) to prevent crashes of the simulation system. [Normalisation] Observation normalisation, which is normalisation of the state space, is performed with rolling mean and standard deviation, gathered over all episodes steps of all training iterations, where each simulated flight corresponds to a training iteration. These observations are normalised before going into the actor neural network during training. The flight aid reward function is preferably normalised. The opponent reward function may optionally be normalised. Reward normalisation may be performed with rolling standard deviation over all training iterations of the rolling sum of discounted reward per episode step. Reward normalisation may be performed during training over all agents acting in parallel. Rewards may be calculated after having gathered the data for each of the episode steps for the current training iteration. In the optimization phase, the gathered data for each episode step are divided into smaller batches of data, preferably each smaller batch having an equal amount of data. These smaller batches are iterated over to perform gradient ascent. The advantage normalisation is performed using only the mean and standard deviation of the current smaller batch of data. [Training] The flight aid agent is preferably trained using more than one simulated flight. In other words, the flight aid agent may be subject to multiple training iterations, over a plurality of simulated flights. Each simulated flight is initiated with the simulated aircraft in initial simulated flight conditions. The initial simulated flight conditions may be varied between different simulated flights, such that the flight aid agent may learn to react to different flight conditions. The initial flight conditions configured to be varied between different simulated 22 flights may include one or more of: angle of attack, true airspeed and altitude of the simulated aircraft. Additionally, or alternatively, the target flight conditions vary between different simulated flights. For some scenarios, it may be preferable to perform more than one iteration, or simulated flight, with the same initial simulated flight conditions and / or the same target flight conditions. For example, if the flight aid agent does not achieve the target flight conditions during a simulated flight initiated at particular initial simulated flight conditions, it may be appropriate to run multiple iterations of the simulated flight without changing the initial simulated flight conditions. In this way, the flight aid agent may be given the opportunity to learn what action is needed to achieve the desired outcome under these initial simulated flight conditions. For example, the simulated flight may be repeated with the same initial simulated flight conditions until the achieved flight conditions are within a predetermined range of the target flight conditions are achieved one or more times. Preferably, the simulated flight may be repeated with the same initial simulated flight conditions until the achieved flight conditions are consistently within a predetermined range of the target flight conditions are achieved one or more times. [Further detail of the simulation experiments] The above-described training methods have been explored in example training experiments. The initial experiments have been conducted by only training the flight aid agent. This allows to define state space and reward functions for the flight aid agent before considering the added complexity of an opponent agent, which is introduced in subsequent experiments. In particular, in subsequent experiments the flight aid agent is trained further by introducing failures which the opponent agent is configured to control. In all experiments described herein, the simulated aircraft and its surroundings are simulated using a simulation software. The simulation provides 6 degrees of freedom. The reinforcement learning environment is created by running this simulation and is realised by utilizing code from an open AI reinforcement learning framework for custom simulation environments. A suitable machine learning algorithm should be used to train the flight aid agent and / or the opponent agent. For example, a proximal policy optimization algorithm may be used to train the flight aid agent and / or the opponent agent. The above-described training experiments have been performed using a proximal policy optimization algorithm. Proximal policy optimization is a preferred algorithm due to enabling continuous actions of the agent. Furthermore, the selected proximal policy optimization algorithm may beneficially use fewer 23 neural networks than other algorithms. Additionally, the selected proximal policy optimization algorithm is an on-policy algorithm which has convergence guarantees and is less sensitive to disturbances than other algorithm types. An alternative algorithm, other than a proximal policy optimization algorithm, may 5 be used. Preferably, the algorithm should allow for continuous actions (e.g., continuous control commands from the flight aid agent). Other on-policy algorithms, for example, an on-policy maximum a posteriori policy optimization algorithm (e.g., V-MPO), may also be suitable. 10            Aspects of the present disclosure have been described with particular reference to the examples illustrated. While specific examples are shown in the drawings and are herein described in detail, it should be understood, however, that the drawings and detailed description are not intended to limit the invention to the particular form disclosed. It will be appreciated that variations and modifications may be made to the examples described within 15 the scope of the present invention, as defined by the claims.

Claims

1. A method for training a reinforcement learning agent to provide inputs to an automated system on an aircraft, the method comprising steps of:providing a machine learning environment including a simulated aircraft configured to perform a simulated flight;providing, as the reinforcement learning agent, a flight aid agent configured to provide control commands to adjust one or more flight parameters of the simulated aircraft,rewarding the flight aid agent based on a flight aid reward function, wherein the flight aid reward function is configured to provide a reward based on a difference between achieved flight conditions and target flight conditions; andproviding, as an adversarial reinforcement learning agent configured to act in opposition to the flight aid agent, an opponent agent configured to manipulate the simulated aircraft.

2. The method of claim 1, wherein the flight aid agent is configured to provide control commands to adjust one or more flight parameters of the simulated aircraft including one or more of: thrust, elevator, rudder, ailerons, and flaps.

3. The method of either of claims 1 and 2, wherein the simulated aircraft comprises a plurality of flight condition sensors configured to measure flight condition data during the simulated flight, and wherein during the simulated flight a state space of the flight aid agent comprises data gathered by the plurality of flight condition sensors.

4. The method of claim 3, wherein during the simulated flight a state space of the opponent agent comprises:data gathered by the plurality of flight condition sensors, andactual flight conditions of the simulated aircraft during the simulated flight.

5. The method of either of claims 3 and 4, wherein the opponent agent is configured to manipulate the simulated aircraft by controlling factors affecting the accuracy of one or more of the plurality of flight condition sensors.

6. The method of any one of claims 1 to 5, wherein during the simulated flight a state space of the opponent agent comprises:the control commands provided by the flight aid agent to the one or more flight parameters of the simulated aircraft, andan actual control provided to the simulated aircraft during the simulated flight.

7. The method of any one of claims 1 to 6, wherein the opponent agent is configured to manipulate the simulated aircraft by controlling factors affecting the response of the flight parameters to control commands provided by the flight aid agent.

8. The method of any one of claims 1 to 7, wherein the target flight conditions include one or more of the following flight conditions: true airspeed, angle of attack, heading, altitude, absolute vertical acceleration and absolute vertical speed of the simulated aircraft.

9. The method of any one of claims 1 to 8, wherein the flight aid reward function is configured to provide a lower reward if flight conditions exceed an upper threshold than if the flight conditions do not exceed the upper threshold for at least one of the following flight conditions: absolute vertical acceleration and absolute vertical speed of the simulated aircraft.

10. The method of any one of claims 1 to 9, wherein the flight aid agent is configured to determine the control command and a clipped control command at each time step of the simulated flight;wherein if a magnitude of the control command is below an upper limit of the flight control of the simulated aircraft, a magnitude of the clipped control command is equal to the magnitude of the control command applied at the time step,wherein if the magnitude of the control command exceeds the upper limit of the flight control of the flight control of the simulated aircraft, the magnitude of the clipped control command corresponds to the exceeded upper limit; andwherein the flight aid reward function is configured to provide the flight aid agent with a greater reward the smaller the difference between the magnitude of control command and the magnitude of clipped control command.

11. The method of any one of claims 1 to 10, wherein the flight aid reward function is normalised.

12. The method of any one of claims 1 to 11, further comprising rewarding the opponent agent based on an opponent reward function.

13. The method of claim 12, wherein the opponent reward function is configured to provide a reward based on a difference between the achieved flight conditions and the target flight conditions, wherein the opponent agent receives a greater reward the greater the difference between the achieved flight conditions and the target flight conditions.

14. The method of any one of claims 1 to 13, wherein the simulated flight is configured to be terminated if the flight conditions reach or exceed a termination condition, wherein the termination condition is a maximum deviation from one or more of the target flight conditions.

15. The method of claim 14, wherein in response to a termination condition being reached, the flight aid agent receives a negative reward and / or the opponent agent receives a positive reward.

16. The method of any one of claims 1 to 15, wherein the training is repeated for a plurality of simulated flights, wherein each of the plurality of simulated flights is initiated with the simulated aircraft in initial simulated flight conditions, and wherein the initial simulated flight conditions vary between different simulated flights.

17. The method of claim 16, wherein initial flight conditions configured to be varied between different simulated flights include one or more of: angle of attack, true airspeed and altitude of the simulated aircraft.

18. The method of any one of claims 1 to 17, wherein the training is repeated for a plurality of simulated flights, wherein each of the plurality of simulated flights is initiated with target flight conditions, and wherein the target flight conditions vary between different simulated flights.

19. The method of any one of claims 1 to 18, wherein a proximal policy optimization algorithm is used to train the flight aid agent and / or the opponent agent.

20. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of the preceding claims.

21. A system comprising a flight aid agent trained using the method of any one of claims 1 to 19, the system being configured to provide inputs to an automated system on an aircraft based on control commands from the flight aid agent.

22. The system of claim 21, wherein the automated system is an autopilot.

23. The system of either of claims 21 and 22, wherein the system is configured to provide audible and / or visual information to a pilot of the aircraft.

Images