Systems and methods for training neural networks to control aircraft

Abstract

The present disclosure relates to systems and methods for training a neural network to control an aircraft. An apparatus includes a control input generator and a neural network trainer. A flight simulator is configured to generate first state data in response to a first control input from the control input generator, and to provide the first state data to a first neural network to generate a candidate second control input. The control input generator is further configured to select a second control input from between the candidate second control input and a randomized offset control input based on a random offset applied to the first control input based on a random value. The first simulator is configured to generate second state data in response to the second control input from the control input generator. The neural network trainer is configured to update a weight of the first neural network based at least in part on the first state data and the second state data.

Description

Technical Field

[0001] The present disclosure generally relates to training a neural network to control an aircraft. Background Art

[0002] Although advances in artificial intelligence and machine learning have made self-driving vehicles an achievable goal, aircraft continue to be operated by pilots. Collecting data from human pilots to train autonomous aircraft can be time-consuming and expensive. Training data collected based on previous pilot experience will not cover potential new scenarios that the aircraft may encounter. Summary of the invention

[0003] In a specific implementation, an apparatus for training a neural network to control an aircraft includes a memory, a control input generator, and a neural network trainer. The memory is configured to store a first neural network. The control input generator is configured to provide the first control input to a flight simulator. The flight simulator is configured to generate first state data in response to the first control input, and provide the first state data as an input to the first neural network to generate a candidate second control input. The control input generator is also configured to select a second control input of the flight simulator based on a random value. The second control input is selected from between the candidate second control input and a randomized offset control input. The randomized offset control input is based on a random offset applied to the first control input. The control input generator is further configured to provide the second control input to the flight simulator. The flight simulator is configured to generate second state data in response to the second control input. The neural network trainer is configured to update the weight of the first neural network based at least in part on the first state data and the second state data.

[0004] In another specific implementation, a method for training a neural network to control an aircraft includes: providing a first control input to a flight simulator. The flight simulator generates first state data in response to the first control input, and provides the first state data as an input to a first neural network to generate a candidate second control input. The method also includes: selecting a second control input of the flight simulator based on a random value. The second control input is selected from between the candidate second control input and a randomized offset control input. The randomized offset control input is based on a random offset applied to the first control input. The method further includes: providing the second control input to the flight simulator. The flight simulator generates second state data in response to the second control input. The method also includes: updating the weights of the first neural network based at least in part on the first state data and the second state data.

[0005] In another specific implementation, a computer-readable storage device stores instructions that, when executed by one or more processors, cause the one or more processors to initiate, execute, or control operations to train a neural network to control an aircraft. The operations include: providing a first control input to a flight simulator. The flight simulator generates first state data in response to the first control input, and provides the first state data as an input to a first neural network to generate a candidate second control input. The operations also include: selecting a second control input of the flight simulator based on a random value. The second control input is selected from between the candidate second control input and a randomized offset control input. The randomized offset control input is based on a random offset applied to the first control input. The operations further include: providing the second control input to the flight simulator. The flight simulator generates second state data in response to the second control input. The operations also include: updating the weights of the first neural network based at least in part on the first state data and the second state data.

[0006] The features, functions, and advantages described herein may be achieved independently in various implementations or may be combined in yet other implementations, further details of which may be found with reference to the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram illustrating a system configured to train a neural network to control an aircraft.

[0008] Figure 2 yes Figure 1 A diagram of an example of a neural network of a system.

[0009] Figure 3 is a diagram of a flow chart illustrating an example of a method of training a neural network to control an aircraft.

[0010] Figure 4 It is shown that Figure 1 Flowchart of the neural network's aircraft life cycle.

[0011] Figure 5 is a diagram of an aircraft configured to be controlled by a neural network.

[0012] Figure 6 is a block diagram of a computing environment including a computing device configured to support aspects of computer-implemented methods and computer-executable program instructions (or code) according to the present disclosure. DETAILED DESCRIPTION

[0013] Aspects disclosed herein present systems and methods for training a neural network to control an aircraft. A neural network is trained by providing a control input to a flight simulator, determining a reward value based on state data generated by the flight simulator in response to the control input, and updating a weight of the neural network based on the reward value. The state data indicates a state of an aircraft simulation of a specific aircraft type in the flight simulator. The neural network is configured to generate a control input responsive to the state data based on the weights of the neural network. The trained neural network can be used to control an aircraft of a specific aircraft type (e.g., a real-world aircraft). Training a neural network based on a flight simulator consumes fewer resources (e.g., time and money) than pilot data. Because a flight simulator can be designed to simulate a variety of scenarios, including scenarios that pilots do not often encounter, a neural network trained using a flight simulator may also be more robust.

[0014] The accompanying drawings and the following description illustrate specific exemplary embodiments. It should be understood that, although not explicitly described or illustrated herein, those skilled in the art will be able to design various arrangements that embody the principles described herein and are included within the scope of the claims that follow this description. In addition, any examples described herein are intended to aid in understanding the principles of the present disclosure and are to be interpreted as not limiting. Therefore, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0015] Specific implementations are described herein with reference to the accompanying drawings. In the description, common features are represented by common reference numerals throughout the accompanying drawings. As used herein, various terms are used only for the purpose of describing specific implementations and are not intended to be limiting. For example, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are also intended to include plural forms. In addition, some features described herein are singular in some implementations and plural in other implementations. For ease of reference herein, these features are generally introduced as "one or more" features and are subsequently referenced in the singular unless aspects related to multiple features are being described.

[0016] The terms "comprise," "comprises," and "comprising" are used interchangeably with "include," "includes," or "including." Additionally, the term "wherein" is used interchangeably with the term "wherein." As used herein, "exemplary" indicates an example, implementation, and / or aspect, and should not be construed as limiting or indicating a preferred or preferred implementation. As used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements (such as structures, components, operations, etc.) do not, by themselves, indicate any priority or order of an element relative to another element, but rather merely distinguish an element from another element having the same name (but for use of the ordinal term). As used herein, the term "group" refers to a grouping of one or more elements, and the term "plurality" refers to a plurality of elements.

[0017] As used herein, "generate", "calculate", "use", "select", "access" and "determine" are interchangeable unless the context indicates otherwise. For example, "generate", "calculate" or "determine" a parameter (or signal) may refer to actively generating, calculating or determining a parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) such as one that has been generated by another component or device. As used herein, "coupling" may include "communicative coupling", "electrical coupling" or "physical coupling", and may also (or optionally) include any combination thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). As an illustrative, non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some implementations, two devices (or components) that are communicatively coupled (such as in electrical communication) can send and receive electrical signals (digital or analog) directly or indirectly (such as via one or more wires, buses, networks, etc.). As used herein, "directly coupled" is used to describe two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without the need for intermediate components.

[0018] Figure 1An example of a system 100 configured to train a neural network to control an aircraft is depicted. The system 100 includes a flight simulator 146 communicatively coupled to a target neural network 106, a master neural network 104, a memory 132, or a combination thereof. For example, the flight simulator 146 is configured to exchange data with the target neural network 106, the master neural network 104, the memory 132, or a combination thereof. The system 100 includes a reward calculator 142 coupled to the flight simulator 146, the memory 132, or both. The system 100 includes a neural network trainer 140 coupled to the memory 132, the reward calculator 142, the target neural network 106, the master neural network 104, or a combination thereof. The system 100 also includes a control input generator 144 coupled to the memory 132, the master neural network 104, the offset calculator 148, the flight simulator 146, or a combination thereof.

[0019] In a specific aspect, the neural network trainer 140 is configured to train the master neural network 104 based on a deep Q-network technique (DQN) technique. For example, the neural network trainer 140 trains the master neural network 104 using the target neural network 106 that corresponds to a delayed copy of the master neural network 104. To illustrate, the neural network trainer 140 is configured to update the weights of the target neural network 106 by copying the weights of the master neural network 104 in response to determining that at least a threshold count of updates have been performed on the weights of the master neural network 104 after a previous update of the weights of the target neural network 106.

[0020] It should be noted that in the following description, Figure 1 Various functions performed by the system 100 are described as being performed by specific components or modules. However, this division of components and modules is for illustration only. In alternative aspects, the functions described herein as being performed by specific components or modules may be divided among multiple components or modules. Furthermore, in alternative aspects, Figure 1 Two or more components or modules may be integrated into a single component or module. Figure 1 Each component or module shown may be implemented using hardware (e.g., a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a controller, etc.), software (e.g., instructions executed by a processor), or any combination thereof.

[0021] In a specific aspect, the neural network trainer 140 is configured to train the master neural network 104 using a reinforcement learning mechanism. The reinforcement learning mechanism attempts to increase or maximize the reward value (calculated by the reward calculator 142). In a specific example, the flight simulator 146 generates state data representing the state of the aircraft simulation. The master neural network 104 generates a control input based on the state data. For example, the master neural network 104 uses the state data as an input and generates a plurality of predicted master reward values ​​for a plurality of outputs. As used herein, "predicted master reward value" refers to the reward value that the master neural network 104 predicts will be generated if a specific one of the plurality of outputs is provided as a control input to the flight simulator 146. Each of the plurality of outputs corresponds to a different control input. In a specific example, if the master neural network 104 is trained to control the elevator setting of the aircraft, the plurality of outputs correspond to various elevator setting values ​​(e.g., elevator position angles) that can be provided as input to the flight simulator 146. In this example, the predicted master reward value for a specific elevator setting value indicates the reward value that the master neural network 104 predicts the reward calculator 142 will generate if the specific elevator setting value is provided as a control input to the flight simulator 146. In a specific implementation, predicted master reward value 117 is a vector including a set of reward values, the set of reward values ​​including one predicted reward value for each possible control input. For example, when the control input corresponds to an elevator setting value, predicted master reward value 117 may indicate a predicted reward value for each allowed elevator setting value. Alternatively, predicted master reward value 117 may include a single value corresponding to the highest predicted reward value of the vector.

[0022] The predicted primary reward value, the highest predicted reward value, or the vector of information identifying the control input associated with the highest predicted primary reward value among the plurality of primary reward values ​​is provided to control input generator 144 to indicate candidate control inputs 107. Control input generator 144 selects or generates control inputs that are provided to flight simulator 146. Flight simulator 146 generates new state data based on the control inputs provided by control input generator 144. Reward calculator 142 determines a reward value (e.g., an actual reward value or a detected reward value) based on a change in the simulated state of the aircraft indicated by the state data (e.g., based on a comparison of previous state data with the new state data).

[0023] The neural network trainer 140 updates the master neural network 104 based at least in part on the reward value. For example, the neural network trainer 140 determines the weights 121 based on the reward value, and updates the weighted node-connections of the master neural network 104 based on the weights 121. Thus, the master neural network 104 is trained to generate control inputs that produce "good" reward values ​​(e.g., positive reward values), and to refrain from generating control inputs that produce "bad" reward values ​​(e.g., negative reward values).

[0024] In a particular aspect, the neural network trainer 140 is configured to train the master neural network 104 based on the target neural network 106, which is a delayed copy of the master neural network 104. For example, the neural network trainer 140 updates the master neural network 104 based at least in part on the predicted target reward value 119 generated by the target neural network 106 for a particular simulation time step (t+1) and the predicted master reward value 127 generated by the master neural network 104 for a previous simulation time step (t). As used herein, a "predicted target reward value" refers to a reward value that the target neural network 106 predicts will result if a particular control input is provided to the flight simulator 146. Like the master neural network 104, the target neural network 106 generates a plurality of predicted target reward values ​​based on the state data 105 from the flight simulator 146, wherein each predicted target reward value 119 corresponds to a possible control input. In a particular implementation, the predicted target reward value 119 is a vector including a set of reward values, the set of reward values ​​including one predicted reward value for each possible control input. For example, when the control input corresponds to an elevator setting, predicted target reward value 119 may indicate a predicted reward value for each allowed elevator setting.Alternatively, predicted target reward value 119 may include a single value corresponding to the highest predicted reward value of the vector.

[0025] Neural network trainer 140 updates master neural network 104 based on predicted master reward value 127 and predicted target reward value 119. In a particular aspect, because target neural network 106 is updated less frequently, target neural network 106 is more stable than master neural network 104. Updating master neural network 104 based on the predicted target reward value of target neural network 106 (e.g., a delayed neural network) reduces the likelihood that master neural network 104 will fall into an unstable feedback loop.

[0026] In a particular aspect, control input generator 144 is configured to randomly select between candidate control inputs 107 based on outputs of master neural network 104 and randomized control inputs. Control input generator 144 is configured to provide the selected control inputs to flight simulator 146. Using randomized control inputs in addition to control inputs generated by master neural network 104 enables more robust training of master neural network 104. For example, training of master neural network 104 may include state data in response to control inputs (e.g., randomized control inputs) that are unlikely to be generated by master neural network 104.

[0027] In particular aspects, one or more components of the system 100 (e.g., the master neural network 104, the target neural network 106, the neural network trainer 140, the reward calculator 142, the control input generator 144, the flight simulator 146, the offset calculator 148, or a combination thereof) may be implemented at least in part by one or more processors executing one or more instructions. One or more components may be implemented as a single processor or multiple processors, such as a multi-core configuration, a multi-processor configuration, a distributed computing configuration, a cloud computing configuration, or any combination thereof. In some implementations, one or more portions of one or more components of the system 100 are implemented by a processor using dedicated hardware, firmware, or a combination thereof.

[0028] During operation, the flight simulator 146 generates state data 105 indicating the state of the aircraft simulation. In a specific example, the flight simulator 146 is configured to generate state data 105 at a specific simulation time step. For example, the flight simulator 146 generates state data 105 corresponding to a specific simulation time step (t+1). The simulation time step represents a specific time interval (e.g., 5 minutes) that the aircraft simulation passes through. In a specific aspect, the flight simulation can be real-time (e.g., for every 5 seconds of real-world clock time, 5 seconds of simulated flight occurs). Alternatively, the flight simulation can occur at different rates (e.g., during 120 seconds of real-world time, 120 minutes of simulated flight occurs). In addition, the simulation time steps can be evenly spaced or unevenly spaced. For example, a first time interval can be used during the takeoff phase of the simulated flight, and a second time interval can be used during the cruise phase of the simulated flight.

[0029] The state data 105 is a vector including values ​​indicating the state of the aircraft during a particular simulation time step (t+1) of the flight simulation (e.g., at the beginning, middle, or end of the particular simulation time step (t+1) of the flight simulation). In a particular aspect, the state data 105 indicates the altitude, speed, pitch, elevator setting, altitude change, speed change, pitch change, or a combination thereof of the aircraft during the particular simulation time step (t+1).

[0030] The flight simulator 146 stores the state data 105 associated with the specific simulation time step (t+1) in the memory 132 for use in reward value calculations associated with subsequent simulation time steps. The flight simulator 146 provides the state data 105 to the reward calculator 142, the target neural network 106, the master neural network 104, or a combination thereof. Alternatively, the reward calculator 142, the target neural network 106, the master neural network 104, or a combination thereof retrieves the state data 105 from the flight simulator 146 or the memory 132.

[0031] The reward calculator 142 retrieves the state data 115 of the previous simulation time step (t) from the memory 132 in response to receiving the state data 105 of the specific simulation time step (t+1) from the flight simulator 146. The reward calculator 142 determines the reward value 111 for the specific simulation time step (t+1) based on a comparison of the state data 105 of the specific simulation time step (t+1) with the state data 115 of the previous simulation time step (t). In a specific aspect, the reward calculator 142 determines the reward value 111 based on a reward function. The reward function can be based on a configuration setting, a default value, a user input, or a combination thereof. In a specific aspect, the reward calculator 142 determines the reward value 111 (R) during the takeoff simulation based on the following reward function:

[0032]

[0033] In a specific example, state data 115 indicates a first altitude and a first pitch of the aircraft simulation, and state data 105 indicates a second altitude and a second pitch of the aircraft simulation. Reward calculator 142 determines an altitude change based on a comparison of the first altitude and the second altitude (e.g., altitude change=second altitude-first altitude). Reward calculator 142 determines a pitch change based on a comparison of the first pitch and the second pitch (e.g., pitch change=second pitch-first pitch).

[0034] In a specific aspect, the reward calculator 142 determines that the pitch is converging in response to determining that the second pitch is less than a first threshold pitch (e.g., 12 degrees) and the pitch change is positive or the second pitch is greater than a second threshold pitch (e.g., 13 degrees) and the pitch change is negative. The reward calculator 142 sets the reward value 111 to a positive value (e.g., 1) in response to determining that the altitude change is greater than a first threshold altitude (e.g., 1 meter) and the second pitch is greater than or equal to the first threshold pitch (e.g., 12 degrees), the second pitch is less than or equal to the second threshold pitch (e.g., 13 degrees), or the pitch is converging. In this example, because the aircraft is gaining altitude above the first threshold altitude and the pitch is converging or is already between the first pitch threshold and the second pitch threshold, the positive value indicates that the status data 105 indicates a "good" status for takeoff.

[0035] Optionally, the reward calculator 142 sets the reward value 111 to a negative value (e.g., -1) in response to determining that the altitude change is less than a second threshold altitude (e.g., -1 meter). In this example, because the aircraft loses altitude above the second threshold altitude, the negative value indicates that the state data 105 indicates a "bad" state for takeoff.

[0036] In a specific aspect, the reward calculator 142 sets the reward value 111 to a neutral value (e.g., 0) in response to determining that the altitude change is greater than or equal to the second threshold altitude (e.g., -1 meter) and less than or equal to the first threshold altitude (e.g., 1 meter). In this example, because the altitude change of the aircraft is not large (e.g., between the first altitude threshold and the second altitude threshold), the neutral value indicates a "neutral" state for takeoff. Optionally, the reward calculator 142 sets the reward value 111 to a neutral value (e.g., 0) in response to determining that the altitude change is greater than the first threshold altitude (e.g., 1 meter), the second pitch is less than the first pitch threshold (e.g., 12) or greater than the second pitch threshold (e.g., 13), and the pitch is not converging. In this example, because the aircraft is gaining altitude, but the pitch is too low or too high and is not converging, the neutral value indicates a "neutral" state for takeoff.

[0037] It should be understood that the reward function for takeoff based on altitude and pitch is provided as an illustrative, non-limiting example. In other examples, the reward calculator 142 is configured to determine the reward value based on a reward function associated with various phases of flight (e.g., cruising, landing, turning, taxiing, parking, or a combination thereof). The reward function can be based on various aircraft state data (e.g., pitch, yaw, roll, heading, altitude, fuel consumption, temperature sensor data, other types of sensor data, or a combination thereof).

[0038] Reward calculator 142 provides reward value 111 to neural network trainer 140. Alternatively, neural network trainer 140 retrieves reward value 111 from reward calculator 142 or from memory 132. In response to receiving reward value 111 for a particular simulation time step (t+1), neural network trainer 140 retrieves predicted primary reward value 127 for a previous simulation time step (t) and data indicating control input 113 provided to flight simulator 146 in the previous simulation time step (t) from memory 132. For example, state data 105 is generated based on control input 113 for the previous simulation time step (t), and master neural network 104 previously generated predicted primary reward value 127 for candidate control input for the previous simulation time step (t) for selecting control input 113.

[0039] In a particular aspect, the neural network trainer 140 determines the weight 121 based on the predicted primary reward value 127 of the candidate control input for the previous simulation time step (t) in response to determining that the control input 113 corresponds to the candidate control input generated by the master neural network 104 for the previous simulation time step (t). In a particular implementation, the neural network trainer 140 refrains from determining the weight 121 based on the predicted primary reward value 127 and refrains from updating the master neural network 104 in response to determining that the control input 113 corresponds to the randomized offset control input generated by the control input generator 144 for the previous simulation time step (t). In an alternative implementation, the neural network trainer 140 retrieves the predicted primary reward value 127 corresponding to the randomized offset control input (compared to the candidate control input) for the previous simulation time step (t) in response to determining that the control input 113 corresponds to the randomized offset control input for the previous simulation time step (t). As described herein, the neural network trainer 140 determines the weight 121 based on the predicted primary reward value 127.

[0040] The target neural network 106 generates a predicted target reward value 119 based on the state data 105 at a specific simulation step (t+1). Figure 2 In the example shown, the neural network 220 includes a plurality of input nodes 202, a hidden layer including a plurality of hidden layer nodes 204, and a plurality of output nodes 206. In a specific aspect, the input nodes 202 are fully connected to the hidden layer via a first plurality of weighted node-connections. In a specific aspect, the hidden layer is fully connected to the output nodes 206 via a second plurality of weighted node-connections. In a specific aspect, the hidden layer includes a first count (e.g., 40) of hidden layer nodes 204. A higher count of hidden layer nodes 204 enables the neural network 220 to account for higher variability. A lower count of hidden layer nodes 204 enables faster processing time to generate values ​​for the output nodes 206. Although Figure 2 The neural network 220 is shown having a single hidden layer, however in other implementations, the neural network 220 includes multiple hidden layers.

[0041] In a specific aspect, neural network 220 corresponds to Figure 1The target neural network 106, the main neural network 104, or both. In a specific example, each of the input nodes 202 is associated with a specific type of data (e.g., a specific value of the state data 105). For illustration, the first input node is associated with pitch, the second input node is associated with height, the third input node is associated with speed, the fourth input node is associated with elevator setting, the fifth input node is associated with pitch change, the sixth input node is associated with height change, and the seventh input node is associated with speed change, or a combination thereof. It should be understood that seven input nodes are provided as an illustrative example, and in other examples, the input nodes 202 include less than seven input nodes or more than seven input nodes.

[0042] In a specific aspect, the normalized values ​​are provided to one or more of the input nodes 202. For example, a normalized pitch value, a normalized altitude, and a normalized speed are provided to a first input node, a second input node, and a third input node, respectively. For illustration, the range of valid pitch values ​​is from a first pitch value (e.g., -60 degrees) to a second pitch value (e.g., 60 degrees). The pitch value (e.g., 30 degrees) is normalized (e.g., by the flight simulator 146, the neural network 220, or another component of the system 100) to generate a normalized pitch value (e.g., 0.5) within a first normalized pitch value (e.g., -1) and a second normalized pitch value (e.g., 1). The normalized pitch value is provided to the first input node.

[0043] The neural network 220 (e.g., a neural network engine) determines the value of each of the hidden layer nodes 204 as a weighted sum of the values ​​applied to the input nodes 202. The neural network 220 determines the output value of the output node 206 as a weighted sum of the values ​​of the hidden layer nodes 204. For example, the neural network 220 generates the output value (e.g., predicted reward value) of each of the output nodes 206 as a weighted sum of the values ​​of the hidden layer nodes 204.

[0044] Each of the output nodes 206 corresponds to a specific control input of the flight simulator 146. In a specific example, the neural network 220 is configured to generate a control input to control an elevator setting of the aircraft simulation, and the elevator setting has a range of values ​​(e.g., control inputs) between a first elevator setting (e.g., -1) and a second elevator setting (e.g., 1). In this example, each of the output nodes 206 corresponds to a specific value of the range. For illustration, the neural network 220 includes a first output node, a second output node, a third output node, a fourth output node, and a fifth output node corresponding to a first control input (e.g., -1), a second control input (e.g., -0.5), a third control input (e.g., 0), a fourth control input (e.g., 0.5), and a fifth control input (e.g., 1), respectively. Although Figure 2 Five output nodes 206 are shown, but in other implementations, the neural network 220 includes fewer than five or more than five output nodes 206. In particular aspects, a higher count of output nodes 206 (e.g., seventeen instead of five) enables more gradual changes and finer aircraft control than a lower count of output nodes 206; however, a lower count of output nodes 206 enables faster processing times to determine the values ​​of the output nodes 206.

[0045] In a particular aspect, neural network 220 outputs a vector of predicted reward values ​​or data identifying output node 206 having the highest predicted reward value among the predicted reward values ​​generated by neural network 220. For example, the output data may include or identify predicted reward value 219, a control input associated with output node 206, or both. In a particular aspect, neural network 220 outputs data (e.g., a vector) indicating the control inputs and corresponding predicted reward values.

[0046] In a specific example, the target neural network 106 (e.g., the target neural network engine) applies the pitch (e.g., normalized pitch), height (e.g., normalized height), speed (e.g., normalized speed), and elevator setting indicated by the state data 105 to the first input node, the second input node, the third input node, and the fourth input node, respectively. In some implementations, the target neural network 106 (e.g., the target neural network engine) determines a pitch change based on a comparison of the first pitch indicated by the state data 115 with the pitch indicated by the state data 105, determines a height change based on a comparison of the first height indicated by the state data 115 with the height indicated by the state data 105, determines a speed change based on a comparison of the first speed indicated by the state data 115 with the speed indicated by the state data 105, or a combination thereof. In such an implementation, the target neural network 106 (e.g., the target neural network engine) applies the pitch change, height change, and speed change to the fifth input node, the sixth input node, and the seventh input node, respectively. The target neural network 106 generates an output value (e.g., a predicted reward value) of the output node 206. In a particular aspect, predicted reward value 219 (eg, the highest predicted reward value among the predicted reward values) corresponds to predicted target reward value 119 for a particular simulation time step (t+1).

[0047] return Figure 1 , the target neural network 106 provides the predicted target reward value 119 for the specific simulation time step (t+1) to the neural network trainer 140, or the neural network trainer 140 retrieves the predicted target reward value 119 from the target neural network 106 or the memory 132. The neural network trainer 140 determines the weight 121 based on the predicted target reward value 119 for the specific simulation time step (t+1), the predicted primary reward value 127 for the previous simulation time step (t), the reward value 111 for the specific simulation time step (t+1), or a combination thereof. In a specific aspect, the weight 121 is determined to reduce (e.g., minimize) the following loss (or cost) function:

[0048]

[0049] Among them, L t (θ t ) corresponds to the training of the main neural network 104 (e.g., by weights θ t denoted) (e.g., determining the weights 121 of the main neural network 104). t+1The reward value 111 corresponding to a specific simulation time step (t+1). Q(S, a; θ) represents a function of a neural network (the main neural network 104 or the target neural network 106) with weights θ given a set of inputs S (e.g., state data) with output values ​​of the neural network indexed by a (e.g., control inputs or actions). corresponding to the output a with the maximum value by the target neural network 106 (e.g., by the weight Q(S) represents the predicted target reward value 119 generated for a specific simulation time step (t+1). t , a t θ t ) corresponds to the predicted primary reward value 127 generated by the primary neural network 104 for the previous simulation time step (t). γ corresponds to a discount value (e.g., 0.99). In particular aspects, the discount value is based on a configuration setting, a default value, a user input, or a combination thereof. In particular aspects, the loss function corresponds to the expectation of the squared difference between the predicted target reward value 119 at a particular simulation time step (t+1) and the predicted primary reward value 127 at the previous simulation time step (t).

[0050] In a particular aspect, reducing (e.g., minimizing) the loss (or cost) function results in updating the master neural network 104 as indicated by the following formula:

[0051]

[0052] Among them, θ t Corresponding to the main neural network 104, θ t+1 corresponds to the updated version of the main neural network 104 after applying the weights 121, α corresponds to the learning rate, and Q(S t , a t θ t ) corresponds to the predicted primary reward value 127 for the previous simulation time step (t). In particular aspects, the learning rate is based on a configuration setting, a default value, a user input, or a combination thereof. In particular aspects, the neural network trainer 140 determines based on the following formula

[0053]

[0054] Among them, R t+1 The reward value 111 corresponding to the specific simulation time step (t+1), The predicted target reward value 119 corresponds to the specific simulation time step (t+1), and γ corresponds to the discount value (e.g., 0.99). The neural network trainer 140 updates the main neural network 104 based on the weights 121. For example, when the main neural network 104 corresponds to Figure 2When the neural network 220 is trained, the neural network trainer 140 updates the weights of the first plurality of weighted node-connections between the input nodes 202 and the hidden layer nodes 204, the second plurality of weighted node-connections between the hidden layer nodes 204 and the output nodes 206, or a combination thereof.

[0055] In a particular aspect, a simulation time step corresponds to an action cycle. The action cycle data associated with a particular simulation time step (t+1) includes state data 115, state data 105, reward value 111, control input 113, candidate control input associated with a previous simulation time step (t), predicted target reward value 119, predicted primary reward value 127, weight 121, or a combination thereof.

[0056] The neural network trainer 140 updates the master neural network 104 during the learning cycle. In a specific aspect, the neural network trainer 140 updates the master neural network 104 based on the action cycle data of a single action cycle. For example, the neural network trainer 140 updates the master neural network 104 at a first time based on the first action cycle data of the previous simulation time step (t), and updates the master neural network 104 at a second time based on the second action cycle data of the specific simulation time step (t+1). In an optional aspect, the neural network trainer 140 updates the master neural network 104 at a specific time based on the action cycle data associated with multiple action cycles. In a specific example, the memory 132 includes a first-in-first-out (FIFO) memory buffer that stores action cycle data associated with a threshold number of action cycles (e.g., 12000). In a specific aspect, the threshold number of action cycles corresponds to a replay memory size. The neural network trainer 140 retrieves the action cycle data associated with a batch size of action cycles (e.g., 128) from the memory 132 as a mini-batch. In a specific aspect, the mini-batch includes a set of action cycle data for a random action cycle. For example, the mini-batch includes action cycle data of 128 random action cycles among the 12,000 most recent action cycles. The neural network trainer 140 updates the main neural network 104 at a specific time based on the mini-batch. In a specific example, the mini-batch includes first action cycle data associated with a specific simulation time step (t+1) and second action cycle data associated with a second simulation time step. The neural network trainer 140 determines the weight 121 based on the first action cycle data, and determines the second weight based on the second action cycle data. The neural network trainer 140 updates the main neural network 104 based on the weight 121, the second weight, or a combination thereof.

[0057] Master neural network 104 (e.g., master neural network engine) generates a predicted master reward value for state data 105. When weights (e.g., weights 121) of master neural network 104 are different from weights of target neural network 106, the predicted master reward value generated by master neural network 104 for state data 105 may be different from the predicted target reward value generated by target neural network 106 for state data 105.

[0058] Primary neural network 104 (e.g., primary neural network engine) generates an output indicating a predicted primary reward value 117 (or at least the highest predicted primary reward value of the predicted primary reward values). Figure 1 , predicted primary reward value 117 indicates or corresponds to candidate control input 107 at a particular simulation time step (t+1) to control input generator 144. Master neural network 104 stores predicted primary reward value 117 associated with candidate control input 107 at the particular simulation time step (t+1) in memory 132. In a particular aspect, master neural network 104 stores predicted primary reward value 117 and the corresponding control input to memory 132.

[0059] The control input generator 144 retrieves the control input 113 of the previous simulation step (t) from the memory 132 in response to receiving the candidate control input 107 for the specific simulation step (t+1). The offset calculator 148 (e.g., a random number generator) generates a random offset 109 between a first offset (e.g., -0.5) and a second offset (e.g., 0.5). The control input generator 144 generates a randomized offset control input 191 for the specific simulation time step (t+1) based on the random offset 109 and the control input 113 of the previous simulation time step (t) (e.g., randomized offset control input 191=control input 113+random offset 109). Therefore, the control input generator 144 generates the randomized offset control input 191 by randomly selecting a value from a range (e.g., -0.5 to +0.5) centered on the value of the control input 113. In a specific aspect, the randomized offset control input 191 is between a first elevator setting (e.g., -1) and a second elevator setting (e.g., 1).

[0060] In a specific aspect, the control input generator 144 determines that the control input 113 corresponds to a first output node (e.g., the 8th output node) of the main neural network 104. The control input generator 144 identifies the second output node (e.g., the 10th output node) based on the first output node (e.g., the 8th output node) and the random offset 109 (e.g., 2). For example, the second identifier of the second output node is based on the sum of the first identifier of the first output node and the random offset 109 (e.g., second identifier=(first identifier+random offset 109) modulo (count of output nodes)). The control input generator 144 designates the specific control input corresponding to the second output node as the randomized offset control input 191. In this aspect, the random offset 109 includes a random value between a first value (e.g., -2) and a second value (e.g., 2). Therefore, the control input generator 144 generates the randomized offset control input 191 by randomly selecting a value from a range centered on the value of the control input 113. The range is from a first control input corresponding to a first output node (e.g., the -2 output node) to a second control input corresponding to a second output node (e.g., the +2 output node). Generating the randomized offset control input 191 by randomly selecting values ​​from a range centered around the value of the control input 113 corresponds to executing a "random-walk" scheme. In some aspects, the random-walk scheme is suitable for aircraft controls that tend to be relatively continuous in real life (e.g., elevator settings), compared to aircraft controls that tend to have discrete states (such as flaps that vary between extended and unextended).

[0061] The control input generator 144 selects one of the candidate control inputs 107 or the randomized offset control input 191 as the control input 103 for a specific simulation time step (t+1). For example, the control input generator 144 generates a specific random value between a first value (e.g., 0.0) and a second value (e.g., 1.0). The control input generator 144 determines the randomized configuration value 193 based on an iteration count (e.g., t+1) of training the master neural network 104. For example, the control input generator 144 sets the randomized configuration value 193 to a first value (e.g., 0.0) in response to determining that the iteration count (e.g., t+1) is less than an observation threshold (e.g., 3000). Setting the randomized configuration value 193 to the first value increases the likelihood that the randomized offset control input 191 is selected during an observation period corresponding to a training iteration that is smaller than the observation threshold.

[0062] In a specific aspect, the control input generator 144 sets the randomized configuration value 193 between a first value (e.g., 0.01) and a second value (e.g., 1.0) based on the iteration count (e.g., t+1) using a linear function in response to determining that the iteration count (e.g., t+1) is greater than or equal to the observation threshold (e.g., 3000). For example, the linear function generates a randomized configuration value 193 that increases linearly from a first value (e.g., 0.01) to a second value (e.g., 1.0) from a first training iteration (e.g., 3000) to a second training iteration (e.g., 6000). Linearly increasing the randomized configuration value 193 linearly reduces the likelihood that the randomized offset control input 191 is selected during an exploration period following the observation period.

[0063] Control input generator 144 selects randomized offset control input 191 as control input 103 in response to determining that the specific random value is greater than randomized configuration value 193. In a specific aspect, selecting the randomized offset control input for at least some simulation time steps during the observation period enables exploration of more combinations of aircraft simulation states and control inputs and increases the robustness of master neural network 104. Optionally, control input generator 144 selects candidate control input 107 as control input 103 in response to determining that the specific random value is less than or equal to randomized configuration value 193. In a specific aspect, selecting more candidate control inputs during the exploration period enables the behavior of master neural network 104 to converge.

[0064] In a particular aspect, the control input generator 144 stores data identifying the control input 103 at the particular simulation time step (t+1) in the memory 132. The control input generator 144 provides the control input 103 at the particular simulation time step (t+1) to the flight simulator 146. In a particular aspect, the neural network trainer 140 updates the target neural network 106 by replicating the master neural network 104 after updating the master neural network 104 based on the weights 121 and in response to determining that the master neural network 104 has been updated at least a threshold number of times after a previous update of the target neural network 106. For example, the neural network trainer 140 updates the target neural network 106 to have the same weights (e.g., weights 121) as the master neural network 104.

[0065] In a specific aspect, the flight simulator 146 generates second state data for a next simulation time step (t+2) based on the control input 103 in response to determining that the specific simulation time step (t+1) is less than the exploration threshold (e.g., 6000). Training of the master neural network 104 continues based on the state data 105 and the second state data according to the techniques described herein. Optionally, the flight simulator 146 generates an output indicating that the master neural network 104 is trained in response to determining that the specific simulation time step (t+1) is greater than or equal to the exploration threshold (e.g., 6000). In a specific aspect, the master neural network 104 is verified and provided (e.g., uploaded) to the aircraft to control the aircraft (e.g., the elevator settings of the aircraft). For example, one or more sensors of the aircraft provide state data to the master neural network 104 and update the elevator settings of the aircraft based on the control input received from the master neural network 104.

[0066] Thus, the system 100 enables training of the master neural network 104 to control an aircraft. Training the master neural network 104 based on the flight simulator 146 consumes fewer resources (e.g., time and money) than pilot data. Because the flight simulator 146 can be designed to simulate a variety of scenarios, including scenarios that pilots do not often encounter, the master neural network 104 trained using the flight simulator 146 may also be more robust.

[0067] Figure 3 A method 300 of training a neural network to control an aircraft is shown. In particular aspects, Figure 1 The control input generator 144, the flight simulator 146, the neural network trainer 140, the system 100, or a combination thereof may perform one or more operations of the method 300.

[0068] The method 300 includes providing a first control input to a flight simulator at 302. For example, as shown in FIG. Figure 1 As described, Figure 1 The control input generator 144 provides the control input 113 to the flight simulator 146. The flight simulator 146 generates the first state data in response to the control input 113.

[0069] The method 300 also includes providing the first state data as an input to the first neural network to generate a candidate second control input at 304. For example, as shown in FIG. Figure 1 As described, Figure 1 The flight simulator 146 provides the first state data to the main neural network 104 to generate the candidate control input 107.

[0070] The method 300 further includes selecting a second control input of the flight simulator based on the random value at 306. For example, as shown in FIG. Figure 1 As described, Figure 1The control input generator 144 selects the control input 103 of the flight simulator 146 based on the random value. Figure 1 As described, control input 103 is selected from candidate control input 107 and randomized offset control input 191. Figure 1 As depicted, randomized offset control input 191 is based on random offset 109 applied to control input 113 .

[0071] The method 300 also includes providing a second control input to the flight simulator at 308. For example, the control input generator 144 provides the control input 103 to the flight simulator 146. Figure 1 As depicted, flight simulator 146 generates second state data in response to control input 103 .

[0072] The method 300 further includes updating the weights of the first neural network based at least in part on the first state data and the second state data at 310. For example, as shown in FIG. Figure 1 As described, neural network trainer 140 updates the weights of master neural network 104 based at least in part on the first state data and the second state data.

[0073] refer to Figure 4 , a flow chart illustrating the life cycle of an aircraft configured to be controlled by a neural network is shown and designated as method 400. During pre-production, the exemplary method 400 includes, at 402, an aircraft (such as reference Figure 5 During the specification and design of the aircraft, method 400 may include specification and design of a neural network-based aircraft control system. The neural network-based aircraft control system includes or is configured to operate using master neural network 104. At 404, method 400 includes material procurement, which may include procurement of materials for the neural network-based aircraft control system.

[0074] During production, method 400 includes, at 406, component and subassembly manufacturing, and at 408, system integration of the aircraft. For example, method 400 may include component and subassembly manufacturing of a neural network-based aircraft control system and system integration of the neural network-based aircraft control system. For example, master neural network 104 may be loaded into the neural network-based aircraft control system. Method 400 includes, at 410, certification and delivery of the aircraft, and at 412, placing the aircraft in service. Certification and delivery may include certifying the neural network-based aircraft control system to place the neural network-based aircraft control system in service. When in service by a customer, the aircraft may be scheduled for routine maintenance and servicing (which may also include modification, reconfiguration, refurbishment, etc.). For example, a new version of master neural network 104 may be loaded into the neural network-based aircraft control system. At 414, method 400 includes performing maintenance and servicing on the aircraft, which may include performing maintenance and servicing on the neural network-based aircraft control system.

[0075] Each process of method 400 may be performed or implemented by a system integrator, a third party, and / or an operator (e.g., a customer). For the purpose of this description, a system integrator may include, but is not limited to, any number of aircraft manufacturers and main system subcontractors; a third party may include, but is not limited to, any number of vendors, subcontractors, and suppliers; and an operator may be an airline, a leasing company, a military entity, a service organization, etc.

[0076] Aspects of the present disclosure may be described in the context of an example of a vehicle. Specific examples of vehicles are Figure 5 Aircraft 500 is shown.

[0077] exist Figure 5 In the example of , aircraft 500 includes fuselage 518 and interior 522 with multiple systems 520. Examples of multiple systems 520 include one or more of propulsion system 524, electrical system 526, environmental system 528, hydraulic system 530, and neural network-based aircraft control system 532 that includes or is configured to access master neural network 104. Any number of other systems may be included. Neural network-based aircraft control system 532 is configured to receive sensor data from one or more sensors 536 and control the operation of one or more other systems 520 of aircraft 500, such as the position of one or more flight surfaces 534, based on the output of master neural network 104 in response to the sensor data.

[0078] Figure 66 is a block diagram of a computing environment 600 including a computing device 610 configured to support various aspects of computer-implemented methods and computer-executable program instructions (or codes) according to the present disclosure. For example, the computing device 610 or a portion thereof is configured to execute instructions to initiate, execute, or control a reference Figures 1 to 5 In particular aspects, the computing device 610 includes Figure 1 One or more components of the system 100, one or more servers, one or more virtual devices, or a combination thereof.

[0079] The computing device 610 includes one or more processors 620. The processor 620 is configured to communicate with a system memory 630, one or more storage devices 640, one or more input / output interfaces 650, one or more communication interfaces 660, or any combination thereof. The system memory 630 includes volatile memory devices (e.g., random access memory (RAM) devices), non-volatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory), or both. The system memory 630 stores an operating system 632, which may include a basic input / output system for starting the computing device 610 and a complete operating system for enabling the computing device 610 to interact with a user, other programs, and other devices. The system memory 630 stores system (program) data 636. For example, system memory 630 stores master neural network 104, target neural network 106, control input 103, control input 113, random offset 109, candidate control input 107, randomized offset control input 191, randomized configuration value 193, predicted master reward value 117, predicted master reward value 127, predicted target reward value 119, reward value 111, state data 105, state data 115, weight 121, or a combination thereof. In a specific aspect, system memory 630 includes Figure 1 Memory 132.

[0080] System memory 630 includes one or more applications 634 (eg, a set of instructions) that are executed by processor 620. As an example, one or more applications 634 include programs that can be executed by processor 620 to initiate, control, or execute a reference program. Figures 1 to 5 For illustration, one or more applications 634 include instructions 682 executable by processor 620 to initiate, control, or perform one or more operations described with reference to master neural network 104, target neural network 106, neural network trainer 140, reward calculator 142, control input generator 144, flight simulator 146, offset calculator 148, or a combination thereof.

[0081] In a specific implementation, the system memory 630 includes a non-transitory computer-readable medium (e.g., a computer-readable storage device) storing instructions 682, which, when executed by the processor 620, cause the processor 620 to initiate, execute, or control operations to train a neural network to control an aircraft. The operations include providing a first control input to a flight simulator. The flight simulator generates first state data in response to the first control input, and provides the first state data as an input to the first neural network to generate a candidate second control input. The operations also include selecting a second control input of the flight simulator based on a random value. The second control input is selected from between the candidate second control input and a randomized offset control input. The randomized offset control input is based on a random offset applied to the first control input. The operations further include providing the second control input to the flight simulator. The flight simulator generates second state data in response to the second control input. The operations also include updating a weight of the first neural network based at least in part on the first state data and the second state data.

[0082] One or more storage devices 640 include non-volatile storage devices, such as magnetic disks, optical disks, or flash memory devices. In a specific example, storage devices 640 include both removable memory devices and non-removable memory devices. Storage devices 640 are configured to store operating systems, images of operating systems, applications (e.g., one or more applications 634), and program data (e.g., program data 636). In specific aspects, system memory 630, storage devices 640, or both include tangible computer-readable media. In specific aspects, one or more storage devices 640 are external to computing device 610.

[0083] One or more input / output interfaces 650 enable the computing device 610 to communicate with one or more input / output devices 670 to facilitate user interaction. For example, one or more input / output interfaces 650 may include a display interface, an input interface, or both. For example, the output / output interface 650 is adapted to receive input from a user, receive input from another computing device, or a combination thereof. In some implementations, the input / output interface 650 complies with one or more standard interface protocols, including a serial interface (e.g., a universal serial bus (USB) interface or an Institute of Electrical and Electronics Engineers (IEEE) interface standard), a parallel interface, a display adapter, an audio adapter, or a custom interface ("IEEE" is a registered trademark of the Institute of Electrical and Electronics Engineers of Piscataway, New Jersey). In some implementations, the input / output device 670 includes one or more user interface devices and displays, including some combinations of buttons, keyboards, pointing devices, displays, speakers, microphones, touch screens, and other devices.

[0084] The processor 620 is configured to communicate with a device or controller 680 via one or more communication interfaces 660. For example, the one or more communication interfaces 660 may include a network interface. The device or controller 680 may include, for example Figure 5 In a specific example, neural network trainer 140 provides master neural network 104 to aircraft 500 via communication interface 660 in response to determining that training of master neural network 104 is completed, receiving user input, or both.

[0085] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device) stores instructions that, when executed by one or more processors, cause the one or more processors to initiate, execute, or control operations to perform some or all of the above-described functions. For example, the instructions may be executable to implement Figures 1 to 5 In some implementations, Figures 1 to 5 Part or all of one or more of the operations or methods may be implemented by one or more processors (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs)) that execute instructions, by dedicated hardware circuits, or any combination thereof.

[0086] The description of the examples described herein is intended to provide a general understanding of the structures of various implementations. The description is not intended to be used as a complete description of all elements and features of the devices and systems utilizing the structures or methods described herein. Upon reviewing this disclosure, many other implementations may be apparent to those skilled in the art. Other implementations may be utilized and derived from this disclosure so that structural and logical replacements and changes may be made without departing from the scope of this disclosure. For example, method operations may be performed in an order different from that shown in the accompanying drawings, or one or more method operations may be omitted. Therefore, this disclosure and the accompanying drawings should be considered illustrative rather than restrictive.

[0087] In addition, although specific examples have been shown and described herein, it should be understood that any subsequent arrangement designed to achieve the same or similar results can replace the specific implementation shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various implementations. Upon reviewing the description, the combination of the above implementations with other implementations not specifically described herein will be apparent to those skilled in the art.

[0088] It is understood that the abstract of the present disclosure will not be used to interpret or limit the scope or meaning of the claims when submitting it. In addition, in the aforementioned detailed description, various features may be grouped together or described in a single implementation for the purpose of simplifying the present disclosure. The above examples illustrate but do not limit the present disclosure. It should also be understood that many modifications and variations are possible according to the principles of the present disclosure. As reflected in the following claims, the subject matter claimed for protection may be directed to less than all features of any one of the disclosed examples. Therefore, the scope of the present disclosure is defined by the following claims and their equivalents.

Claims

1. An apparatus for training a neural network to control an aircraft, the apparatus comprising: a memory configured to store the first neural network; The control input generator is configured as: providing a first control input to a flight simulator, wherein the flight simulator is configured to generate first state data in response to the first control input, and providing the first state data as an input to the first neural network to generate a candidate second control input; selecting a second control input of the flight simulator based on a random value, wherein the control input generator selects a randomized offset control input as the second control input in response to determining that the random value is greater than a randomized configuration value, and selects the candidate second control input as the second control input in response to determining that the random value is less than or equal to the randomized configuration value, and wherein the randomized offset control input is based on a random offset applied to the first control input; and providing the second control input to the flight simulator, wherein the flight simulator is configured to generate second state data in response to the second control input; and a neural network trainer configured to update weights of the first neural network based at least in part on the first state data and the second state data, The first neural network is configured to receive sensor data from one or more sensors and to control operation of one or more systems of the aircraft based on an output of the first neural network in response to the sensor data.

2. The device according to claim 1, wherein: The control input generator is configured to generate the randomized offset control input by randomly selecting a value from a range centered around a value of the first control input.

3. The device according to claim 1, wherein: The control input generator is configured to determine a reward value based on a change in altitude and pitch indicated by the second state data, and wherein the weight is updated based at least in part on the reward value.

4. The device according to claim 3, wherein: The first neural network includes a plurality of output nodes corresponding to a plurality of control inputs, wherein a first output node among the plurality of output nodes corresponds to the candidate second control input, wherein the first neural network is configured to receive the first state data and generate a plurality of predicted reward values ​​as output values ​​of the plurality of output nodes, and wherein a first predicted reward value of the first output node is a highest predicted reward value among the plurality of predicted reward values.

5. The device according to claim 4, wherein: The memory is further configured to store a second neural network, and wherein the neural network trainer is further configured to: providing the second state data as a second input to the second neural network, wherein the second neural network includes a plurality of second output nodes corresponding to the plurality of control inputs, and wherein the second neural network is configured to receive the second state data and generate a plurality of second predicted reward values ​​as second output values ​​of the plurality of second output nodes; and determining that the second predicted reward value of the second output node is a highest predicted reward value among the plurality of second predicted reward values; Wherein, the weight of the first neural network is updated based on the reward value, the first predicted reward value and the second predicted reward value.

6. The device according to claim 5, wherein: The second neural network is a delayed copy of the first neural network.

7. A method of training a neural network to control an aircraft, the method comprising: providing a first control input to a flight simulator, wherein the flight simulator generates first state data in response to the first control input, and providing the first state data as input to a first neural network to generate a candidate second control input; selecting a second control input of the flight simulator based on a random value, wherein a randomized offset control input is selected as the second control input in response to determining that the random value is greater than a randomized configuration value, and the candidate second control input is selected as the second control input in response to determining that the random value is less than or equal to the randomized configuration value, and wherein the randomized offset control input is based on a random offset applied to the first control input; providing the second control input to the flight simulator, wherein the flight simulator generates second state data in response to the second control input; and updating weights of the first neural network based at least in part on the first state data and the second state data, The first neural network is configured to receive sensor data from one or more sensors and to control operation of one or more systems of the aircraft based on an output of the first neural network in response to the sensor data.

8. The method according to claim 7, further comprising: A randomized configuration value is updated based on an iteration count of training the first neural network.

9. The method according to claim 7, further comprising: The randomized offset control input is generated by randomly selecting a value from a range centered around a value of the first control input.

10. The method according to claim 7, wherein: The first status data indicates at least one of an altitude, a speed, a pitch, an elevator setting, an altitude change, a speed change, and a pitch change.

11. The method according to claim 7, further comprising: A reward value is determined based on the change in altitude and pitch indicated by the second state data, wherein the weight is updated based at least in part on the reward value.

12. The method according to claim 11, wherein: The first neural network includes a plurality of output nodes corresponding to a plurality of control inputs, wherein a first output node corresponds to the candidate second control input, wherein the first neural network is configured to receive the first state data and generate a plurality of predicted reward values ​​as output values ​​of the plurality of output nodes, and wherein a first predicted reward value of the first output node is a highest predicted reward value among the plurality of predicted reward values.

13. The method according to claim 12, further comprising: providing the second state data as a second input to a second neural network, wherein the second neural network includes a plurality of second output nodes corresponding to the plurality of control inputs, and wherein the second neural network is configured to receive the second state data and generate a plurality of second predicted reward values ​​as second output values ​​of the plurality of second output nodes; and determining that the second predicted reward value of the second output node is a highest predicted reward value among the plurality of second predicted reward values; Wherein, the weight of the first neural network is updated based on the reward value, the first predicted reward value and the second predicted reward value.

14. The method according to claim 13, wherein: The second neural network is a delayed copy of the first neural network.

15. The method according to claim 13, further comprising: Based on determining that an iteration count of training the first neural network since a previous update of a second weight of the second neural network is greater than a threshold, updating the second weight of the second neural network based on the weight of the first neural network.

16. A computer readable storage device storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising: A first control input is provided to a flight simulator, wherein: The flight simulator generates first state data in response to the first control input, and provides the first state data as input to a first neural network to generate a candidate second control input; selecting a second control input of the flight simulator based on a random value, wherein a randomized offset control input is selected as the second control input in response to determining that the random value is greater than a randomized configuration value, and the candidate second control input is selected as the second control input in response to determining that the random value is less than or equal to the randomized configuration value, and wherein the randomized offset control input is based on a random offset applied to the first control input; providing the second control input to the flight simulator, wherein the flight simulator generates second state data in response to the second control input; and updating weights of the first neural network based at least in part on the first state data and the second state data, The first neural network is configured to receive sensor data from one or more sensors and to control operation of one or more systems of the aircraft based on an output of the first neural network in response to the sensor data.

17. The computer readable storage device of claim 16, wherein: The first status data indicates at least one of an altitude, a speed, a pitch, an elevator setting, an altitude change, a speed change, and a pitch change.

18. The computer readable storage device of claim 16, wherein: The operations further include determining a reward value based on a change in altitude and pitch indicated by the second state data, wherein the weight is updated based at least in part on the reward value.

Images