System, computer program product, and method for autonomous behavior generation

Abstract

The present disclosure provides systems, computer program products, and methods for autonomous behavior generation. The present disclosure relates to autonomous behavior generation using hierarchical reinforcement learning. A system for autonomous behavior generation is described. The system includes both a high-level controller and a low-level controller. The high-level controller receives observations from the environment and, using a neural network, selects a high-level behavior based on the observations. The low-level controller generates output commands for scripted actions based on the selected high-level behavior. After generating the output commands, the system can implement actions, such as causing a device to perform the scripted action.

Description

[0001] Government rights

[0002] This invention was made with Government support under U.S. Government Contract No. HR0011-19-90018. The Government has certain rights in this invention.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application is a continuation-in-part of U.S. application serial number No. 16 / 792,869, filed on February 17, 2020, which is a non-provisional patent application of U.S. provisional application No. 62 / 814,133, filed on March 5, 2019, the entire contents of which are incorporated herein by reference.

[0005] This application also claims the benefit of and is a non-provisional patent application of U.S. Provisional Application No. 62 / 953,008, filed December 23, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0006] The present invention relates to learning systems, and more particularly to learning systems that allow for autonomous behavior generation using hierarchical reinforcement learning. Background Art

[0007] Reinforcement learning (RL) systems are used in a variety of applications to learn from past decisions or scenarios in order to enhance new decision-making actions. Many researchers have attempted to improve the accuracy of such RL systems. For example, dynamic scripting is a related approach described by Pieter Spronck of Tilburg University. Specifically, dynamic scripting was introduced by Pieter Spronck, Marc Ponsen, Ida Sprinkhuizen-Kuyper, and Eric Postma (2006) in the paper Adaptive Game AI with Dynamic Scripting. Machine Learning, Vol. 63, No. 3, pp. 217-248, (Springer DOI: 10.1007 / s10994-006-6205-6) and presented by Armon Toubman, Jan-Joris Roessingh, Pieter Spronck, Aske Plaat, and Jaap van den Herik (2014) in Dynamic Scripting with Team Coordination in Air Combat Simulation, Proceedings of the 27th International Conference on Industrial, Engineering & Other Applications of Applied Intelligent Systems, Springer-Verlag ( IEAAIE 2014 The authors describe the proposed method in a 2011 paper titled "Reinforcement Learning for Behavior," which was presented at the IEEE Conference on Reinforcement Learning (CRI), the publications of which are incorporated herein by reference. Although Spronck et al. have made some progress in RL systems, their approach uses machine learning models other than neural networks for high-level behavior selection. Such models are limited in their ability to make complex decisions.

[0008] In other work, U.S. Patent No. 6,473,851 (the '851 patent), entitled "System for combining multiple input control policies to provide a compositional output control policy," also incorporated herein by reference, describes an approach with the same primary limitations as dynamic scripting. Although the work described in the '851 patent incorporates stochastic policy mixing, the resulting model is also limited in its ability to be used for complex decision making.

[0009] Therefore, there is a continuing need for systems that use specialized reinforcement learning techniques integrated with advanced neural network models to provide complex high-level decision making. Summary of the Invention

[0010] The present disclosure provides a system for autonomous behavior generation. In one aspect, the system includes one or more processors and one or more associated memories. Each associated memory is a non-transitory computer-readable medium encoded with executable instructions, so that when the executable instructions are executed, the associated one or more processors perform several operations, including: receiving observations from the environment through a high-level controller and selecting a high-level behavior based on the observations from the environment using a neural network; generating output commands for scripted actions based on the selected high-level behavior through a low-level controller; and causing a device to perform the scripted action.

[0011] In yet another aspect, the system performs the following operations: training the neural network using reinforcement learning.

[0012] In another aspect, causing the device to perform the scripted action includes controlling an aircraft in the flight scenario.

[0013] In yet another aspect, the system performs the following operations: training a reinforcement learning agent within the high-level controller using a softmax learning function to generate probabilities of selecting different high-level behaviors.

[0014] On the other hand, the same neural network that selects the behavior also generates state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

[0015] In yet another aspect, the system further includes a second neural network that generates a state-value output for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

[0016] In yet another aspect, the neural network generates an action-value output based on observations of the environment, and wherein the high-level controller selects a high-level behavior using a softmax function on the action-value output.

[0017] Furthermore, the high-level controller selects behaviors less frequently than the low-level controllers select scripted actions.

[0018] In another aspect, an additional set of neural networks are trained using reinforcement learning such that each neural network determines how long to perform one of the high-level behaviors.

[0019] Finally, the present invention also includes a computer program product and a computer-implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium, the computer-readable instructions being executable by a computer having one or more processors such that, when the instructions are executed, the one or more processors perform the operations listed herein. Alternatively, the computer-implemented method includes an act of causing a computer to execute the instructions and perform the resulting operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The objects, features and advantages of the present invention will become apparent from the following detailed description of various aspects of the present invention, taken in conjunction with the following drawings, in which:

[0021] Figure 1 is a block diagram depicting components of a system according to various embodiments of the present invention;

[0022] Figure 2 is an illustrative diagram of a computer program product embodying one aspect of the present invention;

[0023] Figure 3 is a flow chart depicting a high-level controller having a policy neural network utilizing a reinforcement learning trainer and a low-level controller that generates actions corresponding to selected behaviors;

[0024] Figure 4 is an illustration of pseudocode depicting a desired implementation utilizing an advanced reinforcement learning agent and scripted behaviors;

[0025] Figure 5 is an example diagram depicting high-level selection of behaviors and corresponding low-level controller decisions that generate directional control used to adjust the heading of an autonomous platform;

[0026] Figure 6 is an illustrative diagram depicting aspects of utilizing an additional neural network to limit the number of times a high-level controller can switch to different behaviors;

[0027] Figure 7 is a table describing the action space;

[0028] Figure 8 is a graph depicting the results of experiments in a simple scenario using the simulator;

[0029] Figure 9 is a block diagram depicting control of an apparatus according to various embodiments;

[0030] Figure 10 is an illustration depicting aspects including a single neural network according to various embodiments;

[0031] Figure 11 is an illustration diagram depicting example pseudocode according to various embodiments; and

[0032] Figure 12 is an illustration diagram depicting example pseudocode according to various embodiments. DETAILED DESCRIPTION

[0033] The present invention relates to learning systems, and more specifically, to learning systems that allow for autonomous behavior generation using hierarchical reinforcement learning. The following description is presented to enable one of ordinary skill in the art to make and use the invention and incorporate it into the context of a particular application. Various modifications and multiple uses for different applications will be apparent to those skilled in the art, and the general principles defined herein can be applied to a wide range of aspects. Therefore, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0034] In the following detailed description, numerous specific details are set forth to provide a more detailed understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without being limited to these specific details. In other cases, known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the present invention.

[0035] The reader's attention is directed to all papers and documents filed concurrently with this specification and made available for public inspection with this specification, the contents of all of which are incorporated herein by reference. All features disclosed in this specification (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise. Therefore, unless expressly stated otherwise, each feature disclosed is merely one example of a general series of equivalent or similar features.

[0036] Before describing the present invention in detail, a description of the major aspects of the present invention is first provided. Subsequently, an introduction is provided to provide the reader with a general understanding of the present invention. Finally, specific details of various embodiments of the present invention are provided to facilitate an understanding of specific aspects.

[0037] (1) Main aspects

[0038] Various embodiments of the present invention include three "main" aspects. The first aspect is a system for autonomous behavior generation using hierarchical reinforcement learning. This system typically takes the form of computer system operating software or otherwise performing operations, or a "hard-coded" instruction set, which can be incorporated into a variety of devices providing different functions. The second major aspect is a method, typically in the form of software, operating using a data processing system (computer). The third major aspect is a computer program product. A computer program product typically represents computer-readable instructions stored on a non-transitory computer-readable medium, such as an optical storage device (e.g., a compact disc (CD) or digital versatile disc (DVD)) or a magnetic storage device (e.g., a floppy disk or magnetic tape). Other non-limiting examples of computer-readable media include hard disks, read-only memories (ROMs), and flash-type memories. These aspects are described in more detail below.

[0039] Figure 1 A block diagram depicting an example of a system of the present invention (i.e., computer system 100) is provided in . Computer system 100 is configured to perform calculations, processes, operations, and / or functions associated with a program or algorithm, or other means for causing one or more processors to perform the associated operations. In one aspect, certain processes and steps discussed herein are implemented as a series of instructions (e.g., a software program) residing in a computer-readable memory unit and executed by one or more processors of computer system 100. When executed, the instructions cause computer system 100 to perform specific actions and exhibit specific behaviors, as described herein. In various aspects, computer system 100 can be embodied in any device operable to perform the functions described herein as appropriate for a particular application, such as a desktop computer, a mobile phone or smartphone, a tablet computer, a computer embodied in a mobile platform, or one or more other devices that can individually and / or collectively execute instructions to perform the associated operations / processes.

[0040] The computer system 100 may include an address / data bus 102 configured to transmit information. Additionally, one or more data processing units, such as a processor 104 (or multiple processors), are coupled to the address / data bus 102 and / or associated memory to enable the one or more processors 104 to perform the operations described herein. The one or more processors 104 are configured to process information and instructions and to cause related operations to be performed, such as operating high-level and low-level controllers, as described in further detail below. In one aspect, the processor 104 is a microprocessor. Alternatively, the processor 104 may be a different type of processor, such as a parallel processor, an application-specific integrated circuit (ASIC), a programmable logic array (PLA), a complex programmable logic device (CPLD), or a field-programmable gate array (FPGA), or any other processing component operable to perform the related operations.

[0041] Computer system 100 is configured to utilize one or more data storage units. Computer system 100 may include a volatile memory unit 106 (e.g., random access memory ("RAM"), static RAM, dynamic RAM, etc.) coupled to address / data bus 102, wherein volatile memory unit 106 is configured to store information and instructions for processor 104. Computer system 100 may also include a non-volatile memory unit 108 (e.g., read-only memory ("ROM"), programmable ROM ("PROM"), erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory, etc.) coupled to address / data bus 102, wherein non-volatile memory unit 108 is configured to store static information and instructions for processor 104. Alternatively, computer system 100 may execute instructions retrieved from an online data storage unit, such as in "cloud" computing. In one aspect, computer system 100 may also include one or more interfaces, such as interface 110, coupled to address / data bus 102. The one or more interfaces are configured to enable the computer system 100 to connect to other electronic devices and computer systems. The communication interface implemented by the one or more interfaces can include wired (e.g., serial cable, modem, network adapter, etc.) and / or wireless (e.g., wireless modem, wireless network adapter, etc.) communication technology.

[0042] In one aspect, computer system 100 may include an input device 112 coupled to address / data bus 102, wherein input device 112 is configured to communicate information and command selections to processor 104. According to one aspect, input device 112 is an alphanumeric input device (such as a keyboard), which may include alphanumeric keys and / or function keys. Alternatively, input device 112 may be an input device other than an alphanumeric input device. In one aspect, computer system 100 may include a cursor control device 114 coupled to address / data bus 102, wherein cursor control device 114 is configured to communicate user input information and / or command selections to processor 104. In one aspect, cursor control device 114 is implemented using a device such as a mouse, trackball, trackpad, optical tracking device, or touch screen. Notwithstanding the foregoing, in one aspect, cursor control device 114 is directed and / or activated via input from input device 112, such as in response to the use of special keys and key sequence commands associated with input device 112. In the alternative, cursor control device 114 is configured to be managed or directed via voice commands.

[0043] In one aspect, computer system 100 may also include one or more optional computer-usable data storage devices, such as a storage device 116 coupled to address / data bus 102. Storage device 116 is configured to store information and / or computer-executable instructions. In one aspect, storage device 116 is a storage device such as a magnetic or optical drive (e.g., a hard disk drive ("HDD"), a floppy disk, a compact disk read-only memory ("CD-ROM"), or a digital versatile disk ("DVD"). In one aspect, a display device 118 is coupled to address / data bus 102, wherein display device 118 is configured to display video and / or graphics. In one aspect, display device 118 may include a cathode ray tube ("CRT"), a liquid crystal display ("LCD"), a field emission display ("FED"), a plasma display, or any other display device suitable for displaying video and / or graphical images and user-readable alphanumeric characters.

[0044] The computer system 100 presented herein is an example computing environment according to one aspect. However, the non-limiting example of computer system 100 is not strictly limited to being a computer system. For example, one aspect provides that computer system 100 represents a type of data processing analysis that can be used according to various aspects described herein. In addition, other computing systems can also be implemented. Indeed, the spirit and scope of the present technology are not limited to any single data processing environment. Therefore, in one aspect, computer-executable instructions (such as program modules) executed by a computer are used to control or implement one or more operations of various aspects of the present technology. In one implementation, such program modules include routines, programs, objects, components, and / or data structures configured to perform specific tasks or implement specific abstract data types. In addition, one aspect provides for implementing one or more aspects of the present technology by utilizing one or more distributed computing environments, such as in which tasks are performed by remote processing devices linked via a communication network, or in which various program modules are located in local and remote computer storage media, including memory-storage devices.

[0045] Figure 2 , a diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in FIG. The computer program product is depicted as a floppy disk 200 or an optical disk such as a CD or DVD 202. However, as previously mentioned, the computer program product generally refers to computer-readable instructions stored on any compatible, non-transitory computer-readable medium. The term "instructions," as used with respect to the present invention, generally indicates a set of operations to be performed on a computer and may refer to a fragment of an entire program or a single, separate software module. Non-limiting examples of "instructions" include computer program code (source or object code) and "hard-coded" electronics (i.e., computer operations encoded into a computer chip). The "instructions" are stored on any non-transitory computer-readable medium, such as in a computer's memory or on a floppy disk, CD-ROM, and flash drive. Regardless, the instructions are encoded on a non-transitory computer-readable medium.

[0046] (2) Introduction

[0047] As mentioned above, the present disclosure provides a learning system that allows for autonomous behavior generation using hierarchical reinforcement learning. This learning system is implemented as a control system with two distinct but complementary components: a high-level component and a low-level component, which work together to generate complex autonomous behaviors for autonomous platforms such as vehicles and other robotic systems. The high-level component uses a deep neural network to determine the optimal probability of selecting from a set of available behaviors. The low-level component uses traditional rule-based artificial intelligence (AI) to generate control outputs (commands) that cause the autonomous platform to generate the selected behavior. In some aspects, the low-level component can also be implemented as a machine learning system that utilizes one or more additional deep neural networks.

[0048] As will be appreciated, the systems and processes described herein have numerous applications. For example, the system can be implemented in autonomous air-to-air engagement systems or used in simulation software designed to model air-to-air engagements for training or decision-making purposes. Experimental results in simulators demonstrate that the present invention yields an order of magnitude improvement in training speed over existing solutions utilizing traditional non-hierarchical reinforcement learning architectures. Furthermore, the present invention can be more easily applied to new problems than state-of-the-art reinforcement learning methods because it can be used with simple, sparse reward functions, rather than complex, dense reward functions that require manual design and trial-and-error experimentation whenever the system is used in a new environment. As an additional potential application, the system can be incorporated into autonomous cars and trucks. In this case, high-level behaviors could include driving along a road, executing an emergency stop, or turning at an intersection. Low-level behaviors for ground vehicles would include steering, acceleration, and braking commands, as well as associated turn signals, transmission shifts, etc. Specific details regarding the present invention are provided below.

[0049] (3) Specific details of each implementation method

[0050] like Figure 3 As shown, the present disclosure provides a high-level controller 300 and a low-level controller 302. The high-level controller 300 accepts observations 304 from an environment 306 (simulated or real) and uses a neural network 308 to select high-level behaviors based on behavior probabilities. The ability to select high-level behaviors based on behavior probabilities is well known to those skilled in the art of machine learning. For example, open source software can be used to implement the neural network that performs high-level behavior selection, such as the TensorFlowCore r1.15 Module: tf software provided by tensorflow.org.

[0051] Neural network 308 is a fully connected network that multiplies an input observation vector by a learnable weight matrix; adds a learnable bias vector to the result; applies a nonlinear function such as a rectified linear unit (ReLU) to this result; and then repeats this process for each neuron layer by applying the results from the previous layer as input. The final layer produces an output for each possible high-level action, and the final layer does not apply an element-wise nonlinearity such as ReLU to the results. Instead, the final layer samples an integer from the softmax distribution defined by these outputs. The high-level action indexed by this integer is then selected. Using a softmax activation instead of a ReLU in the final layer is well known to machine learning practitioners and those skilled in the art. ReLU is not used in the final layer because it would cause the action probabilities to sum to something other than 1. In contrast, the softmax operation produces a probability distribution that guarantees that all probabilities sum to 1. Each output of the neural network affects the relative probability of selecting the corresponding action.

[0052] Non-limiting examples of such high-level behaviors include "lead pursuit," "lag pursuit," "pure pursuit," or "avoidance." Other examples include "hold position," "stop," "swing," "collision avoidance," and so on. It should be appreciated that the specific behavior options depend on the application implementing the system. Once a behavior 310 has been selected and passed to the low-level controller, the low-level controller 302 generates output actions 312 (scripted behaviors) that directly control the system's motion. Thus, output actions 312 are commands or signals that, in some aspects, are sent to the mobile platform's actuators 314 (wheels, motors, engines, flaps, rotors, etc.) to cause the mobile platform to execute the scripted behavior. For example, if an autonomous aircraft in a one-on-one engagement scenario selects "pure pursuit," the low-level controller will generate signals that direct the aircraft to lead or otherwise fly directly toward its opponent. Other examples of scripted behaviors include "lag pursuit" (moving behind the opponent), "lead pursuit" (moving in front of the opponent), and "avoidance" (moving away from the opponent). In an alternative embodiment, the low-level controller 302 may also be implemented using a deep neural network.

[0053] The high-level controller's neural network 308 is trained using a reinforcement learning agent 318 in conjunction with a trainer 316. For each training episode, the system keeps track of the high-level behavior it has chosen, the observations resulting from applying the corresponding low-level actions to the environment, and the rewards obtained from the reward function of the same environment. After each episode is completed, the trainer 316 uses a variant of a gradient descent optimizer to update the neural network 308 connection weights in a manner that increases the expected value of future rewards. As a non-limiting example, the neural network connection weights are updated using the RMSProp gradient descent optimizer with a learning rate of 0.0007, momentum of 0, and an epsilon of 1e-10. RMSProp gradient descent optimizers are well known to those skilled in the art of machine learning. For example, the open source RMSProp gradient descent optimizer can be found at www.tensorflow.org. Other optimizers, such as Momentum, Nesterov, Adam, or AdaGrad, can be readily used.

[0054] A desired embodiment uses A3C (Asynchronous Advantage Actor Critic) to parallelize the execution of episodes by the trainer 316 during training. This is described in "Asynchronous Methods for Deep Reinforcement Learning", Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:1928-1937, 2016, by Mnih et al., the entire contents of which are incorporated herein by reference. Thus, in some aspects, the trainer 316 uses a softmax function to train the reinforcement learning agent 318 to generate probabilities for choosing different actions. Other policy-based reinforcement learning algorithms can also be used instead of Actor-Critic, such as: A2C, PPO, TRPO, ACKTR, IMPALA, DDPG, and TD3.

[0055] Some of these methods (including the desired implementation using A3C) include a value network in addition to the policy network. These minor details should be familiar to those with knowledge of reinforcement learning. In this example, two separate networks are used, each with a single hidden layer of 36 neurons and ReLU activations. Alternatively, a combined network with two outputs can be used—one for generating action probabilities and one for providing the value function. In yet another alternative, a value-based method such as Q-learning can be used, in which case the policy network is replaced by a value network and an ε-greedy policy.

[0056] As a specific example implementation, Figure 4 Example pseudocode for a desired implementation utilizing a high-level reinforcement learning agent and low-level scripted behaviors is provided. It should be understood that while specific examples of pseudocode are provided, other example code or pseudocode can be implemented to provide the operations described herein. The high-level reinforcement learning agent selects from a variety of scripted behaviors. These are essentially high-level behaviors that generate low-level behaviors. For example, the high-level agent may select a "pure pursuit" high-level behavior. Scripted rules (e.g., control laws similar to those used in missile guidance systems) translate this into low-level behaviors, such as moving the ailerons to roll the aircraft while simultaneously moving the elevator to command a certain angle of attack.

[0057] In addition, four alternative embodiments or aspects are provided herein, each of which provides a mechanism that limits the number of times the high-level controller is given the option to switch to a different behavior. It should be understood that the alternative embodiments can be implemented in air combat or any other autonomous vehicle application or other applicable application.

[0058] First alternative ( Figure 11 ) still performs high-level behavior selection at a fixed frequency, but this frequency is lower than the update rate of the low-level controller. This has been shown to provide a slight improvement in performance over the preferred embodiment at the expense of increased complexity. Specifically, Figure 5 An example of high-level selection of behavior 500 (e.g., lagging pursuit, pure pursuit, and leading pursuit) is provided at a frequency that is 1 / 8 the frequency of the low-level controller 502. In this non-limiting example, the low-level controller 502 generates specific scripted actions (e.g., turning direction (left / right)) used to adjust the direction of travel of the mobile platform. Alternative implementations can be used with additional neural networks that limit the time when the high-level controller can switch to different behaviors.

[0059] Another alternative embodiment ( Figure 12Figure 1 depicts example pseudocode for this, using traditional rule-based AI to specify the termination conditions for each behavior. Once a behavior has been selected, execution continues until the termination condition is reached, at which point a high-level controller selects a new behavior.

[0060] Figure 6 Yet another alternative aspect is shown. Figure 6 The illustrated aspect includes additional neural networks 600, one for each state 602 through 608. Note that the states listed below provide specific example implementations; however, the invention is not limited in this regard, as any suitable state or action may be implemented given a particular application. Figure 6Also shown are the state transition probabilities between tactical states. These neural networks 600 are in addition to the policy neural network 601 and are selected from a set of seven available tactical states (each corresponding to one of seven specific behaviors) described and illustrated herein. Neural networks 600 limit the time when a high-level controller can switch to a different behavior. Whenever a selected low-level controller generates an output action 610, the corresponding neural network 600 of that low-level controller generates a probability of continuing the current behavior or handing control back to the high-level controller, which can decide to switch to a different behavior. The overall behavior in this regard is best described in terms of a state machine. The system begins in state 609, meaning that no behavior has yet been selected. From this state, the system transitions to one of states 602 through 608, where the probabilities are determined by applying a softmax function to the output of the policy neural network 601. Once the system is in a state 602 through 608, it executes the high-level behavior corresponding to that specific state 611. For example, the "decelerate" behavior sets the target speed of the aircraft to a value 5 m / s lower than the aircraft's current value. The "pursue" behavior causes the aircraft to turn toward the opponent. The "avoid" behavior causes the aircraft to turn away from the opponent. After each time step, the tactical neural network 600, corresponding to the current state 602 through 608, is used to determine the probability of remaining in the current state or the probability of transitioning back to state 609. In the first case, the system ensures the current behavior continues. In the second case, the policy neural network 601 selects the state 602 through 608 corresponding to the new behavior and repeats the process. Since the tactical network 600 only outputs two probabilities (whose sum is 1), it uses a logistic output instead of a softmax output. The probability of transitioning back to 609 is calculated by applying a logistic function to the output of a single neuron in 600, and the probability of remaining in the current state is determined by subtracting (1 minus this probability). The logistic function is a well-known method used to generate probability distributions in neural networks (similar to softmax). In this case, there are only two possible alternatives (transition back or remain), so a neural network with a single output neuron and a logistic function is used. Since the output of the logistic function is always between 0 and 1, it is guaranteed to be a valid probability. Since the probabilities must sum to 1, the system can then find the remaining probability by subtracting (1 - transition probability). It should be understood that all high-level behaviors or "strategies" are the same as they appear in the desired implementation—for example, "lead pursuit," "lag pursuit," "pure pursuit," "avoidance," and so on. The decisions made by the "tactical" networks are precisely timed—each "tactical" network specifies the conditions under which the corresponding behavior ends.

[0061] Another optional aspect is similar to the previous aspect; however, this aspect uses a single neural network with multiple outputs rather than separate neural networks. Figure 10 As shown, this single neural network generates all the transition probabilities needed to make the following decision: which action state to choose (e.g., Figure 6 States 602 to 608 (shown as states p70, p71, ..., p75, p76) are shown, along with probabilities for each state, as well as whether to continue in the current behavior state (with probabilities p00, p11, ..., p55, p66) or transition to a new behavior state (with probabilities p07, p17, ..., p57, p67). These states correspond to high-level behaviors. The lowest-level behavior of an aircraft involves direct control of flight surfaces. For example, a low-level behavior corresponding to the high-level "pure pursuit" would involve rolling the aircraft in the direction of the opponent and raising the elevator to increase the angle of attack so as to point at the opponent. To decide between the seven available behaviors, a neural network head 1000 with a softmax activation function is used. However, to decide whether to remain in the current behavior state or transition to state 609, a logistic function is used to determine the probability of transitioning to 609, and (1 - this probability) is used to determine the probability of remaining in the current state 602 to 608.

[0062] Experiments were performed using a simulated environment. The simulator models the movement of two or more aircraft in a two-dimensional map. In an example embodiment, the policy network selects one of 14 possible discrete actions for the platform it is controlling, such as Figure 7 shown.

[0063] The softmax output layer of the neural network produces an integer from 0 to 13. This integer corresponds to Figure 7The "Index" column in [ 1 ] specifies whether to perform lead pursuit, lagging pursuit, pure pursuit, or evasion, but in the first two cases, also specifies the amount of lead or lag. When the neural network selects indexes 0 through 5 that trigger the lagging pursuit behavior 700, the neural network causes the platform it is controlling to pursue a point behind its opponent (as a negative intercept offset 708). Pure pursuit 702 and lead pursuit 704 are similar, except that in each case the point is at or in front of the target. Evasive action 706 causes the platform to turn away from its opponent and increase speed as much as possible to enable evasion. Each platform's weapon engagement zone is modeled as a simple circular sector defined by a radius and a center angle. In the experiments, the radius of the zone was set to 2 kilometers, and the center angle was set to 30 degrees. Each episode ends when one of the platforms enters the other platform's weapon engagement zone, at which point a reward of +5000 is awarded to the platform in the firing position and a reward of -5000 is awarded to the platform about to be fired. If neither platform enters the other platform's engagement zone within 1000 time steps, a draw is declared and neither platform receives a reward (zero value).

[0064] In reinforcement learning applications, the environment typically provides the agent with a more informative “dense reward” function that provides a continuous spectrum of outcome desirability beyond just winning or losing. For example, some previous experiments that did not utilize a hierarchical approach used a reward function that included small rewards at various steps for getting closer and closer to the opponent, even if a win was not ultimately achieved. This is necessary for bootstrapping learning. Otherwise, the probability that an untrained platform ever reaches its opponent is too low, the reward received is small, and the policy never even starts to improve. However, a surprising advantage of the new hierarchical approach is that now a simple sparse reward function provides enough feedback to train the agent from scratch. This makes the approach much easier to apply to new scenarios / decision-making environments because it eliminates the need for trial-and-error reward function design.

[0065] In addition to rewards, the environment also provides each agent with the distance, approach speed, position, direction of travel, and cross speed of its opponent after each simulation time step. This allows the agent to choose an action based on the current observed state of the environment.

[0066] Figure 8Experimental results are shown, depicting a simple scenario executed using a simulator that provides 10,000 episodes mapped from a starting point. The baseline 800 results use pure reinforcement learning. It takes approximately 2,500 episodes of experience before the agent learns to win more episodes than it loses. In contrast, the desired embodiment 802 of the present invention uses one of its scripted policies, achieving seemingly instantaneous learning by comparison. In fact, the prior knowledge encoded in the scripted policy greatly simplifies the reinforcement learning task. Experiments were also conducted on an alternative embodiment 804 in which high-level actions are selected 256 times less frequently than low-level actions. This makes it easier for the reinforcement learning module to find a winning strategy because it only needs to select actions four times per episode instead of 1,000 times (assuming each episode lasts 1,000 steps). These results demonstrate that the novel approach has advantages over the two simpler methods on which it is built. It can be significantly faster than reinforcement learning using a flat architecture and more effective than a simple scripted (traditional) AI opponent.

[0067] (3.1) Device control

[0068] like Figure 9 As shown, low-level controller 302 generates command signals to allow the system (via one or more processors, etc.) to control device 900 (e.g., a mobile device display, a virtual reality display, an augmented reality display, a computer monitor, a motor, an engine, a machine, a drone, a camera, etc.). For example, control of device 900 can be used to cause the device to move based on a scripted action or otherwise initiate a physical action (i.e., perform a scripted action).

[0069] In a primary embodiment, an unmanned aerial vehicle (UAV) or drone (either simulated on a computer or a real UAV or drone with hardware actuators connected to the system described herein) can be controlled using a set of behaviors selected using a high-level controller. One such behavior implements a guidance law (e.g., proportional navigation) that causes the controlled aircraft to pursue another aircraft based on the relative position, orientation, and / or velocity of the second aircraft determined using radar, visual cameras, infrared cameras, acoustic sensors, LIDAR, or any other sensor that can provide this information. Instead of a single pursuit behavior, there can also be multiple behaviors from Figure 7Lead, lag, and pure pursuit behaviors. Evasion behaviors may also be included. When selected, this behavior activates a guidance law that causes the aircraft to increase its distance from an aircraft sensed by its sensors. The Weapon Fire behavior, when selected, causes the aircraft to initiate a series of actuator movements and / or firings that activate a weapon, such as a gun or missile, with the goal of destroying or otherwise disabling another aircraft. The Weapon Support behavior, when selected, causes the aircraft to maintain a position that allows it to illuminate an enemy aircraft with its radar transmitter in order to provide support for a radar-guided missile. Any other typical aircraft procedures or maneuvers that can be automated using a series of actuator movements may be included in the set of available behaviors.

[0070] In some embodiments, a drone or other autonomous vehicle can be controlled to move to an area where a target is located based on imagery obtained from the environment. In still other embodiments, a camera can be controlled to orient toward or track the target. In other words, an actuator or motor is activated to move or zoom in on the location of the target. In another aspect, if the system is searching for a specific target and if the target is not determined to be within the camera's field of view, the camera can be rotated or turned to view other areas within the scene until the target is detected.

[0071] Additionally, in a non-limiting example of an autonomous vehicle having multiple sensors (such as cameras that can be used to detect objects in the environment surrounding the vehicle), the system can cause the autonomous vehicle to perform vehicle maneuvers. For example, if the vehicle sensors detect an object in the vehicle's path, the system can be used to perform precise vehicle maneuvers (scripted actions) by controlling vehicle components to avoid a collision. For example, if the vehicle is a car and the object is a stop sign, the system can cause the autonomous vehicle to apply a functional response, such as a braking operation, to stop the vehicle. Other appropriate responses may include one or more of a steering operation, a throttle operation to accelerate or decelerate, or a decision to maintain the direction and speed of travel without changing. The response can be suitable for avoiding a collision, increasing travel speed, or improving efficiency.

[0072] Finally, while the present invention has been described in terms of several embodiments, a person of ordinary skill in the art will readily recognize that the present invention may have other applications in other contexts. It should be noted that many embodiments and implementations are possible. Moreover, the appended claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any "means for..." language is intended to induce a means-plus-function reading of the element and claim, and any element that does not specifically use the "means for..." language should not be interpreted as a means-plus-function element, even if the claim otherwise includes the word "means." Furthermore, while particular method steps have been stated in a particular order, these method steps may occur in any desired order and fall within the scope of the present invention.

Claims

1. A system for autonomous behavior generation, the system comprising: One or more processors and one or more associated memories, each associated memory being a non-transitory computer-readable medium having executable instructions encoded thereon such that, when the executable instructions are executed, the associated one or more processors perform the following operations: receiving, by a high-level controller, observations from an environment and, using a neural network, selecting a high-level behavior from a set of scripted behaviors based on the observations from the environment; training a reinforcement learning agent within the high-level controller using a softmax learning function to generate probabilities for selecting different high-level behaviors; generating, by the low-level controller, output commands for scripted actions corresponding to the selected one of the high-level behaviors; as well as causing a device to perform the scripted action, wherein causing the device to perform the scripted action includes controlling a vehicle, The set of additional neural networks is located outside the neural network, and each additional neural network determines how long to execute one of the high-level behaviors.

2. The system of claim 1 , further comprising the operation of training the neural network using reinforcement learning.

3. The system according to claim 1, wherein: Causing the device to perform the scripted action includes controlling an aircraft in a flight scene.

4. The system according to claim 1, wherein: The same neural network that selects actions also produces state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

5. The system of claim 1 , further comprising a second neural network that generates state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

6. The system according to claim 1, wherein: The neural network produces an action-value output based on observations of the environment, and wherein the high-level controller selects the high-level behavior using a softmax function on the action-value output.

7. The system according to claim 1, wherein: The high-level controller selects behaviors less frequently than the low-level controller selects scripted actions.

8. The system according to claim 1, wherein: The additional set of neural networks is trained using reinforcement learning.

9. A computer program product for autonomous behavior generation, the computer program product comprising: A non-transitory computer-readable medium having executable instructions encoded thereon, such that when the instructions are executed by one or more processors, the one or more processors perform the following operations: receiving, by a high-level controller, observations from an environment and, using a neural network, selecting a high-level behavior from a set of scripted behaviors based on the observations from the environment; training a reinforcement learning agent within the high-level controller using a softmax learning function to generate probabilities for selecting different high-level behaviors; generating, by the low-level controller, output commands for scripted actions corresponding to the selected one of the high-level behaviors; as well as causing a device to perform the scripted action, wherein causing the device to perform the scripted action includes controlling a vehicle, The set of additional neural networks is located outside the neural network, and each additional neural network determines how long to execute one of the high-level behaviors.

10. The computer program product of claim 9, further comprising instructions encoded on a non-transitory medium to cause the one or more processors to train the neural network using reinforcement learning.

11. The computer program product of claim 9, wherein: Causing the device to perform the scripted action includes controlling an aircraft in a flight scene.

12. The computer program product of claim 9, wherein: The same neural network that selects actions also produces state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

13. The computer program product of claim 9, further comprising instructions encoded on a non-transitory medium to cause the one or more processors to use a second neural network to generate a state-value output for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

14. The computer program product of claim 9, wherein: The neural network generates an action-value output based on observations of the environment, and wherein the high-level controller selects the high-level behavior using a softmax function on the action-value output.

15. The computer program product of claim 9, wherein: The high-level controller selects behaviors less frequently than the low-level controller selects scripted actions.

16. The computer program product of claim 9, wherein: The additional set of neural networks is trained using reinforcement learning.

17. A computer-implemented method for autonomous behavior generation, the method comprising the following acts: causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium such that, when executing the instructions, the one or more processors perform the following operations: receiving, by a high-level controller, observations from an environment and, using a neural network, selecting a high-level behavior from a set of scripted behaviors based on the observations from the environment; training a reinforcement learning agent within the high-level controller using a softmax learning function to generate probabilities for selecting different high-level behaviors; generating, by the low-level controller, output commands for scripted actions corresponding to the selected one of the high-level behaviors; as well as causing a device to perform the scripted action, wherein causing the device to perform the scripted action includes controlling a vehicle, The set of additional neural networks is located outside the neural network, and each additional neural network determines how long to execute one of the high-level behaviors.

18. The method of claim 17, further comprising the operation of training the neural network using reinforcement learning.

19. The method according to claim 17, wherein Causing the device to perform the scripted action includes controlling an aircraft in a flight scene.

20. The method according to claim 17, wherein The same neural network that selects actions also produces state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

21. The method of claim 17, further comprising a second neural network that generates state-value outputs for use as a reinforcement learning baseline or for reinforcement learning bootstrapping.

22. The method according to claim 17, wherein The neural network produces an action-value output based on observations of the environment, and wherein the high-level controller selects the high-level behavior using a softmax function on the action-value output.

23. The method according to claim 17, wherein The high-level controller selects behaviors less frequently than the low-level controller selects scripted actions.

24. The method according to claim 17, wherein The additional set of neural networks is trained using reinforcement learning.

Images