Method and device for training neural network model for behavior simulation

Abstract

A computer-implemented method for training a neural network model for behavioral imitation using generative adversarial imitation learning (GAIL) is disclosed. The method includes obtaining presentation data representing a behavior of a presenter for a task, where the presentation data includes a sequence of state-action pairs, where a state corresponds to a condition for a decision and an action corresponds to a decision made by the presenter; updating a discriminator of the neural network to maximize a first objective function, where the first objective function is modified with a first term for promoting the discriminator to become indistinguishable between a behavior of the demonstrator and a state-action pair generated by a policy generator of the neural network; and updating the policy generator to minimize a second objective function, where the second objective function is modified with a second term for promoting convergence of the policy generator to a behavior of the presenter. Several additional aspects are disclosed.

Description

Technical Field

[0001] The various aspects of this disclosure generally relate to artificial intelligence, and in particular to methods and apparatus for training neural network models for behavioral imitation. Background Technology

[0002] Imitation learning (IL) refers to the problem of learning to perform a task from expert demonstrations. It has attracted considerable attention in many real-world applications because it can effectively solve sequential decision-making problems, to name just a few, such as chess, esports, autonomous driving, and intelligent robots. IL has two main approaches: Behavior Cloning (BC), which learns a policy from state-action pairs derived from expert trajectories as a supervised learning problem; and Inverse Reinforcement Learning (IRL), which finds the cost function under conditions where the expert is uniquely optimal.

[0003] However, BC tends to succeed only when using large amounts of data, and IRL learns a cost function that explains expert behavior but doesn't directly tell the learner how to act. Given that the learner's true goal is often to imitate the expert's actions, a framework called Generative Adversarial Imitation Learning (GAIL) is proposed, utilizing the backbone of Generative Adversarial Networks (GANs) to tell the agent how to act by directly learning policies from experts.

[0004] Unfortunately, because GAIL also alternately trains the discriminator and policy generator, it inherits the unstable nature of GANs. GAIL's training process exhibits oscillating behavior and fails to reach optimal equilibrium. Therefore, a method is desired for training neural networks using GAIL for behavioral mimicry that can push GAIL towards equilibrium and achieve stability. Summary of the Invention

[0005] The following is a simplified overview of one or more aspects to provide a basic understanding of them. This overview is not a comprehensive summary of all anticipated aspects, and is neither intended to identify key or important elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0006] Generative Adversarial Imitation Learning (GAIL) is a framework that aims to explicitly learn how to act by learning policies directly from expert demonstrations, rather than first recovering the expert's cost function using IRL and then extracting the policy from the cost function using RL. GAIL is a powerful method for learning sequential decision-making policies, making it widely used in many real-world applications, such as, but not limited to, chess, e-sports, autonomous driving, and intelligent robots.

[0007] However, like its backbone GAN, GAIL trains the discriminator and policy generator alternately, resulting in GAIL's training process inheriting the unstable nature of GANs. It exhibits oscillating behavior and is unstable in reaching optimal equilibrium. Therefore, a method is needed to push GAIL towards equilibrium and achieve stability during training.

[0008] In one aspect, a computer implementation method for training a neural network model for behavioral imitation using Generative Adversarial Imitation Learning (GAIL) is disclosed. The method includes: obtaining demonstration data representing the behavior of a demonstrator in response to a task, wherein the demonstration data includes a sequence of state-action pairs, where a state corresponds to a condition for decision-making and an action corresponds to a decision made by the demonstrator; updating a discriminator of the neural network to maximize a first objective function, wherein the first objective function is modified using a first term that drives the discriminator to become indistinguishable between the demonstrator's behavior and state-action pairs generated by a policy generator of the neural network; and updating the policy generator to minimize a second objective function, wherein the second objective function is modified using a second term that drives the policy generator to converge to the demonstrator's behavior.

[0009] On the other hand, the first and second items are determined based on the first controller and the second controller, respectively, which push the dynamic system corresponding to the discriminator and the policy generator to the objective function corresponding to the equilibrium of the dynamic system.

[0010] On the other hand, the dynamic system corresponding to the discriminator and the policy generator is determined by the derivatives of the first objective function and the second objective function with respect to time.

[0011] On the other hand, the dynamic system reaches equilibrium when the output of the discriminator is equal to 1 / 2 and the output of the policy generator matches the behavior of the presenter.

[0012] On the other hand, the first controller and the second controller are constrained by the principle of asymptotic stability in order to keep the dynamic system stable near the equilibrium.

[0013] On the other hand, the demonstration data includes records of driver behavior and road conditions during driving, and the neural network is trained for autonomous driving.

[0014] On the other hand, the demonstration data includes records of human movement paths and the human's surrounding environment during movement, and wherein the neural network is trained for robot control.

[0015] In one aspect, a method is disclosed for controlling an agent using a neural network model trained by a method as described in one embodiment of the present invention. The method includes: the agent collecting environmental data related to the task; obtaining a current state based on the environmental data; inferring an action for the current state using the trained neural network model; and controlling the agent based on the inferred action for the current state.

[0016] In one aspect, a vehicle capable of autonomous driving is disclosed. The vehicle includes: one or more sensors configured to collect at least a portion of environmental data related to autonomous driving of the vehicle; one or more processors; and one or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform operations as described in one embodiment of the embodiments disclosed herein.

[0017] In one aspect, a robot is disclosed. The robot includes: one or more sensors configured to collect at least a portion of environmental data surrounding the robot; one or more processors; and one or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform operations as described in one embodiment of the embodiments disclosed herein.

[0018] In one aspect, a computer system is disclosed. The computer system includes one or more processors; and one or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of one of the methods disclosed herein.

[0019] In one aspect, one or more computer-readable storage media are disclosed that store computer-executable instructions, which, when executed, cause one or more processors to perform the operation of one of the methods disclosed herein.

[0020] In one aspect, a computer program product including computer-executable instructions is disclosed, which, when executed, cause one or more processors to perform the operation of one of the methods disclosed herein. Attached Figure Description

[0021] The disclosed aspects will be described in conjunction with the accompanying drawings, which are provided for illustrative purposes and not for limiting the scope of the disclosure.

[0022] Figure 1 An exemplary block diagram 100 of an example apparatus according to various aspects of this disclosure is shown.

[0023] Figure 2 An exemplary block diagram 200 of GAIL according to various aspects of this disclosure is shown.

[0024] Figure 3 An exemplary flowchart 300 is shown, according to various aspects of this disclosure, for training a neural network model to perform behavioral imitation using Generative Adversarial Imitation Learning (GAIL).

[0025] Figure 4 An exemplary flowchart 400 is shown, illustrating various aspects of this disclosure for controlling an agent using a trained neural network model.

[0026] Figure 5 An exemplary computer system 500 according to various aspects of this disclosure is shown. Detailed Implementation

[0027] This disclosure will now be discussed with reference to several exemplary implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement embodiments of this disclosure, and not to impose any limitation on the scope of this disclosure.

[0028] Various embodiments will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References to examples and embodiments are for illustrative purposes and are not intended to limit the scope of this disclosure. It should be noted that the word "based on" as used in this disclosure should be understood as "at least based on" and not "based on only" or "only based on".

[0029] Applying the methods disclosed herein may involve using user-related information, such as driving routine information during movement or driving, road conditions, and user behavior records. It should be noted that the use of user-related information requires user authorization and may not exceed the scope of that authorization.

[0030] In reinforcement learning (RL), the optimal policy is typically learned by evaluating cumulative rewards, which is simple, straightforward, and performs well when large amounts of training data are available. However, in sequential decision-making situations, learners cannot frequently receive rewards for each action, and methods based on cumulative rewards have a huge search space. Imitation learning (IL) has been developed to efficiently solve multi-step decision-making problems.

[0031] There are two main approaches to Inverse Reinforcement Learning (IL): Behavior Cloning (BC), which learns a policy from state-action pairs derived from expert trajectories as a supervised learning problem; and Inverse Reinforcement Learning (IRL), which finds the cost function under which the expert is uniquely optimal. However, BC tends to succeed only when using large amounts of data, and IRL learns a cost function that explains the expert's behavior but does not directly tell the learner how to act. Given that the learner's true goal is often to imitate the expert's actions, a framework called Generative Adversarial Imitation Learning (GAIL) is proposed, utilizing the backbone of Generative Adversarial Networks (GANs) to tell the agent how to act by directly learning policies from experts.

[0032] Figure 1 An exemplary block diagram 100 of an example apparatus according to various aspects of this disclosure is shown.

[0033] Figure 1 The illustrated device 100 may be a vehicle such as an autonomous vehicle, a self-controlled machine such as a robot, an intelligent agent capable of mimicking an expert's demonstration of a specific task, or may be part of a vehicle, robot, intelligent agent, etc. Autonomous vehicles are considered in the following description. Figure 1 Examples of devices in the text, but not limited to these.

[0034] The vehicle may be equipped with sensors 110 to sense information related to the road environment or surrounding environment in which the vehicle is traveling. The terms road environment and / or surrounding environment may also be collectively referred to as state, condition, situation, context, etc. Figure 1 Only one sensor 110 is shown in the diagram, but more than one sensor is possible. In one embodiment, sensor 110 may be one or more of a camera, LiDAR, radar, sonar, ultrasonic sensor, proximity sensor, infrared sensor, wheel speed sensor, rain sensor, and / or any combination thereof. It is possible to equip sensor 110 with other suitable types of sensors on the vehicle.

[0035] The vehicle may include a processing system 120. The processing system 120 may be implemented in various ways, for example, the processing system 120 may include one or more processors and / or controllers and one or more memories, the processors and / or controllers may execute software to perform various operations or functions, such as the operations or functions according to various aspects of this disclosure.

[0036] The processing system 120 can receive sensing data from the sensor 110 and perform various operations by analyzing the sensing data. Figure 1In the example, the processing system 120 may include a state detection module 120-1 and an action determination module 120-2. It should be understood that modules 120-1 and 120-2 may be implemented in various ways, for example, they may be implemented as software modules or functions that can be executed by a processor and / or a controller.

[0037] The state detection module 120-1 can be configured to detect a state upon which the vehicle will perform an action. The state can refer to weather, the vehicle's absolute speed, the presence of an obstacle, the relative speed to an obstacle (such as a car or pedestrian), the distance to the obstacle, the azimuth relative to the obstacle, and / or any combination thereof. It should be understood that the state includes other suitable types of information, such as navigation information from a navigation system. In one embodiment, state information can be collected by sensor 110 and provided to processing system 120, and the state detection module 120-1 can detect the state based on the state information.

[0038] The action determination module 120-2 can be configured to determine the action to be performed by the vehicle based on the state from the state detection module 120-1. The action may refer to acceleration or deceleration, braking, steering, reversing, illuminating turn signals, activating hazard lights, and / or any combination thereof. In one embodiment, the action to be performed may be indicated by the action determination module 120-2 to the corresponding control unit of the vehicle.

[0039] Alternatively, the action determination module 120-2 can be implemented using a trained neural network model trained to mimic the behavior of a human driver in order to make highly similar decisions when faced with similar situations. In one embodiment, the action determination module 120-2 can obtain the state of the current time step from the state detection module 120-1 and infer the action to be performed for the current time step based on the obtained state.

[0040] Figure 1 This is shown as an example only, and other implementations are possible.

[0041] Figure 2 An exemplary block diagram 200 of GAIL according to various aspects of this disclosure is shown. Figure 2 In the middle, box 201 can represent the policy generator. And box 202 can represent a discriminator. .

[0042] In one embodiment, GAIL can be used to train a neural network model, which can be used as described above. Figure 1 The action determination module 120-2 described in the document.

[0043] In one embodiment, an expert demonstration can be used to train a neural network model, and the expert demonstration may include multiple trajectories. The trajectory includes multiple state-action pairs. Similar to the description of the state detection module 120-1 and the action determination module 120-2, state and action can refer to various types of information. Still using an autonomous vehicle as an example, the trajectory could be a driving record from a skilled driver, which includes information relative to time. A series of state-action pairs This reflects the conditions encountered by the vehicle along the trajectory and the corresponding decisions made by the driver. The complete set of state-action pairs can be named the policy. The goal of training is to make the neural network model perform the same actions when faced with the same state.

[0044] like Figure 2 As shown, state It can be sampled from the training dataset or generated randomly. Then, the state can be... Input into box 201 (Strategy Generator) ), based on the parameters of the state and policy generator Generate action Then you can put the state-action pairs... Input to box 202 (Discriminator) ), based on the parameters of the discriminator To determine whether the received state-action pair is an expert operation.

[0045] Similar to its backbone GAN, GAIL is trained by alternately training its discriminator 202 and policy generator 201. The discriminator 202 is trained to distinguish the generated policy from the expert policy as clearly as possible. Conversely, the policy generator 201 is trained to generate policies that are close to expert policies to the point of confusing the trained discriminator 202. GAIL's objective function is: (1) in, It is in state and actions The discriminator used below, It represents the generator's strategy. Given an expert strategy, and It is a strategy causal entropy.

[0046] The objective functions for the discriminator and generator, respectively, can be written as: (2) (3) Equation (2) aims to make the discriminator output the generation strategy. And for expert strategy output Equation (3) aims to minimize the effect of the agent's exploitation of the policy. The distribution of state-action pairs encountered in navigation environments and the expert's strategy utilization. Differences between state-action pairs encountered in navigation environments.

[0047] After training a neural network with GAIL, the ideal solution would be for the discriminator to be unable to distinguish between the generated policy and the expert policy, i.e., Furthermore, the distribution of the generated strategies is the same as that of the expert strategies.

[0048] Figure 2 This is shown as an example only, and other implementations are possible.

[0049] Unfortunately, GAIL inherits the unstable nature of GANs; its training process exhibits oscillating behavior and fails to reach optimal equilibrium. In this disclosure, to bring GAIL towards equilibrium, a modeling of GAIL as a dynamic system is presented, employing a controller within control theory for both the policy generator and discriminator.

[0050] First, the variational method is used to obtain the objective functions (2) and (3) relative to time. The derivatives of the equations are used to obtain the differential equations representing the training dynamics of the discriminator and the generator: (4) (5) in, and They represent time. The discriminator network and generator network are both time-dependent. And it is introduced by change. Because of The time is constant during training dynamics, therefore time can be omitted for expert policies. .

[0051] For simplicity, let's define... , and The following dynamic equations can be obtained: (6) (7)

[0052] As mentioned above, the ideal goal of GAIL is to estimate expert policies using policy generators. This makes it impossible for the discriminator to distinguish between expert policies and generated policies, which can be represented as and ,along with It becomes infinity. Ideally, it is expected to: (8) (9)

[0053] In other words, the objective function for the discriminator and the policy generator can be defined as: (10) in, .

[0054] Based on the definition of equilibrium in control theory, if Then point It is a system The equilibrium. This equilibrium can also be called a fixed point, critical point, or steady state. As can be seen from equations (7) and (10), The inability to make equation (7) equal to 0 indicates that GAIL cannot converge to the objective function, which means that the policy generator cannot converge to the expert policy.

[0055] Therefore, based on control theory that treats the training of GAIL as a dynamic system, a method for pushing GAIL to its objective function via a controller is disclosed: (11) (12) in, and These are the controllers used for the discriminator and the policy generator, respectively.

[0056] Since the derivative of the discriminator with respect to time evaluated at the objective function can be equal to 0 using equations (6) and (10), the discriminator can approach its objective function. Therefore, it is disclosed that... A linear controller is designed for use in the discriminator to keep equation (11) equal to 0 at the objective function. Equation (13) below is shown as an example and not a limitation; any linear controller capable of keeping equation (11) equal to 0 at the objective function can be applied. (13)

[0057] On the other hand, using equations (7) and (10), the derivative of the policy generator with respect to time evaluated at its objective function is not equal to 0, as previously stated. Therefore, The disclosed equation (12) is designed to be adjusted to 0 at the objective function. The following equation (14) is shown as an example and not a limitation, and any controller that can make equation (12) equal to 0 at the objective function can be applied. (14)

[0058] By using a controller from a dynamic system perspective to push GAIL toward a given objective function, the loss function of GAIL can be adjusted based on the modified training dynamics, such that both the discriminator and policy generator of GAIL converge to their ideal optimal solutions. In one embodiment, the loss function can be adjusted according to the modified training dynamics of equations (11)-(14) as follows: (15) (16)

[0059] Equations (15)-(16) are merely examples and may vary depending on the controller used.

[0060] To proceed with further steps, in addition to making the policy generator converge to the expert policy, it is expected that the policy generator can be stabilized under the expert policy.

[0061] Example controller and For example, let's analyze further. and These are hyperparameters introduced by the controller to avoid oscillation. and The scope can have specific boundaries.

[0062] It is revealed that the training dynamics of GAIL should be constrained by the principle of asymptotic stability in order to keep the dynamic system stable around equilibrium. Taking equations (13)-(14) as an example, and It should be subject to the following inequalities: (17)

[0063] The inequalities above are merely examples and may vary depending on the controller used.

[0064] By using a constrained controller as disclosed herein, GAIL's discriminator and policy generator can converge to their ideal optimal solutions and remain stable around these optimal solutions. That is, as training progresses, the policy generator is able to converge to an expert policy and remain stable as time reaches infinity.

[0065] Figure 3 An exemplary flowchart is shown for training a neural network model for behavioral imitation using Generative Adversarial Imitation Learning (GAIL) according to various aspects of this disclosure. As described below, some or all of the features shown may be omitted in implementations within the scope of this disclosure, and some of the shown features may not be required for all implementations. Furthermore, some blocks may be executed in parallel or in a different order. In some examples, the method may be performed by any suitable means or unit for performing the functions or algorithms described below.

[0066] The method begins at box 301, where presentation data representing the presenter’s behavior toward the task is obtained, wherein the presentation data includes a sequence of state-action pairs, wherein the state corresponds to a condition for a decision and the action corresponds to a decision made by the presenter.

[0067] The method then proceeds to box 302, where the discriminator of the neural network is updated to maximize a first objective function, wherein the first objective function is modified using a first term that is used to make the discriminator indistinguishable between the demonstrator’s behavior and the state-action pairs generated by the policy generator of the neural network.

[0068] The method then proceeds to box 303, where the policy generator is updated to minimize a second objective function, which is modified using a second term to push the policy generator to converge to the presenter's behavior.

[0069] In one embodiment, the first and second terms are determined based on the first and second controllers, respectively, to push the dynamic system corresponding to the discriminator and the policy generator to the objective function corresponding to the equilibrium of the dynamic system.

[0070] In one embodiment, the first term and the second term can be determined as equations (13)-(14).

[0071] In one embodiment, the dynamic system corresponding to the discriminator and the policy generator is determined by the derivatives of the first objective function and the second objective function with respect to time.

[0072] In one embodiment, the dynamic system can be defined as equations (4)-(5).

[0073] In one embodiment, the dynamic system reaches equilibrium when the output of the discriminator is equal to 1 / 2 and the output of the policy generator matches the behavior of the presenter.

[0074] In one embodiment, the equilibrium of a dynamic system can be described by equations (8)-(9).

[0075] In one embodiment, the first and second controllers are constrained by the principle of asymptotic stability in order to keep the dynamic system stable around equilibrium.

[0076] In one embodiment, the first controller and the second controller are subject to inequality (17).

[0077] Figure 3 This is shown as an example only, and other implementations are possible.

[0078] In one aspect of this disclosure, a method for training a neural network model for autonomous driving is proposed. The method for training the neural network model can be used with any of the embodiments described herein, such as those referenced... Figure 2-3 The described embodiments include demonstration data comprising records of the driver's behavior and the road environment during driving. Autonomous driving can also be referred to as intelligent driving, AI driving, etc.

[0079] In one aspect of this disclosure, a method for training a neural network model for robot control is proposed. The method for training the neural network model can be used with any of the embodiments described herein, such as those referenced... Figure 2-3 The described embodiments include demonstration data comprising human movement paths and records of the human's surrounding environment during movement. Robot control can include various actions of robot movement, such as walking, jumping, searching for paths across obstacles, and robotic arms performing tasks like humans.

[0080] Figure 4 An exemplary flowchart 400 for controlling an agent using a trained neural network model according to various aspects of this disclosure is shown. As described below, some or all of the shown features may be omitted in implementations within the scope of this disclosure, and some shown features may not be required for all embodiments. Furthermore, some blocks may be executed in parallel or in a different order. In some examples, the method may be performed by any suitable means or unit for performing the functions or algorithms described below.

[0081] The method begins at box 401, where an agent collects task-related environmental data.

[0082] In one embodiment, Figure 1 The sensor 110 described herein can be used to collect environmental data.

[0083] Then, the method proceeds to box 402, where the current state is obtained based on environmental data.

[0084] In one embodiment, Figure 1 The state detection module 120-1 described herein can be used to obtain the current state based on environmental data.

[0085] The method then proceeds to box 403, where a trained neural network model is used to infer the action for the current time instance based on the state of the current time instance.

[0086] In one embodiment, Figure 1 The action determination module 120-2 described herein can be used to infer the action for the current time instance based on the state of the current time instance using a trained neural network model.

[0087] The method then proceeds to box 404, where the agent is controlled based on the inferred action for the current time instance.

[0088] In one embodiment, instructions for controlling the intelligent agent can be transmitted by the action determination module 120-2.

[0089] Figure 4 This is shown as an example only, and other implementations are possible.

[0090] In one aspect of this disclosure, a vehicle capable of autonomous driving is provided. For example, such as Figure 1 As shown, the vehicle includes: one or more sensors configured to collect at least a portion of environmental data related to autonomous driving of the vehicle; one or more processors; and one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of the methods described in the embodiments of this disclosure.

[0091] In one aspect of this disclosure, a robot is provided. For example, such as Figure 1 As shown, it can also represent the structure of a robot, which includes: one or more sensors configured to collect at least a portion of environmental data around the robot; one or more processors; and one or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform the operations of the methods described in the embodiments of this disclosure.

[0092] Figure 5An exemplary computer system 500 according to various aspects of this disclosure is illustrated. The computer system may include at least one processor 510. The computer system may also include at least one storage device 520. It should be understood that the storage device 520 may store computer-executable instructions that, when executed, cause the processor 510 to perform actions as described above. Figure 1-4 Any operation described according to embodiments of this disclosure.

[0093] Embodiments of this disclosure may be embodied in one or more computer-readable media, such as non-transitory computer-readable media. A non-transitory computer-readable medium may store computer-executable instructions that, when executed, cause one or more processors to perform actions as described above. Figure 1-4 Any operation described according to embodiments of this disclosure.

[0094] Embodiments of this disclosure can be embodied in a computer program product including computer-executable instructions that, when executed, cause one or more processors to perform according to the combination Figure 1-4 Any operation of the embodiments described in this disclosure.

[0095] It should be understood that all operations in the above methods are merely exemplary, and this disclosure is not limited to any operation in the methods or the order of such operations, and should cover all other equivalents under the same or similar concepts.

[0096] It should also be understood that all modules in the aforementioned apparatus can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any of these modules can be further functionally divided into sub-modules or combined together.

[0097] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit them to the aspects shown herein. All structural and functional equivalents of elements throughout the various aspects described in this disclosure that are known or will be known hereafter by those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A computer-implemented method for training a neural network model for behavioral imitation using Generative Adversarial Imitation Learning (GAIL), the method comprising: Obtain demonstration data representing the demonstrator's behavior in response to a task, wherein the demonstration data includes a sequence of state-action pairs, where a state corresponds to a condition used for decision-making, and an action corresponds to a decision made by the demonstrator. The discriminator of the neural network is updated to maximize a first objective function, wherein the first objective function is modified using a first term that drives the discriminator to become indistinguishable between the demonstrator's behavior and the state-action pairs generated by the policy generator of the neural network; and The policy generator is updated to minimize a second objective function, which is modified using a second term that drives the policy generator to converge to the behavior of the demonstrator.

2. The computer-implemented method of claim 1, wherein, The first and second items are determined based on the first controller and the second controller, respectively, which push the dynamic system corresponding to the discriminator and the policy generator to the objective function corresponding to the equilibrium of the dynamic system.

3. The computer-implemented method of claim 2, wherein, The dynamic system corresponding to the discriminator and the policy generator is determined by the derivatives of the first objective function and the second objective function with respect to time.

4. The computer-implemented method according to claim 2, wherein, The dynamic system reaches equilibrium when the output of the discriminator is equal to 1 / 2 and the output of the policy generator matches the behavior of the presenter.

5. The computer-implemented method according to claim 2, wherein, The first and second controllers are constrained by the principle of asymptotic stability in order to keep the dynamic system stable near the equilibrium.

6. The computer-implemented method according to claim 1, wherein, The demonstration data includes records of driver behavior and road conditions during driving, and the neural network is trained for autonomous driving.

7. The computer-implemented method according to claim 1, wherein, The demonstration data includes records of the human's movement path and the human's surrounding environment during movement, and the neural network is trained for robot control.

8. A method for controlling an agent using a neural network model trained by the method as described in any one of claims 1 to 7, comprising: The intelligent agent collects environmental data related to the task; The current state is obtained based on the environmental data; The action for the current moment is inferred based on the state at the current moment using a trained neural network model; as well as The agent is controlled based on the inferred action for the current moment.

9. A vehicle capable of autonomous driving, comprising: One or more sensors are configured to collect at least a portion of environmental data related to the autonomous driving of the vehicle; One or more processors; as well as One or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the method as described in any one of claims 1 to 8.

10. A robot comprising: One or more sensors are configured to collect at least a portion of environmental data surrounding the robot; One or more processors; as well as One or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the method as described in any one of claims 1 to 8.

11. A computer system, comprising: One or more processors; as well as One or more storage devices storing computer-executable instructions, which, when executed, cause the one or more processors to perform the operations of the method as described in any one of claims 1 to 10.

12. One or more computer-readable storage media storing computer-executable instructions, which, when executed, cause one or more processors to perform the operations of the method as described in any one of claims 1 to 10.

13. A computer program product comprising computer-executable instructions, which, when executed, cause one or more processors to perform the operations of the method as described in any one of claims 1 to 10.

Images