Agent control method
Patent Information
- Application Number
- JP2022168137
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-21
- Filing Date
- 2022-10-20
- Publication Date
- 2025-08-13
- Estimated Expiration
- 2042-10-20
AI Technical Summary
Current machine learning methods for predicting human behavior in autonomous systems face challenges due to the diversity of human intentions and preferences, requiring precise hyperparameter tuning and complex training procedures, especially in Variational Auto-Encoders (VAEs), which are difficult to apply effectively.
A neural network-based method that trains on agent behavior data, incorporating individual agent-specific parameters and using Gaussian mixture models for efficient prediction, allowing for robust and adaptable behavior forecasting without the need for complex architectures.
Enables efficient and robust prediction of diverse human behaviors, facilitating safe operation of autonomous systems by simplifying hyperparameter tuning and enabling quick adaptation to new conditions.
Smart Images
Figure 00000010_0000 
Figure 00000010_0001 
Figure 00000011_0000
Abstract
Description
[Technical Field]
[0001] This disclosure relates to a method for controlling an agent. [Background technology]
[0002] In the field of autonomous systems, a crucial task is to reliably control an agent and predict the behavior of objects (e.g., people) moving around a controlled agent (e.g., a robot) for purposes such as collision avoidance.
[0003] In particular, predicting human behavior within a short timeframe of 1 to 10 seconds is a fundamental task. Hospital robots, for example, must predict the behavior of people such as nurses, doctors, and patients to ensure timely space in aisles. In the case of autonomous driving, vehicles must anticipate other road users cutting into their lanes. Currently, the most accurate prediction methods almost without exception fall into the category of machine learning.
[0004] One of the challenges in this task is that human behavior is influenced by individual desires, preferences, and intentions, and therefore can be extremely diverse. Some drivers, for example, prefer to match the speed of a slower vehicle in front rather than overtake it, while others want to avoid wasting time and would rather overtake. Therefore, accurate prediction requires accommodating this variability in behavior, and in the case of machine learning approaches, this must be learned from the data.
[0005] For this purpose, according to the current leading methods in machine learning, a variation of the Variational Auto-Encoder (VAE) approach is used. The premise in this case is that the variable to be predicted, y ~ p(.|x,z), depends not only on the input data x, but also on a latent variable z that follows a prior distribution p(z). This variable does not exist directly in the training data, but must be estimated. For this purpose, the posterior distribution p(z|x,y) is considered. For this training, p(y|x,z) is used in the decoder network g θ (y,x,z) determines the recognition network h Ψ Modeled by (z,x,y), and further minimizing the Evidence Lower Bound (ELBO) ELBO(x,y,θ,Ψ)=-KL(h Ψ (z,x,y)||p(z))+E z~hΨ(.,x,y) [log(g θ The model is trained using (y,x,z)), where KL represents the Kullback-Leibler divergence between distributions and is extracted to regularize the latent variables or their distributions. The second part of the ELBO is called the reconstruction loss and is used to train the decoder to make the best possible predictions.
[0006] This approach allows for very good modeling of several complex probability distributions. However, successfully applying this methodology is extremely difficult. For example, the problem of overregularization in VAEs arises, requiring precise tuning of the algorithm's hyperparameters, particularly the weighting of the Kullback-Leibler term, in ELBOs. Moreover, training VAEs is more difficult than training conventional neural networks because additional random variables must be extracted during training to calculate the reconstruction loss. [Overview of the project] [Problems that the invention aims to solve]
[0007] Therefore, a control method for the agent is desired, which includes efficient prediction of objects (detected in the agent's vicinity). [Means for solving the problem]
[0008] Disclosure of the invention A method for controlling agents according to various embodiments is provided, the method comprising: training a neural network using training data containing samples of the behavior of a number of agents, wherein the output of the neural network has predictions of the behavior, and this output depends on network parameters that are commonly trained for all training data and on further parameters that are individually trained for each of the number of agents; fitting a probability distribution to the values of the further parameters for the agent resulting from the training; sampling values from a probability distribution for further agents around the agent; and controlling the agent, taking into account the predictions of the behavior of the further agents that the neural network outputs for the sampled values of the further agents.
[0009] The method described above allows for the generation of various possible behavioral patterns of other agents (especially human agents) in the context of controlling an autonomous system. In this way, when controlling an autonomous system, it is possible to determine trajectories or behavioral patterns that avoid collisions with other agents or other interference from other agents. On the other hand, various behaviors of human agents can be generated during the development phase of the autonomous system, and in this way, it is possible to verify that the autonomous system is functioning properly in simulation-based evaluation (i.e., simulated control).
[0010] The method described above allows for the consideration of object behavior when controlling an agent by using effectively trainable behavior predictions. This method can be implemented using established components (neural networks, probabilistic models), where hyperparameter tuning is significantly simpler than, for example, the VAE approach, and training is more robust overall. This allows for the rapid and easy retraining of the model without expert knowledge for new application scenarios, such as predicting driver behavior in other countries; in other words, a single agent can be quickly adapted to modified conditions, thereby enhancing safety, for example, in autonomous driving applications.
[0011] Various examples are given below.
[0012] Example 1 is a method for controlling a robot as described above.
[0013] Example 2 is the method described in Example 1, in which the neural network is trained to map its input to predict the behavior of a certain agent, and the input to the neural network includes state information of the agent whose behavior the neural network should predict, and further parameters.
[0014] In other words, the dependency between the output of a neural network and additional parameters can be easily achieved by adding additional parameters to the input of the neural network (as a trainable portion of the input). Thus, no special network architecture is required for implementation, and existing network architectures with the corresponding input dimensions can be used.
[0015] Example 3 is the method described in Example 2, in which the input to the neural network includes state information relating to a control state in which the behavior of the agent should be predicted.
[0016] Therefore, the neural network can incorporate the state of the control situation into its prediction. The input to the neural network can be, for example, traffic situation information in the form of a list of vehicles with positions and speeds, extended by additional parameters.
[0017] Example 4 is the method described in any one of Examples 1 to 3, where the probability distribution is a Gaussian mixture model.
[0018] For the purpose of modeling complex distributions, the Gaussian mixture model can be efficiently trained and sampled from.
[0019] Example 5 is the method described in any one of Examples 1 to 4, which includes the step of training network parameters and additional parameters by adapting the network parameters and additional parameters to minimize the loss between the samples of the agent's behavior and the behavior predicted by the neural network respectively.
[0020] In other words, additional parameters are incorporated into the training of the neural network. However, in this case, the additional parameters have different (trainable) values for the training samples of different agents, because these parameters are trained individually for each agent whose behavior samples are included in the training data.
[0021] Example 6 is a control device configured to implement the method described in any one of Examples 1 to 5.
[0022] Example 7 is a computer program that, when executed by a processor, includes instructions for causing the processor to operate such that the processor performs the method described in any one of Examples 1 to 5.
[0023] Example 8 is a computer-readable medium that stores instructions for operating a processor so that, when executed by the processor, the processor performs the method described in any one of Examples 1 to 5.
[0024] In the drawings, similar reference numerals generally relate to the same parts in all different viewpoints. The drawings are not necessarily to scale, and instead the focus is generally on illustrating the principles of the invention. In the following description, various embodiments will be described with reference to the following drawings. [Brief explanation of the drawing]
[0025] [Figure 1] This is a diagram of a vehicle. [Figure 2] This figure shows one example of prediction using a neural network. [Figure 3] This diagram shows a flowchart illustrating a control method for a robot that picks up and inspects an object, according to one embodiment. [Modes for carrying out the invention]
[0026] The following detailed description relates to the accompanying drawings illustrating specific details and embodiments of the present disclosure that enable the implementation of the present invention. Other embodiments may be used, and structural, logical, and electrical modifications may be implemented without departing from the scope of protection of the present invention. Since some embodiments of the present disclosure may be combined with one or more other embodiments of the present disclosure for the purpose of forming new embodiments, the various embodiments of the present disclosure are not necessarily mutually exclusive.
[0027] Various examples will be described in more detail below.
[0028] Figure 1 shows vehicle 100.
[0029] In the embodiment shown in Figure 1, the vehicle control device 102 is provided on the vehicle 100, for example, a passenger car or a truck.
[0030] The vehicle control device 102 includes data processing components, such as a processor (for example, a CPU (central unit)) 103, control software for operating the vehicle control device 102, and a storage device 104 for storing data processed by the processor 103.
[0031] For example, stored control software (computer program) includes instructions for causing the processor 103 to operate so that the processor 103 implements one or more neural networks 107 when the processor executes it.
[0032] The data stored in the storage device 104 may include, for example, image data captured by one or more cameras 105. One or more cameras 105 can, for example, take one or more grayscale or color photographs of the vehicle 100.
[0033] The vehicle control device 102 can use image data (or similar data from other sources, such as other types of sensors, or similar vehicle-to-vehicle communication) to detect objects 108 around the vehicle 100, in particular other vehicles, pedestrians i.e., people, or animals.
[0034] The vehicle control device 102 can inspect sensor data and control the vehicle 100 according to the results, that is, it can determine control actions for the vehicle and signal to the individual actuators of the vehicle. Therefore, the vehicle control device 102 can control actuators 106 (e.g., brakes) for purposes such as controlling the vehicle's speed or braking the vehicle.
[0035] In cases where there are movable objects 108, that is, objects that move on their own, in the surrounding area, what is required for effective control is that the vehicle control device 102 predicts how those objects 108 will move. For example, it is desirable that the vehicle control device 102 be able to predict whether a person will step onto the road or whether another vehicle will turn.
[0036] For this purpose, the vehicle control device 102 performs predictions of human behavior, such as that of a driver or pedestrian, but also performs predictions of the behavior of animals, for example, or, in some cases, other autonomous devices. These moving objects are also considered, in the following terms, to be other agents present around the controlled agent. A training dataset is used to train the behavior prediction model, and this training dataset includes data (i.e., training samples) representing the behavior of other agents, and this data includes, for example, the trajectories of agents moving in various traffic conditions.
[0037] According to various embodiments, an approach based on the premise that the variability in behavior is defined, in particular, by the different preferences of human agents is used for behavior prediction. These preferences can, for example, be assumed to be constant over a period of 1 to 10 seconds. Thus, according to various embodiments, the prediction model realized by the decoder in the VAE-based approach is extended only by the preference vector z. However, in this case, the complex architecture required for training in the VAE approach is avoided. In contrast, the vector z is treated as a trainable model parameter z a that is specific to each other agent a in the training dataset in which the data is included. There are multiple training samples within the training data for each agent, and as long as the dimension of z is sufficiently small (as is common in, for example, VAE) and regularization techniques such as weight decay are used, these z a can be learned.
[0038] For new agents, i.e., agents not included in the training dataset, in order to be able to predict behavior, the learned values of z a are collected after training, and in an additional second step, a distribution model is estimated for the dataset of z a . For this, various methods established to fit a probability model capable of efficiently extracting random variables, such as a Gaussian mixture model, to the data can be used. In doing so, the possible preferences for a new agent can be identified by sampling from such a probability distribution, and possible behaviors can be generated using the prediction model.
[0039] That is, instead of the complex training procedure as in the case of the VAE-based prediction approach, in two steps, relatively simple training methods each based on an established stable routine are used.
[0040] Accordingly, according to various embodiments, the control device of an agent, for example, in the case of a vehicle 100, the vehicle control device 102, performs behavioral prediction for other agents. That is, this means that the control device predicts the behavior y of agent a (different from the controlled agent) based on input data x. In this case, the variable x can be a vector of input data, a matrix including a time series of input data, or a multichannel image, depending on the specific application. The same applies to the output y.
[0041] For this purpose, according to various embodiments, the following prediction model is used. That is, the input data x of this prediction model (for example, representing a position) is element z a It is only expanded, and the value of this element is learned individually for each agent in which the training data contains training samples. In the case of a vector or time series of input data, z a The value (or in some cases, a vector of values) can be easily attached to x. If the input data is an image, z a Depending on the value of [the parameter], an additional channel with a fixed value can be provided.
[0042] The input data for this extended predictive model is:
number
number
number
number
number
number
[0043] Figure 2 shows one example of prediction using neural network 201.
[0044] The input data is a state vector 202 representing the state of the agent whose trajectory should be predicted. Each state vector is z a The value of is extended by 203. The resulting extended state vector 204 forms the input to the neural network 201, which in this embodiment receives a set of such extended state vectors 204 (i.e., time series 207) and predicts the trajectory 205 from it. In other words, in this case, the output of the neural network is information about the trajectory 205. a The value 203 is sampled from a probability distribution for the current prediction (or for the agent as well), and in this case, this value is constant for all state vectors 204 used for the current prediction. This can be kept constant for all predictions associated with the same agent, or it can be sampled anew for each prediction (even if it is associated with the same agent).
[0045] To train the neural network 201 (along with the probability distribution 206), first, for each agent in the training data where a training sample exists, z a The (initial) value of is randomly sampled, and this z a The neural network 201 is trained (individually for each agent) along with its original parameters (i.e., network parameters, typically weights). Depending on the application, various loss functions such as cross-entropy loss can be used. After this training, for each agent for which a training sample exists in the training data, z a The individual behavior of each agent is encoded by the value of g(z). Here, in order to be able to predict the behavior of new agents, that is, agents for which there are no training samples in the training data, g(z) is used. a The variable z represented by ) a The probability distribution 206 is modeled.
[0046] For this purpose, any method can be used to fit the probability distribution to the data supplying the probability distribution, and this allows for, for example, a Gaussian mixture model, z a The value of z can be efficiently sampled. Then, for a new agent, z a Sample the value of,
number
[0047] Therefore, according to various embodiments, the following is performed for training and prediction:
[0048] training 1. Neural Networks
number
number
number
number
[0049] Implementation of predictions 1. For further agents, the trained probability distribution model g(z) a ) from z a Extract the value. 2. Extended Input
number
[0050] In summary, various embodiments provide a method as shown in Figure 3.
[0051] Figure 3 shows a flowchart 300 illustrating a control method for a robot that picks up and inspects an object, according to one embodiment.
[0052] In 301, a neural network is trained using training data containing samples of the behavior of a large number of agents, in which case the output of the neural network has predictions of behavior, and this output depends on network parameters that are commonly trained for all training data, and on further parameters that are individually (i.e., uniquely) trained for each of the large number of agents.
[0053] In step 302, a probability distribution is fitted to the values of further parameters for the agent that result from the training.
[0054] In step 303, a value is sampled (i.e., extracted) from the probability distribution of further agents in the vicinity of the agent in question.
[0055] In step 304, the agent is controlled by considering further agent behavior predictions output by the neural network based on the sampled values of the agent.
[0056] The method shown in Figure 3 can be carried out by one or more computers equipped with one or more data processing units. The term “data processing unit” can be understood as any kind of entity that enables the processing of data or signals. Data or signals can be processed, for example, according to at least one (i.e., one or more) special functions performed by the data processing unit. A data processing unit may include, or consist of, analog circuits, digital circuits, logic circuits, microprocessors, microcontrollers, central units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate array (FPGA) integrated circuits, or any combination thereof. Any other method for implementing the individual functions described in more detail herein can also be understood as a data processing unit or logic circuit device. One or more of the method steps described in detail herein can be performed (e.g., implemented) by a data processing unit, through one or more special functions performed by that data processing unit.
[0057] Various embodiments can receive sensor signals from various sensors such as video, radar, LiDAR, ultrasound, motion, acceleration, and thermography, and can be used, for example, to acquire sensor data for detecting objects (i.e., other agents) as input to a neural network that predicts their behavior.
[0058] A neural network can output either a single continuous range of values or multiple values belonging to a single continuous range. For example, it can perform regression on the future behavior of a human agent whose behavior is represented in its output by one or more positions, trajectories, or occupation states.
[0059] The embodiments can be used to train machine learning systems and to control agents, such as physical systems like robots or vehicles. In particular, the embodiments can be applied to control and monitor the execution of operational tasks, for example, on an assembly line.
[0060] The controlled agent can be a robotic device, meaning that control signals for the robotic device can be formed. The term "robotic device" can be interpreted as relating to some physical system itself (equipped with mechanical parts whose motion is controlled), such as computer-controlled machines, vehicles, household appliances, power tools, manufacturing machinery, personal assistants, or access control systems. Control rules for the physical system are learned, and the physical system is controlled accordingly.
[0061] In the case of autonomous vehicles or mobile robots, the robotic device detects pedestrians (e.g., using a LiDAR sensor) and calculates their positions and velocities. Each control device (e.g., vehicle control device 102) samples possible preferences for each pedestrian and uses a trained predictive model (i.e., a trained neural network) to predict the pedestrian's future trajectory from their current position.
[0062] However, the aforementioned approach can be applied to all kinds of agents (for example, agents that are merely simulated and do not physically exist).
[0063] While specific embodiments have been illustrated and described herein, it will be obvious to those skilled in the art that these specific embodiments can be replaced with a variety of alternative and / or equivalent implementations without departing from the scope of protection of the present invention. This application encompasses any adaptation or modification of the specific embodiments discussed herein. Therefore, the present invention is intended to be limited only by the claims and their equivalents.
Claims
1. A method for controlling an agent, comprising: training a neural network for a plurality of agents using training data comprising examples of the agents' behavior, the output of the neural network comprising a prediction for behavior, the output depending on network parameters commonly trained on all the training data, and on further parameters individually trained for each agent of the plurality of agents; fitting a probability distribution to values of the further parameters for the agent resulting from the training; sampling values from the probability distribution for further agents in a neighborhood of the agent; controlling said agent taking into account a behavior prediction of said further agent output by said neural network for said sampled values of said further agent; A method for controlling an agent, including:
2. 2. The method of claim 1, wherein the neural network is trained to map inputs of the neural network to a behavior prediction for an agent, the inputs of the neural network including state information for the agent whose behavior the neural network is to predict and the further parameter.
3. The method of claim 2 , wherein the inputs of the neural network include state information regarding a control situation for which the behavior of the agent is to be predicted.
4. The method of claim 1 , wherein the probability distribution is a Gaussian mixture model.
5. 2. The method of claim 1, comprising training the network parameters and the further parameters by adapting the network parameters and the further parameters to minimize a loss between samples of the agent's behavior and the behavior predicted by the neural network, respectively.
6. A control device configured to implement the method according to any one of claims 1 to 5.
7. A computer program comprising instructions for operating a processor, when executed by the processor, such that the processor performs a method according to any one of claims 1 to 5.
8. A computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 5.