Information processing device, and information processing method

Abstract

This information processing device calculates the speed and position of an agent by means of multi-agent reinforcement learning, and comprises a calculation unit for stochastically calculating the speed and position of the agent by using the acceleration of the agent, which is output from a Boid model to which the speed and position of another agent are input.

Description

Information processing device and information processing method

[0001] The present invention relates to the technical field of multi-agent reinforcement learning.

[0002] Multi-Agent Reinforcement Learning (MARL) is known as prior art 1 (Non-Patent Document 1). MARL is a multi-agent version of reinforcement learning that maximizes a reward predetermined by the environment.

[0003] Furthermore, the Boids model (hereinafter referred to as the Boid model) is known as prior art 2 (Non-Patent Document 2). The Boid model is a model that mimics the behavior of flocks of birds, fish, insects, etc. by using three rules: separation, cohesion, and alignment.

[0004] Niu, Yaru, Rohan R. Paleja and Matthew Craig Gombolay. "Multi-Agent Graph-Attention Communication and Teaming." AAMAS '21: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems May 2021Pages 964-973Craig W. Reynolds: Flocks, Herds, and Schools:A Distributed Behavioral Model https: / / www.cs.toronto.edu / ~dt / siggraph97-course / cwr87 /

[0005] It is desirable to be able to coordinate and control a system consisting of many moving objects (e.g., artificial life forms, cars, drones, robots, etc.) in an optimal manner.

[0006] However, in Conventional Technique 1, each agent can only obtain private / local and noisy observation information, making it difficult to learn a policy that will bring the next action closer to global optimality. Furthermore, Conventional Technique 2 is a rule-based technique and does not have terms that can be learned, making it difficult to build an advanced multi-agent control system using Conventional Technique 2.

[0007] The present invention has been made in consideration of the above points, and aims to provide a technology that enables optimal cooperative control of a system made up of a large number of moving objects.

[0008] According to the disclosed technology, there is provided an information processing device that calculates the speed and position of an agent through multi-agent reinforcement learning, the information processing device including a calculation unit that probabilistically calculates the speed and position of the agent using the acceleration of the agent output from a void model to which the speeds and positions of other agents are input.

[0009] The disclosed technology provides a technology that enables cooperative control of a system made up of a large number of moving objects in an overall optimal manner.

[0010] 1 is a diagram illustrating an example of the configuration of an information processing device 100. FIG. 1 is a diagram illustrating algorithm 1 of MARL. FIG. 2 is a diagram illustrating an image of three rules of the void model. FIG. 3 is a diagram illustrating algorithm 2 of the void model. FIG. 4 is a diagram illustrating algorithm 3 in Example 1. FIG. 5 is a diagram illustrating algorithm 4 in Example 2. FIG. 6 is a diagram illustrating lemma 1. FIG. 7 is a diagram illustrating lemma 2. FIG. 8 is a diagram illustrating Theorem 1. FIG. 9 is a diagram illustrating an experiment in Example 4-1. FIG. 10 is a diagram illustrating an experiment in Example 4-1. FIG. 11 is a diagram illustrating an experiment result in Example 4-1. FIG. 12 is a diagram illustrating an experiment result in Example 4-1. FIG. 13 is a diagram illustrating an experiment in Example 4-2. FIG. 14 is a diagram illustrating an experiment result in Example 4-1. FIG. 15 is a diagram illustrating an example of the hardware configuration of an information processing device 100.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] In the description of the text of the specification, for convenience of description, the symbols attached to the beginning of letters are written as " － "p" is an example. In addition, the text of the specification uses regular font for characters such as sets, matrices, and vectors. It is clear from the context that a certain character means a set, matrix, vector, etc.

[0013] 1 shows an example of the configuration of an information processing device 100 according to this embodiment. As shown in FIG. 1, the information processing device 100 includes an input unit 110, a calculation unit 120, an output unit 130, and a data storage unit 140.

[0014] The data storage unit 140 stores parameters of the neural network model. Input data to the model is input from the input unit 110. The calculation unit 120 performs model calculations on the input data using the parameters read from the data storage unit 140. The output unit 130 outputs output data from the model obtained by the calculation unit 120.

[0015] The information processing device 100 may function as one agent in the first to fourth embodiments described below, or may perform the processing of multiple agents.

[0016] The following describes the specific operation of the information processing device 100. First, to facilitate understanding of the technology according to the present invention, two conventional technologies (conventional technology 1 and conventional technology 2) and their respective problems will be described in more detail, followed by a description of the technology according to the embodiment of the present invention. In the following description, an "algorithm" refers to a processing procedure executed by the calculation unit 120.

[0017] (Prior Art 1 (Reinforcement Learning)) As mentioned above, multi-agent reinforcement learning (MARL) is known as Prior Art 1. MARL is a multi-agent version of reinforcement learning that maximizes a reward predetermined by the environment.

[0018] For example, a reward system is provided in which agents are not eaten by predators, and the movement control (speed and direction) of the multi-agent is acquired through learning. In MARL, hidden states with a certain degree of dimension (256 to 1024 dimensions) are often communicated between agents.

[0019] First, we will outline the architecture of a neural network that is commonly used in conventional MARL. This architecture is mainly composed of the following three elements: 1) to 3).

[0020] 1) Policy Network π θ Probabilistic behavioral sampling using.

[0021] 2) Feedback from the environment (e.g., rewards) ｉ , and the noisy private observation o ｉ ).

[0022] 3) A subset of agents j (∈N ｉ ) and the hidden state h ｊ By exchanging ｉ , c ｉ The transition is performed in chronological order.

[0023] The above is the explanation of 1) to 3). Figure 2 shows an example of an algorithm (neural network architecture) that iteratively updates the actions and hidden states of N agents ({1, ..., N}∈N) over T time steps (k = 0, ..., T-1) (referred to as Algorithm 1). In the following explanation of the algorithm, line numbers will be referred to as "steps." The subject of the operation is the calculation unit 120 of the information processing device 100.

[0024] In the algorithm 1 shown in FIG. 2, in step 3, the calculation unit 120 calculates the behavior x of each agent. ｉ Policy Network π θ The calculation unit 120 also performs probabilistic sampling from the action x ｉ is fed back to the environment, and reward r ｉ and the noisy private observation o ｉ Get.

[0025] In step 4, the calculation unit 120 calculates the hidden state h ｉ and the state c received through communication with other agents ｉ And private observations ｉ and function F θ By inputting ｉ Predict.

[0026] In step 5, agent j sends hidden state h ｊ That is, in steps 5 and 6, the calculation unit 120 communicates with other agents (multiple agents are possible) to transmit the hidden state h ｊ and converts it into a function G θ Convert using c ｉ get.

[0027] (Problem of Prior Art 1) Prior art 1 MARL has the following preconditions 1 and 2, which causes a problem that uncertainty (which may be interpreted as noise) remains in the policy.

[0028] Precondition 1 (noisy private observation): Information that each agent can observe ｉ is not information that can grasp the entire environment, but is private / local observation information and noisy observation information.

[0029] Precondition 2 (limited interaction): Assuming that the number of agents N is very large (10 to 10,000), N-to-N communication is impossible, and only limited communication can be performed with some agents, for example, those in nearby areas.

[0030] Under the above preconditions 1 and 2, it is difficult to learn a policy that brings the next action closer to global optimality, and uncertainty remains in the policy.

[0031] (Prior Art 2: Boyd Method) As mentioned above, the Boyd model is known as a model that imitates a flock of birds, fish, insects, etc., and is often used in CG and the like.

[0032] The void model consists of the following three rules. Figure 3 shows an image of each rule.

[0033] 1) Separation: The agent's position is moved away from the center of the group of neighboring agents.

[0034] 2) Cohesion: The agent's position moves closer to the center of the group of neighboring agents.

[0035] 3) Alignment: Makes an agent's speed approach the average speed of its neighboring agents.

[0036] By using the above three rules, it is possible to create CG that looks like a flock of actual birds. The Boyd model does not involve learning. Non-Patent Document 2, which discloses the Boyd model, describes the three rules but does not disclose their formulation, and there are various implementations.

[0037] Using differential equations, the Boyd model-based multi-agent dynamics can be expressed, for example, by the following equations (1) to (3).

[0038] On the left side of equation (1), the three rules of the void model (Alignment, Cohesion, Separation) are summarized as a function f ｉ and use this as the acceleration of agent i. ｉ (t) is the set of agents j in the neighborhood of agent i at time t. v represents the velocity and p represents the position. The neighboring agents are those within a radius D from agent i. ＣＯＭAs shown on the right side of equation (1), it is assumed that nearby agents share their speeds and positions.

[0039] The left side of equation (1) is the acceleration of agent i. Equation (2) shows that the derivative of position is velocity. Equation (3) shows the initial condition.

[0040] The above differential equation can be discretized using the Euler method to obtain the algorithm shown in Figure 4. In Algorithm 2 shown in Figure 4, the velocity and position are calculated using a radius D ＣＯＭ The "Boids" in Figure 4 are the functions f in Eq. (1). ｉ is a function (void model) equivalent to

[0041] (Issues with Prior Art 2) Prior Art 2 is a rule-based method and does not have any terms that can be learned. Therefore, it is difficult to build an advanced multi-agent control system using Prior Art 2 alone.

[0042] (Overview of Technology According to the Embodiments) It is desirable to be able to optimally control a system made up of a large number of moving objects (e.g., artificial life forms, cars, drones, robots, etc.) in a coordinated manner. However, Conventional Techniques 1 and 2 have the above-mentioned problems.

[0043] Below, we will explain the technology that solves the above problems and makes it possible to build an advanced multi-agent control system.

[0044] As specific techniques, examples 1 to 4 will be described below. The outlines of examples 1 to 4 are as follows.

[0045] Example 1: Example 1 is an example of stochastic multi-agent dynamics using a void model, which integrates prior art 1 and prior art 2.

[0046] Example 2: Example 2 is an example in which "Flock Momentum," which is an implementation of the Boyd model, is applied to Example 1. Flock momentum is an implementation that ensures that there is theoretical advantage in integrating the Boyd model into MARL.

[0047] Example 3: Example 3 is an example that uses only Alignment, Cohesion, and Separation, excluding Cohesion and Separation from the method used in Example 2. In Example 3, theoretical analysis is conducted to prove that Flock momentum can be used to improve the efficiency of MARL, and it is theoretically shown that there is a high possibility that the efficiency of MARL can be improved when the reward is static (i.e., does not change over time). Furthermore, it is shown that, of the three rules of the Boyd model, Alignment is essential, but Cohesion / Separation may not be necessary.

[0048] Example 4: In Example 4, an example of application of the technology according to the present embodiment to applications will be described. Specifically, application to artificial life and traffic flow control will be described.

[0049] Each example will be described below.

[0050] First Embodiment First, probabilistic multi-agent control (differential equation representation) using a void model will be described.

[0051] The learnable probabilistic multi-agent dynamics in Example 1 can be obtained by incorporating elements of MARL into the multi-agent dynamics of the Boyd model (equations (1) to (3)).

[0052] The policy is a Gaussian distribution Norm(μ θ (h ｉ (t)), σ ２ I), the stochastic behavior sampling in the third line of Algorithm 1 (Fig. 2) is x ｉ （ｋ＋１） = μ θ (h ｉ（ｋ） ) + σε ｉ （ｋ） where ε ｉ ∼Norm(0,I).

[0053] In the stochastic multi-agent dynamics in Example 1, behaviors are associated with the speed and position of agents, and the update differences can be considered to follow a Gaussian distribution.

[0054] Therefore, the learnable drift term μ θ and the Wiener process term B with diffusion coefficient σ ｉ Integrating (t) (Brownian motion) into velocity transitions is a natural extension of the multi-agent dynamics of the Boyd model (equations (1)-(3)). Furthermore, the learnable term F θ and G θ A hidden state transition using is introduced.

[0055] The learnable stochastic multi-agent dynamics in the first embodiment is given by the following equations (4) to (8).

[0056] The transitions of h(t) are given by differential equations because the corresponding transitions of behavior (velocity and position) are also described by differential equations.

[0057] That is, in the first embodiment, probabilistic action selection and probabilistic hidden state transitions, as used in MARL, are introduced into the void model (expression of differential equations).

[0058] Regarding equation (4), the acceleration of the Boyd model is f ｉ However, in Example 1, the learnable drift term μ θ and Wiener process term B ｉ (t) (Brownian motion) can be added.

[0059] In addition, in equations (6) and (7), the hidden state h ｉ The transition of and the auxiliary variable c ｉ has been introduced.

[0060] By discretizing the above differential equation, a neural network (NN) structure can be obtained. An example of the NN structure is shown in FIG. 5. In the first embodiment, the calculation unit 120 of the information processing device 100 performs calculations according to the algorithm shown in FIG. 5. Note that the parameters can be learned using, for example, an actor-critic method.

[0061] Algorithm 3 shown in Figure 5 involves alternating transitions between velocity, position, and hidden states. To achieve this, the Euler-Maruyama algorithm is applied to the stochastic differential equation for acceleration, and the Euler algorithm is applied to the ordinary differential equations for position and hidden variables. By explicitly introducing feedback from the environment and communication between agents, as used in the prior art MARL, the NN structure shown in Figure 5 can be obtained.

[0062] Algorithm 3 in Figure 5 shows the update rule over N agents and T time steps, including feedback from the environment.

[0063] In step 3, the velocity and position of other agents in the neighborhood of agent i are input to the Boyd model to obtain the acceleration of agent i at the next time. In step 4, the drift term μ θ and acceleration, σε ｉ （ｋ） Based on this, the velocity of agent i at the next time can be obtained. ｉ ∼Norm(0,I). In step 5, the position of agent i at the next time is obtained.

[0064] In step 6, the reward r is calculated from the feedback environment of the velocity and position. ｉ And private observations ｉ In step 7, the hidden state h ｉ and the state c received through communication with other agents ｉ And private observations ｉ and function F θ By inputting ｉIn step 8, agent j sends the hidden state h ｊ , velocity v ｊ , position p ｊ is sent.

[0065] In step 9, the hidden state h of the other agent is ｊ function G θ Enter it in c ｉ get.

[0066] Here, Δt represents the time interval. Comparing the conventional MARL neural architecture (Fig. 2) with the probabilistic multi-agent dynamics of Example 1 (Fig. 5), the main differences are as follows: 1) and 2).

[0067] 1) Action x in Figure 2 ｉ The stochastic sampling of is associated with stochastic velocity transitions in FIG. 5, which leads to an iterative update rule involving auxiliary positions.

[0068] 2) As shown in step 3 of Figure 5, the dynamics of the void model f ｉ （ｋ） will be integrated.

[0069] According to the first embodiment, it is possible to realize stochastic multi-agent control using the differential equations (4) to (8) and the void model expressed as a neural network by discretizing the differential equations.

[0070] (Example 2) As mentioned above, there is uncertainty in the implementation of the void model, which means that there are many implementation patterns for the three rules of Alignment, Cohesion, and Separation.

[0071] In the second embodiment, an example of implementation of a void model that is easy to theoretically guarantee will be described based on theoretical analysis that will be explained in the third embodiment later.

[0072] That is, in the second embodiment, the void model term f is added to the algorithm 3 (FIG. 5) described in the first embodiment. ｉ (Step 3 in Figure 5) is implemented using Flock momentum as shown in Figure 6 (Algorithm 4).

[0073] In step 3, the calculation unit 120 that executes Flock momentum shown in FIG. 6 calculates the number of neighboring agents j (e.g., agent i) in a radius D ＣＯＭ Calculate the average position and average velocity of the object (within

[0074] In step 4, as an alignment process, the calculation unit 120 applies acceleration to agent i so that the speed of agent i approaches the average speed of the other agents.

[0075] In the Cohesion + Separation process in step 5, the calculation unit 120 performs processing according to a recursive update rule, in which if agent i is too close to the average position, agent i is moved away from the average position, and if agent i is too far from the average position, agent j is moved closer to the average position.

[0076] The function ψ in the second embodiment is ψ( － p ｉ -p ｉ ) = Clip(φ( － p ｉ -p ｉ ), η φ , η ｇ ) This clipping function (Clip) is expressed as ||φ( － p ｉ -p ｉ ) ||≦η φ , and ||g ｉ ||≦η ｇ This is a function that guarantees that

[0077] In step 9, the calculation unit 120 calculates the acceleration of agent i using the value of Alignment + Cohesion + Separation calculated in steps 4 and 5. This acceleration value becomes the return value of the void function.

[0078] In Example 3, we will explain a theoretical analysis of the proposed method in Example 2 (Algorithm 3 + Algorithm 4) to see whether it is possible to achieve efficiency that improves rewards under a static reward function that does not change over time. We will also explain the proposed method in Example 3 (a method that does not use Cohesion + Separation) based on that analysis.

[0079] To facilitate theoretical analysis, we restrict how rewards are obtained from the environment (reward function). That is, we assume a simple MARL task in which the reward function depends only on each agent's velocity and position (both of which are differentiable) and is time-independent (static). For example, in a predator-prey task, a reward that depends on the distance from a fixed-position predator fits this assumption. In this case, we can define the reward function to be maximized as a differentiable function with respect to each agent's d-dimensional velocity and position (R:R ２ｄ →R). Each agent's reward is ～ r ｉ ≒R({v ｉ , p ｉ}) and ∇R({v ｉ , p ｉ})∈R ２ｄ In this specific scenario, the following assumptions are used:

[0080] Assumption 1: (β-smoothness). There exists a β such that for any a, a′∈R ２ｄ For ||∇R(a)-∇R(a')||≦β||a-a'||.

[0081] Lemma 1 (Descent lemma): Under Prerequisite 1, if N agents follow Algorithm 3 (Figure 5) and Δt≦1 / (4β), we can obtain Lemma 1 shown in Figure 7. Lemma 1 shown in Figure 7 indicates that, under the condition that the reward is determined by the position and velocity of each agent and is a time-invariant (static) function, the smaller the gradient of the reward function on the left-hand side, the more likely it is that a large reward will be obtained. Therefore, we want to make the right-hand side as small as possible.

[0082] As shown in Figure 7, the right-hand side consists of three terms: (1) a term that depends on the initial condition, (2) a term that depends on the Wiener process, and (3) a drift term μ, a direction in which the reward increases, ∇R, and a void term (f ｉ ) and the balance of (1) and (2). Of these three, (1) and (2) are uncontrollable terms. For (3), if the void term is set appropriately, it may be possible to reduce the policy uncertainty (u), which can contribute to reducing the right-hand side.

[0083] Next, u ｉ （ｋ） and introduce conditions on the interaction graph. First, we introduce the following additional matrix notation: （ｋ） = [u １ （ｋ） , ..., u Ｎ （ｋ） ], ～ F （ｋ） = [ ～ f １ （ｋ） , ..., ～ f Ｎ （ｋ） ] ・Adjacent average matrix A （ｋ） ∈R Ｎ×Ｎ , which is used to calculate the average speed and position of the N agents in step 3 of Figure 6 .

[0084] ・Mixing matrix M （ｋ） = (1-γ Ａ Δt)I+γ Ａ ΔtA （ｋ―１） , which is the hyperparameter {γ Ａ , Δt, D ＣＯＭ}.

[0085] Also, ||| Ｆ represents the Frobenius norm. If u ｉ （ｋ） + ～ f ｉ （ｋ） If follows a zero-mean distribution with finite variance, averaging with the mixing matrix reduces the norm to

[0086] Condition 1 (variance-reduced uncertainty): Update time interval Δt (>0), alignment coefficient γ Ａ (>0), and the interaction radius D ＣＯＭ (>0), then ξ(γ Ａ , Δt, D ＣＯＭ ) ∈ (0,1] such that for any U （ｋ） ∈R2 ｄ×Ｎ , ～ F （ｋ） ∈R ２ｄ×Ｎ In E||(U （ｋ） + ～ F （ｋ） ) M （ｋ） || ２ Ｆ ≦(1−ξ(γ Ａ , Δt, D ＣＯＭ ))E||U （ｋ） + ～ F （ｋ） || ２ Ｆ where E is taken for a randomly sampled mixing matrix. In Lemma 2 and Theorem 1 below, ξ(γ Ａ , Δt, D ＣＯＭ ) is simply represented as ξ. Other prerequisites are as follows:

[0087] Precondition 2 (Bounded hidden state update): There is η Ｆ exists and for any h ｉ , c ｉ ∈R ｍ , o ｉ ∈R ｌ , N ｉ For ||F θ (h ｉ , c ｉ , o ｉ ) ||≦η Ｆ Meet the following.

[0088] Precondition 3 (Bounded drift coefficient): There is η μ exists and for any h ｉ ∈R ｍ For ||μ θ (h ｉ ) ||≦η μ Meet the following.

[0089] Precondition 4 (ν-smoothness): There exists a certain ν, and for any h ｉ , h' ｉ ∈R ｍ For ||μ θ (h ｉ )-μ θ (h' ｉ ) ||≦ν||h ｉ -h' ｉ Satisfies ||.

[0090] Lemma 2 (Iterative Uncertainty Lemma): Under Condition 1 and Preconditions 1-4, if N agents follow Algorithm 3 (5) with flock momentum applied in Algorithm 4 (Figure 6), then there exists a Δt that satisfies the equation in Figure 8.

[0091] Figure 8 (Lemma 2) shows that we have theoretically proven that the terms in Figure 7 decrease over time when the Boids term is implemented with Flock momentum (Figure 6) under Condition 1 and Preconditions 1 to 4. In order to constrain the terms that appear as errors, Flock momentum (Figure 6) incorporates recursion into the calculation of Cohesion+Separation.

[0092] Combining Lemma 1 and Lemma 2 leads to the following Theorem 1.

[0093] Theorem 1 (Uncertainty reduction using flock momentum): Under Condition 1 and Preconditions 1-4, if N agents follow Algorithm 3 (Fig. 5) with flock momentum (Fig. 6), there exists a Δt that satisfies the equation in Fig. 9. Fig. 9 shows that the left-hand side (the gradient norm of the reward function) is successfully suppressed.

[0094] The above theoretical analysis leads to the following (1) and (2).

[0095] (1) As stated in Condition 1, it is important to connect agents so that the norm of the uncertainty term (u) is reduced by averaging the speed and position of neighboring agents.

[0096] (2) Regarding the void term, alignment is essential (γ Ａ> 0), but Cohesion and Separation are not necessarily required (γ ＣＳ = 0 is allowed). However, from the viewpoint of expanding the observation area of ​​the entire multi-agent, it is preferable to have three rules.

[0097] As described above, cohesion and separation are not necessarily required, and therefore in Example 3, the calculation unit 120 executes processing that excludes the processing related to cohesion and separation (step 5 and the second term on the right-hand side of step 9) from flock momentum (FIG. 6) as processing of the void function in algorithm 3 of FIG. 5. In other words, while executing algorithm 3 of FIG. 5, the calculation unit 120 executes flock momentum (FIG. 6) for the void function (void model), but this processing excludes the processing related to cohesion and separation (step 5 and the second term on the right-hand side of step 9).

[0098] (Example 4) Next, an explanation will be given of Example 4. Below, an application to artificial life will be explained as Example 4-1, and an application to traffic control will be explained as Example 4-2.

[0099] Example 4-1: Application to Artificial Life In Example 4-1, as shown in Fig. 10, behaviors such as N = 1000 agents (Prey) escaping from a predator were learned using the MARL-based method according to this embodiment. Experiments were conducted for Example 2 (Algorithm 3 + Algorithm 4) and Example 3 (Algorithm 3 + Algorithm 4 (Alignment only)).

[0100] The experiment is shown in Figure 11. In Figure 11, C1 and C2 indicate the comparative methods (C2 is MARL, Prior Art 1), and P1 and P2 indicate the proposed method. Specifically, the experiment is as follows.

[0101] C1: No inter-agent communication, no void model C2: Inter-agent communication, no void model P1: Inter-agent communication, with void model (using 3 rules, corresponding to Example 2) P2: Inter-agent communication, with void model (using only Alignment, corresponding to Example 3) Figures 12 and 13 show the evaluation results. Figure 12 shows the results when the predator does not move, and Figure 13 shows the results when the predator moves. Furthermore, AMD is the average minimum distance between the agent and the predator, CARP is the central area positioning ratio, and AR is the average reward in the range from -100 to 0. As shown in Figures 12 and 13, the proposed method can obtain better results than the conventional method.

[0102] Example 4-2: Application to traffic control In Example 4-2, an experiment was conducted on traffic control when N agents (=N vehicles) were traveling on a road, as shown in Fig. 14. The travel route was assumed to be given in advance, and only the speed (a scalar value) was controlled.

[0103] As shown in FIG. 15(b1), the travel route of an agent traveling through a 2D intersection is a given, so the agents can determine from each other's information whether there is a possibility of a collision.

[0104] As shown in FIG. 15(b2), a one-dimensional map is created that represents the distance relationship between each agent and surrounding vehicles, with each agent at the center. This map is generated for each vehicle. With the technology of this embodiment, a void model adjusts the speed and position of each agent on the one-dimensional map. Experiments using the comparison method and the proposed method showed that agents can obtain higher rewards with the proposed method.

[0105] The experimental results for the aforementioned C1, C2, P1, and P2 are shown in Figure 16. In Figure 16, ANV is the average speed normalized to the range of [0, 1]. SR is the stagnation rate to be reduced. Specifically, it is the proportion of agents whose speed is less than the threshold. AR is the average reward.

[0106] As can be seen from Figure 16, P2 (Alignment only) produced good results. This is consistent with the general idea that traffic jams are less likely to occur if the vehicle speed is as close as possible to the surrounding vehicles.

[0107] (Hardware Configuration Example) The information processing device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0108] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0109] Fig. 17 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 17 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0110] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0111] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the information processing device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0112] (Effects of the embodiment) As described above, the technology described in this embodiment makes it possible to realize a technology that enables cooperative control of a system made up of a large number of moving bodies in an overall optimal manner.

[0113] The following additional notes are provided regarding the above-described embodiments.

[0114] <Additional Notes> (Additional Item 1) An information processing device that calculates the speed and position of an agent through multi-agent reinforcement learning, comprising: a calculation unit that probabilistically calculates the speed and position of the agent using the acceleration of the agent output from a void model to which the velocities and positions of other agents have been input. (Additional Item 2) The information processing device according to Additional Item 1, wherein the void model calculates the acceleration of the agent by performing a process that causes the speed of the agent to approach the average speed of the other agents. (Additional Item 3) The information processing device according to Additional Item 1, wherein the void model calculates the acceleration of the agent by performing a first process that causes the speed of the agent to approach the average speed of the other agents, and a recursive second process that causes the position of the agent to move away if it is too close to the average position of the other agents, and causes the position of the agent to move closer if it is too far. (Supplementary Item 4) An information processing method executed by an information processing device that calculates the speed and position of an agent by multi-agent reinforcement learning, the information processing method probabilistically calculating the speed and position of the agent using the acceleration of the agent output from a void model to which the speeds and positions of other agents have been input.

[0115] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0116] REFERENCE SIGNS LIST 100 Information processing device 110 Input unit 120 Calculation unit 130 Output unit 140 Data storage unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that calculates the speed and position of an agent using multi-agent reinforcement learning, comprising a calculation unit that probabilistically calculates the speed and position of the agent using the acceleration of the agent output from a void model that inputs the speeds and positions of other agents.

2. The information processing device according to claim 1, wherein the void model calculates the acceleration of the agent by performing processing to make the speed of the agent approach the average speed of the other agents.

3. The information processing device according to claim 1, wherein the void model calculates the acceleration of the agent by performing a first process that causes the agent's speed to approach the average speed of the other agents, and a recursive second process that moves the agent away if the agent's position is too close to the average position of the other agents, and moves the agent closer if the agent is too far away.

4. An information processing method executed by an information processing device that calculates the speed and position of an agent through multi-agent reinforcement learning, wherein the speed and position of the agent are calculated probabilistically using the acceleration of the agent output from a void model to which the speeds and positions of other agents are input.

Images