Reinforcement learning computing device and reinforcement learning method

Physical Reservoir Computing addresses the computational challenges of deep reinforcement learning by offloading calculations to a physical medium, enhancing efficiency and reducing resource requirements in reinforcement learning systems.

JP7590675B2Active Publication Date: 2024-11-27NIPPON TELEGRAPH & TELEPHONE CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023526735
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-06-09
Publication Date
2024-11-27
Estimated Expiration
2041-06-09

AI Technical Summary

Technical Problem

Deep reinforcement learning methods face significant computational challenges due to high processing loads, particularly in tasks involving large amounts of data and requiring high-speed processing, which are exacerbated by the need for extensive calculation resources.

Method used

Implementing physical Reservoir Computing (RC) to offload calculation tasks to a physical medium, utilizing a computer system with a force layer and an arbitrary matrix representing mutual coupling within the physical medium, where the spectral radius is between 0.6 and 1.2, thereby reducing the computational burden.

Benefits of technology

This approach effectively reduces calculation resources and enhances the efficiency of reinforcement learning by leveraging physical RC, allowing for high-speed processing and minimizing the size and cost of reinforcement learning devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590675000010
    Figure 0007590675000010
  • Figure 0007590675000011
    Figure 0007590675000011
  • Figure 0007590675000012
    Figure 0007590675000012
Patent Text Reader

Abstract

The purpose of the present invention is to provide a computing device for reinforcement learning and a reinforcement learning method for executing highly efficient reinforcement learning in which physical reservoir computing (RC) is applied to reinforcement learning having a high computation processing load and computation resources are effectively reduced. Therefore, the computing device (112) for reinforcement learning according to the present invention comprises a physical medium (115) that includes an RC algorithm having an input layer for inputting a subject state st, an intermediate layer for learning a subject action at on the basis of input information inputted to the input layer, and an output layer for outputting output information relating to an action at, and also includes an action determination algorithm for outputting information relating to the subject action on the basis of the action at outputted from the RC algorithm, the RC algorithm executing at least some of the functions to physically convert the state st and learn a subject strategy.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a computation device for reinforcement learning that is provided with an algorithm for carrying out reinforcement learning. [Background technology]

[0002] In recent years, research and development of deep reinforcement learning using deep neural networks (DNNs) has progressed. Deep neural networks have been reported to exhibit excellent learning performance in a wide range of fields, including game playing, robotics, and Go, and many highly generalizable methods have been proposed that can learn policies for a variety of tasks. In general, in deep reinforcement learning, a state variable s t and the action variable a t , the resulting reward R t Using this, we can calculate the action value function Q(s t ,a t ), state value function V(s t ) is calculated as the output of the neural network to determine a policy. In reinforcement learning, time-series information such as sensor data is often handled as input information, so a recurrent neural network (RNN) model such as LSTM (Long short-term memory) is often incorporated. Generally, the calculation processing and learning costs are high, so there is a demand for reducing these processing loads.

[0003] Recently, a model called Reservoir Computing (hereinafter, also referred to as "RC") has been proposed as a type of RNN. RC consists of an input layer where input information is connected to each neuron, a middle layer where each neuron is connected to each other, and an output layer that adds up and outputs the signals of each neuron. Such a configuration is described, for example, in Patent Document 1.

[0004] RC differs from general RNN in that the input and intermediate layer networks are fixed, and the only variables used for learning are the weight coefficients of the output layer. This type of RC method can significantly reduce the variables to be learned, making it suitable for time series learning that requires a huge amount of data and high-speed processing. RC generates most of the connections between nodes using random mapping. If this random mapping is calculated on a computer, it requires a lot of computing resources as the number of nodes increases, and it is one of the factors that makes the computing system including the computer larger and more expensive. To solve this problem, research and demonstration of a configuration called physical RC, in which the mapping is calculated by a physical system, is progressing. With physical RC, part of the calculation can be outsourced to a physical element, so efficient learning is possible even on devices with low computing resources. Physical RC is described in, for example, Patent Document 2. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Herbert, Jaeger. and Harald, Haas. (2 APRIL 2004). Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. VOL 304 Science. [Non-Patent Document 2] Gouhei, Tanaka.et al.(2019). Recent advances in physical reservoir computing. Neural Networks.100-123. Summary of the Invention

[0006] The present disclosure aims to provide a computation device for reinforcement learning and a reinforcement learning method that apply physical RC to reinforcement learning, which requires a large amount of computational processing, to effectively reduce computational resources and perform highly efficient reinforcement learning.

[0007] In order to achieve the above object, a computation device for reinforcement learning according to one aspect of the present invention is a computation device for reinforcement learning that learns a policy for an object to execute a task, the computation device including a computer system having an input layer for inputting input information related to a state of the object, an intermediate layer for learning a policy of the object based on the input information input to the input layer, and an output layer for outputting output information related to the policy, and an action decision unit for outputting information related to an action to be taken by the object based on the output information output from the computer system, the computer system including a physical medium that physically converts the input information and executes at least a part of the function of learning the policy of the object. , Ω corresponding to the mutual coupling within the physical medium is an arbitrary matrix representing the mutual coupling, and the spectrum radius α of the matrix Ω is 0.6 or more and 1.2 or less.

[0008] A computational method for reinforcement learning according to one aspect of the present invention is a computational method for reinforcement learning in which an object learns a policy for executing a task, the computational method including a computer system having an input layer for inputting information, an intermediate layer for performing a calculation based on the input information input to the input layer, and an output layer for outputting a result of the calculation, the computational method comprising the steps of: inputting input information about a state of the object to the input layer; and the intermediate layer learning the policy of the object based on the input information. and outputting information about the behavior of the target based on information about the policy obtained by the learning, wherein at least a part of the learning of the policy of the target is performed by a physical medium that physically converts the input information. The matrix Ω, which corresponds to the mutual coupling within the physical medium, is an arbitrary matrix representing the mutual coupling, and the spectrum radius α of the matrix Ω is equal to or greater than 0.6 and equal to or less than 1.2.

[0009] According to the above aspects, it is possible to provide a computation device for reinforcement learning and a reinforcement learning method that apply physical RC to reinforcement learning, which requires a large amount of computational processing, effectively reduce computational resources, and perform highly efficient reinforcement learning. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining the form of a general RC model. [Diagram 2] FIG. 2 is a diagram for explaining an RC model of a computation system having a computation device for reinforcement learning according to the first embodiment. [Diagram 3] FIG. 2 is a schematic diagram illustrating the configuration of a calculation device used in the first embodiment. [Figure 4] 4 is a diagram for explaining a hardware configuration of the arithmetic device shown in FIG. 3. [Diagram 5] 13(a) and 13(b) are diagrams for explaining a computation device for reinforcement learning according to a second embodiment of the present invention. [Figure 6] FIG. 11 is a diagram for explaining a calculation device according to a third embodiment of the present invention. [Figure 7] 13(a) and 13(b) are graphs showing the state of learning in one embodiment of the present invention. [Figure 8] 1A is a diagram for explaining a pre-processing unit of the present embodiment, and FIG. 1B is a graph showing the state of learning in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a first embodiment, a second embodiment, and a third embodiment of the present invention will be described. Note that the first embodiment to the third embodiment are examples of the technical ideas, configurations, procedures, etc. of the present invention, and do not limit the specific configurations, conditions, parameters, etc. of the present invention. First, this specification will explain Reservoir Computing (hereinafter, also referred to as "RC") before explaining the first embodiment, the second embodiment, and the third embodiment.

[0012] FIG. 1 is a diagram for explaining the form of a general RC model. The RC model 10 shown in FIG. 1 is composed of an input layer Ri in which an input signal, which is input information, is coupled to each neuron, an intermediate layer Rr in which each neuron is coupled to each other, and an output layer Ro that adds up and outputs the signals of each neuron. In this specification, the coupling of an input signal to each neuron is also referred to as "input". Equation (1) is an equation that determines an input signal u(n) that is input to the input layer Ri. Equation (2) is an equation that determines an output signal y(n) that is output from the output layer Ro when an input signal u(n) is input to the input layer Ri.

number

[0013] In the above formulas (1) and (2), N is the number of neurons, x i (n) is the state of the i-th neuron at time step n, and Ω ij ,mi,η i ,ω i are coefficients that respectively represent the mutual connections between neurons, the connections of input signals to neurons, the connections from output signals to FB (Feedback) signals to each neuron, and the connections from each neuron to output signals. f(·) represents the nonlinear response in each neuron, and tanh(·) is often used. The major difference between an RC network and a general recurrent neural network (RNN) is that the network of the input layer Ri and the hidden layer Rr is fixed, and the only variable used for learning is the weight coefficient of the output layer Ro. The RC method has a great advantage in time series learning, which requires a huge amount of data and high-speed processing, because it can significantly reduce the variables to be learned.

[0014] [First embodiment] (Reinforcement learning with RC (Q-learning)) FIG. 2 is a diagram for explaining an RC model 10 of a computing system having a computing device 112 for reinforcement learning according to the first embodiment. The RC model 10 of the first embodiment has an agent 111, a state measurement device 113, a computing device 112, and a behavior control device 114. In the RC model 10, the computing device 112 learns a strategy for the agent 111 to execute a task. The state of the agent 111 at time t is s t and agent 111 is in state s t Depending on the multiple actions a t The agent 111 can select the action t As a result, the reward R t State s t is measured by the state measurement device 113. Also, the action a t is an action a according to the policy by the action control device 114. t It is controlled to achieve the above.

[0015] As a specific example of the above configuration, for example, a model for controlling a multi-joint robot arm can be considered. In the control of a robot arm, state s t corresponds to the sensor data (position, angle, acceleration, etc.) of each joint of the robot arm. t corresponds to the control signal (rotation amount, displacement amount, etc.) that drives the robot arm. Reward R t is a virtual value obtained by software when the robot arm grasps an object or performs a desired operation. In a system for controlling a robot arm, the state measuring device 113 corresponds to a device such as an ammeter or a voltmeter that acquires sensor data such as current or voltage, and the behavior control device 114 corresponds to a driving device such as a voltage source or a current source.

[0016] The configuration shown in FIG. 2 can also be applied to learning games and shogi. In such a learning system, state s t is an image of the screen or board, and action a t corresponds to a possible move. Also, the reward R tcorresponds to points obtained by an action, etc. Note that the method of the first embodiment is applicable regardless of the input or control amount that depends on the above tasks, and can be widely used for tasks other than those mentioned above.

[0017] The arithmetic unit 112 determines the state s t , action a t and reward R t The behavior of the agent 111 is determined based on the information of the state s measured via the state measurement device 113. The method is described below. FIG. 3 is a schematic diagram of the configuration of the calculation device 112 used in the first embodiment. The calculation device 112 includes an RC algorithm 211 and a behavior decision algorithm 212. The RC algorithm 211 has an RC model 10, and performs reinforcement learning using an RC network. In the first embodiment, the state s measured via the state measurement device 113 is t is regarded as the input signal u(t) at time t in equation (1) and input to the calculation device 112. This signal is transformed by the RC algorithm 211 and the behavior decision algorithm 212 into an intermediate state x(t) and a final output y(t) as described in equation (2). Here, the input and output dimensions of the RC model are the state s t and Action a t The dimensions are set to be equal to those of

[0018] The only learning variable is ω, and the other variables m and Ω are generated according to the configuration of the physical medium 115 (FIG. 4) described later, and learning is not performed. This makes the first embodiment superior to a normal deep learning model in that learning can be performed with a small number of parameters. In the first embodiment, the RC algorithm 211 generates an output y(t) of the RC algorithm 211 in the state s t Actions in a t The action value function Q(s t ,a t The variable ω in equation (2) is learned so that the action value function Q(s t ,a t ) based on a t Decide and take action tThe behavior control device 114 outputs the information about the behavior a t The behavior of the agent 111 is controlled based on the above.

[0019] In the above-described configuration, the agent 111 corresponds to the object, and the information about the state is the state s t Information on the measures will be provided to Action a t State s t is input to the input layer Ri of the RC model 10, and in the intermediate layer Rr, the state s t Based on the agent's action a t is learned. The behavior obtained as a result of learning is a t is output to the behavior decision algorithm 212. The behavior decision algorithm 212 t Agent 111's Action a t Decide and take action t to the behavior control device 114. The RC model 10 is a model of computational processing that mimics neurons, and the RC algorithm 211 realizes computational processing based on the RC model 10 and outputs a reward R t The action value function Q(s t ,a t ) The behavior decision algorithm 212 is a program that decides the behavior of the agent 111 from the processing result performed by the RC algorithm 211. The arithmetic device 112 is a concept that includes software including such a program and hardware for running the program. In the first embodiment, the RC model 10 corresponds to a computer system.

[0020] In addition, the above configuration is a configuration in which the agent 111 performs an action a when performing a task. t This reinforcement learning calculation method is to apply a state s t and a process of inputting state s t Based on the action of agent 111, t The process of learning and the behavior obtained by learning tand outputting information about the behavior of the agent 111 based on the behavior a of the agent 111. t In the process of learning, action a t At least a part of the process of learning the policy executes a computational method for reinforcement learning, which is executed by a physical medium 115 that physically transforms the state st.

[0021] The learning is performed, for example, so as to minimize a TD (Temporal Differential) error L defined by the following equation (3).

number

[0022] In the above formula (3), γ is the discount rate, which is a hyperparameter that indicates how much future rewards are discounted. Typically, it has a value smaller than 1, such as 0.99, and is close to 1. Since the only learning variable is ω,

number

[0023] From this point, the gradient of ω is updated, for example, using the following equation (5).

number

[0024] In equation (5), λ is the learning rate. Although equation (5) is an update rule based on a simple gradient descent method, various optimization algorithms used in the field of machine learning, such as Adam-optimizer and Stochastic Gradient Descent (SGD), can be used. In addition, although equation (3) calculates the TD error for one step, it is also possible to calculate the TD error for n steps. In that case, the following equation (6) is used as the cost function.

number

[0025] The stability and convergence of learning can be improved by setting n in equation (6) to a value greater than 1. It is desirable for n to be a value equal to or less than 10. By considering the partial derivative with respect to ω in equation (6), it is possible to derive the update rule in the same way as equations (5) and (6).

[0026] In general deep reinforcement learning, experimental replay, which randomly samples data sequences, is often used to sample learning data. However, since the first embodiment uses an RC model, which is a type of RNN, such a method cannot be used. To avoid this, a method of inputting time series data (episode) during one trial as is, a sampling method of extracting data of a certain length from within an episode, and a method of specifying a certain time within an episode and reproducing the internal state with the prior data can be used. The method of inputting time series data (episode) during one trial as is and the sampling method of extracting data of a certain length from within an episode are described in, for example, Matthew, Hausknecht. and Peter, Stone. (11,Jan,2017). Deep recurrent q-learning for partially observable. MDPs.Austin.the University of Texas (hereinafter also referred to as "Reference 1"). A technique for specifying a time within an episode and reproducing the internal state using prior data is described, for example, in Steven, Kapturowski, et al. (2019) Recurrent experience replay in distributed reinforcement learning. Published as a conference paper at ICLR. (hereinafter referred to as "Reference 2").

[0027] In the behavior decision algorithm, for example, the ε-greedy method is used. The ε-greedy method selects an action based on a policy π with a probability ε, and selects an action with the highest action value function Q(s, a) calculated above with a probability of (1-ε). The policy π is typically generated from a uniform probability distribution, and all possible actions are selected equally. The calculation device 112 requires many calculation resources for the numerical calculation described in equation (1), and therefore requires more power and execution time as the network becomes larger in scale. In the first embodiment, the calculation corresponding to equation (1) in the RC model in FIG. 3 is performed by the physical medium 115.

[0028] The physical medium 115 may, for example, receive input information as light and convert physical parameters such as the amplitude, wavelength, and frequency of the input light. Alternatively, the physical medium 115 may receive input information as an electrical signal and convert physical parameters such as the value and frequency of the electrical signal. Furthermore, the physical medium 115 may receive input information as a liquid and convert physical parameters such as the pressure and flow rate of the liquid. The conversion is not limited to converting a numerical value without changing the type of parameter, but also includes changing the parameter to another parameter. Learning is performed by changing the state s t Depending on the action a, the task is more likely to be accomplished. t It refers to the accumulation or selection of behavioral information so as to make a choice.

[0029] 4 is a diagram for explaining the hardware configuration of the arithmetic device 112 of the first embodiment that performs an operation corresponding to the formula (1) in a physical medium 115. The arithmetic device 112 shown in FIG. 4 includes signal conversion devices 116 and 117, a physical medium 115, an electronic arithmetic device 119, and a storage unit 118. tis converted from digital information into an analog physical signal via a signal conversion device 116. The converted physical signal is input to a physical medium 115. Here, the physical signal refers to, for example, a current, a voltage, or light intensity. Conversion into a physical signal is desirably performed by a signaling method according to the configuration of the physical medium 115. Signal propagation in the physical medium 115 is described by physical laws determined for each system, but it has been reported, for example, in the above-mentioned Non-Patent Document 1 that an operation equivalent to Equation (1) can be performed in various physical systems.

[0030] The signal propagated through the physical medium 115 is measured by the signal conversion device 117 and converted back into digital information. This measured signal can be considered equivalent to x(t) in equation (1). The measured signal is held in a storage unit 118 consisting of a memory, a hard disk, and the like. In addition, the storage unit 118 also defines and stores programs and parameters for executing equations (2) to (5) at the same time. Information including the measured signal, the programs, and the parameters is transferred to an electronic calculation device 119 equipped with a central processing unit (CPU), a graphics processing unit (GPU), and the like, and the electronic calculation device 119 performs reinforcement learning by calculating equations (2) to (5). With this configuration, the first embodiment has an excellent effect of being able to automatically calculate equation (1) during signal propagation. In addition, the first embodiment with this configuration automatically performs calculations equivalent to equation (1) within the laws of physics in the physical medium 115, thereby reducing and solving the problem of increased calculation resources.

[0031] Next, the physical medium 115 of the first embodiment will be described. The physical medium 115 may be a known one, and the first embodiment uses a configuration in which the physical medium 115 performs calculations using an optical circuit. Calculations using such optical circuits are described, for example, in J, Bueno. et al. (2018). Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5, 756-760 (hereinafter also referred to as "Reference 3"), L, Larger. et al, (2012) Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. Optics express 20, 3241 (hereinafter also referred to as "Reference 4"), and M, Nakajima. et al. (2021) Scalable reservoir computing on coherent linear photonic processor communications. physics.4. 20 (hereinafter also referred to as "Reference 5"). The first embodiment allows high-speed optical input and output exceeding 100 Gbit / s via the optical communication device, and therefore allows high-speed calculations.

[0032] In order to input and output light at high speed, the signal conversion device 116 is composed of a digital-to-analog converter (DAC) and an optical modulator, converts the digital signal into optical intensity and optical phase, and introduces it into the optical physical medium 115. When converting the output light back into a digital signal, the signal conversion device 117 is composed of an optical receiver and an analog-to-digital converter (ADC). The optical physical medium 115 may be, for example, a spatial optical system consisting of a lens, mirror, and spatial modulator as described in Reference 3, or may be configured to use an optical fiber ring as described in Reference 4. While such a configuration makes the device relatively large, it is also possible to integrate a small arithmetic circuit by using an optical integrated circuit as described in Reference 5.

[0033] The physical medium 115 is preferably a nonlinear optical element so that nonlinear conversion is possible. The nonlinear element is realized, for example, by configuring the physical medium 115 with a nonlinear optical material (for example, LiNbO3, etc.) or by a configuration that utilizes gain saturation of an optical amplifier. In the first embodiment, it is preferable to physically implement the physical medium 115 so that the spectral radius (maximum eigenvalue of the matrix) of the matrix Ω in the physical formula (1) is in the range of 0.8 to 1.2. With such a configuration, the first embodiment can realize an operation near the chaos transition point and can improve the performance of retaining RC memory. When the physical medium 115 is an optical medium that does not include a gain medium, the operation near the chaos transition corresponds to an operation executed by a configuration that reduces optical loss as much as possible. Note that, when the physical medium 115 is configured to include a gain medium such as an amplifier, the function related to memory retention can be improved by adjusting the gain.

[0034] Furthermore, the first embodiment is not limited to performing all of the functions equivalent to the calculation shown in equation (1) by the physical medium 115. For example, the first embodiment may perform part of the result obtained by equation (1) by the physical medium 115. And, part of the calculation expressed by equation (1) may be performed by the electronic calculation device 119. More specifically, for example, part of the calculation shown in equation (1) may be performed in advance by the electronic calculation device 119 or the like. Such pre-processing is described, for example, in Reference 4. In the pre-processing described in Reference 4, processing of the second term of equation (1) (input mask processing) is performed in advance, and the result is used as a force signal to the physical medium 115. According to such pre-processing, the state s t Even if the dimensions of the intermediate state x(t) are large, the physical medium 115 can be easily implemented.

[0035] In addition, other aspects of the physical medium 115 include, for example, a memristor such as ReRAM. Such an aspect is a configuration that utilizes a physical phenomenon in which the resistance value changes depending on the history of the current that has passed through the resistive cell. In this configuration, a DAC / ADC is used for the signal conversion devices 116 and 117. Since such a configuration is implemented with an analog electronic circuit, it has high interface affinity and is advantageous for miniaturization. Such a configuration is described, for example, in Yanan, Zhong. et al. (2021). Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing. Nature Communications 12:408. (hereinafter referred to as "Reference 6"). In addition, memristors are advantageous in that they can be formed by semiconductor processes and therefore can easily form large-scale nodes. In addition to such configurations, magnetic circuits, fluids, soft materials, etc. can also be used as aspects of the physical medium 115. This point is summarized, for example, in Reference 1.

[0036] As described above, in the first embodiment, physical RC is applied to reinforcement learning, which requires a large amount of computational processing, and computational resources can be effectively reduced. This makes it possible to prevent the reinforcement learning computation device that performs reinforcement learning from becoming larger and more expensive. Alternatively, when the reinforcement learning computation device that performs reinforcement learning has a surplus of computational resources due to the miniaturization of the reinforcement learning computation device, more advanced computational processing can be performed. Such a first embodiment can provide a reinforcement learning computation device and a reinforcement learning method that perform highly efficient reinforcement learning.

[0037] [Second embodiment] (Reinforcement learning with RC (Actor-critic type)) Next, a second embodiment of the present invention will be described. The reinforcement learning calculation device of the second embodiment is different in that the policy π is also learned (policy on-type learning) as opposed to a method in which the policy π is generated with uniform probability and not learned (policy off-type learning). The second embodiment, which performs policy on-type learning, is capable of faster convergence of a solution than the first embodiment.

[0038] 5(a) and 5(b) are diagrams for explaining arithmetic devices 512 and 513 for reinforcement learning according to the second embodiment. FIG. 5(a) is a schematic diagram of the arithmetic device 512 for reinforcement learning according to the second embodiment. FIG. 5(b) is a schematic diagram of an arithmetic device 513 which is a modified example of the arithmetic device 512 shown in FIG. 5(a). The arithmetic device 512 differs from the arithmetic device 112 of the first embodiment in that it includes an RC algorithm 311 having two RC models 10a and 10b, and a behavior decision algorithm 312 which selects a behavior different from the behavior decision algorithm 212.

[0039] The signal flow in the arithmetic device 512 of the second embodiment shown in FIG. 5(a) is similar to that in the arithmetic device 112 of FIG. 2 described in the first embodiment. However, the arithmetic device 512 differs from the arithmetic device 112 in the configuration of the RC network and the learning method. In the configuration shown in FIG. 5(a), the state s tis input to the two RC models 10a and 10b as the input signal u(t) in equation (1). The input and output dimensions of the RC models 10a and 10b are the state s t , action a t The learning variables in the RC algorithm 311 are the weights ω a and ω c This has the advantage that the RC algorithm 311 can learn using a small number of parameters compared to a normal deep learning model. The RC models 10a and 10b each execute different processes called "Actor" and "Critic".

[0040] Here, the actor decides the possible actions of the agent 111, and the critic decides the state S output from the state measurement device 113. t The RC algorithm 311 estimates the value of the state by collecting information from the action value function π(s t ,a t ) and the output of the critic is the state value function V(s t ) and learning is performed.

[0041] The behavior decision algorithm 312 of the calculation device 310, unlike the first embodiment, selects the behavior of the agent 111 shown in FIG. 2 based on the policy π learned on the actor side. The behavior is selected, for example, based on the cost functions L a ,L c are executed to minimize each of the following:

number

[0042] ω in Equation (7) and Equation (8) a ,ω c By considering the partial derivative with respect to, we can derive the update rule similar to equations (5) and (6).c For , a cost function that takes into account up to n steps ahead may be used, as shown in the following equation (9).

number

[0043] In formula (9), by setting n to a value greater than 1, the stability and convergence of learning can be improved. It is desirable to set n to a value of 10 or less, and ω in formula (9) c By considering the partial differential with respect to, the update rule can be derived similarly to equations (5) and (6). The arithmetic device 310 configured as above can be realized by connecting two RC models 10a and 10b in parallel and making the behavior-decision-making algorithm 212 decide the behavior of the agent 111 by combining information about the policy and information about the value.

[0044] In the above, different RC models 10a, 10b are used on the actor side and the critic side, but the input layer and the intermediate layer may be shared and only the output layer may be different, as in the RC algorithm 411 in Fig. 5(b). This allows the calculation results of the same physical medium 115 to be shared in the RC algorithm 411, so that calculations can be performed by a calculation device 410 having one RC model 10, as in the first embodiment.

[0045] When the RC algorithms 311 and 411 shown in Figures 5(a) and 5(b) are configured as hardware, they are configured to include signal conversion devices 116 and 117, a physical medium 115, a memory unit 118, and an electronic calculation device 119, similar to the calculation device 112 shown in Figure 4 of the first embodiment.

[0046] [Third embodiment] (Adding a random convolutional layer) Next, a third embodiment of the present disclosure will be described. Fig. 6 is a diagram for explaining a calculation device 612 of the third embodiment. The calculation device 612 has a pre-processing unit 613 provided before the RC algorithm 211, together with the RC algorithm 211 and the behavior decision algorithm 212 shown in Fig. 2 of the first embodiment. The pre-processing unit 613 has a function of performing pre-processing using a random convolution layer and a pooling layer, and performs pre-processing on the state s output from the state measurement device 113. t The state s t´ The RC algorithm 211 converts the state s t´ The RC model is calculated based on the above.

[0047] The pre-processing unit 613 has two layers: a kernel filter that performs convolution, and a kernel filter that performs pooling. Convolution refers to a process of compressing pixel values ​​of a small region of an image into a smaller region as one feature, thereby reducing the amount of image data. Pooling refers to a process of reducing the amount of data by thinning out the image. The coefficients of the kernel filter in the random convolution layer are generated, for example, by a random number table in the [-1:1] interval, and are not subsequently learned. Therefore, in the third embodiment, the RC algorithm 211 is similar to that in the first embodiment, and in the second embodiment, the state s t Similarly to processing state s t´ In the third embodiment, when it is necessary to handle high-dimensional input information corresponding to the number of pixels, such as in a game or an in-vehicle camera, the dimension is compressed in advance by a convolutional neural network, and the amount of processing in the RC algorithm 211 can be reduced. EXAMPLES

[0048] (Verification by numerical calculation) An example of reinforcement learning performed using the above first to third embodiments will be described. In this example, a task called the cart-pole problem, which is generally used in tests of reinforcement learning, is executed on a computer simulation. The goal of this task is to control the position of a cart moving in one-dimensional space so that a pole standing on the cart does not fall over. In state st The speed, position, angle and angular velocity of the cart at each time are given as t At that time, you can choose whether the stick will move left or right. If the angle is more than ±20.9° or the position changes by more than ±2.4, you fail and receive a reward R of -200. t Otherwise, a reward Rt of +1 is obtained at each time. In order to implement such a task, this embodiment applies a calculation device having two RC models of the actor and the critic of the second embodiment.

[0049] In addition, an optical medium is adopted as the physical medium of the RC model, and tanh() based on the saturation behavior of a semiconductor optical amplifier (SOA) is used as a nonlinear function. The input coupling m is calculated in advance and then introduced into the physical medium 115, and is generated from a uniform random number in the [0:1] interval. The physical medium 115 is assumed to be constructed on a fiber base, and Ω, which corresponds to the mutual coupling within the medium, reflects the ring topology as shown in the following equation. Such a medium is described, for example, in Francois, Duport. et al. (2012). All-optical reservoir computing. Vol. 20, No. 20 / Optics Express, 22783-22795. (hereinafter referred to as "Reference 7").

number

[0050] The gain α of the return light in the ring is approximately equal to the spectral radius of Ω. Learning in the RC model 10 is performed by the method described in equations (7) and (9). n and γ in equation (9) are set to n=5 and γ=0.99, and the number of nodes in the RC model 10 is set to 64.

[0051] 7(a) and 7(b) are graphs showing the state of learning in this embodiment. In FIG. 7(a), the horizontal axis shows the number of trials, and the vertical axis shows the number of steps in which the stick was not knocked down. The solid line in FIG. 7(a) shows learning using the RC model of this embodiment. The dashed line shows the result of similar learning performed using a three-layer fully connected neural network for comparison. As is clear from FIG. 7(a), in this embodiment, the number of steps in which the stick was not knocked down increases with fewer trials compared to the dashed line, and convergence is faster than when using a fully connected neural network. Furthermore, according to the RC algorithm, the only learning variable is ω, and other variables are processed by the physical medium 115 described later, so learning is possible with fewer parameters.

[0052] Figure 7(b) shows the relationship between the gain α of the return light in the ring and the average number of trials required for convergence. Here, convergence is defined as successfully holding the rod for 200 steps without tilting it five times in a row. As is clear from Figure 7(b), when α is 1.2 or more, the number of trials required for convergence increases, while when α is between 0.6 and 1.0, the number of trials required for convergence is small and stable. This reflects the tendency that when α exceeds 1.2, the behavior of the optical medium becomes chaotic, and the performance of the RC algorithm improves near the chaos transition point of the optical medium. Therefore, it is desirable to set the spectral radius of the RC algorithm to a value as large as possible without reaching the chaos transition point.

[0053] In this embodiment, as another example of learning, the task is to play a computer game Pong (by Atari) against a computer and to win. tare pre-processed by the pre-processing unit described in the third embodiment and input to the RC model. FIG. 8(a) is a diagram for explaining the pre-processing unit 613 used in this embodiment. FIG. 8(b) is a graph showing the state of learning. The pre-processing unit 613 has a convolution filter having a plurality of intermediate layers 3130, 3131, and 3133, and a pooling layer 3135. A calculation device having the RC model 10 is provided behind the convolution filter. In the intermediate layer 3130, an 8×8 kernel filter 3137 performs convolution processing on 84×84×4 input data while sliding by 4×4 pixels. In the intermediate layer 3131, a 4×4 kernel filter 3132 performs convolution processing on 20×20×32 data while sliding by 2×2 pixels. In the intermediate layer 3133, a 3×3 kernel filter 3134 performs convolution processing on 9×9×64 data while sliding by 1×1 pixels. As a result, a feature map is output each time a convolution process is performed. The feature maps are reduced in number of pixels by a pooling layer 3135, and then combined to form a combination layer 3136.

[0054] Learning was performed using the method shown in equations (7) and (9). In equation (9), n = 5, γ = 0, 99, and the number of RC nodes is 4000. The horizontal axis of FIG. 8(b) indicates the number of trials, and the vertical axis indicates the score difference (the value obtained by subtracting the target's score from the computer's score). When the score difference is "+", it indicates that the target has won. As is clear from FIG. 8(b), the score difference increases as the number of trials increases, and then converges stably. Such results indicate that even a relatively difficult task such as game playing can be learned using the RC model of the present disclosure. [Explanation of symbols]

[0055] 10,10a,10b...RC Model 111...Agent 112,512,513,612...Arithmetic unit 113 Condition measuring device 114... Behavior control device 115 Physical medium 116,117 Signal conversion device 118...Storage section 119...electronic arithmetic unit 211,311,411...RC algorithm 212,312,412...Action decision algorithm 613...Pretreatment section 3130, 3131, 3133... Middle layer 3132, 3134, 3137... Kernel filters 3135 Pooling Layer 3136...Binding layer

Claims

1. A computing device for reinforcement learning that learns a strategy for a subject to execute a task, comprising: A computer system having an input layer that receives input information related to a state of an object, an intermediate layer that learns the policy of the object based on the input information received in the input layer, and an output layer that outputs output information related to the policy; a behavior determining unit that outputs information regarding the target's behavior based on the output information output from the computer system, the computer system includes a physical medium that physically transforms the input information and performs at least a portion of the function of learning the strategy of the subject; A computation device for reinforcement learning, wherein Ω corresponding to mutual coupling within the physical medium is an arbitrary matrix representing mutual coupling, and a spectral radius α of the matrix Ω is equal to or greater than 0.6 and equal to or less than 1.

2.

2. The reinforcement learning arithmetic device of claim 1, wherein Ω reflects the following ring topology, and a gain α of the returned light within the ring, which corresponds to a spectral radius of the matrix form of the ring topology, is set between 0.6 and 1.

2.

3. The reinforcement learning computation device according to claim 1 , wherein the computer system comprises: an actor unit that determines the policy of the target based on the input information; and a critic unit that estimates a value of the state based on the input information.

4. The reinforcement learning computation device according to claim 3 , comprising: a first computer system functioning as the actor unit; and a second computer system functioning as the critic unit.

5. 5. The reinforcement learning arithmetic device according to claim 1, further comprising a pre-processing unit for compressing the input information, the pre-processing unit being provided in a front stage of the computer system.

6. The computation device for reinforcement learning according to claim 1 , wherein the physical medium is a nonlinear optical element.

7. 7. The reinforcement learning calculation device according to claim 1, wherein the computer system includes an input conversion unit that converts the input information corresponding to the physical medium, an output conversion unit that converts the output information output from the physical medium according to a calculation process to be performed later, a calculation unit that executes a calculation process related to reinforcement learning based on the output information converted by the output conversion unit, and outputs information regarding the behavior of the target based on a calculation result of the calculation unit.

8. 1. A computational method for reinforcement learning in which a subject learns a strategy for performing a task, comprising: A computer system having an input layer for inputting information, an intermediate layer for performing a calculation based on the input information input to the input layer, and an output layer for outputting a result of the calculation, inputting input information regarding a state of the subject into the input layer; the intermediate layer learning the policy for the object based on the input information; and outputting information about the target's behavior based on information about the strategy obtained by the learning. The step of learning the policy of the subject is at least partially performed by a physical medium that physically transforms the input information, Ω, which corresponds to the mutual coupling within the physical medium, is an arbitrary matrix representing the mutual coupling, and the spectral radius α of the matrix Ω is 0.6 or more and 1.2 or less. A computational method for reinforcement learning.

Citation Information

Patent Citations

  • Optical signal processing circuit

    JP2018200391A