Learning device, learning method, and program

Abstract

The present invention provides a training device comprising: a data acquisition means that acquires data based on observations related to an action to be learned; a training means that trains a model to be trained by using the data based on observations related to an action to be learned; a storage means that stores the data based on observations related to an action to be learned; and a forgetting / retraining means that uses the data stored by the storage means to carry out retraining and forgetting the training of the model to be trained.

Description

Learning device, learning method, and recording medium

[0001] The present invention relates to a learning device, a learning method, and a recording medium.

[0002] In some cases, a model is retrained using accumulated data. For example, a model learning device described in Patent Literature 1 inputs test data into a model trained using training data during a test period and outputs the degree of anomaly of the data. This model learning device also retrains the model using the accumulated test data and training data.

[0003] International Publication No. 2019 / 138655

[0004] When relearning a model that includes uncertainty using accumulated data, it is preferable to be able to avoid underestimating the uncertainty of the model by using the same data multiple times.

[0005] An example of an object of the present invention is to provide a learning device, a learning method, and a recording medium that can solve the above-mentioned problems.

[0006] According to a first aspect of the present invention, a learning device includes a data acquisition means for acquiring data based on observations of the behavior of a learning object, a learning means for learning a model of the learning object using the data based on the observations of the behavior of the learning object, a storage means for storing the data based on the observations of the behavior of the learning object, and a forgetting / relearning means for forgetting and relearning the learning of the model of the learning object using the data stored in the storage means.

[0007] According to a second aspect of the present invention, a learning method includes a computer acquiring observation-based data regarding the behavior of a learning object, learning a model of the learning object using the observation-based data regarding the behavior of the learning object, storing the observation-based data regarding the behavior of the learning object, and forgetting and relearning the learning of the model of the learning object using the stored data.

[0008] According to a third aspect of the present invention, a recording medium is a recording medium having recorded thereon a program that causes a computer to acquire data based on observations of the behavior of a learning object, learn a model of the learning object using the data based on observations of the behavior of the learning object, store the data based on observations of the behavior of the learning object, and forget and re-learn the model of the learning object using the stored data.

[0009] According to the present invention, when re-learning a model that includes uncertainty using accumulated data, it is expected that it will be possible to avoid underestimating the uncertainty of the model by using the same data multiple times.

[0010] FIG. 1 is a diagram showing an example of the configuration of a learning system according to at least one embodiment. FIG. 2 is a diagram showing an example of the configuration of a learning device according to at least one embodiment. FIG. 3 is a diagram showing an example of data input / output in a learning system according to at least one embodiment. FIG. 4 is a diagram showing an example of a processing procedure by which a learning device according to at least one embodiment learns a target model. FIG. 5 is a diagram showing a first example of a change in error of a diffusion coefficient in an experiment according to at least one embodiment. FIG. 6 is a diagram showing a second example of a change in error of a diffusion coefficient in an experiment according to at least one embodiment. FIG. 7 is a diagram showing an example of processing time when forgetting and relearning are performed collectively using a plurality of training data according to at least one embodiment. FIG. 8 is a diagram showing an example of the configuration of a learning device according to at least one embodiment. FIG. 9 is a diagram showing an example of a processing procedure in a learning method according to at least one embodiment. FIG. 10 is a schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0011] The following describes embodiments of the present invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Below, characters with a dot (.) above them may be indicated by adding a "'" next to them. For example, a v with a dot above it may also be written as v'. Furthermore, characters with a circumflex may be indicated by adding a "^" next to them. For example, a D with a circumflex may also be written as D^.

[0012] First Embodiment Fig. 1 is a diagram illustrating an example of the configuration of a learning system according to at least one embodiment. In the example of Fig. 1, the learning system 1 includes a learning device 100 and an environment 920. The environment 920 includes a learning object 910.

[0013] The learning object 910 is the subject of model learning by the learning system 1. Model learning here refers to adjusting the model, such as updating the parameter values ​​of the model, using training data. Model learning can also be referred to as model training. Updating parameter values ​​through model learning is also referred to as parameter learning.

[0014] In the following, an example will be described in which the learning object 910 is a mobile robot. However, the learning object 910 is not limited to a specific one, and various objects can be used for modeling the behavior in response to a control command. The behavior of the learning object 910 referred to here may be a process that does not involve movement of the learning object 910.

[0015] For example, the learning object 910 may be some kind of device such as a machine tool, or may be some kind of equipment such as a factory, a factory line, or a power plant.

[0016] In the following, an example will be described in which the learning device 100 controls the learning object 910. However, the form of control over the learning object 910 is not limited to the form in which the learning device 100 controls the learning object 910. For example, a control device configured as a device separate from the learning device 100 may control the learning object 910. Alternatively, the learning object 910 may operate autonomously. When the learning object 910 operates autonomously, the behavior determined by the learning object 910 itself can be used as a control command.

[0017] In the following, an example will be described in which the learning device 100 determines a control command for the learning object 910 based on a model of the learning object 910. However, the control of the learning object 910 may be performed independently of the learning of the model by the learning device 100.

[0018] The environment 920 is the operating environment of the learning object 910. The environment 920 is an observation target for the learning device 100 to learn a model of the learning object 910. As described above, the environment 920 is assumed to include the learning object 910. Therefore, at least one of the learning object 910 and its surrounding environment is an observation target.

[0019] The learning device 100 learns a model of the learning object 910. The model of the learning object 910 here is a model that receives inputs of the state of the environment 920 and a control command for the learning object 910, and outputs information about the behavior of the learning object 910. The model of the learning object 910 is also referred to as an object model, or simply as a model. In the following, an example will be described in which the object model is expressed as a stochastic differential equation. However, the object model may be expressed in a format other than a stochastic differential equation, such as receiving inputs of the state of the environment 920 and a control command for the learning object 910, and outputting the next state of the environment 920.

[0020] Furthermore, the object model includes a stochastic element, such as by expressing the object model in the form of a stochastic differential equation, because the behavior of the learning object 910 may include a stochastic element.

[0021] It is conceivable that the learning object 910 will behave differently in response to the same control command due to the influence of noise, noise, and the like, among the state items of the environment 920 that are not being observed. For example, if the learning object 910 is a mobile robot, it is conceivable that the learning object 910 will behave differently in response to the same control command due to the friction coefficient of the road surface on which the learning object 910 is traveling, noise mixed into the current to the motor that drives the learning object 910, and the like.

[0022] However, the learning device 100 can also be applied to cases where the target model is expressed as a deterministic model that does not include probabilistic elements.

[0023] The learning device 100 learns the target model. In particular, the learning device 100 learns the target model each time state information of the environment 920 is acquired. This learning can be considered online learning. Furthermore, the learning device 100 stores, as training data, data used to learn the target model each time state information of the environment 920 is acquired. When a predetermined number of pieces of training data or more have been accumulated, the learning device 100 learns the target model using the accumulated training data in addition to learning the target model each time state information of the environment 920 is acquired. In this case, the learning device 100 forgets the learning of the target model and then trains the target model. Learning using accumulated training data is also referred to as relearning. The learning device 100 may be configured using a computer.

[0024] The state information of the environment 920 handled by the learning device 100 may be information obtained by measuring the environment 920 with a sensor or the like, or may be information calculated from the measurement values. In this case, the measurement values ​​are referred to as observation data of the state of the environment 920, or simply as observation data. This observation data is an example of observation data related to the behavior of the learning target.

[0025] For example, when the learning device 100 acquires the acceleration of the learning object 910 as one item of the state information of the environment 920, the learning device 100 may acquire the measured value of acceleration using an acceleration sensor, or may acquire a calculated value of acceleration calculated from the position or velocity of the learning object 910.

[0026] Fig. 2 is a diagram showing an example of the configuration of the learning device 100. In the configuration shown in Fig. 2, the learning device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a storage unit 180, and a processing unit 190. The processing unit 190 includes an object control unit 191, a data acquisition unit 192, a learning unit 193, a sampling unit 194, and a forgetting / relearning unit 195.

[0027] The communication unit 110 communicates with other devices. For example, the communication unit 110 acquires state information of the environment 920 from the learning object 910 and / or sensors provided in the environment 920. The communication unit 110 also transmits control commands to the learning object 910.

[0028] The display unit 120 has a display screen such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 may display information related to the learning of the target model, such as parameter values ​​of the target model. The display unit 120 may also display information related to the control of the target model, such as control commands for the target model.

[0029] The operation input unit 130 includes input devices such as a keyboard and a mouse, and accepts user operations. For example, the operation input unit 130 may accept a user operation for inputting instructions related to the learning of the target model, such as a user operation for specifying a start condition for relearning the target model. The operation input unit 130 may also accept a user operation for inputting instructions related to the control of the learning object 910, such as a user operation for instructing the learning object 910 to perform an operation.

[0030] The storage unit 180 stores various types of data. For example, the storage unit 180 stores a target model and parameter values ​​of the target model. The storage unit 180 also stores training data used to learn the target model. Both the individual pieces of training data stored in the storage unit 180 and the entire training data are referred to as training data. The storage unit 180 is configured using a storage device included in the learning device 100.

[0031] Processing unit 190 performs various processes by controlling each unit of learning device 100. The functions of processing unit 190 may be performed by a CPU (Central Processing Unit) included in learning device 100 reading and executing a program from storage unit 180.

[0032] The object control unit 191 controls the learning object 910. Specifically, the object control unit 191 determines a control command for the learning object 910. Then, the object control unit 191 transmits the determined control command to the learning object 910 via the communication unit 110.

[0033] There is no particular limitation on the method by which the object control unit 191 determines the control command for the learning object 910. For example, the object control unit 191 may search for a control command for the learning object 910 using an evaluation function that indicates a better evaluation the smaller the difference between the action that the learning object 910 is desired to perform and the action of the learning object 910 calculated using the object model, but the method is not limited to this.

[0034] The data acquisition unit 192 acquires state information of the environment 920. The data acquisition unit 192 may acquire information obtained by measuring the environment 920 with a sensor or the like as the state information, or may acquire information calculated from the measurement values ​​as the state information. Alternatively, the data acquisition unit 192 may acquire a combination of information obtained by measuring the environment 920 with a sensor or the like and information calculated from the measurement values ​​as the state information.

[0035] Both the data for each item of state information acquired by the data acquisition unit 192 and the data combining the data for each item of state information acquired by the data acquisition unit 192 are referred to as state information. The state information acquired by the data acquisition unit 192 corresponds to an example of data based on observations related to the behavior of the learning target.

[0036] The learning unit 193 learns the target model. In particular, the learning unit 193 learns the target model each time state information of the environment 920 is acquired. For example, the learning unit 193 may acquire state information of the environment 920 for each time step and learn the target model. As described above with respect to the learning device 100, the learning performed by the learning unit 193 can be considered as online learning. The learning unit 193 stores the data used for learning in the storage unit 180 as training data for re-learning.

[0037] The sampling unit 194 samples individual training data to be used for relearning the target model from among all the training data stored in the storage unit 180. The method by which the sampling unit 194 samples the training data is not limited to a specific method. For example, the sampling unit 194 may randomly sample the training data. Alternatively, the sampling unit 194 may sample the training data such that training data that has not been sampled for a longer period of time is more likely to be sampled.

[0038] The forgetting / relearning unit 195 re-learns the target model using the training data sampled by the sampling unit 194. In particular, the forgetting / relearning unit 195 re-learns the target model after forgetting the learning of the target model using the training data sampled by the sampling unit 194.

[0039] The following further describes the learning of the target model by the learning device 100. Here, it is assumed that the model shown in equation (1) is used as the target model.

[0040]

[0041] t indicates time. dt indicates the amount of change in time, i.e., time. x is a vector indicating the state information of the environment 920. However, the number of elements of x may be one. In other words, x may be represented by a scalar. x ｔ is a vector indicating the state information of the environment 920 at time t. In this way, the value of something at time t is indicated by adding a subscript t to the variable indicating that thing. ｔ is a vector indicating the amount of change in the state information of the environment 920 at time t.

[0042] u is a vector indicating a control command for the learning object 910. However, the number of elements of u may be one. In other words, u may be represented by a scalar. ｔ is a vector indicating a control command for the learning object 910 at time t.

[0043] f(x, u) indicates an element of the behavior of the learning object 910 in response to the state information x and the control command u that can be calculated from the structure of the learning object 910. Therefore, the function f is assumed to be known.

[0044] g ξ (x, u) indicates an unknown bias element in the behavior of the learning object 910 in response to the state information x and the control command u. In other words, once the value of the state information x and the value of the control command u are determined, ξ The value of (x, u) is uniquely determined, but the function g ξ is assumed to be unknown.

[0045] g ξ (x, u) corresponds to an example of a deterministic partial model. The deterministic partial model here is a partial model that outputs a value deterministically in response to state information and control commands. ξ (x, u) can be a vector, but for simplicity, it is assumed to be a scalar. ξ When (x, u) is treated as a vector, for example, a method of creating a model for the number of dimensions of the vector can be considered. ξ As a model representing (x, u), the model shown in equation (2) is used.

[0046]

[0047] φ(x, u) denotes a vector that takes a value according to the state information x and the control command u. Various vectors whose values ​​can be calculated can be used as the vector φ(x, u). For example, the vector shown in Equation (3) may be used as the vector φ(x, u), but is not limited to this.

[0048]

[0049] x Ｌ and y Ｌ indicates the x-coordinate value and y-coordinate value of the position of the learning object 910. θ indicates the orientation of the learning object 910. v indicates the velocity of the learning object 910. ω indicates the angular velocity of the orientation of the learning object 910. v' indicates the acceleration of the learning object 910. ω' indicates each acceleration in the orientation of the learning object 910. The superscript T attached to a vector or matrix indicates the transpose of the vector or matrix.

[0050] ξ is the function g ξ The number of elements of the parameter ξ vector is the same as the number of elements of the vector φ(x, u). The parameter ξ is expressed as in equation (4).

[0051]

[0052] N represents an integer obtained by subtracting 1 from the number of elements of the vector φ(x, u). ０ , ..., ξ Ｎ and ξ denote scalar parameters. Updating the value of the parameter ξ by learning the target model is also called bias learning. The parameters of the bias elements of the model are also called bias parameters. The parameter ξ is an example of a bias parameter.

[0053] D ψ(x, u) is a coefficient indicating the magnitude of an unknown stochastic element in the behavior of the learning object 910 in response to the state information x and the control command u. The stochastic element here is an element that does not necessarily take the same value even for the same state information x and the same control command u. In equation (1), the state information x ｔ and control command u ｔ The amount of change in the probabilistic element of the behavior of the learning object 910 relative to ψ (x ｔ , u ｔ ) A Ｒ It is indicated by dw.

[0054] w denotes a vector of random numbers. ｔ denotes the vector of random numbers at time t. ｔ is the random number w at time t ｔ It is a random number generated from a normal distribution with mean 0 and variance dt. Ｒ represents the properties of the system. For example, if we consider a wheeled mobile robot as the system and assume that the generation of random numbers appears as the angle of the wheels, then A Ｒ may convert the rotation angle of the wheels into the amount of movement of the robot.

[0055] The probabilistic element in the behavior of the learning object 910 does not necessarily take the same value even for the same state information x and the same control command u. ψ The value of (x, u) is uniquely determined once the value of the state information x and the value of the control command u are determined. ψ is assumed to be unknown. ψ (x, u) is also called the diffusion coefficient.

[0056] D ψ (x, u) corresponds to an example of a stochastic partial model. The stochastic partial model here is a partial model that outputs a value stochastically in response to state information and control commands. Here, the diffusion coefficient D ψ As a model representing (x, u), the model shown in equation (5) is used.

[0057]

[0058] ψ is a function Dψ is a vector indicating the parameters of the parameter ψ. The number of elements in the vector of the parameter ψ is the same as the number of elements in the vector φ(x, u). The parameter ψ is expressed as in equation (6). Updating the value of the parameter ψ by learning the target model is also called learning the diffusion coefficient. The parameter of the variance of the probabilistic element of the model (the degree of dispersion of the probability distribution) is also called the variance parameter or the diffusion coefficient parameter. The parameter ψ is an example of a variance parameter (diffusion coefficient parameter).

[0059]

[0060] ψ ０ , ..., ψ Ｎ and σ and σ each represent a scalar parameter. The learning unit 193 learns the target model expressed by equations (1) to (6) so that the target model conforms to the control command determined by the target control unit 191 and the state information of the environment 920 acquired by the data acquisition unit 192.

[0061] In the learning of the target model, the learning unit 193 uses a coefficient λ for updating the value of the parameter ξ. ξ The value of may be calculated based on equation (7).

[0062]

[0063] x ｎ is a vector indicating the value of the state information x of the environment 920. In equation (7), x ｎ indicates the value of the state information x acquired by the data acquisition unit 192. ｎ is a vector indicating the value of the control command u. In equation (7), u ｎ The object control unit 191 sets the state information x ｎ The coefficient λ is the value of the control command u determined based on ξ is also referred to as a learning coefficient. The learning unit 193 uses a coefficient matrix Λ to update the value of the parameter ξ. ξ The value of may be updated based on equation (8).

[0064]

[0065] Coefficient matrix Λξ The initial value of may be set in advance to a constant multiple of a unit matrix including a zero matrix. Then, the learning unit 193 may update the value of the parameter ξ in learning based on equation (9).

[0066]

[0067] The superscript -1 on a matrix indicates the inverse matrix. ｎ indicates the bias obtained from the observed data. The bias is (dx-f ｎ ) is calculated as follows. (g ｎ −φ(x ｎ , u ｎ ) Ｔ ξ) can be considered as calculating the error by subtracting the calculated bias value from the observed bias value. ξ (Λ ξ ) ―１ φ(x ｎ , u ｎ ) is (g ｎ −φ(x ｎ , u ｎ ) Ｔ ξ) is a coefficient that is multiplied by λ ξ (Λ ξ ) ―１ φ(x ｎ , u ｎ ) is g ｎ This is to emphasize data when noise is large, avoiding the effect of data with small noise on learning being small, and to adjust the influence of that data on the entire model by reducing the amount of updates as learning progresses.

[0068] In equation (9), the coefficient matrix Λ updated in equation (8) is ξ The initial value of the parameter ξ is, for example, set in advance to [0,...,0]. Ｔ In addition, the learning unit 193 may set the coefficient λ for calculating the value of the parameter ψ as ψ The value of may be calculated based on equation (10).

[0069]

[0070] Coefficient λ ψ is also referred to as a learning coefficient. The learning unit 193 uses a coefficient matrix Λ to update the value of the parameter ψ. ψ The value of may be updated based on equation (11).

[0071]

[0072] Coefficient matrix Λ ψ The initial value of may be set in advance to a constant multiple of a unit matrix including a zero matrix. Then, the learning unit 193 may update the value of the parameter ψ in learning based on equation (12).

[0073]

[0074] D ｎ indicates the diffusion coefficient value (variance) obtained from the observation data. For example, (dx-f ｎ -g ｎ ) is D ｎ １／２ and A Ｒ Since it is a random number generated from a probability distribution based on dw, A Ｒ and dw is a scalar, D ｎ is (dx-f ｎ -g ｎ ) ２ / (A Ｒ ２ dt) can be calculated. (D ｎ −φ(x ｎ , u ｎ ) Ｔ ψ) can be thought of as calculating the error by subtracting the calculated diffusion coefficient from the observed diffusion coefficient.

[0075] λ ψ (Λ ψ ) －１ φ(x ｎ , u ｎ ) is (D ｎ −φ(x ｎ , u ｎ ) Ｔ λ is a coefficient that is multiplied by λ ψ (Λ ψ ) －１ φ(x ｎ , u ｎ ) is Dｎ This is to avoid the effect of emphasizing data when the noise contained in it is large, thereby reducing the influence of data on learning when the noise is small, and to adjust the influence of that data on the entire model by reducing the amount of updates as learning progresses.

[0076] In equation (12), the coefficient matrix Λ updated in equation (11) is ψ The initial value of the parameter ψ is set in advance to, for example, [0,...,0]. Ｔ However, the method by which the learning unit 193 learns the target model is not limited to a specific method.

[0077] Although the above description has been given of the case where the learning device 100 learns the target model, it is also possible for the learning device 100 to forget the learning of the target model. Forgetting the learning of the target model here refers to canceling out the update of the parameter values ​​in the learning of the target model. Forgetting the learning of the target model is also simply referred to as forgetting. For example, the learning device 100 calculates the coefficient matrix Λ based on equation (13). ξ The value of may be updated.

[0078]

[0079] During forgetting, the learning device 100 uses the coefficient λ calculated during learning. ξ To this end, the storage unit 180 uses the value of the coefficient λ ξ Then, the learning device 100 may update the value of the parameter ξ in the forgetting process based on equation (14).

[0080]

[0081] In equation (14), the coefficient matrix Λ updated in equation (13) is ξ The value of the parameter ξ is used. Updating the value of the parameter ξ by forgetting is also called forgetting of the bias. Furthermore, the learning device 100 uses the coefficient matrix Λ based on the equation (15). ψ The value of may be updated.

[0082]

[0083] During forgetting, the learning device 100 uses the coefficient λ calculated during learning. ψ To this end, the storage unit 180 uses the value of the coefficient λ ψ Then, the learning device 100 may update the value of the parameter ψ in the forgetting process based on equation (16).

[0084]

[0085] In equation (16), the coefficient matrix Λ updated in equation (15) is ψ The value of the parameter ψ is used. Updating the value of the parameter ψ by forgetting is also called forgetting of the diffusion coefficient. However, the method by which the learning device 100 performs forgetting is not limited to a specific method.

[0086] The forgetting / relearning unit 195 performs relearning of the target model expressed by equations (1) to (6) so that the target model conforms to the control commands and state information included in the training data stored in the storage unit 180. In particular, the forgetting / relearning unit 195 performs relearning after forgetting.

[0087] Here, an example will be described in which the learning device 100 uses a vector expressed as in equation (17) as φ(x, u).

[0088]

[0089] Furthermore, the individual training data stored in the storage unit 180 is (x, u, dx, λ ξ , λ ψ , D), where (x, u, dx, λ ξ , λ ψ ,D) are x, u, dx, λ ξ , λ ψ , and data including each data of D. Here, dx is expressed as in equation (18).

[0090]

[0091] dx Ｌ is the x-coordinate value x of the position of the learning object 910 Ｌ This indicates the amount of change in Ｌis the y-coordinate value y of the position of the learning object 910 Ｌ dθ represents the amount of change in the orientation θ of the learning object 910. Here, ξ is expressed as ｛ｘＬ，ｙＬ，θ｝ ψ is also written as ψ ｛ｘＬ，ｙＬ，θ｝ Also written as λ ξ λ ξ ｛ｘＬ，ｙＬ，θ｝ Also written as λ ψ λ ψ ｛ｘＬ，ｙＬ，θ｝ It can also be written as Λ ξ Λ ξ ｛ｘＬ，ｙＬ，θ｝ It can also be written as Λ ψ Λ ψ ｛ｘＬ，ｙＬ，θ｝ Also written as D. ｛ｘＬ，ｙＬ，θ｝ It can also be written as:

[0092] In the following formula, the subscript {x Ｌ , y Ｌ , θ} means that the calculation is performed separately for each element. ｛ｘＬ，ｙＬ，θ｝ =B {ｘＬ，ｙＬ，θ} +C {ｘＬ，ｙＬ，θ} The description is A ｘＬ =B ｘＬ +C ｘＬ and A yＬ =B yＬ +C yＬ and A θ =B θ +C θ Show that the following holds true.

[0093] The forgetting and relearning unit 195 updates the parameters ξ in the forgetting and relearning processes in the following procedure: (1) The forgetting and relearning unit 195 updates the coefficient matrix Λ based on the equation (19). ξ ｛ｘＬ，ｙＬ，θ｝ Update the value of

[0094]

[0095] (2) The forgetting / relearning unit 195 calculates the forgetting parameter ξ based on the equation (20). ｛ｘＬ，ｙＬ，θ｝ Update the value of .

[0096]

[0097] (3) The forgetting and relearning unit 195 calculates the coefficient λ based on the equation (21). ξ ｛ｘＬ，ｙＬ，θ｝ Calculate the value of

[0098]

[0099] (4) The forgetting and relearning unit 195 calculates the coefficient matrix Λ based on the equation (22). ξ ｛ｘＬ，ｙＬ，θ｝ Update the value of

[0100]

[0101] (5) The forgetting and relearning unit 195 calculates the parameter ξ in relearning based on the equation (23). ｛ｘＬ，ｙＬ，θ｝ Update the value of .

[0102]

[0103] (6) The forgetting and relearning unit 195 uses the coefficient λ calculated in (3) ξ ｛ｘＬ，ｙＬ，θ｝ The value of is overwritten and saved in the storage unit 180.

[0104] The forgetting and relearning unit 195 updates the parameter ψ in the forgetting and relearning processes in the following procedure: (7) The forgetting and relearning unit 195 updates the coefficient matrix Λ based on the equation (24). ψ ｛ｘＬ，ｙＬ，θ｝ Update the value of

[0105]

[0106] (8) The forgetting / relearning unit 195 calculates the forgetting parameter ψ based on the equation (25). ｛ｘＬ，ｙＬ，θ｝ Update the value of λ ψ ｛ｘＬ，ｙＬ，θ｝ and D ｛ｘＬ，ｙＬ，θ｝ The value retrieved from the storage unit 180 is used.

[0107]

[0108] (9) The forgetting and relearning unit 195 calculates D ｛ｘＬ，ｙＬ，θ｝ Calculate the value of

[0109]

[0110] The left side of equation (26) ({dxＬ , dy Ｌ , dθ}-(f ｛ｘＬ，ｙＬ，θ｝ (x, u)+φ(x, u) Ｔ ξ ｛ｘＬ，ｙＬ，θ｝ ) is the right-hand side (√D ｛ｘＬ，ｙＬ，θ｝ (ARdw) ｛ｘＬ，ｙＬ，θ｝ ) In this case, if the variance is calculated assuming that the mean of the stochastic elements is 0, it is expressed as in equation (27).

[0111]

[0112] V represents the variance. E represents the expected value. ｛ｘＬ，ｙＬ，θ｝ When the equation for is expressed as Equation (28),

[0113]

[0114] √dt is a vector in which the square roots of the amount of change in time are arranged, and the number of dimensions differs depending on the system.

[0115] (10) The forgetting and relearning unit 195 calculates the coefficient λ based on the equation (29). ψ ｛ｘＬ，ｙＬ，θ｝ Calculate the value of

[0116]

[0117] (11) The forgetting and relearning unit 195 calculates the coefficient matrix Λ based on the equation (30). ψ ｛ｘＬ，ｙＬ，θ｝ Update the value of

[0118]

[0119] (12) The forgetting and relearning unit 195 calculates the parameter ψ in relearning based on the equation (31). ｛ｘＬ，ｙＬ，θ｝ Update the value of .

[0120]

[0121] (13) The forgetting and relearning unit 195 uses the D calculated in (9) ｛ｘＬ，ｙＬ，θ｝ and the coefficient λ calculated in (10) ψ ｛ｘＬ，ｙＬ，θ｝ The value of is overwritten and saved in the storage unit 180.

[0122] In equations (19) to (31), the subscript {x Ｌ , y Ｌ , θ} is provided as an index for identifying state information. The storage unit 180 distinguishes each variable using this index and stores the value of the variable. As a result, the storage unit 180 stores, for each piece of state information, various data used in learning using that state information (learning by the learning unit 193 or relearning by the forgetting / relearning unit 195) as training data.

[0123] This allows the forgetting / relearning unit 195 to recalculate the values ​​of past learning using the training data used for relearning in the forgetting process and offset them from the parameter values ​​of the learning target. In other words, the forgetting / relearning unit 195 can pinpoint and forget the results of past learning using the training data used for relearning.

[0124] The forgetting / relearning unit 195 may perform forgetting and relearning collectively using multiple pieces of training data. Here, the learning device 100 performs forgetting and relearning collectively using k+1 pieces of training data, where k is an integer greater than or equal to 0. For example, as shown in equation (32), the k+1 coefficients λ are grouped into a diagonal matrix to form a matrix Ω ［ｎ－ｋ，ｎ］ ｛ξ，ψ｝ Let's say.

[0125]

[0126] Here, {ξ, ψ} represents either ξ or ψ. The forgetting and relearning unit 195 calculates a coefficient λ for updating the value of the parameter ξ. ξ ｎ－ｋ , ..., λ ξ ｎ A diagonal matrix Ω with ξ [ｎ－ｋ，ｎ] and the coefficient λ for updating the value of the parameter ψ. ψ ｎ－ｋ , ..., λ ψ ｎ A diagonal matrix Ω with ψ [ｎ－ｋ，ｎ] and n−k, . . . , n represent indices for identifying k+1 training data.

[0127] Also, as shown in equation (33), φ(x ｎ－ｋ , u ｎ－ｋ ), ..., φ(x ｎ , u ｎ ) is expressed as a matrix that summarizes the vertical vectors, and the matrix Φ ［ｎ－ｋ，ｎ］ Let's say.

[0128]

[0129] Also, as shown in equation (34), g ｎ－ｋ , ..., g ｎ are grouped into vertical vectors to form vector g ［ｎ－ｋ，ｎ］ Let's say.

[0130]

[0131] Also, as shown in equation (35), D ｎ－ｋ , ..., D ｎ Vector D ［ｎ－ｋ，ｎ］ Let's say.

[0132]

[0133] In the relearning, the forgetting / relearning unit 195 calculates the coefficient matrix Λ based on the equation (36). ξ and the coefficient matrix Λ ψ Update the value of

[0134]

[0135] As described above, {ξ, ψ} represents either ξ or ψ. The forgetting and relearning unit 195 calculates the matrix Ω ξ [ｎ－ｋ，ｎ] Using the above, the coefficient matrix Λ is calculated based on the equation (36). ξ and update the value of the matrix Ω ψ [ｎ－ｋ，ｎ] Using the above, the coefficient matrix Λ is calculated based on the equation (36). ψ Update the value of

[0136] Then, the forgetting / relearning unit 195 updates the value of the parameter ξ in the relearning and updates the value of the parameter ψ in the relearning based on equation (37).

[0137]

[0138] As described above, {ξ, ψ} represents either ξ or ψ. The forgetting and relearning unit 195 calculates the matrix Ω ξ [ｎ－ｋ，ｎ] and the vector Λ ξ The value of the parameter ξ is updated based on the equation (37) using ψ [ｎ－ｋ，ｎ] and the vector Λ ψ is used to update the value of the parameter ψ based on equation (37).

[0139] In the forgetting process, the forgetting / relearning unit 195 calculates the coefficient matrix Λ based on the equation (38). ξ and the coefficient matrix Λ ψ Update the value of

[0140]

[0141] As described above, {ξ, ψ} represents either ξ or ψ. The forgetting and relearning unit 195 calculates the matrix Ω ξ [ｎ－ｋ，ｎ] Using the above, the coefficient matrix Λ is calculated based on the equation (38). ξ and update the value of the matrix Ω ψ [ｎ－ｋ，ｎ] Using the above, the coefficient matrix Λ is calculated based on the equation (38). ψ Update the value of

[0142] Then, the forgetting / relearning unit 195 updates the value of the forgetting parameter ξ and the value of the forgetting parameter ψ based on equation (39).

[0143]

[0144] As described above, {ξ, ψ} represents either ξ or ψ. The forgetting and relearning unit 195 calculates the matrix Ω ξ [ｎ－ｋ，ｎ] and the vector Λ ξ , and update the value of the parameter ξ based on equation (39) using ψ [ｎ－ｋ，ｎ] and the vector Λ ψ is used to update the value of the parameter ψ based on equation (39).

[0145] Fig. 3 is a diagram showing an example of data input / output in the learning system 1. In the example of Fig. 3, the object control unit 191 determines a control command u for the learning object 910 using the object model, and transmits the determined control command u to the learning object 910 via the communication unit 110. The object control unit 191 also outputs the determined control command u to the learning unit 193. The learning object 910 receives the control command u determined by the object control unit 191, and operates in accordance with the received control command u.

[0146] The data acquisition unit 192 acquires observation information on the state of the environment 920 and outputs state information x of the environment 920 to the learning unit 193. The learning unit 193 acquires the control command u determined by the object control unit 191 and the state information x obtained from the data acquisition unit 192. The learning unit 193 learns the object model based on the obtained control command u and state information x, and outputs the updated model as the learning result to the object control unit 191. Specifically, the learning unit 193 may be configured to output the updated value of the parameter ξ and the updated value of the parameter ψ to the object control unit 191.

[0147] Furthermore, the learning unit 193 stores data including the control command u, the state information x, the amount of change dx, the learning coefficient λ, and the observation diffusion coefficient D in the storage unit 180 as training data for the forgetting and relearning unit 195 to perform forgetting and relearning. The observation diffusion coefficient D here is a diffusion coefficient calculated from the deviation of the actual movement amount of the learning object 910 from the ideal movement amount (calculated movement amount) of the learning object 910 when control is executed.

[0148] Furthermore, the learning unit 193 determines whether the number of pieces of training data stored in the storage unit 180 is equal to or greater than a predetermined threshold. If it is determined that the number of pieces of training data is equal to or greater than the predetermined threshold, the learning unit 193 outputs a trigger to prompt forgetting and relearning to the sampling unit 194 and the forgetting / relearning unit 195, and further outputs the updated values ​​of the parameter ξ and the updated values ​​of the parameter ψ to the forgetting / relearning unit 195 as learning results.

[0149] When the sampling unit 194 acquires a trigger from the learning unit 193, it samples the training data stored in the storage unit 180 and outputs the sampled training data to the forgetting and relearning unit 195. The sampled training data is also referred to as sampling data.

[0150] When the forgetting / relearning unit 195 acquires a trigger from the learning unit 193, it performs forgetting and relearning using the training data sampled by the sampling unit 194. As a result, the forgetting / relearning unit 195 further updates the values ​​of the parameter ξ and the parameter ψ acquired from the learning unit 193. The forgetting / relearning unit 195 outputs the updated model as the learning result to the target control unit 191. Specifically, the learning unit 193 may be configured to output the updated values ​​of the parameter ξ and the updated values ​​of the parameter ψ to the target control unit 191.

[0151] 4 is a diagram showing an example of a processing procedure for the learning device 100 to learn a target model. In the processing of FIG. 4, the learning unit 193 acquires data for learning the target model (step S101). Specifically, the learning unit 193 acquires a control command for the learning target 910 determined by the target control unit 191 and state information acquired by the data acquisition unit 192.

[0152] Next, the learning unit 193 learns the bias using the data obtained in step S101 (step S102), and then learns the diffusion coefficient using the data obtained in step S101 and the value of the parameter ξ obtained in step S102 (step S103).

[0153] Next, the learning unit 193 stores data (training data) to be used by the forgetting / relearning unit 195 for relearning the target model in the storage unit 180 (step S104). Next, the learning unit 193 determines whether the number of pieces of training data stored in the storage unit 180 is equal to or greater than a predetermined threshold (step S105).

[0154] If the learning unit 193 determines that the number of pieces of training data is less than the threshold (step S105: NO), the process returns to step S101. On the other hand, if the learning unit 193 determines that the number of pieces of training data is equal to or greater than the threshold (step S105: YES), the forgetting / relearning unit 195 forgets and re-learns the bias (step S201).

[0155] Next, the forgetting / relearning unit 195 forgets and re-learns the diffusion coefficient (step S202), and then updates the values ​​of the learning coefficient and the observed diffusion coefficient (step S203).

[0156] Next, the processing unit 190 determines whether a learning termination condition is met (step S204). The termination condition here is not limited to a specific one. For example, the termination condition here may be, but is not limited to, that the loop from steps S101 to S204 has been repeated a predetermined number of times.

[0157] If the processing unit 190 determines that the learning termination condition is not met (step S204: NO), the process returns to step S101. On the other hand, if the processing unit 190 determines that the learning termination condition is met (step S204: NO), the learning device 100 ends the process of FIG.

[0158] Next, a description will be given of an experiment on the learning device 100. An experiment was conducted in which the movement of a binary mobile robot was simulated using a model including a bias and a diffusion coefficient, and the learning device 100 was used to learn this model.

[0159] In the experiment, control commands for the robot were prepared for 100,000 steps at random speeds and angular velocities. Then, the bias and diffusion coefficient of the model were estimated from the control commands and the observed data of the position and posture of the robot operating according to the control commands. In the simulation, the bias g due to acceleration on the x-axis of the robot's coordinate system was calculated. ｐ was set as shown in equation (40).

[0160]

[0161] In other words, when acceleration occurs, the robot moves in the x-axis direction 1.13 times the calculated movement distance based on the control command. θ was set as shown in equation (41).

[0162]

[0163] When angular acceleration occurs, the robot rotates 1.14 times larger than the calculated rotation amount based on the control command. Also, the diffusion coefficient D ｐ was set as shown in equation (42).

[0164]

[0165] That is, the velocity of the robot on the x-axis of the coordinate system is determined according to a probability distribution having a variance equal to the sum of 0.3 times the magnitude of the calculated velocity based on the control command and 0.2 times the magnitude of the calculated angular velocity of the robot's orientation based on the control command. θ was set as shown in equation (43).

[0166]

[0167] That is, the orientation of the robot was determined according to a probability distribution having a variance equal to the sum of 0.1 times the magnitude of the calculated velocity based on the control command and 0.2 times the magnitude of the calculated angular velocity of the orientation of the robot based on the control command. Furthermore, the basis function φ(x, u) used in model training was expressed by Equation (44).

[0168]

[0169] 5 is a diagram showing a first example of the change in the error of the diffusion coefficient in the experiment. ｐ The graph in FIG. 5 shows the change in the error. The error was calculated using evaluation data prepared separately. The horizontal axis of the graph in FIG. 5 indicates the time step. The vertical axis indicates the index value of the error in the diffusion coefficient. The index value shown in Equation (45) is used as the index value of the error in the diffusion coefficient.

[0170]

[0171] N indicates the number of evaluation data. D indicates the actual value of the diffusion coefficient. In FIG. 5, D is the diffusion coefficient D on the x-axis of the robot coordinate system. ｐ , and D^ indicates the set value in the evaluation data. D^ indicates the calculated value of the diffusion coefficient. In FIG. 5, D^ is the diffusion coefficient D^ on the x-axis of the robot's coordinate system in the evaluation data, calculated by the learning unit 193 or the forgetting and relearning unit 195. ｐ It can be evaluated that the smaller the error index value shown in equation (45), the smaller the error in the diffusion coefficient.

[0172] The line L11 represents the diffusion coefficient D ｐ The line L12 shows the change in the index value of the error of the diffusion coefficient D ｐ The line L13 shows the change in the index value of the error of the diffusion coefficient D when the forgetting and relearning unit 195 performs forgetting and relearning 100 times each time the learning unit 193 performs learning. ｐ The change in the error index value is shown.

[0173] In the example of FIG. 5 , the diffusion coefficient D is smaller when the forgetting / relearning unit 195 performs forgetting and relearning once every time the learning unit 193 performs learning (see line L12) and when the forgetting / relearning unit 195 performs forgetting and relearning 100 times every time the learning unit 193 performs learning (see line L13) than when the forgetting / relearning unit 195 does not perform forgetting or relearning (see line L11). ｐ It can be said that the error is decreasing rapidly.

[0174] Fig. 6 shows a second example of the change in the error of the diffusion coefficient in the experiment. θ 6 shows how the error changes. The horizontal axis of the graph in FIG. 6 represents the time step. The vertical axis represents the index value of the error in the diffusion coefficient. In FIG. 6 as well, the index value shown in Equation (45) is used as the index value of the error in the diffusion coefficient.

[0175] In FIG. 6, D is the diffusion coefficient D in the robot's orientation. θ 6 shows the set values ​​in the evaluation data. In FIG. 6, D^ calculated by the learning unit 193 or the forgetting and relearning unit 195 is the diffusion coefficient D in the robot orientation in the evaluation data. θ Indicates the value of

[0176] The line L21 represents the diffusion coefficient D θ The line L22 shows the change in the index value of the error of the diffusion coefficient D θ The line L23 shows the change in the index value of the error of the diffusion coefficient D when the forgetting and relearning unit 195 performs forgetting and relearning 100 times each time the learning unit 193 performs learning. θ The change in the error index value is shown.

[0177] In the example of FIG. 6 , the diffusion coefficient D is smaller when the forgetting / relearning unit 195 performs forgetting and relearning once every time the learning unit 193 performs learning (see line L22) and when the forgetting / relearning unit 195 performs forgetting and relearning 100 times every time the learning unit 193 performs learning (see line L23) than when the forgetting / relearning unit 195 does not perform forgetting or relearning (see line L21). θ It can be said that the error is decreasing rapidly.

[0178] Figure 7 shows an example of the processing time when forgetting and relearning are performed collectively using multiple training data. The vertical axis of the graph in Figure 7 represents time. Bars B11 and B12 represent the time required when the relearning size is 1. The relearning size here refers to the number of training data used for each collectively performed forgetting and relearning.

[0179] Thus, a relearning size of 1 indicates a single lumped forgetting and relearning iteration using one training data. A relearning size of 10 indicates a single lumped forgetting and relearning iteration using ten training data. A relearning size of 100 indicates a single lumped forgetting and relearning iteration using 100 training data. Figure 7 shows the time required for 1000 lumped forgetting and relearning iterations.

[0180] The bar B11 indicates the time required when the forgetting and relearning processes are performed sequentially without being grouped together. Therefore, the bar B11 indicates the time required when the forgetting and relearning processes are repeated sequentially 1000 times, which is 0.49 seconds.

[0181] Bar B12 shows the time required for performing forgetting and relearning all at once. Since the relearning size is 1, bar B12 shows the time required for repeating forgetting and relearning 1,000 times using one training data, which was 0.49 seconds.

[0182] Bars B21 and B22 show the time required when the forgetting and relearning size is 10. Bar B21 shows the time required when forgetting and relearning are performed sequentially rather than all at once. Therefore, bar B21 shows the time required to repeat forgetting and relearning sequentially 10,000 times, which was 1.64 seconds.

[0183] Bar B22 shows the time required for performing forgetting and relearning all at once. Since the relearning size is 10, bar B22 shows the time required for repeating forgetting and relearning 10 times using 10 training data, which was 0.51 seconds.

[0184] Bars B31 and B32 show the time required when the relearning size is 10. Bar B31 shows the time required when forgetting and relearning are performed sequentially without batching. Therefore, bar B31 shows the time required to repeat forgetting and relearning sequentially 100,000 times, which was 12.69 seconds.

[0185] Bar B32 shows the time required for performing forgetting and relearning all at once. Since the relearning size is 100, bar B32 shows the time required for repeating forgetting and relearning 100 times using 100 training data, which was 0.59 seconds.

[0186] In this way, as exemplified by equations (32) to (39), by performing forgetting and relearning collectively using matrix operations, training results were obtained that showed that the increase in the required time could be suppressed even if the number of training data used in one forgetting and relearning session increased.

[0187] As described above, the data acquisition unit 192 acquires state information. The learning unit 193 uses training data including the state information to learn a target model. The storage unit 180 stores the state information. The forgetting / relearning unit 195 uses the training data stored in the storage unit 180 to forget and relearn the target model.

[0188] According to the learning device 100, in terms of forgetting and relearning the learning of a target model, when relearning a model that includes uncertainty using accumulated training data, it is expected that it will be possible to avoid underestimating the uncertainty of the model by using the same training data multiple times.

[0189] One possible method for learning a model is to perform online learning, but this takes time. For example, when using a model to control a learning target, it takes time for the model learning to progress to the point where the learning target can be made to perform a desired operation.

[0190] Another method of model training is to alternately learn bias parameters and variance parameters using accumulated training data, but this method requires waiting for the training data to be accumulated, and therefore cannot be used for online learning. If learning is performed after the accumulation of sequentially obtained training data, it also takes time for the model training to progress and for the target to perform the desired operation.

[0191] In contrast, in the learning device 100, the learning unit 193 learns the target model every time new data is obtained, so that the target model can be learned without waiting for the accumulation of training data.

[0192] In addition, since the forgetting / relearning unit 195 uses the training data stored in the memory unit 180 to learn (relearn) the target model, it is expected that the learning of the target model will progress more quickly compared to when only online learning is performed.

[0193] Furthermore, by having the forgetting / relearning unit 195 forget the learning of the target model, it is possible to forget the learning that uses uncertain parameter values ​​(bias parameter values ​​and variance parameter values) at a stage when the learning of the target model has not progressed. In this respect, the learning device 100 can avoid underestimating the uncertainty of the target model, and it is expected that the learning of the target model will progress more quickly than if the learning of the target model is not forgotten.

[0194] The target model also includes a deterministic partial model, which is a partial model that deterministically outputs a value in accordance with state information and a control command for the learning object 910, and a stochastic partial model, which is a partial model that probabilistically outputs a value in accordance with state information and a control command for the learning object 910. The forgetting / relearning unit 195 alternately performs forgetting and relearning of the deterministic partial model and forgetting and relearning of the stochastic partial model, using training data that includes data based on observations of the behavior of the learning object 910 and control commands for the learning object 910.

[0195] The learning device 100 can improve the accuracy of the target model by alternately retraining a deterministic partial model and a probabilistic partial model. In particular, the learning device 100 can alternately retrain a deterministic partial model and a probabilistic partial model, thereby allowing a probabilistic partial model that has progressed to a certain degree of learning to be used when retraining a deterministic partial model, and allowing a deterministic partial model that has progressed to a certain degree of learning to be used when retraining a stochastic partial model. In this respect, the learning device 100 is expected to enable efficient training of the target model.

[0196] Furthermore, the forgetting / relearning unit 195 forgets the learning of the target model using sampling data sampled from the training data stored in the storage unit 180, and then re-learns the target model using the sampling data. The learning device 100 can forget previous learning (learning by the learning unit 193 or relearning by the forgetting / relearning unit 195) using the data used for re-learning. In this respect, the learning device 100 is expected to be able to avoid underestimating the uncertainty of the target model by repeating learning using the same data.

[0197] Furthermore, the forgetting / relearning unit 195 uses matrix operations to collectively perform forgetting and relearning of the target model using multiple pieces of sampling data sampled from the training data stored in the storage unit 180. This is expected to prevent the learning device 100 from increasing in time even if the number of pieces of training data used in one forgetting and relearning operation increases.

[0198] Second Embodiment Fig. 8 is a diagram illustrating an example of the configuration of a learning device according to at least one embodiment. In the configuration illustrated in Fig. 8, a learning device 610 includes a data acquisition unit 611, a learning unit 612, a storage unit 613, and a forgetting / relearning unit 614.

[0199] With this configuration, the data acquisition unit 611 acquires data based on observations of the behavior of the learning target. The learning unit 612 uses the data based on observations of the behavior of the learning target to learn a model of the learning target. The memory unit 613 stores the data based on observations of the behavior of the learning target. The forgetting / relearning unit 614 uses the data stored in the memory unit 613 to forget and relearn the learning of the model of the learning target.

[0200] The data acquisition unit 611 corresponds to an example of data acquisition means. The learning unit 612 corresponds to an example of learning means. The storage unit 613 corresponds to an example of storage means. The forgetting / relearning unit 614 corresponds to an example of forgetting / relearning means.

[0201] The learning device 610 is expected to be able to forget and re-learn model learning, and when re-learning a model that includes uncertainty using accumulated data, to avoid underestimating the uncertainty of the model by using the same data multiple times.

[0202] One possible method for learning a model is to perform online learning, but this takes time. For example, when using a model to control a learning target, it takes time for the model learning to progress to the point where the learning target can be made to perform a desired operation.

[0203] Another method of model learning is to use accumulated training data to learn the model, but in this case, it is necessary to wait for the training data to be accumulated, and it cannot be used for online learning. If learning is performed after waiting for the accumulation of training data that is obtained sequentially, it also takes time for the model learning to progress and for the learning target to be able to perform the desired operation.

[0204] In contrast to this, in the learning device 610, the learning unit 612 learns the model every time new data is obtained, so that the model can be learned without waiting for the accumulation of training data.

[0205] Furthermore, since the forgetting / relearning unit 614 uses the training data stored in the memory unit 613 to learn (relearn) the model, it is expected that model learning will progress more quickly compared to when only online learning is performed.

[0206] Furthermore, by performing forgetting of model learning by the forgetting / relearning unit 614, it is possible to forget learning that uses uncertain parameter values ​​at a stage when model learning has not progressed. In this respect, the learning device 610 can avoid underestimating the uncertainty of the model, and it is expected that model learning will progress more quickly than if forgetting of model learning is not performed.

[0207] Third Embodiment Fig. 9 is a diagram showing an example of a processing procedure in a learning method according to at least one embodiment. The learning method shown in Fig. 9 includes acquiring data (step S611), learning (step S612), storing (step S613), and forgetting and relearning (step S614).

[0208] In acquiring data (step S611), the computer acquires data based on observations of the motion of the learning target. In learning (step S612), the computer uses the data based on observations of the motion of the learning target to learn a model of the learning target. In storing (step S613), the computer stores the data based on observations of the motion of the learning target. In forgetting and relearning (step S614), the computer uses the stored data to forget and relearn the model of the learning target.

[0209] According to the learning method shown in FIG. 9 , in that model learning is forgotten and then re-learned, when re-learning a model that includes uncertainty using accumulated data, it is expected that it will be possible to avoid underestimating the uncertainty of the model by using the same data multiple times.

[0210] One possible method for learning a model is to perform online learning, but this takes time. For example, when using a model to control a learning target, it takes time for the model learning to progress to the point where the learning target can be made to perform a desired operation.

[0211] Another method of model learning is to use accumulated training data to learn the model, but in this case, it is necessary to wait for the training data to be accumulated, and it cannot be used for online learning. If learning is performed after waiting for the accumulation of training data that is obtained sequentially, it also takes time for the model learning to progress and for the learning target to be able to perform the desired operation.

[0212] In contrast to this, in the learning method shown in FIG. 9, model learning is performed every time new data is obtained, so that model learning can be performed without waiting for the accumulation of training data.

[0213] Furthermore, in the learning method shown in FIG. 9, the model is learned (relearned) using stored training data, which is expected to result in faster model learning than when only online learning is performed.

[0214] Furthermore, by performing forgetting of model learning by the forgetting / relearning unit 614, it is possible to forget learning that uses uncertain parameter values ​​at a stage when model learning has not progressed. In this respect, the learning method shown in Fig. 9 can avoid underestimating the uncertainty of the model, and it is expected that model learning will progress more quickly than when forgetting of model learning is not performed.

[0215] 10 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. In the configuration shown in FIG. 10, a computer 700 includes a CPU (Central Processing Unit) 710, a main memory device 720, an auxiliary memory device 730, and an interface 740.

[0216] One or more of the learning devices 100 and 610, or a portion thereof, may be implemented in a computer 700. In this case, the operation of each of the above-described processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-described processing in accordance with the program. The CPU 710 also allocates storage areas in the main storage device 720 corresponding to each of the above-described storage units in accordance with the program. Communication between each device and other devices is performed by an interface 740 having a communication function and communicating under the control of the CPU 710.

[0217] When learning device 100 is implemented in computer 700, the operations of processing unit 190 and each of its units are stored in the form of a program in auxiliary storage device 730. CPU 710 reads the program from auxiliary storage device 730, loads it into main storage device 720, and executes the above-described processing in accordance with the program.

[0218] Furthermore, the CPU 710 allocates a storage area for the storage unit 180 in the main storage device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and performing communication under the control of the CPU 710. Display of images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and receiving user operations under the control of the CPU 710.

[0219] When the learning device 610 is implemented in the computer 700, the operations of the data acquisition unit 611, learning unit 612, and forgetting / relearning unit 614, and the operations of writing and reading data to and from the memory unit 613 are stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, loads it into the main storage device 720, and executes the above-mentioned processing in accordance with the program.

[0220] Furthermore, CPU 710 allocates a storage area for storage unit 613 in main memory device 720 in accordance with the program. Communication between learning device 610 and other devices is achieved by interface 740 having a communication function and performing communication under the control of CPU 710. Interaction between learning device 610 and a user is achieved by interface 740 having a display device and an input device, displaying various images under the control of CPU 710, and accepting user operations.

[0221] Alternatively, a program for executing all or part of the processing performed by the learning devices 100 and 610 may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform the processing of each unit. The term "computer system" as used herein includes hardware such as an operating system (OS) and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, read-only memories (ROMs), and compact disc read-only memories (CD-ROMs), as well as storage devices such as hard disks built into computer systems. The program may be designed to implement part of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already stored in the computer system.

[0222] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0223] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0224] (Supplementary Note 1) A learning device comprising: a data acquisition means for acquiring data based on observations regarding the behavior of a learning object; a learning means for learning a model of the learning object using the data based on the observations regarding the behavior of the learning object; a storage means for storing the data based on the observations regarding the behavior of the learning object; and a forgetting / relearning means for forgetting and relearning the learning of the model of the learning object using the data stored in the storage means.

[0225] (Supplementary Note 2) The model of the learning object includes a deterministic partial model which is a partial model that deterministically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and a stochastic partial model which is a partial model that probabilistically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and the forgetting / relearning means alternately forgets and re-learns the deterministic partial model and forgets and re-learns the stochastic partial model using training data including data based on observations of the behavior of the learning object and a control command for the learning object. (Supplementary Note 2) The learning device according to Supplementary Note 1,

[0226] (Supplementary Note 3) The learning device according to Supplementary Note 1 or Supplementary Note 2, wherein the forgetting / relearning means performs forgetting of the learning of the model to be learned using sampling data sampled from the data stored in the storage means, and performs relearning of the model to be learned using the sampling data.

[0227] (Supplementary Note 4) The learning device according to any one of Supplementary Notes 1 to 3, wherein the forgetting / relearning means performs forgetting and relearning of the learning of the model to be learned collectively using a plurality of sampling data sampled from the data stored in the storage means by using a matrix operation.

[0228] (Supplementary Note 5) A learning method comprising: a computer acquiring data based on observations of the behavior of a learning object; learning a model of the learning object using the data based on observations of the behavior of the learning object; storing the data based on observations of the behavior of the learning object; and forgetting and relearning the learning of the model of the learning object using the stored data.

[0229] (Supplementary Note 6) The learning method according to Supplementary Note 5, wherein the model of the learning object includes a deterministic partial model which is a partial model that deterministically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and a stochastic partial model which is a partial model that probabilistically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and wherein forgetting and relearning the model of the learning object involves the computer alternately forgetting and relearning the deterministic partial model and forgetting and relearning the stochastic partial model using training data including data based on observations of the behavior of the learning object and a control command for the learning object.

[0230] (Supplementary Note 7) The learning method according to Supplementary Note 5 or Supplementary Note 6, wherein in forgetting and relearning the learning of the model to be learned, the computer uses sampling data sampled from the data stored in the storage means to forget the learning of the model to be learned, and uses the sampling data to relearn the model to be learned.

[0231] (Supplementary Note 8) The learning device according to any one of Supplementary Notes 5 to 7, wherein in forgetting and relearning the learning of the model to be learned, the computer uses a plurality of sampling data sampled from stored data to forget and relearn the learning of the model to be learned all at once by using matrix operations.

[0232] (Supplementary Note 9) A recording medium having recorded thereon a program that causes a computer to perform the following operations: acquiring data based on observations of the behavior of a learning object; learning a model of the learning object using the data based on observations of the behavior of the learning object; storing the data based on observations of the behavior of the learning object; and forgetting and relearning the learning of the model of the learning object using the stored data.

[0233] (Supplementary Note 10) The recording medium according to Supplementary Note 9, wherein the model of the learning object includes a deterministic partial model which is a partial model that deterministically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and a stochastic partial model which is a partial model that probabilistically outputs a value in response to data based on observations of the behavior of the learning object and a control command for the learning object, and in forgetting and relearning the model of the learning object, the program causes the computer to alternately forget and relearn the deterministic partial model and the stochastic partial model using training data including data based on observations of the behavior of the learning object and a control command for the learning object.

[0234] (Supplementary Note 11) The recording medium described in Supplementary Note 9 or Supplementary Note 10, wherein in performing the unlearning and relearning of the model to be learned, the program causes the computer to perform the unlearning of the model to be learned using sampling data sampled from the data stored in the storage means, and to perform the relearning of the model to be learned using the sampling data.

[0235] (Supplementary Note 12) The recording medium according to any one of Supplementary Notes 9 to 11, wherein, in performing the forgetting and relearning of the model to be learned, the program causes the computer to execute the forgetting and relearning of the model to be learned collectively using a plurality of sampling data sampled from stored data by using matrix operations.

[0236] The present invention may be applied to a learning device, a learning method, and a recording medium.

[0237] 1 Learning system 100, 610 Learning device 110 Communication unit 120 Display unit 130 Operation input unit 180, 613 Storage unit 190 Processing unit 191 Object control unit 192, 611 Data acquisition unit 193, 612 Learning unit 194 Sampling unit 195, 614 Forgetting and relearning unit 910 Learning object 920 Environment

Claims

1. A data acquisition means for acquiring data based on observations of the behavior of the target to be learned, A learning means for training a model of the target to be trained using data based on observations of the target operation, A storage means for storing data based on observations of the behavior of the object to be learned, A forgetting and relearning means that uses the data stored in the memory means to forget and retrain the model to be learned, A learning device equipped with the following features.

2. The model to be learned includes a deterministic submodel which deterministically outputs values ​​in response to data based on observations of the operation of the model to be learned and control commands to the model to be learned, and a probabilistic submodel which probabilistically outputs values ​​in response to data based on observations of the operation of the model to be learned and control commands to the model to be learned. The forgetting and relearning means alternately forgets and relearns the learning of the deterministic submodel and forgets and relearns the learning of the probabilistic submodel, using training data that includes data based on observations of the operation of the learning target and control commands for the learning target. The learning device according to claim 1.

3. The forgetting and relearning means forgets the learning of the model to be learned using sampled data sampled from the data stored by the memory means, and then retrains the model to be learned using the sampled data. A learning device according to claim 1 or claim 2.

4. The forgetting and relearning means uses multiple sampled data points sampled from the data stored by the memory means to perform forgetting and relearning of the model to be learned in a unified manner using matrix operations. A learning device according to claim 1 or claim 2.

5. Computers We obtain data based on observations of the behavior of the target of learning, Using data based on observations of the target behavior, the model of the target behavior is trained. The system stores data based on observations regarding the behavior of the target being learned, Using the stored data, the model to be trained is forgotten and retrained. A learning method that includes this.

6. On the computer, Obtaining data based on observations of the behavior of the target of learning, The process involves training the model of the target of learning using data based on observations of the target of learning behavior, The process involves storing data based on observations of the behavior of the subject being learned, Using the stored data, the model to be trained is to be forgotten and retrained. A program that executes the command.