Machine learning device, prediction control device, machine learning method, prediction control method, and computer-readable recording medium

The machine learning device addresses convergence issues in regression problems by separating gradient calculations for variance and mean parameters, thereby stabilizing learning and enhancing accuracy.

WO2024241396A1PCT designated stage expired Publication Date: 2024-11-28NEC CORP

Patent Information

Application Number
PCT/JP2023/018817
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-19
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional machine learning devices face challenges with slow convergence and instability when solving regression problems involving heteroscedasticity, primarily due to the fluctuating noise term in the Gaussian distribution variance function model affecting the gradient values of the mean function model parameters.

Method used

A machine learning device that calculates and updates gradient values for both the variance and mean parameters using separate gradient value calculation units, minimizing the influence of the variance model's output on the mean model's parameters, thereby stabilizing the learning process and improving convergence.

Benefits of technology

The solution enhances the convergence of machine learning by stabilizing the gradient values of the mean function model parameters, leading to improved accuracy and efficiency in regression problem solving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023018817_28112024_PF_FP_ABST
    Figure JP2023018817_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A machine learning device 10 comprises: a first gradient value calculation unit 11 that calculates a gradient value for a variance parameter of a mean negative log likelihood, which is calculated using a plurality of training data containing explanatory and objective variables, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; a second gradient value calculation unit 12 that calculates a gradient value for a mean parameter of the mean squared error between the output value of the Gaussian distribution mean function model and the objective variable when the explanatory variable and the mean parameter are treated as input, which is calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; and a parameter update unit 13 that updates the variance parameter using the gradient value for the variance parameter of the mean negative log likelihood and updates the mean parameter using the gradient value for the mean parameter of the mean squared error.
Need to check novelty before this filing date? Find Prior Art

Description

Machine learning device, prediction control device, machine learning method, prediction control method, and computer-readable recording medium

[0001] The present disclosure relates to a machine learning device and a machine learning method for machine learning a model for solving a regression problem, a predictive control device and a predictive control method using this model, and a computer-readable recording medium on which a program for realizing these is recorded.

[0002] In recent years, attempts have been made to solve regression problems involving heteroscedasticity using machine learning. In these attempts, machine learning is performed using a model in which the mean and variance parameters of a Gaussian distribution depend on the input. For example, Non-Patent Document 1 discloses a machine learning device that performs such machine learning.

[0003] A conventional machine learning device disclosed in Non-Patent Document 1 will be specifically described with reference to FIG. 11 . FIG. 11 shows an example of a conventional machine learning device. The machine learning device 200 shown in FIG. 11 performs machine learning to generate a model for solving a regression problem. As shown in FIG. 11 , the machine learning device 200 includes a batch data generation unit 210, a gradient value calculation unit 220, a parameter update unit 230, and a storage unit 240.

[0004] The machine learning device 200 shown in Fig. 11 operates as shown in Fig. 12. Fig. 12 is a flow diagram showing the operation of the machine learning device shown in Fig. 11.

[0005] First, a user inputs parameters of a Gaussian distribution mean function model (hereinafter referred to as "mean parameters"), parameters of a Gaussian distribution variance function model (hereinafter referred to as "variance parameters"), and supervised training data via an input device. At this time, each input parameter is an initial parameter.

[0006] As a result, as shown in FIG. 2, the machine learning device 200 accepts the input data, and the memory unit 240 stores the input mean parameters, the input variance parameters, and the input supervised training data (step C1).

[0007] Next, the batch data generation unit 210 creates batch data from the supervised training data recorded in the storage unit 240, and records the created batch data in the storage unit 240 (step C2).

[0008] Next, the gradient value calculation unit first calculates an average negative log-likelihood from the batch data, the mean parameter, and the variance parameter recorded in the storage unit 240. Then, the gradient value calculation unit calculates the gradient value of the mean parameter and the gradient value of the variance parameter using the calculated average negative log-likelihood (step C3).

[0009] The parameters used at this time are the initial input parameters the first time the loop processing described below is performed, and thereafter they are parameters updated by the parameter update unit 230 described below.

[0010] Next, the parameter update unit 230 updates the mean parameter and the variance parameter based on the calculated gradient values ​​(step C4), and further determines whether a user-specified termination condition is satisfied (step C5).

[0011] If the termination condition is satisfied in step C5, the parameter update unit 230 outputs the parameters at that time that are recorded in the storage unit 240 (step C6), which ends the processing in the machine learning device 200.

[0012] On the other hand, if the termination condition is not satisfied in step C5, the updated parameters are set as new initial parameters, and the loop process is executed again from the batch data creation step (step C2) described above.

[0013] Here, the regression problem involving heteroscedasticity will be described in detail below. The regression problem involving heteroscedasticity can be formulated by the following Equation 1.

[0014]

[0015] In the above equation (1), Normal represents a Gaussian distribution, x represents an explanatory variable, y represents a target variable generated from a Gaussian distribution conditioned on x, μ(x, θ) represents a Gaussian distribution mean function model (true mean function) that depends on x, and ν(x, Φ) represents a Gaussian distribution variance function model (true variance function) that depends on x. The machine learning device 200 shown in FIG. 11 performs machine learning using the Gaussian distribution mean function model μ(x, θ) and the Gaussian distribution variance function model ν(x, Φ) with the goal of bringing each model closer to the true mean function and true variance function. Here, the loss function used for this regression problem is the negative log likelihood (hereinafter also referred to as "NLL"), expressed by the following equation (2), in accordance with the concept of maximum likelihood estimation:

[0016]

[0017] In the above equation 2, θ is a parameter of the Gaussian distribution mean function model (hereinafter referred to as the “mean parameter”), Φ is a parameter of the Gaussian distribution variance function model (hereinafter referred to as the “variance parameter”), μ(x, θ) is a Gaussian distribution mean function model expressed by the mean parameter θ, and ν(x, Φ) is a Gaussian distribution variance function model expressed by the variance parameter Φ.

[0018] 11 calculates the gradient value of the mean parameter θ and the gradient value of the variance parameter Φ in the average negative log-likelihood for the batch data, and updates these parameters using an update method such as a stochastic gradient method. This series of updates can be generally summarized as follows:

[0019]

[0020]

[0021] In the above formulas 3 and 4, the subscript t represents the number of updates. t=0 is the first update. In addition, in the above formulas 3 and 4, η t is the learning rate at update number t, N Btrepresents the sample size of the batch data created at update count t. Furthermore, the second term i in the above equations 3 and 4 is the sample size of the batch data B t represents the index of the training data contained in the second term. The quantity in the square brackets in the second term corresponds to the average negative log-likelihood.

[0022] Kurtland Chua, et al. “Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models” NeurIPS 2018

[0023] However, the above-described machine learning device 200 has problems in that learning takes a long time and learning is unstable. The reasons for this will be explained below.

[0024] First, in the loss function (NLL formula) shown in the above equation 2, the Gaussian distribution variance function model ν(x, Φ) is a noise term and therefore fluctuates significantly during learning. Then, in the loss function (NLL formula) shown in the above equation 2, there is a term that includes both the Gaussian distribution mean function model μ(x, θ) and the Gaussian distribution variance function model ν(x, Φ), that is, "(y-μ(x, θ)) 2 In the term " / ν(x, Φ)", the Gaussian distribution variance function model ν(x, Φ) is located in the denominator. Therefore, fluctuations in the Gaussian distribution variance function model ν(x, Φ) slow down the convergence of the mean parameter θ, which is a model parameter of the Gaussian distribution mean function model μ(x, θ), and also significantly affect the gradient value corresponding to the mean parameter θ. As a result, the above-mentioned problem occurs in the above-mentioned machine learning device.

[0025] An example of the objective of the present disclosure is to solve the above problem and reduce the influence of the output value of the Gaussian distribution variance function model on the gradient value for the parameters of the Gaussian distribution mean function model, thereby improving convergence in machine learning.

[0026] In order to achieve the above object, a machine learning device according to one aspect of the present disclosure comprises: a first gradient value calculation means that calculates a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a response variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; a second gradient value calculation means that calculates a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the response variable when the explanatory variables and the mean parameters are input, the mean squared error being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; and a parameter update unit that updates the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and updates the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0027] Furthermore, to achieve the above object, a prediction control device according to one aspect of the present disclosure includes: a signal acquisition unit that acquires a signal indicating a state of a controlled object; and a prediction control unit that inputs the level of the acquired signal as an explanatory variable to a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model, and generates an input signal to be input to the controlled object based on an output from the machine learning model that serves as a dependent variable, wherein the machine learning model calculates a gradient value for the variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variables and the dependent variable, and a variance parameter of the Gaussian distribution variance function model that depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model that depends on the explanatory variables; and calculates a gradient value for the mean parameter of a mean square error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are input, calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model that depends on the explanatory variables. The Gaussian distribution variance function model is constructed by updating a variance parameter of the Gaussian distribution variance function model using a gradient value for the variance parameter of the average negative log-likelihood, and updating a mean parameter of the Gaussian distribution mean function model using a gradient value for the mean parameter of the mean squared error.

[0028] In order to achieve the above object, a machine learning method according to one aspect of the present disclosure comprises: calculating a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a response variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculating a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the response variable when the explanatory variables and mean parameters are input, the mean squared error being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; updating the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood; and updating the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0029] Furthermore, to achieve the above object, a predictive control method according to one aspect of the present disclosure includes: acquiring a signal indicating a state of a controlled object; inputting the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model; and generating an input signal to be input to the controlled object based on an output from the machine learning model, the output being a dependent variable; the machine learning model calculates a gradient value for the variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variables and the dependent variable, a variance parameter of the Gaussian distribution variance function model that depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model that depends on the explanatory variables; and calculating a gradient value for the mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are input, the gradient value being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model that depends on the explanatory variables; The Gaussian distribution variance function model is constructed by updating a variance parameter of the Gaussian distribution variance function model using a gradient value for the variance parameter of the average negative log-likelihood, and updating a mean parameter of the Gaussian distribution mean function model using a gradient value for the mean parameter of the mean squared error.

[0030] To achieve the above object, a first computer-readable recording medium according to one aspect of the present disclosure is characterized in that it records a program including the following instructions on a computer: calculate a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a response variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculate a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the response variable when the explanatory variables and mean parameters are input, calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; update the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and update the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0031] Furthermore, to achieve the above object, a second computer-readable recording medium according to one aspect of the present disclosure provides a computer-readable recording medium having recorded thereon a program including instructions to: cause a computer to acquire a signal indicating a state of a controlled object; input the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model; and generate an input signal to be input to the controlled object based on the output from the machine learning model, which serves as a dependent variable; wherein the machine learning model calculates a gradient value for the variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variables and the dependent variable, a variance parameter of the Gaussian distribution variance function model that depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model that depends on the explanatory variables; and calculates a gradient value for the mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are used as input, calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model that depends on the explanatory variables; The Gaussian distribution variance function model is constructed by updating a variance parameter of the Gaussian distribution variance function model using a gradient value for the variance parameter of the average negative log-likelihood, and updating a mean parameter of the Gaussian distribution mean function model using a gradient value for the mean parameter of the mean squared error.

[0032] As described above, according to the present disclosure, it is possible to reduce the influence of the output value of the Gaussian distribution variance function model on the gradient value for the parameters of the Gaussian distribution mean function model, thereby improving the convergence in machine learning.

[0033] FIG. 1 is a configuration diagram showing a schematic configuration of a machine learning device according to the first embodiment. FIG. 2 is a configuration diagram specifically showing the configuration of a machine learning device 10 according to the first embodiment. FIG. 3 is a flow diagram showing the operation of the machine learning device according to the first embodiment. FIG. 4 is a configuration diagram showing a schematic configuration of a prediction control device according to the second embodiment. FIG. 5 is a configuration diagram specifically showing the configuration of a prediction control device 30 according to the second embodiment. FIG. 6 is a flow diagram showing the operation of the prediction control device according to the second embodiment. FIG. 7 is a block diagram showing an example of a computer that realizes the machine learning device according to the first embodiment and the prediction control device according to the second embodiment. FIG. 8 is a diagram showing an example of a hydraulic excavator used in the examples. FIG. 9 is a diagram showing an example of a reference trajectory used in the examples. FIG. 10 is a diagram explaining the effects of the examples. FIG. 11 shows an example of a conventional machine learning device. FIG. 12 is a flow diagram showing the operation of the machine learning device shown in FIG. 11.

[0034] First Embodiment First, in a first embodiment, a machine learning device, a machine learning method, and a program will be described with reference to FIGS. 1 to 3. FIG.

[0035] [Device Configuration] First, a schematic configuration of the machine learning device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of the machine learning device according to the first embodiment.

[0036] 1, a machine learning device 10 according to a first embodiment is a device that executes machine learning of a model for solving a regression problem. As shown in FIG. 1, the machine learning device 10 includes a first gradient value calculation unit 11, a second gradient value calculation unit 12, and a parameter update unit 13.

[0037] The first gradient value calculation unit 11 calculates a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data sets including explanatory variables and a target variable, the variance parameter, and a mean parameter. Here, the variance parameter is a parameter of a Gaussian distribution variance function model. The mean parameter is a parameter of a Gaussian distribution mean function model that depends on the explanatory variables.

[0038] The second gradient value calculation unit 12 calculates a gradient value for the mean parameter of the mean square error between the output value of the Gaussian distribution mean function model and the objective variable when the explanatory variables and the mean parameter are input, the gradient value being calculated using a plurality of training data and the mean parameter.

[0039] The parameter update unit 13 updates the variance parameter Φ using a gradient value for the variance parameter Φ of the average negative log-likelihood.The parameter update unit 13 also updates the mean parameter θ using a gradient value for the mean squared error.

[0040] In this way, in the machine learning device 10, the gradient value for updating the mean parameter θ is calculated from the mean square error between the output value μ(x, θ) of the Gaussian distribution mean function model and the objective variable y. In other words, the influence of the output value of the Gaussian distribution variance function model v(x, Φ) on the gradient value for the mean parameter θ of the Gaussian distribution mean function model is suppressed. As a result, the convergence in machine learning is improved.

[0041] Next, the configuration and functions of the machine learning device 10 according to the first embodiment will be specifically described with reference to Fig. 2. Fig. 2 is a block diagram specifically showing the configuration of the machine learning device 10 according to the first embodiment.

[0042] 2 , in the first embodiment, the machine learning device 10 includes, in addition to the first gradient value calculation unit 11, the second gradient value calculation unit 12, and the parameter update unit 13, a data acquisition unit 14, a batch data generation unit 15, and a storage unit 16. The machine learning device 10 is also connected to a user's terminal device 20 via a network or the like so as to be able to communicate data therewith.

[0043] In the first embodiment, a user transmits a large amount of training data to the machine learning device 10 via the terminal device 20. The transmitted training data includes data that serve as explanatory variables and teacher data that serve as target variables. In the first embodiment, the user can also transmit initial parameters for the mean parameter θ and the variance parameter Φ to the machine learning device 10 via the terminal device 20.

[0044] When the training data and initial parameters are transmitted from the terminal device 20 , the data acquisition unit 14 acquires them and stores the acquired training data and initial parameters in the storage unit 16 .

[0045] The batch data generation unit 15 extracts some of the training data from the large amount of training data stored in the storage unit 16, and generates the extracted training data as batch data. The batch data may be stored in the storage unit 16 again.

[0046] In the first embodiment, the first gradient value calculation unit 11 first acquires the batch data generated by the batch data generation unit 15, the initial parameters of the mean parameter θ stored in the storage unit 16, and the initial parameters of the variance parameter Φ also stored in the storage unit 16. Then, the first gradient value calculation unit 11 calculates a gradient value for the variance parameter of the mean negative log-likelihood calculated from these.

[0047] Here, as in the Background Art section, the explanatory variable is x, the objective variable is y, the Gaussian distribution variance function model is v(x, Φ), and the Gaussian distribution mean function model is μ(x, θ). Also, the variance parameter is Φ, and the mean parameter is θ. In this case, the negative logarithmic likelihood is, for example, "0.5*[log2πν(x, Φ)+(y-μ(x, θ)) 2 / ν(x, Φ)].

[0048] Therefore, the first gradient value calculation unit 11 calculates the negative log-likelihood “0.5*[log2πν(x,Φ)+(y−μ(x,θ)) 2 / ν(x, Φ)]" and then calculates an average negative log-likelihood using each calculated negative log-likelihood. The first gradient value calculation unit 11 then calculates a gradient value of the calculated average negative log-likelihood for the acquired dispersion parameter Φ.

[0049] In the first embodiment, the second gradient value calculation unit 12 first acquires the batch data generated by the batch data generation unit 15 and the initial parameter of the mean parameter θ stored in the storage unit 16. Then, the second gradient value calculation unit 12 calculates a gradient value for the mean parameter of the mean square error calculated from these.

[0050] Here, the squared error is the squared error between the output value μ(x, θ) of the Gaussian distribution mean function model when the explanatory variables and mean parameters are input, and the objective variable y, and is expressed as "(y - μ(x, θ))^2".

[0051] Therefore, the second gradient value calculation unit 12 calculates the square error (y−μ(x, θ)) for each training data constituting the batch data. 2 and then calculate each calculated square error (y-μ(x, θ)) 2 Furthermore, the second gradient value calculation unit 12 calculates the gradient value of the mean square error for the acquired average parameter θ.

[0052] If the parameter update unit 13 has not yet performed an update, the initial parameters are acquired as the mean parameter θ and the variance parameter Φ. On the other hand, if the parameter update unit 13 has already performed an update, the updated mean parameter θ and variance parameter Φ are acquired.

[0053] In the first embodiment, the parameter updater 13 updates the variance parameter Φ and the mean parameter θ so as to minimize the loss function that calculates the sum of the above-mentioned average negative log-likelihood and the above-mentioned mean squared error.

[0054] An example of the loss function is the function shown in the following equation 5. In the following equation 5, the second argument θ is a constant set based on the average parameter θ. Specifically, θ is set to the same value as the first argument θ, but is treated as a constant when updating the average parameter θ (when calculating the gradient).

[0055]

[0056] Specifically, the parameter update unit 13 applies the gradient value calculated by the first gradient value calculation unit 11 to the following equation (6) to perform the stochastic gradient descent method and optimize the variance parameter Φ so that the batch average of the loss function is minimized. In the following equation (6), the second term on the right side indicates the gradient value. Also, in equation (6), η t denotes the learning rate at update number t, and N Bt indicates the sample size of the batch data created at update count t. Bt indicates the batch data.

[0057]

[0058] Furthermore, the parameter update unit 13 applies the gradient value calculated by the second gradient value calculation unit 12 to the following equation (7) to perform stochastic gradient descent and optimize the average parameter θ so that the batch average of the loss function is minimized. In the following equation (7), the second term on the right side indicates the gradient value. Similarly, in equation (7), η t denotes the learning rate at update number t, and N Bt denotes the sample size of batch data created at update count t, and Bt denotes batch data.

[0059]

[0060] Thus, in the function of Equation 5 above, essentially, the loss function for the mean parameter θ is trained using the mean double error, and the loss function for the variance parameter is trained using the negative log-likelihood, and the two are trained independently. As a result, the influence of the output value of the Gaussian distribution variance function model ν(x, Φ) on the gradient value for the mean parameter θ of the Gaussian distribution mean function model is suppressed. This improves convergence in machine learning.

[0061] Thereafter, the parameter update unit 13 stores the updated variance parameter Φ and the updated mean parameter θ in the storage unit 16. The variance parameter Φ and the mean parameter θ stored in the storage unit 16 are used as parameters of a machine learning model used in a prediction control device, which will be described later.

[0062] [Device Operation] Next, the operation of the machine learning device 10 in embodiment 1 will be described using FIG. 3. FIG. 3 is a flow diagram showing the operation of the machine learning device in embodiment 1. In the following description, FIGS. 1 and 2 will be referenced as appropriate. In addition, in embodiment 1, a machine learning method is implemented by operating the machine learning device 10. Therefore, the description of the machine learning method in embodiment 1 will be replaced by the following description of the operation of the machine learning device 10.

[0063] First, a user transmits any supervised training data, any initial parameters for a mean parameter θ, and any initial parameters for a variance parameter to the machine learning device 10 via the terminal device 20. As a result, as shown in Fig. 3 , the data acquisition unit 14 acquires the training data and the initial parameters transmitted from the terminal device 20 and stores them in the storage unit 16 (step S1).

[0064] Next, the batch data generation unit 15 extracts some of the training data from the large amount of training data stored in the storage unit 16, and generates the extracted training data as batch data (step S2). The batch data generation unit 15 also stores the created batch data in the storage unit 16.

[0065] Next, the first gradient value calculation unit 11 acquires the batch data generated in step S2 and the initial parameters of the mean parameter θ and the arbitrary initial parameters of the variance parameter Φ stored in the storage unit 16 in step S1. Then, using these, the first gradient value calculation unit 11 calculates a gradient value for the variance parameter Φ of the average negative log-likelihood, which is calculated using the batch data, the mean parameter θ, and the variance parameter Φ (step S3).

[0066] Specifically, in step S3, the first gradient value calculation unit 11 calculates the negative log-likelihood “0.5*[log2πν(x,Φ)+(y−μ(x,θ)) 2 / ν(x, Φ)]" and then calculates an average negative log-likelihood using each calculated negative log-likelihood. The first gradient value calculation unit 11 then calculates a gradient value of the calculated average negative log-likelihood for the acquired dispersion parameter Φ.

[0067] If the parameters have already been updated in step S5 (to be described later), the first gradient value calculation unit 11 acquires the latest updated variance parameter Φ and the latest mean parameter θ instead of the initial parameters, and executes step S3.

[0068] Next, the second gradient value calculation unit 12 acquires the batch data generated in step S2 and the initial parameters of the mean parameter θ stored in the storage unit 16 in step S1. Then, using these, the second gradient value calculation unit 12 calculates a gradient value for the mean parameter θ of the mean squared error, which is calculated using the batch data and the mean parameter θ (step S4).

[0069] Specifically, in step S4, the second gradient value calculation unit 12 calculates the square error (y−μ(x, θ)) for each training data constituting the batch data. 2 and then calculate each calculated square error (y-μ(x, θ)) 2 Furthermore, the second gradient value calculation unit 12 calculates the gradient value of the mean square error for the acquired average parameter θ.

[0070] If the parameter update in step S5 described later has already been performed, the second gradient value calculation unit 12 also acquires the latest updated average parameter θ instead of the initial parameter, and executes step S4.

[0071] The parameter update unit 13 updates the variance parameter Φ using the gradient value calculated in step S2 so as to minimize the loss function (see above equation 5) that calculates the sum of the average negative log-likelihood and the mean squared error, and further updates the mean parameter θ using the gradient value calculated in step S3 (step S5). The parameter update unit 13 also stores the updated variance parameter Φ and the updated parameter θ in the storage unit 16. Furthermore, if the updated variance parameter Φ and the updated parameter θ are already stored, the parameter update unit 13 overwrites them with the newly updated variance parameter Φ and parameter θ.

[0072] Next, the parameter update unit 13 determines whether a termination condition specified in advance by the user is satisfied (step S6), such as whether the number of updates of the variance parameter Φ and the mean parameter θ reaches a predetermined number.

[0073] If the determination result in step S6 indicates that the termination condition is satisfied (S6: Yes), the processing in the machine learning device 10 ends. Thereafter, the variance parameter Φ and the mean parameter θ stored in the storage unit 16 are used as parameters of a machine learning model used in a prediction control device, which will be described later.

[0074] On the other hand, if the result of the determination in step S6 is that the termination condition is not satisfied (S6: No), each step is executed again from step S2. In this case, in step S2, the batch data generation unit 15 newly extracts multiple training data that have not yet been converted into batch data, and generates these as batch data.

[0075] [Effects of First Embodiment] As described above, in the first embodiment, the machine learning device 10 calculates the gradient value for the mean parameter θ of the Gaussian distribution mean function model using the mean squared error that does not depend on the variance parameter Φ of the Gaussian distribution variance function model, by the first gradient value calculation unit 11. Therefore, the gradient value for the mean parameter θ of the Gaussian distribution mean function model is stabilized because its scale does not depend on the output value of the Gaussian distribution variance function model, which fluctuates during the learning process, and as a result, the convergence of the mean parameter of the Gaussian distribution mean function model is improved.

[0076] Furthermore, when the output value of the Gaussian distribution mean function model always minimizes the mean square error, this output value also minimizes the average negative log-likelihood. Therefore, according to the first embodiment, a bias caused by changing the loss function from the negative log-likelihood is not created, and the gradient value for the mean parameter θ can be stabilized, resulting in improved convergence of the mean parameter.

[0077] [Program] The program in the first embodiment may be any program that causes a computer to execute steps S1 to S6 shown in Fig. 3. By installing and executing this program on a computer, the machine learning device 10 and machine learning method in the first embodiment can be realized. In this case, the processor of the computer functions as the first gradient value calculation unit 11, the second gradient value calculation unit 12, the parameter update unit 13, the data acquisition unit 14, and the batch data generation unit 15, and performs processing.

[0078] In the first embodiment, the storage unit 16 can be realized by a storage device such as a hard disk provided in the computer. The storage unit 16 may also be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0079] The program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the first gradient value calculation unit 11, the second gradient value calculation unit 12, the parameter update unit 13, the data acquisition unit 14, and the batch data generation unit 15.

[0080] Second Embodiment Next, a prediction control device, a prediction control method, and a program according to a second embodiment will be described with reference to FIGS. 4 and 5. FIG.

[0081] [Device Configuration] First, the schematic configuration of the prediction control device in the second embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing the schematic configuration of the prediction control device in the second embodiment.

[0082] 4, a prediction control device 30 according to the second embodiment is a device for performing predictive control on a control target. As shown in FIG. 4, the prediction control device 30 includes a signal acquisition unit 31, a prediction control unit 32, and a machine learning model 33.

[0083] Of these, the signal acquisition unit 31 acquires a signal indicating the state of the controlled object. The prediction control unit 32 inputs the level of the acquired signal as an explanatory variable to a machine learning model 33 having a Gaussian distribution variance function model and a Gaussian distribution mean function model. The prediction control unit 32 also generates an input signal to be input to the controlled object based on the output from the machine learning model 33, which serves as a target variable.

[0084] The machine learning model 33 is constructed according to the following procedure: First, a gradient value for the variance parameter of the average negative log-likelihood calculated using a plurality of training data sets including explanatory variables and a target variable, and the variance parameter of a Gaussian distribution variance function model and the mean parameter of a Gaussian distribution mean function model, which depend on the explanatory variables, is calculated.

[0085] Next, a gradient value for the mean parameter of the Gaussian distribution mean function model, which is calculated using a plurality of training data and the mean parameter of the Gaussian distribution mean function model that depends on the explanatory variables, is calculated for the mean square error between the output value of the Gaussian distribution mean function model and the objective variable when the explanatory variables and the mean parameter are input.

[0086] Then, the variance parameters of the Gaussian distribution variance function model are updated using the gradient value for the variance parameter of the average negative log-likelihood, and the mean parameters of the Gaussian distribution mean function model are updated using the gradient value for the mean parameter of the mean squared error. The machine learning model 33 is constructed using the updated variance parameters and the updated mean parameters.

[0087] Next, the configuration and functions of the prediction control device 30 in the second embodiment will be specifically described with reference to Fig. 5. Fig. 5 is a configuration diagram specifically showing the configuration of the prediction control device 30 in the second embodiment.

[0088] As shown in FIG. 5 , the predictive control device 30 , together with the controlled object 40 and the state observer 50 , constitutes a model predictive control system 100 .

[0089] The control target 40 is a device, a system, or the like that receives an input signal generated by the prediction control device 30 and operates based on the received input signal. In the second embodiment, the control target 40 may be any controllable device or system. Examples of the control target 40 include a robot, an automobile, a backhoe, a ship, a chemical plant system, a power plant system, and a trading system.

[0090] Furthermore, the control target 40 does not necessarily have to have the function of receiving an input signal and the function of executing processing based on the received input signal. In this case, an operator of the control target 40 controls the control target 40 in accordance with instructions from the input signal.

[0091] The state observer 50 observes the state of the control object 21 and generates a signal indicating the observed state. The state observer 50 then transmits the generated signal (hereinafter referred to as a "state signal") to the predictive control device 30. A specific example of the state observer 50 is an inertial measurement unit (IMU) equipped with an acceleration sensor or the like. In the example of FIG. 5 , the state observer 50 constitutes a part of the control object 21.

[0092] In the predictive control device 30, in the second embodiment, the signal acquisition unit 31 acquires a state signal from the state observation device 50 and inputs the acquired state signal to the predictive control unit 31. When the state observation device 50 is an inertial measurement unit, the signal acquisition unit 31 acquires, as the state signal, a signal indicating the angular velocity or acceleration during rotational movement of each part of the control object 40, a signal indicating the speed of the control object 40, a signal indicating the position of the control object 40, or the like.

[0093] In the second embodiment, the machine learning model 33 is constructed using the variance parameter Φ and the mean parameter θ updated by the machine learning device 10 in the first embodiment. Specifically, the machine learning model 33 is a neural network or the like constructed in a memory by the machine learning engine using the mean parameter θ and the variance parameter Φ updated in the first embodiment.

[0094] In the second embodiment, the prediction control unit 32 receives a state signal input from the signal acquisition unit 31, and inputs the received state signal to the machine learning model 33. Then, the prediction control unit 32 receives an output from the machine learning model 33, generates an input signal to be input to the control target 40 based on the received output, and transmits the generated input signal to the control target 40. In this way, the prediction control unit 32 executes model predictive control.

[0095] Furthermore, the algorithm for model predictive control executed by the prediction control unit 32 is not particularly limited as long as it corresponds to the machine learning model 33 used and the format of the objective function. For example, when a Gaussian distribution mean function model and a Gaussian distribution variance function model are expressed using a nonlinear model such as a multilayer perceptron, it is possible to use an algorithm for model predictive control such as ITMPC (Information Theoretic Model Predictive Control) or iLQG (Iterative Linear Quadratic Gaussian).

[0096] Furthermore, since the aleatoric uncertainty of state transitions can be evaluated using a Gaussian distribution variance function model, a mechanism for utilizing the evaluation of aleatoric uncertainty may be incorporated into the algorithm for model predictive control.

[0097] Specifically, in the case of a task where it is important to suppress aleatory uncertainty, an approach such as risk-averse model predictive control may be adopted. Also, in the model predictive control algorithm, the cost function may be designed so that the cost increases as the output value of the Gaussian distribution variance function model increases.

[0098] Furthermore, when ITMPC is used as an algorithm for model predictive control, state fluctuations corresponding to the output values ​​of the Gaussian distribution variance function model may be added when generating trajectory samples. In this case, it becomes possible to calculate a control input that maximizes the objective function while taking into account the magnitude of accidental uncertainty. Note that application examples of the Gaussian distribution variance function model to model predictive control are not limited to the above examples.

[0099] [Device Operation] Next, the operation of the predictive control device 30 in the second embodiment will be described with reference to FIG. 6. FIG. 6 is a flow diagram showing the operation of the predictive control device in the second embodiment. In the following description, reference will be made to FIGS. 4 and 5 as appropriate. In the second embodiment, a predictive control method is implemented by operating the predictive control device 30. Therefore, the description of the predictive control method in the second embodiment will be replaced by the following description of the operation of the predictive control device 30.

[0100] First, as a premise, the controlled object 40 operates, and the state observing device 50 observes the state of the controlled object 21 and outputs a state signal indicating the observed state. As a result, as shown in Fig. 6, the signal acquiring unit 31 acquires the state signal from the state observing device 50 and inputs the acquired state signal to the predictive control unit 31 (step S11).

[0101] Next, the prediction control unit 32 receives the state signal input from the signal acquisition unit 31 in step S11 and determines whether a preset termination condition is satisfied (step S12). The termination condition may be, for example, that the observed state has reached a target.

[0102] If the determination result in step S12 indicates that the termination condition is satisfied (S12: Yes), the processing in the prediction control device 30 ends. On the other hand, if the determination result in step S12 indicates that the termination condition is not satisfied (S12: No), step S13 is executed.

[0103] In step S13 , the prediction control unit 32 inputs the state signal input from the signal acquisition unit 31 in step S11 to the machine learning model 33 .

[0104] Next, the predictive control unit 32 receives the output from the machine learning model 33, generates an input signal to be input to the control object 40 based on the received output, and transmits the generated input signal to the control object 40 (step S14).

[0105] After step S14 is executed, the controlled object 40 receives the input signal transmitted from the predictive control device 30 and executes an operation according to the input signal. After step S14 is executed, step S11 is executed again.

[0106] [Effects of Second Embodiment] As described above, in the second embodiment, the machine learning model 33 is constructed using the parameters updated in the first embodiment, and predictive control is performed using this machine learning model 33. As described in the first embodiment, the convergence is improved in the machine learning of the machine learning model 33. Therefore, according to the second embodiment, highly accurate predictive control is possible.

[0107] [Program] The program in the second embodiment may be a program that causes a computer to execute steps S11 to S14 shown in FIG. 6. By installing and executing this program on a computer, the predictive control device 30 and predictive control method in the second embodiment can be realized. In this case, the processor of the computer functions as the signal acquisition unit 31 and the predictive control unit 32 and performs processing. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device. Furthermore, the computer may be a control computer installed in the control target 40.

[0108] The program in the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as either the signal acquisition unit 31 or the prediction control unit 32.

[0109] (Physical Configuration) Here, a computer that realizes the machine learning device 10 by executing the program in the first embodiment and a computer that realizes the prediction control device by executing the program in the second embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of a computer that realizes the machine learning device in the first embodiment and the prediction control device in the second embodiment.

[0110] 7, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0111] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0112] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0113] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0114] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0115] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0116] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0117] The machine learning device and the prediction control device can be realized not by a computer on which a program is installed, but by hardware corresponding to each part, such as an electronic circuit. Furthermore, the machine learning device and the prediction control device may be partially realized by a program and the remaining part by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 7 .

[0118] Next, Example 1, which specifically illustrates the machine learning device 10, and Example 2, which specifically illustrates the prediction control device 30, will be described below with reference to Figures 8 to 10. In the following description, the above-mentioned Figures 1 to 7 will be referred to as appropriate.

[0119] Furthermore, in the following first and second embodiments, the controlled object 40 is assumed to be a hydraulic excavator equipped with a remote control system. Fig. 8 is a diagram showing an example of a hydraulic excavator used in the embodiments.

[0120] The remote control system is connected to the prediction control device 30 via Wi-Fi communication and receives input signals from the prediction control device 30. Then, the remote control device remotely controls the operation lever of the hydraulic excavator, which is the controlled object 40, in accordance with the received input signals.

[0121] Specifically, the remote control system includes a communication device for Wi-Fi communication, a servo motor for operating the control lever, a servo motor controller, and a power supply device for supplying power to the servo motor.

[0122] In addition, in a hydraulic excavator, the movable range of the operating lever is limited to a range corresponding to the rotational movement around the bucket axis, arm axis, and boom axis of the excavator, as shown in FIG. bucket indicates the rotation angle around the bucket axis, and θ arm indicates the rotation angle around the arm axis, and θ boom indicates the rotation angle around the boom axis.

[0123] In the backhoe shown in Figure 8, the direction and angle of rotation of each axis are determined by the direction and angle of tilt of the control lever. Therefore, the direction and angle of tilt of the control lever are specified by the input signal. In this embodiment, for convenience, the input signal (control input) u at a discrete time t expressed by an arbitrary natural number is t is defined as the following equation 8.

[0124]

[0125] In the above equation 8, each component represents the direction and angle of tilt of the control lever corresponding to the rotational movement of the bucket shaft, arm shaft, and boom shaft, and is expressed as a numerical value between -1.0 and +1.0. Note that the positive or negative value represents the direction of rotational movement (the direction in which the control lever is tilted), and the absolute value of the value represents the degree of tilt (angle). Therefore, 0 corresponds to no tilt, and 1 corresponds to maximum tilt. The control cycle is 80 milliseconds.

[0126] The state observer 50 observes the state of the control object 40 and transmits a state signal indicating the observed state to the predictive control device 40. In this embodiment, the state observer 50 is an inertial measurement unit (IMU) provided in the control object 40. The IMU measures the joint angles of the excavator, i.e., the three angles θ shown in FIG. 8, at discrete time t. bucket , θ arm , θ boom These three angles are defined as the state of the controlled object 40. The state signal is expressed by the following equation (9).

[0127]

[0128] The observation period of the state observer 50 is synchronized with the control period of the predictive control device 30, and is set so that the observation period of the state observer 50 occurs immediately after the timing at which the predictive control device 30 transmits an input signal. Unless otherwise specified, all angles below are in degrees.

[0129] Furthermore, this embodiment shows an example in which a user applies the predictive control device 30 as a model predictive control system 100, and the predictive control device 30 is utilized to automate the soil leveling operation using the bucket of a hydraulic excavator.

[0130] Example 1 Example 1 will be described along the steps shown in FIG. 3 in the first embodiment.

[0131] [Step S1] A user transmits any supervised training data, any initial parameters for a mean parameter θ, and any initial parameters for a variance parameter to the machine learning device 10 via the terminal device 20. As a result, the data acquisition unit 14 acquires the training data and the initial parameters transmitted from the terminal device 20 and stores them in the storage unit 16.

[0132] In Example 1, the supervised training data is data acquired from multiple soil break-in operations performed by a human operator. In the supervised training data, the explanatory variables are the state and control input at a certain discrete time t, and the objective variable is the state at discrete time t+1. It is assumed that the supervised training data follows the state transition equation of the following equation (10).

[0133]

[0134] Here, x≦t indicates the entire history of the three angles observed up to time t, and u≦t indicates the entire history of lever inputs performed up to time t. The main reasons for including the entire history are that with hydraulic excavators, it takes time for control inputs to be reflected in increases or decreases in angle (hydraulic delay), the inertia of the arm cannot be ignored, and communication delays exist in the remote control system. To address these issues, the entire history is used as the input for each model.

[0135] The variance associated with the state transition is mainly due to the randomness of communication delays in the remote control system. For example, the larger the absolute value of the control input, the greater the change in the angle at the next time due to the difference in input timing caused by communication delays.

[0136] Furthermore, x t+1 is a three-dimensional vector, the output of the Gaussian distribution mean function model μ(x≦t, u≦t) is a three-dimensional vector value. More generally, the output of the Gaussian distribution variance function model ν(x≦t, u≦t) is a symmetric matrix with 3 rows and 3 columns. However, for simplicity of explanation, in this embodiment, it is assumed that the off-diagonal components are 0, and the output of the Gaussian distribution variance function is its diagonal components, which are expressed as three-dimensional vector values.

[0137] The Gaussian distribution mean function model and Gaussian distribution variance function model that model the above equation (10) are deep neural network models that have a long-short-term memory (LSTM) layer in the first layer and a multilayer perceptron in the second layer. The number of nodes in each gate of the LSTM is 100, the multilayer perceptron has two hidden layers, each with 100 nodes, and the activation function is a tanh function. Random values ​​generated using pseudorandom numbers are used as initial parameters.

[0138] [Step S2] The batch data generation unit 15 extracts some of the training data from the large amount of training data stored in the storage unit 16 and generates the extracted training data as batch data. The batch data generation unit 15 also stores the created batch data in the storage unit 16. In this embodiment, the batch data generation unit 15 generates the batch data by randomly extracting 256 pieces of training data from the storage unit 16.

[0139] [Step S3] The first gradient value calculation unit 11 acquires the batch data generated in step S2 and the initial parameters of the mean parameter θ and the arbitrary variance parameter Φ stored in the storage unit 16 in step S1. Then, using these, the first gradient value calculation unit 11 calculates a gradient value for the variance parameter Φ of the average negative log-likelihood, which is calculated using the batch data, the mean parameter θ, and the variance parameter Φ. In this embodiment, the Gaussian distribution variance function model is represented by a deep neural network. Therefore, the first gradient value calculation unit 11 calculates the gradient value of the variance parameter Φ using the backpropagation algorithm.

[0140] [Step S4] The second gradient value calculation unit 12 acquires the batch data generated in step S2 and the initial parameter of the mean parameter θ stored in the storage unit 16 in step S1. Then, using these, the second gradient value calculation unit 12 calculates a gradient value for the mean parameter of the mean squared error calculated using the batch data and the mean parameter θ. In this embodiment, the Gaussian distribution mean function model is represented by a multilayer perceptron. Therefore, the second gradient value calculation unit 12 calculates the gradient value of the mean parameter θ using the backpropagation algorithm.

[0141] [Step S5] The parameter update unit 13 updates the variance parameter Φ stored in the storage unit 16 with the gradient value calculated in step S3, and updates the mean parameter θ stored in the storage unit 16 with the gradient value calculated in step S4. In this embodiment, the parameter update unit 13 uses a stochastic gradient method to update each parameter at a learning rate of 0.001.

[0142] [Step S6] After each parameter is updated, the loop process is generally resumed from step S2 (step S6: No). However, if a termination condition specified by the user is satisfied after each parameter is updated (step S6: Yes), the loop process is not resumed and the process in the machine learning device 10 ends. In this embodiment, the termination condition is, for example, that the total number of updates reaches 10,000.

[0143] Example 2 Example 2 will be described along the steps shown in FIG. 6 in the second embodiment.

[0144] [Step S11] The state observing device 50 observes the state of the controlled object 40 and outputs a state signal indicating the observed state. As a result, the signal acquiring unit 31 acquires the state signal from the state observing device 50 and inputs the acquired state signal to the prediction control unit 31.

[0145] [Step S12] Next, the prediction control unit 32 receives the state signal input from the signal acquisition unit 31 in step S11 and determines whether a preset termination condition is satisfied. In this embodiment, the termination condition is that the observed state reaches the final target coordinates of the reference trajectory described below.

[0146] If it is determined in step S12 that the termination condition is satisfied (S12: Yes), the processing in the prediction control device 30 ends.

[0147] [Step S13] If the result of the determination in step S12 is that the termination condition is not satisfied (S12: No), the prediction control unit 32 inputs the state signal input from the signal acquisition unit 31 in step S11 to the machine learning model 33.

[0148] [Step S14] Next, the prediction control unit 32 receives the output from the machine learning model 33, generates an input signal to be input to the control object 40 based on the received output, and transmits the generated input signal to the control object 40.

[0149] After step S14 is executed, the controlled object 40 receives the input signal transmitted from the predictive control device 30 and executes an operation according to the input signal. After step S14 is executed, step S11 is executed again.

[0150] In this embodiment, ITMPC is used as an algorithm for model predictive control. The prediction horizon H in ITMPC is set to 20, and the target variable to be optimized is defined as the control input for H steps from the current discrete time t to t+H-1. The algorithm for model predictive control is expressed as the following equation 11.

[0151]

[0152] Hereinafter, not only the control input but also other variables will be expressed as "discrete time: number of steps" to indicate that the variables at each time up to H steps ahead are included. ITMPC simulates the state transition of Equation 10 using the trained Gaussian distribution mean function and Gaussian distribution variance function model obtained by the machine learning device 10. In other words, the state x of the control object 40 received from the state observation device 50 t and the input plan u of a certain lever t:H Given x, we can use both models recursively to find t+1:H can be generated by Monte Carlo sampling. t+1:H And, u t:H The total cost function for and is defined as follows:

[0153]

[0154] Here, C is an instantaneous cost function. In this embodiment, in order to construct a trajectory tracking type predictive control system 100, the instantaneous cost function is defined as shown in the following Expression 13.

[0155]

[0156] Here, p x,s+1 , p y,s+1 , p θ、s+1 are the x and y coordinates [m] and azimuth angle of the bucket tip (point p) shown in FIG. 8 at discrete time s+1, and are determined by the structure of the excavator as θ s+1 It is calculated geometrically based on pref x,s+1 , p ref y,s+1 , p ref θ、s+1 are target coordinates at discrete time s+1. As shown in Fig. 9, the reference trajectory in this embodiment is constructed so that the height of the bucket tip from the ground plane is maintained and the bucket blade is always kept horizontally. Fig. 9 is a diagram showing an example of the reference trajectory used in this embodiment.

[0157] For example, suppose the reference trajectory is constructed to move horizontally between a point 0.63 m and a point 1.43 m from the excavator's pivot axis in the x direction while maintaining the bucket tip at a height of 0.89 m from the ground plane.

[0158] In FIG. 9, each arrow represents a target coordinate that constitutes a reference trajectory, and the arrow tip indicates p ref x and p ref y The arrow indicates the direction of p ref θ In addition, to avoid sudden starts and stops, the placement of the target coordinates is adjusted to have three phases: acceleration, constant velocity, and deceleration. Also, while FIG. 9 shows an example in which the reference trajectory is a task of horizontal movement forward, the reference trajectory may also be a task of horizontal movement backward, and is constructed using the same concept.

[0159] In addition, in the above formula 13, a x , a y , a θ are coefficients that determine the cost weight of each term, and in this embodiment, they are set to 10000, 10000, and 10, respectively. The composite function of the total cost function and the state transition function as described above is defined as the objective function of ITMPC in this embodiment.

[0160] (Effects of the Example) Fig. 10 is a diagram illustrating the effects of the example. Fig. 10 shows the transition of prediction accuracy by a machine learning model when the negative log-likelihood in the conventional method is used as the loss function and when the loss function used in the proposed method of the present invention is used.

[0161] This performance evaluation uses supervised data from records of multiple attempts by a human operator to perform leveling, and the explanatory and objective variables are standardized in preprocessing. The horizontal axis in Figure 10 represents the number of epochs, while the vertical axis, NLL, represents the negative log-likelihood, and RMSE, the root mean square error. Both the negative log-likelihood and the root mean square error are calculated using standardized values. Figure 10 shows the average value obtained from the results of five trials, with a 95% confidence interval.

[0162] According to the results shown in Figure 10, the proposed method of the present invention is superior in terms of both final accuracy and early learning accuracy in terms of RMSE. First, at the end of learning after 10,000 epochs, an approximately 7-fold improvement in accuracy was observed. Furthermore, the proposed method can achieve the accuracy achieved by the conventional method in 10,000 epochs with 1 / 200th the number of epochs. In terms of NLL, the conventional method is superior to the proposed method in terms of early learning accuracy, but the difference is not as large as the RMSE. Furthermore, after 6,000 epochs, the proposed method achieves the same accuracy as the conventional method. Furthermore, regarding the stability of the NLL transition, the learning curve of the conventional method shows multiple spikes, while the learning curve of the proposed method does not. Therefore, the proposed method is expected to converge with a higher probability.

[0163] As described above, the machine learning device 10 according to this embodiment can obtain a model that can predict the next state with one-seventh the bias of the conventional method at the end of learning. Therefore, according to this embodiment, the control performance of model predictive control can be improved compared to the conventional method.

[0164] Some or all of the above-described embodiments and examples can be expressed by (Appendix 1) to (Appendix 18) described below, but are not limited to the following descriptions.

[0165] (Supplementary Note 1) A machine learning device comprising: a first gradient value calculation unit that calculates a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a dependent variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; a second gradient value calculation unit that calculates a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameters are input, the gradient value being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; and a parameter update unit that updates the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and updates the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0166] (Supplementary Note 2) The machine learning device according to Supplementary Note 1, wherein the parameter update unit updates the variance parameter and the mean parameter so as to minimize a loss function that calculates a sum of the average negative log-likelihood and the mean squared error.

[0167] (Supplementary Note 3) The machine learning device according to Supplementary Note 2, wherein the loss function is expressed by the above-mentioned formula 5, where x is an explanatory variable, y is a target variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0168] (Supplementary Note 4) A prediction control unit is provided which acquires a signal indicating a state of a controlled object; and inputs the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model, and generates an input signal to be input to the controlled object based on an output from the machine learning model which serves as a dependent variable, wherein the machine learning model calculates a gradient value for the variance parameter of an average negative logarithm likelihood calculated using a plurality of training data including the explanatory variables and the dependent variable, a variance parameter of the Gaussian distribution variance function model which depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model, the gradient value for the mean parameter of a mean square error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are input, the gradient value being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model which depends on the explanatory variables; updates the variance parameter of the Gaussian distribution variance function model using the gradient value for the variance parameter of the average negative logarithm likelihood, and calculates a gradient value for the mean parameter of the mean square error. a prediction control device configured by updating a mean parameter of the Gaussian distribution mean function model using a Gaussian distribution mean function model.

[0169] (Supplementary Note 5) The prediction control device according to Supplementary Note 4, wherein the machine learning model is constructed by updating the variance parameter and the mean parameter so as to minimize a loss function that calculates a sum of the average negative log-likelihood and the mean squared error.

[0170] (Supplementary Note 6) The prediction control device according to Supplementary Note 5, wherein the loss function is expressed by the above formula 5, where x is an explanatory variable, y is a target variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0171] (Supplementary Note 7) A machine learning method comprising: calculating a gradient value for a variance parameter of an average negative logarithm likelihood calculated using a plurality of training data including explanatory variables and a dependent variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculating a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameters are input, the mean squared error being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; updating the variance parameter using the gradient value for the variance parameter of the average negative logarithm likelihood, and updating the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0172] (Supplementary Note 8) The machine learning method according to Supplementary Note 7, wherein in updating the parameters, the variance parameter and the mean parameter are updated so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

[0173] (Supplementary Note 9) The machine learning method according to Supplementary Note 8, wherein the loss function is expressed by the above formula 5, where x is an explanatory variable, y is a target variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0174] (Supplementary Note 10) A signal indicating a state of a controlled object is acquired, and the level of the acquired signal is input as an explanatory variable to a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model, and an input signal to be input to the controlled object is generated based on an output from the machine learning model, which serves as a dependent variable; the machine learning model is constructed by: calculating a gradient value for the variance parameter of an average negative logarithm likelihood, calculated using a plurality of training data including the explanatory variables and the dependent variable, a variance parameter of the Gaussian distribution variance function model that depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model; calculating a gradient value for the mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are input, calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model that depends on the explanatory variables; updating the variance parameter of the Gaussian distribution variance function model using the gradient value for the variance parameter of the average negative logarithm likelihood, and updating the mean parameter of the Gaussian distribution mean function model using the gradient value for the mean parameter of the mean squared error. A predictive control method comprising:

[0175] (Supplementary Note 11) The predictive control method according to Supplementary Note 10, wherein the machine learning model is constructed by updating the variance parameter and the mean parameter so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

[0176] (Supplementary Note 12) The predictive control method according to Supplementary Note 11, wherein the loss function is expressed by the above-mentioned formula 5, where x is an explanatory variable, y is a response variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0177] (Supplementary Note 13) A computer-readable recording medium having recorded thereon a program including instructions to: cause a computer to calculate a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a response variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculate a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the response variable when the explanatory variables and mean parameters are input, calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; update the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and update the mean parameter using the gradient value for the mean parameter of the mean squared error.

[0178] (Supplementary Note 14) The computer-readable recording medium according to Supplementary Note 13, wherein in updating the parameters, the variance parameter and the mean parameter are updated so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

[0179] (Supplementary Note 15) The computer-readable recording medium according to Supplementary Note 14, wherein the loss function is expressed by the above-mentioned formula 5, where x is an explanatory variable, y is a target variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0180] (Supplementary Note 16) A program including instructions is recorded on a computer to: acquire a signal indicating a state of a controlled object; input the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model; and generate an input signal to be input to the controlled object based on an output from the machine learning model, which serves as a dependent variable; wherein the machine learning model calculates a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variables and the dependent variable, a variance parameter of a Gaussian distribution variance function model that depends on the explanatory variables, and a mean parameter of a Gaussian distribution mean function model that depends on the explanatory variables; calculates a gradient value for a mean parameter of a mean square error between an output value of the Gaussian distribution mean function model and the dependent variable when the explanatory variables and the mean parameter are used as input, which is calculated using the plurality of training data and the mean parameter of a Gaussian distribution mean function model that depends on the explanatory variables; a gradient value for the variance parameter of the average negative log-likelihood, and a gradient value for the mean parameter of the mean squared error, the gradient value being used to update a variance parameter of the Gaussian distribution variance function model; and a gradient value for the mean parameter of the mean squared error, the gradient value being used to update a mean parameter of the Gaussian distribution mean function model.

[0181] (Supplementary Note 17) The computer-readable storage medium according to Supplementary Note 16, wherein the machine learning model is constructed by updating the variance parameter and the mean parameter so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

[0182] (Supplementary Note 18) The computer-readable recording medium according to Supplementary Note 17, wherein the loss function is expressed by the above formula 5, where x is an explanatory variable, y is a target variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

[0183] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0184] As described above, the present disclosure can reduce the influence of the output value of a Gaussian distribution variance function model on the gradient value of the parameters of a Gaussian distribution mean function model, thereby improving the convergence in machine learning. The present disclosure can be applied to applications such as machine learning systems for solving regression problems. It can also be applied to applications such as control systems that utilize regression models, such as model predictive control.

[0185] 10 Machine learning device 11 First gradient value calculation unit 12 Second gradient value calculation unit 13 Parameter update unit 14 Data acquisition unit 15 Batch data generation unit 16 Storage unit 20 Terminal device 30 Prediction control device 31 Signal acquisition unit 32 Prediction control unit 33 Machine learning model 40 Control target 50 State observation device 100 Model predictive control system 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims

1. A machine learning device comprising: a first gradient value calculation means for calculating a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a target variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; a second gradient value calculation means for calculating a gradient value for a mean parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the target variable when the explanatory variables and the mean parameters are input, the mean squared error being calculated using the plurality of training data and the mean parameter of the Gaussian distribution mean function model; and a parameter update means for updating the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and updating the mean parameter using the gradient value for the mean parameter of the mean squared error.

2. The machine learning device according to claim 1, wherein the parameter update means updates the variance parameter and the mean parameter so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

3. The machine learning device according to claim 2, wherein the loss function is expressed by the following equation 14, where x is an explanatory variable, y is an objective variable, μ(x, θ) is a Gaussian distribution mean function model, θ is a mean parameter, θ hat is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

4. A signal acquisition means for acquiring a signal indicating a state of a controlled object; and a predictive control means for inputting the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution average function model, and generating an input signal to be input to the controlled object based on an output from the machine learning model which serves as a target variable, wherein the machine learning model calculates a gradient value for the variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variables and the target variable, a variance parameter of a Gaussian distribution variance function model that depends on the explanatory variables, and an average parameter of a Gaussian distribution average function model that depends on the explanatory variables; calculates a gradient value for the average parameter of a mean squared error between an output value of the Gaussian distribution average function model and the target variable when the explanatory variables and the average parameter are input, calculated using the plurality of training data and an average parameter of a Gaussian distribution average function model that depends on the explanatory variables; a variance parameter of the Gaussian distribution variance function model is updated using a gradient value for the variance parameter of an average negative log-likelihood, and a mean parameter of the Gaussian distribution mean function model is updated using a gradient value for the mean parameter of a mean squared error.

5. The prediction control device according to claim 4, wherein the machine learning model is constructed by updating the variance parameter and the mean parameter so as to minimize a loss function that calculates the sum of the average negative log-likelihood and the mean squared error.

6. The prediction control device according to claim 5, wherein the loss function is expressed by the following equation 15, where x is an explanatory variable, y is an objective variable, μ(x) is a Gaussian distribution mean function model, θ is a mean parameter, θ is a constant set based on the mean parameter θ, v(x, Φ) is a Gaussian distribution variance function model, and Φ is a variance parameter.

7. A machine learning method comprising: calculating a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a target variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculating a gradient value for an average parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the target variable when the explanatory variables and the mean parameters are input, the mean squared error being calculated using the plurality of training data and the mean parameter of a Gaussian distribution mean function model; updating the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood; and updating the mean parameter using the gradient value for the mean parameter of the mean squared error.

8. A predictive control method comprising: acquiring a signal indicating a state of a controlled object; inputting a level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution mean function model; and generating an input signal to be input to the controlled object based on an output from the machine learning model, which is a target variable; the machine learning model is constructed by: calculating a gradient value for the variance parameter of an average negative logarithm likelihood calculated using a plurality of training data including the explanatory variables and the target variable, a variance parameter of a Gaussian distribution variance function model that depends on the explanatory variables, and an average parameter of a Gaussian distribution mean function model; calculating a gradient value for an average parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the target variable when the explanatory variables and the average parameter are input, the gradient value being calculated using the plurality of training data and the average parameter of a Gaussian distribution mean function model that depends on the explanatory variables; updating the variance parameter of the Gaussian distribution variance function model using the gradient value for the variance parameter of the average negative logarithm likelihood, and updating the average parameter of the Gaussian distribution mean function model using the gradient value for the average parameter of the mean squared error.

9. A computer-readable recording medium having recorded thereon a program including instructions to: cause a computer to calculate a gradient value for a variance parameter of an average negative log-likelihood calculated using a plurality of training data including explanatory variables and a response variable, a variance parameter of a Gaussian distribution variance function model, and a mean parameter of a Gaussian distribution mean function model; calculate a gradient value for an average parameter of a mean squared error between an output value of the Gaussian distribution mean function model and the response variable when the explanatory variables and the mean parameters are input, calculated using the plurality of training data and the mean parameter of a Gaussian distribution mean function model; update the variance parameter using the gradient value for the variance parameter of the average negative log-likelihood, and update the mean parameter using the gradient value for the mean parameter of the mean squared error.

10. A program including instructions is recorded on a computer to acquire a signal indicating a state of a controlled object, input the level of the acquired signal as an explanatory variable into a machine learning model having a Gaussian distribution variance function model and a Gaussian distribution average function model, and generate an input signal to be input to the controlled object based on an output from the machine learning model which is a target variable, wherein the machine learning model calculates a gradient value for the variance parameter of an average negative log-likelihood calculated using a plurality of training data including the explanatory variable and the target variable, a variance parameter of a Gaussian distribution variance function model that depends on the explanatory variables, and an average parameter of a Gaussian distribution average function model that depends on the explanatory variables, calculates a gradient value for the average parameter of a mean squared error between an output value of the Gaussian distribution average function model and the target variable when the explanatory variable and the average parameter are input, calculated using the plurality of training data and the average parameter of a Gaussian distribution average function model that depends on the explanatory variables, a gradient value for the variance parameter of an average negative log-likelihood is used to update a variance parameter of the Gaussian distribution variance function model, and a gradient value for the mean parameter of a mean squared error is used to update a mean parameter of the Gaussian distribution mean function model.

Citation Information

Patent Citations

  • Information processing device, learning device, information processing method, learning method, information processing program, and learning program

    JP2021140484A

  • Neural network systems implementing conditional neural processes for efficient learning

    WO2019155065A1

Cited By

  • Clock error drift modeling method and system based on statistical likelihood optimization

    CN121598812A

  • Soft soil foundation settlement probability prediction method based on transfer learning enhancement

    CN121786399A

  • Landslide dominant flow channel identification method and system based on width learning algorithm

    CN121787610A