Machine learning programs, machine learning methods, and machine learning devices

Abstract

To minimize an average error of a control target point including speed control with respect to a goal.SOLUTION: A machine learning program causes a computer to execute: an acquisition step of acquiring observation information including information on a speed of a control target point at a control target time; an output step of outputting control information including information on speed control of the control target point, the control information being determined in accordance with the observation information and a control policy; a corrected reward determination step of determining a corrected reward obtained by correcting a reward in accordance with a speed of the control target point included in the observation information, the reward being higher as an error between a value of an evaluation index and a goal is smaller, the evaluation index being a value of an index other than a speed derived from the observation information; and a learning step of performing reinforcement learning of the control policy based on the observation information and the corrected reward.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a machine learning program, a machine learning method, and a machine learning apparatus. [Background technology]

[0002] Attempts have been made to apply reinforcement learning to the learning of various controls. Patent Document 1 discloses a method for learning speed control to minimize deviation from the commanded path by performing reinforcement learning based on the deviation from the commanded path. Non-Patent Document 1 discloses a method for learning welding control, including welding speed, using reinforcement learning in laser welding by calculating a reward based on the difference between the desired bead width and the generated bead width. Non-Patent Document 2 discloses a method for learning a control strategy when controlling a system to satisfy a given objective by replacing the objective in accumulated experience data and using it for learning. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Patent No. 6077617 [Non-patent literature]

[0004] [Non-Patent Document 1] M. Schmitz, F. Pinsker, A. Ruhri, B. Jiang and G. Safronov, “Enabling Rewards for Reinforcement Learning in Laser Beam Welding processes through Deep Learning,” 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 14-17 December, 2020. [Overview of the project] [Problems that the invention aims to solve]

[0005] Reinforcement learning is a method for learning a strategy that maximizes the expected value of the discounted cumulative reward. The discounted cumulative reward is calculated by summing the rewards obtained from the current time onward, multiplying each reward by a smaller weight as the time difference from the current time increases. By performing reinforcement learning using rewards calculated based on errors, as described in Patent Document 1 and Non-Patent Document 1, it is possible to learn a control method that minimizes errors. However, when the velocity of the controlled point changes, the distance traveled per unit time fluctuates depending on the velocity, so the discounted cumulative error fluctuates not only due to the error calculated from the trajectory but also due to the velocity. For this reason, it has been difficult to minimize the average error of the trajectory of the controlled point with respect to the target trajectory, including velocity control, with respect to the target trajectory in conventional techniques.

[0006] The problem that this invention aims to solve is to provide a machine learning program, a machine learning method, and a machine learning apparatus that can minimize the average error of a controlled point, including speed control, relative to a target. [Means for solving the problem]

[0007] The machine learning program of the embodiment is a machine learning program that causes a computer to execute the following steps: an acquisition step of acquiring observation information including information about the velocity of a control target point at a control target time; an output step of outputting control information including information about velocity control of the control target point, determined according to the observation information and the control policy; a corrected reward determination step of determining a corrected reward in which the smaller the error between the evaluation index of the values ​​of indicators other than velocity derived from the observation information and the target, the higher the reward, according to the velocity of the control target point included in the observation information; and a learning step of reinforcing the control policy from the observation information and the corrected reward. [Brief explanation of the drawing]

[0008] [Figure 1] A schematic diagram of the learning system. [Figure 2]A diagram showing the correspondence between the target evaluation indicator at the position of the controlled point and the evaluation indicator that was actually achieved. [Figure 3] Functional block diagram of a machine learning device. [Figure 4] A schematic diagram of the display screen. [Figure 5A] A schematic diagram of the display screen. [Figure 5B] A schematic diagram of the display screen. [Figure 6] A flowchart illustrating the flow of information processing. [Figure 7] Hardware. [Modes for carrying out the invention]

[0009] The machine learning program, machine learning method, and machine learning apparatus of this embodiment will be described in detail below with reference to the attached drawings.

[0010] Figure 1 is a schematic diagram of an example of the learning system 1 of this embodiment.

[0011] The learning system 1 comprises a machine learning device 10 and a controlled device 20. The machine learning device 10 and the controlled device 20 are connected in a communication manner.

[0012] The machine learning device 10 is an information processing device that performs reinforcement learning. In other words, the machine learning device 10 is the agent that is the main subject of learning. The machine learning device 10 is a computer for executing the machine learning program of this embodiment.

[0013] The controlled device 20 is the object controlled by the machine learning device 10. In other words, the controlled device 20 is the target to which control information determined according to the control strategy learned by the machine learning device 10 is applied.

[0014] The controlled device 20 may be, for example, a robot such as a Cartesian coordinate robot or an articulated robot, a machine tool such as a laser processing or laser welding machine, or an unmanned mobile device such as an automated guided vehicle or a drone. The controlled device 20 may also be a computer simulator that simulates the operation of these devices.

[0015] The machine learning device 10 learns a control policy so that the controlled points controlled by the controlled device 20 achieve the target in the evaluation index. In other words, the machine learning device 10 learns a control policy that minimizes the average error of the controlled points relative to the target.

[0016] A control target point is a point that is controlled at each of the consecutive control target time points along a time series. If the control target device 20 is a robot, the control target point is, for example, the tip of the robot arm or a specific position on the end effector. If the control target device 20 is a machine tool such as a laser processing or laser welding machine, the control target point is, for example, the laser irradiation point during laser processing. If the control target device 20 is an unmanned mobile body such as an automated guided vehicle or a drone, the control target point is, for example, the center of gravity of the unmanned mobile body.

[0017] In reinforcement learning, the learning of the machine learning device 10 progresses through interaction between the machine learning device 10, which is the main learning agent, and the controlled device 20, which is the target of the control.

[0018] Specifically, the controlled device 20 outputs observational information of the controlled point at each controlled time point to the machine learning device 10. The machine learning device 10 determines control information representing the action based on the observational information and control strategy acquired from the controlled device 20, and outputs it to the controlled device 20. The machine learning device 10 learns by repeating this series of processes.

[0019] Observation information refers to information that represents the state of the controlled point at the controlled time, and is information necessary for controlling the controlled device 20. In this embodiment, the observation information includes at least information regarding the velocity of the controlled point at the controlled time.

[0020] Information regarding the velocity of the controlled point only needs to be information that can identify the velocity of the controlled point at the controlled time. More specifically, the information regarding the velocity of the controlled point represents at least one of the following at the controlled time: position, velocity, acceleration, and distance traveled per unit time.

[0021] Control information refers to information used to control the behavior of a controlled point. In this embodiment, the control information includes at least information regarding the velocity control of the controlled point.

[0022] Specifically, if the controlled device 20 is a drone, the control information includes the velocity or acceleration in each direction (forward, backward, left, right, up, and down), while the observation information includes the drone's position, velocity, and surrounding information, which are necessary for controlling the drone. Surrounding information includes, for example, images of the surroundings taken by a camera, depth images, and occupied grid maps.

[0023] If the controlled device 20 is a multi-joint robot, the control information includes the torque and angle of each joint, and the position, orientation, and velocity of the controlled point. The observation information includes the angle and angular velocity of each joint, the position, orientation, and velocity of the controlled point, and information about the working environment, which are all necessary for controlling the multi-joint robot. Information about the working environment includes, for example, images of the surroundings taken by a camera, depth images, etc.

[0024] If the controlled device 20 is a laser welding machine, the control information includes welding speed, welding acceleration, laser power, and spot diameter. The observation information includes information necessary for controlling the laser welding machine, such as the laser irradiation position, irradiation speed, spot diameter, gap between materials, width of the bead or molten pool, and information around the welding position. Information around the welding position includes, for example, images of the area around the welding position taken by a camera, and temperature distribution.

[0025] Next, the basic concept of reinforcement learning will be described. In this embodiment, a form using goal-conditioned reinforcement learning as reinforcement learning will be described.

[0026] Goal-conditioned reinforcement learning is a method of learning a control policy that determines an action a t from the state s t input at a certain control target time t when a goal g is given.

[0027] The state s t corresponds to the observation information at the control target time t or a part thereof. The action a t corresponds to the control information.

[0028] The control policy is a probability distribution represented by π(a t |s t , g). The control policy π(a t |s t , g) is learned, for example, by a neural network that outputs probability values or parameters of a probability model.

[0029] Goal-conditioned reinforcement learning aims to learn a control policy π(a t |s t , g) that maximizes the expected value of the discounted cumulative reward represented by the following formula (1). The discounted cumulative reward is the sum of the rewards obtained after the current time, multiplied by weights that become smaller as the time difference from the current time increases.

[0030]

Equation

[0031] In Equation (1), r(s t , a t , g) represents the reward calculated at time t + 1 as a result of taking the action a t in the state s when the goal g is given. In Equation (1), γ represents the discount rate. k is an integer greater than or equal to 0.​​

[0032] The discount rate γ is a parameter between 0 and 1 that adjusts how much consideration is given to distant future rewards when making decisions about actions. In other words, the discount rate γ is a hyperparameter for adjusting how far into the future is considered. The discount rate γ is a parameter used to discount rewards that are obtained in the distant future more heavily. The discount rate γ also plays a role in regularization, which stabilizes learning.

[0033] Various algorithms are known for reinforcement learning. Many of them use a value function V(s t ,g) and the action-value function Q(s t ,a t Includes the learning step of ,g).

[0034] Value function V(s t ,g) is a state s given the target g. t From the current control policy π(a t |s t This is an estimate of the discounted cumulative reward obtained by acting according to (g). Value function V(s t When using a method called TD (Temporal Difference) learning, the value of g) is updated (learned) by the update formula shown in equation (2) below.

[0035]

number

[0036] In equation (2), α represents the learning rate.

[0037] Action value function Q(s t ,a t ,g) is a state s given the target g. t Action a t After taking the current control policy π(a t |s t This is an estimate of the discounted cumulative reward obtained when acting according to (g). Action value function Q(s t ,at The value of g) is updated (learned) in TD learning by the update formula expressed by equation (3) below.

[0038]

number

[0039] In equation (3), equation (4) below is generally difficult to calculate.

[0040]

number

[0041] Therefore, instead of equation (4) in equation (3), we have the value function V(s t Using ,g), or the control policy π(a|s t+1 The action value function Q(s) of only action a sampled according to g). t+1 They use ,a,g).

[0042] Value function V(s t ,g) and the action-value function Q(s t ,a t g) is learned, for example, by a linear model or a neural network.

[0043] Next, we will explain a method for learning a control policy that enables the controlled object to achieve the target in the evaluation metric using goal-conditional reinforcement learning.

[0044] Figure 2 shows an example of the correspondence between the target evaluation index g at the position x of the controlled point and the evaluation index f(x) that was actually achieved.

[0045] Position x is the distance from the reference position on the predetermined trajectory of the controlled point. The trajectory of the controlled point is the trajectory of the controlled point at each predetermined controlled time. If the predetermined trajectory of the controlled point is a straight line, the reference position can be used as the origin and the predetermined trajectory can be used as the x-axis.

[0046] The target is the target value for the evaluation metric f(x).

[0047] The evaluation index f(x) is the value of an index other than velocity, derived from the observed information, and represents the index that was actually achieved. Derivation means any of the following: calculation, operation, determination, identification, or reading. As mentioned above, the observed information includes, for example, information necessary for controlling the drone, such as the drone's position, velocity, and surrounding information, depending on the type of controlled device 20; information necessary for controlling a multi-joint robot, such as the angle and angular velocity of each joint, the position, attitude, and velocity of the controlled point, and information about the working environment; welding speed, welding acceleration, laser power, and spot diameter. The observed information also includes information necessary for controlling the laser welding machine, such as the laser irradiation position, irradiation speed, spot diameter, gap between materials, width of the bead or molten pool, and information around the welding position. The evaluation index f(x) is the value of an index other than velocity, derived from this observed information.

[0048] Specifically, the evaluation index f(x) and target g are as follows: For example, in the case of drone control, the evaluation index f(x) is the distance between the set trajectory (a pre-set path of the controlled point) and the current position of the controlled point, and the target g is set to 0 (zero). In the case of laser welding control, the bead width or penetration depth is used as the evaluation index f(x), and a positive constant is set to the target g.

[0049] In goal-conditional reinforcement learning, reinforcement learning is performed to achieve the goal. Therefore, the smaller the error d(x) between the goal g and the actually achieved evaluation metric f(x), the larger the reward r(s) t ,a t The reward is defined such that ,g) is given. For the error d(x), for example, L1 distance or L2 distance can be used. In goal-conditional reinforcement learning, the error d(x) is integrated from time t to t+1 and multiplied by -1 to obtain the reward r(s) as shown in equation (5) below. t ,a t Define and learn (g).

[0050]

number

[0051] In equation (5), x(t) represents the position of the controlled point at time t. Time t has the same meaning as controlled time t.

[0052] The definition of reward expressed by equation (5) is the control policy π(a) that minimizes the expected value of the objective function expressed by equation (6) below. t |s t This means learning g).

[0053]

number

[0054] Equation (6) is the objective function that represents the average error of the controlled point relative to the target. More specifically, the average error expressed by equation (6) represents the integral value obtained by integrating the error d(x) between the target trajectory, which is the predetermined trajectory of the controlled point, and the actual trajectory of the controlled point, along the target trajectory.

[0055] Furthermore, as an approximation of the definition of reward expressed by equation (5), using only the error d(x) at time t+1, the reward r(s t ,a t ,g) to r(s t ,a t Another method is to define it as ,g)=d(x(t+1)). The definition of reward expressed by this equation is the control policy π(a) that minimizes the expected value of the objective function expressed by equation (7) below, which is obtained by discretely calculating equation (6) above. t |s t This corresponds to learning g).

[0056]

number

[0057] In controlling a controlled device 20 such as a drone, if the goal is to minimize the difference between the predetermined trajectory of the controlled point and the actual trajectory, the objective function, with a discount rate γ for reinforcement learning, is expressed by the following equation (8).

[0058]

number

[0059] Equation (8) is the objective function that represents the average error of the controlled point relative to the target. More specifically, the average error expressed by equation (8) represents the integral value obtained by integrating the error d(x) between the target trajectory, which is the predetermined trajectory of the controlled point, and the actual trajectory of the controlled point, along the target trajectory.

[0060] In the case of a controlled device 20 such as a laser welding machine, if the goal is to set the bead width or penetration depth to a predetermined value, the same equation (8) becomes the objective function.

[0061] Here, when the velocity of the controlled point is constant, minimizing the expected value of the objective function expressed by equation (6) is the same as minimizing the objective function expressed by equation (8). However, when the velocity of the controlled point is not constant, minimizing the expected value of the objective function expressed by equation (6) and minimizing the objective function expressed by equation (8) are different. In detail, for example, when the value of the error d(x) is large, the deviation from the target g is larger when the velocity is large compared to when the velocity is small, and the effect on equation (8) is greater. However, since equation (6) integrates the error d(x) with respect to time, the effect of velocity is not taken into consideration.

[0062] Therefore, in conventional reinforcement learning, when learning a control policy for a controlled point, including velocity control, it was not possible to perform reinforcement learning that optimizes the original objective function, equation (8). In other words, in conventional reinforcement learning, when learning a control policy for a controlled point, including velocity control, it was difficult to minimize the average error of the controlled point relative to the target.

[0063] Therefore, in the machine learning device 10 of this embodiment, the control policy is reinforced and learned using a corrected reward, which is obtained by correcting the reward according to the velocity of the control target point included in the observation information. By using a corrected reward, the machine learning device 10 of this embodiment can ensure that changes in velocity do not affect the value of the average error, and can learn a control policy that minimizes the average error.

[0064] Furthermore, the machine learning device 10 of this embodiment reinforces the control policy by using a corrected discount rate, which is obtained by correcting the reward discount rate according to the distance moved by the controlled point, instead of the reward discount rate. By using a corrected discount rate, the machine learning device 10 of this embodiment can ensure that changes in speed do not affect the value of the discounted cumulative reward, and can learn a control policy that minimizes the average error.

[0065] In other words, the machine learning device 10 of this embodiment provides a method for learning a control strategy that minimizes the average error expressed by equation (8) above, taking into account the influence of speed.

[0066] Therefore, in this embodiment, the reward is defined by the following formula (9).

[0067]

number

[0068] With the reward expressed by equation (9) above, equation (8) above can be expressed by equation (10) below.

[0069]

number

[0070] As described above, goal-conditional reinforcement learning employs a control policy π(a) that maximizes the expected value of the discounted cumulative reward expressed by equation (1) above. t |s tThe learning objective is to learn g). Therefore, in order to maximize the expected value of the discounted cumulative reward expressed by equation (10) above, the discount rate γ at time t+k is k This needs to be replaced with the following equation (11).

[0071]

number

[0072] Therefore, in this embodiment, the value function V(s t The TD learning of g) is determined by the update formula expressed by equation (12) below.

[0073]

number

[0074] Furthermore, in this embodiment, the action value function Q(s t ,a t The TD learning of g) is determined by the update formula expressed by equation (13) below.

[0075]

number

[0076] In other words, in this embodiment, the value function V(s t The above equation (2), which is the update formula for ,g), and the action-value function Q(s t ,a t Instead of the discount rate γ in equation (3) above, which is the update formula for ,g), a corrected discount rate obtained by correcting the discount rate γ by the velocity is used. The corrected discount rate is expressed by the following equation (14).

[0077]

number

[0078] In formula (14), x (t+1) -x (t)This is the velocity v(t) at time t and the control period δ. T And using v(t)δ T It can also be calculated as follows: That is, the adjusted discount rate can also be expressed by the following formula (15).

[0079]

number

[0080] Furthermore, in this embodiment, as a corrected reward obtained by correcting the reward according to the velocity of the controlled point included in the observation information, the above equation (9) or an approximate calculation of the above equation (9), such as the following equations (16) to (19), is used.

[0081]

number

[0082] In other words, the machine learning device 10 of this embodiment reinforces the control policy using a corrected reward, which is obtained by correcting the reward according to the velocity of the control target point included in the observation information. By using the corrected reward, the machine learning device 10 of this embodiment can learn a control policy that minimizes the average error.

[0083] Furthermore, the machine learning device 10 of this embodiment further reinforces the control policy by using a corrected discount rate, which is obtained by correcting the reward discount rate according to the distance moved by the controlled point, instead of the reward discount rate. By using a corrected discount rate, the machine learning device 10 of this embodiment can ensure that changes in speed do not affect the value of the discounted cumulative reward, and can further learn a control policy that minimizes the average error.

[0084] The problem with goal-conditional reinforcement learning is that, in the early stages of learning, it is difficult to obtain an action sequence that achieves the goal g, requiring a very large number of action searches. Therefore, in the machine learning device 10 of this embodiment, in order to further improve the efficiency of goal-conditional reinforcement learning, the goal of an action sequence that failed to achieve the goal is replaced and used as an action sequence that achieved the goal for learning. For example, the action sequence a0, a1, a2, ... a t As a result of performing the action, unfortunately, goal g was not achieved, but another goal g' was achieved. In this case, the action sequence a0, a1, a2, ... a t While this is a failure case, by replacing the goal with another goal g', it can be used as a success case for learning. In this way, learning can be made more efficient not only by learning from failure cases as they are, but also by creating success cases by replacing the goals and using them for learning.

[0085] Therefore, in the machine learning device 10 of this embodiment, control strategies can be efficiently learned by setting multiple evaluation indicators as targets.

[0086] Next, the configuration of the machine learning device 10 in this embodiment will be described in detail.

[0087] Figure 3 is a functional block diagram of an example of the machine learning device 10 of this embodiment.

[0088] The machine learning device 10 comprises a communication unit 12, a UI (user interface) unit 14, and a storage unit 16. The communication unit 12, UI unit 14, storage unit 16, and control unit 18 are communicated together via a bus 19 or the like.

[0089] The communication unit 12 communicates with external information processing devices such as the controlled device 20 via a network or the like. The UI unit 14 has a display function and an input function. The display function displays various types of information. Examples of the display function include a display, projection device, etc. The input function accepts user input. Examples of the input function include a pointing device such as a mouse and touchpad, a keyboard, etc. A touch panel that integrates the display function and the input function may also be used. The storage unit 16 stores various types of information.

[0090] The UI unit 14 and the storage unit 16 may be configured to be connected to the control unit 18 via wired or wireless means in a manner that allows communication. At least one of the UI unit 14 and the storage unit 16 may be connected to the control unit 18 via a network or the like.

[0091] Furthermore, at least one of the UI unit 14 and the storage unit 16 may be located outside the machine learning device 10. Alternatively, at least one of the one or more functional units included in the UI unit 14, the storage unit 16, and the control unit 18 may be mounted on an external information processing device that is connected to the machine learning device 10 via a network or the like in a way that allows communication.

[0092] The control unit 18 performs information processing in the machine learning device 10. The control unit 18 includes an acquisition unit 18A, a learning unit 18B, an output unit 18C, an experience data editing unit 18D, a target setting unit 18E, a corrected reward determination unit 18F, and a corrected discount rate determination unit 18G.

[0093] The acquisition unit 18A, learning unit 18B, output unit 18C, experience data editing unit 18D, goal setting unit 18E, corrected reward determination unit 18F, and corrected discount rate determination unit 18G are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (CPU (Transparent Processing Unit), i.e., by software, executing a program on it. Each of the above units may also be implemented by a dedicated IC or other processor, i.e., by hardware. Each of the above units may also be implemented by using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units.

[0094] The acquisition unit 18A acquires observation information. As described above, the observation information represents the state of the controlled point at controlled time t, and includes information regarding the velocity of the controlled point at controlled time t. The observation information also includes the target g for the evaluation index. The acquisition unit 18A sequentially acquires the observation information output sequentially from the controlled device 20 at each controlled time t. Each time the acquisition unit 18A acquires observation information for controlled time t, it outputs the acquired observation information to the learning unit 18B.

[0095] The learning unit 18B processes the observation information for the controlled time t received from the acquisition unit 18A, such as extracting, scaling, and clipping some of the data, to create a state s that the observation information can be used for reinforcement learning. t Convert to [the specified format]. If the observation information includes an image, the learning unit 18B may perform image processing or image recognition processing.

[0096] Next, the learning unit 18B processes the observation information of the controlled target time t received from the acquisition unit 18A and sets the current control policy π(a t |s t Using g), action a t To decide.

[0097] In detail, the learning unit 18B extracts the target g from the observation information to determine the control policy π(a t |s t, identify (g). Then, the learning unit 18B samples an action a t |s t , g) according to the control policy π(a t . The learning unit 18B may determine the action a t with the maximum probability. If the control policy π(a t |s t , g) is configured to directly output an action a t |s t , g) from the state s t and the target g, the learning unit 18B may use the state s t |s t , g) and the control policy π(a t to directly determine the action a t |s t , g). Also, the learning unit 18B may sample an action a t randomly without using the control policy π(a

[0098] for a certain period from the start. The learning unit 18B outputs the action a t determined by these processes to the output unit 18C.

[0099] The output unit 18C outputs control information including information related to the speed control of the control target point, which is determined according to the observation information and the control policy. Specifically, the output unit 18C receives the action a t from the learning unit 18B. The output unit 18C performs processes such as scaling on the action a t received from the learning unit 18B to convert the action a t into control information and output it to the control target device 20.

[0100] Also, the learning unit 18B uses the data for learning as experience data and stores it in the storage unit 16. Specifically, the learning unit 18B uses, as the experience data corresponding to the control target time t, the target g, the achieved evaluation index f(x(t)), the speed v(t) of the control target point or the speed x(t) - x(t - 1), the state s t-1 at one control time before, and the action a t-1The experience data including [specific content] is stored in the storage unit 16.

[0101] Using the reinforcement learning algorithm to be used, the learning unit 18B calculates the state s t , the value V(s t , g) of the value function, the value Q(s t , a t , g) of the action value function, the action a <000009�>'s probability value π(a t-1 |s t-1 , g), etc., may be further included in the experience data and stored in the storage unit 16.

[0102] The learning unit 18B further performs a process of updating the control policy π(a t |s t , g), the value function V(s t , g), and the action value function Q(s t [[ID=3๐]], a t , g) at a certain frequency. This update process corresponds to learning. Details of this learning will be described later.

[0103] The storage unit 16 stores the experience data input from the learning unit 18B up to a predetermined maximum number of experience data. When the experience data stored in the storage unit 16 exceeds the maximum value, the control unit 18 discards the old experience data.

[0104] When the learning unit 18B performs the update process of the control policy π(a t |s t , g), the value function V(s t , g), and the action value function Q(s t , a t , g), the experience data editing unit 18D randomly samples a certain number (M) of experience data from the storage unit 16. M is an integer greater than or equal to 1. Also, the experience data editing unit 18D identifies, as a series of peripheral experience data corresponding to each of the M sampled experience data, other experience data corresponding to other control target times t' within a predetermined period from the control target time t corresponding to each of the M sampled experience data.

[0105] Then, the experience data editing unit 18D uses the M sampled experience data and the surrounding experience data sequences corresponding to each of the M experience data to generate MK edited experience data for the learning unit 18B to use for training. K is an integer greater than or equal to 1.

[0106] In detail, the experience data editing unit 18D outputs the M sampled experience data and the surrounding experience data series corresponding to each of the M experience data to the target setting unit 18E.

[0107] The target setting unit 18E sets a target g based on a group of evaluation indices selected from a group of evaluation indices, consisting of a first empirical data set containing an evaluation index f(x(t)) derived from observation information acquired by the acquisition unit 18A, and one or more second empirical data sets each containing an evaluation index f(x(t')) derived from one or more other observation information at a different control target time t'. j Set it as follows.

[0108] The first set of empirical data corresponds to M sampled empirical data. The second set of empirical data corresponds to the surrounding empirical data series that correspond to each of the M empirical data.

[0109] In detail, the goal setting unit 18E sets K goals for each of the M pieces of experience data received from the experience data editing unit 18D. j Determine the value of j, which is expressed by equation (20) below.

[0110]

number

[0111] First, the goal setting unit 18E sets the goal g contained in each of the M experience data as goal g0 (j=0). Next, the goal setting unit 18E randomly samples K-1 evaluation indices f(x(t')) contained in the second experience data, which is the experience data that constitutes the surrounding experience data series corresponding to each of the M experience data, and sets K-1 goal g j (j=1,2,...,K-1) is determined. Through these processes, the target setting unit 18E determines K targets g for each of the M empirical data. j Set (j=0,1,2,...,K-1).

[0112] The target setting unit 18E may limit the sampling range of K-1 evaluation indices f(x(t')) to evaluation indices f(x(t')) achieved at a controlled time t' that is later than the controlled time t of the empirical data.

[0113] Furthermore, the target setting unit 18E adds noise to each of the K-1 evaluation indices f(x(t')) sampled from the evaluation indices f(x(t')) included in the second experience data, and sets the K-1 target g j You may also set it as (j=1,2,...,K-1). For the noise, you can use noise generated according to a probability distribution such as a Gaussian distribution or a uniform distribution.

[0114] Furthermore, the goal setting unit 18E replaces the second set of experience data, which is the experience data constituting the surrounding experience data series corresponding to each of the M pieces of experience data (first set of experience data), with a value randomly selected from the range of possible evaluation indicators f(x(t)) contained in each of the M pieces of experience data, and sets K-1 goals g j You can also set it as (j=1,2,...,K-1).

[0115] The target setting unit 18E sets K targets g according to the target selection method selected by the user. j (j=1,2,...,K-1) may be set. Also, the target setting unit 18E sets K-1 targets g, which is the number selected by the user. jYou can also sample (j=1,2,...,K-1).

[0116] For example, the goal setting unit 18E displays a display screen on the UI unit 14 for receiving input regarding the goal selection method and the number of goals to be selected.

[0117] Figure 4 is a schematic diagram of an example of the display screen 30. For example, the target setting unit 18E displays the display screen 30 on the UI unit 14.

[0118] The display screen 30 has a section for selecting the target selection method, and the target g j Includes an input field for the number of selections, and

[0119] Examples of goal selection methods include "no additions," "random," "future," and "future (with noise)." "No additions" means selecting a new goal g other than the goals g included in the empirical data. j "No addition" indicates no addition. "Random" indicates a random selection. "Future" indicates selection from the evaluation index f(x(t')) included in the second empirical data corresponding to the controlled time t' which is future to the controlled time t corresponding to the empirical data. "Future (Noise Added)" indicates adding noise to the evaluation index f(x(t')) included in the second empirical data corresponding to the future controlled time t'.

[0120] Target g included in display screen 30 j The input field for the number of selections represents the number of K-1 selections to be chosen from the evaluation indicators of the second set of empirical data.

[0121] The user inputs the desired target selection method and the number of targets to be selected by operating the UI unit 14 while viewing the display screen 30. The target setting unit 18E, according to the target selection method selected by the user via the display screen 30, selects K-1 targets g entered by the user. j You can select this from the evaluation indicators for the second set of experience data.

[0122] For example, consider a scenario where the user selects "No additions" via the display screen 30. In this case, the goal setting unit 18E selects only the goal g included in the M experience data. j This is selected as the target, and no other targets are selected. Therefore, in this case, K is forced to be 1.

[0123] Let's also consider a scenario where "random" is selected by the user. In this case, the goal setting unit 18E should randomly sample K-1 achieved evaluation metrics f(x(t')) from each of the second set of experience data, which are the experience data that constitute the surrounding experience data series corresponding to each of the M experience data (first experience data). Then, the goal setting unit 18E should add the sampled evaluation metrics f(x(t')) as the goal j (j=1,2,...,K-1).

[0124] Furthermore, we consider a scenario where "future" is selected by the user. In this case, the goal setting unit 18E should randomly sample K-1 evaluation indices f(x(t')) achieved at the control target time t' (second experience data) that is in the future from the M experience data (first experience data). Then, the goal setting unit 18E should add the sampled evaluation indices f(x(t')) as the goal j (j=1,2,...,K-1).

[0125] Furthermore, consider a scenario where "Future (Noise Added)" is selected by the user. In this case, the target setting unit 18E randomly samples K-1 evaluation indices f(x(t')) achieved at a future control target time t' (second experience data) from M experience data (first experience data). Then, the target setting unit 18E adds noise to the sampled evaluation indices f(x(t')) to create a noisy evaluation indice, which is then added as the target j (j=1,2,...,K-1). As for the noise, as mentioned above, noise generated according to a probability distribution such as a Gaussian distribution can be used.

[0126] Target g selected in target setting unit 18E j However, if the first set of experiential data consists entirely of items similar to target g, the contribution to improving learning efficiency decreases. However, by adding a noisy evaluation metric as target j, target g j This allows for an increase in the variety of values, thereby improving learning efficiency.

[0127] Furthermore, as mentioned above, the target g included in the display screen 30 j The input field for the number of selections represents the number of K-1 selections to be made from the evaluation indicators of the second set of experience data. The target setting unit 18E sets the number of K-1 targets g, which are the number of selections entered by the user via the display screen 30. j By selecting from the evaluation indicators of the second set of experiential data, K-1 target g j We just need to sample (j=1,2,...,K-1).

[0128] Returning to Figure 3, let's continue the explanation. Through the above processing by the target setting unit 18E, K targets g are set for each of the M experience data. j (j=0,1,2,...,K-1) is set.

[0129] The goal setting unit 18E sets K goals g for each of the M pieces of experience data. j (j=0,1,2,...,K-1), that is, MK targets g j Output this to the Experience Data Editorial Department 18D.

[0130] The experience data editing unit 18D, for each of the M experience data, selects the evaluation index f(x(t+1)) and speed information v(t+1) or speed information x(t+1)-x(t) contained in each of the M experience data, and K targets g received from the target setting unit 18E. j (j=0,1,2,...,K-1) is output to the corrected reward determination unit 18F.

[0131] The corrected reward determination unit 18F determines the evaluation index f(x(t+1)) and target g of the values ​​of indicators other than velocity derived from the observation information. jA corrected reward is determined by adjusting the reward for smaller errors, according to the velocity of the controlled point included in the observation information.

[0132] In detail, the corrected reward determination unit 18F determines a corrected reward that is adjusted so that the reward decreases as the speed of the controlled point increases.

[0133] In this embodiment, the correction reward determination unit 18F determines the K targets g set by the target setting unit 18E. j For each of the (j=0,1,2,...,K-1) values, an evaluation index f(x(t+1)) and K target g values ​​are derived from the acquired observational information. j The smaller the error with each of the values ​​(j=0,1,2,...,K-1), the higher the reward calculated. Then, the corrected reward determination unit 18F determines a corrected reward by correcting the calculated reward according to the velocity information of the controlled point included in the observation information.

[0134] Specifically, the corrected reward determination unit 18F receives evaluation index f(x(t+1)) and speed information v(t+1) or speed information x(t+1)-x(t) included in the experience data from the experience data editing unit 18D, and K targets g corresponding to the experience data. j The corrected reward is calculated using (j=0,1,2,...,K-1).

[0135] For calculating the corrected reward, the above formula (9) or an approximate calculation of the above formula (9), such as the above formulas (16) to (19), is used. The corrected reward determination unit 18F determines the target g j The d(x(t+1)) in these equations can be calculated using the L1 distance or L2 distance between each of the values ​​(j=0,1,2,...,K-1) and the evaluation index f(x(t+1)) included in the empirical data.

[0136] The adjustment reward determination unit 18F determines the adjustment reward by calculating the adjustment reward. Alternatively, the adjustment reward determination unit 18F may determine the adjustment reward by receiving the adjustment reward calculated by an external device or the like that calculates the adjustment reward.

[0137] The corrected reward determination unit 18F determines K target g for each M experience data. j Corrected reward r(s) for each of (j=0,1,2,...,K-1) t ,a t ,g j ) is output to the Experience Data Editorial Department 18D.

[0138] The experience data editing department 18D identifies the states s included in each of the M experience data. t , action a t , evaluation index f(x(t+1)), and speed information v(t+1) or speed information x(t+1)-x(t), and target g received from target setting unit 18E j The target g received from the correction reward determination unit 18F j Correction reward r(s) t ,a t ,g j ) and the edited experience data including are generated. That is, for one experience data, the experience data editing unit 18D replaces the target g with K target g. j The K targets g are set, j Correction reward r(s) corresponding to each t ,a t ,g j The system generates K edited experience data sets, each with further settings. The experience data editing unit 18D then outputs MK edited experience data sets generated from the M experience data sets to the learning unit 18B.

[0139] The learning unit 18B processes observational information and corrected reward r(s) t ,a t ,g j ) from control policy π(a t |s t Reinforce learning is performed on g).

[0140] In other words, the learning unit 18B uses MK edited empirical data received from the empirical data editing unit 18D to determine the control policy π(a t |s t ,g), value function V(s t ,g), and the action-value function Q(s t ,at Perform the process of updating g).

[0141] First, the learning unit 18B outputs the velocity information v(t+1) or velocity information x(t+1)-x(t) contained in each of the MK edited experience data received from the experience data editing unit 18D to the correction discount rate determination unit 18G.

[0142] The corrected discount rate determination unit 18G determines a corrected discount rate by correcting the discount rate γ of the corrected reward according to the velocity of the controlled point derived from the observation information. Specifically, the corrected discount rate determination unit 18G determines a corrected discount rate that is discounted more as the velocity of the controlled point increases (i.e., the value of the discount rate γ becomes smaller).

[0143] Specifically, the corrected discount rate determination unit 18G calculates the corrected discount rate as the power of the discount rate γ, where the speed information v(t+1) or speed information x(t+1)-x(t) is the exponent of the power. That is, the corrected discount rate determination unit 18G calculates the corrected discount rate at the controlled time t+1 using the following equation (21) or equation (22).

[0144]

number

[0145] The corrected discount rate determination unit 18G may determine a corrected discount rate obtained by correcting the discount rate corresponding to the input discount rate for the input speed that was received, according to the speed of the point to be controlled.

[0146] While users can directly input the discount rate by operating the UI unit 14, it is not intuitively clear how much the reward will be discounted. Therefore, it is preferable for the correction discount rate determination unit 18G to display a display screen on the UI unit 14 that allows for more intuitive setting of the input discount rate.

[0147] Figure 5A is a schematic diagram of an example of the display screen 32. The correction discount rate determination unit 18G displays the display screen 32 on the UI unit 14. The display screen 32 includes an input field for the distance traveled per unit time and an input field for the input discount rate for that distance (displayed as "Discount Rate" on the display screen 32). Figure 5A shows the input field for the distance traveled per unit time as an example of an input field for input speed. However, the display screen 32 may also have an input field for speed instead of the input field for the distance traveled per unit time. By providing input fields for input speed, such as the distance traveled per unit time or speed, along with the input discount rate on the display screen 32, users can see how much the reward is discounted relative to the speed, and thus input their desired discount rate for speed (distance traveled per unit time) more intuitively.

[0148] The user operates the UI unit 14 while viewing the display screen 32 to input the distance traveled per unit time and the input discount rate, which is the percentage by which the error and reward are discounted for that distance traveled.

[0149] Assume a scenario where the user inputs a travel distance X and a user-desired input discount rate G for the travel distance X through an operation instruction from the user to the UI unit 14.

[0150] In this case, the corrected discount rate determination unit 18G calculates the discount rate γ from the input discount rate G at the travel distance X using the following formula (23).

[0151]

number

[0152] The corrected discount rate determination unit 18G then calculates the corrected discount rate using the discount rate γ calculated by equation (23) and either equation (14) or equation (15). Based on these calculations, the corrected discount rate determination unit 18G determines the corrected discount rate.

[0153] Furthermore, for confirmation purposes, the correction discount rate determination unit 18G may display correspondence information on the UI unit 14 that represents the correspondence between the determined correction discount rate and the distance traveled per unit time.

[0154] Figure 5B is a schematic diagram of an example of the display screen 34. For example, the corrected discount rate determination unit 18G displays the display screen 34 on the UI unit 14. The display screen 34 includes a graph as correspondence information, which includes a line diagram DC that shows the correspondence between the corrected discount rate and the distance traveled. Note that the correspondence information does not need to be a graph; it can be any information that shows the correspondence between the corrected discount rate and the distance traveled.

[0155] Thus, the corrected discount rate determination unit 18G may determine a corrected discount rate by correcting the discount rate γ, which corresponds to the input discount rate for the input speed received from the user, according to the speed of the controlled point. When the conditions of the controlled device 20 change, such as the environment of an unmanned mobile body or robot, or the material used for laser welding, the appropriate discount rate is also expected to change. For this reason, by allowing the user to set and change the discount rate, the corrected discount rate determination unit 18G can determine a corrected discount rate according to the conditions of the controlled device 20.

[0156] Returning to Figure 3, let's continue the explanation. The corrected discount rate determination unit 18G outputs the calculated corrected discount rate to the learning unit 18B.

[0157] The learning unit 18B uses the MK edited empirical data received from the empirical data editing unit 18D and the corrected discount rate received from the corrected discount rate determination unit 18G to determine the control policy π(a t |s t ,g), value function V(s t ,g), and the action-value function Q(s t ,a t The process updates (learns) g).

[0158] When using a reinforcement learning algorithm called a policy-on type, the learning unit 18B may sample the aforementioned experience data when a certain number of experience data have been stored in the storage unit 16, or when the drone flight or welding is completed, and perform update processing using the edited experience data generated based on the experience data.

[0159] On the other hand, when using a reinforcement learning algorithm called an off-policy type, the learning unit 18B may sample a certain number of experience data from the storage unit 16 each time or every few times, and perform update processing using edited experience data generated based on the experience data. In the case of an off-policy type, experience data is stored in the storage unit 16 until the maximum number of experience data is reached, and if the maximum is exceeded, the oldest experience data may be discarded.

[0160] The learning unit 18B determines the control policy π(a t |s t ,g), value function V(s t ,g), and the action-value function Q(s t ,a t Any reinforcement learning algorithm can be used to update ,g). In this embodiment, it is preferable that the learning unit 18B performs these update processes using the corrected discount rate received from the corrected discount rate determination unit 18G instead of the discount rate γ. For example, the value function V(s t ,g) and the action-value function Q(s t ,a t When learning at least one of ,g), the learning unit 18B uses equations (2) and (3) above to perform the value function V(s t ,g) and the action-value function Q(s t ,a t You just need to update g).

[0161] The learning unit 18B should process according to the reinforcement learning algorithm being used, except that it uses a corrected discount rate instead of the discount rate γ.

[0162] Next, an example of the information processing flow performed by the machine learning device 10 of this embodiment will be described.

[0163] Figure 6 is a flowchart showing an example of the information processing flow performed by the machine learning device 10 of this embodiment.

[0164] The acquisition unit 18A acquires observation information for the controlled time t from the controlled device 20 (step S100).

[0165] The learning unit 18B uses the observation information acquired in step S100 to determine the state s t And calculate target g, and action a t The action a determined in step S102 is determined (step S102). The output unit 18C outputs the action a determined in step S102. t This is converted into control information and output to the controlled device 200 (step S104).

[0166] The learning unit 18B stores data corresponding to the observation information of the controlled time t acquired in step S100 in the storage unit 16 (step S106). As described above, the empirical data includes the target g, the achieved evaluation index f(x(t)), the velocity information of the controlled point, which is velocity v(t) or velocity (velocity expressed as the distance traveled per unit time with one control cycle as the unit) x(t)-x(t-1), and the state s one control time before. t-1 And, action a t-1 This includes,

[0167] The learning unit 18B determines whether or not it is time to perform learning (step S108). That is, the learning unit 18B determines the control policy π(a t |s t ,g), value function V(s t ,g), and the action-value function Q(s t ,a t The learning unit 18B determines whether it is time to perform the update process in step (g). For example, the learning unit 18B performs the update process at regular intervals of the controlled time. Therefore, the learning unit 18B makes the decision in step S108 by determining whether a certain period of controlled time has elapsed since the last learning execution.

[0168] If the decision in step S108 is negative (Step S108: No), the process proceeds to step S110. In step S110, the control unit 18 determines whether or not to terminate the process (Step S110). If the decision in step S110 is positive (Step S110: Yes), the routine terminates. If the decision in step S110 is negative (Step S110: No), the process returns to step S100.

[0169] On the other hand, if the learning unit 18B determines that it is time to perform learning (step S108: Yes), the process proceeds to step S112.

[0170] In step S112, the experience data editing unit 18D acquires M pieces of experience data by randomly sampling a fixed number (M pieces) of experience data from the storage unit 16 (step S112). The experience data editing unit 18D outputs the sampled M pieces of experience data and the surrounding experience data sequences corresponding to each of the M pieces of experience data to the target setting unit 18E.

[0171] The goal setting unit 18E, based on the group of first experience data, which consists of M pieces of experience data acquired in step S112, and second experience data, which constitutes a series of surrounding experience data corresponding to each of the M pieces of experience data, selects K evaluation indicators from a plurality of evaluation indicators included in the group, and sets K goals g j Set (j=0,1,2,...,K-1) (Step S114).

[0172] The experience data editing unit 18D, for each of the M experience data, selects the evaluation index f(x(t+1)) and speed information v(t+1) or speed information x(t+1)-x(t) contained in each of the M experience data, and K targets g received from the target setting unit 18E. j (j=0,1,2,...,K-1) is output to the corrected reward determination unit 18F.

[0173] The corrected reward determination unit 18F uses the evaluation index f(x(t+1)) and speed information v(t+1) or speed information x(t+1)-x(t) included in the experience data acquired in step S112, and the K target g set in step 114 corresponding to the experience data. j Using (j=0,1,2,...,K-1), the corrected reward r(s t ,a t ,g j ) is determined (step S116).

[0174] The experience data editing department 18D identifies the states s included in each of the M experience data. t , action a t , evaluation index f(x(t+1)), and speed information v(t+1) or speed information x(t+1)-x(t), and target g received from target setting unit 18E j The target g received from the correction reward determination unit 18F j Correction reward r(s) t ,a t ,g j The system generates edited experience data including ) and . The experience data editing unit 18D outputs MK edited experience data generated from M experience data to the learning unit 18B. The learning unit 18B outputs the velocity information v(t+1) or velocity information x(t+1)-x(t) contained in each of the MK edited experience data received from the experience data editing unit 18D to the correction discount rate determination unit 18G.

[0175] The corrected discount rate determination unit 18G determines the corrected discount rate as a power of the discount rate γ, where the speed information v(t+1) or speed information x(t+1)-x(t) received from the learning unit 18B is the exponent of the power (step S118).

[0176] The learning unit 18B uses the MK edited empirical data received from the empirical data editing unit 18D and the corrected discount rate received from the corrected discount rate determination unit 18G to determine the control policy π(a t |s t ,g), value function V(s t ,g), and the action-value function Q(s t ,at The process of updating (learning) g) is performed (step S120). Then, proceed to step S110 above.

[0177] As described above, the machine learning device 10 of this embodiment comprises an acquisition unit 18A, an output unit 18C, a corrected reward determination unit 18F, and a learning unit 18B. The acquisition unit 18A acquires observation information including information about the velocity of the control target point at the control target time. The output unit 18C outputs control information including information about velocity control of the control target point, which is determined according to the observation information and the control policy. The corrected reward determination unit 18F determines a corrected reward, which is calculated by correcting the reward for smaller errors between the evaluation index of the values ​​of indicators other than velocity derived from the observation information and the target, according to the velocity of the control target point included in the observation information. The learning unit 18B reinforces the control policy from the observation information and the corrected reward.

[0178] Thus, the machine learning device 10 of this embodiment reinforces the control policy using a corrected reward, which is a reward adjusted according to the velocity of the control target point included in the observation information. By using a corrected reward adjusted according to velocity, the machine learning device 10 of this embodiment can learn a control policy that minimizes the average error of the control target point relative to the target, including velocity control.

[0179] Therefore, the machine learning device 10 of this embodiment can minimize the average error of the controlled point with respect to the target, including speed control.

[0180] Furthermore, the machine learning device 10 of this embodiment further reinforces the control policy by using a corrected discount rate, which is obtained by correcting the reward discount rate according to the distance moved by the controlled point, instead of the reward discount rate. By using a corrected discount rate, the machine learning device 10 of this embodiment can ensure that changes in speed do not affect the value of the discounted cumulative reward, and can further learn a control policy that minimizes the average error.

[0181] Furthermore, the machine learning device 10 of this embodiment also includes a target setting unit 18E. Based on a group of first empirical data including evaluation indicators derived from acquired observation information, and one or more second empirical data each including evaluation indicators derived from one or more other observation information with different control target times than the said observation information, the target setting unit 18E selects K evaluation indicators from the group of evaluation indicators to set K targets g j Set it as follows.

[0182] Then, the corrected reward determination unit 18F determines the set K targets g j For each of these, an evaluation index and K target g are derived from the acquired observational information. j The smaller the error between each of these values, the higher the reward, and a corrected reward is determined by adjusting for the speed of the controlled point. Then, the learning unit 18B receives the target g from the target setting unit 18E and determines a corrected reward. j The target g received from the correction reward determination unit 18F j Correction reward r(s) t ,a t ,g j Reinforcement learning is performed based on edited experience data including ).

[0183] Therefore, the learning unit 18B performs action a t When determining the target g, not only the target g that was set, but also the evaluation indicator that was ultimately achieved was used as the target g. j Edited experience data can be used for training. Therefore, in addition to the above effects, the machine learning device 10 of this embodiment can significantly improve learning efficiency.

[0184] (modified version) In the above embodiment, a configuration in which the control unit 18 comprises a target setting unit 18E and a corrected discount rate determination unit 18G was described as an example. However, the control unit 18 may also be configured without comprising at least one of the target setting unit 18E and the corrected discount rate determination unit 18G.

[0185] In the case of a configuration without a target setting unit 18E, the corrected reward determination unit 18F determines K targets g jInstead of (j = 0, 1, 2, ..., K - 1), the correction reward may be determined for each of the M pieces of empirical data by using the target g included in each of the M pieces of empirical data acquired by the empirical data editing unit 18D.

[0186] Also, in the case of a configuration not provided with the correction discount rate determination unit 18G, the learning unit 18B uses the discount rate γ, which is the discount rate before correction, instead of the correction discount rate, for the control policy π(a t |s t , g), the value function V(s t , g), and the action value function Q(s t , a t , g) to perform an update (learning) process. Specifically, in this case, the learning unit 18B may use the above equations (2) and (3) for the update of the value function V(s t , g) and the action value function Q(s t , a t , g). This is a method of approximating by substituting the above equations (12) and (13) with the above equations (2) and (3) respectively, and is effective when the change in speed is small. This method has the advantage that existing reinforcement learning processes can be applied as they are.

[0187] Next, an example of the hardware configuration of the machine learning device 10 of the above embodiment will be described.

[0188] FIG. 7 is a hardware configuration diagram of an example of the machine learning device 10 of the above embodiment.

[0189] The machine learning device 10 of the above embodiment includes a control device such as a CPU (Central Processing Unit) 90B, a storage device such as a ROM (Read Only Memory) 90C, a RAM (Random Access Memory) 90D, and a HDD (hard disk drive) 90E, an I / F unit 90A that is an interface with various devices, and a bus 90F that connects each unit, and has a hardware configuration using a normal computer.

[0190] In the machine learning device 10 of the above embodiment, each of the above components is realized on a computer by the CPU 90B reading a program from ROM 90C onto RAM 90D and executing it.

[0191] The program for executing each of the above processes performed by the machine learning device 10 of the above embodiment may be stored in the HDD 90E. Alternatively, the program for executing each of the above processes performed by the machine learning device 10 of the above embodiment may be pre-installed and provided in the ROM 90C.

[0192] Furthermore, the program for executing the above-described process performed by the machine learning device 10 of the above embodiment may be provided as a computer program product by being stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD). Alternatively, the program for executing the above-described process performed by the machine learning device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the program for executing the above-described process performed by the machine learning device 10 of the above embodiment may be provided or distributed via a network such as the Internet.

[0193] Although embodiments of the present invention have been described above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. This novel embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0194] 10 Machine Learning Devices 14 UI section 18A Acquisition Department 18B Learning Department 18C Output Section 18E Goal Setting Department 18F Compensation compensation determination department 18G Correction Discount Rate Determination Unit 20 Controlled Devices

Claims

1. An acquisition step to acquire observational information including information about the velocity of the controlled point at the controlled time, An output step that outputs control information including information regarding the velocity control of the control target point, determined according to the observation information and control strategy, A corrected reward determination step in which a corrected reward is determined by correcting the value of an indicator other than velocity derived from the observation information, and the smaller the error between the evaluation index and the target, the higher the reward, according to the velocity of the controlled point included in the observation information. A learning step in which the control strategy is reinforced and learned from the observation information and the correction reward, A machine learning program that instructs a computer to perform a task.

2. The aforementioned step of determining the corrected reward is: A corrected reward is determined by correcting the aforementioned reward so that it becomes lower as the speed of the controlled point increases. The machine learning program according to claim 1.

3. The system includes a step to determine a corrected discount rate, which is obtained by correcting the discount rate of the corrected reward according to the velocity of the controlled point derived from the observation information. The aforementioned learning steps are: The control policy is reinforced and learned from the observation information, the corrected reward, and the corrected discount rate. The machine learning program according to claim 1.

4. The aforementioned step of determining the corrected discount rate is: The corrected discount rate is determined such that the value of the discount rate decreases as the speed of the controlled point increases. The machine learning program according to claim 3.

5. The aforementioned step of determining the corrected discount rate is: The discount rate corresponding to the input discount rate for the input speed at which the input was received is corrected according to the speed of the controlled point to determine the corrected discount rate. The machine learning program according to claim 3.

6. The goal setting step includes setting a group of evaluation indicators selected from a group of evaluation indicators included in the first group, based on a group of first empirical data including the evaluation indicators derived from the acquired observation information, and a group of one or more second empirical data each including the evaluation indicators derived from one or more other observation information with different control target times than the said observation information, as the goal. The aforementioned step of determining the corrected reward is: For each of the set targets, a corrected reward is determined, which is calculated by adjusting the reward according to the velocity of the controlled point included in the observation information, with a higher reward being given the smaller the error between the evaluation index derived from the acquired observation information and each of the targets. The machine learning program according to claim 1.

7. The aforementioned goal-setting step is, The evaluation index included in the first experience data and the noise-added evaluation index, which is the evaluation index included in the second experience data with added noise, are set as the target. The machine learning program according to claim 6.

8. The aforementioned goal-setting step is, The goal is set according to the goal selection method selected by the user. The machine learning program according to claim 6.

9. The aforementioned goal-setting step is, Set the number of targets selected by the user. The machine learning program according to claim 6.

10. An acquisition step to acquire observational information including information about the velocity of the controlled point at the controlled time, An output step that outputs control information including information regarding the velocity control of the control target point, determined according to the observation information and control strategy, A corrected reward determination step in which a corrected reward is determined by correcting the value of an indicator other than velocity derived from the observation information, and the smaller the error between the evaluation index and the target, the higher the reward, according to the velocity of the controlled point included in the observation information. A learning step in which the control strategy is reinforced and learned from the observation information and the correction reward, Machine learning methods including

11. An acquisition unit that acquires observation information including information regarding the velocity of the controlled point at the controlled time, An output unit that outputs control information including information regarding the velocity control of the control target point, determined according to the observation information and control strategy, A corrected reward determination unit determines a corrected reward that is adjusted according to the velocity of the controlled point included in the observation information, with a higher reward for smaller errors between the evaluation index of the values ​​of indicators other than velocity derived from the observation information and the target, and a corrected reward for smaller errors. A learning unit that reinforces the control strategy from the observation information and the correction reward, A machine learning device equipped with the following features.

Images