Machine learning program, machine learning method, and machine learning device

The machine learning program addresses the challenge of minimizing average error in control target points by using corrected rewards and discount rates to adapt to speed changes, optimizing the control policy for improved trajectory accuracy.

JP2024162483AActive Publication Date: 2024-11-21KK TOSHIBA

Patent Information

Application Number
JP2023078022
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-10
Publication Date
2024-11-21
Estimated Expiration
2043-05-10

AI Technical Summary

Technical Problem

Conventional reinforcement learning methods struggle to minimize the average error of control target points, including speed control, due to changes in speed affecting the discount cumulative error, making it difficult to optimize the trajectory of control targets.

Method used

A machine learning program that performs reinforcement learning using corrected rewards and discount rates adjusted according to the speed of the control target point, minimizing the average error by integrating speed effects into the learning process.

Benefits of technology

The solution enables the learning of a control policy that effectively minimizes the average error of control target points, including speed control, by accounting for speed changes and optimizing the trajectory.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162483000001_ABST
    Figure 2024162483000001_ABST
Patent Text Reader

Abstract

To minimize an average error of a control target point including speed control with respect to a goal.SOLUTION: A machine learning program causes a computer to execute: an acquisition step of acquiring observation information including information on a speed of a control target point at a control target time; an output step of outputting control information including information on speed control of the control target point, the control information being determined in accordance with the observation information and a control policy; a corrected reward determination step of determining a corrected reward obtained by correcting a reward in accordance with a speed of the control target point included in the observation information, the reward being higher as an error between a value of an evaluation index and a goal is smaller, the evaluation index being a value of an index other than a speed derived from the observation information; and a learning step of performing reinforcement learning of the control policy based on the observation information and the corrected reward.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] An embodiment of the present invention relates to a machine learning program, a machine learning method, and a machine learning device. [Background technology]

[0002] Attempts have been made to apply reinforcement learning to various control learning. Patent Document 1 discloses a method of learning speed control so as to minimize deviation from a command path by calculating a reward based on deviation from the command path and performing reinforcement learning. Non-Patent Document 1 discloses a method of learning welding control including welding speed by calculating a reward based on the difference between a desired bead width and a generated bead width in laser welding. Non-Patent Document 2 discloses a method of learning a control policy by replacing the target of accumulated experience data and using it for learning when controlling a system to meet a given target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6077617 [Non-patent literature]

[0004] [Non-Patent Document 1] M. Schmitz, F. Pinsker, A. Ruhri, B. Jiang and G. Safronov, “Enabling Rewards for Reinforcement Learning in Laser Beam Welding processes through Deep Learning,” 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 14-17 December, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] Reinforcement learning is a method for learning a strategy to maximize the expected value of the discounted cumulative reward. The discounted cumulative reward is the sum of rewards obtained after the current time, multiplied by a smaller weight as the time difference from the current time increases. If reinforcement learning is performed using a reward calculated based on the error as in Patent Document 1 and Non-Patent Document 1, a control method that reduces the error can be learned. However, when the speed of the control target point changes, the moving distance per unit time varies depending on the speed, so that not only the error calculated by the trajectory but also the discounted cumulative error varies depending on the speed. For this reason, in the conventional technology, it was difficult to minimize the average error of the trajectory of the control target point, including the speed control, with respect to the target trajectory.

[0006] The problem to be solved by the present invention is to provide a machine learning program, a machine learning method, and a machine learning device that can minimize the average error of a control target point, including speed control, with respect to a target. [Means for solving the problem]

[0007] The machine learning program of the embodiment is a machine learning program for causing a computer to execute the following steps: an acquisition step of acquiring observation information including information regarding the speed of the control point at the control time; an output step of outputting control information including information regarding speed control of the control point, determined according to the observation information and a control strategy; a corrected reward determination step of determining a corrected reward, corrected according to the speed of the control point included in the observation information, such that the smaller the error between an evaluation index of the value of an index other than the speed derived from the observation information and a target, the higher the reward; and a learning step of reinforcement learning the control strategy from the observation information and the corrected reward. [Brief description of the drawings]

[0008] [Figure 1] Schematic diagram of the learning system. [Diagram 2]13 is a diagram showing the correspondence between the target evaluation index at the position of a control point and the evaluation index that is actually achieved. [Diagram 3] FIG. 1 is a functional block diagram of a machine learning device. [Figure 4] FIG. [Figure 5A] FIG. [Figure 5B] FIG. [Figure 6] 1 is a flowchart showing the flow of information processing. [Figure 7] Hardware. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The machine learning program, the machine learning method, and the machine learning device of the present embodiment will be described in detail below with reference to the accompanying drawings.

[0010] FIG. 1 is a schematic diagram of an example of a learning system 1 according to the present embodiment.

[0011] The learning system 1 includes a machine learning device 10 and a control target device 20. The machine learning device 10 and the control target device 20 are connected to each other so as to be able to communicate with each other.

[0012] The machine learning device 10 is an information processing device that performs reinforcement learning. In other words, the machine learning device 10 is an agent that is the subject of learning. The machine learning device 10 is a computer for executing the machine learning program of this embodiment.

[0013] The control target device 20 is an object to be controlled by the machine learning device 10. In other words, the control target device 20 is an application target of control information determined in accordance with the control strategy learned by the machine learning device 10.

[0014] The controlled device 20 is, for example, a robot such as a Cartesian coordinate robot or an articulated robot, a machine tool for laser processing or laser welding, an unmanned mobile object such as an unmanned carrier or a drone, etc. The controlled device 20 may be a computer simulator that simulates the operation of these devices.

[0015] The machine learning device 10 learns a control policy such that the control object points controlled by the control target device 20 achieve a target in the evaluation index. In other words, the machine learning device 10 learns a control policy that minimizes the average error of the control object points with respect to the target.

[0016] The control target point is a point that is the object of control at each of successive control target times along a time series. If the control target device 20 is a robot, the control target point is, for example, the tip of a robot arm or a specific position of an end effector. If the control target device 20 is a machine tool for laser processing or laser welding, for example, the control target point is, for example, a laser irradiation point during laser processing. If the control target device 20 is an unmanned mobile object such as an unmanned carrier or drone, the control target point is, for example, the center of gravity of the unmanned mobile object.

[0017] In reinforcement learning, learning of the machine learning device 10 progresses through interactions between the machine learning device 10, which is the subject of learning, and the control target device 20, which is the control target.

[0018] Specifically, the control target device 20 outputs observation information of the control target point at each control target time to the machine learning device 10. The machine learning device 10 determines control information representing an action according to the observation information and control measure acquired from the control target device 20, and outputs the control information to the control target device 20. The learning of the machine learning device 10 progresses as this series of processing steps is repeated.

[0019] The observation information is information that indicates the state of the control target point at the control target time, and is information necessary for controlling the control target device 20. In this embodiment, the observation information includes at least information related to the speed of the control target point at the control target time.

[0020] The information on the velocity of the control point may be any information that can identify the velocity of the control point at the control time. More specifically, the information on the velocity of the control point is information that indicates at least one of the position, velocity, acceleration, and moving distance per unit time of the control point at the control time.

[0021] The control information is information used to control the behavior of the control point. In this embodiment, the control information includes at least information related to speed control of the control point.

[0022] Specifically, when the controlled device 20 is a drone, the control information is the speed or acceleration in each of the forward, backward, left, right, up, and down directions, and the observation information is information necessary for controlling the drone, such as the drone's position, speed, and surrounding information. The surrounding information is, for example, an image of the surroundings taken by a camera, a range image, an occupancy grid map, and the like.

[0023] When the controlled device 20 is a multi-joint robot, the control information is the torque and angle of each joint, and the position, posture, and speed of the point to be controlled. The observation information is information necessary for controlling the multi-joint robot, such as the angle and angular velocity of each joint, the position, posture, and speed of the point to be controlled, and information on the working environment. Information on the working environment is, for example, an image of the surroundings taken by a camera, a range image, etc.

[0024] When the controlled device 20 is a laser welding machine, the control information includes the welding speed, welding acceleration, laser power, spot diameter, etc. The observation information includes the laser irradiation position, irradiation speed, spot diameter, gap between materials, width of the bead or molten pool, information about the periphery of the welding position, etc. The information about the periphery of the welding position includes, for example, an image of the periphery of the welding position captured by a camera, temperature distribution, etc.

[0025] Next, a basic concept of reinforcement learning will be described. In this embodiment, a form of reinforcement learning using goal-conditional reinforcement learning will be described.

[0026] Goal-conditional reinforcement learning is a method to determine the state s input at a certain control object time t when a goal g is given. t From, Action a t This is a method for learning a control strategy that determines

[0027] Status t corresponds to the observed information at the time t to be controlled or a part of it. t corresponds to control information.

[0028] The control strategy is π(a t |s t ,g) is the probability distribution represented by the control strategy π(a t |s t , g) are trained, for example, with a neural network that outputs probability values ​​or parameters of a probabilistic model.

[0029] Goal-conditional reinforcement learning is a control policy π(a t |s t The purpose of this learning is to learn (g). The discounted cumulative reward is the sum of the rewards obtained after the current time, with smaller weights applied to the rewards obtained after the current time, the greater the time difference from the current time.

[0030]

number

[0031] In equation (1), r(s t ,a t ,g) is the state s given a goal g t Action a t represents the reward calculated at time t+1 as a result of performing the above. In formula (1), γ represents the discount rate. k is an integer equal to or greater than 0.

[0032] The discount rate γ is a parameter between 0 and 1 that adjusts how much future rewards are taken into consideration when deciding on actions. In other words, the discount rate γ is a hyperparameter that adjusts how far into the future to consider. The discount rate γ is a parameter that is used to discount rewards that are obtained further into the future. The discount rate γ also plays a role in regularization to stabilize learning.

[0033] There are many known algorithms for reinforcement learning. Most of them use a value function V(s t ,g) and the action value function Q(s t ,a t ,g) learning steps.

[0034] Value function V(s t ,g) is the state s given a goal g. t to the current control strategy π(a t |s t ,g) according to the value function V(s t When a method called TD (Temporal Difference) learning is used, the values ​​of (g) are updated (learned) by the update formula represented by the following formula (2).

[0035]

number

[0036] In equation (2), α represents the learning rate.

[0037] Action value function Q(s t ,a t ,g) is the state s given a goal g. t Action a t After taking the current control strategy π(a t |s t The action value function Q(s t ,at In TD learning, the values ​​of (g) are updated (learned) by the update equation expressed by the following equation (3).

[0038]

number

[0039] In formula (3), formula (4) below is generally difficult to calculate.

[0040]

number

[0041] For this reason, we replace equation (4) in equation (3) with the value function V(s t ,g) or control strategy π(a|s t+1 , g) is the action value function Q(s t+1 ,a,g) can be used.

[0042] Value function V(s t ,g) and the action value function Q(s t ,a t ,g) are trained, for example, using a linear model or a neural network.

[0043] Next, we explain a method for learning a control policy that enables a controlled object to achieve a goal in an evaluation index, using goal-conditioned reinforcement learning.

[0044] FIG. 2 is a diagram showing an example of the correspondence relationship between a target g of an evaluation index at a position x of a control target point and an actually achieved evaluation index f(x).

[0045] Position x is the distance from the reference position on the trajectory of the controlled points that is planned in advance. The trajectory of the controlled points is the trajectory of the controlled points at each of the controlled time periods that are planned in advance. If the trajectory of the controlled points that is planned is a straight line, the reference position can be used as the origin and the planned trajectory can be used as the x-axis.

[0046] The target is the target value of the evaluation index f(x).

[0047] The evaluation index f(x) is a value of an index other than the speed derived from the observation information, and is an index that has actually been achieved. Derive means any of calculation, calculation, determination, identification, and reading. As described above, the observation information includes, for example, information necessary for controlling a drone, such as the drone's position, speed, and surrounding information, information necessary for controlling a multi-joint robot, such as the angle and angular velocity of each joint, the position, attitude, and speed of a control target point, and information on the work environment, depending on the type of the controlled device 20, welding speed, welding acceleration, laser power, spot diameter, etc. The observation information includes information necessary for controlling a laser welding machine, such as the laser irradiation position, irradiation speed, spot diameter, gap between materials, width of a bead or molten pool, and information around the welding position. The evaluation index f(x) is a value of an index other than the speed among these indexes derived from the observation information.

[0048] Specifically, for example, in the case of drone control, the evaluation index f(x) is the distance between a preset trajectory, which is the trajectory of the control point set in advance, and the current position of the control point, and the distance 0 (zero) is given as the target g. In the case of laser welding control, the bead width or penetration depth is given as the evaluation index f(x), and a positive constant is given as the target g.

[0049] In goal-conditioned reinforcement learning, reinforcement learning is performed to achieve a goal. For this reason, the smaller the error d(x) between the goal g and the actually achieved evaluation index f(x), the larger the reward r(s t ,a t The reward is defined so that the error d(x) is given. For example, the L1 distance or the L2 distance can be used as the error d(x). In goal-conditional reinforcement learning, the error d(x) is integrated between time t and t+1, multiplied by -1, and the reward r(s t ,a t ,g) are defined and learned.

[0050]

number

[0051] In equation (5), x(t) represents the position of the control target point at time t, which is the same as the control target time t.

[0052] The reward defined by equation (5) is the control policy π(a t |s t ,g) learning.

[0053]

number

[0054] Equation (6) is an objective function that represents the average error of the control point with respect to the target. In detail, the average error represented by equation (6) represents the integral value obtained by integrating the error d(x) between the target trajectory, which is the trajectory of the control point scheduled in advance, and the actual trajectory of the control point, along the target trajectory.

[0055] In addition, as an approximation of the definition of reward expressed by equation (5), we use only the error d(x) at time t+1 to calculate reward r(s t ,a t ,g) to r(s t ,a t ,g)=d(x(t+1)). The reward defined by this formula is the control policy π(a t |s t This corresponds to learning (g).

[0056]

number

[0057] In controlling a control target device 20 such as a drone, when it is desired to minimize the difference between a pre-planned trajectory of a control target point and an actual trajectory, the objective function introducing a discount rate γ for learning by reinforcement learning is expressed by the following equation (8).

[0058]

number

[0059] Equation (8) is an objective function that represents the average error of the control point with respect to the target. In detail, the average error represented by equation (8) represents the integral value obtained by integrating the error d(x) between the target trajectory, which is the trajectory of the control point scheduled in advance, and the actual trajectory of the control point, along the target trajectory.

[0060] In the controlled device 20 for laser welding or the like, when it is desired to set the bead width or penetration depth to a predetermined value, the formula (8) is also the objective function.

[0061] Here, when the speed of the control target point is constant, minimization of the expected value of the objective function expressed by the above formula (6) is the same as minimization of the objective function expressed by the above formula (8). However, when the speed of the control target point is not constant, minimization of the expected value of the objective function expressed by the above formula (6) differs from minimization of the objective function expressed by the above formula (8). In detail, for example, when the value of the error d(x) is large, the deviation from the target g is larger when the speed is high than when the speed is low, and the influence on formula (8) is large. However, since formula (6) integrates the error d(x) with respect to time, the influence of the speed is not taken into consideration.

[0062] For this reason, in conventional reinforcement learning, when learning a control policy for a control point including speed control by reinforcement learning, it was not possible to perform reinforcement learning that optimizes the original objective function, Equation (8). In other words, in conventional reinforcement learning, when learning a control policy for a control point including speed control by reinforcement learning, it was difficult to minimize the average error of the control point with respect to the target.

[0063] Therefore, the machine learning device 10 of this embodiment performs reinforcement learning of a control policy using a corrected reward obtained by correcting the reward according to the speed of the control target point included in the observation information. By using the corrected reward, the machine learning device 10 of this embodiment can prevent the change in speed from affecting the value of the average error, and can learn a control policy that minimizes the average error.

[0064] Furthermore, the machine learning device 10 of this embodiment further performs reinforcement learning of the control policy using a corrected discount rate obtained by correcting the discount rate of the reward according to the moving distance of the control target point, instead of the discount rate of the reward. By using the corrected discount rate, the machine learning device 10 of this embodiment can prevent a change in speed from affecting the value of the discounted cumulative reward, and can learn a control policy that minimizes the average error.

[0065] That is, the machine learning device 10 of this embodiment provides a method for learning a control policy that minimizes the average error expressed by the above formula (8) while taking into account the speed effect.

[0066] Therefore, in this embodiment, the reward is defined by the following formula (9).

[0067]

number

[0068] With the reward expressed by the above formula (9), the above formula (8) can be expressed by the following formula (10).

[0069]

number

[0070] As described above, goal-conditional reinforcement learning maximizes the expected value of the discounted cumulative reward expressed by the above equation (1). t |s t, g). Therefore, in order to maximize the expected value of the discounted cumulative reward expressed by the above formula (10), the discount rate γ k needs to be replaced with the following equation (11).

[0071]

number

[0072] Therefore, in this embodiment, the value function V(s t ,g) TD learning is determined by the update equation expressed by the following equation (12).

[0073]

number

[0074] In this embodiment, the action value function Q(s t ,a t ,g) TD learning is determined by the update equation expressed by the following equation (13).

[0075]

number

[0076] That is, in this embodiment, the value function V(s t , g), and the action value function Q(s t ,a t Instead of the discount rate γ in the above equation (3), which is the update equation for (g), a corrected discount rate obtained by correcting the discount rate γ by the speed is used. The corrected discount rate is expressed by the following equation (14).

[0077]

number

[0078] In formula (14), x (t+1) -x (t)is the velocity v(t) at time t and the control period δ T and v(t)δ T That is, the adjusted discount rate can be expressed by the following formula (15).

[0079]

number

[0080] In addition, in this embodiment, the above formula (9) or an approximation of the above formula (9), for example, the following formulas (16) to (19), etc., are used as a corrected reward obtained by correcting the reward according to the speed of the control target point included in the observation information.

[0081]

number

[0082] That is, in the machine learning device 10 of this embodiment, reinforcement learning of a control policy is performed using a corrected reward obtained by correcting the reward according to the speed of the control target point included in the observation information. By using the corrected reward, the machine learning device 10 of this embodiment can learn a control policy that minimizes the average error.

[0083] Furthermore, the machine learning device 10 of this embodiment further performs reinforcement learning of the control policy using a corrected discount rate obtained by correcting the discount rate of the reward according to the moving distance of the control target point, instead of the discount rate of the reward. By using the corrected discount rate, the machine learning device 10 of this embodiment can prevent a change in speed from affecting the value of the discounted cumulative reward, and can further learn a control policy that minimizes the average error.

[0084] Here, the problem with goal-conditional reinforcement learning is that it is difficult to obtain an action sequence that achieves the goal g in the early stages of learning, and so a large number of action searches are required for learning. Therefore, in the machine learning device 10 of this embodiment, in order to further increase the efficiency of goal-conditional reinforcement learning, the goal of an action sequence that did not achieve the goal is replaced with another action sequence that achieved the goal, and used for learning. For example, the action sequence a0, a1, a2,..., a t As a result of executing the above, unfortunately, goal g is not achieved and another goal g' is achieved. In this case, the action sequence a0, a1, a2,...a t is a failure example, but if the goal is replaced with another goal g', it can be used as a success example for learning. In this way, instead of simply learning a failure example as a failure example, it is possible to create a success example by replacing the goal and use it for learning, thereby making learning more efficient.

[0085] For this reason, in the machine learning device 10 of this embodiment, by setting multiple evaluation indexes as targets, it is possible to efficiently learn a control policy.

[0086] Next, the configuration of the machine learning device 10 in this embodiment will be described in detail.

[0087] FIG. 3 is a functional block diagram of an example of the machine learning device 10 of this embodiment.

[0088] The machine learning device 10 includes a communication unit 12, a UI (user interface) unit 14, and a storage unit 16. The communication unit 12, the UI unit 14, the storage unit 16, and the control unit 18 are communicatively connected via a bus 19 or the like.

[0089] The communication unit 12 communicates with an external information processing device such as the control target device 20 via a network or the like. The UI unit 14 has a display function and an input function. The display function displays various information. The display function is, for example, a display, a projection device, etc. The input function accepts operation input by the user. The input function is, for example, a pointing device such as a mouse and a touch pad, a keyboard, etc. The display function and the input function may be integrated into a touch panel. The memory unit 16 stores various information.

[0090] The UI unit 14 and the storage unit 16 may be configured to be communicably connected to the control unit 18 via a wired or wireless connection. At least one of the UI unit 14 and the storage unit 16 may be connected to the control unit 18 via a network or the like.

[0091] At least one of the UI unit 14 and the storage unit 16 may be provided outside the machine learning device 10. At least one of the UI unit 14, the storage unit 16, and one or more functional units included in the control unit 18 may be mounted on an external information processing device communicatively connected to the machine learning device 10 via a network or the like.

[0092] The control unit 18 executes information processing in the machine learning device 10. The control unit 18 includes an acquisition unit 18A, a learning unit 18B, an output unit 18C, an experience data editing unit 18D, a goal setting unit 18E, a corrected remuneration determination unit 18F, and a corrected discount rate determination unit 18G.

[0093] The acquisition unit 18A, the learning unit 18B, the output unit 18C, the experience data editing unit 18D, the goal setting unit 18E, the corrected remuneration determination unit 18F, and the corrected discount rate determination unit 18G are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (tral processing unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC, that is, by hardware. Each of the above units may be realized by using both software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0094] The acquisition unit 18A acquires observation information. As described above, the observation information is information that represents the state of the control target point at the control target time t, and includes information on the speed of the control target point at the control target time t. The observation information also includes a target g for the evaluation index. The acquisition unit 18A sequentially acquires the observation information that is sequentially output from the control target device 20 for each control target time t. The acquisition unit 18A outputs the acquired observation information to the learning unit 18B every time the acquisition unit 18A acquires the observation information for the control target time t.

[0095] The learning unit 18B performs processing such as extraction of some data, scaling, and clipping on the observation information at the control target time t received from the acquisition unit 18A, and converts the observation information into a state s t When the observation information includes an image, the learning unit 18B may perform image processing or image recognition processing.

[0096] Next, the learning unit 18B applies the current control measure π(a t |s t ,g) to perform action a t Determine.

[0097] In detail, the learning unit 18B extracts a target g from the observation information to obtain a control policy π(a t |s tThen, the learning unit 18B identifies a control policy π(a t |s t ,g) according to action a t The learning unit 18B samples the action a with the highest probability. t The state s t and direct action a from goal g t The control policy π(a t |s t , g) is configured, the learning unit 18B t and the target g and the control strategy π(a t |s t ,g) and direct action a t In addition, the learning unit 18B may determine the control strategy π(a t |s t ,g) without using random action a t may be sampled.

[0098] The learning unit 18B determines the behavior a determined by these processes. t is output to output unit 18C.

[0099] The output unit 18C outputs control information including information on the speed control of the control target point determined according to the observation information and the control measure. t The output unit 18C receives the action a received from the learning unit 18B. t By performing scaling and other processing on the action a t into control information and outputs it to the controlled device 20.

[0100] The learning unit 18B stores data used for learning as experience data in the storage unit 16. In detail, the learning unit 18B stores the target g, the achieved evaluation index f(x(t)), the velocity v(t) or the velocity x(t)-x(t-1) of the control target point, and the state s one control time before as experience data corresponding to the control target time t. t-1 and action a one control time before t-1The experience data including the above is stored in the storage unit 16.

[0101] According to the reinforcement learning algorithm used, the learning unit 18B learns the state s t , the value of the value function V(s t ,g), the value of the action value function Q(s t ,a t ,g), action a t-1 The probability value of π(a t-1 |s t-1 , g) etc. may be further included in the experience data and stored in the storage unit 16.

[0102] The learning unit 18B further derives the control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t , g) is updated. This update process corresponds to learning. The details of this learning will be described later.

[0103] The memory unit 16 stores the experience data input from the learning unit 18B up to a predetermined maximum number of experience data. When the experience data stored in the memory unit 16 exceeds the maximum number, the control unit 18 discards the oldest experience data.

[0104] The learning unit 18B determines the control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t When performing the update process of (g), the empirical data editing unit 18D randomly samples a fixed number (M pieces) of empirical data from the storage unit 16. M is an integer equal to or greater than 1. The empirical data editing unit 18D also uses a control target time t corresponding to each of the sampled M pieces of empirical data as a reference, and identifies other empirical data corresponding to other control target times t' within a predetermined period from the reference control target time t as surrounding empirical data series corresponding to each of the M pieces of empirical data.

[0105] Then, the empirical data editing unit 18D generates MK pieces of edited empirical data to be used for learning by the learning unit 18B, using the M pieces of sampled empirical data and surrounding empirical data series corresponding to each of the M pieces of empirical data, where K is an integer equal to or greater than 1.

[0106] In detail, empirical data editing unit 18D outputs the sampled M pieces of empirical data and surrounding empirical data series corresponding to each of the M pieces of empirical data to goal setting unit 18E.

[0107] The target setting unit 18E sets a plurality of evaluation indexes selected from the plurality of evaluation indexes included in the group based on a group of first empirical data including an evaluation index f(x(t)) derived from the observation information acquired by the acquisition unit 18A and one or more second empirical data each including an evaluation index f(x(t')) derived from one or more other observation information at a control target time t' different from the observation information. j Set as.

[0108] The first empirical data corresponds to the sampled M pieces of empirical data, and the second empirical data corresponds to a series of surrounding empirical data corresponding to each of the M pieces of empirical data.

[0109] In detail, the goal setting unit 18E sets K goals g for each of the M pieces of experience data received from the experience data editing unit 18D. j j is expressed by the following formula (20).

[0110]

number

[0111] First, the target setting unit 18E sets the target g included in each of the M pieces of empirical data as the target g0 (j=0). Next, the target setting unit 18E randomly samples K-1 evaluation indices f(x(t')) included in the second empirical data, which is empirical data constituting a peripheral empirical data series corresponding to each of the M pieces of empirical data, and sets the K-1 targets g j (j=1, 2, ..., K-1). Through these processes, the target setting unit 18E determines K targets g j Set (j=0,1,2,...,K-1).

[0112] The goal setting unit 18E may limit the sampling range of the K-1 evaluation indexes f(x(t')) to the evaluation indexes f(x(t')) achieved at a control target time t' that is in the future than the control target time t of the empirical data.

[0113] In addition, the target setting unit 18E sets the noise-added evaluation indexes obtained by adding noise to each of the K-1 evaluation indexes f(x(t')) sampled from the evaluation indexes f(x(t')) included in the second empirical data as the K-1 targets g j (j=1, 2, ..., K-1). The noise may be generated according to a probability distribution such as a Gaussian distribution or a uniform distribution.

[0114] In addition, the target setting unit 18E sets a value randomly selected from within a possible range of the evaluation index f(x(t)) included in each of the M pieces of empirical data, instead of the second empirical data which is empirical data constituting a peripheral empirical data series corresponding to each of the M pieces of empirical data (first empirical data), as the K-1 targets g j It may be set as (j=1,2,...,K-1).

[0115] The target setting unit 18E sets K targets g according to the target selection method selected by the user. j (j=1, 2, ..., K-1) may be set. The goal setting unit 18E may set K-1 goals g j(j=1,2,...,K-1) may be sampled.

[0116] For example, the goal setting unit 18E displays on the UI unit 14 a display screen for receiving input of a goal selection method and the number of goals to be selected.

[0117] 4 is a schematic diagram of an example of the display screen 30. For example, the goal setting unit 18E displays the display screen 30 on the UI unit 14.

[0118] The display screen 30 includes a selection field for selecting a target selection method and a selection field for selecting a target g j and an input field for the number of selections.

[0119] The target selection method may be, for example, "no addition", "random", "future", "future (with noise)", etc. "No addition" is to select a new target g other than the target g included in the experience data. j "Future" represents a selection from the evaluation index f(x(t')) included in the second empirical data corresponding to a control target time t' that is in the future of the control target time t corresponding to the empirical data. "Future (noise addition)" represents the addition of noise to the evaluation index f(x(t')) included in the second empirical data corresponding to the future control target time t'.

[0120] Target g included in display screen 30 j The input field for the number of selections indicates the number of (K-1) selections to be selected from the evaluation indexes of the second empirical data.

[0121] The user operates the UI unit 14 while viewing the display screen 30 to input a desired target selection method and the number of targets to be selected. The target setting unit 18E sets the number of K-1 targets g input by the user according to the target selection method selected by the user via the display screen 30. j may be selected from the evaluation indexes of the second empirical data.

[0122] For example, assume that the user selects "no addition" via the display screen 30. In this case, the goal setting unit 18E sets only the goal g included in the M pieces of experience data as the goal g j Therefore, in this case, K=1 is forced.

[0123] Also, assume that the user has selected "random". In this case, the goal setting unit 18E may randomly sample K-1 of the achieved evaluation indexes f(x(t')) included in each of the second empirical data, which is empirical data constituting a peripheral empirical data series corresponding to each of the M empirical data (first empirical data). Then, the goal setting unit 18E may add the sampled evaluation indexes f(x(t')) as the goal j (j=1, 2, ..., K-1).

[0124] Also, assume that the user selects "future." In this case, the goal setting unit 18E randomly samples K-1 evaluation indices f(x(t')) achieved at the empirical data (second empirical data) at the control target time t' in the future from M pieces of empirical data (first empirical data). Then, the goal setting unit 18E adds the sampled evaluation indices f(x(t')) as goals j (j=1, 2, ..., K-1).

[0125] Also, assume that the user has selected "future (with noise)". In this case, the goal setting unit 18E randomly samples K-1 evaluation indices f(x(t')) achieved in empirical data (second empirical data) at a control target time t' in the future from M pieces of empirical data (first empirical data). Then, the goal setting unit 18E adds noise to the sampled evaluation index f(x(t')) to obtain a noise-added evaluation index as the target j (j=1, 2, ..., K-1). As the noise, as described above, noise generated according to a probability distribution such as a Gaussian distribution may be used.

[0126] The target g selected in the target setting section 18E j However, if the first empirical data contains only objects similar to the target g, the contribution to improving the learning efficiency decreases. However, by adding the noise-added evaluation index as the target j, the contribution to improving the learning efficiency of the target g j This increases the variation in values, thereby improving learning efficiency.

[0127] As described above, the target g included in the display screen 30 j The input field for the number of selections indicates the number of K-1 selections to be selected from the evaluation indexes of the second experience data. The goal setting unit 18E sets the number of K-1 goals g j By selecting from the evaluation index of the second empirical data, the K-1 targets g j Just sample (j=1,2,...,K-1).

[0128] Returning to FIG. 3, the above-mentioned process by the target setting unit 18E sets K targets g j (j=0,1,2,...,K-1) is set.

[0129] The target setting unit 18E sets K targets g for each of the M pieces of experience data. j (j=0,1,2,...,K-1), i.e., MK targets g j is output to the experience data editing unit 18D.

[0130] The empirical data editing unit 18D calculates, for each of the M pieces of empirical data, the evaluation index f(x(t+1)) and the speed information v(t+1) or the speed information x(t+1)-x(t) included in each of the M pieces of empirical data, and the K targets g(t+1) and g(t+1) received from the target setting unit 18E. j (j=0, 1, 2, ..., K-1) is output to the corrected remuneration determination unit 18F.

[0131] The corrected reward determination unit 18F calculates the evaluation index f(x(t+1)) of the value of the index other than the speed derived from the observation information and the target g jThe smaller the error with respect to the target velocity, the higher the reward is, and a corrected reward is determined based on the velocity of the control point contained in the observation information.

[0132] In detail, the corrected reward determination unit 18F determines a corrected reward by correcting the reward so that the reward becomes lower as the speed of the control target point becomes faster.

[0133] In this embodiment, the corrected remuneration determination unit 18F determines the K targets g set by the target setting unit 18E. j For each (j=0,1,2,...,K-1), the evaluation index f(x(t+1)) derived from the acquired observation information and K targets g j The smaller the error with each of (j=0, 1, 2, ..., K-1), the higher the reward is calculated. Then, the corrected reward determination unit 18F determines a corrected reward by correcting the calculated reward according to the speed information of the control target point included in the observation information.

[0134] Specifically, the corrected remuneration determination unit 18F determines the evaluation index f(x(t+1)) and the speed information v(t+1) or the speed information x(t+1)-x(t) included in the experience data received from the experience data editing unit 18D, and the K targets g corresponding to the experience data. j (j=0,1,2,...,K-1) and calculate the corrected reward.

[0135] The calculation of the corrected reward uses the above formula (9) or an approximation of the above formula (9), for example, the above formulas (16) to (19). j d(x(t+1)) in these equations can be calculated using the L1 distance or L2 distance between each of (j=0,1,2,...,K-1) and the evaluation index f(x(t+1)) included in the empirical data.

[0136] The correction remuneration determination unit 18F determines the correction remuneration by calculating the correction remuneration. The correction remuneration determination unit 18F may determine the correction remuneration by accepting the correction remuneration calculated by an external device that calculates the correction remuneration.

[0137] The corrected reward determination unit 18F determines the K targets g j (j=0,1,2,...,K-1) t ,a t ,g j ) is output to experience data editing unit 18D.

[0138] The empirical data editing unit 18D edits the state s included in each of the M pieces of empirical data. t , action a t , evaluation index f(x(t+1)), and speed information v(t+1) or speed information x(t+1)-x(t), and the target g received from the target setting unit 18E. j and the target g received from the correction remuneration determination unit 18F j Correction reward r(s t ,a t ,g j That is, the experience data editing unit 18D generates edited experience data including K targets g j is set, and the K targets g j The corrected reward r(s t ,a t ,g j ) is further set. Then, experience data editing unit 18D outputs MK pieces of edited experience data generated from the M pieces of experience data to learning unit 18B.

[0139] The learning unit 18B uses the observation information and the corrected reward r(s t ,a t ,g j ) to the control strategy π(a t |s t ,g) are reinforced learning.

[0140] That is, the learning unit 18B uses the MK edited empirical data received from the empirical data editing unit 18D to derive the control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,at , g) is updated.

[0141] First, learning unit 18B outputs speed information v(t+1) or speed information x(t+1)-x(t) contained in each of the MK edited experience data received from experience data editing unit 18D to corrected discount rate determination unit 18G.

[0142] The corrected discount rate determination unit 18G determines a corrected discount rate by correcting the discount rate γ of the corrected reward according to the speed of the control target point derived from the observation information. In detail, the corrected discount rate determination unit 18G determines a corrected discount rate such that the faster the speed of the control target point is, the larger the discount (i.e., the smaller the value of the discount rate γ becomes).

[0143] Specifically, correction discount rate determination unit 18G calculates the correction discount rate as the power of discount rate γ with speed information v(t+1) or speed information x(t+1)-x(t) as the exponent. That is, correction discount rate determination unit 18G calculates the correction discount rate at control target time t+1 using the following formula (21) or formula (22).

[0144]

number

[0145] Corrected discount rate determination unit 18G may determine a corrected discount rate by correcting the discount rate according to the input discount rate for the input speed at which the input was received, according to the speed of the control target point.

[0146] Although the user may directly input the input discount rate by operating UI unit 14, it is difficult for the user to intuitively understand how much the remuneration will be discounted. Therefore, it is preferable that correction discount rate determination unit 18G displays on UI unit 14 a display screen that enables the user to intuitively set the input discount rate.

[0147] FIG. 5A is a schematic diagram of an example of the display screen 32. The correction discount rate determination unit 18G displays the display screen 32 on the UI unit 14. The display screen 32 includes an input field for the distance traveled per unit time, and an input field for the input discount rate for the distance traveled (displayed as "discount rate" on the display screen 32). FIG. 5A shows an input field for the distance traveled per unit time as an input field for the input speed, as an example. However, the display screen 32 may have an input field for the speed instead of the input field for the distance traveled per unit time. By providing an input field for the input speed, such as the distance traveled per unit time or the speed, together with the input discount rate on the display screen 32, the user can see how much the remuneration is discounted for the speed, and can more intuitively input a desired discount rate for the speed (distance traveled per unit time).

[0148] The user operates the UI unit 14 while viewing the display screen 32 to input a travel distance per unit time and an input discount rate that is a rate at which the error and reward are discounted for that travel distance.

[0149] Assume that a user operates the UI unit 14 to input a travel distance X and an input discount rate G desired by the user for the travel distance X.

[0150] In this case, corrected discount rate determination unit 18G calculates discount rate γ from the input discount rate G for the travel distance X according to the following formula (23).

[0151]

number

[0152] Then, correction discount rate determination unit 18G may calculate the correction discount rate by using discount rate γ calculated by equation (23) and the above equation (14) or equation (15). Through these calculations, correction discount rate determination unit 18G determines the correction discount rate.

[0153] For confirmation, correction discount rate determination unit 18G may display on UI unit 14 correspondence information indicating the correspondence between the determined correction discount rate and the travel distance per unit time.

[0154] 5B is a schematic diagram of an example of display screen 34. For example, corrected discount rate determination unit 18G displays display screen 34 on UI unit 14. Display screen 34 includes, as correspondence information, a graph including a line DC that represents the correspondence between the corrected discount rate and the traveled distance. Note that the correspondence information is not limited to a graph as long as it is information that represents the correspondence between the corrected discount rate and the traveled distance.

[0155] In this way, corrected discount rate determination unit 18G may determine a corrected discount rate by correcting discount rate γ according to the input discount rate for the input speed at which an input by the user is accepted, according to the speed of the control target point. If the conditions of control target device 20 change, such as the environment of the unmanned moving body or robot, or the materials for laser welding, it is considered that the appropriate discount rate will also change. For this reason, by allowing the discount rate to be set and changed by the user, corrected discount rate determination unit 18G can determine a corrected discount rate according to the conditions of control target device 20.

[0156] Returning to Fig. 3, the explanation will continue. Corrected discount rate determination unit 18G outputs the calculated corrected discount rate to learning unit 18B.

[0157] Learning unit 18B calculates control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t , g) is updated (learned).

[0158] When using a reinforcement learning algorithm called a policy-on type, the learning unit 18B may sample the above-mentioned empirical data when a certain amount of empirical data is stored in the memory unit 16, or when the drone flight or welding is completed, and perform an update process using edited empirical data generated based on the empirical data.

[0159] On the other hand, when a reinforcement learning algorithm called a policy-off type is used, the learning unit 18B may sample a certain number of empirical data from the storage unit 16 every time or once every few times, and perform an update process using edited empirical data generated based on the empirical data. In the case of the policy-off type, the empirical data may be stored in the storage unit 16 until the number of empirical data reaches a predetermined maximum value, and when the maximum value is exceeded, the oldest empirical data may be discarded.

[0160] The learning unit 18B determines the control strategy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t Any reinforcement learning algorithm can be used to update the value function V(s , g). In this embodiment, it is preferable that the learning unit 18B performs these update processes using a correction discount rate received from the correction discount rate determination unit 18G instead of the discount rate γ. For example, the value function V(s t ,g) and the action value function Q(s t ,a t , g), the learning unit 18B uses the above formulas (2) and (3) to calculate the value function V(s t ,g) and the action value function Q(s t ,a t ,g) can be updated.

[0161] The learning unit 18B may perform processing in accordance with the reinforcement learning algorithm to be used, except that the correction discount rate is used instead of the discount rate γ.

[0162] Next, an example of the flow of information processing executed by the machine learning device 10 of this embodiment will be described.

[0163] FIG. 6 is a flowchart showing an example of the flow of information processing executed by the machine learning device 10 of this embodiment.

[0164] The acquiring unit 18A acquires observation information at a control target time t from the control target device 20 (step S100).

[0165] The learning unit 18B determines the state s t and goal g are calculated, and action a t The output unit 18C determines the action a determined in step S102. t into control information and outputs it to the controlled device 200 (step S104).

[0166] The learning unit 18B stores data corresponding to the observation information at the control target time t acquired in step S100 in the storage unit 16 (step S106). As described above, the experience data includes the target g, the achieved evaluation index f(x(t)), the speed information of the control target point, i.e., the speed v(t) or the speed (speed represented by the moving distance per unit time with one control period as the unit) x(t)-x(t-1), and the state s one control time before. t-1 and Action a t-1 Including,

[0167] The learning unit 18B determines whether or not it is time to execute learning (step S108). t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t , g) is performed. For example, the learning unit 18B performs the update process at every certain control target time. Therefore, the learning unit 18B performs the determination in step S108 by determining whether a period of the certain control target time has elapsed since the previous learning execution.

[0168] If the determination in step S108 is negative (step S108: No), the process proceeds to step S110. In step S110, the control unit 18 determines whether or not to end the process (step S110). If the determination in step S110 is positive (step S110: Yes), the process ends this routine. If the determination in step S110 is negative (step S110: No), the process returns to step S100.

[0169] On the other hand, if the learning unit 18B determines that it is time to execute learning (step S108: Yes), the process proceeds to step S112.

[0170] In step S112, the empirical data editing unit 18D randomly samples a certain number (M pieces) of empirical data from the storage unit 16 to acquire M pieces of empirical data (step S112). The empirical data editing unit 18D outputs the sampled M pieces of empirical data and surrounding empirical data series corresponding to each of the M pieces of empirical data to the goal setting unit 18E.

[0171] The target setting unit 18E sets K evaluation indexes selected from a plurality of evaluation indexes included in a group of first empirical data, which is M pieces of empirical data acquired in step S112, and second empirical data constituting a peripheral empirical data series corresponding to each of the M pieces of empirical data, as K targets g j (j=0, 1, 2, ..., K-1) is set (step S114).

[0172] The empirical data editing unit 18D calculates, for each of the M pieces of empirical data, the evaluation index f(x(t+1)) and the speed information v(t+1) or the speed information x(t+1)-x(t) included in each of the M pieces of empirical data, and the K targets g(t+1) and g(t+1) received from the target setting unit 18E. j (j=0, 1, 2, ..., K-1) is output to the corrected remuneration determination unit 18F.

[0173] The corrected remuneration determination unit 18F determines the evaluation index f(x(t+1)) and the speed information v(t+1) or the speed information x(t+1)-x(t) included in the experience data acquired in step S112, and the K targets g set in step 114 corresponding to the experience data. j (j=0,1,2,...,K-1), and the corrected reward r(s t ,a t ,g j ) is determined (step S116).

[0174] The empirical data editing unit 18D edits the state s included in each of the M pieces of empirical data. t , action a t , evaluation index f(x(t+1)), and speed information v(t+1) or speed information x(t+1)-x(t), and the target g received from the target setting unit 18E. j and the target g received from the correction remuneration determination unit 18F j Correction reward r(s t ,a t ,g j ). Then, experience data editing unit 18D outputs the M K edited experience data generated from the M experience data to learning unit 18B. Learning unit 18B outputs speed information v(t+1) or speed information x(t+1)-x(t) included in each of the M K edited experience data received from experience data editing unit 18D to corrected discount rate determination unit 18G.

[0175] Correction discount rate determination unit 18G determines the power of discount rate γ with the speed information v(t+1) or speed information x(t+1)-x(t) received from learning unit 18B as the correction discount rate (step S118).

[0176] Learning unit 18B calculates control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,at , g) is updated (learned) (step S120). Then, the process proceeds to step S110.

[0177] As described above, the machine learning device 10 of this embodiment includes an acquisition unit 18A, an output unit 18C, a corrected reward determination unit 18F, and a learning unit 18B. The acquisition unit 18A acquires observation information including information on the speed of the control target point at the control target time. The output unit 18C outputs control information including information on the speed control of the control target point determined according to the observation information and the control measure. The corrected reward determination unit 18F determines a corrected reward that is higher as the error between the evaluation index of the value of the index other than the speed derived from the observation information and the target is smaller, and is corrected according to the speed of the control target point included in the observation information. The learning unit 18B performs reinforcement learning of the control measure from the observation information and the corrected reward.

[0178] In this way, the machine learning device 10 of this embodiment performs reinforcement learning of a control policy using a corrected reward corrected according to the speed of the control object point included in the observation information. By using the corrected reward corrected according to the speed, the machine learning device 10 of this embodiment can learn a control policy that minimizes the average error of the control object point with respect to the target, including speed control.

[0179] Therefore, the machine learning device 10 of this embodiment can minimize the average error of the control point, including the speed control, with respect to the target.

[0180] Furthermore, the machine learning device 10 of this embodiment further performs reinforcement learning of the control policy using a corrected discount rate obtained by correcting the discount rate of the reward according to the moving distance of the control target point, instead of the discount rate of the reward. By using the corrected discount rate, the machine learning device 10 of this embodiment can prevent a change in speed from affecting the value of the discounted cumulative reward, and can further learn a control policy that minimizes the average error.

[0181] The machine learning device 10 of this embodiment further includes a goal setting unit 18E. Based on a group of first empirical data including evaluation indexes derived from the acquired observation information and one or more second empirical data including evaluation indexes derived from one or more other observation information at a different control target time from the acquired observation information, the goal setting unit 18E sets K evaluation indexes selected from the multiple evaluation indexes included in the group as K targets g j Set as.

[0182] Then, the corrected reward determination unit 18F determines the set K targets g j For each of the K targets g j The smaller the error between each of the targets is, the higher the reward is. The learning unit 18B then determines a corrected reward by correcting the reward according to the speed of the control target point. j and the target g received from the correction remuneration determination unit 18F j Correction reward r(s t ,a t ,g j ) and perform reinforcement learning based on edited experience data including.

[0183] For this reason, the learning unit 18B performs the action a t Not only the target g that was set when determining the target g, but also the evaluation indicators that were ultimately achieved are considered as the target g j The edited experience data can be used for learning. Therefore, in addition to the above effects, the machine learning device 10 of the present embodiment can significantly improve learning efficiency.

[0184] (Modification) In the above embodiment, an example has been described in which control unit 18 includes goal setting unit 18E and corrected discount rate determination unit 18G. However, control unit 18 may be configured without including at least one of goal setting unit 18E and corrected discount rate determination unit 18G.

[0185] In the case of a configuration that does not include the goal setting unit 18E, the corrected remuneration determination unit 18F sets K goals g jInstead of (j=0, 1, 2, . . . , K-1), the corrected reward may be determined for each of the M pieces of experience data using the goal g included in each of the M pieces of experience data acquired by the experience data editing unit 18D.

[0186] In addition, in the case of a configuration that does not include the correction discount rate determination unit 18G, the learning unit 18B uses the discount rate γ, which is the discount rate before correction, instead of the correction discount rate, to determine the control policy π(a t |s t ,g), value function V(s t ,g), and the action value function Q(s t ,a t In this case, the learning unit 18B performs a process of updating (learning) the value function V(s t ,g) and the action value function Q(s t ,a t , g) can be updated using the above formulas (2) and (3). This is an approximation method that substitutes the above formulas (12) and (13) for the above formulas (2) and (3), respectively, and is effective when there is little change in speed. This method has the advantage that existing reinforcement learning processing can be applied as is.

[0187] Next, an example of the hardware configuration of the machine learning device 10 of the above embodiment will be described.

[0188] FIG. 7 is a diagram illustrating an example of a hardware configuration of the machine learning device 10 according to the above embodiment.

[0189] The machine learning device 10 of the above embodiment includes a control device such as a CPU (Central Processing Unit) 90B, storage devices such as a ROM (Read Only Memory) 90C, a RAM (Random Access Memory) 90D, and a HDD (Hard Disk Drive) 90E, an I / F unit 90A that interfaces with various devices, and a bus 90F that connects each unit, and has a hardware configuration that utilizes a conventional computer.

[0190] In the machine learning device 10 of the above embodiment, the CPU 90B reads a program from the ROM 90C onto the RAM 90D and executes it, thereby realizing each of the above units on a computer.

[0191] The programs for executing the above-mentioned processes executed by the machine learning device 10 of the above-mentioned embodiment may be stored in the HDD 90E. Also, the programs for executing the above-mentioned processes executed by the machine learning device 10 of the above-mentioned embodiment may be provided by being pre-installed in the ROM 90C.

[0192] The program for executing the above-mentioned processing executed by the machine learning device 10 of the above embodiment may be stored in a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD) in an installable or executable format file and provided as a computer program product. The program for executing the above-mentioned processing executed by the machine learning device 10 of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. The program for executing the above-mentioned processing executed by the machine learning device 10 of the above embodiment may be provided or distributed via a network such as the Internet.

[0193] Although the embodiment of the present invention has been described above, the embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0194] 10 Machine Learning Device 14 UI section 18A Acquisition Department 18B Learning Department 18C Output section 18E Goal Setting Department 18F Compensation compensation determination department 18G Adjustment Discount Rate Determination Department 20 Controlled Device

Claims

1. An acquisition step of acquiring observation information including information regarding a velocity of a control target point at a control target time; an output step of outputting control information including information regarding speed control of the control target point, the control information being determined according to the observation information and a control strategy; a corrected reward determination step for determining a higher reward as the error between the evaluation index of the value of the index other than the speed derived from the observation information and the target is smaller, and correcting the corrected reward according to the speed of the control target point included in the observation information; a learning step of performing reinforcement learning on the control policy from the observation information and the corrected reward; A machine learning program that allows a computer to execute the following.

2. The correction remuneration determination step includes: determining a corrected reward by correcting the reward so that the corrected reward is lower as the speed of the control target point is faster; The machine learning program according to claim 1 .

3. a corrected discount rate determination step for determining a corrected discount rate obtained by correcting the discount rate of the corrected remuneration in accordance with the speed of the control target point derived from the observation information; The learning step includes: The control policy is subjected to reinforcement learning based on the observation information, the corrected reward, and the corrected discount rate. The machine learning program according to claim 1 .

4. The correction discount rate determination step includes: determining the corrected discount rate such that the faster the speed of the control point is, the smaller the value of the discount rate becomes; The machine learning program according to claim 3 .

5. The correction discount rate determination step includes: The discount rate according to the input discount rate for the input speed at which the input was received is corrected according to the speed of the control point to determine the corrected discount rate. The machine learning program according to claim 3 .

6. a target setting step of setting, as the target, a plurality of evaluation indexes selected from a plurality of evaluation indexes included in a group of first empirical data including the evaluation index derived from the acquired observation information and one or more second empirical data including the evaluation index derived from one or more other observation information having a different control target time from the acquired observation information; The correction remuneration determination step includes: For each of the plurality of targets that have been set, a higher reward is determined as the error between the evaluation index derived from the acquired observation information and each of the plurality of targets is smaller, and the corrected reward is determined by correcting the reward according to the speed of the control target point included in the observation information. The machine learning program according to claim 1 .

7. The goal setting step includes: The evaluation index included in the first empirical data and a noise-added evaluation index obtained by adding noise to the evaluation index included in the second empirical data are set as the target. The machine learning program according to claim 6.

8. The goal setting step includes: setting the goal according to a goal selection method selected by a user; The machine learning program according to claim 6.

9. The goal setting step includes: setting a user-selected number of said goals; The machine learning program according to claim 6.

10. An acquisition step of acquiring observation information including information regarding a velocity of a control target point at a control target time; an output step of outputting control information including information regarding speed control of the control target point, the control information being determined according to the observation information and a control strategy; a corrected reward determination step for determining a higher reward as the error between the evaluation index of the value of the index other than the speed derived from the observation information and the target is smaller, and correcting the corrected reward according to the speed of the control target point included in the observation information; a learning step of performing reinforcement learning on the control policy from the observation information and the corrected reward; Machine learning methods, including

11. an acquisition unit that acquires observation information including information regarding a velocity of a control target point at a control target time; an output unit that outputs control information including information regarding speed control of the control target point, the control information being determined in accordance with the observation information and a control strategy; A corrected reward determination unit determines a higher reward as the error between the evaluation index of the value of the index other than the speed derived from the observation information and the target is smaller, and determines a corrected reward by correcting the reward according to the speed of the control target point included in the observation information; a learning unit that performs reinforcement learning of the control policy from the observation information and the corrected reward; A machine learning device comprising:

Citation Information

Patent Citations

  • Automatic driving solution under multi-target complex traffic scene based on reinforcement learning

    CN114701517A

  • Automatic driving method and device based on hierarchical reinforcement learning

    CN116061971A

  • Processing condition adjusting device and machine learning device

    JP2019166559A

  • Information processing device and information processing method

    WO2021075107A1

  • Wiring device

    JP1985077617A

Cited By

  • Machine learning device, machine learning program, trained model generation method, and action plan generation device

    WO2026211479A1