Information processing apparatus, information processing method, and information processing program

Abstract

To make a robot model at the time of simulation perform a more efficient action by using reinforcement learning.SOLUTION: The information processing device includes a target information acquisition section configured to acquire, as target information, a target position of a tip portion of a robot model and a target angle of the tip portion at the target position in a simulation, a current information acquisition section configured to acquire, as current information, a current position of the tip portion and a current angle of the tip portion, an intermediate reward calculation section configured to calculate, as an immediate reward value, a reward to be given to an agent each time the robot model acts, based on a distance from the current position of the tip portion to the target position and an angle difference between the current angle of the tip portion and the target angle of the tip portion at the target position, and calculate, as an intermediate reward value, an integrated value of the immediate reward values from a start of the simulation to a current time, and a reinforcement learning section configured to learn a learning model such that the intermediate reward value at an end of the simulation is maximized.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program, and more particularly to an information processing device, an information processing method, and an information processing program used for reinforcement learning. [Background technology]

[0002] Creating a robot control program using offline teaching has traditionally required a large number of tasks and a long time, so efforts have been made to reduce the amount of work involved by improving the efficiency and automating offline teaching.

[0003] In recent years, advances in computer technology have led to widespread use of three-dimensional simulations of robot behavior. Furthermore, advances in machine learning technology have led to proposals to reduce the effort and time required for offline teaching by using simulations of the behavior of a robot model in a three-dimensional virtual space that mimics the space in which the robot will actually be used, allowing artificial intelligence to reinforce learning efficient robot control. Reinforcement learning in the field of robot control here refers to the process of giving rewards to an agent, the subject of robot control learning, to learn efficient control of a robot model.

[0004] The learning device disclosed in Patent Document 1 includes a learning unit that learns the behavior of a robot arm. The learning unit of the learning device disclosed in Patent Document 1 includes a reward calculation unit that calculates a reward according to the trajectory of the robot arm, and the reward calculation unit performs calculations by increasing the reward as the distance between the tip of the robot arm and the target position becomes shorter. The learning device disclosed in Patent Document 1 increases the reward as the distance between the tip of the robot arm and the target position becomes shorter, thereby enabling the robot arm to learn efficient behavior until it reaches the target position.

[0005] However, the learning device disclosed in Patent Document 1 is insufficient in terms of the efficiency of the robot arm's actions, and there is still a demand for learning more efficient actions. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-82314 Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and an information processing program that can make a robot model perform more efficient actions during simulation using reinforcement learning. [Means for solving the problem]

[0008] That is, the information processing device according to the first aspect is an information processing device that learns a learning model used by an agent that is responsible for a robot model in a simulation, and is characterized by comprising: a target information acquisition unit that acquires the target position of the tip of the robot model in the simulation and the target angle of the tip at the target position as target information; a current information acquisition unit that acquires the current position of the tip and the angle of the tip as current information; an intermediate reward calculation unit that calculates an immediate reward value as a reward to be granted to the agent each time the robot model takes action based on the distance from the current position of the tip to the target position and the angular difference between the current angle of the tip and the target angle of the tip at the target position, and calculates the integrated value of the immediate reward value from the start of the simulation to the present as an intermediate reward value; and a reinforcement learning unit that learns the learning model so that the intermediate reward value at the end of the simulation is maximized.

[0009] In a second aspect, in the information processing device according to the first aspect, the immediate reward value may be calculated based on a reduction in task completion time of the robot model in addition to the distance and the angle difference.

[0010] In a third aspect, the information processing device according to the first aspect may further include a final reward determination unit that determines a reward to be granted to the agent as a final reward value based on the final result of the behavior of the robot model, and the reinforcement learning unit may learn the learning model so that the sum of the intermediate reward value and the final reward value is maximized at the end of the simulation.

[0011] In a fourth aspect, in the information processing device according to the third aspect, the final reward determination unit may determine the final reward value to be the maximum value if, in the final result, the tip reaches the target position within a specified time limit and the angle of the tip at the target position reaches the target angle.

[0012] In a fifth aspect, in the information processing device according to the third aspect, the final reward determination unit may determine the final reward value to be a minimum value when the tip portion comes into contact with the workpiece model during the simulation.

[0013] In a sixth aspect, in the information processing device according to the third aspect, the final reward determination unit may determine the final reward value to be a minimum value if the tip portion does not reach the target angle at the target position within a predetermined time limit.

[0014] According to a seventh aspect, in the information processing device according to the first aspect, the distance and the angle difference may be normalized before use.

[0015] According to an eighth aspect, in the information processing device according to the first aspect, the robot model may be an articulated robot having a plurality of arms.

[0016] An information processing method according to a ninth aspect is a control method used in an information processing device that learns a learning model used by an agent that carries out a robot model in a simulation, and is characterized in that the calculation unit of the information processing device carries out the following: a target information acquisition step that acquires the target position of the tip of the robot model in the simulation and the target angle of the tip at the target position as target information; a current information acquisition step that acquires the current position of the tip and the angle of the tip as current information; an intermediate reward calculation step that calculates an immediate reward value as a reward to be granted to the agent each time the robot model takes action based on the distance from the current position of the tip to the target position and the angular difference between the current angle of the tip and the target angle of the tip at the target position, and calculates the integrated value of the immediate reward value from the start of the simulation to the present as an intermediate reward value; and a reinforcement learning step that learns the learning model so that the intermediate reward value at the end of the simulation is maximized.

[0017] An information processing program according to a tenth aspect is a control program used in an information processing device that learns a learning model used by an agent that carries a robot model in a simulation, and is characterized in that it causes a calculation unit of the information processing device to perform the following: a target information acquisition function that acquires the target position of the tip of the robot model in the simulation and the target angle of the tip at the target position as target information; a current information acquisition function that acquires the current position of the tip and the angle of the tip as current information; an intermediate reward calculation function that calculates a reward to be granted to the agent each time the robot model takes action as an immediate reward value based on the distance from the current position of the tip to the target position and the angular difference between the current angle of the tip and the target angle of the tip at the target position, and calculates the integrated value of the immediate reward values ​​from the start of the simulation to the present as an intermediate reward value; and a reinforcement learning function that learns the learning model so that the intermediate reward value at the end of the simulation is maximized. [Effects of the Invention]

[0018] The information processing device etc. according to the present invention is an information processing device that learns a learning model used by an agent that is responsible for a robot model in a simulation, and is characterized by comprising: a target information acquisition unit that acquires, as target information, a target position of the tip of the robot model in the simulation and a target angle of the tip at the target position; a current information acquisition unit that acquires, as current information, the current position of the tip and the angle of the tip; an intermediate reward calculation unit that calculates, as an immediate reward value, a reward to be granted to the agent each time the robot model takes action, based on the distance from the current position of the tip to the target position and the angular difference between the current angle of the tip and the target angle of the tip at the target position, and calculates, as an intermediate reward value, the integrated value of the immediate reward values ​​from the start of the simulation to the present; and a reinforcement learning unit that learns the learning model so that the intermediate reward value at the end of the simulation is maximized, thereby making it possible to make the robot model behave more efficiently during the simulation using reinforcement learning. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic configuration diagram of a simulation system according to this embodiment and a block diagram of an information processing device. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a robot model according to this embodiment. [Figure 3] FIG. 3 is a diagram for explaining an example of the connection relationship between the components of the robot model according to this embodiment. [Figure 4] FIG. 4 is a diagram for explaining the configuration of the tip of the robot model according to this embodiment. [Figure 5] FIG. 5 is a diagram for explaining the relationship between the tip of the robot model according to this embodiment and the target position and target angle. [Figure 6] FIG. 6 is a diagram for explaining the relationship between the tip of the robot model and the workpiece model according to this embodiment. [Figure 7] FIG. 7 is a flowchart of the information processing program according to this embodiment. [Figure 8]FIG. 8 is a flowchart of an information processing program according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] An information processing device 100 according to an embodiment will be described in detail below with reference to the drawings. The information processing device 100 learns the learning model used by the agent that handles the robot model 10 in the simulation, and reproduces the robot 20, workpiece 22, and object 24 contained in the workspace 300 as three-dimensional models in the virtual space 400 as the robot model 10, workpiece model 12, and object model 14, respectively, and simulates the operation of the robot 20. Note that the diagram is an example, and the information processing device 100 is not limited to the one shown. Also, the diagram is a schematic diagram, and the size ratios of the components of the robot model 10 and the robot 20, the number and positional relationships of the workpiece model 12 and the workpiece 22, and the object model 14 and the objects 24 present in the virtual space 400 and the working space 300 are not strictly accurate.

[0021] <Embodiment> <System configuration> FIG. 1 is a schematic diagram of a simulation system 500 including an information processing device 100 according to this embodiment. The simulation system 500 includes the information processing device 100, an articulated robot 20 having multiple arms, and a robot control device 200. The robot 20 is a painting robot that drives each of its arms under the control of the robot control device 200 to paint a workpiece 22. Note that FIG. 1 shows a painting robot with a spray gun attached to its tip 10f as an example, but the robot model 10 and the robot 20 are not limited to this and may be a welding robot, a transport robot, a pickup robot, or the like. The robot model 10, like the robot 20, is an articulated robot having multiple arms.

[0022] Here, the object 24 is an object present around the robot 20, and may be another workpiece, another robot, control equipment, or the like. The virtual space 400 in which the simulation is performed is displayed on the display unit 141. The display unit 141 is a monitor having a function of displaying various information output from the information processing device 100, and can be realized by, for example, an LCD (Liquid Crystal Display), an organic EL display, or the like. Note that the display unit 141 may also be, for example, a monitor of a mobile terminal or a tablet terminal. An operator can input instructions to the information processing device 100 using an input device such as the keyboard 131 or a mouse (not shown). Note that, in the information processing device 100 according to this embodiment, the display unit 141 is not essential for performing machine learning (reinforcement learning), which will be described later.

[0023] The robot control device 200 drives the robot 20 based on robot control information output by a simulation in the information processing device 100. The control information may include, for example, values ​​such as the target angle and angular velocity of each axis (joint) constituting the robot 20, and the torque and drive power of the drive motor. Note that while FIG. 1 shows a configuration in which the robot control device 200 is connected to the information processing device 100, the present invention is not limited to this. In other words, the robot control device 200 may exist separately from the information processing device 100.

[0024] In the simulation system 500, the information processing device 100 reduces the load in simulating a plurality of movements of the robot 20 and provides an environment for efficient machine learning.

[0025] Next, the configuration of the robot model 10 will be described with reference to Figures 2 and 3. Figure 2 is a diagram for explaining an example of the configuration of the robot model 10, and Figure 3 is a diagram for explaining an example of the connection relationship between the components of the robot model 10. The robot model 10 is a six-axis articulated arm robot. The joints of the arm robot are movable parts, and each rotates around an axis. The robot model 10 includes a base 10a, a first arm 10b, a second arm 10c, a third arm 10d, and an end effector 10e. A spray gun or the like is attached to the tip of the end effector 10e as a tip 10f. The pedestal 10a is the base of the robot model 10, which is placed directly at the installation location in the virtual space 400. The first arm 10b is connected to the pedestal 10a via a first axis. The second arm 10c is connected to the first arm 10b via a second axis. The third arm 10d is connected to the second arm 10c via a third axis and a fourth axis. The end effector 10e is connected to the third arm 10d via a fifth axis and a sixth axis. A rotational angle range is set for each joint.

[0026] <Hardware configuration> The information processing device 100 includes a calculation unit 110, a communication I / F (interface) 120, an input I / F 130, an output I / F 140, and a storage unit 170 as hardware components.

[0027] The calculation unit 110 is typically a processor, and includes a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc., and may be realized by a logic circuit (hardware) formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc., or a dedicated circuit. The calculation unit 110 may execute the functions and methods described in each embodiment by reading a program stored in the storage unit 170 and executing code or instructions included in the read program.

[0028] The storage unit 170 stores various programs and data required for the operation of the information processing device 100. The storage unit 170 may include, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 170 may also include memory (such as a random access memory (RAM) or a read-only memory (ROM)) that provides a working area for the calculation unit 110. The information processing device 100 according to this embodiment functions as a goal information acquisition unit 111, a current information acquisition unit 112, an intermediate reward calculation unit 113, a final reward determination unit 114, and a reinforcement learning unit 115, which will be described later, by the calculation unit 110 executing a program loaded into the memory of the storage unit 170. For example, the storage unit 170 may store motion analysis software for simulating the operation of the robot model 10 in a virtual space.

[0029] The storage unit 170 stores robot configuration information including the shape of each arm constituting the robot model 10, the connection relationships between the arms, and the range of motion of each arm. The robot configuration information may include the length, width, and length of the arms constituting the robot, the distance between joints (effective length), the connection relationships between the arms, and identification information of the connecting joints. For example, the robot configuration information may be extracted from CAD (Computer Aided Design) data of the robot 20. Furthermore, the robot configuration information may include information indicating the range of motion of each arm, such as soft limit values, the rotation direction of each joint, and the range of possible rotation angles. Note that this data may be received and stored by the target information acquisition unit 111 (described later) via the input device (keyboard) 131 input by an operator. Although FIG. 1 illustrates the storage unit 170 as being integrated with the information processing device 100, the data stored in the storage unit 170 may be stored in a database server separate and independent from the information processing device 100.

[0030] The communication I / F 120 is implemented as hardware such as a network adapter, communication software, or a combination of these, and transmits and receives various data to and from an external device such as the robot control device 200, for example.

[0031] The input I / F 130 is a connection interface with an input device for inputting various operations to the information processing device 100. The input device includes, for example, a touch panel, hardware keys such as a keyboard 131, and a pointing device such as a mouse.

[0032] The output I / F 140 is a connection interface with an output device that outputs processing results such as simulation results by the information processing device 100. The output device includes, for example, a display unit (display) 141, a speaker, and the like.

[0033] <Functional configuration> The calculation unit 110 includes a goal information acquisition unit 111 , a current information acquisition unit 112 , an intermediate reward calculation unit 113 , a final reward determination unit 114 , and a reinforcement learning unit 115 . The position and angle of the tip 10f will be described with reference to Figures 4 and 5, and the contact between the tip 10f and the workpiece model 12 will be described with reference to Figure 6. Figure 4 is a diagram for explaining the configuration of the tip 10f of the robot model 10, Figure 5 is a diagram for explaining the relationship between the tip 10f of the robot model 10 and the target position 11 and target angle, and Figure 6 is a diagram for explaining the relationship between the tip 10f of the robot model 10 and the workpiece model 12.

[0034] The target information acquisition unit 111 acquires the target position 11 of the tip 10f of the robot model 10 in the simulation and the target angle of the tip 10f at the target position 11 as target information. A reference point 10g of the tip 10f serves as a reference for the position of the tip 10f, and a target position 11 of the tip 10f serves as a target for the reference point 10g. That is, the robot model 10 is controlled so that the reference point 10g reaches the target position 11. The target angle of the tip portion 10f is a target angle of the tip portion 10f when the reference point 10g of the tip portion 10f reaches the target position 11, and defines the direction in which the tip portion 10f should face at the target position 11. In other words, the robot model 10 is controlled so that the angle of the tip portion 10f becomes the target angle when the reference point 10g reaches the target position 11. The angle of the tip 10f refers to the angle of a predetermined normal direction 10h (hereinafter referred to as the normal direction of the tip 10f) determined in one direction relative to the direction in which the tip 10f faces. The target angle is set using the angle of the normal direction 10h of the tip 10f, and refers to the angle of the normal direction 11a (see FIG. 5) of the tip 10f at the time when the reference point 10g reaches the target position 11.

[0035] The current information acquisition unit 112 acquires the current position of the tip portion 10f and the angle of the tip portion 10f as current information. The current position of the tip portion 10f refers to the current position of the reference point 10g, that is, the position of the reference point 10g at the current time point or the time point of interest in the simulation. The current angle of the tip 10f refers to the angle of the normal direction 10h of the current tip 10f, and refers to the angle of the normal direction 10h of the tip 10f at the current time or at a time of interest in the simulation.

[0036] The agent is the entity that learns and is responsible for the robot model 10. The agent receives a state and performs an action based on the state. The action acts on the environment and as a result, a reward is obtained. In reinforcement learning, the agent learns the optimal action according to the evaluation (reward) of the action in that state.

[0037] In this embodiment, the state represents the current state of the environment surrounding the agent. In the posture search of the robot model 10 in this embodiment, the current position of the reference point 10g of the tip 10f, the current angle of the tip 10f, the target position 11, the angle of the tip 10f at the target position 11, etc. are included.

[0038] An action in this embodiment is an operation that an agent performs when in a certain state. In this embodiment, the agent is a robot model 10, and the action is realized by the amount of movement of the actuator of each axis of the robot model 10.

[0039] In this embodiment, the reward represents an evaluation of the agent's behavior in a certain state. In this embodiment, if the tip 10f of the robot model 10 moves closer to the target position 11 after the action, a high immediate reward value is given as a good evaluation. The agent's final result is also evaluated, and the reward is assigned to the agent as a final reward value. The final result refers to a certain state, such as when the angle of the tip 10f of the agent (robot model 10) reaches the target angle at the target position 11, when the tip 10f comes into contact with the workpiece model 12, or when the maximum step is reached without achieving learning. When the final result is obtained, the state is evaluated and the final reward value is determined. In this embodiment, if the angle of the tip 10f cannot reach the target angle at the target position 11 or when the tip 10f comes into contact with the workpiece model 12, the evaluation is low. On the other hand, if the angle of the tip 10f reaches the target angle at the target position 11, the evaluation is high.

[0040] The reward value will be explained below with reference to Table 1, equations (1), (2), and (3). The intermediate reward calculation unit 113 calculates the reward to be given to the agent each time the robot model 10 takes action as an immediate reward value based on the distance 15 (see Figure 5) from the current position of the tip portion 10f to the target position 11 and the angle difference 16 (see Figure 5) between the current angle of the tip portion 10f and the target angle of the tip portion 10f at the target position 11, and calculates the integrated value of the immediate reward value from the start of the simulation to the present as the intermediate reward value. The distance 15 may be expressed as a Euclidean distance.

[0041] The distance evaluation (Rd) in Table 1 is calculated based on the distance 15 from the current position of the tip 10f to the target position 11, and is calculated using the formula shown in Equation (1). The maxStep in Equation (1) indicates the maximum step in one episode of reinforcement learning. The distance 15 and the angle difference 16 are normalized before use, and the normalized distance (d) is the normalized value of the distance 15, and the normalized angle difference (a) is the normalized value of the angle difference 16. Normalization is a type of data preprocessing in machine learning, and in order to efficiently train a learning model, data is normalized to unify the scales of different feature quantities. In this embodiment, the scales of the different feature quantities are unified by normalizing the distance 15 and the angle difference 16. The angle evaluation (Ra) in Table 1 is calculated based on the angle difference 16 between the current angle of the tip portion 10f and the target angle of the tip portion 10f at the target position 11, and is calculated using the formula expressed by equation (2). The distance evaluation (Rd) and the angle evaluation (Ra) are set to negative values ​​as shown in Equation (1) and Equation (2), based on the fact that the fewer the number of actions performed by the robot model 10, the higher the evaluation and the greater the reward given.

[0042] The immediate reward value may be calculated based on the reduction in the time taken by the robot model 10 to complete the task, in addition to the distance 15 and the angle difference 16 . The task achievement time is the time from the start of the simulation to the achievement of the objective of the robot model 10, specifically, the number of actions taken by the robot model 10 from the start of the simulation until the angle of the tip 10f of the robot model 10 reaches the target angle at the time when the reference point 10g of the tip 10f of the robot model 10 reaches the target position 11. The fewer the number of actions taken by the robot model 10 from the start of the simulation to the achievement of the objective of the robot model 10, the more reward can be received, specifically, the reward decreases by a value of (-1 / maximum number of learning steps) each time the number of actions taken by the robot model 10 increases by one.

[0043] The total immediate reward (R) is the sum of the immediate reward value due to the reduction in task completion time, the distance evaluation (Rd), and the angle evaluation (Ra), and is expressed by equation (3). The intermediate reward value is the cumulative value of the total immediate reward (R) from the start of the simulation to the completion of the task. In addition, the total immediate reward (R) may be expressed as the sum of the distance rating (Rd) and the angle rating (Ra).

[0044] [Table 1]

[0045]

number

[0046]

number

[0047]

number

[0048] The final reward determination unit 114 determines the reward to be given to the agent based on the final result of the action of the robot model 10 as the final reward value.

[0049] The final reward determination unit 114 determines the final reward value to be the maximum value when, in the final result, the tip portion 10f reaches the target position 11 within a predetermined time limit and the angle of the tip portion 10f at the target position 11 reaches the target angle. In this case, as shown in No. 4 in Table 1, the final reward value is 1. The predetermined time limit may be the time it takes to reach the maximum number of steps (maxStep) in learning.

[0050] The final reward determination unit 114 determines the final reward value to be the minimum value if the tip 10f comes into contact with the workpiece model 12 during the simulation. In this case, as shown in No. 6 in Table 1, the final reward value is −1. Referring to FIG. 6, the contact between the tip 10f and the workpiece model 12 will be described. A "Ray Perception Sensor" is attached to the tip of the tip portion 10f. The "Ray Perception Sensor" is a type of sensor used to determine contact with an object in ML-Agents (Unity Machine Learning Agents), a reinforcement learning framework from Unity (registered trademark), and acquires information such as the distance from the robot model 10, which is an agent, to the workpiece model 12 and the type of workpiece model 12. The length of a ray 17 extending from the tip of the tip portion 10f represents the distance to the workpiece model 12 (see FIG. 6(a)). When the tip portion 10f comes into contact with the workpiece model 12, a contact mark 18 is displayed instead of the ray 17 (see FIG. 6(b)).

[0051] The final reward determination unit 114 determines the final reward value to be the minimum value if the tip end portion 10f does not reach the target angle at the target position 11 within a predetermined time limit. In this case, as shown in No. 5 in Table 1, the final reward value is −1.

[0052] The reinforcement learning unit 115 learns the learning model so that the intermediate reward value at the end of the simulation is maximized. Furthermore, the reinforcement learning unit 115 may train the learning model so that the sum of the intermediate reward value and the final reward value is maximized at the end of the simulation.

[0053] (Information processing program and information processing method) Next, an information processing program according to an embodiment of the present disclosure will be described together with an information processing method with reference to Fig. 7. Fig. 7 is a flowchart of the information processing program according to this embodiment. The information processing method is executed by the calculation unit 110 of the information processing device 100 based on the information processing program.

[0054] The information processing program includes a goal information acquisition step S111, a current information acquisition step S112, an intermediate reward calculation step S113, and a reinforcement learning step S115. The information processing program causes the calculation unit 110 to perform various functions such as a goal information acquisition function, a current information acquisition function, an intermediate reward calculation function, and a reinforcement learning function. These functions are executed in the order shown in the flowchart of Fig. 7, but the order can also be changed as appropriate. The various functions overlap with the description of the various functional units of the information processing device 100 described above, and therefore detailed description thereof will be omitted.

[0055] The target information acquisition function acquires the target position 11 of the tip 10f of the robot model 10 in the simulation and the target angle of the tip 10f at the target position 11 as target information (S111: target information acquisition step).

[0056] The current information acquisition function acquires the current position of the tip portion 10f and the angle of the tip portion 10f as current information (S112: current information acquisition step).

[0057] The intermediate reward calculation function calculates the reward given to the agent each time the robot model 10 takes action as an immediate reward value based on the distance 15 from the current position of the tip 10f to the target position 11 and the angle difference 16 between the current angle of the tip 10f and the target angle of the tip 10f at the target position 11, and calculates the cumulative value of the immediate reward value from the start of the simulation to the present as the intermediate reward value (S113: intermediate reward calculation step).

[0058] The reinforcement learning function trains the learning model so that the intermediate reward value at the end of the simulation is maximized (S115: reinforcement learning step).

[0059] (Information processing program and information processing method according to other embodiments) An information processing program according to another embodiment will be described together with an information processing method with reference to Fig. 8. Fig. 8 is a flowchart of the information processing program according to another embodiment. The information processing program according to FIG. 8 differs from the information processing program according to FIG. 7 in that a final reward determination step S114 is added, and further, the processing content of the reinforcement learning step S115 differs in accordance with this addition. The information processing program according to FIG. 8 will be described below together with the information processing method. The information processing program according to FIG. 8 includes a goal information acquisition step S111, a current information acquisition step S112, an intermediate reward calculation step S113, a final reward determination step S114, and a reinforcement learning step S115. The information processing program according to Fig. 8 causes the calculation unit 110 to perform various functions such as a goal information acquisition function, a current information acquisition function, an intermediate reward calculation function, a final reward determination function, and a reinforcement learning function. These functions are executed in the order shown in the flowchart of Fig. 8, but the order can be changed as appropriate. The various functions overlap with the description of the information processing program according to Fig. 7 above, and therefore the overlapping description will be omitted.

[0060] The final reward determination function determines the reward to be given to the agent based on the final result of the action of the robot model 10 as the final reward value (S114: final reward determination step).

[0061] The reinforcement learning function trains the learning model so that the sum of the intermediate reward value and the final reward value is maximized at the end of the simulation (S115: reinforcement learning step).

[0062] According to the information processing device 100 of this embodiment described above, the intermediate reward value is calculated based on the distance 15 and the angle difference 16, so that the robot model 10 can be made to behave more efficiently using reinforcement learning compared to when the intermediate reward value is calculated based only on the distance 15.

[0063] Furthermore, according to the information processing device 100 of this embodiment described above, the intermediate reward value is calculated based on the distance 15, the angle difference 16, and the shortening of the task completion time of the robot model 10. Therefore, compared to the case where the intermediate reward value is calculated based only on the distance 15, reinforcement learning can be used to make the robot model 10 behave more efficiently.

[0064] Furthermore, according to the information processing device 100 of this embodiment described above, the reward value given to the agent is the sum of the intermediate reward value and the final reward value, so that the robot model 10 can be made to behave more efficiently using reinforcement learning compared to when only one of the intermediate reward value and the final reward value is used.

[0065] It should be noted that the present invention is not limited to the information processing device 100, the information processing method, and the information processing program according to the above-described embodiments, and can be embodied in various other modified examples or application examples without departing from the gist of the present invention as set forth in the claims. Also, although the term "information" is used in the above-described embodiments, the term "information" can be replaced with "data," and the term "data" can be replaced with "information." [Explanation of symbols]

[0066] 10 Robot Models 10a Pedestal 10b First arm 10c Second Arm 10d Third Arm 10e End Effector 10f Tip 10g reference point 10h Normal direction 11 Target position 11a Normal direction 12 Workpiece model 14 Object Models 15 distance 16 Angle difference 17 rays of light 18 Contact Mark 20. Robot 22 Workpiece 24 Object 100 Information processing device 110 Arithmetic section 111 Target information acquisition unit 112 Current Information Acquisition Department 113 Interim Remuneration Calculation Department 114 Final Remuneration Determination Department 115 Reinforcement Learning Department 120 Communication I / F (Interface) 130 Input I / F 131 Keyboard (input section) 140 Output I / F 141 Display section 170 Storage section 200 Robot control device 300 workspace 400 Virtual Space 500 Simulation System

Claims

1. An information processing device that learns a learning model used by an agent that handles a robot model in a simulation, a target information acquisition unit that acquires, as target information, a target position of a tip of the robot model in the simulation and a target angle of the tip of the robot model at the target position; a current information acquisition unit that acquires the current position and angle of the tip portion as current information; an intermediate reward calculation unit that calculates a reward to be given to the agent each time the robot model takes action as an immediate reward value based on the distance from the current position of the tip part to the target position and the angular difference between the current angle of the tip part and the target angle of the tip part at the target position, and calculates an intermediate reward value as an integrated value of the immediate reward value from the start of the simulation to the present; a reinforcement learning unit that learns the learning model so that the intermediate reward value at the end of the simulation is maximized; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the immediate reward value is calculated based on a reduction in a task completion time of the robot model in addition to the distance and the angular difference.

3. a final reward determination unit that determines a reward to be given to the agent as a final reward value based on a final result of the behavior of the robot model; Further provided with The information processing apparatus according to claim 1 , wherein the reinforcement learning unit learns the learning model so that a sum of the intermediate reward value and the final reward value at the end of the simulation is maximized.

4. The information processing device according to claim 3, characterized in that the final reward determination unit determines the final reward value to be a maximum value if, in the final result, the tip portion reaches the target position within a predetermined time limit and the angle of the tip portion at the target position reaches a target angle.

5. 4. The information processing apparatus according to claim 3, wherein the final reward determination unit determines the final reward value to be a minimum value when the tip portion comes into contact with a workpiece model during the simulation.

6. 4. The information processing device according to claim 3, wherein the final reward determination unit determines the final reward value to be a minimum value when the tip portion does not reach the target angle at the target position within a predetermined time limit.

7. 2. The information processing apparatus according to claim 1, wherein the distance and the angular difference are normalized before use.

8. The information processing apparatus according to claim 1 , wherein the robot model is an articulated robot having a plurality of arms.

9. A control method used in an information processing device that learns a learning model used by an agent that handles a robot model in a simulation, comprising: The calculation unit of the information processing device a target information acquisition step of acquiring a target position of a tip end of the robot model in the simulation and a target angle of the tip end at the target position as target information; a current information acquiring step of acquiring a current position of the tip portion and an angle of the tip portion as current information; an intermediate reward calculation step of calculating an immediate reward value, which is a reward to be given to the agent each time the robot model takes action, based on the distance from the current position of the tip part to the target position and the angular difference between the current angle of the tip part and the target angle of the tip part at the target position, and calculating an intermediate reward value as an integrated value of the immediate reward value from the start of the simulation to the present; a reinforcement learning step of learning the learning model so that the intermediate reward value at the end of the simulation is maximized; An information processing method characterized by carrying out the above.

10. A control program used in an information processing device that learns a learning model used by an agent that handles a robot model in a simulation, comprising: a calculation unit of the information processing device, a target information acquisition function that acquires, as target information, a target position of a tip of the robot model in the simulation and a target angle of the tip of the robot model at the target position; a current information acquisition function that acquires the current position and angle of the tip portion as current information; an intermediate reward calculation function that calculates a reward to be given to the agent each time the robot model takes action as an immediate reward value based on the distance from the current position of the tip part to the target position and the angular difference between the current angle of the tip part and the target angle of the tip part at the target position, and calculates an intermediate reward value as an integrated value of the immediate reward value from the start of the simulation to the present; a reinforcement learning function that learns the learning model so that the intermediate reward value at the end of the simulation is maximized; An information processing program characterized by performing the above.

Images