Robot, robot control method, and robot control program

Abstract

To provide a robot that operates on the basis of learning using a plurality of parameters, a robot control method, and a robot control program.SOLUTION: The robot includes an acquisition section configured to acquire a plurality of learning parameters defining a relationship between a distal end portion of the robot and a target point when the robot acts on a workpiece, the relationship being a state of the robot, an action learning section configured to perform reinforcement learning by the robot acting from a reference point to the target point based on the plurality of learning parameters, and an action control section configured to control an action based on a result of the reinforcement learning by the action learning section.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a robot, a robot control method, and a robot control program. [Background technology]

[0002] There is a robot system that uses a model constructed using machine learning to construct a predetermined motion pattern when the robot grasps an object (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-162857 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the robot system described in Patent Document 1 has few parameters for constructing a model, and as an example, the relationship (parameters) of the tip of the gripping part or the like of the robot with respect to an object (such as the position and direction of the tip) cannot be accurately expressed. In this way, when the number of parameters for learning is relatively small, there is a risk that the robot's desired operation (for example, automated robot operation) cannot be realized.

[0005] The present disclosure provides a robot that operates based on learning using a plurality of parameters, a method for controlling the robot, and a program for controlling the robot. [Means for solving the problem]

[0006] One embodiment of the robot includes an acquisition unit that acquires multiple learning parameters that define the state of the robot, i.e., the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be processed by the robot; a behavioral learning unit that performs reinforcement learning by having the robot act from a reference point to a target point based on the multiple learning parameters; and a behavioral control unit that controls behavior based on the results of reinforcement learning by the behavioral learning unit. [Effects of the Invention]

[0007] The robot, robot control method, and robot control program of the present disclosure can operate based on learning using multiple parameters. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a robot according to an embodiment. [Figure 2] FIG. 2 is a block diagram for explaining a robot (control unit) according to an embodiment. [Figure 3] 10A and 10B are diagrams for explaining a reference point, a tip of a robot, and a target point. [Figure 4] 10 is a flowchart illustrating a method for controlling a robot according to an embodiment. [Figure 5] FIG. 10 is a diagram for explaining an example of a comparative experiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment will be described below.

[0010] [Robot 100 Overview] First, an overview of a robot 100 according to an embodiment will be described. FIG. 1 is a diagram illustrating a robot 100 according to an embodiment.

[0011] The robot 100 may be an articulated robot (having a plurality of joints 102) that includes a plurality of arms 101 and joints 102 that connect adjacent arms 101, or may be a robot that has a single joint 102. The joint 102 may include, for example, a motor (not shown) that moves the arm 101.

[0012] The robot 100 may be a paint dispensing robot that dispenses paint onto the workpiece 200 from a dispensing unit (not shown) at the tip 103 (tip 103 of the arm 101). In this case, for example, the dispensing unit may be provided with an opening (spray nozzle) that receives paint via a pipe arranged on the arm 101 or the like and dispenses the paint onto the workpiece 200. Alternatively, robot 100 may be a welding robot that performs welding on workpiece 200 using a welding portion at tip 103 (tip 103 of arm 101). The welding may be of various types, including spot welding and arc welding. In this case, for example, the welding portion may include a gun or the like for performing welding, and may have shapes and functions for performing various types of welding. Alternatively, the robot 100 may be a gripping robot that grips an object with a gripping unit (not shown) disposed at the tip 103 (tip 103 of the arm 101). In this case, for example, the gripping unit may be a manipulator in which the base ends of a plurality of gripping pieces are movably connected, and the gripping pieces can be moved closer to or farther away from each other based on the base ends to grip the object. The robot 100 is not limited to the above-described example, and may be a robot used for various purposes.

[0013] The robot 100 acquires multiple learning parameters that define the relationship between the state of its own robot, i.e., the state of its tip 103 (the tip 103 of the arm 101), and the target point 201 when the robot takes action on the workpiece 200. The state of the robot may be the state in which the robot operates. The target point 201 may be a target arrival point of the tip 103 of the arm 101 when the robot 100 operates. Alternatively, the target point 201 may be an operating point (processing point) when the robot 100 operates (processes) the workpiece 200. As a specific example, the target point 201 may be a dispensing point when dispensing paint onto the workpiece 200, a welding point when welding the workpiece 200, a gripping point when gripping an object, or the like. The learning parameters are a plurality of parameters that define the relationship between the tip 103 of the robot 100 (tip 103 of the arm 101) and the target location (target point 201) of the tip 103, and specific examples will be described later.

[0014] The robot 100 performs reinforcement learning by moving from the reference point 121 to the target point 201 based on a plurality of learning parameters, the details of which will be described later. That is, the robot 100 performs reinforcement learning by operating (repeatedly trying) while changing information (for example, parameter values) that defines each of the plurality of learning parameters. That is, the robot 100 learns its own behavior by taking an action in response to changes in the plurality of parameters (changes in state). In other words, the robot 100 (agent) performs learning (reinforcement learning) by setting the state relatively finely by using a plurality of states that are learning parameters, and performing an action based on the state.

[0015] The robot 100 behaves based on the results of the reinforcement learning. That is, after the reinforcement learning, the robot 100 actually performs an action on the workpiece 200. That is, as an example, the robot 100 performs various actions such as discharging paint onto the workpiece, welding the workpiece 200, or grasping an object.

[0016] [More about Robot 100] Next, the robot 100 according to an embodiment will be described in detail. FIG. 2 is a block diagram for explaining the robot 100 (control unit 110) according to an embodiment. FIG. 3 is a diagram for explaining the reference point 121, the tip 103 of the robot, and the target point 201. As shown in FIG.

[0017] 2, the robot 100 includes, for example, a control unit 110. The control unit 110 includes, for example, an acquisition unit 111, a behavior learning unit 112, and a behavior control unit 113. The control unit 110 may be configured, for example, by an arithmetic processing unit of the robot 100. The control unit 110 (for example, an arithmetic processing unit) may realize the functions of each unit (for example, the acquisition unit 111, the behavior learning unit 112, and the behavior control unit 113) by, for example, appropriately reading and executing various programs stored in a storage unit (not shown) such as a memory.

[0018] The acquisition unit 111 acquires a plurality of learning parameters that define the state of the robot 100, that is, the relationship between the tip 103 of the robot 100 and the target point 201 when the robot 100 acts on the workpiece 200 to be processed. The acquisition unit 111 may acquire a plurality of learning parameters from an external device (not shown), for example, via a communication unit (communication interface) (not shown). The external device may be a user terminal used by a user of the robot 100, a server, or the like. The user terminal may be, for example, a desktop, a laptop, or the like. Alternatively, the acquisition unit 111 may acquire multiple learning parameters from an external memory (not shown), for example, when an external memory (not shown) in which multiple learning parameters are stored is connected to an interface (not shown) of the robot 100.

[0019] The state of the robot 100 may be the state when the robot is moving (acting), or the like. The state represents the current state of the environment surrounding the robot 100 (agent). In the reinforcement learning (posture search of the robot 100) of this embodiment, the "state" is represented by a plurality of learning parameters. The target point 201 (see FIG. 3 ) may be a target arrival point of the tip 103 of the arm 101 when the robot 100 operates. Alternatively, the target point 201 may be an operating point (processing point) when the robot 100 operates (processes) the workpiece 200. As a specific example, the target point 201 may be a dispensing point when dispensing paint onto the workpiece 200, a welding point when welding the workpiece 200, a gripping point when gripping an object, or the like. The learning parameters may be a plurality of parameters that define the relationship between the tip 103 of the robot 100 (the tip 103 of the arm 101) and the target location (target point 201) of the tip 103.

[0020] The behavior learning unit 112 performs reinforcement learning by having the robot 100 take action from the reference point 121 to the target point 201 based on a plurality of learning parameters. That is, the behavior learning unit 112 performs reinforcement learning by operating (repeatedly trying) while changing information (for example, parameter values) that defines each of a plurality of learning parameters. That is, the behavior learning unit 112 learns the behavior by having the robot itself take action in accordance with changes in a plurality of parameters. In other words, the behavior learning unit 112 sets the states relatively finely by using a plurality of states that are learning parameters, and performs learning (reinforcement learning) by having the robot 100 (agent) take actions based on the states.

[0021] The behavior learning unit 112 may perform reinforcement learning based on multiple learning parameters, including at least two of the following (1) to (7). The behavior learning unit 112 may also perform normalization as preprocessing of the learning parameters when performing reinforcement learning. The normalization (preprocessing) may be, for example, a process of aligning the units of different learning parameters and relatively lowering the values ​​within a predetermined range by multiplying the values ​​by a certain coefficient. More specifically, the normalization (preprocessing) may be a process of transforming the learning data used in reinforcement learning, i.e., the values ​​of the explanatory variables and the objective variable, so that they are easier to handle in reinforcement learning calculations. This normalization (preprocessing) relatively lowers the values ​​of the explanatory variables, thereby enabling the robot 100 (behavior learning unit 112) to suppress fluctuations in weight updates of the neural network and facilitate weight convergence. Furthermore, this normalization (preprocessing) relatively reduces the value of the objective variable, thereby relatively reducing the value of the loss function and facilitating error convergence.

[0022] That is, the behavior learning unit 112 may perform reinforcement learning based on a plurality of learning parameters, for example, parameters including the following (1) to (4): Note that FIG. 3 shows an example of the reference point 121, the tip 103 of the robot 100 (arm 101), and the target point 201. (1) Normalized relative position from reference point 121 to target point 201 (2) Normalized relative position from the reference point 121 to the tip 103 (3) A normalized direction vector indicating the direction from the tip 103 to the target point 201 (4) The normalized Euclidean distance between the tip 103 and the target point 201

[0023] The reference point 121 may be a reference position of the robot 100 (articulated robot), that is, a home position of the posture of the arm 101, or the like. The target point 201 may be a target, or may be a portion of the workpiece 200 to be processed (machined) by the tip 103 of the robot 100. The above (1) is a learning parameter that normalizes the relative position from the reference point 121 to the target point 201. That is, the learning parameter (1) is based on, for example, the distance from the reference point 121 of the arm 101 to the target point 201 (target), and is fixed for each step of movement of the arm 101, so is a parameter that does not change during learning of one step. As an example, when the coordinates of the reference point 121 are set to [0.0, 0.0, 0.0], the coordinates of the target point 201 (target) may be defined as the relative position.

[0024] The tip 103 is the tip 103 of the arm 101, and if the tip 103 of the arm 101 is shaped like a gun, it may be referred to as the "gun tip." The "gun tip" is also called the "target point," and the term "gun tip" will be used hereinafter as a representative term. The above (2) is a learning parameter that normalizes the relative position from the reference position (reference point 121) of the robot 100 to the tip 103 (tip of the gun) of the arm 101. That is, the learning parameter (2) is based on, for example, the distance from the reference position (reference point 121) of the robot 100 to the tip 103 (tip of the gun) of the arm 101, and is a parameter that is constantly monitored because it changes for each step of the movement of the arm 101 and depending on the learning situation.

[0025] Regarding the normalization (preprocessing) of (1) and (2) above, for example, each value indicating the three-dimensional coordinates X, Y, and Z is divided by the maximum value of each value so that the value falls within the range of -1.0 to 1.0. As a specific example, if the X values ​​are

[0100] ,

[0200] , and

[0300] , the behavior learning unit 112 (robot 100) normalizes each of the three values ​​(X values) by dividing each of the three X values ​​by "300." Note that the Y and Z values ​​can also be normalized in the same way as the X value. Furthermore, for (1) and (2) above, the values ​​fall within the range of -1.0 to 1.0 due to normalization, and there are three components of the three-dimensional coordinates (X, Y, Z), so there are three sizes.

[0026] The above (3) is a learning parameter obtained by converting the direction toward the aim point 201 (target) into a normalized vector of X, Y, and Z rotations, with the tip 103 (tip of the gun) of the robot 100 as the reference point. In other words, the learning parameter of the above (3) is a vector that indicates the direction from the tip 103 (tip of the gun) of the robot 100 toward the aim point 201 (target). Because it is a normalized vector, each component of the X, Y, and Z rotations falls within the range of -1.0 to 1.0. Regarding (3) above, the value falls within the range of -1.0 to 1.0 due to normalization, and there are three sizes due to the rotation (three components (RX, RY, RZ)) of the three-dimensional coordinates (X, Y, Z).

[0027] The above (4) is a learning parameter obtained by normalizing the Euclidean distance between the tip 103 (tip of the gun) of the robot 100 and the target point 201 (target). The learning parameter in the above (4) is a parameter that indicates how far the tip 103 (tip of the gun) of the robot 100 is from the target point 201 (target). The normalization (preprocessing) in the above (4) is performed, for example, by dividing the value of the Euclidean distance (multiple distances) for each step by the maximum value of the multiple distance values, so that the value falls within the range of -1.0 to 1.0. The distance is expressed as one component, so it has one size.

[0028] The behavior learning unit 112 may perform reinforcement learning based on the multiple learning parameters, for example, parameters including the following (5): (5) A normalized angular difference between the direction 103a (see FIG. 3) in which the tip 103 actually faces and the preset reference direction 201a (see FIG. 3) in which the tip 103 faces relative to the target point 201.

[0029] Regarding (5) above, this is a learning parameter that normalizes the difference (angle difference) between the direction 103a (angle) in which the tip 103 (gun tip) of the robot 100 is actually pointing and the orientation (predetermined processing (processing) orientation) (the orientation 201a in which the tip 103 (gun tip) is pointing) (angle) of the tip 103 (gun tip) when processing (machining) with respect to the target point 201 (target). This learning parameter represents the amount of rotation required to align the tip 103 (gun tip). Regarding the normalization (preprocessing) of (5) above, for example, the value of the angle difference (multiple angle differences) for each step is divided by the maximum value of the multiple angle differences so that it falls within the range of -1.0 to 1.0. The angle difference is expressed as one component and therefore has one size.

[0030] When the robot 100 is an articulated robot, the behavior learning unit 112 may perform reinforcement learning based on a plurality of learning parameters including, for example, the following (6): (6) Normalized angles of each of the multiple arms 101 of the robot 100 (articulated robot)

[0031] The above parameter (6) is a learning parameter obtained by normalizing the angle of the direction 101a (see FIG. 3) in which each of the multiple arms 101 constituting the articulated robot is currently facing (the current angle of the joint 102 of each arm 101 (each axis)), i.e., the angle of the joint 102 (actuator) that moves each arm 101. As an example, if the articulated robot has axes 1 (first arm) to 6 (sixth arm), normalization is performed so that the angle falls within the range of 0.0 to 1.0 according to the soft limits (upper and lower angle limits) of each axis. In other words, the parameter (6) is parameterized in the same unit. This makes it possible to treat the parameter (6) as a continuous value between the upper and lower limits of the normalized angle. The parameter (6) has a size corresponding to the number of arms 101 (axes), i.e., six sizes if there are six arms 101 (axes).

[0032] The behavior learning unit 112 may perform reinforcement learning based on the following parameter (7), for example, as the plurality of learning parameters. (7) Whether the tip 103 has come into contact with the workpiece 200 (collision determination)

[0033] Regarding (7) above, as an expression indicating the proximity between the tip 103 (gun tip) of the arm 101 and the workpiece 200, a finite-length rod (ray) (not shown) may be attached to the tip 103 (gun tip), and the learning parameter may indicate whether the ray has come into contact with the workpiece 200. The length of the rod (ray) may be changeable. The value may take the value of "0" or "1" by indicating that the tip 103 has come into contact with the workpiece 200 as a value of "0" and that the tip 103 has not come into contact with the workpiece 200 as a value of "1." As an example of this embodiment, five rods (rays) may be attached to the tip 103, and each of the five rods (rays) may be expressed using three sizes: whether or not it has hit the workpiece 200; whether or not the rod (ray) has contacted the workpiece 200; and if the rod (ray) has contacted the workpiece 200, the distance from the origin (reference point 121) of the rod (ray) when it has contacted the workpiece 200 (a value of "1" is set for non-contact). The above-mentioned rods (rays) may be defined not only in the tip 103 of the arm 101, but also in the extension direction or perpendicular direction of the shape of the tip 103 (gun tip). Even in this case, the number of rods (rays) may be arbitrary, and the number of elements to be observed may be increased by increasing the size. This allows the robot 100 of this embodiment to simplify the conventional contact determination, which requires a large amount of calculation. It should be noted that the above content is merely an example, and one to four rods (rays) may be attached to the tip portion 103, or six or more rods (rays) may be attached to the tip portion 103. In this case, each rod (ray) may take three sizes. Furthermore, the size that each rod (ray) takes is not limited to the three sizes described above, and it may take at least one of the three sizes described above, or a size different from the three sizes described above. In other words, the purpose of the above content is to reduce (compress) the amount of calculation, which was a heavy load in conventional contact detection.

[0034] The behavior control unit 113 controls the behavior based on the results of reinforcement learning by the behavior learning unit 112. That is, after reinforcement learning, the behavior control unit 113 actually performs an action on the workpiece 200. That is, as an example, based on the control of the behavior control unit 113, the robot 100 performs various actions such as discharging paint onto the workpiece, welding the workpiece 200, or gripping an object.

[0035] The behavior control unit 113 may control the behavior of the joint unit 102 based on the results of reinforcement learning. That is, the robot 100 may drive the motors and actuators (not shown) of the joint unit 102 by the behavior control unit 113 performing control based on the results of reinforcement learning, and move the arm 101 in response to the driving of the motor, thereby performing an operation on the workpiece 200.

[0036] [Control method] Next, a method for controlling the robot 100 according to an embodiment will be described. FIG. 4 is a flowchart illustrating a method for controlling the robot 100 according to an embodiment.

[0037] In step ST101, the acquisition unit 111 acquires a plurality of learning parameters that define the state of the robot 100, that is, the relationship between the tip 103 of the robot 100 and the target point 201 when the robot 100 acts on the workpiece 200.

[0038] In step ST102, the behavior learning unit 112 performs reinforcement learning by having the robot 100 act from the reference point 121 to the target point 201 based on the plurality of learning parameters acquired in step ST101. That is, the behavior learning unit 112 may perform reinforcement learning based on, for example, any two or more of the following parameters (1) to (7) as the plurality of learning parameters. (1) Normalized relative position from reference point 121 to target point 201 (2) Normalized relative position from the reference point 121 to the tip 103 (3) A normalized direction vector indicating the direction from the tip 103 to the target point 201 (4) The normalized Euclidean distance between the tip 103 and the target point 201 (5) A normalized angular difference between the actual direction of the tip 103 and a preset reference direction of the tip 103 relative to the target point 201. (6) Normalized angles of each of the multiple arms 101 of the robot 100 (articulated robot) (7) Whether the tip 103 has come into contact with the workpiece 200 (collision determination)

[0039] In step ST103, the behavior control unit 113 controls the behavior based on the result of the reinforcement learning in step ST102. The behavior control unit 113 may control the behavior (movement) of the joint unit 102 of the robot 100 (for example, an articulated robot, etc.) based on the result of the reinforcement learning.

[0040] [Experimental Results] Next, a comparative experiment between the robot of this embodiment and a comparative (conventional) robot will be described. FIG. 5 is a diagram for explaining an example of a comparative experiment. The vertical axis in Figure 5 shows the reward during reinforcement learning, and a value of "1" indicates a high reward. The horizontal axis in Figure 5 shows the number of learning steps. In FIG. 5, the dashed line indicates the robot of this embodiment, and the solid line indicates a comparative example (conventional).

[0041] The robot of this embodiment, shown by the dashed line, performed reinforcement learning using the learning parameters (1) to (7) described above, while the comparison (conventional) robot, shown by the solid line, performed reinforcement learning using a smaller number of learning parameters than this embodiment. As shown by the dashed line representing the robot of the present embodiment, it can be seen that at the stage where the number of learning steps S is small (S < 4M), the reward is higher compared to reinforcement learning using a small number of learning parameters of the comparison target (conventional). Also, at the stage where the subsequent learning steps increase (S < 10M), in the present embodiment shown by the dashed line, the reward increases gently compared to the comparison target (conventional), and then (10M < S), the reward becomes "1". That is, in the robot of the present embodiment, at all stages of the number of learning steps, the variation in the reward is less compared to the comparison target (conventional). As a result, it can be understood that the stability when operating based on reinforcement learning is higher compared to the comparison target (conventional). From the above, it can be seen that when performing reinforcement learning using the robot of the present embodiment, that is, the learning parameters (1) to (7) described above, it is superior compared to the case where the number of learning parameters of the comparison target (conventional) is less than that of the present embodiment.

[0042] [Regarding Functions and Circuits] Next, the functions and circuits of the control unit 110 of the robot 100 described above will be explained. Each part of the control unit 110 of the robot 100 may be realized as a function of a computer's arithmetic processing unit or the like. That is, the acquisition unit 111, the action learning unit 112, and the action control unit (control unit 110) of the robot 100 may be respectively realized as an acquisition function, an action learning function, and an action control function (control function) by a computer's arithmetic processing unit or the like. The control program of the robot 100 can cause the robot 100 (computer) to realize the above-described various functions. The control program of the robot 100 may be recorded in a non-temporary storage medium readable by the robot 100 (computer), such as a memory, a solid-state drive, a hard disk drive, or an optical disk. The storage medium may be, for example, rephrased as a non-temporary computer-readable medium that stores the control program of the robot 100. As described above, each unit of the control unit 110 of the robot 100 may be realized by an arithmetic processing unit of a computer or the like. The arithmetic processing unit or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the control unit 110 of the robot 100 may be realized as a circuit that constitutes the arithmetic processing unit or the like. In other words, the acquisition unit 111, behavior learning unit 112, and behavior control unit 113 (control unit 110) of the robot 100 may be realized as an acquisition circuit, behavior learning circuit, and behavior control circuit (control circuit) that constitute the arithmetic processing unit of a computer or the like. The storage unit (memory, etc.) of the robot 100 may be realized as a storage function including the functions of an arithmetic processing unit, etc. The storage unit (memory, etc.) of the robot 100 may be realized as a memory circuit by being configured with an integrated circuit, etc. The storage unit (memory, etc.) of the robot 100 may be realized as a storage device by being configured with a plurality of devices, for example.

[0043] The robot 100 can be configured by combining one or any combination of the above-described multiple units. In this disclosure, the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0044] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that each aspect described below is an example at the time of filing, and this embodiment is not limited to the aspects described below. In other words, this embodiment is not limited to the aspects described below, and may be realized by appropriately combining the above-mentioned parts. Furthermore, a lower-level aspect may be able to cite any of the higher-level aspects. The effects described below are merely examples, and the effects of each aspect are not limited to those described below. Each aspect may, for example, achieve at least one of the effects described below.

[0045] (Aspect 1) One embodiment of the robot includes an acquisition unit that acquires multiple learning parameters that define the state of the robot, i.e., the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be processed by the robot; a behavioral learning unit that performs reinforcement learning by having the robot act from a reference point to a target point based on the multiple learning parameters; and a behavioral control unit that controls behavior based on the results of reinforcement learning by the behavioral learning unit. This allows the robot to operate based on the results of learning (reinforcement learning) using multiple parameters. Also, by using the multiple learning parameters that have been set, the robot can more accurately represent the relationship between the tip (gun tip) and the target point, allowing it to function as a state.

[0046] (Aspect 2) In one embodiment, the robot is a multi-joint robot having a plurality of arms and joints connecting adjacent arms, and the behavior control unit may control the behavior of the joints based on the results of reinforcement learning. This allows the robot, even if it is an articulated robot equipped with multiple arms and joints that move each arm, to operate based on the results of learning (reinforcement learning) using multiple parameters.

[0047] (Aspect 3) In one embodiment, the robot may be a paint dispensing robot that dispenses paint onto a workpiece from a dispensing portion at the tip, or a welding robot that welds onto a workpiece using a welding portion at the tip. This allows the robot, even if it is a variety of robots such as a paint-dispensing robot or a welding robot, to operate based on the results of learning (reinforcement learning) using a plurality of parameters.

[0048] (Aspect 4) In one embodiment of the robot, the behavioral learning unit may perform reinforcement learning based on multiple learning parameters including a normalized relative position from the reference point to the target point, a normalized relative position from the reference point to the tip, a normalized direction vector indicating the direction from the tip to the target point, and a normalized Euclidean distance between the tip and the target point. As a result, the robot can transform the learning data used in reinforcement learning, i.e., the values ​​of the explanatory variables and the objective variable, through normalization so that they are easier to handle in reinforcement learning calculations. Also, by relatively lowering the values ​​of the explanatory variables through normalization (preprocessing), the robot can suppress the fluctuation in the weight updates of the neural network, making it easier for the weights to converge. Also, by relatively lowering the values ​​of the objective variable through normalization (preprocessing), the robot can relatively lower the value of the loss function, making it easier for the error to converge.

[0049] (Aspect 5) In one embodiment of the robot, the behavioral learning unit may perform reinforcement learning based on multiple learning parameters, including a normalized angular difference between the direction in which the tip portion actually faces and a predetermined reference direction in which the tip portion faces relative to the target point. This allows the robot to achieve the same effect as the normalization described above.

[0050] (Aspect 6) In one aspect of the robot, if the robot is an articulated robot, the behavior learning unit may perform reinforcement learning based on multiple learning parameters including normalized angles of each of the robot's multiple arms. This allows the robot to achieve the same effect as the normalization described above.

[0051] (Aspect 7) In one aspect of the robot, the behavior learning unit may perform reinforcement learning based on a parameter indicating whether or not the tip portion has come into contact with the workpiece, as the plurality of learning parameters. This allows the robot to operate based on the results of learning using multiple parameters (reinforcement learning).

[0052] (Aspect 8) In one embodiment of the robot control method, the robot executes an acquisition step in which the robot acquires multiple learning parameters that define the state of the robot, i.e., the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be machined; a behavioral learning step in which the robot performs reinforcement learning by acting from a reference point to a target point based on the multiple learning parameters; and a behavioral control step in which the robot controls its behavior based on the results of the reinforcement learning in the behavioral learning step. As a result, the control method can achieve the same effects as the robot of the above-described aspect.

[0053] (Aspect 9) One embodiment of a robot control program provides the robot with an acquisition function that acquires multiple learning parameters that define the state of the robot, i.e., the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be processed; a behavioral learning function that performs reinforcement learning by having the robot act from a reference point to a target point based on the multiple learning parameters; and a behavioral control function that controls behavior based on the results of reinforcement learning from the behavioral learning function. As a result, the control program can achieve the same effect as the robot of the above-described aspect. [Explanation of symbols]

[0054] 100 robots 101 Arm 102 Joints 103 Tip 110 control section 111 Acquisition Department 112 Behavioral Learning Department 113 Behavior Control Unit 121 Reference point 200 Workpiece 201 target point

Claims

1. an acquisition unit that acquires a plurality of learning parameters that define the state of the robot, that is, the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be machined; a behavior learning unit that performs reinforcement learning by causing the robot to act from a reference point to a target point based on the plurality of learning parameters; a behavior control unit that controls behavior based on the results of reinforcement learning by the behavior learning unit; A robot equipped with:

2. the robot is an articulated robot including a plurality of arms and joints connecting adjacent arms, The behavior control unit controls the behavior of the joint unit based on the result of reinforcement learning. The robot of claim 1 .

3. The robot is a paint dispensing robot that dispenses paint onto a workpiece from a dispensing portion at the tip, or a welding robot that welds onto a workpiece using a welding portion at the tip. The robot according to claim 1 or 2.

4. The behavior learning unit includes, as the plurality of learning parameters: a normalized relative position from the reference point to the target point; a normalized relative position from the reference point to the tip; a normalized direction vector indicating a direction from the tip to the target point; a normalized Euclidean distance between the tip and the target point; Reinforcement learning is performed based on parameters including The robot of claim 1 .

5. The behavior learning unit performs reinforcement learning based on the plurality of learning parameters, which include a normalized angle difference between a direction in which the tip portion actually faces and a predetermined reference direction in which the tip portion faces relative to the target point. The robot of claim 1 .

6. When the robot is an articulated robot, the behavior learning unit performs reinforcement learning based on parameters including normalized angles of each of a plurality of arms of the robot as the plurality of learning parameters. The robot of claim 1 .

7. The behavior learning unit performs reinforcement learning based on a parameter indicating whether or not the tip portion has come into contact with a workpiece as the plurality of learning parameters. The robot of claim 1 .

8. The robot an acquisition step of acquiring a plurality of learning parameters that define the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be machined, the learning parameters being the state of the robot; a behavior learning step in which the robot performs reinforcement learning by acting from a reference point to a target point based on the plurality of learning parameters; a behavior control step of controlling behavior based on a result of reinforcement learning in the behavior learning step; A method for controlling a robot that performs the above.

9. To the robot, an acquisition function for acquiring a plurality of learning parameters that define the relationship between the tip of the robot and a target point when the robot acts on a workpiece to be machined, which is a state of the robot; a behavior learning function that performs reinforcement learning by causing the robot to act from a reference point to a target point based on the plurality of learning parameters; a behavior control function that controls behavior based on the results of reinforcement learning of the behavior learning function; A robot control program that makes this possible.

Images