Robot control system, robot control method, and computer program

Abstract

To generate a track for keeping optimality to some degree while more flexibly avoiding a collision between robots as compared with a conventional way.SOLUTION: A server 2 includes: a machine learning section 407 which learns a table showing eligibility for avoiding a collision between a tool of a first robot and a second robot for each of a plurality of actions by executing learning processing based on a reinforcement learning algorithm; an optimum path inference section 408 which infers an eligible action to be optimum for avoiding a collision when the tool is brought close to the second robot of the plurality of actions on the basis of a model; and a robot control section 404 which controls the first robot so that the tool is moved along a first standard path 51 when the tool is not brought close to the second robot and controls a first robot 3A so that eligible action inferred by the optimum path inference section 408 is executed when the tool is brought close to the second robot.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This invention relates to a trajectory generation technique that maintains a certain degree of optimality while avoiding collisions between multiple robots. [Background technology]

[0002] Traditionally, robots, such as industrial robots, have been used to mass-produce and rapidly manufacture products such as automobiles, machinery, electrical equipment, electronic devices, food products, cosmetics, and pharmaceuticals. Robots contribute to labor and efficiency improvements. Furthermore, multiple robots are sometimes deployed in the same space to perform tasks. When these robots are given a target orientation, technologies have been proposed to achieve the most efficient trajectory generation possible while avoiding collisions between the robots.

[0003] The method described in Patent Document 1 is a method for efficiently avoiding collisions between two robots in a robot system comprising a first robot having a first work area and a second robot having a second work area. The first work area overlaps with the second work area, thereby defining the overlapping work area, and the method includes the steps of providing first operation information related to the operation of the first robot performed by the first robot, retrieving the first operation information, and determining, with respect to the operation of a plurality of second robots, whether the operation of the second robot involves a risk of collision between the first robot and the second robot. If it is determined that the operation of the second robot involves a risk of collision between the first robot and the second robot, another robot is selected from the available robots as the second robot, and the steps of providing the first operation information and determining whether there is a risk are repeated. If it is determined that there is no risk, the operation of the second robot is performed.

[0004] The method described in Non-Patent Document 1 involves generating trajectories for each arm of multiple robots, creating a graph with the position along the path as a parameter, and adjusting the velocity to obtain trajectories that do not interfere with each other.

[0005] According to the method described in Patent Document 2, a list of motion commands is generated that includes the start and end points of the trajectories of multiple robot arms, and the order in which each trajectory is generated is determined based on the motion command list. For a specific robot arm in the motion command list, a trajectory is generated based on the start and end points so as to avoid obstacle spaces registered in the obstacle memory for the trajectory generation of other robot arms. Then, when the robot arm is operated along the generated trajectory, the swept space swept by the body of the arm is added to the obstacle memory as an obstacle space that other robot arms should avoid. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 6556245 [Patent Document 2] Japanese Patent Publication No. 2019-193975 [Non-patent literature]

[0007] [Non-Patent Document 1] G. Sanchez and J.-C. Latombe, "On delaying collision checking in PRM planning: Application to multi-robot coordination" International Journal of Robotics Research, vol.21, no.1, pp.5-26, 2002 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Incidentally, a wide variety of products are being developed one after another to meet the needs of consumers, and introducing a dedicated robot for each product would be costly. Therefore, general-purpose robots are being introduced, and the robots are being used to perform tasks specific to each product.

[0009] The conventional methods described above can avoid anticipated collisions between robots, but they struggle to respond flexibly to unexpected situations. For example, they have difficulty responding if there is a delay in the movement of one of the robots. In other words, conventional methods lack versatility, and it is difficult to achieve efficient motion planning while avoiding collisions even when motion delays occur. Furthermore, conventional learning methods require a great deal of training time and motion planning time to learn avoidance strategies for all collisions. Moreover, if the robot shape becomes complex or the number of robots increases, a great deal of computation time is required to optimize the trajectory to the target posture of the robots while avoiding collisions.

[0010] In view of these problems, the present invention aims to enable the realization of motion planning that avoids collisions between robots more flexibly than conventional methods while maintaining a certain degree of optimality. [Means for solving the problem]

[0011] A robot control system according to one embodiment of the present invention includes a learning means that learns a model representing the suitability of each of a plurality of actions to avoid collisions between a moving object, which is all or part of the robot, and an object, by performing a learning process based on a reinforcement learning algorithm, A determination means for determining whether the moving object and the object are approaching each other, Among the multiple actions described above, the moving target and the object are approaching each other. If it is determined that An inference means that infers a suitable action that is optimal for avoiding the collision based on the model, and the moving object and the object are not in close proximity. It was determined to be so. In this case, the robot is controlled so that the object to be moved moves along a predetermined path, and the object to be moved and the object are approaching each other. It was determined to be so.In such cases, the control means controls the robot to perform the qualified action inferred by the inference means. The model represents the Q value for each of the multiple actions as the eligibility for each of the multiple states of the positional relationship between the moving target and the object, and the learning means determines that the robot and the object are approaching, and if the Q value for the current state of the robot and the object among the multiple states has not yet been learned, it uses the Q learning algorithm as the reinforcement learning algorithm to execute the learning process.

[0012] Preferably, each of the aforementioned actions is movement or stopping in a different direction from one another.

[0013] The moving object is the robot arm and a tool provided at the tip of the arm, and the determination means determines that the moving object and the object are approaching if the object is present in a predetermined area around the tool, and determines that they are not approaching if the object is not present. [Effects of the Invention]

[0014] According to the present invention, multiple robots can efficiently search for a near-optimal trajectory to reach a target posture while more flexibly avoiding collisions between robots than in conventional methods. Specifically, even if an unexpected situation occurs, such as one robot malfunctioning or experiencing a delay in operation, it is possible to generate a trajectory that maintains a certain degree of optimality while avoiding collisions between robot arms. [Brief explanation of the drawing]

[0015] [Figure 1] This is a flowchart illustrating an example of the overall processing flow of a robotic system. [Figure 2] This figure shows examples of robot mechanism models and control models. [Figure 3] This diagram shows an example of a server hardware configuration. [Figure 4] This figure shows an example of a server's functional configuration. [Figure 5] This figure shows examples of the initial and target poses of the first and second robots. [Figure 6] This flowchart illustrates an example of the process flow for pathfinding using the RRT (Rapidly-Exploring Random Trees) algorithm. [Figure 7] This figure shows examples of the postures of the first and second robots while they are moving. [Figure 8]This is a perspective view showing an example of a monitoring area. [Figure 9] This diagram shows examples of how the monitoring area looks from directly above, directly to the side, and directly below. [Figure 10] This diagram shows the vectors at the center of each monitoring area. [Figure 11] This figure shows an example of a lookup table. [Figure 12] This flowchart illustrates an example of the processing flow for machine learning using the Q-learning algorithm. [Figure 13] This flowchart illustrates an example of the overall processing flow by an action planning program. [Modes for carrying out the invention]

[0016] [1. Overall System Configuration] Figure 1 is a flowchart illustrating an example of the overall processing flow of robot system 1. Figure 2 shows an example of the mechanism model and control model of robot 3. Figure 3 shows an example of the hardware configuration of server 2. Figure 4 shows an example of the functional configuration of server 2.

[0017] As shown in Figure 1, the robot system 1 consists of a server 2 and multiple robots 3, and these robots 3 cooperate to process the workpiece and produce a product.

[0018] Robot 3, as shown in Figure 1 or Figure 2, consists of a base 30, a first arm 311, a second arm 312, a third arm 313, a fourth arm 314, a fifth arm 315, a tool 32, a first drive unit 331, a second drive unit 332, a third drive unit 333, a fourth drive unit 334, a fifth drive unit 335, a sixth drive unit 336, a controller 34, and a communication interface 35, among other components. In Figure 2, dotted lines represent wired or wireless communication paths. Solid lines with arrows represent power transmission.

[0019] In this embodiment, the base 30 is installed on a horizontal surface. The first arm 311 has its base end supported by the base 30 and rotates around a first axis 391 perpendicular to the mounting surface of the base 30. The first drive unit 331 consists of a motor, a reduction gear, and an angle sensor, and rotates the first arm 311. The second drive unit 332 to the sixth drive unit 336 are also composed of a motor, a reduction gear, and an angle sensor, similar to the first drive unit 331.

[0020] The second arm 312 has its base end supported by the tip of the first arm 311 and pivots around the second axis 392, which is perpendicular to both the first axis 391 and the longitudinal direction of the second arm 312. The second drive unit 332 causes the second arm 312 to pivot.

[0021] The third arm 313 has its base end supported by the tip of the second arm 312, and pivots around a third axis 393 that is parallel to the second axis 392. The third drive unit 333 causes the third arm 313 to pivot.

[0022] The fourth arm 314 has its base end supported by the tip of the third arm 313 and rotates around a fourth axis 394 parallel to the longitudinal direction of the third arm 313. The fourth drive unit 334 rotates the fourth arm 314.

[0023] The fifth arm 315 has its base end supported by the tip of the fourth arm 314, and pivots around the fifth axis 395, which is perpendicular to the fourth axis 394. The fifth drive unit 335 causes the fifth arm 315 to pivot.

[0024] The tool 32 has its base end supported at the tip of the fifth arm 315 and rotates around a sixth axis 396 parallel to the longitudinal direction of the fifth arm 315. The sixth drive unit 336 rotates the tool 32. Depending on the task, a hand or a screw fastening device may be used as the tool 32.

[0025] The angle sensor of the first drive unit 331 detects the angle θ1 between the current position of the first arm 311 and the reference position in the coordinate system of the base 30. In other words, it detects how much the first arm 311 has rotated from the reference position. The angle sensor of the second drive unit 332 detects the angle θ2 between the first arm 311 and the second arm 312.

[0026] The angle sensor of the third drive unit 333 detects the angle θ3 between the second arm 312 and the third arm 313. The angle sensor of the fourth drive unit 334 detects the angle θ4 between the current position of the fourth arm 314 and the reference position of the third arm 313 in the coordinate system of the third arm 313. In other words, it detects how much the fourth arm 314 has rotated from its reference position.

[0027] The angle sensor of the fifth drive unit 335 detects the angle θ5 between the fourth arm 314 and the fifth arm 315. The angle sensor of the sixth drive unit 336 detects the angle θ6 between the current position of the tool 32 and the reference position in the coordinate system of the fifth arm 315. In other words, it detects how much the tool 32 has rotated from its reference position.

[0028] The angle sensors of the first drive unit 331 through the sixth drive unit 336 transmit the detected angles θ1 to θ6 to the server 2 via the communication interface 35. Since the lengths of the link sections of the first arm 311 through the fifth arm 315 and the length of the tool 32 are all constant (invariant), the posture of the robot 3 can be uniquely determined if the angles θ1 to θ6 are known.

[0029] The communication interface 35 is either a wired interface device such as a USB (Universal Serial Bus) adapter or a NIC (Network Interface Card), or a wireless communication device such as a Bluetooth adapter or a Wi-Fi adapter, which sends and receives data to and from the server 2.

[0030] The controller 34 controls the first to sixth drive units 331 to 336 so that the first arm 311 to the fifth arm 315 and the tool 32 assume the posture specified in the commands received by the communication interface 35 from the server 2. Alternatively, it controls the tool 32 to operate according to the commands.

[0031] Server 2 directs each of the multiple robots 3 to assume a predetermined posture for processing the workpiece. During this process, it also takes measures to avoid collisions between the robots 3. The following explanation will use a laptop computer as Server 2 as an example.

[0032] As shown in Figure 3, Server 2 consists of a main processor 20, RAM (Random Access Memory) 21, ROM (Read Only Memory) 22, auxiliary storage device 23, communication interface 24, display 25, keyboard 26, and pointing device 27, among others.

[0033] The ROM 22 or auxiliary storage device 23 has computer programs installed on it, including the operating system and the operation planning program 40.

[0034] RAM21 is the main memory of server 2. Computer programs such as the operation planning program 40 are loaded into RAM21 as needed.

[0035] The main processor 20 executes the computer program loaded into RAM 21. The main processor 20 can be a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit), among others.

[0036] The communication interface 24 transmits and receives data to and from the robot 3. The communication interface 24 uses a communication device conforming to the standards adopted by the robot 3.

[0037] The display 25 displays a screen for inputting commands or data, or a screen showing the results of calculations performed by the main processor 20, etc.

[0038] The keyboard 26 and pointing device 27 are input devices for the operator to input commands or data, etc.

[0039] According to the motion planning program 40, functions such as the initial condition storage unit 401, motion planning unit 402, path storage unit 403, robot control unit 404, state determination unit 405, table storage unit 406, machine learning unit 407, and optimal path inference unit 408 shown in Figure 4 are realized. With these functions, the paths (trajectories) of each of the multiple robots 3 are searched using the RRT* (Rapidly-exploring Random Tree star) algorithm, and each robot 3 is controlled according to its respective path. Furthermore, path changes are performed by reinforcement learning to avoid collisions.

[0040] The following describes the processes of each part and each robot 3 shown in Figure 4, using the example of a case where two robots 3 are used to process a workpiece.

[0041] The two robots 3 may be referred to as "First Robot 3A" and "Second Robot 3B" respectively. Similarly, the base 30, tool 32, and angles θ1 to θ6 of the first robot 3A may be referred to as "Base 30A," "Tool 32A," and "Angle θ 1_A ~θ 6_A The second robot 3B's base 30, tool 32, and angles θ1 to θ6 are described as "base 30B", "tool 32B", and "angle θ 1_B ~θ 6_B It may be written as follows:

[0042] [2. Initial Data Storage] Figure 5 shows examples of the initial and target postures of the first robot 3A and the second robot 3B.

[0043] The initial condition memory unit 401 (see FIG. 4) stores, as information regarding the initial conditions and the like of the first robot 3A and the second robot 3B, the first installation position P shown in FIG. 5(A) or FIG. 5(B) O_A and the second installation position P O_B in the coordinates in the work space 10 for each, as well as the first initial posture F S_A , the second initial posture F S_B , the first target posture F G_A , and the second target posture F G_B . In addition, the lengths of each link of the first robot 3A, the length of the tool 32A, the lengths of each link of the second robot 3B, and the length of the tool 32B are stored.

[0044] The work space 10 is the space where the first robot 3A and the second robot 3B exist. The first installation position P O_A and the second installation position P O_B are the positions where the bases 30A of the first robot 3A and the bases 30B of the second robot 3B are installed, respectively.

[0045] The first initial posture F S_A and the second initial posture F S_B are the initial postures of the first robot 3A and the second robot 3B, respectively, and are specified as follows. When the operator inputs a predetermined command, the angles θ 1_A ~θ 6_A are detected by the six angle sensors of the first robot 3A and transmitted to the server 2. These angles θ 1_A ~θ 6_A represent the first initial posture F S_A . Similarly, the angles θ 1_B ~θ 6_B are detected by the six angle sensors of the second robot 3B and transmitted to the server 2. These angles θ 1_B ~θ 6_B represent the second initial posture F S_B .

[0046] The first target posture FG_A and the second target attitude F G_B These are the target postures of the first robot 3A and the second robot 3B, respectively, and the first target posture F G_A The angle θ at that time 1_A ~θ 6_A and the second target attitude F G_B The angle θ at that time 1_B ~θ 6_B These are specified by the operator entering them. Rather than directly entering these angles, the first target attitude F is manually entered. G_A and the second target attitude F G_B The first robot 3A and the second robot 3B each take the angle θ, 1_A ~θ 6_A and angle θ 1_B ~θ 6_B It may also be obtained by detecting it.

[0047] In addition, the three-dimensional shapes of the base 30A, the first arm 311A ​​to the fifth arm 315A, and the tool 32A of the first robot 3A, as well as the three-dimensional shapes of the base 30B, the first arm 311B to the fifth arm 315B, and the tool 32B of the second robot 3B, are stored in the initial condition storage unit 401.

[0048] [3. Control of Robot 3] When the operator inputs the start command to server 2, the first robot 3A will return to the first initial posture F S_A From the first target posture F G_A The second robot 3B is changed to the second initial posture F. S_B From the first target posture F G_B The process of changing to this is performed by the operation planning unit 402 or the optimal path inference unit 408.

[0049] [3.1 Travel following standard routes] Figure 6 is a flowchart illustrating an example of the process flow for pathfinding using the RRT algorithm. Figure 7 shows examples of the postures of the first robot 3A and the second robot 3B during movement.

[0050] The operation planning unit 402 (see Figure 4) searches for the first standard path 51 and the second standard path 52 using the RRT* algorithm.

[0051] The first standard path 51 leads the first robot 3A to the first initial posture F S_A From the first target posture F G_A This is the path of tool 32A when changing to the second standard path 52, which moves the second robot 3B to the second initial posture F. S_B From the second target posture F G_B This is the path of tool 32B when changing to [this].

[0052] The RRT* algorithm is a pathfinding method that is an extension of the RRT algorithm.

[0053] Both the RRT algorithm and the RRT* algorithm are algorithms that search for a path from an initial position (the first node) to a target position (the target node) by connecting branches (line segments) while avoiding obstacles.

[0054] According to the RRT algorithm, a path is found in the procedure shown in Figure 6. The computer adds the first node to the node list (#901). If a straight branch of a certain distance D1 or less can be extended from the added node toward the target node without hitting any obstacles (#902, Yes), that branch is added to the node list (#908), and the RRT algorithm terminates. Step #902 is performed each time a node is added to the node list.

[0055] If it is not possible to extend a branch in this way (No in #902), the computer randomly samples one point from the workspace 10 (#903). Hereafter, the sampled point will be referred to as the "sample point". The node closest to the sample point from the nodes stored in the node list is selected as the parent node (#904). A straight branch is extended from the parent node toward the sample point at a distance D1 (#905).

[0056] If the branch hits an obstacle (Yes in #906), the computer returns to step #903 and starts again, sampling any point in workspace 10, etc.

[0057] On the other hand, if the branch does not hit an obstacle (No in #906), the computer adds the parent node along with the branch to the node list (#907). Then, if a straight branch of less than or equal to distance D1 can be extended from the added node to the target node without hitting any obstacles (Yes in #902), the branch is added to the node list (#908), and the RRT algorithm is terminated. The operation planning unit 402 repeatedly executes steps #903 to #907 until a branch that reaches the target node is obtained.

[0058] By performing this process, the first node and the target node are connected by one or more branches. These connected branches represent the search result, or the desired path. The RRT algorithm is thus simple. However, it does not guarantee optimality.

[0059] On the other hand, according to the RRT* algorithm, when adding a new node in step #907, the computer considers not only the branches connecting the new node to the parent node, but also the branches connecting the new node to each of the parent node's neighboring nodes (provided they do not encounter obstacles). It then selects the branch with the lowest travel cost from among these branches and adds it to the node list. Furthermore, the search may continue even after the first node and the target node are connected by one or more branches. This may lead to the discovery of branches with even lower travel costs, improving the path.

[0060] In other words, the RRT* algorithm calculates the shortest branch to an existing node each time a new node is added, so it can find a path with a lower movement cost than the RRT algorithm. Also, if new nodes are added indefinitely, the shortest path will always be found. However, it is difficult to determine whether the shortest path has been found or not, and the processing time increases as the number of new nodes added increases, so it is desirable to set an upper limit on the search time or the number of steps and then execute the RRT* algorithm.

[0061] The motion planning unit 402 treats objects (e.g., workpieces) that actually exist in the workspace 10 as obstacles when searching for the first standard path 51 and the second standard path 52 using the RRT* algorithm. When searching for the first standard path 51, it also considers areas outside the range of motion of the joint angle of the arm of the first robot 3A as obstacles. Similarly, when searching for the second standard path 52, it considers areas outside the range of motion of the joint angle of the arm of the second robot 3B as obstacles. Information regarding obstacles can also be stored in advance in the initial condition storage unit 401.

[0062] The first standard route 51 and the second standard route 52, once obtained by the operation planning unit 402, are stored in the route storage unit 403.

[0063] The "MoveIt!" tool, which is included in the Robot Operating System (ROS) software for controlling robots, can perform pathfinding based on the RRT* algorithm. Therefore, the motion planning unit 402 may be implemented using this tool.

[0064] By the way, in both the first standard path 51 and the second standard path 52, the first robot 3A is in the first initial posture F S_A Furthermore, the second robot 3B is in the second initial posture F S_B The search is performed when this condition is met. Therefore, if the posture of the second robot 3B changes thereafter, the first robot 3A may collide with the second robot 3B when the posture of the first robot 3A is changed based on the first standard path 51. The same applies if the posture of the first robot 3A changes. A method for avoiding collisions between the two robots 3 will be described later.

[0065] When a start command is input, the motion planning unit 402 calculates the first standard path 51 in the manner described above. However, if the first standard path 51 has been calculated in the past for the same initial posture and target posture combination of the first robot 3A, the calculation may be skipped, and the first standard path 51 stored in the path storage unit 403 may be used in the subsequent processes. The same applies to the second standard path 52.

[0066] The robot control unit 404 controls the first robot 3A and the second robot 3B as follows:

[0067] When the first robot 3A is in the first normal state, the robot control unit 404 controls the first robot 3A so that the tool 32A moves along the first standard path 51. However, when it is in the first avoidance state, the robot control unit 404 controls the first robot 3A so that the tool 32A moves in the direction inferred by the optimal path inference unit 408. The processing of the optimal path inference unit 408 will be described later.

[0068] The first avoidance state is a state in which tool 32A is so close to the second robot 3B that it is about to make contact, as shown in Figure 7, and a collision must be avoided. The first normal state is a state in which tool 32A is not very close to the second robot 3B and there is no need to avoid it. Whether it is the first normal state or the first avoidance state is determined by the state determination unit 405, as will be described later.

[0069] Similarly, when the second robot 3B is in the second normal state, the robot control unit 404 controls the second robot 3B so that the tool 32B moves along the second standard path 52, and when it is in the second avoidance state, the robot control unit 404 controls the second robot 3B so that the tool 32B moves in the direction inferred by the optimal path reasoning unit 408. The second avoidance state is a state in which the tool 32B is so close to the first robot 3A that it is about to come into contact with it and a collision must be avoided, and the second normal state is a state in which the tool 32B is not very close to the first robot 3A and there is no need to avoid it.

[0070] The robot control unit 404 also instructs the first robot 3A or the second robot 3B to perform machining on the workpiece using tools 32A and 32B.

[0071] [3.2 Determining the state] Figure 8 shows the monitoring area M. 1_A ~M 13_A This is a perspective view showing an example. Figure 9 shows the monitoring area M 1_A ~M 13_A This figure shows examples of what the area looks like from directly above, directly to the side, and directly below. Figure 10 shows the monitoring area M. 1_A ~M 13_A This diagram shows the vectors at the center of each.

[0072] When a start command is input, the state determination unit 405 (see Figure 4) determines that the posture of the first robot 3A is the first target posture F. G_AUntil then, the system continues to determine in real time whether the first robot 3A is in the first avoidance state or the first normal state. If it is in the first avoidance state, it continues to determine the proximity state. The proximity state is the state of how close tool 32A is to other robot regions, and the monitoring region M 1_A ~M 13_A This is represented by whether each of them intersects with the domain of another robot. Therefore, the proximity state is 2 13 Yes, it is possible. Furthermore, the proximity state can be said to represent the relative positional relationship between tool 32A and other robotic regions.

[0073] Similarly, the state determination unit 405 determines that the posture of the second robot 3B is the second target posture F G_B Until then, the system continues to determine in real time whether the second robot 3B is in the second avoidance state or the second normal state, and if it is in the second avoidance state, it continues to determine that it is in an approach state. The following describes how the first robot 3A determines whether it is in the first avoidance state or the first normal state, and how it determines that it is in an approach state.

[0074] The state determination unit 405 determines that the posture of the first robot 3A is the first target posture F G_A Until then, the first monitoring position P will be monitored in real time. M_A , 13 monitoring areas M as shown in Figure 8 or Figure 9 1_A ~M 13_A It detects the position of each robot, as well as the positions of other robot areas. 1_A M 2_A M 3_A ... M 13_A Each of these is assigned a monitoring area number: "1", "2", "3", ..., "13".

[0075] First monitoring position P M_A This is the position of tool 32A in the workspace 10 (see Figure 5), the current posture of the first robot 3A, and the first installation position P. O_A It can be determined based on the following. The current posture is determined in real time from the first robot 3A by angle θ. 1_A ~θ6_A It can be identified by obtaining it. Alternatively, it may be identified by simulating the operation of the first robot 3A.

[0076] Monitoring area M 13_A has its center at the first monitoring position P M_A and is a spherical three-dimensional area with a radius length of length L S

[0077] Monitoring area M 7_A ~M 9_A is a spherical three-dimensional area with a radius length of length L 13_A similar to the monitoring area M S As shown in Fig. 9(A), the monitoring areas M 7_A ~M 9_A are arranged horizontally so as to surround the upper part of the monitoring area M 13_A and the four monitoring areas M 13_A 、M 7_A ~M 9_A are in contact with each other.

[0078] Monitoring area M 1_A ~M 6_A is a spherical three-dimensional area with a radius length of length L 13_A similar to the monitoring area M S As shown in Fig. 9(B), the monitoring areas M 1_A ~M 6_A are arranged horizontally so as to surround the side of the monitoring area M 13_A while being in contact with it. 13_A

[0079] Monitoring area M 10_A ~M 12_A is a spherical three-dimensional area with a radius length of length L 13_A similar to the monitoring area M S As shown in Fig. 9(C), the monitoring areas M 10_A ~M 12_A are arranged horizontally so as to surround the lower part of the monitoring area M 1_A and the four monitoring areas M 10_A ~M 13_A are in contact with each other. ​

[0080] Length L S Tool 32A monitors area M 13_A It is set to fit just within the area. Alternatively, the radial cross-section of tool 32A is within the monitoring area M. 13_A It's set to fit just right.

[0081] As tool 32A moves, the first monitoring position P in the workspace 10 M_A Because it changes, the monitoring area M 1_A ~M 13_A The position of each in the workspace 10 also changes. First monitoring position P M_A From monitoring area M 1_A ~M 13_A The direction vector to the center of each is of length L. S If the length is, for example, 0.05 meters, it is as shown in Figure 10. S This is changed depending on the diameter of the robot arm used. Also, the length L S By changing this setting, you can adjust how much margin of error is allowed for collision avoidance.

[0082] The other robot domain is the area within the work domain occupied by another robot 3 (in this example, the second robot 3B), and can be identified by the current posture of the second robot 3B and the three-dimensional shape of each part of the second robot 3B. The posture of the second robot 3B, like the posture of the first robot 3A, can be determined in real time from the angle θ of the second robot 3B. 1_B ~θ 6_B This can be identified by obtaining [a certain value], or by simulating the operation of the second robot 3B.

[0083] The status determination unit 405 monitors area M 1_A ~M 13_A If any of the above intersects with the domain of another robot, it is determined that the first robot 3A is in the first avoidance state; otherwise, it is determined that it is in the first normal state.

[0084] Furthermore, if the first avoidance state is present, the state determination unit 405 determines the approach state. That is, the monitoring area M 1_A ~M 13_A It determines whether each of these areas intersects with another robotic domain.

[0085] The state determination unit 405 determines, by the same method, whether the second robot 3B is in a second avoidance state or a second normal state, and determines whether it is in an approach state.

[0086] [3.3 Reinforcement Learning and Avoidance Path Inference] Figure 11 shows an example of a lookup table 60. Figure 12 is a flowchart illustrating an example of the processing flow of machine learning using the Q learning algorithm.

[0087] As described above, the robot control unit 404 (see Figure 4) moves tool 32A to avoid collision with the second robot 3B when the first avoidance state is in effect. Similarly, when the second avoidance state is in effect, it moves tool 32B to avoid collision with the first robot 3A. Such avoidance actions are inferred based on a lookup table (Q-table), which is a model obtained by machine learning using the Q-learning algorithm, one of the reinforcement learning algorithms.

[0088] The following describes the machine learning and behavioral reasoning processes, using the control of the first robot 3A as an example. These processes are performed by the table storage unit 406, the machine learning unit 407, and the optimal path reasoning unit 408 (see Figure 4).

[0089] The table storage unit 406 has the lookup table (Q table) 60 shown in Figure 11 stored in it beforehand.

[0090] In the lookup table 60, "State" is the proximity state between tool 32A and other robots, i.e., the monitoring area M 1_A ~M 13_AEach of them is in a state of whether or not it intersects with another robot domain, (b1,b2,b3,…,b 13 ) is represented by b. i If = 1, then the monitoring area M i_A This indicates that it intersects with other robotics domains, i If i = 0, it indicates that the lines intersect. However, 1 ≤ i ≤ 13.

[0091] "Action" is the first monitoring position P M_A From 13 monitoring areas M 1_A ~M 13_A This is the movement of tool 32A with its direction of travel being towards any one of the centers. The distance of movement is a constant distance D4. Therefore, the monitoring area M 1_A Movement of tool 32A at a distance D4 toward the center, monitoring area M 2_A Movement of tool 32A at a distance D4 toward the center, ..., monitoring area M 13_A There are 13 possible actions, such as moving tool 32A a distance D4 toward the center of the area. However, the monitoring area M 13_A The center is the first monitoring position P. M_A Since it overlaps with the monitoring area M, 13_A Movement toward the center can be considered essentially a stop (cessation).

[0092] Therefore, the number of combinations of actions and states is 13 × 2 13 There are several possibilities. The lookup table 60 shows one Q-value for each combination. Note that the initial values ​​of these Q-values ​​are predetermined (for example, zero).

[0093] The machine learning unit 407 consists of an action decision unit 471, a state transition calculation unit 472, a reward calculation unit 473, and a table update unit 474, and performs machine learning using the Q-learning algorithm in the procedure shown in Figure 12. The action decision unit 471 and the table update unit 474 correspond to the agent, the state transition calculation unit 472 and the reward calculation unit 473 correspond to the environment, and the lookup table 60 corresponds to the policy.

[0094] The action decision unit 471 sets state s to the approach state at which the first robot 3A enters the first avoidance state (#701 in Figure 12). In other words, it performs initialization. Then, it decides which action a to be performed by the first robot 3A in state s from the 13 actions described above (#702). Action a may be determined simply by randomness, or it may be determined by an ε-greedy policy or a softmax method.

[0095] The state transition calculation unit 472 simulates the first robot 3A performing action a in state s (#703), and after the execution, it checks 13 monitoring areas M 1_A ~M 13_A The system observes (determines) which of the monitoring areas intersects with other robot areas (state s'), and also observes the first monitoring position P. M_A The distance d between this robot and the other robot's domain is calculated (#704). The other robot's domain can be determined by simulating the movement of the second robot 3B in step #703.

[0096] The reward calculation unit 473 calculates the reward r corresponding to the state s' determined by the state transition calculation unit 472 and the calculated distance d. t This will be calculated based on the following compensation regulations (#705).

[0097] <Compensation Regulations> ·r t =-3: When another robot is near this robot's tool 32 and this robot stops. ·r t =-2: When another robot is near this robot's tool 32, and this robot deviates from the path determined by the RRT* algorithm. ·r t =-1: If another robot is near this robot's tool 32, and this robot reaches a result other than the two above (for example, if another robot is near this robot's tool 32 and was able to move along the path determined by the RRT* algorithm) ·r t=2: When the other robot is far away from this robot's tool 32, and this robot was able to move along the path determined by the RRT* algorithm. ·r t =0: When another robot is far away from this robot's tool 32, and this robot deviates from the path determined by the RRT* algorithm. ·r t =1: If another robot is far from the tool 32 of this robot, and this robot reaches a result other than the two above (for example, if another robot is far from the tool 32 of this robot, and this robot stops). In this example, the first robot 3A is the main robot, and the second robot 3B is the other robot. Whether the other robot is nearby or far away can be determined by whether the distance d is greater than or equal to a predetermined distance D2. That is, if the distance d is greater than or equal to distance D2, it is determined to be far away, and if the distance is less than D2, it is determined to be nearby.

[0098] Furthermore, the path determined by the RRT* algorithm is the path of the robot obtained by the motion planning unit 402, and in this example, it is the path of the first robot 3A, i.e., the first standard path 51. As mentioned above, in this example, there are only 13 possible directions of movement for action a, so it is rare for the tool 32A to move in a way that perfectly matches the first standard path 51. Therefore, if the deviation between the position after movement by action a and the first standard path 51 is less than or equal to a predetermined distance D3, it may be determined that the robot was able to move along the path determined by the RRT* algorithm (the first standard path 51).

[0099] Then, the table update unit 474 calculates the current Q value (behavioral evaluation value) based on the following equation (1) (#706), and updates the lookup table 60 based on the following equation (2) (#707). Q π (s t ,a t )=E π [R t |s t =s,a t =a] …… (1) however,

number

number

[0100] On the other hand, state s' is in 13 monitoring areas M 1_A ~M 13_A If none of these areas intersect with other robot areas, that is, if the first robot 3A is in its first normal state (No in #708), then it can be considered that it has avoided other robot areas, i.e., the final state (#709), and the series of processes (episodes) from steps #701 to #708 ends.

[0101] The action decision unit 471 or the table update unit 474 repeatedly executes the episode as long as the termination requirements are not met (No in #710). When the termination requirements are met (Yes in #710), the current machine learning is terminated. The termination requirements are, for example, that the number of times the episode has been executed reaches a predetermined number of times, or that each Q value in the lookup table 60 corresponding to the approach state set as state s in step #701 (the approach state when the first robot 3A enters the first avoidance state) converges to a certain extent or completely.

[0102] When the machine learning performed by the machine learning unit 407 is completed, the optimal path inference unit 408 infers the optimal action, i.e., the optimal direction of movement for tool 32A, based on the lookup table 60. Specifically, 2 13 The system searches the lookup table 60 for a row corresponding to the current approach state among the street approach states, and identifies the action corresponding to the largest Q value among the 13 Q values ​​shown in that row. The direction associated with the identified action is inferred to be the optimal direction of movement.

[0103] The robot control unit 404 then controls the first robot 3A so that the tool 32A moves a distance D4 in the optimal direction inferred by the optimal path inference unit 408.

[0104] After movement, if the state determination unit 405 determines that the first robot 3A is in a state requiring avoidance, machine learning is performed by the machine learning unit 407 in the manner described above, and the optimal path inference unit 408 infers the optimal direction of movement for the tool 32A. Then, the robot control unit 404 controls the first robot 3A so that the tool 32A moves a distance D4 in the inferred optimal direction.

[0105] On the other hand, if the first robot 3A is determined to be in the first normal state, the robot control unit 404 controls the first robot 3A so that the tool 32A returns to the first standard path 51 and moves along the first standard path 51. However, if it is determined again to be in the first avoidance state on the way back to or after returning to the first standard path 51, the machine learning unit 407 and the optimal path inference unit 408 perform the above processing, and the first robot 3A is controlled so that the tool 32A moves a distance D4 in the inferred optimal direction.

[0106] Thus, if the first robot 3A is in a normal state, server 2 controls the first robot 3A so that the tool 32A moves along the first standard path 51. On the other hand, if the first avoidance state is present, server 2 performs machine learning to infer the optimal direction of movement and controls the first robot 3A so that the tool 32A moves in the inferred direction of movement.

[0107] Then, when the first robot 3A reaches the target posture, the robot control unit 404 instructs the tool 32A to perform machining or other operations on the workpiece.

[0108] Note that the lookup table 60 is generated after one machine learning operation. 13 The Q-values ​​related to a portion of the street states are primarily improved (updated). In other words, with each machine learning operation, the lookup table 60 is gradually completed overall.

[0109] Furthermore, if the episode is repeatedly executed until the Q-values ​​of each approaching state set as state s in step #701 converge to a certain extent or completely converge, the convergence of each Q-value of other approaching states becomes easier with each iteration of machine learning, and the number of processes in a single machine learning run decreases. Therefore, the time required for machine learning is shortened. Ultimately, inference can be made based on the lookup table 60 without going through machine learning.

[0110] If the Q values ​​for each approach state determined by the state determination unit 405 have already converged in the lookup table 60 (i.e., have been learned), the machine learning processing by the machine learning unit 407 may be skipped, and the optimal path inference unit 408 may infer the optimal direction of movement.

[0111] The machine learning unit 407 and the optimal path inference unit 408 improve (update) the lookup table for the second robot 3B by performing machine learning in a similar manner, and infer the optimal direction of movement for the tool 32B based on that lookup table. The robot control unit 404 then controls the second robot 3B so that the tool 32B moves in the inferred direction of movement.

[0112] [4. Overall processing flow and effects of this embodiment] Figure 13 is a flowchart illustrating an example of the overall processing flow by the operation planning program 40.

[0113] Next, the overall processing flow of Server 2 based on the motion planning program 40 will be explained using the case of controlling the first robot 3A and the second robot 3B as an example, with reference to the flowchart. Based on the motion planning program 40, Server 2 executes the processing in the procedure shown in Figure 13.

[0114] Server 2 generates the first standard path 51 and the second standard path 52 for the first robot 3A and the second robot 3B, respectively, using the RRT* algorithm (Figure 13, #101).

[0115] Server 2 begins monitoring the status of the first robot 3A and the second robot 3B, and determines the approach status of the first robot 3A while it is in the first avoidance state, and determines the approach status of the second robot 3B while it is in the second avoidance state (#102).

[0116] If the first robot 3A is in the first normal state (Yes in #103), server 2 controls the first robot 3A so that the tool 32A moves along the first standard path 51 (#104). If, as a result of the processing in step #108 described later, the robot returns to the first normal state, server 2 controls the first robot 3A so that the tool 32A returns to the first standard path 51.

[0117] Similarly, if the second robot 3B is in the second normal state (Yes in #103), server 2 controls the first robot 3A so that the tool 32A moves along the first standard path 51 (#104).

[0118] On the other hand, if the first robot 3A is in the first avoidance state (No in #103), server 2 will improve these Q values ​​by performing machine learning (#106) if the Q values ​​corresponding to the current approach state have not been improved, i.e., learned, in the lookup table 60 (No in #105). The machine learning method is as explained in Figure 12. If it has already been learned (Yes in #105), step #106 is skipped.

[0119] Server 2 then infers the optimal direction of travel based on the lookup table 60 (#107) and controls the first robot 3A so that the tool 32A moves a distance D4 in the inferred direction (#108).

[0120] Server 2 performs the same processing when the second robot 3B is in the second avoidance state as it does when the first robot 3A is in the first avoidance state (#105~#108).

[0121] Server 2 selectively executes the processes in step #104 and steps #105 to #108 depending on the state of the first robot 3A. Similarly, it selectively executes the processes in step #104 and steps #105 to #108 depending on the state of the second robot 3B.

[0122] Then, when the first robot 3A reaches the target posture (Yes in #109), the first robot 3A is instructed to perform machining or other operations as needed, and control of the first robot 3A is terminated. Similarly, when the second robot 3B reaches the target posture (Yes in #109), the second robot 3B is instructed to perform machining or other operations as needed, and control of the second robot 3B is terminated.

[0123] According to this embodiment, the first robot 3A or the second robot 3B is controlled according to the optimal direction of travel inferred based on the lookup table 60, so collisions between the first robot 3A and the second robot 3B can be avoided more flexibly than in the conventional method.

[0124] [5. Variations and Application Examples] In this embodiment, the lookup table 60 was prepared as a model using the Q-learning algorithm, but other models may be prepared using other reinforcement learning algorithms. For example, a deep neural network may be generated as a model using a deep reinforcement learning algorithm.

[0125] In this embodiment, the first robot 3A and the second robot 3B were installed in predetermined positions, but the present invention can also be applied when at least one of them is moving.

[0126] In this embodiment, both the first robot 3A and the second robot 3B were controlled to avoid collisions, but only one of them may be controlled. Alternatively, robot 3 may be controlled to avoid collisions with dynamic objects other than robots with arms (e.g., workpieces on a conveyor belt, automatically driven transport wagons, drones, humans, etc.).

[0127] In this embodiment, the case where two robots 3 are present in the workspace 10 was described as an example, but the present invention can also be applied when there are three or more robots.

[0128] The lookup table 60 may be output from server 2 to another computer and used on that computer to control multiple robots to prevent them from colliding with each other.

[0129] In this embodiment, the monitoring area M 1_A ~M 13_A The shape was a sphere, but it may be other shapes. For example, it may be a regular hexagonal prism. Also, the number of monitoring areas may be less than or more than 13. Furthermore, the monitoring areas may be located on parts of the robot 3 other than the tool 32.

[0130] In this embodiment, in the lookup table 60, the state is 2 13 The proximity status of the street was used. In other words, the monitoring area M 1_A ~M 13_A The state used was whether each of these intersected with another robot domain. However, other states may also be used.

[0131] For example, monitoring area M 1_A ~M 13_A The first state is whether each of them intersects with another robot's domain, and the next time point is the monitoring domain M 1_A~M 13_A The second state is whether each of them intersects with another robot's domain, and the monitoring domain M at the next time step. 1_A ~M 13_A The identifier of the monitoring area closest to the first standard path 51 may be used in combination with the identifier. In this case, 13 bits are needed to represent the first state, 13 bits are needed to represent the second state, and 4 bits are needed to represent the identifier. Therefore, a total of 30 bits are needed. Thus, the lookup table 60 contains 2 30 For each street state, there are 13 possible Q values ​​for different actions.

[0132] In this case, the optimal path inference unit 408 obtains these first state, second state, and identifier, and infers the optimal action.

[0133] In this embodiment, the reward calculation unit 473 calculates the reward r based on the above-mentioned reward provision. t The above calculation was performed, but it may also be calculated based on other reward rules. For example, the above reward rules may be further modified to include a condition that "-10" is awarded as a reward if this robot collides with another robot.

[0134] In this embodiment, the machine learning unit 407 used the successful avoidance of other robot regions as a condition for episode termination (see #710 in Figure 12), but it may also use the overlap of tool 32 with other robot regions as a termination condition.

[0135] In this embodiment, the state determination unit 405 determined whether the state was a first normal state or a first avoidance state, and the type of approach state, based on the simulation results. However, the tool 32A may be equipped with a proximity sensor, and the determination may be made based on the detection results from the proximity sensor. Alternatively, the state space may be defined based on information from an external camera image or a camera attached to the robot arm itself, and collision avoidance may be determined.

[0136] In this embodiment, the case where robot 3 is a 6-axis robot was described as an example, but the present invention can also be applied to cases where robot 3 is a multi-joint robot with 7 or more axes or a dual-arm robot.

[0137] Furthermore, the overall configuration or individual parts of the robot system 1, server 2, and robot 3, the content of the processing, the order of processing, etc., can be modified as appropriate in accordance with the spirit of the present invention. [Explanation of symbols]

[0138] 2. Server (Robot Control System) 3 Robots 311 First Arm (Movement Target, Arm) 312 Second Arm (Movement Target, Arm) 313 Third Arm (Moving Object, Arm) 314. Fourth Arm (Movement Target, Arm) 315 Fifth Arm (Moving Object, Arm) 32 Tools (Items to be moved) 3A First Robot (Robot) 3B The Second Robot (Robot, Object, Second Robot) 404 Robot control unit (control means) 405 State determination unit (determination means) 407 Machine Learning Department (Learning Methods) 408 Optimal path inference unit (inference means) 51. The first standard route (route) 52. Second Standard Route (Second Route) 60 Look-up Table (Model) M 1_A ~M 12_A Monitoring area (designated area, second area) M 13_A Monitoring area (designated area, first area)

Claims

1. A learning method that learns a model representing the suitability of each of multiple actions to avoid collisions between the moving target, which is all or part of the robot, and an object, by performing a learning process based on a reinforcement learning algorithm, A determination means for determining whether the moving object and the object are approaching each other, An inference means that, when it is determined that the moving target and the object are approaching each other, infers the most suitable action from among the plurality of actions, which is best suited to avoid the collision, based on the model. A control means that, when it is determined that the moving target and the object are not in close proximity, controls the robot to move the moving target along a pre-calculated path, and when it is determined that the moving target and the object are in close proximity, controls the robot to perform the appropriate action inferred by the inference means. It has, The model represents the Q value for each of the multiple actions as the eligibility for each of the multiple states of the positional relationship between the moving object and the object, The learning means determines that the robot and the object are in close proximity, and if the Q value for the current state of the robot and the object among the multiple states has not yet been learned, it executes the learning process using the Q learning algorithm as the reinforcement learning algorithm. A robot control system characterized by the following features.

2. Each of the aforementioned actions is movement or stopping in a different direction from one another. The robot control system according to claim 1.

3. The object to be moved is the arm of the robot and a tool provided at the tip of the arm, The determination means determines that if the object is present in a predetermined area around the tool, the moving object and the object are approaching each other, and if the object is not present, it determines that they are not approaching each other. A robot control system according to claim 1 or claim 2.

4. The predetermined region comprises a first region encompassing the tool and a plurality of second regions surrounding the first region. Each of the aforementioned multiple states is whether or not the first region and each of the multiple second regions intersect with the object. The robot control system according to claim 3.

5. Each of the aforementioned multiple actions is movement or stopping in either direction of the first region or the multiple second regions. The robot control system according to claim 4.

6. The aforementioned object is a second robot that works in cooperation with the aforementioned robot, The determination means determines whether the moving object and the second robot are approaching each other based on a pre-calculated second path of the second robot. The robot control system according to claim 3.

7. A model learning process is performed by having a computer execute a reinforcement learning algorithm to learn a model that represents the suitability of each of several actions to avoid collisions between the moving target (which is all or part of the robot) and the object, by performing a learning process based on the reinforcement learning algorithm. The computer is made to perform a determination process to determine whether or not the moving object and the object are approaching each other. When it is determined that the moving target and the object are approaching each other, the computer is instructed to perform an inference process based on the model to infer the most suitable action to avoid the collision among the aforementioned multiple actions. If it is determined that the moving target and the object are not in close proximity, the robot is controlled to move along a pre-calculated path; if it is determined that the moving target and the object are in close proximity, the robot is controlled to perform the appropriate action inferred by the inference process performed by the computer, thereby causing the computer to execute a control process. The model represents the Q value for each of the multiple actions as the eligibility for each of the multiple states of the positional relationship between the moving object and the object, When it is determined that the robot and the object are in close proximity, and the Q-value for the current state of the robot and the object among the multiple states has not yet been learned, the model learning process involves causing the computer to execute a learning process using the Q-learning algorithm as the reinforcement learning algorithm. A robot control method characterized by the following:

8. A model learning process is performed by having a computer execute a reinforcement learning algorithm to learn a model that represents, for each of several actions, the suitability of avoiding collisions in each of several states of the positional relationship between the moving target (which is all or part of the robot) and the object, using a Q-value. The computer is made to perform a determination process to determine whether or not the moving object and the object are approaching each other. When it is determined that the moving target and the object are approaching each other, the computer is instructed to perform an inference process based on the model to infer the most suitable action to avoid the collision among the aforementioned multiple actions. If it is determined that the moving target and the object are not in close proximity, the robot is controlled to move along a pre-calculated path; if it is determined that the moving target and the object are in close proximity, the robot is controlled to perform the appropriate action inferred by the inference process performed by the computer, thereby causing the computer to execute a control process. When it is determined that the robot and the object are in close proximity, and the Q-value for the current state of the robot and the object among the multiple states has not yet been learned, the model learning process involves causing the computer to execute a learning process using the Q-learning algorithm as the reinforcement learning algorithm. A computer program characterized by the following features.

Images