CONTROL SYSTEM, MOTION PLANNING DEVICE, CONTROL DEVICE, MOTION PLANNING AND CONTROL METHOD, MOTION PLANNING METHOD, AND CONTROL METHOD

Abstract

To appropriately and efficiently manage the work process of a robot and execute control of the robot.SOLUTION: A motion planning device: creates first unit state transition data that represents a transition from a state, in which a first element that is a component or an assembly and a second element that is a component or an assembly exist independently, to a state in which a third element that is an assembly has been assembled; identifies first grip data that is planned values of relative positions of an end effector of a robot and a first object to be gripped, which is the first element or the second element; and creates a first guide trajectory, which is a final portion of relative trajectories of the first element and the second element for carrying out the transition. A control device causes the end effector to grip the first object to be gripped on the basis of the first grip data and first observation data which is obtained by observation of the position of the first element and the position of the second element, and causes the first object to be gripped to move relatively to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, thereby achieving a state in which the third element has been assembled.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a control system, a motion planning device, a control device, a motion planning and control method, a motion planning method, and a control method. [Background technology]

[0002] A wide variety of parts are required for product assembly work, and in the product manufacturing process, it is necessary to create assembly procedures taking into account the properties of these wide variety of parts, etc. Against this background, techniques for creating part assembly procedures have been proposed.

[0003] For example, there is a technique for generating an assembly sequence taking into consideration the type of parts (see Patent Document 1). This technique analyzes the adjacency relationship of component parts using a predetermined analysis method according to the type of parts, and generates an assembly sequence for the component parts.

[0004] There is also a technology for automating the teaching of an appropriate hand selection task that takes into account compatibility with the part to be gripped (see Patent Document 2). This technology uses a part information DB that stores information about the parts and a hand information DB that stores information about one or more hands to generate teaching information for selecting a hand to grip the part, optimizing the selection of the hand to minimize the assembly time.

[0005] There is also a technique for generating an appropriate assembly sequence (see Patent Document 3). This technique identifies disassembly possible components from an assembly, sets edges between the nodes that are in a geometrically constrained relationship with the components as nodes, generates a disassembly graph in which an evaluation value is assigned to each of the nodes and edges, and determines the disassembly sequence of the assembly using the disassembly graph, thereby generating the assembly sequence.

[0006] There is also technology relating to the generation of motions for robots performing assembly work.

[0007] For example, a method of using machine learning to make a robot learn how to perform a task is being studied (see Non-Patent Document 1). This technology proposes a method of making a robot learn the action of grasping an object by reinforcement learning based on image data obtained from a camera. This method makes it possible to automate at least a part of the series of processes for teaching a robot the action of grasping an object.

[0008] There is also a technology related to a versatile control device (see Patent Document 4). In this technology, a sequence for realizing a final target relative relationship amount is determined from the relative relationship amount at the time when the control of the operation is started, and a control command is generated so as to change the relative relationship amount to the relative relationship amount in the next transition target state. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2019-209446 A [Patent Document 2] International Publication No. 2015 / 178377 [Patent Document 3] International Publication No. 2015 / 177855 [Patent Document 4] JP 2020-082332 A

[0010] [Non-Patent Document 1] Dmitry Kalashnikov, et al. "QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation" arXiv preprint arXiv:1806.10293, 2018. Summary of the Invention [Problem to be solved by the invention]

[0011] However, the above-mentioned technology for creating assembly procedures is merely a method for creating procedure manuals that show the flow of work, and does not take into account how robots operate to assemble parts, etc. Therefore, a method for creating assembly procedures that takes into account the operation of robots is required.

[0012] In addition, the above-mentioned technology for generating the robot's motion may have a drawback in that it may not be possible to properly reflect the motion intended by the administrator who manages the robot when machine learning is introduced. For example, if the motion generated by the model learned by machine learning is different from the motion intended by the administrator, it is difficult to change it because it takes time and cost to re-learn the model to generate the intended motion. In addition, even if the motion is the same, the intention of the work may be different. In addition, the input and the output may not correspond one-to-one. Therefore, the motion of the robot cannot be generated uniquely. In addition, although in machine learning, the model is often learned by learning one task at a time, in reality, the conditions for each task are often given by the administrator from a combination of multiple tasks. Therefore, it is difficult to learn the model in advance while taking into account these many conditions.

[0013] Therefore, it is desirable to create an assembly procedure that allows a manager to easily manage the robot's operation plan for each process, and to control the robot's operation using the created procedure data.

[0014] The present disclosure has been made in consideration of the above circumstances, and aims to provide a control system, a motion planning method, and a control method for appropriately and efficiently managing a robot's work process and controlling the robot, as well as a motion planning device and a control device suitable for configuring such a control system. [Means for solving the problem]

[0015] A control system according to a first aspect of the present disclosure is a control system including a motion planning device and a control device, the motion planning device including a transition creating unit that creates first unit state transition data representing a transition from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled, a gripping data specifying unit that specifies first gripping data that is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly, and a final portion of a relative trajectory of the first element and the second element for performing the transition, the initial position of the first element and the second element. and a guide trajectory creation unit that creates a first guide trajectory that is a common target relative trajectory regardless of the initial relative position of the control device. The control device includes an acquisition unit that acquires the first unit state transition data, the first gripping data, and the first guide trajectory, and acquires first observation data that observes the position of the first element and the position of the second element from a first sensor, and an execution unit that executes a first task to grasp the first gripping object with the end effector based on the first observation data and the first gripping data, and move the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory in a final portion, thereby completing a state in which the third element is assembled in the first unit state transition data.

[0016] A motion planning device according to a second aspect of the present disclosure includes: a transition creation unit that creates first unit state transition data representing a transition from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled; a gripping data identification unit that identifies first gripping data that is a planned value of the relative position of the end effector and the first gripping object when the end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and a guide trajectory creation unit that creates a first guide trajectory that is a final portion of the relative trajectories of the first element and the second element for performing the transition, and is a common target relative trajectory regardless of the initial relative positions of the first element and the second element.

[0017] A control device according to a third aspect of the present disclosure includes: first unit state transition data representing a transition from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled; first gripping data being a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and first gripping data being a final portion of a relative trajectory of the first element and the second element for performing the transition, the final portion being a relative trajectory of the first element and the second element, an acquisition unit that acquires a first guide trajectory, which is a common target relative trajectory regardless of an initial relative position, and acquires first observation data observing the positions of the first element and the second element from a first sensor; and an execution unit that executes a first task to grasp the first grasped object with the end effector based on the first observation data and the first gripping data, and to move the first grasped object, which is one of the first element and the second element, relative to the other element along a target relative trajectory whose final portion includes the first guide trajectory, thereby completing a state in which the third element is assembled in the first unit state transition data. Effect of the Invention

[0018] According to the control system and the operation planning and control method of the present invention, the work process of a robot can be appropriately and efficiently managed and the robot can be controlled. [Brief description of the drawings]

[0019] [Figure 1] FIG. 2 is a diagram showing the configuration of a control system for controlling a robot according to the present embodiment. [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a robot as an example of an object to be controlled. [Diagram 3] FIG. 2 is a block diagram showing the hardware configuration of the motion planning device and a control device. [Figure 4] 2 is a block diagram showing an example of a functional configuration of a motion planning apparatus and a control apparatus. FIG. [Diagram 5] 11 is a diagram showing an example of unit state transition data created in the motion planning device; FIG. [Figure 6] 1A to 1C are diagrams illustrating an example of an assembly procedure and a disassembly procedure using parts A to D. [Figure 7] FIG. 11 is a diagram illustrating an example of a relative position of gripping data. [Figure 8] 13 is an example of a relative trajectory recorded when an assembly of parts A to D is disassembled into an assembly of part D and parts A to C. [Figure 9] FIG. 13 is a diagram showing an example of a guide path. [Figure 10] FIG. 2 is a sequence diagram showing a processing flow of the control system according to the present embodiment. [Figure 11] 13 is a flowchart showing a flow of a motion planning process of the motion planning device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an example of an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or equivalent components and parts are given the same reference numerals. In addition, the dimensional ratios of the drawings are exaggerated for the convenience of explanation and may differ from the actual ratios.

[0021] Fig. 1 is a diagram showing the configuration of a control system for controlling a robot according to this embodiment. As shown in Fig. 1, the control system 1 includes a robot 10, a state observation sensor 14, a motion planning device 20, a state transition control unit 25, and a control device 30. The state transition control unit 25 may be a part of the motion planning device 20 or a part of the control device 30, or may be configured as a device independent of both the motion planning device 20 and the control device 30 as in this embodiment.

[0022] (robot) 2 and 3 are diagrams showing a schematic configuration of a robot 10 as an example of a control target. The robot 10 in this embodiment is a six-axis vertical articulated robot, and an end effector 12 is provided at the tip 11a of an arm 11. The robot 10 performs an assembly task by gripping a part with the end effector 12. In the example of FIG. 3, the end effector 12 is configured to have a pair of hands of a clamping portion 12a, but the end effector 12 may be a suction pad to suck a part. In this specification, gripping a part includes sucking a part.

[0023] As shown in FIG. 2, the robot 10 has an arm 11 with six degrees of freedom having joints J1 to J6. Each of the joints J1 to J6 connects links to each other so that they can rotate in the directions of the arrows C1 to C6 by a motor (not shown). A gripper is connected to the tip of the arm 11 as an end effector 12. Here, a vertical multi-joint robot is taken as an example, but a horizontal multi-joint robot (SCARA robot) may also be used. Also, a six-axis robot is taken as an example, but a multi-joint robot with other degrees of freedom such as a five-axis or seven-axis robot may also be used, or a parallel link robot may also be used.

[0024] (Status Observation Sensor) The state observation sensor 14 observes the state of the robot 10 and outputs the observed data as state observation data. For example, an encoder for the joints of the robot 10, a visual sensor (camera), motion capture, a force-related sensor, etc. are used as the state observation sensor 14. The position and posture of the tip 11a of the arm 11 can be identified from the angle of each joint as the state of the robot 10, and the posture of the part (work object) can be estimated from the visual sensor and / or the force-related sensor. When a marker for motion capture is attached to the end effector 12, the position and posture of the end effector 12 can be identified as the state of the robot 10, and the posture of the part (work object) can be estimated from the position and posture of the end effector 12.

[0025] The force-related sensor is a general term for a force sensor and a torque sensor, and also includes a tactile sensor when the sensor is provided at a portion that comes into contact with a part. The force-related sensor may be provided on the surface of the part where the end effector 12 grips a part or on a joint part within the end effector 12 so as to detect the force that the end effector of the robot 10 receives from the part. The force-related sensor is, for example, a sensor that detects a one-element or multiple-element force of one axis, three axes, or six axes as the state of the robot 10. By using the force-related sensor, it is possible to grasp more accurately how the end effector 12 grips the part, i.e., the attitude of the part, and to perform appropriate control.

[0026] In addition, the position and posture of the end effector 12 itself and the part being gripped by the end effector 12 can also be detected as the state of the robot 10 by a visual sensor.

[0027] In this way, the state of the end effector 12 and the gripped part can be detected by various sensors that are the state observation sensor 14. Furthermore, the detection results of the various sensors can be acquired as state observation data. The state observation sensor 14 is an example of a first sensor. Furthermore, the first sensor may be one sensor, or may be a collective term for sensors in which the sensor that observes the first element and the sensor that observes the second element are different from each other. Furthermore, the state observation sensor 14 is an example of a second sensor. The second sensor may be the same sensor as the first sensor.

[0028] (Motion planning device / control device) Next, the configurations of the motion planning device 20 and the control device 30 will be described.

[0029] FIG. 3 is a block diagram showing the hardware configuration of the motion planning device 20 and the control device 30 according to this embodiment. The motion planning device 20 and the control device 30 can be realized with the same hardware configuration. The motion planning device 20 has a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage 20D, an input unit 20E, a display unit 20F, and a communication interface (I / F) 20G. Each component is connected to be able to communicate with each other via a bus 20H. The control device 30 has a CPU 30A, a ROM 30B, a RAM 30C, a storage 30D, an input unit 30E, a display unit 30F, and a communication I / F 30G. Each component is connected to be able to communicate with each other via a bus 30H. The case of the motion planning device 20 will be described below.

[0030] In this embodiment, programs are stored in ROM 20B or storage 20D. CPU 20A is a central processing unit that executes various programs and controls each configuration. That is, CPU 20A reads out programs from ROM 20B or storage 20D and executes the programs using RAM 20C as a working area. CPU 20A controls each of the above configurations and performs various arithmetic processing according to the programs recorded in ROM 20B or storage 20D.

[0031] The ROM 20B stores various programs and various data. The RAM 20C temporarily stores programs or data as a working area. The storage 20D is configured with an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, and stores various programs including an operating system, and various data.

[0032] The input unit 20E includes a keyboard and a pointing device such as a mouse, and is used to perform various inputs. The display unit 20F is, for example, a liquid crystal display, and displays various information. The display unit 20F may adopt a touch panel system and function as the input unit 20E.

[0033] The communication interface (I / F) 20G is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

[0034] 4 is a block diagram showing an example of a functional configuration of the motion planning device 20 and the control device 30. The motion planning device 20 and the control device 30 may be configured as an integrated unit.

[0035] As shown in Fig. 4, the motion planning device 20 has, as its functional components, a transition creation unit 110, a grip data identification unit 112, and a guide trajectory creation unit 114. Each functional component of the motion planning device 20 is realized by the CPU 20A reading out a program stored in the ROM 20B or the storage 20D, expanding the program in the RAM 20C, and executing the program. The control device 30 has, as its functional components, an acquisition unit 130 and an execution unit 132. Each functional component of the control device 30 is realized by the CPU 30A reading out a program stored in the ROM 30B or the storage 30D, expanding the program in the RAM 30C, and executing the program.

[0036] Here, the state transition data created by the motion planning device 20 of this embodiment will be described. FIG. 5 is a diagram showing an example of unit state transition data created by the motion planning device 20. In this embodiment, an assembly process by the work of the robot 10 is represented by a state transition diagram. As shown in FIG. 5, the entire state transition diagram 201 is decomposed into element units of parts or assemblies, and data representing the transitions of the elements is represented as unit state transition data (202A and 202B). The unit state transition data is a partial element of the composite state transition data including each of the unit state transition data, and by decomposing into unit state transition data, efficient management of control data such as grip data and guide trajectories, which will be described later, becomes possible.

[0037] Hereinafter, each processing unit of the motion planning device 20 will be described. The motion planning device 20 creates each unit state transition data as a motion plan, and accepts registration information required for creating the unit state transition data by input from the user. The registration information includes various information such as part information (part ID, type, etc.), CAD information for each type of end effector 12, assembly procedures (including disassembly procedures), and part grip positions (including postures). The assembly procedures include the relative trajectories of the parts recorded during assembly (or disassembly) in the simulation.

[0038] The transition creation unit 110 creates compound state transition data including each of the unit state transition data linked by their common elements. Each of the unit state transition data is created from an assembly procedure in the registered information.

[0039] 5, the transition creation unit 110 creates first unit state transition data 202A representing a transition from a state in which a first element A and a second element B exist independently to a state in which a third element A', which is an assembly of the first element and the second element, is assembled. The first element A is a part or an assembly. The second element B is also a part or an assembly.

[0040] Next, the transition creation unit 110 creates the next second unit state transition data 202B using the third element A'. The second unit state transition data 202B represents unit state transition data indicating the next assembly process after the first unit state transition data 202A transitions to a state in which the third element A' is being assembled. The transition creation unit 110 creates the second unit state transition data 202B representing a transition from a state in which the third element A' and the fourth element C exist independently to a state in which a fifth element A'' which is an assembly consisting of the third element A' and the fourth element C is being assembled.

[0041] The composite state transition data 201 created by the transition creation unit 110 is in a format in which the first unit state transition data 202A and the second unit state transition data 202B are linked by a third element A', which is a common element between the first unit state transition data 202A and the second unit state transition data 202B.

[0042] In the above description, the unit state transition data 202A is the first unit state transition data, but which unit state transition data is the first unit state transition data can be any one. For example, the unit state transition data 202B may be the first unit state transition data, and the next unit state transition data may be the second unit state transition data. The second unit state transition data is data representing the creation of a new assembly element by combining the element assembled by the transition represented by the first unit state transition data with another element.

[0043] The transition creation section 110 may create only the unit state transition data 202A and 202B without creating the compound state transition data 201.

[0044] Before explaining the grip data specification unit 112 and the guide path creation unit 114, a specific example of an assembly procedure corresponding to the composite state transition data will be described. FIG. 6 is a diagram showing an example of an assembly procedure and a disassembly procedure using parts A to D. When applied to state transitions, each of (1) to (4) corresponds to unit state transition data, and data in a format in which all steps of (1) to (4) are linked corresponds to the composite state transition data. When the work process of assembling parts A and B in (1) is applied to the above-mentioned first unit state transition data, parts A and B correspond to the first element and the second element, respectively, and an assembly obtained by assembling parts A and B corresponds to the third element. (2) to (4) can also be similarly applied to the first unit state transition data. Moreover, when the work process in (2) is applied to the second unit state transition data, an assembly of parts A and B corresponds to the third element, part C corresponds to the fourth element, and an assembly of parts A to C corresponds to the fifth element. Similarly, (3) and (4) can be applied to the second unit state transition data. In the following, the grip data specifying unit 112 and the guide trajectory creating unit 114 will be described using the assembly procedure of parts A to D as an example.

[0045] The grip data identifying unit 112 identifies grip data for each of the unit state transition data based on the grip position of the part in the registration information. The grip data is a planned value of the relative position of the end effector and the gripping object when the end effector 12 of the robot 10 grips the gripping object for assembly. The grip data identifying unit 112 identifies the first grip data when the end effector 12 grips the first element or the second element which is the first gripping object. The grip data identifying unit 112 also identifies the second grip data when the end effector 12 of the robot 10 grips the third element or the fourth element which is the second gripping object. The relative position in the grip data will be described below. Here, the other element with respect to the element set as the gripping object becomes the installation object.

[0046] Fig. 7 is a diagram showing an example of relative positions in gripping data. Fig. 7(A) shows the relative position R1 of the end effector 12 with respect to part A as an arrow when the end effector 12 is a suction pad. Fig. 7(A) shows the relative position R1 of the end effector 12 with respect to part A as an suction pad. Fig. 7(B) shows the relative position R2 of the end effector 12 with respect to part B as an end effector 12 is a gripper (hand) equipped with a pair of clamping parts. Parts C and D are similarly shown.

[0047] The relative position is calculated using the following equation (1). (T obj→tcp ) -1 =T tcp→obj (1) The left side represents the part coordinate system, and the right side represents the relative position (position and posture) from the end effector 12. The gripping data is calculated, for example, as planned values ​​of the target relative position of the object to be gripped and CAD data with an identification ID of the end effector 12 used during gripping. The planned value of the relative position may be input by the user, who is the administrator, by including it in the gripping position of the part in the registered information (the position on the surface of the part to be gripped), or may be automatically calculated by an existing method such as a gripping plan.

[0048] The guide trajectory creation unit 114 creates a guide trajectory for each unit state transition data based on an assembly procedure in the registration information. The guide trajectory is the final part of the relative trajectory between an element that is a gripping object for performing a state transition and an element that is a destination object, and is a trajectory that is a common target relative trajectory regardless of the initial relative positions between these elements. The guide trajectory creation unit 114 creates a first guide trajectory for assembling the first element and the second element. The guide trajectory creation unit 114 also creates a second guide trajectory for assembling the third element and the fourth element. Note that, in addition to the case where the relative trajectory is represented by position information, there is also a case where the trajectory is represented explicitly by speed information without including position information. When the trajectory is represented by speed information, it is represented by a vector amount including directional information.

[0049] In addition, the first guide trajectory can be created as the initial part of the relative trajectory between the first element and the second element recorded when the third element is disassembled into the first element and the second element, or the final part of the relative trajectory between the first element and the second element recorded when the third element is assembled from the first element and the second element. An example of a method for creating the guide trajectory will be described below.

[0050] FIG. 8 is an example of a relative trajectory recorded when an assembly of parts A to D is disassembled into an assembly of parts D and parts A to C. That is, it is an example of the initial part of the relative trajectory of the first and second elements recorded when the third element is disassembled. In FIG. 8, the relative trajectory G1 of the part D is shown, and the reverse trajectory of this relative trajectory G1 can be used as the guide trajectory when gripping and assembling the part D. Such a relative trajectory can be calculated when generating the procedure for disassembling from the target state. As a disassembly procedure, a bounding box (BB) is set for each process, and when the bounding box does not overlap with the part that was moved during disassembly, the disassembly of the process is completed.

[0051] Fig. 9 shows an example of a guide path. Fig. 9(A) shows a guide path (G2) when part A is gripped and lifted. Fig. 9(B) shows a guide path (G3) when part B is attached to part A.

[0052] The state transition control unit 25 acquires the compound state transition data or the unit state transition data, the grip data, and the guide trajectory from the motion planning device 20. When the state transition control unit 25 acquires the compound state transition data, it breaks it down into unit state transition data. When the state transition control unit 25 receives a notification of completion of execution of one state transition from the control device 30, it instructs the control device 30 to execute the next state transition. In addition, the state transition control unit 25 provides the unit state transition data, the grip data, and the guide trajectory to the control device 30 in advance all at once or each time a state transition occurs so that the control device 30 can use the unit state transition data, the grip data, and the guide trajectory for execution of the state transition. The state transition control unit 25 may be included in the motion planning device 20, may be included in the control device 30, or may be a device different from either of them. In addition, the entire control system including the motion planning device 20, the control device 30, and the state transition control unit 25 may be one device.

[0053] Hereinafter, each processing unit of the control device 30 will be described using the state transition in Fig. 5 as an example. Here, one state transition is called a task.

[0054] The acquisition unit 130 acquires the unit state transition data 202A (first unit state transition data) and the corresponding grip data (first grip data) and guide trajectory (first guide trajectory) from the state transition control unit 25. The acquisition unit 130 also acquires observation data (first observation data) that observes the position of element A (first element) and the position of element B (second element) from the state observation sensor 14 (first sensor). Note that the position of element A and the position of element B include the posture. In accordance with the progress of the task, the acquisition unit 130 acquires the unit state transition data 202B (second unit state transition data) and the corresponding grip data (second grip data) and guide trajectory (second guide trajectory) from the state transition control unit 25. The acquisition unit 130 acquires observation data (second observation data) that observes the position of element A' (third element) and the position of element C (fourth element) from the state observation sensor 14 (second sensor).

[0055] The execution unit 132 uses the unit state transition data 202A output from the state transition control unit 25 to execute a task (first task) which is a state transition represented by the unit state transition data 202A. Specifically, the execution unit 132 causes the end effector 12 to grasp a grasp target (first grasp target) which is one of the elements A and B based on the observation data and gripping data for the elements A and B, and moves the grasp target relatively to the other element along a target relative trajectory which includes the guide trajectory in its final portion.

[0056] More specifically, the execution unit 132 grasps the positions of elements A and B in the robot coordinate system from the observation data. Assuming that element A is to be grasped, the start position of the guide path in the robot coordinate system is grasped from the position of element B and the guide path. The operational movement amount of the robot 10 to the start position of the guide path is calculated, and element A is grasped by the end effector 12 based on the grasping data. Element A grasped by the end effector 12 is moved to the start position of the guide path, and then moved along the guide path. The execution unit 132 may execute the above operations by appropriately correcting the observation error, the grasping position error, and the directional error between the robot coordinate system and the guide path.

[0057] Here, an example of movement using the guide trajectory will be described. The execution unit 132 generates a transition trajectory, which is a trajectory for moving a gripping object, which is one of the elements A and B, from its initial position to the starting point position of the guide trajectory or any position on the guide trajectory relative to the other element. In the example of Fig. 9(B), the trajectory ST for gripping the part B to the starting point position of the guide trajectory G3 is an example of a relative trajectory. The transition trajectory may be either a relative trajectory or a trajectory in the robot coordinate system.

[0058] The non-grasped element is basically assumed to be fixed as a target object to be incorporated, but it may be grasped and moved by the end effector 12 of another robot 10. Also, if movement occurs due to contact between parts during the task, it may be observed again.

[0059] In addition, the execution unit 1232 executes a task (second task) of assembling element A'' (fifth element) from element A' (third element) and element C (fourth element) in Figure 5 using the next unit state transition data (second unit state transition data) output from the state transition control unit 25 according to the progress of the task, in the same way as the preceding task.

[0060] Next, the operation of the control system 1 will be described.

[0061] 10 is a sequence diagram showing the flow of processing in the control system 1 of this embodiment. The CPU 20A functions as each part of the motion planning device 20 to perform motion planning processing, and the CPU 30A functions as each part of the control device 30 to perform control processing.

[0062] In step S100, the motion planning device 20 creates composite state transition data and control data by connecting each of the unit state transition data. The control data is each of the grip data and each of the guide trajectories corresponding to each of the created unit state transition data. The details of the process in step S100 will be described later.

[0063] In step S102, the motion planning device 20 outputs the compound state transition data and the control data to the state transition control unit 25. The state transition control unit 25 divides the compound state transition data into unit state transition data.

[0064] In step S104, the state transition control unit 25 receives a task start instruction and starts the task to be processed. When the task start instruction is received, the first task to be processed is started, and the task to be processed is updated according to the progress of the task. The task start instruction may be received by the control device 30.

[0065] In step S106, the state transition control unit 25 notifies the control device 30 of the start of the task.

[0066] In step S108, the state transition control unit 25 outputs to the control device 30 the unit state transition data corresponding to the task to be processed and the control data corresponding to the unit state transition data.

[0067] In step S110, upon receiving a notification of the start of the task, the acquisition unit 130 of the control device 30 acquires the observation data observed by the state observation sensor 14.

[0068] In step S112, the execution unit 132 of the control device 30 calculates the transition trajectory of the robot 10 to the starting position of the guide trajectory based on the observation data and the gripping data.

[0069] In step S114, the execution unit 132 of the control device 30 executes the task to be processed. That is, the end effector 12 grasps the element to be grasped based on the grasping data, and moves the grasped object relative to the other element along the calculated transition trajectory and guide trajectory.

[0070] In step S116, the execution unit 132 of the control device 30 judges whether or not the end point of the guide trajectory has been reached in the execution of the task. If it is judged that the end point has been reached, the process proceeds to step S118, and if it is judged that the end point has not been reached, the judgment of step S116 is repeated.

[0071] In step S118, the execution unit 132 of the control device 30 notifies the state transition control unit 25 of the completion of the task.

[0072] In step S120, state transition control unit 25 determines whether or not the processing has been completed up to the final state of the compound state transition data. If it is determined that the processing has been completed up to the final state, the process proceeds to step S124. If it is determined that the processing has not been completed up to the final state, the process proceeds to step S122.

[0073] In step S122, state transition control unit 25 updates the task to be processed, and returns to step S108. Task update is a process of moving the task to be processed to the next task. In step S108 after the task update, unit state transition data and control data (gripping data and guide trajectory) corresponding to the task to be processed are output to control device 30, and the subsequent processes are repeated.

[0074] In step S124, the state transition control unit 25 ends the task, and the processing of the control system 1 ends.

[0075] FIG. 11 is a flowchart showing the flow of the motion planning process of the motion planning device 20 in step S100.

[0076] In step S200, the CPU 20A receives registration information input by a user. The registration information includes various information such as part information (part ID, type, etc.), CAD information for each type of end effector 12, assembly procedures, and part gripping positions (including postures).

[0077] In step S202, transition creation section 110 creates compound state transition data by linking each of the unit state transition data, based on the assembly procedure in the registered information.

[0078] In step S204, the grip data identifying unit 112 identifies grip data for each piece of unit state transition data based on the grip position of the part in the registered information.

[0079] In step S206, the guide trajectory generating unit 114 generates a guide trajectory for each piece of unit state transition data based on the assembly procedure (recorded relative trajectories of parts) in the registered information.

[0080] In step S208, the CPU 20A displays each of the created unit state transition data, the grip data, and the guide path on the display unit 20F. Note that the display here is not limited to the display unit 20F, and may be via another device.

[0081] In step S210, CPU 20A determines whether or not a correction instruction has been received from the user. If it is determined that a correction instruction has been received, the process proceeds to step S212, and if it is determined that a correction instruction has not been received, the process proceeds to step S102.

[0082] In step S212, CPU 20A reflects the contents of the correction instruction in each of the unit state transition data, the grip data, and the guide trajectory. The correction instruction is, for example, correction of the state transition diagram of each of the unit state transition data, addition of a waypoint, correction of the grip position of the part in the grip data, correction of the relative trajectory in the guide trajectory, etc. In the output of step S102, composite state transition data consisting of each of the final unit state transition data, and control data which is the grip data and the guide trajectory are output to state transition control unit 25.

[0083] As described above, according to the control system 1 of the present embodiment, composite state transition data and control data are created by linking each of the unit state transition data, and tasks are controlled by the unit state transition data. This makes it possible to appropriately and efficiently manage the work process of the robot.

[0084] In the above embodiment, the operation planning process or the control process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. In addition, the operation planning process or the control process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0085] In the above embodiment, the program is pre-stored (installed) in the ROM 20B (30B) or the storage 20D (30D), but the present invention is not limited to this. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may be downloaded from an external device via a network. [Explanation of symbols]

[0086] 1. Robot System 10. Robot 11 Arm 12 Gripper 14 Condition Observation Sensor 20 Motion planning device 25 State transition control section 30 Control device

Claims

1. A control system including a motion planning device and a control device, The motion planning device includes: a transition creation unit that breaks down a part or an assembly into element units and creates first unit state transition data that represents a transition of elements from a state in which a first element, which is a part or an assembly, and a second element, which is a part or an assembly, exist independently to a state in which a third element, which is an assembly consisting of the first element and the second element, is assembled; a gripping data specifying unit that specifies first gripping data that is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and a guide trajectory creation unit that creates a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is a common target relative trajectory regardless of an initial relative position of the first element and the second element; The control device includes: an acquisition unit that acquires the first unit state transition data, the first gripping data, and the first guide trajectory, and acquires first observation data that observes a position of the first element and a position of the second element from a first sensor; and an execution unit that executes a first task of gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, to complete a state in which the third element is assembled in the first unit state transition data, In the motion planning device, The transition creation unit further creates second unit state transition data representing a transition of an element from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled; the gripping data specifying unit further specifies second gripping data which is a planned value of a relative position between the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element which is a second gripping object for assembly; The guide trajectory creation unit further creates a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element; The processor of the motion planning device displays each of the created first unit state transition data and the second unit state transition data, the first grip data and the second grip data, and the first guide trajectory and the second guide trajectory on a display unit, When a processor of the motion planning device receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory; In the control device, the acquisition unit further acquires the second unit state transition data, the second gripping data, and the second guide trajectory according to progress of the task, and acquires second observation data observing a position of the third element and a position of the fourth element from a second sensor; the execution unit, according to progress of the task, further executes a second task of gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion, thereby completing a state in which the fifth element is assembled in the second unit state transition data. Control system.

2. The transition creation unit creates composite state transition data including the first unit state transition data and the second unit state transition data in a form in which common parts of each other are aggregated. The control system of claim 1 .

3. The control system further includes a state transition control unit, the state transition control unit outputs the second unit state transition data to the execution unit of the control device after the first task is completed; the execution unit of the control device executes the second task using the output second unit state transition data. A control system according to claim 1 or 2.

4. A control system as described in any one of claims 1 to 3, wherein the execution unit of the control device generates a transition trajectory, which is a trajectory that moves the first grasped object, which is one of the first element and the second element, relative to the other element from its initial position to the starting position of the first guide trajectory or any position on the first guide trajectory.

5. A transition creation unit that breaks down a part or assembly into element units and creates first unit state transition data that represents a transition of elements from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled; a gripping data specifying unit that specifies first gripping data that is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; a guide trajectory generation unit that generates a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is a common target relative trajectory regardless of an initial relative position of the first element and the second element; Including, The transition creation unit further creates second unit state transition data representing a transition of an element from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled; the gripping data specifying unit further specifies second gripping data which is a planned value of a relative position between the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element which is a second gripping object for assembly; The guide trajectory creation unit further creates a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition, and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element; The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit, When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory. Motion planning device.

6. The transition creation unit creates composite state transition data including the first unit state transition data and the second unit state transition data in a form in which common parts of each other are aggregated. The motion planning device according to claim 5 .

7. 7. The motion planning device according to claim 5, wherein the first guide trajectory is an initial portion of a relative trajectory of the first element and the second element recorded when the third element is disassembled into the first element and the second element, or a final portion of the relative trajectory of the first element and the second element recorded when the third element is assembled from the first element and the second element.

8. First unit state transition data representing a transition of an element from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled, when a part or assembly is decomposed into element units; First gripping data is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and obtain a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is set as a common target relative trajectory regardless of initial relative positions of the first element and the second element; an acquisition unit that acquires first observation data that observes a position of the first element and a position of the second element from a first sensor; an execution unit that executes a first task of gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, to complete a state in which the third element is assembled in the first unit state transition data; Including, The acquisition unit further acquires second unit state transition data representing a transition from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled; Second gripping data is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; and obtaining a second guide trajectory, which is a final portion of the relative trajectories of the third element and the fourth element for performing the transition, and which is set as a common target relative trajectory regardless of initial relative positions of the third element and the fourth element; acquiring second observation data from a second sensor that observes a position of the third element and a position of the fourth element; When a correction instruction is received from a user at a device from which the data is acquired by the acquisition unit, at least one of the following corrections is executed in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory; the execution unit, according to progress of the task, further executes a second task of gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion, thereby completing a state in which the fifth element is assembled in the second unit state transition data. Control device.

9. The control device according to claim 8, wherein the execution unit generates a transition trajectory, which is a transition portion before the final portion of the relative trajectory, and which is a trajectory that moves the first grasped object, which is one of the first element and the second element, from its initial position to a starting position of the first guide trajectory or any position on the first guide trajectory relative to the other element, until the first grasped object transitions from its real space relative position to a target relative trajectory of the first guide trajectory.

10. A motion planning and control method for performing motion planning and control, comprising: In the motion plan, disassembling the part or assembly into element units, and creating first unit state transition data representing a transition of the elements from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled; Identifying first gripping data which is a planned value of a relative position between an end effector of a robot and a first gripping object when the end effector grasps the first element or the second element which is a first gripping object for assembly; creating a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and which is a common target relative trajectory regardless of an initial relative position of the first element and the second element; In the control, acquiring the first unit state transition data, the first gripping data, and the first guide path, and acquiring first observation data that observes a position of the first element and a position of the second element from a first sensor; gripping the first gripping object by the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion, thereby executing a first task for completing a state in which the third element is assembled in the first unit state transition data; Further, in the motion plan, In the creation of the transition data, second unit state transition data is created which represents a transition of an element from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled; The identification of the gripping data further includes identifying second gripping data that is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; The guide trajectory generation further includes generating a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element; The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit, When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory; Furthermore, in the control, Regarding the acquisition, the second unit state transition data, the second gripping data, and the second guide trajectory are acquired according to a progress of a task, and second observation data observing a position of the third element and a position of the fourth element are acquired from a second sensor; Regarding the execution, according to the progress of the task, a second task is executed to complete a state in which the fifth element is assembled in the second unit state transition data by gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion thereof. A method of planning and controlling operations for causing a computer to execute a process.

11. A part or assembly is decomposed into element units, and first unit state transition data is created which represents a transition of the elements from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled; Identifying first gripping data which is a planned value of a relative position between an end effector of a robot and a first gripping object when the end effector grasps the first element or the second element which is a first gripping object for assembly; creating a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and which is a common target relative trajectory regardless of an initial relative position of the first element and the second element; Furthermore, in the motion planning, In the creation of the transition data, second unit state transition data is created which represents a transition of an element from a state in which the third element and a fourth element which is a part or an assembly exist independently to a state in which a fifth element which is an assembly including the third element and the fourth element is assembled; The identification of the gripping data further includes identifying second gripping data that is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; The guide trajectory generation further includes generating a second guide trajectory, which is a final portion of the relative trajectory of the third element and the fourth element for performing the transition and is set as a common target relative trajectory regardless of the initial relative positions of the third element and the fourth element; The processor displays the created first unit state transition data and the created second unit state transition data, the created first grip data and the created second grip data, and the created first guide trajectory and the created second guide trajectory on a display unit, When the processor receives a correction instruction from a user, the processor executes at least one of the following corrections in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory. A motion planning method for causing a computer to execute a process.

12. First unit state transition data representing a transition of an element from a state in which a first element which is a part or assembly and a second element which is a part or assembly exist independently to a state in which a third element which is an assembly consisting of the first element and the second element is assembled, when a part or assembly is decomposed into element units; First gripping data is a planned value of a relative position of an end effector and a first gripping object when an end effector of a robot grips the first element or the second element, which is a first gripping object, for assembly; and obtain a first guide trajectory, which is a final portion of the relative trajectory of the first element and the second element for performing the transition, and is set as a common target relative trajectory regardless of initial relative positions of the first element and the second element; acquiring first observation data from a first sensor that observes a position of the first element and a position of the second element; a first task is executed to complete a state in which the third element is assembled in the first unit state transition data by gripping the first gripping object with the end effector based on the first observation data and the first gripping data, and moving the first gripping object, which is one of the first element and the second element, relative to the other element along a target relative trajectory that includes the first guide trajectory as a final portion thereof; Furthermore, in the control, second unit state transition data representing a transition from a state in which the third element and a fourth element, which is a part or an assembly, exist independently to a state in which a fifth element, which is an assembly including the third element and the fourth element, is assembled, with respect to the acquisition; Second gripping data is a planned value of a relative position of the end effector and the second gripping object when the end effector of the robot grips the third element or the fourth element, which is a second gripping object, for assembly; and obtaining a second guide trajectory, which is a final portion of the relative trajectories of the third element and the fourth element for performing the transition, and which is set as a common target relative trajectory regardless of initial relative positions of the third element and the fourth element; acquiring second observation data from a second sensor that observes a position of the third element and a position of the fourth element; When a correction instruction is received from a user by a device from which the data is acquired, at least one of the following corrections is executed in response to the correction instruction: correction of state transition diagrams of the first unit state transition data and the second unit state transition data, addition of waypoints to the first guide trajectory and the second guide trajectory, correction of gripping positions of parts in the first gripping data and the second gripping data, and correction of relative trajectories in the first guide trajectory and the second guide trajectory; Regarding the execution, according to the progress of the task, a second task is executed to complete a state in which the fifth element is assembled in the second unit state transition data by gripping the second gripping object by the end effector based on the second observation data and the second gripping data, and moving the second gripping object, which is one of the third element and the fourth element, relative to the other element along a target relative trajectory that includes the second guide trajectory in a final portion thereof. A control method for causing a computer to execute processing.

Images