Robot system, method for controlling a robot system, method for manufacturing articles, program and recording medium

Abstract

This technology provides advantages for robots in holding workpieces. [Solution] The robot system comprises a robot having an end effector, a sensor that detects the state of one or more workpieces placed in a workpiece storage section, and a control device that controls the robot. The control device is configured to perform the following: a selection process that uses the detection results obtained from the sensor to select a target workpiece from one or more workpieces to be held by the end effector; an information processing that, if the end effector does not meet the holding conditions necessary for holding the target workpiece, uses the detection results to obtain information about the area in which the target workpiece can move within the workpiece storage section; and a control process that controls the robot to move the target workpiece to the area by bringing the end effector into contact with the target workpiece.

Description

Technical Field

[0001] The present invention relates to the technology of robots.

Background Art

[0002] Conventionally, control of a robot for taking out a plurality of workpieces stacked in a workpiece storage unit one by one to the robot is known. However, regarding the workpiece existing at the end of the workpiece storage unit, even if the workpiece can be recognized by a camera, the robot may not be able to take out the workpiece existing at the end of the workpiece storage unit due to interference of the robot with the wall of the workpiece storage unit. If workpieces remain in the workpiece storage unit, the number of replacements of the workpiece storage unit increases to produce a predetermined number of products, which takes time to produce a predetermined number of products, and there is a risk that the production efficiency of the products will decrease.

[0003] In contrast, Patent Document 1 discloses a workpiece supply device provided with a protruding portion that can advance and retreat upward from the inner bottom surface of a container body. Further, Patent Document 2 discloses that when a robot fails to grip a part, the robot is made to perform an operation of pulling the part to the center of a box.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] However, while both Patent Document 1 and Patent Document 2 can change the state of the workpiece, it is not guaranteed that the workpiece will be in a state where it can be held by the robot after the state has been changed.

[0007] This invention provides a technology that is advantageous for robots to hold workpieces. [Means for solving the problem]

[0008] A first aspect of the present invention is a robot system comprising: a robot having an end effector; a sensor for detecting the state of one or more workpieces arranged in a workpiece storage section; and a control device for controlling the robot, wherein the control device is configured to perform the following: a selection process for selecting a target workpiece from the one or more workpieces to be held by the end effector using the detection results obtained from the sensor; an information processing for obtaining information about a region in the workpiece storage section to which the target workpiece can move, using the detection results, if the end effector does not satisfy the holding conditions necessary for holding the target workpiece; and a control process for controlling the robot to bring the end effector into contact with the target workpiece and move the target workpiece to the region.

[0009] A second aspect of the present invention is a method for controlling a robot system comprising a robot having an end effector, a sensor for detecting one or more workpieces arranged in a workpiece storage section, and a control device for controlling the robot, wherein the control device uses the detection results obtained from the sensor to select a target workpiece from the one or more workpieces to be held by the end effector, and if the control device does not satisfy the holding conditions necessary for the end effector to hold the target workpiece, it uses the detection results to obtain information on the area in which the target workpiece can move within the workpiece storage section, and the control device controls the robot to bring the end effector into contact with the target workpiece and move the target workpiece to the area. [Effects of the Invention]

[0010] According to the present invention, a technique is provided that is advantageous for a robot to hold a workpiece. [Brief explanation of the drawing]

[0011] [Figure 1] (a) is an explanatory diagram of the robot system according to the first embodiment. (b) is an explanatory diagram showing the inside of the workpiece housing as seen from the sensor according to the first embodiment. [Figure 2] This is a block diagram of the hardware configuration of the robot controller according to the first embodiment. [Figure 3] This is a functional block diagram of the robot controller according to the first embodiment. [Figure 4] This is a control flowchart of the robot controller according to the first embodiment. [Figure 5] This is an explanatory diagram of the gripping point according to the first embodiment. [Figure 6] This is a control flowchart for the workpiece movement process according to the first embodiment. [Figure 7] This is a schematic diagram illustrating the empty area according to the first embodiment. [Figure 8]It is a schematic diagram for explaining a method of selecting contact information according to the first embodiment. [Figure 9] It is a schematic diagram for explaining a calculation process of the operation amount of the end effector according to the first embodiment. [Figure 10] It is an explanatory diagram of the operation of the end effector realized by the operation plan of the first embodiment. [Figure 11] It is a flowchart of the calculation process of the contact information according to the first embodiment. [Figure 12] It is a conceptual diagram for explaining a method of calculating candidates for contact information according to the first embodiment. [Figure 13] It is an explanatory diagram of a determination process of whether or not the entry condition according to the first embodiment is satisfied. [Figure 14] It is a schematic diagram for explaining an interference determination between the work accommodation part and the end effector according to the first embodiment. [Figure 15] It is a schematic diagram for explaining an interference determination between the work in the work accommodation part and the end effector 5 according to the first embodiment. [Figure 16] It is a schematic diagram for explaining a method of setting candidates for contact information according to the first embodiment. [Figure 17] It is an explanatory diagram of the end effector according to the first embodiment. [Figure 18] It is an explanatory diagram of the 3D model according to the first embodiment. [Figure 19] It is a flowchart showing a process of setting candidates for contact information according to the first embodiment. <000年096> [Figure 20] It is a schematic diagram for explaining a method of determining the occurrence of a moment according to the first embodiment. [Figure 21] It is a schematic diagram for explaining a process of setting candidates for contact information according to the first embodiment. [Figure 22] It is a schematic diagram for explaining the rotation of the 3D model of the end effector according to the first embodiment. [Figure 23] It is a schematic diagram for explaining an interference determination according to the first embodiment. [Figure 24]This is a flowchart of the process for recognizing free space according to the first embodiment. [Figure 25] This is a schematic diagram of an image generated by the environment recognition unit according to the first embodiment. [Figure 26] This is a schematic diagram of an image generated by the environment recognition unit according to the first embodiment. [Figure 27] This is a schematic diagram of a distance image according to the first embodiment. [Figure 28] This is a schematic diagram of a binarized image according to the first embodiment. [Figure 29] This is an explanatory diagram showing the maximum distance of the minimum available space according to the first embodiment. [Figure 30] This is a schematic diagram of an image generated by the environment recognition unit according to the first embodiment. [Figure 31] This is an explanatory diagram of information processing in another example according to the first embodiment. [Figure 32] This is an explanatory diagram of an example of a display image according to the first embodiment. [Figure 33] This is an explanatory diagram of an example of a display image according to the first embodiment. [Figure 34] This is a flowchart showing the process of setting candidate contact information according to the second embodiment. [Figure 35] This is a flowchart for performing the process of removing a workpiece from the path according to the third embodiment. [Figure 36] This is a schematic diagram illustrating the process according to the third embodiment. [Figure 37] This is a schematic diagram illustrating the distance calculation method according to the third embodiment. [Figure 38] This is an explanatory diagram of the training data according to the fourth embodiment. [Figure 39] This is an explanatory diagram of the reasoning according to the fourth embodiment. [Figure 40] This is an explanatory diagram of the masking process according to the fourth embodiment. [Modes for carrying out the invention]

[0012] [System Configuration] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0013] For the purpose of this explanation, the following terms are defined: "Position" refers to a three-dimensional vector (x, y, z) representing a point relative to a given Euclidean coordinate system. "Orientation" refers to a 3x3 rotation matrix used to superimpose one Euclidean coordinate system onto the other.

[0014] [First Embodiment] Figure 1(a) is an explanatory diagram of a robot system 1000 according to the first embodiment. The robot system 1000 comprises a robot 7, a robot controller 1 connected to the robot 7 and controlling the robot 7, a sensor controller 2 connected to the robot controller 1, and a sensor 3 connected to the sensor controller 2. The robot controller 1 is an example of a control device.

[0015] The workpiece storage section 6 is a box-shaped storage container with an opening at the top, capable of accommodating multiple bulk-stacked workpieces 20. For example, the workpiece storage section 6 includes an inner bottom surface 61 on which the workpieces 20 are placed, and an inner wall surface 62 arranged on the outer periphery of the inner bottom surface 61. Each of the multiple workpieces 20 can be held by the robot 7. Furthermore, the multiple workpieces 20 are identical in shape and size to each other.

[0016] Robot 7 is, for example, an industrial robot. Robot 7 can transport a workpiece 20 and perform a predetermined production operation. The predetermined production operation is, for example, an assembly operation in which workpiece 20 is assembled to another workpiece. By robot 7 performing the predetermined production operation, a product, which is an example of an article, is manufactured. The article may be a final product or an intermediate product. Robot 7 has a robot arm 4 and an end effector 5 provided at the tip, which is an example of a predetermined part of the robot arm 4. Robot arm 4 is, for example, a vertically articulated robot arm. End effector 5 is, for example, a robot hand. The robot hand has parallel chucks with two fingers 51 and, like robot arm 4, can open and close the two fingers 51 based on command values ​​from robot controller 1. The end effector 5 may be a vacuum gripper that holds the workpiece 20 by suction.

[0017] Sensor 3 is positioned above the workpiece housing 6, so that the entire opening of the workpiece housing 6, i.e., the entire inner bottom surface 61, is within the field of view of sensor 3. Sensor 3 is fixed to, for example, a frame (not shown).

[0018] In the first embodiment, sensor 3 is an imaging device such as a digital camera. Sensor 3 generates image information as detection information. Detection information is the detection result. Sensor 3 is preferably a depth sensing camera, such as an RGBD camera. A depth sensing camera generates image information (image data) including depth information as detection information.

[0019] The sensor 3 can be any sensor that generates detection information that allows the robot controller 1 to acquire the arrangement status (e.g., position and orientation information) of the workpieces 20 placed on the inner bottom surface 61 of the workpiece storage section 6. For example, the sensor 3 may be an RGB camera that generates image information without depth information. The sensor 3 may also be fixed to the robot arm 4. The sensor 3 may also be a depth sensor using infrared light, and may scan the inside of the workpiece storage section 6 with infrared light to generate depth information as detection information. In this way, the sensor 3 detects the workpieces visible to the sensor 3 from among the multiple workpieces 20 placed inside the workpiece storage section 6, i.e., on the inner bottom surface 61, and outputs the detection information to the robot controller 1 via the sensor controller 2.

[0020] Figure 1(b) is an explanatory diagram showing the state of the inside of the workpiece storage section 6 as seen from the sensor 3 according to the first embodiment. In the example of Figure 1(b), most of the stacked workpieces 20 have been unloaded by the robot 7, and one or more workpieces 20 remain in the workpiece storage section 6. A portion of the inner bottom surface 61 of the workpiece storage section 6 is exposed between the one or more workpieces 20. Therefore, in the example of Figure 1(b), the sensor 3 detects the state of the inner bottom surface 61 of the workpiece storage section 6 and the one or more workpieces 20 placed on the inner bottom surface 61, and outputs the detected information to the robot controller 1 via the sensor controller 2. The following describes the case where one or more workpieces 20 are stored inside the workpiece storage section 6, as in the example of Figure 1(b).

[0021] Figure 2 is a block diagram of the hardware configuration of the robot controller 1 according to the first embodiment. Figure 3 is a functional block diagram of the robot controller 1 according to the first embodiment. The robot controller 1 includes a CPU (Central Processing Unit) 101, which is an example of a processor. The robot controller 1 also includes RAM (Random Access Memory) 102 as a primary storage device and an HDD (Hard Disk Drive) 103 as a secondary storage device. Note that RAM 102 and HDD 103 are examples of storage devices, and the storage devices are not limited to these examples.

[0022] The CPU 101 functions as the contact information recognition unit 10, environment recognition unit 11, operation planning unit 12, and control unit 13 shown in Figure 3 by executing program 107. Program 107 is stored in the HDD 103, which is a secondary storage device.

[0023] Furthermore, the robot controller 1 includes a sensor interface 104, a robot interface 105, and a general-purpose interface 106 as connection interfaces to external devices. The sensor interface 104 is connected to the sensor controller 2, enabling the transmission and reception of information with the sensor controller 2. The robot interface 105 is connected to the robot 7, enabling the transmission and reception of information with the robot 7. The general-purpose interface 106 is a general-purpose interface to external devices, such as USB (Universal Serial Bus) or HDMI (High-Definition Multimedia Interface). For example, the general-purpose interface 106 is connected to an input device 108 such as a mouse and keyboard, and a display 109, receiving information from the input device 108 and transmitting image information to the display 109. The sensor interface 104 and the robot interface 105 can be any standard corresponding to the sensor controller 2 and robot 7, respectively, and may also be general-purpose interfaces such as Ethernet. These components of the robot controller 1 are bus-connected to each other, enabling the transmission and reception of data between them.

[0024] In this embodiment, the non-temporary recording medium readable by the computer is the HDD 103, and the program 107 is recorded on the HDD 103, but this is not the only possible representation. The program 107 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 107 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, etc. The program 107 may also be obtained from a network (not shown).

[0025] The contact information recognition unit 10 and the environment recognition unit 11 shown in Figure 3 are connected to the sensor controller 2 and can receive detection information (values) generated by the detection processing of the sensor 3 from the sensor controller 2. If the sensor 3 is an imaging device such as a camera, the detection information is image information obtained by imaging.

[0026] In this embodiment, the sensor 3 is composed of one camera, but it is not limited to this, and may be composed of multiple cameras, for example, two cameras connected to the contact information recognition unit 10 and the environment recognition unit 11, respectively.

[0027] Each of the contact information recognition unit 10 and the environment recognition unit 11 is configured to transmit the recognition result to the motion planning unit 12. The motion planning unit 12 is configured to plan the movements of the robot arm 4 and the end effector 5 based on the received information and to transmit the motion plan to the control unit 13. The control unit 13 converts the received motion plan into robot command values ​​and transmits the robot command values ​​to the robot arm 4 or the end effector 5. The robot arm 4, upon receiving the robot command values, operates according to the received robot command values, and the end effector 5, upon receiving the robot command values, operates according to the received robot command values.

[0028] [Overview of robot operation] Figure 4 is a control flowchart of the robot controller 1 according to the first embodiment. As the control cycle shown in Figure 4 is repeated, multiple workpieces 20 are unloaded one by one from the workpiece storage unit 6 by the robot 7. Next, as shown in Figure 1(b), the state in which one or more workpieces 20 remain in the workpiece storage unit 6 will be described. The robot controller 1 controls the robot 7 to pick up one or more workpieces 20 stored in the workpiece storage unit 6.

[0029] First, in step S35, the robot controller 1 instructs the sensor controller 2 to perform detection processing using the sensor 3 and obtains detection information, which is the detection result, from the sensor 3. The robot controller 1 uses the detection information to detect one or more workpieces 20 within the workpiece storage unit 6. That is, the robot controller 1 obtains existence information of the workpieces 20 within the workpiece storage unit 6. Existence information is information representing the coordinate position (x, y, z) of the workpiece 20 relative to the coordinate system of the sensor 3 (sensor coordinate system). Then, the robot controller 1 selects one target workpiece 21 to be held from among the one or more workpieces 20 arranged inside the workpiece storage unit 6. The target workpiece 21 may be selected randomly from among the one or more workpieces 20, or it may be selected according to the priority order assigned to the one or more workpieces 20. The process in step S35 is an example of a selection process.

[0030] If one or more workpieces 20 constitute a single workpiece, that workpiece is selected as the target workpiece 21. If one or more workpieces 20 constitute two or more workpieces, one of the two or more workpieces is selected as the target workpiece 21. In this case, any workpiece 22 other than the target workpiece 21 may become the target workpiece 21 in subsequent control cycles.

[0031] Any method can be applied to detect the workpiece 20. For example, if a depth sensor is used for sensor 3, the depth information within the workpiece housing 6 obtained by scanning can be sampled into point cloud information, and the workpiece 20 can be determined by performing a matching process against a 3D model of the workpiece 20. Alternatively, if an RGB camera is used for sensor 3, the workpiece 20 may be detected using a machine learning algorithm that detects the presence of objects, such as an SSD (Single-Shot Detector).

[0032] Next, in step S36, the robot controller 1 performs a process to detect gripping points that can be held by the end effector 5 for the target workpiece 21 selected in step S35. The storage device, such as the HDD 103, stores workpiece information (3D model) of the workpiece 20 and multiple candidate gripping points associated with the workpiece information. The CPU 101 of the robot controller 1 extracts gripping points from these multiple candidate gripping points that can be held by the end effector 5.

[0033] The gripping point is information including the position and orientation of the end effector 5 and the degree of opening of the two fingers 51, such that the end effector 5 can hold the workpiece 20 without dropping it when the end effector 5 is moved near the workpiece 20 and the fingers 51 of the end effector 5 are opened or closed. The position and orientation of the end effector 5 are, for example, the position and orientation relative to the workpiece coordinate system.

[0034] Figure 5 is an explanatory diagram of a gripping point according to the first embodiment. The gripping point shown in the example of Figure 5 is information about the position and orientation of the end effector 5 and the degree of opening of the two fingers 51 of the end effector 5 when the two fingers 51 of the end effector 5 are opened by a predetermined width indicated by the arrow 500 and the end effector 5 is moved to the vicinity of the workpiece 20. The end effector 5 can hold the workpiece 20 by performing a holding operation in which the two fingers 51 are closed after the end effector 5 has moved to the gripping point.

[0035] When the end effector 5 is made to perform a holding operation by closing its two fingers 51 at the gripping point, the two fingers 51 will contact two locations on the workpiece 20. Two normal vectors pointing inward on the workpiece 20, with each of these two locations as the reference, must be opposite to each other. The condition for the two normal vectors to be opposite to each other is, for example, that the angle between the two normal vectors is 180 degrees ± 1 degree or less. ± 1 degree is an acceptable range. Note that the acceptable range is not limited to ± 1 degree and may be set to any value by the user, for example. However, the larger the acceptable range, the lower the possibility that the end effector 5 can stably hold the workpiece 20, so an acceptable range of ± 1 degree is preferable.

[0036] Furthermore, the end effector 5 and the workpiece 20 must not interfere with each other (for example, collide) at the gripping point before and during the holding operation of the end effector 5.

[0037] The above describes the gripping point when the end effector 5 has two parallel chucks, but it is not limited to this. For example, if the end effector 5 is a vacuum gripper that performs vacuum suction, the gripping point is information about the position and orientation of the end effector 5 that allows the end effector 5 to contact the workpiece 20 and hold the workpiece 20 by suction.

[0038] Furthermore, multiple candidate gripping points are calculated in advance for each workpiece using 3D simulation or the like, linked to workpiece information, and stored in a storage device such as the HDD 103. Since multiple candidate gripping points are set in the workpiece coordinate system, the robot controller 1 can easily detect a gripping point from among the multiple candidate gripping points by calculating the position and orientation of the target workpiece 21 detected in step S35.

[0039] Furthermore, the position and orientation of the target workpiece 21 can be calculated, for example, using pattern matching or other processing when sensor 3 is an RGB camera. Even when sensor 3 is a depth-sensing camera, the position and orientation can be calculated using the 3D matching method used in step S35.

[0040] In step S36, a gripping point is selected from among multiple candidate gripping points stored in the HDD 103 or the like, that is in a position and orientation that allows the robot arm 4 to move, and does not interfere with workpieces 22 or workpiece housing 6 other than the target workpiece 21 that the robot 7 is intended to hold.

[0041] In step S37, the robot controller 1 determines whether the end effector 5 satisfies the necessary holding conditions for holding the target workpiece 21. In this embodiment, since the robot controller 1 performs the process of detecting gripping points in step S36, it determines whether it has detected one or more gripping points on the target workpiece 21. That is, in this embodiment, the necessary holding condition for the end effector 5 to hold the target workpiece 21 is the condition that one or more gripping points have been detected on the target workpiece 21.

[0042] If step S37 is YES, that is, if the holding condition is met, in step S38, the robot controller 1 moves the two fingers 51 of the end effector 5 in the opening or closing direction based on the opening degree information included in the gripping point, moves the robot arm 4 based on the position and orientation information included in the gripping point, and then causes the end effector 5 to perform a holding operation, thereby causing the robot 7 to hold the target workpiece 21. Next, in step S39, the robot controller 1 controls the robot 7 to remove the held target workpiece 21 from the workpiece storage unit 6.

[0043] As the control cycle from step S35 to step S39 is repeated, the workpieces 20 stored in the workpiece storage unit 6 are removed from the workpiece storage unit 6, and the number of workpieces 20 in the workpiece storage unit 6 decreases. In some cases, due to interference conditions of the robot 7, the gripping point may not be detected in step S36. For example, a workpiece 20 located at the edge of the inner bottom surface 61 of the workpiece storage unit 6 may not have its gripping point detected because the robot 7 may interfere with the inner wall surface 62 of the workpiece storage unit 6. If step S37 is NO, that is, if the holding condition is not met, the robot controller 1 proceeds to the process in step S40.

[0044] In step S40, the robot controller 1 moves the target workpiece 21 within the workpiece storage unit 6 using the end effector 5, performing a workpiece movement process that changes the placement state of the target workpiece 21 within the workpiece storage unit 6. The control operation shown in Figure 4 above is repeated until the workpiece 20 is no longer detected within the workpiece storage unit 6.

[0045] [Overview of workpiece movement process] The workpiece movement process in step S40 shown in Figure 4 will be described in detail below. Figure 6 is a control flowchart of the workpiece movement process according to the first embodiment.

[0046] In step S1, the contact information recognition unit 10 calculates one or more contact pieces of information from the detection information acquired from the sensor 3. The contact pieces of information are control information used to control the robot 7.

[0047] The contact information includes information on the position and orientation in which the end effector 5 can contact the target workpiece 21, and information on the external force vector applied to the target workpiece 21. The position and orientation in which the end effector 5 can contact the target workpiece 21 is the position and orientation in which the end effector 5 can contact the target workpiece 21 without interfering with other workpieces 20 or the workpiece housing 6.

[0048] The external force vector is represented by a three-dimensional vector of (x,y,z). The specific definition of the vector and the recognition processing method will be described later. Note that contact information can be calculated for any workpiece 20 inside the workpiece housing 6, not just the workpiece 20 located at the edge of the inner bottom surface 61 of the workpiece housing 6 as shown in Figure 1(b).

[0049] Next, in step S2, the environmental recognition unit 11 uses the detection information acquired from the sensor 3 to recognize the available area on the inner bottom surface 61 of the work storage unit 6 in which the target workpiece 21 can move, thereby acquiring information about the available area. The available area is a portion of the inner bottom surface 61 of the work storage unit 6 that is exposed from one or more workpieces 20. The information about the available area is stored in the RAM 102 or HDD 103. The process in step S2 is an example of information processing.

[0050] Figure 7 is a schematic diagram illustrating the empty area 33 according to the first embodiment. The empty area 33 is determined by computer simulation using a virtual model of the workpiece 20. The work area 25 is defined as the area with the largest projection region (orthogonal projection) obtained by projecting the workpiece 20 onto the inner bottom surface 61 of the workpiece housing 6 in the direction from the sensor 3 toward the inner bottom surface 61 of the workpiece housing 6. The direction from the sensor 3 toward the inner bottom surface 61 of the workpiece housing 6 is, for example, a direction perpendicular to the inner bottom surface 61. The work area 25 is larger than the projection region obtained by projecting the target workpiece 21 onto the inner bottom surface 61 of the workpiece housing 6. In this embodiment, the empty area 33 is larger than the work area 25. The environment recognition unit 11 defines the area extended outward by a predetermined amount from the work area 25 as the minimum empty area 26. The predetermined amount is preferably the thickness of the fingertip of the end effector 5 from the work area 25. The empty area 33 is larger than the minimum empty area 26. The specific method for recognizing the empty area 33 will be described later. The work area 25 may also be defined as the projected area obtained by projecting the target workpiece 21 onto the inner bottom surface 61 of the workpiece housing 6.

[0051] Although the explanation uses the example of performing step S1 first, it is not limited to this; step S2 may be performed first, or steps S1 and S2 may be performed simultaneously. Performing steps S1 and S2 simultaneously can shorten the execution time per picking cycle and improve the production efficiency of the manufactured goods.

[0052] In step S3, the motion planning unit 12 selects from one or more contact pieces of information obtained in step S1 the contact piece to be used in step S5-1, which moves the target workpiece 21 in the direction of the empty area 33 obtained in step S2. The method for selecting the contact piece will be described later.

[0053] In step S4, the motion planning unit 12 calculates the amount of movement of the end effector 5 based on the contact information selected in step S3 and the information on the center position of the empty area 33. The method for calculating the amount of movement will be described later.

[0054] In step S5, the motion planning unit 12 plans the movement of the robot 7 based on the contact information selected in step S3 and the amount of movement calculated in step S4. The method for planning the movement of the robot 7 will be described later.

[0055] The planned operation is transmitted to the control unit 13. In step S5-1, the control unit 13 converts the planned operation into a robot command value and transmits the robot command value to the robot 7. The robot 7 operates according to the robot command value. The control unit 13 controls the robot 7 to move the target workpiece 21 to the empty area 33 by bringing the end effector 5 into contact with the target workpiece 21. The process in step S5-1 is an example of a control process.

[0056] After moving the target workpiece 21, in the next control cycle, the control unit 13 controls the robot 7 to have the end effector 5 hold the target workpiece 21.

[0057] Figure 8 is a schematic diagram illustrating the method of selecting contact information performed by the motion planning unit 12 according to the first embodiment. In step S1 of Figure 6, the contact information recognition unit 10 is assumed to have recognized two pieces of contact information 31 and 32 for the target workpiece 21. Also, in step S2 of Figure 6, the environment recognition unit 11 is assumed to have recognized an empty area 33 from the detection information (image information) acquired from the sensor 3.

[0058] Each of the contact information 31 and 32 contains information about the external force vector applied to the target workpiece 21. The arrows in Figure 8 indicate the external force vectors. A dotted line 310 overlapping the external force vector included in contact information 31 and a dotted line 320 overlapping the external force vector included in contact information 32 are shown. In Figure 8, the area shown by the dashed rectangle is the empty area 33. In Figure 8, the intersection of two mutually orthogonal dotted lines 331 and 332 within the empty area 33 indicates the center 34 of the empty area 33. Each dotted line 331 and 332 passes through the center of two opposing sides. Note that the center 34 may also be the centroid of the empty area 33. Furthermore, although the empty area 33 shown in Figure 8 is rectangular, it is not limited to this shape; for example, the empty area 33 may be circular.

[0059] The motion planning unit 12 calculates the shortest Euclidean distance between line 310 and the center 34, and the shortest Euclidean distance between line 320 and the center 34. The motion planning unit 12 selects contact information that includes an external force vector that coincides with the line with the shortest distance among the two calculated shortest Euclidean distances. This selects contact information that allows the target workpiece 21 to move into the empty area 33. In the example in Figure 8, among the contact information 31 and 32, contact information 31, which has the shortest Euclidean distance, is selected. Note that contact information whose shortest Euclidean distance exceeds a threshold may not be selected.

[0060] Figure 9 is a schematic diagram illustrating the calculation process for the amount of motion of the end effector 5 according to the first embodiment. The starting point of the external force vector is defined as the starting point of motion 40. The intersection point 41 is defined as the point where the perpendicular line drawn from the center 34 of the empty region 33 to the line 310 intersects with the line 310.

[0061] The motion planning unit 12 calculates the amount of movement of the end effector 5 using the Euclidean distance from the motion starting point 40 to the intersection point 41 of the external force vector included in the contact information 31 selected in step S3. When moving the target workpiece 21, the robot 7 is controlled so that the end effector 5 moves in a straight line. In this case, the Euclidean distance from the motion starting point 40 to the intersection point 41 becomes the amount of movement of the end effector 5.

[0062] Figure 10 is an explanatory diagram of the operation of the end effector 5 realized by the operation plan of the first embodiment. In step S5-1, the control unit 13 moves the end effector 5 in the entry direction from the position and orientation 130 in which it begins to approach the target workpiece 21 in the workpiece housing 6. The control unit 13 moves the end effector 5 in a predetermined direction (direction of operation) from the position and orientation 132 in which the end effector 5 has entered the vicinity of the target workpiece 21, and brings the end effector 5 into contact with the target workpiece 21. The position and orientation of the end effector 5 in contact with the target workpiece 21 is defined as position and orientation 131. The control unit 13 applies an external force in a predetermined direction (direction of operation) by bringing the fingers 51 of the end effector 5 into contact with the target workpiece 21, and controls the robot arm 4 to move the target workpiece 21 in the predetermined direction (direction of operation). That is, the control unit 13 controls the robot arm 4 so that the end effector 5 moves in a straight line. At that time, the control unit 13 controls the robot arm 4 to maintain a constant posture of the end effector 5. The predetermined direction (direction of movement) is the direction of the external force vector, that is, the direction of the external force acting on the target workpiece 21. Note that in Figure 10, the positions and postures 130, 132, and 131 are the same as each other. When the end effector 5 contacts the side of the target workpiece 21, an external force is applied to the target workpiece 21 from the end effector 5 in the direction of the end effector 5's movement, and the target workpiece 21 is pressed in the direction of the end effector 5's movement and moves in the direction of the end effector 5's movement. In this case, the end effector 5 does not hold the target workpiece 21, but slides it.

[0063] Hereinafter, position and attitude 130 may also be referred to as approach position and attitude 130, position and attitude 132 as entry position and attitude 132, and position and attitude 131 as contact position and attitude 131.

[0064] Furthermore, below, the position among position and attitude 130 may be referred to as the approach position, and the attitude among position and attitude 130 may be referred to as the approach attitude. The position among position and attitude 132 may be referred to as the entry position, and the attitude among position and attitude 132 may be referred to as the entry attitude. The position among position and attitude 131 may be referred to as the contact position, and the attitude among position and attitude 131 may be referred to as the contact attitude. The approach position is an example of the first position, the entry position is an example of the second position, and the contact position is an example of the third position.

[0065] Figure 11 is a flowchart of the contact information calculation process (processing in step S1) according to the first embodiment. First, in step S6, the contact information recognition unit 10 calculates the position and orientation of the target workpiece 21.

[0066] In the first embodiment, the contact information recognition unit 10 acquires an RGBD image from the sensor 3, which captures the target workpiece 21, as detection information. The contact information recognition unit 10 performs a matching process on the RGBD image using a 3D model of the workpiece 20, and calculates the position and orientation of the target workpiece 21 with respect to the sensor coordinate system of the sensor 3.

[0067] The means for calculating the position and orientation of the target workpiece 21 are not limited to those described above. For example, as an alternative, the contact information recognition unit 10 may calculate the position and orientation of the target workpiece 21 by acquiring point cloud data of the target workpiece 21 using the sensor 3 and performing fitting processing using a 3D model of the workpiece 20. Existing algorithms such as the ICP (Iterative Closest Point) method can be used for the fitting processing. Alternatively, as another alternative, the contact information recognition unit 10 may determine the position and orientation of the target workpiece 21 by machine learning using the captured image acquired from the sensor 3.

[0068] Next, in step S7, the contact information recognition unit 10 calculates candidate contact information. Figure 12 is a conceptual diagram illustrating the method for calculating candidate contact information according to the first embodiment. The case in which the target workpiece 21 is a workpiece to be held detected by the sensor 3, and the contact information recognition unit 10 calculates candidate contact information for the target workpiece 21 will be described.

[0069] First, as a preliminary step before calculating candidate contact information for the target workpiece 21, the contact point Pi and the position and orientation of the contact point Pi relative to the workpiece coordinate system of the reference workpiece 50 are set. The position and orientation of the reference workpiece 50 relative to the sensor coordinate system can be arbitrary. Also, workpiece 50 is a workpiece with the same shape and size as workpiece 20. Furthermore, workpiece 50 may be a 3D model of workpiece 20.

[0070] A contact point Pi is associated with one or more candidate contact information. In some cases, multiple candidate contact information may be associated with a single contact point. Therefore, the subscript i allows us to determine which candidate contact information on the same point Pi corresponds to. Ri is a 4x4 homogeneous transformation matrix formed by arranging the position vector and the rotation matrix representing the orientation. The method for setting candidate contact information will be described later.

[0071] The position and orientation of the target workpiece 21 relative to the sensor coordinate system have already been calculated in step S6. Therefore, a simultaneous transformation matrix R can be calculated to transform the position and orientation of the reference workpiece 50 to the position and orientation of the target workpiece 21. Thus, the position and orientation Ri' of the contact point Pi' on the target workpiece 21 relative to the work coordinate system can be obtained by a matrix operation of R × Pi. Because multiple candidate contact information is pre-associated with the reference workpiece 50, all candidate contact information for the target workpiece 21 can be calculated using the method described above.

[0072] The contact information recognition unit 10 calculates the position and orientation of the contact information candidate linked to the target workpiece 21 based on the workpiece coordinate system using the method described above, and then converts the reference coordinate system of the calculated position and orientation of the contact information candidate from the workpiece coordinate system to the sensor coordinate system. For the conversion, the simultaneous conversion matrix representing the position and orientation of the workpiece 21 calculated in step S6 is multiplied from the left by the aforementioned Ri'. Next, the contact information recognition unit 10 converts the position and orientation of the contact information candidate based on the sensor coordinate system to the position and orientation of the contact information candidate based on the coordinate system of the end effector 5. The conversion matrix from the sensor coordinate system to the end effector coordinate system can be obtained by known means. For example, the internal and external parameters of the sensor 3 can be calibrated and a matrix can be obtained to convert between the orientation of the sensor coordinate system and the end effector coordinate system.

[0073] Next, in step S8, the contact information recognition unit 10 determines whether or not the entry conditions are met for all the recognized contact information candidates. Figure 13 is an explanatory diagram of the determination process for whether or not the entry conditions are met according to the first embodiment. Here, the end effector 5 of the robot 7 shown in Figure 1(a) can approach the target workpiece 21 only from the opening side (upper side) of the workpiece housing 6. The entry condition is, for example, that when the robot moves to the posture (approach posture) among the position and posture 130 shown in Figure 10, the tip of the end effector 5 is not facing the opening side of the workpiece housing 6.

[0074] Therefore, in step S8, the contact information recognition unit 10 excludes, as shown in the example on the right in Figure 13, any candidate contact information in which the tip of the end effector 5 is facing the opening of the workpiece housing 6 in the approach posture based on the end effector coordinate system calculated in step S7 of Figure 11, because it does not satisfy the entry conditions, i.e., it is impossible to approach.

[0075] Next, in step S9, the contact information recognition unit 10 performs interference determination. The interference determination is performed to determine whether there is interference between the end effector 5 and the workpiece housing unit 6, and whether there is interference between the end effector 5 and workpieces 22 other than the target workpiece 21 within the workpiece housing unit 6.

[0076] Figure 14 is a schematic diagram illustrating interference determination between the workpiece housing 6 and the end effector 5 according to the first embodiment. The region 100 shown in Figure 14 is the three-dimensional region of the smallest rectangular parallelepiped in which the end effector 5 is inscribed. Region 100 is positioned at an approachable position and orientation 130 included in the candidate contact information. The workpiece housing 6 is positioned in advance, and the positions of all inner wall surfaces 62 of the workpiece housing 6 are known to the robot controller 1. In this case, interference determination can be made by performing an intersection determination to determine whether the inner wall surfaces 62 of the workpiece housing 6 intersect with each edge of region 100. If the inner wall surfaces 62 of the workpiece housing 6 intersect with at least one edge of region 100, it is determined that there is "interference". If none of the edges of region 100 intersect with the inner wall surfaces 62 of the workpiece housing 6, it is determined that there is "no interference".

[0077] Figure 15 is a schematic diagram illustrating interference detection between the workpiece in the workpiece housing 6 and the end effector 5 according to the first embodiment. Spatial information within the workpiece housing 6 can be acquired as point cloud information from the sensor 3.

[0078] Region 111 is defined as the area of ​​the tip of the end effector 5, i.e., the finger 51, positioned at the approach position and orientation 130 determined from the contact information. The contact information recognition unit 10 determines that there is "no interference" if no points from the point cloud 110 are present within region 111. The contact information recognition unit 10 determines that there is "interference" if even one point from the point cloud is present within region 111.

[0079] However, all detected point clouds include the point cloud 112 of the target workpiece 21. Since the point cloud 112 of the target workpiece 21 is a contact target, it needs to be excluded from the interference determination. Therefore, the contact information recognition unit 10 excludes the point cloud 110 from the point cloud 110 if the distance from the coordinate origin of the target workpiece 21 is less than or equal to a threshold, and identifies it as the point cloud 112 of the target workpiece 21. The threshold is the radius of the sphere that circumscribes the target workpiece 21. However, the threshold may be arbitrarily set by the user. In Figure 15, the point indicated by the dashed arrow in the point cloud 110 is included in the region 111, so it is determined that "interference exists".

[0080] In step S10, the contact information recognition unit 10 recognizes a candidate for contact information as contact information if it determined in step S8 that it is permissible to enter and in step S9 that there is no interference. The above recognition method is not limited to the workpiece 20 located at the edge of the workpiece storage unit 6, but can be applied to any workpiece located within the workpiece storage unit 6 whose position and orientation can be detected.

[0081] Next, a method for setting candidate contact information will be described. Figure 16 is a schematic diagram illustrating a method for setting candidate contact information according to the first embodiment. The contact information includes at least information on the contact position of the end effector 5 that can contact the target workpiece 21, with reference to the work coordinate system of the target workpiece 21. In this embodiment, the contact information includes information on the contact position and orientation 131 of the end effector 5 that can contact the target workpiece 21, with reference to the work coordinate system of the target workpiece 21, as well as information on the external force vector applied to the target workpiece 21. The external force vector is a vector of external force applied to the target workpiece 21 that moves the target workpiece 21 to an empty area without rotating the target workpiece 21.

[0082] In this embodiment, in order to set candidate contact information, three positions and orientations are defined based on the work coordinate system of the target workpiece 21: the position and orientation (approach position and orientation) 130 in which the end effector 5 begins to approach the target workpiece 21; the position and orientation (entry position and orientation) 132 in which the end effector 5 approaches the contact portion of the target workpiece 21 in a predetermined direction and enters the vicinity of the contact portion of the target workpiece 21; and the position and orientation (contact position and orientation) 131 in which the end effector 5 contacts the contact portion of the target workpiece 21. Of these positions and orientations 130, 132, and 131, the contact position and orientation 130 is set as the contactable position and orientation in the contact information.

[0083] The entry position and orientation 132, and the approach position and orientation 130 are determined from the contact position and orientation 131 using a method described later. In addition, the inverse vector 74, which is in the opposite direction to the normal vector in the minute plane 70 on the target workpiece 21 that the end effector 5 contacts, is included as information about the external force vector in the contact information.

[0084] Figure 17 is an explanatory diagram of the end effector 5 according to the first embodiment. The fingers 51 of the end effector 5 are provided with a minute surface 140 that contacts the target workpiece 21. A coordinate system is defined for the minute surface 140 with its origin at its center. Note that the coordinate system of the minute surface 140 and the end effector coordinate system of the end effector 5 are different from each other.

[0085] Next, we will explain how to set specific contact information candidates. The contact information candidates are set, for example, by a 3D simulation using a 3D model. The contact information recognition unit 10 takes in the 3D model (information) of the end effector 5 and the 3D model (information) of the workpiece 20.

[0086] The 3D models of the end effector 5 and the workpiece 20 are represented by meshes. Figure 18 is an explanatory diagram of the 3D model 400 according to the first embodiment. As shown in Figure 18, the 3D model 400 is a collection of multiple planes 401. A plane 401 is, for example, a triangular plane. A minute plane 70 in the target workpiece 21 that the end effector 5 contacts can be considered as a plane in the 3D model of the target workpiece 21 represented by a mesh.

[0087] Figure 19 is a flowchart showing the process of setting contact information candidates according to the first embodiment. First, the contact information recognition unit 10 selects any one plane from the mesh that constitutes the 3D model of the target workpiece 21. For example, if the 3D model 400 shown in Figure 18 is the 3D model of the target workpiece 21 to be held, then any one is selected from the group of triangular planes that constitute the 3D model 400. The selection method and selection order are not limited.

[0088] In step S15, the contact information recognition unit 10 determines whether the normal vector 73 in the triangular mesh generates a moment on the target workpiece 21.

[0089] Figure 20 is a schematic diagram illustrating a method for determining the generation of a moment according to the first embodiment. When the starting point of the normal vector 73 in the triangular mesh is the contact point Pi, the angle θ is the smaller of the two angles (superior angle and inferior angle) between the vector 72 passing through the contact point Pi from the centroid 71 of the target workpiece 21 and the normal vector 73.

[0090] The contact information recognition unit 10 determines "no moment" if the angle θ is smaller than the threshold θt. If the angle θ is greater than or equal to the threshold, the contact information recognition unit 10 determines "moment present" and proceeds to determine the next mesh. The threshold θt should be set to a value within the range of less than ±90 degrees, but it is preferable to set it to a value within the range of ±1 degree. This is because if the absolute value of the threshold is too large, the moment generated when the end effector 5 applies an external force to the target workpiece 21 may cause the target workpiece 21 to not move as intended.

[0091] If it is determined that there is no moment in step S15, the contact information recognition unit 10 proceeds to the process in step S16.

[0092] In step S16, the contact information recognition unit 10 sets the inverse vector 74 of the normal vector 73 as the external force vector in the contact information. In this case, the contact information recognition unit 10 sets the origin of the minute surface 140 of the end effector 5 to coincide with the contact point Pi.

[0093] In step S17, the contact information recognition unit 10 moves the 3D model of the end effector 5 by a predetermined distance in the direction of the normal vector 73.

[0094] Figure 21 is a schematic diagram illustrating the process of setting candidate contact information according to the first embodiment. As shown in Figure 21, the contact information recognition unit 10 moves the 3D model 90 of the end effector 5 by a distance d1 from the contact point Pi in the direction of the normal vector 73.

[0095] In step S18, the contact information recognition unit 10 searches for candidate entry positions and orientations. Specifically, after moving the 3D model 90 in step S17, the contact information recognition unit 10 rotates the 3D model 90 with the center of the minute surface 140 shown in Figure 17 as the origin, and searches for states in which the 3D model 90 can take as an entry position and orientation.

[0096] In this case, if the 3D model 90 is rotated continuously, the number of positions and orientations that are subject to interference determination, as described later, will increase, causing the computation time to diverge. Therefore, in the first embodiment, the contact information recognition unit 10 searches by quantizing the rotation angle of the 3D model 90.

[0097] Figure 22 is a schematic diagram illustrating the rotation of the 3D model 90 of the end effector 5 according to the first embodiment. As shown in Figure 22, the contact information recognition unit 10 sets a sphere 410 centered on the minute surface 140, divides the sphere 410 at a constant latitude and constant longitude, and defines a curved surface 411 enclosed by two adjacent meridians and two adjacent parallels. At this time, the angle (θ,φ) of the straight line passing through the center of the curved surface 411 is used as the search angle. The parallels and meridians can be set to any width that allows the calculation to be performed in a realistic amount of time using the user's computing resources. However, the quantization method is not limited to this method.

[0098] In step S19, the contact information recognition unit 10 performs interference detection on the position and orientation of the rotated 3D model 90. Existing methods such as ray casting can be used for interference detection.

[0099] Figure 23 is a schematic diagram illustrating interference detection according to the first embodiment. In the raycasting method, a ray is projected perpendicularly from the mesh of the 3D model, and if it passes through the mesh along the way, the position of the intersection and the distance from the mesh of the 3D model to the intersection can be obtained. In the first embodiment, a ray is projected from the tip surface of the finger of the 3D model 90 of the end effector 5 to the base of the finger. If the 3D model 90 is interfering with another object model, the mesh 91 constituting the interfering object model intersects with the 3D model 90 of the end effector 5. Therefore, the intersection is obtained by the raycasting method. The position and orientation of the 3D model 90 that is determined to have no interference in the interference detection are taken as the entry position and orientation.

[0100] Figure 21 shows that there was no interference at position and orientation 190 and position and orientation 191. Once the entry position and orientation are determined, the contact position and orientation are obtained by offsetting them by a distance d1 in the opposite direction to the normal vector 73.

[0101] In step S20, the contact information recognition unit 10 moves the 3D model 90 of the end effector 5 by an arbitrary distance d2 in the direction of the root of the end effector 5, as shown in Figure 21, for the positions and orientations 190 and 191 where no interference was determined in step S19. The position and orientation of the 3D model 90 after the movement are designated as candidates for the approach position and orientation.

[0102] Next, in step S21, the contact information recognition unit 10 performs interference determination on candidate approach positions and attitudes in the same manner as for entry positions and attitudes. If it is determined that there is "no interference" in step S21, the contact information recognition unit 10 proceeds to the process in step S22. In step S22, the contact information recognition unit 10 sets the candidate approach positions and attitudes as the approachable positions and attitudes for contact information. In Figure 21, position and attitude 192 becomes the approachable position and attitude and is set as a candidate for contact information.

[0103] By the method described above, candidate contact information is set, including information on the position and orientation that the end effector 5 can approach, as well as information on external force vectors. Note that the method for setting candidate contact information is not limited to the example described above.

[0104] Next, we will specifically explain the process of recognizing the free area in step S2 of Figure 6. Figure 24 is a flowchart of the process of recognizing the free area (process of step S2) according to the first embodiment.

[0105] In step S11, the environmental recognition unit 11 images the state of the inner bottom surface 61 of the workpiece storage unit 6. Spatial information within the workpiece storage unit 6 can be acquired as point cloud information by the sensor 3. The environmental recognition unit 11 identifies the position of the point cloud with a Z coordinate below a certain level as the inner bottom surface 61 of the workpiece storage unit 6. Through this process, the environmental recognition unit 11 images the arrangement state of the workpieces 20 on the inner bottom surface 61 of the workpiece storage unit 6.

[0106] Figure 25 is a schematic diagram of the image IM1 generated by the environment recognition unit 11 according to the first embodiment. In step S11, the environment recognition unit 11 generates the image IM1 shown in Figure 25. In Figure 25, the inner bottom surface 120 corresponding to the inner bottom surface 61 of the work storage unit 6, the workpiece 121 corresponding to the target workpiece 21 to be held (moved), and the workpiece 122 corresponding to the other workpiece 22 are drawn. Image IM1 is an 8-bit binary image, with the areas where workpieces 121 and 122 exist having a gray value of 0, and the exposed portion of the inner bottom surface 120 having a gray value of 255. The bit depth of image IM1 can be any value, and the gray value of the exposed portion can also be any value as long as it is a positive value other than 0. The sizes of the imaged inner bottom surface 120 and workpieces 121 and 122 can be any size as long as their relative relationship is maintained. However, it is necessary that the position of each pixel can be converted to a position based on the sensor coordinate system.

[0107] Figure 26 is a schematic diagram of the image IM2 generated by the environment recognition unit 11 according to the first embodiment. In step S12, the environment recognition unit 11 generates image IM2 by adding pixels with a gray value of 0 along the outer edge of image IM1 to the image IM1 generated in step S11 in order to add information about the wall of the work housing unit 6. The number of pixels added in the thickness direction of the wall may be 1 pixel or more than 1 pixel.

[0108] Figure 27 is a schematic diagram of the distance image IM3 according to the first embodiment. In step S13, the environment recognition unit 11 calculates the distance image IM3 from the image IM2 generated in step S12. Specifically, the environment recognition unit 11 selects each of the multiple pixels of image IM2 as a pixel of interest, calculates the Euclidean distance between the pixel of interest and the closest pixel with a gray value of 0, and calculates the distance image IM3 by making the pixel of interest a pixel with a gray value corresponding to the Euclidean distance.

[0109] The conversion method from image IM2 to distance image IM3 can be calculated using the technique disclosed in "CR Maurer; Rensheng Qi; V. Raghavan, A linear time algorithm for computing exact Euclidean distance transforms of binary images in arbitrary dimensions."

[0110] Figure 28 is a schematic diagram of the binarized image IM4 according to the first embodiment. In step S14, the environment recognition unit 11 generates a binarized image IM4 by binarizing the distance image IM3 obtained by conversion in step S13. The binarization threshold is set to, for example, the maximum distance (in pixels) of the free area to be searched. As a result, only areas whose distance from the workpiece is greater than or equal to the threshold are extracted. Figure 29 is an explanatory diagram showing the maximum distance 27 of the minimum free area 26 shown in Figure 7 according to the first embodiment. The threshold is set to, for example, the maximum distance 27.

[0111] In Figure 28, the white area 124 is the area that is at least 27 units away from the walls of the workpiece housing 6 and the workpiece, corresponding to the maximum distance 27 of the minimum free area 26. Therefore, by operating the end effector 5 to a position corresponding to any pixel within area 124, the target workpiece 21 can be moved to the free area 33. Each of the pixels constituting area 124 becomes the center 34 of the free area 33 determined in step S2 of Figure 6. The threshold can be any value as long as it is greater than or equal to 27 units of the maximum distance 26 of the minimum free area 26, and the larger the threshold, the more areas are found that do not have workpieces or walls around them.

[0112] The environment recognition unit 11 selects an empty area 33 to which the target workpiece 21 will move, using a predetermined method. For example, the environment recognition unit 11 may select the center 34 of the empty area 33 with the shortest Euclidean distance from the approach position from area 124. A specific selection method will be explained using Figure 30. IM5 in Figure 30 represents the external force vector 720 of the contact information and the contactable position 721 selected in the situation of IM1 in Figure 25. At this time, the environment recognition unit 11 creates an IM6 of the same size as IM4 in Figure 28, where only the pixels corresponding to the contactable position 721 have a gray value of 0. Next, IM6 is converted to IM7 using the same method used when converting from IM3 to IM4. Next, IM7 is masked with IM4 to obtain the Euclidean distance between each pixel of area 124 and the contactable position 721. The pixel of area 124 with the shortest Euclidean distance is selected as area 33.

[0113] The environmental recognition unit 11 may select any two or more points from region 124. Figure 31 is an explanatory diagram of information processing in another example according to the first embodiment. As shown in Figure 31, the environmental recognition unit 11 uses the detection information from the sensor 3 to acquire information of the empty region 33 as the first region and information of the empty region 150 as the second region. The empty region 150, like the empty region 33, is a region in the work storage unit 6 where the target workpiece 21 can be moved.

[0114] Regarding the information about the empty area 33, the Euclidean distance between the line overlapping the external force vector and the center 34 of the empty area 33 is evaluated from the information about the external force vector applied to the target workpiece 21 contained in the contact information 31. In the example in Figure 31, the contact information 31 is selected as contact information for moving the target workpiece 21 to the empty area 33.

[0115] Regarding the information on the empty area 150, the Euclidean distance between the external force vector and the center 151 of the empty area 150 is evaluated based on the coordinate point where the line obtained from the external force vector information contained in the contact information 31 is closest to the center 34 of the empty area 33. In the example in Figure 31, the contact information 152 is selected as contact information for moving the target workpiece 21 from the empty area 33 to the empty area 150. The environment recognition unit 11 may recognize the empty area 33 and the empty area 150 under different conditions. For example, when the environment recognition unit 11 recognizes the empty area 150 as the destination, it can increase the threshold of the binarization process to move the target workpiece 21 to a position away from surrounding workpieces and the walls of the workpiece housing 6. That is, in the control process of step S5-1, the control unit 13 makes contact with the target workpiece 21 with the end effector 5 to move the target workpiece 21 to the empty area 33, and then controls the robot 7 to move the target workpiece 21 from the empty area 33 to the empty area 150.

[0116] Next, we will explain the specific method for selecting contact information in step S3 of Figure 6. The motion planning unit 12 can select any one point from the region 124, which shows candidate empty areas obtained by the flowchart in Figure 24, and then select contact information using the method in Figure 8. The amount of operation of the end effector in step S4 of Figure 6 can be calculated using the method described above with reference to Figure 9.

[0117] Next, the method for planning the movement of the robot 7 in step S5 of Figure 6 will be explained. When the robot 7 moves so that the end effector 5 moves according to the contact information selected in step S3, the end effector 5 moves to the position and orientation 130 shown in Figure 16. In order for the end effector 5 to move from this position and orientation 130 to position and orientation 131, the end effector 5 must move a distance d2 in the Z direction relative to the coordinate system of the end effector 5 from position and orientation 130, and then move a distance d1 in the direction of movement of the end effector 5. After that, the end effector 5 can move the target workpiece 21 by moving by the amount of movement determined in step S4 in the direction of movement. Then, in the control processing of step S5-1, the control unit 13 controls the robot 7 according to the movement plan created using the contact information in step S5. That is, in the control processing of step S5-1, the control unit 13 brings the end effector 5 into contact with the target workpiece 21 so that the direction of the external force acting on the target workpiece 21 is towards the region 33. The control unit 13 then moves the end effector 5, thereby moving the target workpiece 21 in the direction of the end effector 5's movement.

[0118] Specifically, the control process in step S5-1 includes, as shown in Figure 16, a first control process SA1 that moves the end effector 5 from the approach position (130) to an entry position (132) facing the target workpiece 21 in a predetermined direction; a second control process SA2 that moves the end effector 5 from the entry position (132) in a predetermined direction to bring the end effector 5 into contact with the target workpiece 21; and a third control process SA3 that, after the end effector 5 has made contact with the target workpiece 21, moves the end effector 5 in a predetermined direction to apply an external force to the target workpiece 21 and move the target workpiece 21. The control unit 13 maintains the end effector 5 in a constant position during these first control process SA1, second control process SA2, and third control process SA3. This operation of the robot 7 increases the reliability of moving the target workpiece 21 to the empty area 33.

[0119] The control unit 13 displays images on the display 109 in Figure 2 that present the recognized contact information, available area, and operation plan results to the user. Figures 32 and 33 are explanatory diagrams of a GUI (Graphical User Interface) 200, which is an example of a display image according to the first embodiment. The GUI 200 includes an overhead image window 201 that displays image information captured of the workpiece housing, a contact information display unit 206 that displays the recognized contact information in a list, and an operation plan display unit 210.

[0120] The overhead image window 201 displays the images captured by sensor 3, which were used to recognize contact information and environmental information. Empty areas are overlaid on the captured images. The center cluster 202 of the empty areas is displayed in white, while the rest is displayed in gray.

[0121] Furthermore, the center of the empty area selected during the operation planning stage is displayed as an X mark 204. Recognized contact information is also displayed as an arrow 203. The data for each contact is displayed as a list on the contact information display unit 206.

[0122] The contact information display unit 206 has a fixed size, and the user can view all contact information data by scrolling the scroll bar 209. When the user clicks on a contact information they want to focus on from the contact information display unit 206, for example, contact information 207, the color of the list changes, and an arrow corresponding to the selected contact information is highlighted, as shown by arrow 205.

[0123] Furthermore, the motion plan result 208 is displayed as an arrow, as shown in Figure 33. The motion plan result 208 is displayed at the time the motion plan is executed by the motion plan unit 12. The arrow indicating the motion plan result 208 extends from the contactable position of the contact information to the destination position of the end effector 5. At this time, in the motion plan display unit 210, the number of the contact information used for the motion plan is displayed in the used contact information display unit 211, the amount of movement of the end effector 5 is displayed in the amount of movement display unit 212, and the predicted destination position of the target workpiece 21 is displayed in the predicted destination display unit 213.

[0124] By implementing the above method, the robot system 1000 shown in Figure 1(a) can plan an operation to move the target workpiece 21 to be held to a position where there are no inner walls 62 of the workpiece housing 6 or other workpieces around it. This makes it possible to hold the target workpiece 21 in the end effector 5, improving the production efficiency of the products by the robot system 1000. Thus, the first embodiment provides a technology that is advantageous for the robot 7 to hold the workpiece 20.

[0125] [Second Embodiment] A second embodiment will now be described. In the first embodiment described above, the case in which the movement of the robot 7 is planned using one contact information was explained as an example. The end effector 5 in Figure 1(a) is equipped with a parallel chuck having two fingers 51, so it is also possible to move the target workpiece 21 by using two contact information to bring the two fingers 51 into contact with the target workpiece 21. This makes it easier to move the target workpiece 21 without rotating it, compared to when an external force is applied from one point. Below, in the second embodiment, the method of setting the contact information will be described as a difference from the first embodiment.

[0126] Figure 34 is a flowchart showing the process of setting candidate contact information according to the second embodiment. In the second embodiment, the contact information recognition unit 10 performs each process in the flowchart shown in Figure 34, in addition to the flowchart shown in Figure 19. In the second embodiment, the contact information recognition unit 10 creates a pair of contact information that can be touched by the two fingers 51 of the end effector 5.

[0127] First, in step S23, the contact information recognition unit 10 groups the set contact information candidates. The external force vector applied to the target workpiece 21 is quantized in the manner shown in Figure 22, and contact information with external force vectors belonging to the same curved surface 411 is grouped together.

[0128] Next, in step S24, the contact information recognition unit 10 subgroups all the contact information within the group created in step S23 into subgroups based on approachable orientations. Subgrouping can be done in the same way as in step S23, by grouping together those that will have the same orientation after quantization.

[0129] Next, in step S25, the contact information recognition unit 10 calculates a composite vector of the external force vectors of any two contact information within the subgroup. In this case, the contact information recognition unit 10 performs vector synthesis for all combinations. However, if the distance between the approachable positions is greater than the maximum opening amount of the two fingers 51 of the end effector 5, vector synthesis is not performed.

[0130] Next, in step S26, the contact information recognition unit 10 checks whether all the composite vectors calculated in step S25 generate a moment. The same method as in step S15 can be used for this check.

[0131] The contact information recognition unit 10 defines two contact information sets, each having a composite vector determined to have no moment, as a pair of contact information sets to be used when bringing the two fingers 51 into contact in step S27. However, the pair of contact information sets consists of the contactable position, the contactable orientation, the opening degree of the end effector, and the external force vector applied to the target workpiece 21.

[0132] For the contactable posture, the quantized contactable postures of the two contact information pieces forming the pair should be used. For the contactable position, the average value of the contactable positions of the two contact information pieces forming the pair should be used. For the end effector opening, the Euclidean distance between the contactable positions of the two contact information pieces forming the pair should be used. For the external force vector applied to the workpiece, the composite vector of the external force vectors of the two contact information pieces forming the pair should be used.

[0133] Next, the method for recognizing contact information pairs and the method for planning operations will be described. First, contact information is recognized in the same manner as in the first embodiment. Then, contact information that can be used to create a predefined pair is selected from the recognized contact information. In this way, a pair of contact information is recognized.

[0134] Once a pair of contact information is created, environmental information is recognized using the flowchart in step S2. Next, contact information is selected in step S3. At this time, the contactable position and orientation of the contact information pair are used for the entry conditions in step S8 and the interference determination in step S9.

[0135] Next, in step S4, the amount of movement of the end effector 5 is calculated. The contactable position and orientation of the contact information pair are used to calculate the amount of movement. Finally, in step S5, the movement of the end effector 5 is planned.

[0136] The control unit 13 causes the end effector 5 to assume the contactable position and orientation of the contact information pair, and controls the end effector 5 to open by the opening degree of the end effector 5 of the contact information pair. The operation can be planned in the same manner as in the first embodiment.

[0137] [Third Embodiment] A third embodiment will now be described. In the first embodiment described above, a method for moving one target workpiece 21 to an empty area was explained. In the third embodiment, a method for planning an operation to secure a path for the target workpiece 21 to move is explained, by moving one or more workpieces that are on the path to move the target workpiece 21 to an empty area, to an empty area that does not affect the movement of the target workpiece 21. It is assumed that two or more workpieces 20 remain in the workpiece storage unit 6.

[0138] Figure 35 is a flowchart for the process of removing a workpiece from the path according to the third embodiment. Figure 36 is a schematic diagram illustrating the process according to the third embodiment. The process in the flowchart of Figure 35 is executed between the process in step S4 and the process in step S5 of Figure 6.

[0139] First, in step S28, the motion planning unit 12 calculates the positions of all workpieces 20 within the field of view of the sensor 3. The method for calculating the positions is the same as the method for calculating the positions and orientations in step S1 of Figure 6. Furthermore, if the positions and orientations of all workpieces 20 within the field of view of the sensor 3 have already been calculated in step S1, step S28 can be omitted.

[0140] Next, in step S29, the motion planning unit 12 determines the path 300 for moving the target workpiece 21. The target workpiece 21 is the workpiece to be held (moved). The path 300 is the path from the calculated position of the target workpiece 21 to a position within the empty area 33 in Figure 9. The path 300 is, for example, a straight line path along the external force vector applied to the target workpiece 21 from the contact information selected in step S3 in Figure 6.

[0141] Next, in step S30, the motion planning unit 12 calculates the distance between the path 300, which was determined in step S29, and the workpieces 22 other than the target workpiece 21. The motion planning unit 12 calculates the shortest Euclidean distance between the path 300 and the workpieces 22 as the distance.

[0142] Next, in step S31, if any of the one or more workpieces 22 are on the path 300, the motion planning unit 12 selects that workpiece to be moved. To explain with a specific example, the motion planning unit 12 performs a threshold determination process on the distance obtained in step S30, and selects workpieces that are less than or equal to the threshold distance to be moved. In this way, among the one or more workpieces 22, the workpiece that is close to the path 300 on which the target workpiece 21 will be moved is selected as the workpiece to be moved.

[0143] The threshold can be the maximum distance 27 (in pixels) of the minimum free area 26 shown in Figure 7, which is defined for the target workpiece 21 used by the environment recognition unit 11. The threshold may also be set to any positive value by the user.

[0144] Next, in step S32, the motion planning unit 12 calculates the shortest Euclidean distance between the path 300 and each pixel of the empty region 33 recognized in step S2 of Figure 6. Figure 37 is a schematic diagram illustrating the distance calculation method according to the third embodiment. The motion planning unit 12 calculates the distance between the path 300 and each pixel 301 that constitutes the white empty region in Figure 37. For simplicity, only a portion of the pixels 301 are shown in Figure 37.

[0145] Next, in step S33, the motion planning unit 12 calculates the empty area to which the workpiece 20 will be moved. This process is similar to the process in step S31, and can be performed by determining the distance between the path 300 calculated in step S32 and the pixels 301 of the empty area using a threshold.

[0146] The threshold should be a value of at least half the maximum distance (in pixels) of the minimum free area 26 defined for the workpiece 20 to be held, as used by the environment recognition unit 11.

[0147] Finally, in step S34, the motion planning unit 12 plans the operation of the end effector 5 in the same manner as in the first embodiment, based on the contact information of the workpiece to be moved selected in step S31 and the information of the available area to be moved, calculated in step S34.

[0148] In this manner, the motion planning unit 12 plans to move other workpieces that are present on the path 300 before moving the target workpiece 21. Then, prior to the control process of step S5-1 shown in Figure 6, the control unit 13 controls the robot 7 to move the other workpieces off the path 300. This prevents interference between the target workpiece 21 and other workpieces 22 when moving the target workpiece 21, making it easier to move the target workpiece 21 into the intended empty area 33.

[0149] [Fourth Embodiment] A fourth embodiment will now be described. In the first embodiment described above, a case was described in which the robot controller 1 calculates the posture of the workpiece and then calculates contact information using predefined contact information candidates. In the fourth embodiment, a method for calculating contact information using machine learning will be described.

[0150] First, let's explain the machine learning model used to calculate contact information. For example, we will use a model called U-Net. U-Net can perform semantic segmentation, taking an image as input and outputting a mask image with specific regions of the image filled in. Note that any model capable of semantic segmentation is acceptable, not just U-Net.

[0151] Machine learning can be performed by a computer. That is, it can be performed by the robot controller 1, or by a computer other than the robot controller 1. The following explanation will use the case where the robot controller 1 performs machine learning as an example.

[0152] The robot controller 1 performs machine learning using the machine learning model described above. The machine learning in this embodiment is supervised learning. Since machine learning requires training data, the method for creating the training data will be explained first.

[0153] Figure 38 is an explanatory diagram of the training data 600 according to the fourth embodiment. One of the multiple training data 600 has image data 610 and ground truth data 620. Image data 610 is input data, for example, an image (raw image) of the workpiece 20. Ground truth data 620 is composed of a set of three types of image data: a contactable position image 601, a contactable angle image 602, and an external force vector image 603.

[0154] Each of the three types of image data 601 to 603 is created from image data 610, which is an image of the workpiece 20 placed in the workpiece storage unit 6, based on pre-set contact information. Therefore, each of the three types of image data 601 to 603 of the correct answer data 620 included in each of the multiple training data 600 has the same number of pixels in the width and height as the image data 610, which is the captured image.

[0155] The contactable location image 601 is a one-channel binary image in which the locations accessible to the workpiece displayed on the image are represented by a gray value of 255, and all other locations by a gray value of 0. Note that the gray values ​​for the accessible locations and other locations do not have to be 255 and 0, as long as they can be distinguished. Also, in the contactable location image 601, the workpiece's position is represented by a dashed line for explanatory purposes in Figure 38, but the actual contactable location image 601 does not have a dashed line representing the workpiece's position.

[0156] The contactable angle image 602 is a three-channel image representing the Rx, Ry, and Rz component values ​​of the contact orientation in the contact information. Figure 38 shows only the channel image for the Rz component, omitting the channel images for the Rx and Ry components. Each channel image corresponds to each component of the contact orientation, and the gray value of a pixel in the channel image represents the value of the corresponding component of the contact orientation at the position of that pixel. Note that Rx, Ry, and Rz must be quantized beforehand to 255 levels from 1 to 255. In Figure 38, the position of the workpiece is shown with a dashed line for explanatory purposes in the contactable angle image 602, but the actual contactable angle image 602 does not have a dashed line representing the position of the workpiece.

[0157] The external force vector image 603 is an image representing the direction of the external force vector at each pixel, and, similar to the contactable angle image 602, represents a 1-channel image in which the direction of the external force vector is quantized as an angle to 255 levels. In this case, the direction of motion represents the angle when rotated clockwise, with the top of the image plane set to 0 degrees. Alternatively, this can be extended to a 2-channel image, where the second channel represents the angle with the vertically upward direction of the paper set to 0 degrees, thereby representing the 3D orientation. In Figure 38, the position of the workpiece is shown with a dashed line for explanatory purposes in the external force vector image 603, but the actual external force vector image 603 does not have a dashed line representing the position of the workpiece.

[0158] Next, we will explain how to train a machine learning model using training data 600 in the learning phase (training process). The input data (example data) used during machine learning is image data 610 of a workpiece placed in the workpiece storage unit 6. The ground truth data 620, which serves as the training target for the machine learning model, is data created by combining three types of images in the channel direction: a contactable position image 601, a contactable angle image 602, and an external force vector image 603. The robot controller 1 performs machine learning using multiple training data 600 to generate a trained machine learning model.

[0159] Next, we will explain how to calculate contact information from the output of the trained machine learning model during the inference phase (inference processing). Figure 39 is an explanatory diagram of the inference according to the fourth embodiment. The robot controller 1 performs inference processing using the trained machine learning model M1.

[0160] The robot controller 1 acquires image data 630, which is an image captured by the sensor 3, and uses the image data 630 as input data for the machine learning model M1. Using the machine learning model M1, the robot controller 1 generates an image of a contactable position 641, an image of a contactable angle 642, and an external force vector image 643 as output data 621 from the input image data 630.

[0161] The contactable position image 641 is an image containing information about the contactable position (third position) of the end effector 5. The contactable angle image 642 is an image containing information about the contactable orientation of the end effector 5. The external force vector image 643 is an image containing information about the external force applied to the target workpiece 21.

[0162] Figure 40 is an explanatory diagram of the masking process according to the fourth embodiment. Next, the robot controller 1 generates image 351 by masking the contactable position image 642 with the contactable angle image 642, and generates image 352 by masking the external force vector image 643 with the contactable position image 641. The robot controller 1 obtains contact position and orientation 131 (Figure 16) information by obtaining the angle of the end effector at the position where the end effector can make contact on image 351, and obtains external force vector information on image 352. In this way, the robot controller 1 obtains contact information (i.e., control information). Thus, in the masked images 351 and 352, information of pixels with a gray value that is not 0 can become contact information.

[0163] As described above, according to the fourth embodiment, the robot controller 1 can calculate contact information using the machine learning model M1. This eliminates the need to calculate the position and orientation of the target workpiece 21, thereby improving the cycle time of the device.

[0164] [Other variations] The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of this disclosure. For example, at least two of the embodiments described above may be combined. Furthermore, the effects described in this embodiment are merely a list of the most preferred effects arising from the embodiments of this disclosure, and the effects of the embodiments of this disclosure are not limited to those described in this embodiment.

[0165] In the embodiments described above, the case in which the robot arm 4 is a vertically articulated robot arm was described, but the invention is not limited to this. The robot arm 4 may be various types of robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot. Furthermore, this disclosure is applicable to machines that can automatically perform movements such as extension and retraction, bending and straightening, vertical movement, horizontal movement, or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0166] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0167] The above disclosure of embodiments includes the following sections.

[0168] (Section 1) A robot with an end effector, A sensor that detects the status of one or more workpieces placed in the workpiece storage area, The robot comprises a control device for controlling the robot, The control device is A selection process is performed to select a target workpiece from among the one or more workpieces to be held by the end effector, using the detection results obtained from the sensor. If the end effector does not satisfy the holding conditions necessary to hold the target workpiece, the workpiece storage unit acquires information about the area in which the target workpiece can move using the detection result, and The system is configured to perform a control process that controls the robot to bring the end effector into contact with the target workpiece and move the target workpiece to the area. A robotic system characterized by the following features.

[0169] (Section 2) The control device controls the robot to move the target workpiece and then cause the end effector to hold the target workpiece. The robot system according to item 1, characterized in that

[0170] (Section 3) The aforementioned region is an area on the inner bottom surface of the workpiece housing that is exposed from one or more workpieces. A robot system according to item 1 or 2, characterized in that it is a robot system according to item 1 or 2.

[0171] (Section 4) The aforementioned region is wider than the projection region obtained by projecting the target workpiece onto the inner bottom surface of the workpiece housing in the direction from the sensor toward the inner bottom surface of the workpiece housing. The robot system according to item 3, characterized in that

[0172] (Section 5) The direction from the sensor toward the inner bottom surface of the workpiece housing is perpendicular to the inner bottom surface of the workpiece housing. The robot system according to item 4, characterized in that

[0173] (Section 6) The aforementioned region is wider than the region that extends outward by a predetermined amount from the projection region. A robot system according to item 4 or 5, characterized by the features described herein.

[0174] (Section 7) The end effector has fingers, The predetermined amount is the thickness of the finger. The robot system according to item 6, characterized in that

[0175] (Section 8) The control device controls the robot to move the end effector in a straight line when moving the target workpiece to the area in the control process. A robot system according to any one of claims 1 to 7, characterized in that it is a robot system according to any one of claims 1 to 7.

[0176] (Section 9) The control device controls the robot to maintain a constant posture of the end effector when moving the target workpiece to the area during the control process. A robot system according to any one of claims 1 to 8, characterized in that it is a robot system according to any one of claims 1 to 8.

[0177] (Section 10) The control device, in the control process, brings the end effector into contact with the target workpiece such that the direction of the external force acting on the target workpiece is toward the region. A robot system according to any one of claims 1 to 9, characterized in that it is a robot system according to any one of claims 1 to 9.

[0178] (Section 11) If there is another workpiece on the path from the target workpiece to the area, the control device controls the robot to move the other workpiece out of the path prior to the control process. A robot system according to any one of claims 1 to 10, characterized in that it is a robot system according to any one of claims 1 to 10.

[0179] (Section 12) The aforementioned region is the first region, In the information processing, the control device acquires information about the second region in the work storage unit where the target work can be moved, using the detection result. The control device, in the control process, controls the robot to move the target workpiece from the first region after bringing the end effector into contact with the target workpiece and moving the target workpiece to the first region. A robot system according to any one of claims 1 to 11, characterized in that it is a robot system according to any one of claims 1 to 11.

[0180] (Section 13) The aforementioned control process is A first control process moves the end effector from a first position to a second position facing the target workpiece in a predetermined direction, A second control process that moves the end effector from the second position in the predetermined direction to bring the end effector into contact with the target workpiece, The process includes a third control process which, after the end effector has come into contact with the target workpiece, moves the end effector in a predetermined direction to apply an external force to the target workpiece and move the target workpiece, A robot system according to any one of claims 1 to 12, characterized in that it is a robot system according to any one of claims 1 to 12.

[0181] (Section 14) The control device maintains the end effector in a constant position during the first control process, the second control process, and the third control process. The robot system according to item 13, characterized in that

[0182] (Section 15) The control device acquires control information used for controlling the robot, which includes at least the information of the first position, and uses the control information when executing the first control process, the second control process and the third control process. A robotic system according to item 13 or 14, characterized by the features described herein.

[0183] (Section 16) The control device is From the detection results, obtain one or more pieces of control information. From among the one or more pieces of control information, the control information to be used in the first control process, the second control process, and the third control process is acquired. The robot system according to item 15, characterized in that

[0184] (Section 17) The control device is The output data is acquired using a trained machine learning model in which the input data is the detection result obtained from the sensor, and the output data is information indicating the first position, information indicating the orientation of the end effector, and information on the external force applied to the target workpiece. The control information is acquired using the output data. The robot system according to item 15, characterized in that

[0185] (Section 18) Equipped with an additional display device, The control device causes the display device to display an image corresponding to the region. A robot system according to any one of claims 1 to 17, characterized in that it is a robot system according to any one of claims 1 to 17.

[0186] (Section 19) Equipped with an additional display device, The control device displays an image on the display device that includes the robot's motion plan in the control process. A robot system according to any one of claims 1 to 17, characterized in that it is a robot system according to any one of claims 1 to 17.

[0187] (Section 20) The sensor is a depth-sensing camera, and as a result of the detection, it acquires image information including depth information. A robot system according to any one of claims 1 to 19, characterized in that it is a robot system according to any one of claims 1 to 19.

[0188] (Section 21) A robot with an end effector, A sensor that detects one or more workpieces placed in the workpiece storage area, A control method for a robot system comprising a control device for controlling the robot, The control device uses the detection results obtained from the sensor to select a target workpiece from among the one or more workpieces to be held by the end effector. If the control device fails to satisfy the holding conditions necessary for the end effector to hold the target workpiece, it acquires information about the area in which the target workpiece can move within the workpiece housing using the detection result. The control device controls the robot to move the target workpiece to the area by bringing the end effector into contact with the target workpiece. A method for controlling a robot system characterized by the following features.

[0189] (Section 22) A robotic system used to manufacture articles, as described in any one of paragraphs 1 through 20. A method for manufacturing an article, characterized by the following:

[0190] (Section 23) A program to cause a computer to execute the control method described in item 21.

[0191] (Section 24) A computer-readable recording medium on which the program described in item 23 is recorded. [Explanation of symbols]

[0192] 1...Robot controller (control device), 3...Sensor, 4...Robot arm, 5...End effector, 6...Workpiece housing, 7...Robot, 20...Workpiece, 21...Target workpiece, 33...Empty area (area), 51...Finger, 61...Inner bottom surface, 62...Inner wall surface, 1000...Robot system

Claims

1. A robot with an end effector, A sensor that detects the status of one or more workpieces placed in the workpiece storage area, The robot comprises a control device for controlling the robot, The control device is A selection process is performed to select a target workpiece from among the one or more workpieces to be held by the end effector, using the detection results obtained from the sensor. If the end effector does not satisfy the holding conditions necessary to hold the target workpiece, the workpiece storage unit acquires information about the area in which the target workpiece can move using the detection result, and The system is configured to perform a control process that controls the robot to bring the end effector into contact with the target workpiece and move the target workpiece to the area. A robotic system characterized by the following features.

2. The control device controls the robot to move the target workpiece and then cause the end effector to hold the target workpiece. The robot system according to feature 1.

3. The aforementioned region is an area on the inner bottom surface of the workpiece housing that is exposed from one or more workpieces. The robot system according to feature 1.

4. The aforementioned region is wider than the projection region obtained by projecting the target workpiece onto the inner bottom surface of the workpiece housing in the direction from the sensor toward the inner bottom surface of the workpiece housing. The robot system according to claim 3, characterized in that it is the same as described in claim 3.

5. The direction from the sensor toward the inner bottom surface of the workpiece housing is perpendicular to the inner bottom surface of the workpiece housing. The robot system according to feature 4.

6. The aforementioned region is wider than the region that extends outward by a predetermined amount from the projection region. The robot system according to feature 4.

7. The end effector has fingers, The predetermined amount is the thickness of the finger. The robot system according to feature 6.

8. The control device controls the robot to move the end effector in a straight line when moving the target workpiece to the area in the control process. The robot system according to feature 1.

9. The control device controls the robot to maintain a constant posture of the end effector when moving the target workpiece to the area during the control process. The robot system according to feature 1.

10. The control device, in the control process, brings the end effector into contact with the target workpiece such that the direction of the external force acting on the target workpiece is toward the region. The robot system according to feature 1.

11. If there is another workpiece on the path from the target workpiece to the area, the control device controls the robot to move the other workpiece out of the path prior to the control process. The robot system according to feature 1.

12. The aforementioned region is the first region, In the information processing, the control device acquires information about the second region in the work storage unit where the target work can be moved, using the detection result. The control device, in the control process, controls the robot to move the target workpiece from the first region after bringing the end effector into contact with the target workpiece and moving the target workpiece to the first region. The robot system according to feature 1.

13. The aforementioned control process is A first control process moves the end effector from a first position to a second position facing the target workpiece in a predetermined direction, A second control process that moves the end effector from the second position in the predetermined direction to bring the end effector into contact with the target workpiece, The process includes a third control process which, after the end effector has come into contact with the target workpiece, moves the end effector in a predetermined direction to apply an external force to the target workpiece and move the target workpiece, The robot system according to feature 1.

14. The control device maintains the end effector in a constant position during the first control process, the second control process, and the third control process. The robot system according to claim 13, characterized in that it is the robot system according to claim 13.

15. The control device acquires control information used for controlling the robot, which includes at least the information of the first position, and uses the control information when executing the first control process, the second control process and the third control process. The robot system according to claim 13, characterized in that it is the robot system according to claim 13.

16. The control device is From the detection results, obtain one or more pieces of control information, From among the one or more pieces of control information, the control information to be used in the first control process, the second control process, and the third control process is acquired. The robot system according to claim 15, characterized in that it is the robot system according to claim 15.

17. The control device is The output data is acquired using a trained machine learning model in which the input data is the detection result obtained from the sensor, and the output data is information indicating the first position, information indicating the orientation of the end effector, and information on the external force applied to the target workpiece. The control information is acquired using the output data. The robot system according to claim 15, characterized in that it is the robot system according to claim 15.

18. Equipped with an additional display device, The control device causes the display device to display an image corresponding to the region. The robot system according to feature 1.

19. Equipped with an additional display device, The control device displays an image on the display device that includes the robot's motion plan in the control process. The robot system according to feature 1.

20. The sensor is a depth-sensing camera, and as a result of the detection, it acquires image information including depth information. The robot system according to feature 1.

21. A robot with an end effector, A sensor that detects one or more workpieces placed in the workpiece storage area, A control method for a robot system comprising a control device for controlling the robot, The control device uses the detection results obtained from the sensor to select a target workpiece from among the one or more workpieces to be held by the end effector. If the control device fails to satisfy the holding conditions necessary for the end effector to hold the target workpiece, it acquires information about the area in which the target workpiece can move within the workpiece housing using the detection result. The control device controls the robot to move the target workpiece to the area by bringing the end effector into contact with the target workpiece. A method for controlling a robot system characterized by the following features.

22. A robot system according to any one of claims 1 to 20 is used to manufacture an article. A method for manufacturing an article, characterized by the following:

23. A program for causing a computer to execute the control method described in claim 21.

24. A computer-readable recording medium having the program described in claim 23 recorded on it.

Images