Control device of a robot device that controls the position of a robot

Abstract

Control device (29) of a robot device (5, 6, 8, 9, 10) which moves a second element (91, 97) by a robot (1) and adjusts the relative position of the second element with respect to a first element (81, 87), comprising a visual sensor (25, 27) receiving the first element and the second element; an operation control unit (43) which outputs a command to drive the robot to the robot; and an image processing unit (31) which processes the image captured by the visual sensor, wherein a first feature section for detecting the position of the first element and a second feature section for detecting the position of the second element are predetermined, wherein the image processing unit comprises a feature amount detection unit (32) which detects, in the image captured by the vision sensor, a first feature amount with respect to a position of the first feature portion and a second feature amount with respect to a position of the second feature portion, a computing unit (33) which calculates the difference between the first feature amount and the second feature amount as a relative position amount, and a command generation unit (34) which generates a movement command for operating the robot, wherein a relative position amount is set in a reference image (61) containing an image of the first feature portion and the second feature portion when the second element is arranged at a target position with respect to the first element, wherein the command generation unit generates a movement command based on the relative position amount in the image (62, 63, 65) captured by the vision sensor and the relative position amount in the reference image containing the image of the first feature portion and the second feature portion, in order to operate the robot so that the second element is arranged at the target position with respect to the first element, wherein the operation control unit changes the position of the robot based on the movement command.

Description

[0001] The present invention relates to a control device of a robot device that controls the position of a robot.

[0002] A robot device equipped with a robot can perform a desired task by attaching a work tool corresponding to the task to the robot. For example, if a hand that grasps a workpiece is attached to the robot as a work tool, the robot device can transport the workpiece to a desired location.

[0003] When a workpiece is transported by a robot device, the workpiece gripped by the robot device can be attached to another workpiece or placed inside another workpiece by controlling the position and attitude of the workpiece. When performing such an operation, it is desirable to accurately position the workpiece gripped by the robot device with respect to the other workpiece. For example, when performing an operation in which one workpiece is joined to another workpiece, the operation may fail if the position and attitude of one workpiece differ from those of the other workpiece.

[0004] In the prior art, a method is known in which, when attaching one workpiece to another, the position of the robot is controlled using an image captured by a camera. For example, a control method is known in which a target image of one workpiece or the other workpiece is prepared in advance. As the workpiece is transported by the robot device, the workpiece is captured by a camera. Then, the position of the robot is adjusted by comparing the image of the workpiece with the target image (see, for example, JP 2013-180380 A and JP 2015-214022 A).

[0005] Furthermore, a controller is known in which a vision sensor coordinate system is calibrated in advance with respect to a robot coordinate system, and the three-dimensional position of the workpiece is calculated based on the position of the workpiece in the vision sensor coordinate system. Furthermore, a controller is known in which a Jacobian matrix can be created in advance with respect to the position and size of a feature portion in images. The position of the robot is corrected based on the position of the feature portion in an image captured by a camera, the position of the feature portion in target data, and the Jacobian matrix (see, for example, JP 2017-170599 A).

[0006] Furthermore, from DE 11 2018 005 783 T5 a device is known which comprises several means and sensors for gripping a core, moving it and measuring its position as well as physical quantities, for example forces and torques acting on the core.

[0007] DE 10 2016 212 695 B4 discloses an industrial robot which, with the aid of imaging sensors and precise control of the limbs, assumes a predetermined position of an effector in an absolute coordinate system.

[0008] Furthermore, DE 10 2012 112 025 B4 shows a method and an interaction of two moving systems with an object, wherein the radiation is used to determine their relative position.

[0009] DE 10 2009 034 529 B4 describes a system that enables a servo-operated manipulator to operate without a calibration target.

[0010] DE 10 2004 049 332 A1 shows a method for the automated positioning of two components using a plurality of industrial robots.

[0011] Furthermore, EP 2 915 635 A2 discloses a robot with a hand and a control unit that controls the hand.

[0012] In the method that uses a Jacobian matrix to regulate the position of the robot, the movement amount of the robot can be calculated by applying the Jacobian matrix to the difference in the feature amount of a feature portion in images. However, there may be cases where the Jacobian matrix cannot be calculated accurately due to measurement errors or the like when calculating the Jacobian matrix. Or the Jacobian matrix is ​​calculated based on the deviation of the position in the vision sensor coordinate system when the position of the robot has been moved by a tiny amount. Therefore, the robot can be driven with good accuracy near the feature portion, but the accuracy decreases with the distance from the feature portion. For example, there is a problem that the accuracy is poor when the robot is moved with a large movement amount.As a result, the position of the robot cannot be regulated with good accuracy.

[0013] In the vision sensor coordinate system calibration method, the three-dimensional position of a feature portion can be detected based on the image from a 2D camera. However, this method requires the position of the vision sensor coordinate system to be calibrated with respect to the robot coordinate system in advance.

[0014] The invention is therefore based on the object of providing a control device for a robot device by means of which a reliable, sensor-based control of the robot device can be achieved.

[0015] According to the invention, the above-mentioned object is achieved by the subject matter of claim 1.

[0016] Specifically, the object is achieved by a control device of a robot device that moves a second element by a robot and adjusts the relative position of the second element with respect to a first element. The control device includes a vision sensor that captures the first element and the second element. The control device includes an operation control unit that outputs a command to drive the robot to the robot, and an image processing unit that processes the image captured by the vision sensor. A first feature section for detecting the position of the first element and a second feature section for detecting the position of the second element are predetermined.The image processing unit includes a feature amount detection unit that detects, in the image captured by the vision sensor, a first feature amount related to a position of the first feature portion and a second feature amount related to a position of the second feature portion. The image processing unit includes a calculation unit that calculates the difference between the first feature amount and the second feature amount as a relative position amount. The image processing unit includes a command generation unit that generates a movement command for operating the robot. A relative position amount in a reference image including an image of the first feature portion and the second feature portion when the second element is located at a target position with respect to the first element is set.The command generation unit generates a movement command based on the relative position amount in the image captured by the vision sensor and the relative position amount in the reference image including the image of the first feature portion and the second feature portion, to operate the robot so that the second element is positioned at the target position with respect to the first element. The operation control unit changes the position of the robot based on the movement command. Fig. 1 is a schematic view of a first robot device in one embodiment. Fig. 2 is an enlarged oblique view of a first workpiece, a second workpiece and a hand in the first robot device. Fig. 3 is a block diagram of the first robot device in the embodiment. Fig. 4 is a first flowchart of a first control of the first robot device in the embodiment. Fig. 5 is a reference image of the first robot device in the embodiment. Fig. 6 is a second flowchart of the first control of the first robot device in the embodiment. Fig. 7 is an image when the position of the second workpiece is shifted with respect to the first workpiece. Fig. 8 is a different picture when the position of the second workpiece is shifted with respect to the first workpiece. Fig. 9 is an enlarged oblique view of a first workpiece, a second workpiece, and a hand in a second robot device of the embodiment. Fig. 10 is an enlarged oblique view of the first workpiece, the second workpiece, and the hand in the second robot device when the position of the second workpiece is shifted with respect to the first workpiece. Fig. 11 is an image taken by a first camera when the position of the second workpiece was shifted with respect to the first workpiece. Fig. 12 is an image taken by a second camera when the position of the second workpiece with respect to the first workpiece was appropriate. Fig. 13 is an image taken by the second camera when the position of the second workpiece was shifted with respect to the first workpiece. Fig. 14 is a flowchart of a third controller in the second robot device. Fig. 15 is an enlarged oblique view of a first workpiece, a second workpiece, and a hand in a third robot device of the embodiment. Fig. 16 is an enlarged oblique view of a first workpiece, a second workpiece, and a hand in a fourth robot device of the embodiment. Fig. 17 is a schematic view of a fifth robot device in the embodiment. Fig. 18 is an enlarged oblique view of a container, a workpiece, and a hand at a proper position of the workpiece with respect to a container. Fig. 19 is an enlarged oblique view of the container, the workpiece and the hand when the position of the workpiece is shifted with respect to the container. Fig. 20 is an image when the position of the workpiece is shifted with respect to the container.

[0017] With reference to Fig. 1 to Fig. 20, a control device of a robot device in one embodiment will be explained. In the present embodiment, a robot device that performs the work of assembling a product and a robot device that arranges workpieces inside a container will be used as examples for explanation.

[0018] Fig. Fig. 1 is a schematic view of a first robot device in the present embodiment. The first robot device 5 includes a hand 2 as a working tool (end effector) and a robot 1 that moves the hand 2. The first robot device 5 performs an operation of attaching a first workpiece 81 as a first member to a second workpiece 91 as a second member.

[0019] The robot 1 is an articulated robot having multiple joint sections. The robot 1 includes a base section 14 and a rotary base 13 supported on the base section 14. The base section 14 is fixed to a fixture surface. The rotary base 13 is configured to rotate relative to the base section 14. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is rotatably supported on the rotary base 13 via a joint section. The upper arm 11 is rotatably supported on the lower arm 12 via a joint section. Furthermore, the upper arm 11 rotates about a rotation axis parallel to the extension direction of the upper arm 11.

[0020] The robot 1 includes a wrist 15 coupled to the end of the upper arm 11. The wrist 15 is rotatably supported on the upper arm 11 via a joint portion. The wrist 15 includes a rotatably configured flange 16. The hand 2 is fixed to the flange 16. The robot 1 of the present embodiment has six drive axes, but is not limited to such a configuration. Any robot capable of moving a work tool can be used.

[0021] The hand 2 is a working tool for gripping and releasing the workpiece 91. The hand 2 has a plurality of claw portions 3. The hand 2 is configured such that the claw portions 3 open and close. The claw portions 3 grip the workpiece 91 by clamping the workpiece 91. The hand 2 of the first robot device 5 has the claw portions 3, but is not limited to such a configuration. Any structure configured to grip a workpiece can be adopted for the hand 2. For example, a hand that grips a workpiece by suction or by magnetic force can also be adopted.

[0022] The robot device 5 in the present embodiment includes a conveyor 75 as a transport machine that transports the first workpiece 81 with respect to the robot 1. The transport machine is arranged near the robot. The conveyor 75 is configured to transport the workpiece 81 to a predetermined position. The conveyor 75 is configured to transport the workpiece 81 at a predetermined transport speed.

[0023] In the first robot device 5 of the present embodiment, the robot 1 attaches the workpiece 91 to the workpiece 81 while the conveyor 75 continues to transport the workpiece 81. That is, the workpiece 81 is moved by the conveyor 75 during the period in which the work of attaching the workpiece 91 is performed. The robot 1 attaches the workpiece 91 to the workpiece 81 while changing its position and posture so as to follow the workpiece 81.

[0024] Fig. 2 shows an enlarged oblique view of the first workpiece, the second workpiece, and the hand in the first robot device. Referring to Fig. 1 and Fig. 2, the second workpiece 91 has a holding portion 94 protruding from the surface of its main body. The hand 2 grips the second workpiece 91 by gripping the holding portion 94 with the claw portions 3.

[0025] The first workpiece 81 has projections 82, 83 projecting from the surface of its main body. The projection 82 and the projection 83 are arranged spaced apart from each other. Furthermore, the workpiece 81 is supported on the conveyor 75 such that the projection 82 and the projection 83 are adjacent to each other in the vertical direction. Openings 82a, 83a are formed on the upper surfaces of the projections 82, 83. The second workpiece 91 has projections 92, 93 projecting from the surface of its main body. Pins 92a, 93a are fixed to the projections 92, 93. The pin 92a and the pin 93a are arranged to be adjacent to each other in the vertical direction. In the first control performed by the first robot device 5, an operation is performed in which the pin 92a is inserted into the opening 82a and the pin 93a is inserted into the opening 83a.

[0026] In the first control of the present embodiment, control is performed to adjust the position of the pins 92a, 93a with respect to the openings 82a, 83a. Before the first control is performed, the posture of the workpiece 91 with respect to the workpiece 81 has been adjusted. That is, the posture of the robot 1 has been adjusted so that the pin 92a and the pin 93a of the workpiece 91 are arranged on a straight line in the vertical direction. Therefore, by executing control to arrange the pin 92a directly above the opening 82a, it is possible to adjust the positions of the pins 92a, 93a with respect to the openings 82a. After the first control of the present embodiment is executed, the pins 92a, 93a are inserted into the openings 82a, 83a by moving the workpiece 91 downward in the vertical direction as shown by the arrow 103, thereby attaching the workpiece 91 to the workpiece 81.

[0027] The robot device 5 includes a camera 25 as a vision sensor that captures an image of the first workpiece 81 and the second workpiece 91. The camera 25 in the present embodiment is a 2D camera. The camera 25 is held on the hand 2 via a holding element 17. The camera 25 in the robot device 5 changes its position and attitude along with the hand 2.

[0028] The camera 25 captures an image when the second workpiece 91 has approached the first workpiece 81. The camera 25 is arranged to capture the surroundings of the area where the second workpiece 91 is assembled with the first workpiece 81. Furthermore, the camera 25 is arranged to capture a first feature portion for detecting the position of the first workpiece 81 and a second feature portion for detecting the position of the second workpiece 91. In the robot device 5, the camera 25 is arranged to capture the workpieces 81, 91 from above the workpiece 81 and the workpiece 91.

[0029] A base coordinate system 5 is defined for the robot device 5. In the example shown in Fig. 1, the origin point of the base coordinate system 51 is located on the base section 14 of the robot 1. The base coordinate system 51 is also called a global coordinate system. The base coordinate system 51 is a coordinate system in which the position of the origin point is fixed and the orientation of the coordinate axes is also fixed. The position and attitude of the base coordinate system 51 do not change even if the position and attitude of the robot 1 change. The base coordinate system 51 has an X-axis, a Y-axis, and a Z-axis as coordinate axes, which are orthogonal to each other. In addition, a W-axis is defined as the coordinate axis around the X-axis. A P-axis is defined as the coordinate axis around the Y-axis. An R-axis is defined as the coordinate axis around the Z-axis.

[0030] The position and attitude of the robot 1 can be expressed by the base coordinate system 51. For example, the position of the robot 1 can be expressed by the position of the tip end point of the tool located at the tip end of the hand 2. Furthermore, a tool coordinate system that moves along with the hand 2 can be set at the tip end point of the tool. The attitude of the robot 1 can be expressed by the orientation of the tool coordinate system with respect to the base coordinate system 51.

[0031] Fig. 3 shows a block diagram of the robot device in the present embodiment. Referring to Fig. 1 to Fig. 3, the robot 1 includes a robot drive device that changes the position and attitude of the robot 1. The robot drive device includes robot drive motors 22 that drive the structural elements such as the arms and wrist, etc. The orientation of the respective structural elements changes by driving the robot drive motors 22.

[0032] The hand 2 includes a hand drive device that drives the hand 2. The hand drive device includes a hand drive motor 21 that drives the claw portions 3 of the hand 2. The claw portions 3 of the hand 2 open and close by driving the hand drive motor 21. However, the claw portions 3 can also be designed to be driven by compressed air. In this case, the hand drive device can include devices that drive the claw portions by compressed air, such as an air pump, a cylinder, or the like.

[0033] A control device 29 of the robot apparatus includes a robot control device 4 that controls the robot 1 and the hand 2. The robot control device 4 includes a computer having a CPU (central processing device) as a processor. The computer has RAM (random access memory), ROM (read only memory), etc., connected to the CPU via a bus. A pre-prepared operation program 41 for controlling the robot 1, the hand 2, and the conveyor 75 is input to the robot control device 4. The robot 1 and the hand 2 transport the workpiece 91 based on the operation program 41. The conveyor 75 transports the workpiece 81 based on the operation program 41.

[0034] The arithmetic processing device of the robot control device 4 includes a storage unit 42 that stores predetermined information. The storage unit 42 stores information related to the control of the robot 1, the hand 2, and the conveyor 75. The storage unit 42 may be constituted by a storage medium capable of storing information, such as a volatile memory, a non-volatile memory, a hard disk, or the like. The operation program 41 is stored in the storage unit 42. The robot control device 4 includes a display 46 that displays any information related to the robot device 5. The display 46 includes, for example, a liquid crystal panel.

[0035] The arithmetic processing device includes an operation control unit 43 that outputs operation commands for the robot 1 and the hand 2. The operation control unit 43 corresponds to the processor, which operates according to the operation program 41. The operation control unit 43 is configured to read the information stored in the storage unit 42. The processor functions as the operation control unit 43 by reading the operation program 41 and executing the control specified in the operation program 41.

[0036] The operation control unit 43 sends operation commands for driving the robot 1 to the robot drive unit 45 based on the operation program 41. The robot drive unit 45 includes electrical circuits that drive the robot drive motors 22. The robot drive unit 45 supplies power to the robot drive motors 22 based on the operation commands. Furthermore, the operation control unit 43 sends operation commands for driving the hand 2 to a hand drive unit 44 based on the operation program 41. The hand drive unit 44 includes electrical circuits that drive the hand drive motor 21. The hand drive unit 44 supplies power to the hand drive motor 21 based on the operation commands. Furthermore, the camera 25 is connected to the robot control device 4 of the present embodiment. The operation control unit 43 sends an image capture command to the camera 25 based on the operation program 41.

[0037] The arithmetic processing device of the robot control device 4 includes an image processing unit 31 that processes the image captured by the camera 25. The image processing unit 31 has a feature amount detection unit 32 that detects, for the respective workpieces 81, 91, a feature amount of the feature portion constituting a predetermined characteristic region. The image processing unit 31 includes a computing unit 33 that calculates the difference between the feature amount of the first workpiece 81 and the feature amount of the second workpiece 91 as a relative amount. The image processing unit 31 has a command generation unit 34 that generates a movement command for driving the robot 1 based on the relative amount calculated by the computing unit 33.

[0038] The image processing unit 31 corresponds to the processor that operates according to the operation program 41. Specifically, the respective units of the feature amount detection unit 32, the arithmetic unit 33, and the operation command generation unit 33 correspond to the processor that operates according to the operation program 41. The processor reads the operation program 41 and acts as the respective unit by executing the controls specified in the operation program 41.

[0039] The robot device 5 includes a state detector that detects the operating state of the robot device 5. The state detector of the present embodiment includes position detectors 23 that detect the position and attitude of the robot 1. The position detectors 23 are mounted on the robot drive motors 22 corresponding to the drive axes of the constituent elements such as arms, etc. For example, the position detectors 23 detect the rotation angle when driving the robot drive motors 22. Based on the output of the position detectors 23, the position and attitude of the robot 1 are detected.

[0040] The control device 29 of the robot device 5 includes a conveyor control device 76 that controls the operation of the conveyor 75. The conveyor control device 76 includes a computing device (a computer) including a CPU and RAM, etc. The conveyor control device 76 is configured to mutually communicate with the robot control device 4. The operation control unit 43 outputs an operation command for driving the conveyor 75 to the conveyor control device 76 based on the operation program 41. The conveyor control device 76 receives the operation command from the robot control device 4 and drives the conveyor 75.

[0041] The control device 29 of the robot device 5 of the present embodiment includes the robot control device 4 that controls the robot 1 and the hand 2, and the conveyor control device 76 that controls the conveyor 75, but this is not limited to this configuration. The robot device 5 may also be configured such that the robot 1, the hand 2, and the conveyor 75 are controlled by a single control device.

[0042] Furthermore, in the control device 29 of the robot device 5 of the present embodiment, the robot control device 4 includes the image processing unit 31 having the image processing function, but this is not limited to this configuration. It is also possible for the control device of the robot device to include an image processing device (computer) having the image processing unit 31. The image processing device may be constituted by an arithmetic processing device having a CPU as a processor. The processor of the image processing device functions as a feature amount detection unit, an arithmetic unit, and an operation command generation unit that operate based on the operation program. The image processing device is configured to mutually communicate with the robot control device.

[0043] Fig. Fig. 4 shows a first flowchart of a first control in the present embodiment. Referring to Fig. 1, Fig. 2 and Fig. 4, in the first control, the workpiece 91 is positioned relative to the workpiece 81 transported by the conveyor 75 using the image captured by the one camera 25. As mentioned above, before the first control, the position of the workpiece 91 relative to the workpiece 81 was previously adjusted. This means that the position of the robot 1 is adjusted when the image is captured by the camera 25. In this control, control is performed to adjust the position of the pin 92a relative to the opening 82a.

[0044] In the present embodiment, before the robot device 5 performs the actual operation, a relative position amount in the reference image is calculated. Then, the robot device 5 performs the operation using the pre-calculated relative position amount. The first flowchart shows the control for calculating the relative position amount in the reference image.

[0045] In step 111, the reference image for performing the first control is generated. Fig. 5 shows the reference image for performing the first control of the present embodiment. The reference image 61 corresponds to an image taken by the camera 25 when the second workpiece 91 was positioned at the target position with respect to the first workpiece 81. In the present embodiment, the reference image 61 is an image taken by the camera 25 when the pins 92a, 93a of the workpiece 91 were positioned directly above the openings 82a, 83a of the workpiece 81. The reference image 61 can be prepared in advance by the operator and stored in the storage unit 42.

[0046] In the present embodiment, the first feature portion for detecting the position of the first workpiece 81 is set in advance. In the first control, the upper surface of the protrusion 82 is set as the feature portion. Furthermore, the second feature portion for detecting the position of the second workpiece 91 is set in advance. In the first control, the upper surface of the protrusion 92 is set as the feature portion. A feature portion is a region whose shape can be detected upon image analysis. A part of the workpiece, a pattern formed on the surface of the workpiece, a line or image displayed on the surface of the workpiece, or the like can be employed for the feature portion. Furthermore, the feature portion is preferably set near the region where the second workpiece 91 comes into contact with the first workpiece 81.

[0047] In the present embodiment, on the first workpiece 81, an adjustment point P1 is set on the upper surface of the protrusion 82, which constitutes the first feature portion. The adjustment point P1 is set at a corner of the protrusion 82. The position of the adjustment point P1 corresponds to the position of the workpiece 81. On the second workpiece 91, a second adjustment point P2 is arranged on the upper surface of the protrusion 92, which constitutes the second feature portion. The second adjustment point P2 is set at a corner of the protrusion 92. The adjustment point P2 corresponds to the position of the workpiece 91. The adjustment points P1, P2 are set in a range included in the image when the camera 25 captures the image.

[0048] With reference to Fig. 3, Fig. 4 and Fig. 5, the feature amount detection unit 32 of the image processing unit 31 detects the first feature portion and the second feature portion of the reference image 61 in step 112. Regarding the method for detecting the feature portions, a base image that is a basis for the respective workpieces 81, 91 may be prepared in advance. By a method such as pattern matching using the base image and the image captured by the camera 25, the feature portions in the image captured by the camera 25 can be detected. In this example, the upper surface of the protrusion 82 and the upper surface of the protrusion 92 can be detected.

[0049] In step 113, the feature amount detection unit 32 detects the first feature amount with respect to the position of the first feature portion and the second feature amount with respect to the position of the second feature portion. In the present embodiment, a screen coordinate system 52 is set for the image captured by the camera 25. The screen coordinate system 52 is a coordinate system in which any point in the image is set as the origin point. The screen coordinate system 52 has a u-axis and a v-axis that are orthogonal to each other. The screen coordinate system 52 corresponds to the vision sensor coordinate system of the camera 25.

[0050] The feature amount related to the position in the present embodiment is the u-axis coordinate value and the v-axis coordinate value of the screen coordinate system 52 in the image. The feature amount detection unit 32 can detect the positions of the setting points P1, P2 set in the feature portions based on the feature portions detected in the reference image 61. As the first feature amount, the feature amount detection unit 32 detects the coordinate values ​​(u1b, v1b) of the setting point P1 in the screen coordinate system 52. Furthermore, the feature amount detection unit 32 detects the coordinate values ​​(u2b, v2b) of the setting point P2 in the screen coordinate system 52 as the second feature amount.

[0051] Then, in step 114, the arithmetic unit 33 of the image processing unit 31 calculates the relative amount with respect to the first feature amount and the second feature amount in the reference image. The arithmetic unit 33 calculates a relative position amount as the relative amount for performing position control of the robot 1. The relative position amount is the difference between the first feature amount and the second feature amount. For example, the arithmetic unit 33 calculates the difference (u1b - u2b, v1b - v2b) between the coordinate values ​​of the first feature amount and the coordinate values ​​of the second feature amount as the relative amount. The relative position amount in the reference image 61 calculated by the arithmetic unit 33 is stored as the reference relative position amount in the storage unit 42.

[0052] In this way, the image processing unit 31 can calculate the relative position amount in the reference image 61. In the present embodiment, the relative position amount in the reference image 61 was calculated in advance and stored in the storage unit 42, but this is not limited to this implementation. The relative position amount in the reference image 61 may also be calculated with each execution of the first control.

[0053] For the reference image 61, it is not necessary to use an image actually captured by the camera 25. For example, three-dimensional data of the respective workpieces 81, 91 can be created by a CAD device (computer-aided design device) or the like. Three-dimensional data can be created when the workpiece 91 is arranged at the target position with respect to the workpiece 81. The reference image 61 can be generated by projecting this three-dimensional data onto a plane extending along a direction corresponding to the orientation of the camera.

[0054] Next, the robot control device 4 executes control to bring the pin 92a of the second workpiece 91 closer to the opening 82a of the first workpiece 81 so that the first feature portion and the second feature portion are positioned within an image pickup range 25a of the camera 25. This control can be performed by any control method. For example, the robot control device 4 detects the position of the workpiece 81 on the conveyor 75 through a specific sensor. The robot control device 4 detects the position of the workpiece 81 based on the moving speed of the conveyor 75. The robot control device 4 can control the position and attitude of the robot 1 so that the workpiece 91 approaches the workpiece 81.

[0055] Fig. Fig. 6 shows a second flowchart of the first control in the present embodiment. Referring to Fig. 3 and Fig. 6, after the robot control device 4 executes the control for approaching the workpiece 91 to the workpiece 81 in step 115, the operation control unit 43 photographs the workpieces 81, 91 through the camera 25.

[0056] Fig. 7 shows an image captured by the camera for adjusting the position of the second workpiece relative to the first workpiece. Included in image 62 are an image of the upper surface of the protrusion 82, which represents the first feature portion, and the upper surface of the protrusion 92, which represents the second feature portion. In image 62, the second workpiece 91 is displaced relative to the first workpiece 81 in the positive direction of the u-axis of the screen coordinate system 52, as shown by arrow 101.

[0057] Fig. Figure 8 shows another image taken by the camera for adjusting the position of the second workpiece relative to the first workpiece. In image 63, the second workpiece 91 is displaced relative to the first workpiece 81 as shown by arrow 102 in the negative direction of the u-axis of the screen coordinate system 52. Referring to Fig. 7 and Fig. 8, a control is carried out in the first control to correct such a deviation of the second workpiece 91. In the following explanation, these Fig. 7 and Fig. 8 the Fig. 7 is used as an example.

[0058] With reference to Fig. 3, Fig. 6 and Fig. 7, in step 116 to step 118, the same control as the control with respect to the reference image 61 is executed. The image processing unit 31 detects the first feature amount and the second feature amount in the image 62 and calculates the relative position amount using the first feature amount and the second feature amount.

[0059] In step 116, the feature amount detection unit 32 detects the first feature portion and the second feature portion in the image 62 captured by the camera 25. Here, the upper surface of the projection 82 of the workpiece 81 is detected as the first feature portion, and the upper surface of the projection 92 of the workpiece 91 is detected as the second feature portion.

[0060] In step 117, the feature amount detection unit 32 detects the first feature amount and the second feature amount in the image 62 captured by the camera 25. As the first feature amount related to the position of the first feature portion, the feature amount detection unit 32 detects the coordinate values ​​(u1m, v1m) of the setting point P1 in the screen coordinate system 52. Furthermore, as the second feature amount related to the position of the second feature portion, the feature amount detection unit 32 detects the coordinate values ​​(u2m, v2m) of the setting point P2 in the screen coordinate system 52.

[0061] In step 118, the arithmetic unit 33 calculates the difference between the first feature amount and the second feature amount as the relative position amount. The relative position amount in the image 62 captured by the camera 25 is the difference (u1m - u2m, v1m - v2m) between the coordinate values ​​of the first feature amount and the coordinate values ​​of the second feature amount.

[0062] When the hand 2 has grasped the second workpiece 91 at the predetermined position, the second feature amount with respect to the second workpiece 91 becomes constant. Therefore, the second feature amount may also be measured in advance and stored in the storage unit 42. That is, the coordinate values ​​of the set point P2 may be stored in advance in the storage unit 42. However, when the hand 2 grasps the second workpiece 91, it may deviate from the desired position. Therefore, in the present embodiment, the second feature amount is also detected based on the actually captured image 62 by the method of pattern matching with a base image.

[0063] Next, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 to position the second workpiece 91 at the target position with respect to the first workpiece 81 based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. The command generation unit 34 of the present embodiment generates the movement command to move the robot 1 so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.

[0064] In step 119, the command generation unit 34 calculates the difference in relative position amounts, which is the difference between the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61. In the present embodiment, the command generation unit 34 calculates the difference in relative position amounts by subtracting the relative position amount in the reference image 61 from the relative position amount in the image 62 captured by the camera 25. The difference in relative position amounts can be expressed as each value with respect to the u-axis and the v-axis by [(u1m - u2m) - (u1b - u2b), (v1m - v2m) - (v1b - v2b)]. In this way, in the present embodiment, the difference in relative position amounts with respect to the u-axis and the difference in relative position amounts with respect to the v-axis are calculated.

[0065] Next, in step 120, the command generation unit 34 determines whether or not the difference in the relative position amounts is within a predetermined determination range. The determination range has been predetermined and stored in the storage unit 42. For example, a determination range for the value with respect to the u-axis and a determination range for the value with respect to the v-axis may be predetermined. The difference in the relative position amounts becomes closer to zero the closer the second workpiece 91 approaches the target position with respect to the first workpiece 81. If the value with respect to the u-axis is within the determination range and the value with respect to the v-axis is within the determination range, it can be determined that the difference in the relative position amounts is within the determination range.That is, the command generation unit 34 can determine that the positioning of the workpiece 91 with respect to the workpiece 81 is completed.

[0066] If at least one of the u-axis value and the v-axis value deviates from the determination range, the command generation unit 34 may determine that the difference in the relative position amounts deviates from the determination range. That is, the command generation unit 34 may determine that the workpiece 91 has not reached the desired position with respect to the workpiece 81.

[0067] If the difference in the relative position amounts in step 120 is within the determination range, control is terminated. If the difference in the relative position amounts in step 120 deviates from the determination range, control proceeds to step 121.

[0068] In step 121, the command generation unit 34 sets the driving method of the robot 1 based on the difference in the relative position amounts. The command generation unit 34 sets the movement direction and the movement amount of the position of the robot 1 in the base coordinate system 51. In the present embodiment, the movement direction of the position of the robot 1 with respect to the difference in the relative position amounts is set in advance. The movement direction of the position of the robot 1 with respect to a positive value or a negative value of the u-axis of the screen coordinate system 52 is set in advance in the base coordinate system 51. For example, in the case of a positive value of the difference in the relative position amounts with respect to the u-axis, a movement direction (1, 1, 0) is set in advance using the coordinate values ​​of the X-axis, the Y-axis, and the Z-axis of the base coordinate system 51.In the case of a positive value of the difference of the relative position amounts with respect to the v-axis, a movement direction (0, 0, 1) is set in advance using the coordinate values ​​of the X-axis, the Y-axis and the Z-axis of the basic coordinate system 51.

[0069] Furthermore, the calculation method for the movement amount of the position of the robot 1 with respect to the difference in the relative position amounts is predetermined. For example, for the movement amount of the position of the robot 1 in a direction corresponding to the u-axis, a value for which the value ((u1m - u2m) - (u1b - u2b)) with respect to the u-axis is multiplied by a predetermined coefficient may be applied. For the movement amount of the position of the robot 1 in a direction corresponding to the v-axis, a value for which the value ((v1m - v2m) - (v1b - v2b)) with respect to the v-axis is multiplied by a predetermined coefficient may be applied. In this way, the movement amount of the position of the robot 1 in the directions corresponding to each axes of the screen coordinate system 52 can be calculated.

[0070] In the present embodiment, based on the difference in the relative position amounts with respect to the u-axis, a movement amount in the X-axis direction, a movement amount in the Y-axis direction, and a movement amount in the Z-axis direction are calculated in the base coordinate system 51. Furthermore, based on the difference in the relative position amounts with respect to the v-axis, a movement amount in the X-axis direction, a movement amount in the Y-axis direction, and a movement amount in the Z-axis direction are calculated in the base coordinate system 51. Therefore, in the base coordinate system 51, two movement amounts (a movement amount with respect to the u-axis and a movement amount with respect to the v-axis) may be calculated for one axis. In this case, the position of the robot 1 in the direction for which two movement amounts have been calculated does not need to be moved.Alternatively, it is possible to calculate a final range of motion by multiplying the respective ranges of motion by a coefficient. Or, any of the ranges of motion can be used.

[0071] Next, in step 122, the robot 1 is driven based on the movement direction and the movement amount of the position of the robot 1. The command generation unit 34 generates a movement command to drive the robot 1 based on the movement direction and the movement amount of the position of the robot 1. The command generation unit 34 outputs the movement command to the operation control unit 43. The operation control unit 43 changes the position of the robot 1 based on the movement command.

[0072] Then, control proceeds to step 115. In the first control, control is performed that repeats steps 115 to 122 until the difference in the relative position amounts enters the determination range. In the first control, if the positioning of the workpiece has not been completed by the first control, the position of the workpiece can be gradually approximated to the desired position by repeating steps 115 to 122.

[0073] In the control of the present embodiment, it is not necessary to calibrate the vision sensor coordinate system with respect to the base coordinate system. Furthermore, it is not necessary to determine a Jacobian matrix in advance to perform the workpiece positioning. Therefore, the workpiece positioning can be performed by a simple method.

[0074] In the first control of the present embodiment, the positioning of the second workpiece 91 is performed while the conveyor 75 transports the first workpiece 81. The control for detecting the first feature amount and the second feature amount by the feature amount detection unit 32, the control for detecting the relative position amount by the arithmetic unit 33, and the control for calculating the movement command by the command generation unit 34 are repeated. Through these controls, control for tracking the position of the robot 1 holding the first workpiece 91 with respect to the position of the workpiece 81 moved by the conveyor 75 can be performed.

[0075] The camera 25 in the present embodiment is held on the hand 2 via a holding member 17. The hand 2 holds the second workpiece 91. Therefore, the relative position and attitude of the workpiece 81 with respect to the camera 25 are constant during the positioning of the second workpiece 91 with respect to the first workpiece 81. Referring to Fig. 5, Fig. 7 and Fig. 8, the position of the workpiece 91 and the projection 92 in the image 62, 63 captured by the camera 25 remains constant even when the relative position of the second workpiece 91 with respect to the first workpiece 81 changes. As a result, the position of the adjustment point P2 of the second feature portion remains constant in the image captured by the camera 25.

[0076] With reference to Fig. 6, with the control described above, with each image capture of the first workpiece 81 and the second workpiece 91 by the camera 25, detection of the second feature portion in the second workpiece 91 and detection of the second feature amount are performed. However, since the coordinate values ​​of the set point P2, which are the second feature amount, are constant, the storage unit 42 may store the second feature amount obtained from the image captured by the camera 25 the first time. In step 115 to step 117, the feature amount detection unit 32 may acquire the second feature amount for images captured by the camera 25 the second time and later from the memory 42. It is sufficient for the feature amount detection unit 32 to perform a first detection of the feature portion and a first detection of the feature amount.By performing such control, the control for calculating the relative position amount from the images captured by the camera 25 at the second time and later can be simplified. This shortens the time for processing the images captured by the camera 25 at the second time and later.

[0077] Incidentally, in the present embodiment, the movement direction and movement speed of the first workpiece 81 by the conveyor 75 are set in advance. The robot control device 4 can perform feedforward control that changes the position of the robot 1 in coordination with the movement of the workpiece 81 by the conveyor 75. In the present embodiment, the workpiece 81 moves at a constant speed.

[0078] The command generation unit 34 calculates the movement direction and the movement speed of the robot 1 at which the position of the robot 1 follows the position of the first workpiece 81 moved by the conveyor 75. For example, the command generation unit 34 calculates the movement direction such that the tip end point of the tool of the robot 1 moves in the movement direction of the workpiece 81. The command generation unit 34 can calculate the movement amount for which the tip end point of the tool of the robot 1 moves at the same movement speed as the workpiece 81 in the same direction as the movement direction of the workpiece 81.Then, the command generation unit 34 can perform the control of the movement direction and the movement amount calculated based on the above-described difference in the relative position amounts, in addition to the control of the movement direction and the movement amount of the position of the robot 1 based on the transportation by the conveyor 75.

[0079] By implementing such control, the change in the position and attitude of the robot 1 associated with the movement of the first workpiece 81 by the conveyor 75 can be performed through feedforward control. Since, in the control based on the difference in the relative position amounts, a deviation in the relative position of the second workpiece 91 with respect to the first workpiece 81 can be corrected, the positioning of the second workpiece 91 with respect to the first workpiece 81 can be performed within a short time.

[0080] Fig. 9 shows an enlarged oblique view of a hand, a first workpiece, and a second workpiece in a second robot device of the present embodiment. In the first robot device 5, the position of the robot 1 is adjusted by a single camera 25, but this is not limited to this configuration. The robot device may adjust the position of the robot 1 using two or more cameras.

[0081] In the second robot device 8, a holding element 18 is fixed to the hand 2. The holding element 18 has a section 18a running on the upper side and a section 18b running on the lower side. A camera 25 as a first vision sensor is fixed to the section 18a running on the upper side, as in the first robot device 5. The camera 25 records an image in a viewing area 25a. A camera 26 as a second vision sensor is fixed to the section 18b running on the lower side. The camera 26 records the first workpiece 81 and the second workpiece 91. In particular, the second camera 26 records an image when the second workpiece 91 has approached the first workpiece 81. The camera 26 is arranged at a position where it can record the projection 83 of the first workpiece 81 and the projection 93 of the second workpiece 91.The underside portion 18b supports the camera 26 so that it can capture the projections 83, 93 from beneath the workpieces 81, 91. The camera 26 captures an image in a viewing area 26a. The camera 26 of the present embodiment is a 2D camera.

[0082] The two cameras 25, 26 are arranged so that their optical axes extend in mutually different directions. In the present embodiment, the camera 26 is arranged so that the optical axis of the camera 26 extends in a direction substantially orthogonal to that of the optical axis of the camera 25. Second control is performed at the second robot device 8 based on the images captured by the two cameras 25, 26. In the second control, the position of the workpiece 91 with respect to the workpiece 81 is adjusted using the images captured by the cameras 25, 26.

[0083] In the second control, a movement command is generated by the first controller based on the image from the camera 25. Furthermore, a movement command is generated based on the image from the camera 26 by the same method as in the first control. In the present embodiment, a third feature section for detecting the position of the first workpiece 81 and a fourth feature section for detecting the position of the second workpiece 91 in the image captured by the camera 26 were set in advance.

[0084] The third feature section is a region different from the first feature section. For example, the side surface of the protrusion 83 of the first workpiece 81 can be defined as the third feature section. A third setting point P3 for determining the position of the first workpiece 81 can be defined in the third feature section. The fourth feature section is a region different from the second feature section. For example, the side surface of the protrusion 93 of the second workpiece can be defined as the fourth feature section. A fourth setting point P4 for determining the position of the second workpiece 91 can be defined in the fourth feature section.

[0085] The feature amount detection unit 32 detects, in the image captured by the camera 26, a third feature amount related to the third feature portion and a fourth feature amount related to the fourth feature portion. In the image captured by the camera 26, the coordinate values ​​of the setting point P3 in the screen coordinate system 52 represent the third feature amount. Furthermore, the coordinate values ​​of the setting point P4 in the screen coordinate system 52 represent the fourth feature amount. The arithmetic unit 33 calculates the difference between the third feature amount and the fourth feature amount as the relative position amount.

[0086] Furthermore, a reference image is generated in advance with respect to the image of the camera 26 when the second workpiece 91 is positioned at the target position with respect to the first workpiece 81. Furthermore, a relative position amount, which is the difference between the third feature amount and the fourth feature amount in the reference image, is determined. The relative position amount in the reference image can be calculated in advance.

[0087] The command generation unit 34 calculates the difference in the relative position amounts based on the relative position amount in the image captured by the camera 26 and the relative position amount in the reference image including an image of the third feature portion and the fourth feature portion. Then, based on the difference in the relative position amounts, the command generation unit 34 generates a movement command to move the robot 1 so that the second workpiece 91 is arranged at the target position with respect to the first workpiece 81. The command generation unit 34 generates the movement command to move the robot 1 so that the relative position amount in the image captured by the camera 26 approaches the relative position amount in the reference image.

[0088] The command generation unit 34 may generate the final movement command output to the operation control unit 43 based on the movement command generated from the image of the camera 25 and the movement command generated from the image of the camera 26. For example, after driving the robot 1 using one of the movement command based on the image of the camera 25 and the movement command based on the image of the camera 26, the command generation unit 34 may drive the robot 1 by the other of these movement commands.

[0089] Or, the movement command based on the image from camera 25 and the movement command based on the image from camera 26 may also be combined. For example, when the movement direction of the position of robot 1 corresponding to the u-axis of the image from camera 25 and the movement direction of the position of robot 1 corresponding to the u-axis of the image from camera 26 coincide, an average value of the movement amounts of robot 1 can be calculated. Such control for regulating the position of robot 1 may be repeatedly executed until the difference in the relative position amounts based on the image from camera 25 falls within the determination range and the difference in the relative position amounts based on the image from camera 26 falls within the determination range.

[0090] In the second robot device 8, position control is performed while the respective cameras 25, 26 capture different feature portions. Therefore, the position of the workpiece 91 can be adjusted more precisely than in a control system in which position adjustment is performed using a single camera. Furthermore, the position of the workpiece can be adjusted based on images from multiple cameras without performing stereometry or the like. Furthermore, the position of the workpiece can be adjusted without using 3D cameras by using multiple 2D cameras.

[0091] The position of the adjustment point P2 in the image captured by the first camera 25 and the position of the adjustment point P4 in the image captured by the second camera 26 do not change. Therefore, as in the first control, the second feature amount and the fourth feature amount detected at the beginning can be stored in the storage unit 42. Then, in a second and subsequent control, the second feature amount and the fourth feature amount stored in the storage unit 42 can be acquired, and the relative position amounts in the respective images can be calculated.

[0092] The second robot device 8 is equipped with a plurality of cameras 25, 26. The control device of such a robot device 8 can, in addition to regulating the position of the workpiece 91, also correct the position of the workpiece 91. The correction of the position of the workpiece 91 is explained below.

[0093] Fig. 10 shows another enlarged oblique view of the hand, the first workpiece, and the second workpiece in the second robot device of the present embodiment. In the first control and the second control, the relative position of the second workpiece 91 with respect to the first workpiece 81 was adjusted before the control for adjusting the position of the second workpiece 91 was executed. However, there may be a situation where the position of the second workpiece 91 with respect to the first workpiece 81 is shifted.

[0094] When the workpiece 91 has reached the target position, the direction in which the pin 92a and the pin 93a are arranged is parallel to the direction in which the opening 82a and the opening 83a are arranged. In the example shown in Fig. 10, the position of the workpiece 91 is slightly rotated from the target position, as shown by arrow 106. The workpiece 91 is slightly rotated about the X-axis of the base coordinate system 51 in the direction from the Y-axis to the Z-axis. The workpiece 91 is slightly rotated from the target position about the rotation axis of the flange 16. As a result, the position of the projection 92 of the workpiece 91 with respect to the projection 82 of the workpiece 81 is shifted toward the negative side of the Y-axis, as shown by arrow 107. Furthermore, the position of the projection 93 of the workpiece 91 with respect to the projection 83 of the workpiece 81 is shifted toward the positive side of the Y-axis, as shown by arrow 108.

[0095] The plurality of cameras 25, 26 in the present embodiment are configured to capture spaced-apart portions of the second workpiece 91. The first camera 25 is arranged to capture the protrusion 92 as a portion located on one side in a predetermined direction of the second workpiece 91. The second camera 26 is arranged to capture the protrusion 93 as a portion located on the other side in the predetermined direction of the second workpiece 91. The predetermined direction mentioned here corresponds to the direction in which the pin 92a and the pin 93a are adjacent to each other.

[0096] Fig. 11 shows the image taken by the first camera when the position of the second workpiece was shifted with respect to the first workpiece. Referring to Fig. 10, Fig. 11 and Fig. 5, which is the reference image, the relative position of the workpiece 91 with respect to the workpiece 81 in the image 66 differs from the relative position of the workpiece 91 with respect to the workpiece 81 in the reference image 61. The projection 92 of the workpiece 91 is shifted with respect to the projection 82 of the workpiece 81 in the direction shown by the arrow 107. The extending direction of the pin 92a fixed to the projection 92 and the extending direction of the opening 82a formed in the projection 82 do not extend parallel, but are directed in different directions.

[0097] Fig. 12 shows an image captured by the second camera when the position of the second workpiece relative to the first workpiece was the target position. In image 67, the extending direction of the pin 93a fixed to the projection 93 and the extending direction of the opening 83a formed in the projection 83 extend parallel. Therefore, the pin 93a can be inserted into the opening 83a by positioning the workpiece 91 relative to the workpiece 81.

[0098] Fig. Figure 13 shows an image taken by the second camera when the position of the second workpiece was shifted relative to the first workpiece. In image 68, the workpiece 91 is slightly rotated relative to the workpiece 81 as shown by arrow 108. Referring to Fig. 10, Fig. 12 and Fig. 13, the relative position of the projection 93 with respect to the projection 83 in Figure 68 is shifted as shown by arrow 108. The extending direction of the pin 93a and the extending direction of the opening 83a do not extend in parallel. Therefore, the pin 93a cannot be inserted into the opening 83 even if the tip end of the pin 93a is positioned directly above the upper end of the opening 83a. In a third control of the present embodiment, control is performed to correct the relative position of the second workpiece 91 with respect to the first workpiece 81.

[0099] Fig. Fig. 14 shows a flowchart of the third control in the second robot device of the present embodiment. In the third control, in addition to the second control, control for correcting the position of the workpiece 91 is performed.

[0100] Steps 131 to 138 correspond to the second control described above. In step 131, the first camera 25 and the second camera 26 capture images. In step 132, the feature amount detection unit 32 detects the first feature portion and the second feature portion in the image captured by the camera 25. In step 133, the feature amount detection unit 32 detects the third feature portion and the fourth feature portion in the image captured by the camera 26.

[0101] In step 134, the feature amount detection unit 32 detects the first feature amount with respect to the position of the first feature portion, the second feature amount with respect to the position of the second feature portion, the third feature amount with respect to the position of the third feature portion, and the fourth feature amount with respect to the position of the fourth feature portion. Referring to Fig. 11, the feature amount detection unit 32 detects the coordinate values ​​of the setting point P1 in the screen coordinate system 52 as the first feature amount in the image 66 captured by the first camera 25. Furthermore, the feature amount detection unit 32 detects the coordinate values ​​of the setting point P2 in the screen coordinate system 52 as the second feature amount. Referring to Fig. 13, the feature amount detection unit 32 detects the coordinate values ​​of the set point P3 in the screen coordinate system 52 in the image 68 as the third feature amount. Furthermore, the feature amount detection unit 32 detects the coordinate values ​​of the set point P4 in the screen coordinate system 52 as the fourth feature amount.

[0102] With reference to Fig. 14, in step 135, the computing unit 33 of the image processing unit 31 calculates a relative position amount based on the first feature amount and the second feature amount. In step 136, the computing unit 33 calculates a relative position amount based on the third feature amount and the fourth feature amount. The computing unit 33 calculates the relative position amounts in the images 66, 68 captured by the respective cameras 25, 26.

[0103] Then, in step 137, the command generation unit 34 of the image processing unit 31 calculates the difference in relative position amounts for the images 66, 67 captured by the respective cameras 25, 26, which is the difference between the relative position amount in the image 66, 68 and the relative position amount in the reference image. In step 138, the command generation unit 34 generates a movement command for the robot 1 for the respective images 66, 68 based on the difference in relative position amounts. The difference in relative position amounts is calculated as a value with respect to the u-axis and the v-axis in the screen coordinate system 52. The command generation unit 34 generates the movement commands for the robot 1 based on the differences in relative position amounts.

[0104] The command generation unit 34 calculates a movement direction and a movement amount of the robot 1 in the base coordinate system 51 based on the image 66 captured by the camera 25. Furthermore, the command generation unit 34 calculates a movement direction and a movement amount of the robot 1 in the base coordinate system 51 based on the image 68 captured by the camera 26. That is, the command generation unit 34 calculates the movement amount along the direction of the coordinate axis of the base coordinate system 51 for each of the coordinate axes of the base coordinate system 51.

[0105] Then, in step 139, the command generation unit 34 determines whether or not the position of the second workpiece 91 with respect to the first workpiece 81 is within a predetermined determination range based on the movement command generated from the image of the first camera 25 and the movement command generated from the image of the second camera 26.

[0106] In the present embodiment, the command generation unit 34 obtains the moving direction of the position of the robot 1 on a predetermined coordinate axis of the base coordinate system 51. Referring to Fig. 10, in this example, the Y-axis was set in advance among the coordinate axes of the base coordinate system 51. The command generation unit 34 acquires the movement command for the Y-axis in the movement command generated from the image 66 of the first camera 25 and the movement command for the Y-axis in the movement command generated from the image 68 of the second camera 26.

[0107] When the Y-axis movement directions obtained from the images 66, 68 captured by the two cameras 25, 26 indicate the same direction, the command generation unit 34 determines that the position of the second workpiece 91 relative to the first workpiece 81 is within the predetermined determination range. However, when the Y-axis movement directions are different directions, the operation command unit 34 determines that the position of the second workpiece 91 relative to the first workpiece 81 deviates from the determination range.

[0108] In the example shown in Fig. 10 to Fig. As shown in Fig. 13, in the movement command based on the image captured by the first camera 25, a command is generated to move the robot 1 in the direction opposite to the arrow 107. And in the movement command based on the image captured by the second camera 26, a command is generated to move the robot 1 in the direction opposite to the arrow 108. Therefore, in step 139, the command generation unit 34 determines that the movement directions for the Y-axis are different based on the images captured by the two cameras. In this case, control proceeds to step 140.

[0109] In the present embodiment, it is determined that the robot 1 is within the determination range when the directions in which the position of the robot 1 is to move indicate the same direction for the predetermined coordinate axis of the base coordinate system, but this is not limited to this. It may also be determined that the robot 1 is within the determination range when the movement directions for the predetermined coordinate axis are different from each other, but the movement amount is minimal.

[0110] In step 140, the command generation unit 34 sets the correction method for the posture of the robot 1. The command generation unit 34 generates a movement command to rotate the workpiece 91 in the direction opposite to the inclination direction of the second workpiece 91 with respect to the first workpiece 81. Referring to Fig. 10, in the present embodiment, control is performed to rotate the workpiece 91 around the X-axis of the base coordinate system 51 in a direction from the Z-axis to the Y-axis. A position correction angle θ representing the rotation amount of the workpiece 91 can be set in advance.

[0111] In step 141, the command generation unit 34 outputs the movement command based on the method for correcting the posture of the robot 1 to the operation control unit 43. The operation control unit 43 corrects the posture of the robot 1 based on the movement command received from the command generation unit 34.

[0112] Then, in step 142, the image processing unit 31 corrects the movement direction of the position of the robot 1 in the base coordinate system 51 with respect to the coordinate axes of the screen coordinate system 52. The difference in the relative position amounts is calculated from the values ​​with respect to the y-axis and v-axis of the screen coordinate system 52. The movement direction of the position of the robot 1 in the base coordinate system 51 with respect to the u-axis and v-axis of the screen coordinate system 52 is set in advance. When the posture of the robot 1 is corrected, the movement direction in the base coordinate system with respect to the u-axis and v-axis of the images captured by the respective cameras 25, 26 also changes. That is, the movement direction of the position of the robot 1 expressed by the coordinate value of the x-axis, y-axis, and z-axis of the base coordinate system changes.

[0113] The image processing unit 31 corrects the movement direction of the position of the robot 1 corresponding to the coordinate axes of the screen coordinate system 52 based on the correction amount of the posture of the robot 1. For example, the movement direction of the position of the robot 1 can be corrected by applying a transformation matrix calculated based on the correction of the posture of the robot 1 to the coordinate values ​​of the X-axis, the Y-axis, and the Z-axis showing the movement direction of the position of the robot 1.

[0114] After step 142, control returns to step 131. Then, control is repeated from step 131 to step 139. If the movement directions for the predetermined coordinate axis of the base coordinate system 51 differ from each other in step 139, control of steps 140, 141, and 142 is executed.

[0115] In step 140, the method for correcting the posture of the robot 1 is determined. It may be that the direction in which the posture of the second workpiece 91 deviates with respect to the first workpiece 81 has reversed due to the posture correction of the robot 1 by the previous posture correction amount θ. In this case, the posture of the robot 1 is corrected in the opposite direction to the previous correction. In addition, a smaller correction amount than the previous correction amount is used. For example, control is executed in which the posture is corrected by a correction amount that is half the previous correction amount (-θ / 2). On the other hand, it is possible that the direction in which the posture of the second workpiece 91 deviates with respect to the first workpiece 81 is the same direction as the previous correction despite the posture of the robot 1 being corrected by the previous correction amount θ.In this case, the attitude is corrected in the same direction and by the same correction amount θ as before. After that, control from step 131 to step 139 can be repeated.

[0116] When the direction in which the posture deviates has been reversed due to the posture correction by the previous control, control may be performed in which the posture of the robot is corrected in the opposite direction and the correction amount is reduced. Through this control, the deviation of the posture of the second workpiece 91 with respect to the first workpiece 81 can be gradually corrected. The posture of the robot 1 can be corrected until the deviation direction of the second workpiece 91 with respect to the first workpiece 81 in the images captured by the first camera 25 and the second camera 26 becomes the same direction.

[0117] When the posture of the second workpiece 91 relative to the first workpiece 81 deviates from the determination range in step 139, any control may be performed. For example, the robot device may be stopped. Furthermore, in step 140, the command generation unit 34 may specify the method for correcting the posture of the robot 1 by any method. For example, a posture correction amount for the workpiece 81 may be calculated based on the movement amounts in the Y-axis direction calculated from the images captured by the two cameras 25, 26.

[0118] If, in step 139, the movement direction based on the image captured by the first camera 25 and the movement direction based on the image captured by the second camera 26 are the same, control proceeds to step 143. In steps 143 to 145, as in the second control, determination and correction of the position of the robot 1 are performed. In step 143, it is determined whether or not the difference in the relative position amounts based on the image captured by the first camera 25 and the difference in the relative position amounts based on the image captured by the second camera 26 are within the predetermined determination range. If, in step 143, at least one of the differences in the relative position amounts deviates from the determination range, control proceeds to step 144.

[0119] In step 144, the command generation unit 34 generates a final movement command for the robot based on the movement commands based on the respective images. Then, in step 145, the operation control unit 43 changes the position of the robot 1 by driving the robot 1 based on the final movement command for the robot. After that, control proceeds to step 131.

[0120] If, in step 143, the differences in the relative position amounts between both images are within the determination range, it can be determined that the position and attitude of the second workpiece 91 with respect to the first workpiece 81 are the target position and attitude. Then, this control can be terminated.

[0121] Since the structure, action, and results of the second robot device 8 are the same as those of the first robot device 5 except for the above, the explanation is not repeated here.

[0122] Fig. 15 shows an enlarged oblique view of a hand, a first workpiece, and a second workpiece in a third robot device of the present embodiment. In the third robot device 9, a holding member 19 is fixed to the hand. The holding member 19 has an upper portion 19a and a lower portion 19b. A camera 25 as a first vision sensor is fixed to the upper portion 19a, as in the second robot device 8. A camera 26 as a second vision sensor is fixed to the lower portion 19b.

[0123] The orientation of the camera 26 of the third robot device 9 differs from the orientation of the camera 26 in the second robot device 8. The underside portion 19a of the holding element 19 holds the camera 26 such that the field of view 26a of the camera 26 coincides with the field of view 25a of the camera 25. The camera 26 of the third robot device 9 is arranged to record the same area as the area recorded by the camera 25. The camera 26 is arranged to record the projection 82 of the first workpiece 81 and the projection 92 of the second workpiece 91.

[0124] In the third robot device 9, a fourth control is performed to position the workpiece 91 with respect to the workpiece 81. In the fourth control, as in the second control of the second robot device 8, a third feature portion and a fourth feature portion are set, and the position of the robot is regulated based on the images from the plurality of cameras 25, 26. In the present embodiment, the side surface of the protrusion 82 is set as the third feature portion. Furthermore, the side surface of the protrusion 92 is set as the fourth feature portion. In the third robot device 9, the third feature portion may be the same as or different from the first feature portion. Furthermore, the fourth feature portion may be the same as or different from the second feature portion.

[0125] Since the other structures, the action and the results of the third robot device 9 are the same as those of the second robot device 8, the explanation is not repeated here.

[0126] Two cameras are provided in the control device of the second robot device 8 and the control device of the third robot device 9, but this is not limited to this configuration. The control device of the robot device may also include three or more cameras. The control device can capture different feature portions through the respective cameras and perform position and attitude control of the workpiece based on the images captured by the respective cameras.

[0127] Fig. 16 shows an enlarged oblique view of a hand, a first workpiece, and a second workpiece in a fourth robot device of the present embodiment. The fourth robot device 10 includes a third camera 28 in addition to a first camera 25 and a second camera 26. The fourth robot device 10 has a holding member 20 held on the hand 2. The holding member 20 has a portion 20a extending on the upper side and holding the camera 25 and a portion 20b extending on the lower side and holding the camera 26. In addition, the holding member 25 has a portion 20c extending laterally and holding the camera 28. The camera 28 has an image pickup area 28a. The camera 28 is arranged so that its optical axis extends in a direction different from the optical axis direction of the camera 25 and the optical axis direction of the camera 26.

[0128] In the fourth robot device 10, a fifth control is performed to adjust the position and attitude of the workpiece 91 with respect to the workpiece 81. In the fifth control, a fifth feature portion is defined on the first workpiece 81 for processing the image captured by the camera 28. Furthermore, a sixth feature portion is defined on the second workpiece 91. For example, the side surface of the protrusion 83 of the workpiece 81 is defined as the fifth feature portion, and the side surface of the protrusion 93 of the workpiece 91 is defined as the sixth feature portion.

[0129] For the image captured by the camera 28, the same control as the first control described above can be performed. The camera 28 is arranged at a position where it can capture the fifth feature portion and the sixth feature portion. A fifth setting point and a sixth setting point for determining the position of the feature portions are set at the fifth feature portion and the sixth feature portion. The feature amount detection unit 32 can detect a fifth feature amount corresponding to the fifth feature portion and a sixth feature amount corresponding to the sixth feature portion based on the positions of the setting points in the screen coordinate system 52. A reference image regarding the camera 28 has been prepared in advance. Then, the fifth feature amount and the sixth feature amount in the reference image can be calculated in advance.In addition, the relative position amount in the reference image can be calculated in advance.

[0130] In the fifth control, the same control as in the third control described above can be performed. In the fifth control, the position and attitude of the second workpiece 91 with respect to the first workpiece 81 are adjusted by three cameras. In the control for adjusting the position of the workpiece 91 with respect to the workpiece 81, the deviation of the position of the second workpiece 91 with respect to the first workpiece 81 can be detected based on the images acquired by the three cameras 25, 26, 28. Therefore, the position adjustment can be performed more accurately than in the case of performing the position adjustment by two cameras.

[0131] In the control for correcting the position of the workpiece 91 with respect to the workpiece 81, the position of the workpiece 91, which is shifted about the Y-axis of the base coordinate system 51, can be corrected based on the image captured by the camera 25 and the image captured by the camera 28. In this way, by increasing the number of cameras, it is possible to increase the directions in which a positional shift is corrected.

[0132] Since the other structures, action, and results of the fourth robot device 10 are the same as those of the first robot device 5, the second robot device 8, and the third robot device 9, the explanation is not repeated here.

[0133] In the first robot device 5, the second robot device 8, the third robot device 9, and the fourth robot device 10 described above, the cameras 25, 26, and 28 are fixed to the hand 2 and move together with the hand 2, but this is not limited to this configuration. The cameras may also be fixed to the device surface. That is, the cameras may be fixed in such a way that the position and attitude of the cameras do not change when the position and attitude of the robot 1 changes.

[0134] Fig. 17 shows a schematic view of a fifth robot device in the present embodiment. The fifth robot device 6 performs an operation of placing a workpiece 97 as a second member inside a casing 87 as a first member. The casing 87 corresponds to the first workpiece, and the workpiece 97 corresponds to the second workpiece. The fifth robot device 6 has a hand 7 attached to the robot 1. The hand 7 is configured to grasp the workpiece 97 by suction. The fifth robot device 6 includes the conveyor 75 as a transport machine that transports the casing 87.

[0135] The fifth robot device 6 includes a camera 27 as a vision sensor, which is mounted on the setup surface via a stand 71. The camera 27 does not move even when the position and attitude of the robot 1 change. The camera 27 is arranged at a position sufficiently far from the conveyor 75 to capture the workpiece 97 and the housing 87 while the robot 1 adjusts the position of the workpiece 97.

[0136] Fig. 18 shows a side view when a workpiece is accommodated in the housing by the fifth robot device. The housing 87 is transported by the conveyor 75 as shown by the arrow 105. The housing 87 has a plurality of wall portions 87a configured to correspond to the shape of the workpiece 97. The fifth robot device 6 accommodates the workpiece 97 in an area surrounded by the wall portions 87a. In the example shown in Fig. 18, the position of the workpiece 97 is appropriate with respect to the area surrounded by the wall portions 87a. Therefore, the workpiece 97 can be accommodated in the housing 87 by lowering the workpiece 97 as shown by arrow 104.

[0137] Fig. Fig. 19 shows another side view when a workpiece is placed in the housing by the fifth robot device. In the example shown in Fig. 19, the position of the workpiece 97 is shifted with respect to the area surrounded by the wall portions 87a. The fifth robot device 6 executes a sixth control. In the sixth control, the position of the robot 1 is regulated so that the workpiece 97 is placed directly above the area surrounded by the wall portions 87a. The position of the workpiece 97 with respect to the housing 87 has been regulated in advance. In the sixth control, the position of the workpiece 97 with respect to the housing 87 is corrected by the same control as in the first control. A reference image in which the position of the workpiece 97 with respect to the housing 87 as in Fig. 18 shown was generated in advance.

[0138] Fig. 20 shows an image taken by the camera. In the image of Fig. 20, the position of the workpiece 97 is shifted with respect to the area surrounded by the wall portions 87a. In the sixth control, the side surface of a wall portion 87a can be set as the first feature portion. A first setting point P1 can be set at a corner of the side surface of the wall portion 87a. The side surface of the workpiece 97 can be set as the second feature portion. A second setting point P2 can be set at a corner of the side surface of the workpiece 97.

[0139] With reference to Fig. 3 and Fig.20, the feature amount detection unit 32 detects a first feature amount related to the position of the first feature portion and a second feature amount related to the position of the second feature portion. The arithmetic unit 33 calculates the relative position amount based on the first feature amount and the second feature amount. The command generation unit 34 generates a movement command to move the robot 1 based on the relative position amount in the image 65 captured by the camera 27 and the relative position amount in the reference image.

[0140] Even in a robot device with a fixed camera, the position of the robot can be controlled using the same control as the first control. Furthermore, by using multiple cameras fixed on the setup surface or the like, the same control as the second control, the third control, and the fourth control can be performed.

[0141] Since the other structures, action, and results of the fifth robot device 6 are the same as those of the first robot device 5, the second robot device 8, the third robot device 9, and the fourth robot device 10, the explanation is not repeated here.

[0142] In the embodiment described above, the vision sensor is a 2D camera, but it is not limited to this configuration. The vision sensor may also be a 3D camera that can detect the three-dimensional position of the elements included in the image. By using a 3D camera as the vision sensor, the position of the first element and the position of the second element in the base coordinate system can be detected even without preparing a base image.

[0143] In the above-described embodiment, a robot device that assembles products and a robot device that inserts workpieces into a case were used as examples, but they are not limited to these embodiments. The present embodiment can be applied to robot devices that perform any operations. For example, the present embodiment can be applied to a robot device that removes workpieces from a case, a robot device that performs spot welding, a robot device that applies an adhesive, or the like.

[0144] The first member and the second member in the above-described embodiment are workpieces or a housing, but are not limited to such configurations. Any member on which a robot performs an operation can be used for the first member. And any member that moves by driving the robot can be used for the second member. Specifically, the second member can be a work tool attached to the robot. For example, in a robot device that performs spot welding, a spot welding gun is attached to the robot as a work tool. In this case, the second member is the spot welding gun. On the spot welding gun, portions that can be detected in an image captured by a camera can be set as the second feature portion, the fourth feature portion, and the sixth feature portion.

[0145] In the above-described embodiment, the feature portions are set on the surface of the workpieces or the housing, but this is not limited to this. The feature portions may be set on structural members of the robot, structural members of the hand, or structural members of the transport machine. For example, in the first controller to the fourth controller, there may be no deviation in holding the workpiece with respect to the hand. Or the position of the robot may be regulated according to the deviation in holding the workpiece with respect to the hand. In these cases, the second feature portion, the fourth feature portion, and the sixth feature portion may be set on the hand if the hand is placed within the camera's shooting range.

[0146] The present disclosure can provide a robot apparatus that precisely controls the position of a robot by a simple method.

[0147] For the individual controls described above, the order of the steps can be changed as desired, as long as the functions and effects are not changed.

Claims

[1] Control device (29) of a robot device (5, 6, 8, 9, 10) which moves a second element (91, 97) by a robot (1) and adjusts the relative position of the second element with respect to a first element (81, 87), comprising a visual sensor (25, 27) receiving the first element and the second element; an operation control unit (43) which outputs a command to drive the robot to the robot; and an image processing unit (31) which processes the image captured by the visual sensor, wherein a first feature section for detecting the position of the first element and a second feature section for detecting the position of the second element are predetermined, wherein the image processing unit comprises a feature amount detection unit (32) which detects, in the image captured by the vision sensor, a first feature amount with respect to a position of the first feature portion and a second feature amount with respect to a position of the second feature portion, a computing unit (33) which calculates the difference between the first feature amount and the second feature amount as a relative position amount, and a command generation unit (34) which generates a movement command for operating the robot, wherein a relative position amount is set in a reference image (61) containing an image of the first feature portion and the second feature portion when the second element is arranged at a target position with respect to the first element, wherein the command generation unit generates a movement command based on the relative position amount in the image (62, 63, 65) captured by the vision sensor and the relative position amount in the reference image containing the image of the first feature portion and the second feature portion, in order to operate the robot so that the second element is arranged at the target position with respect to the first element, wherein the operation control unit changes the position of the robot based on the movement command. [2] The control device according to claim 1, wherein the command generation unit calculates a difference between the relative position amounts, which is the difference between the relative position amount in the image captured by the vision sensor and the relative position amount in the reference image, wherein a movement direction of the position of the robot with respect to the difference in the relative position amounts is determined in advance, wherein a method for calculating a movement amount of the position of the robot with respect to the difference in the relative position amounts is determined in advance, wherein the command generation unit determines the movement direction and the movement amount of the position of the robot based on the difference in the relative position amounts and generates a movement command based on the movement direction and the movement amount of the position of the robot. [3] The control device according to claim 1 or 2, wherein the robot device comprises a transport machine (75) that transports the first member, wherein the control for detecting the first feature amount and the second feature amount by the feature amount detection unit, the control for calculating the relative position amount by the arithmetic unit, and the control for generating the movement command by the command generating unit are repeated during the period in which the transport machine transports the first workpiece. [4] The control device according to claim 3, wherein the moving direction and the moving speed of the first member by the transport machine are set in advance, wherein the command generation unit generates the moving command based on the moving direction and the moving amount of the position of the robot, which position of the robot follows with respect to the position of the first member moved by the transport machine. [5] Control device according to one of claims 1 to 4, wherein the visual sensor forms a first visual sensor (25), wherein a second visual sensor (26) receiving the first element and the second element is provided, wherein a third feature section is defined for detecting the position of the first element and a fourth feature section is defined for detecting the position of the second element, wherein the feature amount detection unit detects a third feature amount with respect to a position of the third feature portion and a fourth feature amount with respect to a position of the fourth feature portion in the image captured by the second vision sensor, wherein the calculation unit calculates the difference between the third feature amount and the fourth feature amount as the relative position amount, where a relative position amount in a reference image, which contains an image of the third feature section and the fourth feature section when the second element is arranged at the target position with respect to the first element, wherein the command generation unit generates a movement command to operate the robot so that the second element is arranged at the target position with respect to the first element based on the relative position amount in the image captured by the second vision sensor and the relative position amount in the reference image including the image of the third feature portion and the fourth feature portion, and generates a movement command based on the movement command generated from the image of the first vision sensor and the movement command generated from the image of the second vision sensor, which is output to the operation control unit. [6] The control device according to claim 5, wherein the first visual sensor is arranged to capture an area located on one side in a predetermined direction of the second element, the second visual sensor is arranged to capture an area located on the other side in the predetermined direction of the second element, wherein it is determined whether or not the position of the second element with respect to the first element is within a predetermined determination range based on the movement command generated from the image of the first visual sensor and the movement command generated from the image of the second visual sensor.

Images