Control device for robot device

Abstract

The invention provides a control device for a robot device. A first feature of the first workpiece and a second feature of the second workpiece are predetermined. The feature amount detection unit detects, in an image captured by a camera, a first feature amount relating to a position of a first feature portion and a second feature amount relating to a position of a second feature portion. The calculation unit calculates a difference between the first feature amount and the second feature amount as a relative position amount. The command generation unit generates a movement command for operating the robot on the basis of the relative position amount in the image captured by the camera and the relative position amount in a predetermined reference image.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of July 29, 2020, application number 202010745050.8, and invention name “Control device for a robot device for controlling the position of a robot”.

[0002] Technical Area

[0003] The present invention relates to a control device for a robot device for controlling the position of the robot. Background Art

[0004] In a robot device including a robot, a work tool corresponding to the work can be mounted on the robot to perform a desired work. For example, by mounting a hand for holding a workpiece as a work tool on the robot, the robot device can transport the workpiece to a desired position.

[0005] When a workpiece is transported by a robot, the workpiece held by the robot can be mounted on another workpiece or placed inside another workpiece by controlling the position and posture of the workpiece. When performing such an operation, it is preferred to strictly align the position of the workpiece held by the robot with the other workpiece. For example, when performing an operation of fitting a workpiece to another workpiece, there is a case where the operation fails when the position and posture of one workpiece deviates from the position and posture of the other workpiece.

[0006] In the prior art, it is known that when a workpiece is mounted on another workpiece, the position of the robot is controlled using an image captured by a camera. For example, a target image of a workpiece or another workpiece can be prepared in advance. When the workpiece is transported by the robot device, the workpiece is photographed by the camera. In addition, it is known that the position of the robot is adjusted by comparing the image of the workpiece with the target image (for example, refer to Japanese Patent Publication No. 2013-180380 and Japanese Patent Publication No. 2015-214022).

[0007] In addition, the following control is known: the visual sensor coordinate system is pre-calibrated relative to the robot coordinate system, and the three-dimensional position of the workpiece is calculated based on the position of the workpiece in the visual sensor coordinate system. Alternatively, a Jacobian matrix related to the position and size of the feature part in the image can be generated in advance. The following control is known (for example, refer to Japanese Patent Publication No. 2017-170599): based on the position of the feature part in the image captured by the camera, the position of the feature part in the target data, and the Jacobian matrix, the position of the robot is corrected. Summary of the invention

[0008] In the method of using the Jacobian matrix to adjust the position of the robot, the movement amount of the robot can be calculated by multiplying the difference in the feature amount of the feature part of the image by the Jacobian matrix. However, sometimes the Jacobian matrix cannot be calculated with high precision due to measurement errors when calculating the Jacobian matrix. In addition, the Jacobian matrix is ​​calculated based on the position offset in the visual sensor coordinate system when the position of the robot is moved by a small amount. Therefore, the robot can be driven with high precision near the feature part, but the accuracy decreases when it is far away from the feature part. For example, there is a problem of poor accuracy when the robot is moved with a large movement amount. As a result, the position of the robot cannot be controlled with high precision sometimes.

[0009] In the method of calibrating the visual sensor coordinate system, the three-dimensional position of the characteristic part can be detected based on the image of the two-dimensional camera. However, in this method, the position of the visual sensor coordinate system needs to be calibrated in advance with respect to the robot coordinate system.

[0010] One embodiment of the present disclosure is a control device for a robot device that adjusts the relative position of the second member relative to the first member by moving the second member by the robot. The control device includes a visual sensor that captures the first member and the second member. The control device includes: an action control unit that sends a command for driving the robot to the robot; and an image processing unit that processes the image captured by the visual sensor. A first characteristic part for detecting the position of the first member and a second characteristic part for detecting the position of the second member are predetermined. The image processing unit includes a feature quantity detection unit that detects a first feature quantity related to the position of the first characteristic part and a second feature quantity related to the position of the second characteristic part in the image captured by the visual sensor. The image processing unit includes a calculation unit that calculates the difference between the first feature quantity and the second feature quantity as a relative position quantity. The image processing unit includes a command generation unit that generates a movement command for causing the robot to perform an action. The relative position quantity in a reference image is determined, and the reference image includes an image of the first characteristic part and the second characteristic part when the second member is configured at a target position relative to the first member. The command generation unit generates a movement command for causing the robot to move so as to arrange the second component at a target position relative to the first component based on the relative position amount in the image captured by the visual sensor and the relative position amount in the reference image including the image of the first characteristic part and the second characteristic part. The motion control unit changes the position of the robot based on the movement command. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of a first robot device in an embodiment.

[0012] Figure 2It is an enlarged perspective view of the first workpiece, the second workpiece, and the hand in the first robot device.

[0013] Figure 3 It is a block diagram of a first robot apparatus in an embodiment.

[0014] Figure 4 This is a first flowchart of the first control of the first robot device in the embodiment.

[0015] Figure 5 It is a reference image of the first robot device in the embodiment.

[0016] Figure 6 This is a second flowchart of the first control of the first robot device in the embodiment.

[0017] Figure 7 This is an image when the position of the second workpiece is offset relative to the first workpiece.

[0018] Figure 8 These are other images when the position of the second workpiece is offset relative to the first workpiece.

[0019] Fig. 9 It is an enlarged perspective view of a first workpiece, a second workpiece, and a hand in the second robot apparatus according to the embodiment.

[0020] Fig.10 This is an enlarged perspective view of the first workpiece, the second workpiece, and the hand when the posture of the second workpiece relative to the first workpiece is shifted in the second robot apparatus.

[0021] Fig.11 It is an image captured by the first camera when the posture of the second workpiece is offset relative to the first workpiece.

[0022] Fig.12 It is the image captured by the second camera when the posture of the second workpiece matches the posture of the first workpiece.

[0023] Fig.13 It is an image captured by the second camera when the posture of the second workpiece is offset relative to the first workpiece.

[0024] Fig.14 This is a flowchart of the third control in the second robot device.

[0025] Fig.15 It is an enlarged perspective view of a first workpiece, a second workpiece, and a hand in a third robot apparatus according to the embodiment.

[0026] Fig.16 This is an enlarged perspective view of a first workpiece, a second workpiece, and a hand in a fourth robot apparatus according to the embodiment.

[0027] Fig.17 This is a schematic diagram of a fifth robot apparatus in the embodiment.

[0028] Fig.18 This is an enlarged schematic diagram of the housing, the workpiece, and the hand when the position of the workpiece and the position of the housing are aligned.

[0029] Fig.19 This is an enlarged schematic diagram of the housing, the workpiece, and the hand when the position of the workpiece relative to the housing is offset.

[0030] Fig. 20 This is an image when the position of the workpiece is offset relative to the housing. DETAILED DESCRIPTION

[0031] Reference Figures 1 to 20 In this embodiment, a robot device that performs a work of assembling a product and a robot device that arranges a workpiece inside a housing are described as examples.

[0032] Figure 1 1 is a schematic diagram of the first robot device in this embodiment. The first robot device 5 includes a hand 2 as a working tool (end effector) and a robot 1 that moves the hand 2. The first robot device 5 performs an operation of attaching a second workpiece 91 as a second member to a first workpiece 81 as a first member.

[0033] The robot 1 is a multi-joint robot including a plurality of joints. The robot 1 includes a base portion 14 and a rotating base 13 supported by the base portion 14. The base portion 14 is fixed to a setting surface. The rotating base 13 is formed to rotate relative to the base portion 14. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is rotatably supported on the rotating base 13 via a joint portion. The upper arm 11 is rotatably supported on the lower arm 12 via a joint portion. In addition, the upper arm 11 rotates around a rotation axis parallel to the direction in which the upper arm 11 extends.

[0034] The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 is supported by the upper arm 11 via a joint so as to be rotatable. The wrist 15 includes a flange 16 formed so as to be rotatable. The hand 2 is fixed to the flange 16. The robot 1 of this embodiment has six drive axes, but is not limited to this. Any robot that can move a working tool can be used.

[0035] The hand 2 is a working tool for holding or releasing the workpiece 91. The hand 2 has a plurality of claws 3. The hand 2 is formed so that the claws 3 are opened or closed. The hand 2 holds the workpiece 91 by clamping the workpiece 91 with the claws 3. The hand 2 of the first robot device 5 has the claws 3, but is not limited to this. The hand can adopt any structure that can hold the workpiece. For example, a hand that holds the workpiece by adsorption or magnetic force can also be adopted.

[0036] The robot device 5 in this embodiment includes a conveyor belt 75 as a conveyor, and the conveyor belt 75 conveys the first workpiece 81 relative to the robot 1. The conveyor belt 75 is arranged near the robot 1. The conveyor belt 75 is formed to convey the workpiece 81 to a predetermined position. The conveyor belt 75 is formed to convey the workpiece 81 at a predetermined moving speed.

[0037] In the first robot device 5 of the present embodiment, the robot 1 attaches the workpiece 91 to the workpiece 81 while the workpiece 81 is continuously conveyed by the conveyor 75. That is, during the operation of attaching the workpiece 91, the workpiece 81 is moved by the conveyor 75. The robot 1 attaches the workpiece 91 to the workpiece 81 while changing its position and posture to follow the workpiece 81.

[0038] exist Figure 2 FIG. 4 shows an enlarged perspective view of the first workpiece, the second workpiece, and the hand in the first robot device. Figure 1 and Figure 2 The second workpiece 91 has a gripping portion 94 protruding from the surface of the main body. The second workpiece 91 is gripped by the hand 2 by the claw 3 grasping the gripping portion 94.

[0039] The first workpiece 81 has protrusions 82 and 83 protruding from the surface of the main body. The protrusions 82 and 83 are arranged separately from each other. In addition, the workpiece 81 is supported on the conveyor belt 75 in a manner that the protrusions 82 and 83 are arranged in the vertical direction. Holes 82a and 83a are formed on the upper surfaces of the respective protrusions 82 and 83. The second workpiece 91 has protrusions 92 and 93 protruding from the surface of the main body. Pins 92a and 93a are fixed to the respective protrusions 92 and 93. The pins 92a and 93a are arranged in a manner that they are arranged in a straight line with each other. In the first control implemented by the first robot device 5, the operation of inserting the pin 92a into the hole 82a and the pin 93a into the hole 83a is implemented.

[0040] In the first control of the present embodiment, control is performed to align the positions of the pins 92a, 93a with the holes 82a, 83a. Before the first control is implemented, the posture of the workpiece 91 relative to the workpiece 81 is adjusted. That is, the posture of the robot 1 is adjusted so that the pins 92a and 93a of the workpiece 91 are arranged on a straight line extending in the vertical direction. Therefore, in the first control, the positions of the pins 92a, 93a and the holes 82a, 83a can be aligned by implementing control to arrange the pins 92a just above the holes 82a. After the first control of the present embodiment is implemented, the workpiece 91 is moved to the lower side in the vertical direction indicated by the arrow 103, and the pins 92a, 93a are inserted into the holes 82a, 83a, so that the workpiece 91 is mounted on the workpiece 81.

[0041] The robot device 5 includes a camera 25 as a visual sensor for capturing images of the first workpiece 81 and the second workpiece 91. The camera 25 in this embodiment is a two-dimensional camera. The camera 25 is supported by the hand 2 via the support member 17. The position and posture of the camera 25 in the robot device 5 change together with the position and posture of the hand 2.

[0042] The camera 25 captures an image when the second workpiece 91 approaches the first workpiece 81. The camera 25 is configured to capture the vicinity of the portion where the second workpiece 91 fits into the first workpiece 81. In addition, the camera 25 is configured to capture a first characteristic portion for detecting the position of the first workpiece 81 and a second characteristic portion for detecting the position of the second workpiece 91. In the robot device 5, the camera 25 is configured to capture the workpieces 81 and 91 from above.

[0043] A reference coordinate system 51 is set in the robot device 5. Figure 1 In the example shown, the origin of the reference coordinate system 51 is arranged at the base portion 14 of the robot 1. The reference coordinate system 51 is also called the world coordinate system. The reference coordinate system 51 is a coordinate system in which the position of the origin is fixed and the orientation of the coordinate axis is fixed. Even if the position and posture of the robot 1 change, the position and orientation of the reference coordinate system 51 do not change. The reference coordinate system 51 has an X-axis, a Y-axis, and a Z-axis that are orthogonal to each other as coordinate axes. In addition, the W axis is set as a coordinate axis around the X axis. The P axis is set as a coordinate axis around the Y axis. The R axis is set as a coordinate axis around the Z axis.

[0044] The position and posture of the robot 1 can be represented by the reference coordinate system 51. For example, the position of the robot 1 can be represented by the position of the tool tip point arranged at the tip of the hand 2. In addition, a tool coordinate system that moves together with the hand 2 can be set at the tool tip point. The posture of the robot 1 can be represented by the orientation of the tool coordinate system relative to the reference coordinate system 51.

[0045] exist Figure 3 A block diagram of the robot device in this embodiment is shown in FIG. Figures 1 to 3 The robot 1 includes a robot driving device for changing the position and posture of the robot 1. The robot driving device includes a robot driving motor 22 for driving structural components such as an arm and a wrist. The robot driving motor 22 is driven to change the orientation of each structural component.

[0046] The hand 2 is provided with a hand driving device for driving the hand 2. The hand driving device includes a hand driving motor 21 for driving the claw 3 of the hand 2. The claw 3 of the hand 2 is opened or closed by driving the hand driving motor 21. In addition, the claw 3 can also be formed to work by using air pressure. In this case, the hand driving device can include a device that drives the claw by using air pressure such as an air pump and an air cylinder.

[0047] The control device 29 of the robot device 5 includes a robot control device 4 for controlling the robot 1 and the hand 2. The robot control device 4 includes a calculation processing device (computer) having a CPU (Central Processing Unit) as a processor. The calculation processing device has a RAM (Random Access Memory) and a ROM (Read Only Memory) connected to the CPU via a bus. An action program 41 prepared in advance for controlling the robot 1, the hand 2, and the conveyor 75 is input to the robot control device 4. The robot 1 and the hand 2 transport the workpiece 91 based on the action program 41. The conveyor 75 transports the workpiece 81 based on the action program 41.

[0048] The operation processing device of the robot control device 4 includes a storage unit 42 for storing predetermined information. The storage unit 42 stores information related to the control of the robot 1, the hand 2, and the conveyor 75. The storage unit 42 can be composed of a storage medium capable of storing information, such as a volatile memory, a non-volatile memory, or a hard disk. The action program 41 is stored in the storage unit 42. The robot control device 4 includes a display 46 for displaying arbitrary information related to the robot device 5. The display 46 includes, for example, a liquid crystal display panel.

[0049] The calculation processing device includes a motion control unit 43 that sends motion instructions to the robot 1 and the hand 2. The motion control unit 43 is equivalent to a processor driven according to the motion program 41. The motion control unit 43 is formed to be able to read information stored in the storage unit 42. The processor functions as the motion control unit 43 by reading the motion program 41 and implementing the control specified in the motion program 41.

[0050] The motion control unit 43 sends a motion command for driving the robot 1 to the robot driving unit 45 based on the motion program 41. The robot driving unit 45 includes a circuit for driving the robot driving motor 22. The robot driving unit 45 supplies power to the robot driving motor 22 based on the motion command. In addition, the motion control unit 43 sends a motion command for driving the hand 2 to the hand driving unit 44 based on the motion program 41. The hand driving unit 44 includes a circuit for driving the hand driving motor 21. The hand driving unit 44 supplies power to the hand driving motor 21 based on the motion command. In addition, a camera 25 is connected to the robot control device 4 of this embodiment. The motion control unit 43 sends a command for capturing an image to the camera 25 based on the motion program 41.

[0051] The operation processing device of the robot control device 4 includes an image processing unit 31 that processes the image captured by the camera 25. The image processing unit 31 has a feature quantity detection unit 32 that detects the feature quantity of a predetermined feature portion, that is, a feature part of each workpiece 81, 91. The image processing unit 31 has a calculation unit 33 that calculates the difference between the feature quantity of the first workpiece 81 and the feature quantity of the second workpiece 91 as a relative quantity. The image processing unit 31 has a command generation unit 34 that generates a movement command for causing the robot 1 to move based on the relative quantity calculated by the calculation unit 33.

[0052] The image processing unit 31 corresponds to a processor driven according to the operation program 41. In particular, each unit of the feature quantity detection unit 32, the calculation unit 33, and the command generation unit 34 corresponds to a processor driven according to the operation program 41. The processor reads the operation program 41 and implements the control specified in the operation program 41 to function as each unit.

[0053] The robot device 5 is provided with a state detector for detecting the operating state of the robot device 5. The state detector of this embodiment includes a position detector 23 for detecting the position and posture of the robot 1. The position detector 23 is mounted on the robot drive motor 22 corresponding to the drive shaft of the structural member such as the arm. For example, the position detector 23 detects the rotation angle when the robot drive motor 22 is driven. Based on the output of the position detector 23, the position and posture of the robot 1 are detected.

[0054] The control device 29 of the robot device 5 is provided with a conveyor belt control device 76 for controlling the operation of the conveyor belt 75. The conveyor belt control device 76 includes a calculation processing device (computer) having a CPU and a RAM, etc. The conveyor belt control device 76 is formed to be able to communicate with the robot control device 4. The motion control unit 43 sends an operation command for driving the conveyor belt 75 to the conveyor belt control device 76 based on the motion program 41. The conveyor belt control device 76 receives the operation command from the robot control device 4 to drive the conveyor belt 75.

[0055] The control device 29 of the robot device 5 of this embodiment includes the robot control device 4 for controlling the robot 1 and the hand 2 and the conveyor control device 76 for controlling the conveyor 75, but the present invention is not limited to this embodiment. The robot device 5 may be configured so that the robot 1, the hand 2 and the conveyor 75 are controlled by one control device.

[0056] Furthermore, in the control device 29 of the robot device 5 of the present embodiment, the robot control device 4 includes an image processing unit 31 having a function of processing an image, but the present invention is not limited to this embodiment. The control device of the robot device may also include an image processing device (computer) having the image processing unit 31. The image processing device can be composed of an operation processing device having a CPU as a processor. The processor of the image processing device functions as a feature quantity detection unit, a calculation unit, and an instruction generation unit driven based on an action program. The image processing device is formed to be able to communicate with the robot control device.

[0057] exist Figure 4 The first flow chart of the first control in this embodiment is shown in FIG. Figure 1 , Figure 2 as well as Figure 4 In the first control, the image captured by one camera 25 is used to align the position of the workpiece 91 with the workpiece 81 conveyed by the conveyor 75. As described above, before the first control, the posture of the workpiece 91 is adjusted in advance with respect to the posture of the workpiece 81. That is, the posture of the robot 1 when the image is captured by the camera 25 is adjusted. In the control here, the control of aligning the position of the pin 92a with the hole 82a is performed.

[0058] In this embodiment, the relative position amount in the reference image is calculated before the robot 5 actually performs the work. Then, the robot 5 performs the work using the relative position amount in the reference image calculated in advance. The first flowchart shows the control of calculating the relative position amount in the reference image.

[0059] In step 111, a reference image for performing the first control is generated. Figure 5 61 is an image captured by the camera 25 when the second workpiece 91 is arranged at the target position relative to the first workpiece 81. In the present embodiment, the reference image 61 is an image captured by the camera 25 when the pins 92a and 93a of the workpiece 91 are arranged just above the holes 82a and 83a of the workpiece 81. The reference image 61 can be prepared in advance by the operator and stored in the storage unit 42.

[0060] In the present embodiment, a first characteristic portion for detecting the position of the first workpiece 81 is predetermined. In the first control, the upper surface of the protrusion 82 is determined as the first characteristic portion. In addition, a second characteristic portion for detecting the position of the second workpiece 91 is predetermined. In the first control, the upper surface of the protrusion 92 is determined as the second characteristic portion. The characteristic portion is a portion whose shape can be detected when an image is analyzed. The characteristic portion can be a portion of the workpiece, a pattern formed on the surface of the workpiece, a line or a figure drawn on the surface of the workpiece, etc. In addition, the characteristic portion is preferably set to be near the portion where the second workpiece 91 contacts the first workpiece 81.

[0061] In the present embodiment, a set point P1 is set on the upper surface of the protrusion 82 as the first characteristic part in the first workpiece 81. The set point P1 is set at the corner of the protrusion 82. The position of the set point P1 corresponds to the position of the workpiece 81. A set point P2 is arranged on the upper surface of the protrusion 92 as the second characteristic part in the second workpiece 91. The set point P2 is set at the corner of the protrusion 92. The set point P2 corresponds to the position of the workpiece 91. The set points P1 and P2 are set in the area included in the image when the camera 25 takes the image.

[0062] Reference Figure 3 , Figure 4 as well as Figure 5 In step 112, the feature amount detection unit 32 of the image processing unit 31 detects the first feature part and the second feature part of the reference image 61. As a method for detecting the feature part, a base image serving as a reference for each workpiece 81, 91 can be prepared in advance. The feature part in the image captured by the camera 25 can be detected by a method such as template matching using the base image and the image captured by the camera 25. In this example, the upper surface of the protrusion 82 and the upper surface of the protrusion 92 can be detected.

[0063] In step 113, the feature quantity detection unit 32 detects a first feature quantity related to the position of the first feature part and a second feature quantity related to the position of the second feature part. In the present embodiment, a screen coordinate system 52 is set in the image captured by the camera 25. The screen coordinate system 52 is a coordinate system when an arbitrary point in the image is set as the origin. The screen coordinate system 52 has a u-axis and a v-axis that are orthogonal to each other. The screen coordinate system 52 corresponds to the visual sensor coordinate system of the camera 25.

[0064] The feature quantity related to the position in this embodiment is the coordinate value of the u axis and the coordinate value of the v axis of the screen coordinate system 52 in the image. The feature quantity detection unit 32 can detect the positions of the set points P1 and P2 set in the characteristic part based on the characteristic part detected in the reference image 61. The feature quantity detection unit 32 detects the coordinate value (u1b, v1b) of the set point P1 on the screen coordinate system 52 as the first feature quantity. In addition, the feature quantity detection unit 32 detects the coordinate value (u2b, v2b) of the set point P2 on the screen coordinate system 52 as the second feature quantity.

[0065] Next, in step 114, the calculation unit 33 of the image processing unit 31 calculates the relative amount related to the first feature amount and the second feature amount in the reference image. The calculation unit 33 calculates the relative position amount as the relative amount to control the position of the robot 1. The relative position amount is the difference between the first feature amount and the second feature amount. For example, the calculation unit 33 calculates the difference (u1b-u2b, v1b-v2b) between the coordinate value of the first feature amount and the coordinate value of the second feature amount as the relative position amount. The relative position amount in the reference image 61 calculated by the calculation unit 33 is stored in the storage unit 42 as the reference relative position amount.

[0066] In this way, the image processing unit 31 can calculate the relative position amount in the reference image 61. In addition, in the present embodiment, the relative position amount in the reference image 61 is calculated in advance and stored in the storage unit 42, but the present invention is not limited to this method. The relative position amount in the reference image 61 may also be calculated each time the first control is performed.

[0067] In addition, the reference image 61 does not need to use an image actually captured by the camera 25. For example, three-dimensional data of each workpiece 81, 91 can be generated by a CAD (Computer Aided Design) device or the like. Three-dimensional data when the workpiece 91 is arranged at a target position relative to the workpiece 81 can be generated. The reference image 61 can also be generated by projecting the three-dimensional data onto a plane in a direction corresponding to the orientation of the camera.

[0068] Next, the robot control device 4 performs the following control: the pin 92a of the second workpiece 91 is brought close to the hole 82a of the first workpiece 81 so as to arrange the first characteristic portion and the second characteristic portion inside the imaging range 25a of the camera 25. This control can be performed by any control. For example, the robot control device 4 detects the position of the workpiece 81 on the conveyor belt 75 by a predetermined sensor. The robot control device 4 detects the position of the workpiece 81 based on the moving speed of the conveyor belt 75. The robot control device 4 can control the position and posture of the robot 1 so that the workpiece 91 approaches the workpiece 81.

[0069] exist Figure 6 A second flowchart of the first control in this embodiment is shown in FIG. Figure 3 and Figure 6 After the robot control device 4 implements control to make the workpiece 91 approach the workpiece 81 , in step 115 , the motion control unit 43 uses the camera 25 to photograph the workpieces 81 , 91 .

[0070] exist Figure 7 6 shows an image captured by a camera in order to adjust the position of the second workpiece relative to the first workpiece. Image 62 includes an image of the upper surface of the protrusion 82 as the first characteristic part and an image of the upper surface of the protrusion 92 as the second characteristic part. In image 62, the second workpiece 91 is offset to the positive side of the u-axis of the screen coordinate system 52 relative to the first workpiece 81 as indicated by arrow 101.

[0071] exist Figure 8 63 shows another image captured by the camera in order to adjust the position of the second workpiece relative to the first workpiece. In the image 63, the second workpiece 91 is offset relative to the first workpiece 81 in the negative direction of the u-axis of the screen coordinate system 52 as indicated by the arrow 102. Figure 7 and Figure 8 In the first control, a control is implemented to correct the offset of the second workpiece 91. Figure 7 and Figure 8 In Figure 7 Take this as an example to illustrate.

[0072] Reference Figure 3 , Figure 6 as well as Figure 7 In steps 116 to 118, the same control as that for the reference image 61 is implemented. The image processing unit 31 detects the first feature amount and the second feature amount in the image 62 captured by the camera 25, and calculates the relative position amount using the first feature amount and the second feature amount.

[0073] In step 116, the feature quantity detection unit 32 detects the first feature portion and the second feature portion in the image 62 captured by the camera 25. Here, the upper surface of the protrusion 82 of the workpiece 81 is detected as the first feature portion, and the upper surface of the protrusion 92 of the workpiece 91 is detected as the second feature portion.

[0074] In step 117, the feature quantity detection unit 32 detects the first feature quantity and the second feature quantity in the image 62 captured by the camera 25. The feature quantity detection unit 32 detects the coordinate value (u1m, v1m) of the set point P1 on the screen coordinate system 52 as the first feature quantity related to the position of the first feature part. In addition, the feature quantity detection unit 32 detects the coordinate value (u2m, v2m) of the set point P2 on the screen coordinate system 52 as the second feature quantity related to the position of the second feature part.

[0075] In step 118, the calculation unit 33 calculates the difference between the first feature value and the second feature value as the relative position value. The relative position value in the image 62 captured by the camera 25 is the difference between the coordinate value of the first feature value and the coordinate value of the second feature value (u1m-u2m, v1m-v2m).

[0076] In addition, when the second workpiece 91 is held at a predetermined position of the hand 2, the second feature quantity related to the second workpiece 91 is fixed. Therefore, the second feature quantity may be measured in advance and stored in the storage unit 42. That is, the coordinate value of the set point P2 may be stored in advance in the storage unit 42. On the other hand, the second workpiece 91 may deviate from the desired position when the hand 2 holds it. Therefore, in the present embodiment, the second feature quantity is also detected based on the actually captured image 62 and by a template matching method with the base image.

[0077] Next, the command generation unit 34 of the image processing unit 31 generates a movement command for the robot 1 based on the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61, so as to arrange the second workpiece 91 at the target position relative to the first workpiece 81. The command generation unit 34 of the present embodiment generates a movement command for causing the robot 1 to operate so that the relative position amount in the image 62 captured by the camera 25 approaches the relative position amount in the reference image 61.

[0078] In step 119, the command generation unit 34 calculates the difference between the relative position amount in the image 62 captured by the camera 25 and the relative position amount in the reference image 61, that is, the difference in relative position amount. In the present embodiment, the command generation unit 34 calculates the difference in relative position amount obtained by subtracting the relative position amount in the reference image 61 from the relative position amount in the image 62 captured by the camera 25. The difference in relative position amount can be expressed as a value related to each u-axis and v-axis by [(u1m-u2m)-(u1b-u2b), (v1m-v2m)-(v1b-v2b)]. In this way, in the present embodiment, the difference in relative position amount related to the u-axis and the difference in relative position amount related to the v-axis are calculated.

[0079] Next, in step 120, the instruction generation unit 34 determines whether the difference in the relative position quantity is within a predetermined determination range. The determination range is predetermined and stored in the storage unit 42. For example, the determination range of the value related to the u-axis and the determination range of the value related to the v-axis can be pre-set. The closer the second workpiece 91 is to the target position relative to the first workpiece 81, the closer the difference in the relative position quantity is to zero. When the value related to the u-axis is within the determination range and the value related to the v-axis is within the determination range, it can be determined that the difference in the relative position quantity is within the determination range. That is, the instruction generation unit 34 can determine that the position alignment of the workpiece 91 and the workpiece 81 is completed.

[0080] On the other hand, when at least one of the value related to the u-axis and the value related to the v-axis deviates from the determination range, the command generation unit 34 can determine that the difference in relative position amount deviates from the determination range. In other words, the command generation unit 34 can determine that the workpiece 91 has not yet reached the desired position relative to the workpiece 81.

[0081] In step 120 , when the relative position difference is within the determination range, the control is terminated. In step 120 , when the relative position difference is out of the determination range, the control is transferred to step 121 .

[0082] In step 121, the command generation unit 34 sets the driving method of the robot 1 based on the difference in relative position amount. The command generation unit 34 sets the moving direction and moving amount of the position of the robot 1 in the reference coordinate system 51. In the present embodiment, the moving direction of the position of the robot corresponding to the difference in relative position amount is predetermined. The moving direction of the position of the robot 1 is determined on the reference coordinate system 51 corresponding to the positive or negative value of the u-axis of the screen coordinate system 52. For example, when the difference in the relative position amount related to the u-axis is a positive value, the moving direction of (1, 1, 0) is predetermined using the coordinate values ​​of the X-axis, Y-axis, and Z-axis of the reference coordinate system 51. In addition, when the difference in the relative position related to the v-axis is a positive value, the moving direction of (0, 0, 1) is predetermined using the coordinate values ​​of the X-axis, Y-axis, and Z-axis of the reference coordinate system 51.

[0083] Furthermore, a method for calculating the amount of movement of the position of the robot 1 corresponding to the difference in relative position amount is predetermined. For example, the amount of movement of the position of the robot 1 in the direction corresponding to the u-axis can be a value obtained by multiplying the value related to the u-axis ((u1m-u2m)-(u1b-u2b)) by a predetermined coefficient. In addition, the amount of movement of the position of the robot 1 in the direction corresponding to the v-axis can be a value obtained by multiplying the value related to the v-axis ((v1m-v2m)-(v1b-v2b)) by a predetermined coefficient. In this way, the amount of movement of the position of the robot 1 can be calculated in the directions corresponding to the axes of the screen coordinate system 52.

[0084] In the present embodiment, the movement amount in the X-axis direction, the movement amount in the Y-axis direction, and the movement amount in the Z-axis direction of the reference coordinate system 51 are calculated based on the difference in the relative position amount related to the u-axis. In addition, the movement amount in the X-axis direction, the movement amount in the Y-axis direction, and the movement amount in the Z-axis direction of the reference coordinate system 51 are calculated based on the difference in the relative position amount related to the v-axis. Therefore, sometimes two movement amounts (the movement amount related to the u-axis and the movement amount related to the v-axis) are calculated for one axis in the reference coordinate system 51. In this case, the position of the robot 1 may not be moved in the direction of the axis for which the two movement amounts are calculated. Alternatively, each movement amount may be multiplied by a coefficient to calculate the final movement amount. Alternatively, the movement amount of either party may be used.

[0085] Next, in step 122, the robot 1 is driven based on the moving direction and the moving amount of the position of the robot 1. The command generation unit 34 generates a movement command for driving the robot 1 based on the moving direction and the moving amount of the position of the robot 1. The command generation unit 34 sends the movement command to the motion control unit 43. The motion control unit 43 changes the position of the robot 1 based on the movement command.

[0086] Then, the control is transferred to step 115. In the first control, the control of steps 115 to 122 is repeated until the difference in the relative position amount is within the determination range. In the first control, even if the position alignment of the workpiece is not completed by one control, the position of the workpiece can be gradually approached to the desired position by repeating steps 115 to 122.

[0087] In the control of this embodiment, it is not necessary to calibrate the visual sensor coordinate system with respect to the reference coordinate system. Alternatively, it is not necessary to calculate the Jacobian matrix in advance for the position alignment of the workpiece. Therefore, the position alignment of the workpiece can be performed by a simple method.

[0088] In the first control of the present embodiment, the position alignment of the second workpiece 91 is performed while the conveyor 75 is conveying the first workpiece 81. The control of detecting the first feature quantity and the second feature quantity by the feature quantity detection unit 32, the control of detecting the relative position quantity by the calculation unit 33, and the control of calculating the movement command by the command generation unit 34 are repeated. Through this control, the position of the robot 1 holding the workpiece 91 can be controlled to follow the position of the workpiece 81 moved by the conveyor 75.

[0089] In addition, the camera 25 in this embodiment is supported by the hand 2 via the support member 17. The hand 2 holds the second workpiece 91. Therefore, during the period of aligning the second workpiece 91 with the first workpiece 81, the relative position and posture of the workpiece 81 with respect to the camera 25 are fixed. Figure 5 , Figure 7 as well as Figure 8 Even if the relative position of the second workpiece 91 to the first workpiece 81 changes, the positions of the workpiece 91 and the protrusion 92 in the images 62 and 63 captured by the camera 25 are fixed. As a result, the position of the set point P2 of the second characteristic part in the image captured by the camera 25 is fixed.

[0090] Reference Figure 6 In the above control, each time the camera 25 captures the first workpiece 81 and the second workpiece 91, the second characteristic portion in the second workpiece 91 is detected, and the second characteristic quantity is detected. On the other hand, since the coordinate value of the set point P2 as the second characteristic quantity is fixed, the storage unit 42 can store in advance the second characteristic quantity obtained from the image captured by the camera 25 for the first time. In steps 115 to 117, the characteristic quantity detection unit 32 can obtain the second characteristic quantity from the storage unit 42 for the image captured by the camera 25 after the second time. The characteristic quantity detection unit 32 only needs to implement the detection of the first characteristic portion and the detection of the first characteristic quantity. By implementing this control, the control of calculating the relative position quantity based on the image captured by the camera after the second time can be simplified. The processing time for the image captured by the camera after the second time is shortened.

[0091] In addition, in this embodiment, the moving direction and moving speed of the first workpiece 81 moved by the conveyor 75 are predetermined. The robot control device 4 can perform feedforward control to change the position of the robot 1 according to the movement of the workpiece 81 by the conveyor 75. In this embodiment, the workpiece 81 moves at a fixed moving speed.

[0092] The command generation unit 34 calculates the moving direction and moving speed of the position of the robot 1 so that the position of the robot 1 follows the position of the first workpiece 81 moved by the conveyor 75. For example, the command generation unit 34 calculates the moving direction so that the tool tip of the robot 1 moves in the moving direction of the workpiece 81. The command generation unit 34 can calculate the moving amount of the tool tip of the robot 1 moving in the same direction as the moving direction of the workpiece 81 at the same moving speed as the workpiece 81. In addition, the command generation unit 34 can control the moving direction and moving amount of the position of the robot 1 based on the conveyance of the conveyor 75, and can also implement the control of the moving direction and moving amount calculated based on the difference in the relative position amount described above.

[0093] By performing this control, the position and posture of the robot 1 related to the movement of the first workpiece 81 by the conveyor 75 can be changed by feedforward control. In the control based on the difference in relative position, it is only necessary to correct the relative position offset of the second workpiece 91 with respect to the first workpiece 81, so the position alignment of the second workpiece 91 and the first workpiece 81 can be performed in a short time.

[0094] exist Fig. 9 2 shows an enlarged stereoscopic view of the hand of the second robot device, the first workpiece, and the second workpiece in this embodiment. In the first robot device 5, the position of the robot 1 is adjusted by one camera 25, but the present invention is not limited to this method. The robot device can use two or more cameras to adjust the position of the robot 1.

[0095] In the second robot device 8, the support member 18 is fixed to the hand 2. The support member 18 has a portion 18a extending upward and a portion 18b extending downward. As in the first robot device 5, a camera 25 is fixed to the portion 18a extending upward as a first visual sensor. The camera 25 captures an image with a camera range 25a. A camera 26 as a second visual sensor is fixed to the portion 18b extending downward. The camera 26 captures an image of the first workpiece 81 and the second workpiece 91. In particular, the camera 26 captures an image when the second workpiece 91 approaches the first workpiece 81. The camera 26 is arranged at a position where it can capture the protrusion 83 of the first workpiece 81 and the protrusion 93 of the second workpiece 91. The portion 18b extending downward supports the camera 26 so that the protrusions 83 and 93 can be captured from the side of the workpieces 81 and 91. The camera 26 captures an image with a camera range 26a. The camera 26 of this embodiment is a two-dimensional camera.

[0096] The two cameras 25 and 26 are arranged so that their optical axes extend in different directions. In the present embodiment, the camera 26 is arranged so that its optical axis extends in a direction substantially orthogonal to the optical axis of the camera 25. In the second robot device 8, the second control is performed based on the images captured by the two cameras 25 and 26. In the second control, the position of the workpiece 91 relative to the workpiece 81 is adjusted using the images captured by the cameras 25 and 26.

[0097] In the second control, the movement command is generated based on the image of the camera 25 by the first control. In addition, the movement command is generated based on the image of the camera 26 by the same method as the first control. In this embodiment, the third characteristic part for detecting the position of the first workpiece 81 and the fourth characteristic part for detecting the position of the second workpiece 91 in the image captured by the camera 26 are predetermined.

[0098] The third characteristic part is a part different from the first characteristic part. As the third characteristic part, for example, the side of the protrusion 83 of the first workpiece 81 can be set. The third setting point P3 for determining the position of the first workpiece 81 can be set in the third characteristic part. The fourth characteristic part is a part different from the second characteristic part. As the fourth characteristic part, for example, the side of the protrusion 93 of the second workpiece 91 can be set. The fourth setting point P4 for determining the position of the second workpiece 91 can be set in the fourth characteristic part.

[0099] The feature quantity detection unit 32 detects a third feature quantity related to the position of the third feature part and a fourth feature quantity related to the position of the fourth feature part in the image captured by the camera 26. In the image captured by the camera 26, the coordinate value of the point P3 on the screen coordinate system 52 is set as the third feature quantity. In addition, the coordinate value of the point P4 on the screen coordinate system 52 is set as the fourth feature quantity. The calculation unit 33 calculates the difference between the third feature quantity and the fourth feature quantity as the relative position quantity.

[0100] In addition, a reference image is prepared in advance that is related to the image of the camera 26 when the second workpiece 91 is arranged at the target position relative to the first workpiece 81. In addition, the relative position amount, which is the difference between the third feature amount and the fourth feature amount in the reference image, is determined in advance. The relative position amount in the reference image can be calculated in advance.

[0101] The command generation unit 34 calculates the difference in the relative position amount based on the relative position amount in the image captured by the camera 26 and the relative position amount in the reference image including the image of the third characteristic part and the fourth characteristic part. Then, the command generation unit 34 generates a movement command for causing the robot 1 to operate so as to arrange the second workpiece 91 at the target position relative to the first workpiece 81 based on the difference in the relative position amount. The command generation unit 34 generates a movement command for causing the robot 1 to operate so that the relative position amount in the image captured by the second camera 26 approaches the relative position amount in the reference image.

[0102] The command generation unit 34 can generate a final movement command to be sent to the motion control unit 43 based on the movement command generated based on the image of the camera 25 and the movement command generated based on the image of the camera 26. For example, the command generation unit 34 can drive the robot 1 by one of the movement command based on the image of the camera 25 and the movement command based on the image of the camera 26, and then drive the robot 1 by the other movement command.

[0103] Alternatively, the movement command based on the image of the camera 25 and the movement command based on the image of the camera 26 may be synthesized. For example, when the movement direction of the position of the robot 1 corresponding to the u-axis of the image of the camera 25 coincides with the movement direction of the position of the robot 1 corresponding to the u-axis of the image of the camera 26, the average value of the movement amount of the robot 1 may be calculated. Such control for adjusting the position of the robot 1 may be repeatedly performed until the difference in the relative position amount based on the image of the camera 25 is within the determination range and the difference in the relative position amount based on the image of the camera 26 is within the determination range.

[0104] In the second robot device 8, each camera 25, 26 captures a different characteristic portion to control the position. Therefore, compared with the control of position adjustment using a single camera, the position of the workpiece 91 can be adjusted with high accuracy. In addition, the position of the workpiece can be adjusted based on images from multiple cameras without stereo measurement or the like. Furthermore, even without using a three-dimensional camera, the position of the workpiece can be adjusted by using multiple two-dimensional cameras.

[0105] In addition, the position of the set point P2 in the image captured by the first camera 25 and the position of the set point P4 in the image captured by the second camera 26 do not change. Therefore, the second feature quantity and the fourth feature quantity detected at the beginning can be stored in the storage unit 42 in the same manner as in the first control. Then, in the second and subsequent controls, the second feature quantity and the fourth feature quantity stored in the storage unit 42 can be acquired to calculate the relative position quantity in each image.

[0106] In addition, a plurality of cameras 25 and 26 are disposed in the second robot device 8. In the control device of such a robot device 8, in addition to adjusting the position of the workpiece 91, the posture of the workpiece 91 can be corrected. Next, correction of the posture of the workpiece 91 will be described.

[0107] exist Fig.10 , other enlarged stereoscopic views of the hand, the first workpiece, and the second workpiece in the second robot device of the present embodiment are shown in FIG. In the first control and the second control, the relative posture of the second workpiece 91 with respect to the first workpiece 81 is adjusted before the control for adjusting the position of the second workpiece 91 is implemented. On the other hand, there is a case where the posture of the second workpiece 91 with respect to the first workpiece 81 is offset.

[0108] When the workpiece 91 is in the target posture, the direction in which the pins 92a and 93a are arranged is parallel to the direction in which the holes 82a and 83a are arranged. Fig.10In the example shown, the posture of the workpiece 91 is slightly rotated from the target posture as indicated by arrow 106. The workpiece 91 is slightly rotated from the Y axis toward the Z axis around the X axis of the reference coordinate system 51. The workpiece 91 is slightly rotated from the target posture around the rotation axis of the flange 16. As a result, the position of the protrusion 92 of the workpiece 91 relative to the protrusion 82 of the workpiece 81 is offset in the direction of the negative side of the Y axis as indicated by arrow 107. In addition, the position of the protrusion 93 of the workpiece 91 relative to the protrusion 83 of the workpiece 81 is offset in the direction of the positive side of the Y axis as indicated by arrow 108.

[0109] The plurality of cameras 25 and 26 in the present embodiment are configured to photograph the portions of the second workpiece 91 that are arranged apart from each other. The first camera 25 is configured to photograph the protruding portion 92 that is a portion arranged on one side in a predetermined direction of the second workpiece 91. In addition, the second camera 26 is configured to photograph the protruding portion 93 that is a portion arranged on the other side in a predetermined direction of the second workpiece 91. The predetermined direction here corresponds to the direction in which the pins 92a and the pins 93a are arranged.

[0110] exist Fig.11 , an image captured by the first camera when the posture of the second workpiece is offset relative to the first workpiece. Fig.10 , Fig.11 and as a reference image Figure 5 , the relative position of the workpiece 91 with respect to the workpiece 81 in the image 66 is different from the relative position of the workpiece 91 with respect to the workpiece 81 in the reference image 61. The protrusion 92 of the workpiece 91 is offset in the direction indicated by the arrow 107 relative to the protrusion 82 of the workpiece 81. In addition, the direction in which the pin 92a fixed to the protrusion 92 extends is not parallel to the direction in which the hole 82a formed in the protrusion 82 extends, but is oriented in different directions.

[0111] exist Fig.12 67 shows an image captured by the second camera when the posture of the second workpiece relative to the first workpiece is the target posture. In the image 67, the direction in which the pin 93a fixed to the protrusion 93 extends is parallel to the direction in which the hole 83a formed in the protrusion 83 extends. Therefore, by aligning the positions of the workpieces 91 and 81, the pin 93a can be inserted into the inside of the hole 83a.

[0112] exist Fig.13 68 shows an image captured by the second camera when the posture of the second workpiece relative to the first workpiece is offset. In image 68, workpiece 91 is slightly rotated relative to workpiece 81 as shown by arrow 108. Fig.10 , Fig.12 as well as Fig.13, the relative position of the protrusion 93 in the image 68 relative to the protrusion 83 is offset as shown by the arrow 108. In addition, the direction in which the pin 93a extends is not parallel to the direction in which the hole 83a extends. Therefore, even if the front end of the pin 93a is arranged directly above the front end of the hole 83a, the pin 93a cannot be inserted into the inside of the hole 83a. In the third control of this embodiment, a control is implemented to correct the relative posture of the second workpiece 91 relative to the first workpiece 81.

[0113] exist Fig.14 A flowchart of the third control in the second robot apparatus of this embodiment is shown in . In the third control, in addition to the second control, control for correcting the posture of the workpiece 91 is performed.

[0114] Steps 131 to 138 are the same as the second control described above. In step 131, the first camera 25 and the second camera 26 are used to capture an image. In step 132, the feature quantity detection unit 32 detects the first feature portion and the second feature portion in the image captured by the first camera 25. In step 133, the feature quantity detection unit 32 detects the third feature portion and the fourth feature portion in the image captured by the second camera 26.

[0115] In step 134, the feature quantity detection unit 32 detects a first feature quantity related to the position of the first feature part, a second feature quantity related to the position of the second feature part, a third feature quantity related to the position of the third feature part, and a fourth feature quantity related to the position of the fourth feature part. Fig.11 The feature quantity detection unit 32 detects the coordinate value of the set point P1 on the screen coordinate system 52 in the image 66 captured by the first camera 25 as the first feature quantity. In addition, the feature quantity detection unit 32 detects the coordinate value of the set point P2 on the screen coordinate system 52 as the second feature quantity. Fig.13 The feature quantity detection unit 32 detects the coordinate value of the set point P3 on the screen coordinate system 52 in the image 68 as the third feature quantity. The feature quantity detection unit 32 detects the coordinate value of the set point P4 on the screen coordinate system 52 as the fourth feature quantity.

[0116] Reference Fig.14 In step 135, the calculation unit 33 of the image processing unit 31 calculates the relative position amount based on the first feature amount and the second feature amount. In step 136, the calculation unit 33 calculates the relative position amount based on the third feature amount and the fourth feature amount. The calculation unit 33 calculates the relative position amount in the images 66 and 68 captured by the respective cameras 25 and 26.

[0117] Next, in step 137, the command generation unit 34 of the image processing unit 31 calculates the difference between the relative position amount in the images 66 and 68 and the relative position amount in the reference image, that is, the difference in relative position amount, for the images 66 and 68 captured by the respective cameras 25 and 26. In step 138, the command generation unit 34 generates a movement command for the robot 1 based on the difference in relative position amount for each of the images 66 and 68. The difference in relative position amount is calculated as a value related to the u-axis and the v-axis in the screen coordinate system 52. The command generation unit 34 generates a movement command for the robot 1 based on the difference in relative position amount.

[0118] The command generation unit 34 calculates the movement direction and movement amount of the position of the robot 1 in the reference coordinate system 51 based on the image 66 captured by the camera 25. In addition, the command generation unit 34 calculates the movement direction and movement amount of the position of the robot 1 in the reference coordinate system 51 based on the image 68 captured by the camera 26. That is, the command generation unit 34 calculates the movement amount in the direction along the coordinate axis of the reference coordinate system 51 for each coordinate axis of the reference coordinate system 51.

[0119] Next, in step 139 , the command generation unit 34 determines whether the posture of the second workpiece 91 relative to the first workpiece 81 is within a predetermined determination range based on the movement command generated based on the image of the first camera 25 and the movement command generated based on the image of the second camera 26 .

[0120] In the present embodiment, the command generation unit 34 acquires the movement direction of the robot 1 at the position on the predetermined coordinate axis of the reference coordinate system 51. Fig.10 In this example, the Y axis is preselected among the coordinate axes of the reference coordinate system 51. The command generation unit 34 obtains the Y axis movement command among the movement commands generated based on the image 66 of the first camera 25 and the Y axis movement command among the movement commands generated based on the image 68 of the second camera 26.

[0121] When the moving directions on the Y axis acquired from the images 66 and 68 captured by the two cameras 25 and 26 are the same, the command generation unit 34 determines that the posture of the second workpiece 91 relative to the first workpiece 81 is within the predetermined determination range. On the other hand, when the moving directions on the Y axis are different from each other, the command generation unit 34 determines that the posture of the second workpiece 91 relative to the first workpiece 81 is out of the determination range.

[0122] exist Figures 10 to 13In the example shown, in the movement command based on the image captured by the first camera 25, a command is generated for moving the robot 1 to the side opposite to the arrow 107. In addition, in the movement command based on the image captured by the second camera 26, a command is generated for moving the robot 1 to the side opposite to the arrow 108. Therefore, in step 139, the command generation unit 34 determines that the movement directions on the Y axis based on the images captured by the two cameras are different from each other. In this case, the control is transferred to step 140.

[0123] In addition, in the present embodiment, when the directions in which the position of the robot should be moved on the predetermined coordinate axes of the reference coordinate system are the same, it is determined that the robot is within the determination range, but the present invention is not limited to this method. Even when the movement directions on the predetermined coordinate axes are different, it can be determined that the robot is within the determination range if the movement amount is small.

[0124] In step 140, the command generation unit 34 sets a method for correcting the posture of the robot 1. The command generation unit 34 generates a movement command for rotating the workpiece 91 in the direction opposite to the direction in which the second workpiece 91 is tilted relative to the first workpiece 81. Fig.10 In the present embodiment, control is performed to rotate the workpiece 91 in the direction from the Z axis toward the Y axis around the X axis of the reference coordinate system 51. The rotation amount of the workpiece 91, that is, the posture correction amount θ can be determined in advance.

[0125] In step 141, the command generation unit 34 sends a movement command based on the method of correcting the posture of the robot 1 to the motion control unit 43. The motion control unit 43 corrects the posture of the robot based on the movement command received from the command generation unit 34.

[0126] Next, in step 142, the image processing unit 31 corrects the moving direction of the position of the robot 1 on the reference coordinate system 51 with respect to the coordinate axes of the screen coordinate system 52. The difference in relative position amount is calculated by the values ​​related to the u-axis and the v-axis of the screen coordinate system 52. The moving direction of the position of the robot 1 in the reference coordinate system 51 corresponding to the u-axis and the v-axis of the screen coordinate system 52 is predetermined. When the posture of the robot 1 is corrected, the moving direction in the reference coordinate system 51 corresponding to the u-axis and the v-axis of the image captured by each camera 25, 26 also changes. That is, the moving direction of the position of the robot 1 represented by the coordinate values ​​of the X-axis, Y-axis, and Z-axis of the reference coordinate system changes.

[0127] The image processing unit 31 corrects the moving direction of the position of the robot 1 corresponding to the coordinate axis of the screen coordinate system 52 based on the correction amount of the posture of the robot 1. For example, the moving direction of the position of the robot 1 can be corrected by multiplying the transformation matrix calculated based on the correction amount of the posture of the robot 1 by the coordinate values ​​on the X-axis, Y-axis, and Z-axis of the reference coordinate system 51 indicating the moving direction of the position of the robot 1.

[0128] After step 142, control returns to step 131. Then, control in steps 131 to 139 is repeated. In step 139, if the movement directions on the predetermined coordinate axes of reference coordinate system 51 are different from each other, control in steps 140, 141, and 142 is performed again.

[0129] In step 140, a method for correcting the posture of the robot 1 is set. Sometimes, by correcting the posture of the robot 1 with the previous posture correction amount θ, the direction of the posture deviation of the second workpiece 91 relative to the first workpiece 81 becomes the opposite direction. In this case, the posture of the robot 1 is corrected in the direction opposite to the previous posture correction. A correction amount that is reduced compared to the previous correction amount is also used. For example, control is implemented to change the posture with a correction amount (-θ / 2) that is half of the previous correction amount. On the other hand, sometimes, even if the posture of the robot 1 is corrected with the previous posture correction amount θ, the direction of the posture deviation of the second workpiece 91 relative to the first workpiece 81 is the same as that in the previous control. In this case, the posture is corrected in the same direction and with the same posture correction amount θ as the previous time. Thereafter, the control of steps 131 to 139 can be repeated again.

[0130] In this way, when the direction of the posture deviation is reversed by the posture correction in the previous control, control can be performed to correct the posture of the robot in the opposite direction and reduce the correction amount. Through this control, the deviation of the posture of the second workpiece 91 relative to the first workpiece 81 can be gradually corrected. The posture of the robot 1 can be corrected in the images captured by the first camera 25 and the second camera 26 until the deviation direction of the second workpiece 91 relative to the first workpiece 81 becomes the same.

[0131] In addition, in step 139, when the posture of the second workpiece relative to the first workpiece deviates from the determination range, arbitrary control can be implemented. For example, the robot device can also be stopped. In addition, in step 140, the command generation unit 34 can set a method for correcting the posture of the robot 1 through arbitrary control. For example, the posture correction amount of the workpiece 81 can also be calculated based on the movement amount in the Y-axis direction calculated from the images captured by the two cameras 25 and 26.

[0132] In step 139 , when the movement direction based on the image captured by the first camera 25 and the movement direction based on the image captured by the second camera 26 are the same, control proceeds to step 143 .

[0133] In step 143 to step 145, the position of the robot 1 is determined and corrected in the same manner as in the second control. In step 143, it is determined whether the difference in the relative position amount based on the image captured by the first camera 25 and the difference in the relative position amount based on the image captured by the second camera 26 are within a predetermined determination range. In step 143, if at least one relative position amount difference is out of the determination range, control is transferred to step 144.

[0134] In step 144, the command generation unit 34 generates a final robot movement command based on the movement command based on each image. Then, in step 145, the motion control unit 43 drives the robot 1 based on the final robot movement command to change the position of the robot 1. After that, control is transferred to step 131.

[0135] In step 143 , when the difference in the relative position amount between the two images is within the determination range, it can be determined that the position and posture of the second workpiece 91 relative to the first workpiece 81 are the target position and the target posture. Then, this control can be ended.

[0136] The structure, function, and effect of the second robot device 8 other than those described above are the same as those of the first robot device 5 , and thus will not be described again here.

[0137] exist Fig.15 2 shows an enlarged stereoscopic view of the hand, the first workpiece, and the second workpiece of the third robot device in this embodiment. In the third robot device 9, the support member 19 is fixed to the hand 2. The support member 19 has a portion 19a extending upward and a portion 19b extending downward. As with the second robot device 8, a camera 25 is fixed to the portion 19a extending upward as a first visual sensor. A camera 26 is fixed to the portion 19b extending downward as a second visual sensor.

[0138] The orientation of the camera 26 of the third robot apparatus 9 is different from that of the camera 26 of the second robot apparatus 8. The portion 19b extending downward of the support member 19 supports the camera 26 so that the imaging range 26a of the camera 26 overlaps the imaging range 25a of the camera 25. The camera 26 of the third robot apparatus 9 is configured to image the same portion as the portion imaged by the camera 25. The camera 26 is configured to be able to image the protruding portion 82 of the workpiece 81 and the protruding portion 92 of the workpiece 91.

[0139] In the third robot device 9, the fourth control is implemented to align the position of the workpiece 91 with respect to the workpiece 81. In the fourth control, similarly to the second control of the second robot device 8, the third characteristic part and the fourth characteristic part are set, and the position of the robot 1 is adjusted based on the images of the plurality of cameras 25 and 26. In the present embodiment, the side surface of the protrusion 82 is set as the third characteristic part. In addition, the side surface of the protrusion 92 is set as the fourth characteristic part. In the third robot device 9, the third characteristic part may be the same as or different from the first characteristic part. In addition, the fourth characteristic part may be the same as or different from the second characteristic part.

[0140] The other structures, functions and effects of the third robot device 9 are the same as those of the second robot device 8 , and thus will not be described again here.

[0141] The control device of the second robot device 8 and the control device of the third robot device 9 are provided with two cameras, but the present invention is not limited to this embodiment. The control device of the robot device may also be provided with three or more cameras. The control device may use each camera to capture different characteristic parts, and implement the control of the position and posture of the workpiece based on the images captured by each camera.

[0142] exist Fig.16 , an enlarged stereoscopic view of the hand, the first workpiece, and the second workpiece of the fourth robot device in the present embodiment is shown. The fourth robot device 10 includes a third camera 28 in addition to the first camera 25 and the second camera 26. The fourth robot device 10 includes a supporting member 20 supported by the hand 2. The supporting member 20 includes a portion 20a that supports the camera 25 and extends upward, and a portion 20b that supports the camera 26 and extends downward. In addition, the supporting member 20 includes a portion 20c that supports the camera 28 and extends laterally. The camera 28 has a camera range 28a. The camera 28 is configured so that the optical axis extends in a direction different from the direction of the optical axis of the camera 25 and the direction of the optical axis of the camera 26.

[0143] In the fourth robot device 10, a fifth control for adjusting the position and posture of the workpiece 91 relative to the workpiece 81 is implemented. In the fifth control, a fifth characteristic portion is set in the first workpiece 81 to process the image captured by the camera 28. In addition, a sixth characteristic portion is set in the second workpiece 91. For example, the side surface of the protrusion 83 of the workpiece 81 is set as the fifth characteristic portion, and the side surface of the protrusion 93 of the workpiece 91 is set as the sixth characteristic portion.

[0144] The same control as the first control described above can also be implemented with respect to the image captured by the camera 28. The camera 28 is arranged at a position where the fifth characteristic part and the sixth characteristic part can be captured. A fifth setting point and a sixth setting point for determining the position of the characteristic part are set at the fifth characteristic part and the sixth characteristic part. The characteristic quantity detection unit 32 can detect the fifth characteristic quantity corresponding to the fifth characteristic part and the sixth characteristic quantity corresponding to the sixth characteristic part based on the position of the setting point in the screen coordinate system 52. A reference image related to the camera 28 is prepared in advance. Then, the fifth characteristic quantity and the sixth characteristic quantity in the reference image can be calculated in advance. In addition, the relative position quantity in the reference image can be calculated in advance.

[0145] In the fifth control, the same control as the third control described above can be implemented. In the fifth control, the position and posture of the second workpiece 91 relative to the first workpiece 81 are adjusted by three cameras. In the control of adjusting the position of the workpiece 91 relative to the workpiece 81, the positional deviation of the second workpiece 91 relative to the first workpiece 81 can be detected based on the images obtained by the three cameras 25, 26, and 28. Therefore, compared with the case where the position adjustment is performed by two cameras, the position adjustment can be performed with higher accuracy.

[0146] In addition, in the control for correcting the posture of the workpiece 91 relative to the workpiece 81, the posture of the workpiece 91 that is offset around the Y axis of the reference coordinate system 51 can be corrected based on the image captured by the camera 25 and the image captured by the camera 28. In this way, by increasing the number of cameras, the directions for correcting the deviation of the posture can be increased.

[0147] The rest of the structure, function and effect of the fourth robot apparatus are the same as those of the first robot apparatus 5 , the second robot apparatus 8 and the third robot apparatus 9 , and therefore will not be described again here.

[0148] In the first robot device 5, the second robot device 8, the third robot device 9, and the fourth robot device 10, the cameras 25, 26, and 28 are fixed to the hand 2 and move together with the hand 2, but the present invention is not limited to this. The camera may be fixed to a setting surface or the like. In other words, the camera may be fixed so that the position and posture of the camera do not change even if the position and posture of the robot 1 change.

[0149] exist Fig.17, a schematic diagram of the fifth robot device in the present embodiment is shown in FIG. The fifth robot device 6 performs an operation of placing a workpiece 97 as a second member inside a housing 87 as a first member. The housing 87 corresponds to the first workpiece, and the workpiece 97 corresponds to the second workpiece. The fifth robot device 6 includes a hand 7 mounted on the robot 1. The hand 7 is formed to hold the workpiece 97 by suction. The fifth robot device 6 includes a conveyor belt 75 as a conveyor for conveying the housing 87.

[0150] The fifth robot device 6 has a camera 27 as a visual sensor fixed to the installation surface via a bracket 71. The camera 27 remains stationary even if the position and posture of the robot 1 change. The camera 27 is arranged at a position far enough from the conveyor belt 75 so that the workpiece 97 and the housing 87 can be photographed while the robot 1 is adjusting the position of the workpiece 97.

[0151] exist Fig.18 105 shows a side view of the fifth robot device 6 when the workpiece 97 is being accommodated in the housing. The housing 87 is transported by the conveyor belt 75 as shown by the arrow 105. The housing 87 has a plurality of wall portions 87a formed corresponding to the shape of the workpiece 97. The fifth robot device 6 accommodates the workpiece 97 in the area surrounded by the wall portions 87a. Fig.18 In the example shown, the workpiece 97 is aligned with the position of the area surrounded by the wall portion 87 a . Therefore, the workpiece 97 can be accommodated in the housing 87 by lowering the workpiece 97 as shown by the arrow 104 .

[0152] exist Fig.19 Another side view is shown in FIG. 1 when the fifth robot device is accommodating the workpiece into the housing. Fig.19 In the example shown, the position of the workpiece 97 is offset relative to the area surrounded by the wall portion 87a. In the fifth robot device 6, the sixth control is implemented. In the sixth control, the position of the robot 1 is adjusted so that the workpiece 97 is arranged directly above the area surrounded by the wall portion 87a. The posture of the workpiece 97 relative to the housing 87 is adjusted in advance. In the sixth control, the offset of the position of the workpiece 97 relative to the position of the housing 87 is corrected by the same control as the first control. Fig.18 The reference image corresponds to the state where the workpiece 97 is aligned relative to the housing 87 as shown.

[0153] exist Fig. 20 The image captured by the camera is shown in Fig. 20In the image, the position of the workpiece 97 is offset relative to the area surrounded by the wall portion 87a. In the sixth control, for example, the side of the wall portion 87a can be set as the first characteristic portion. The first setting point P1 can be set at the corner of the side of the wall portion 87a. In addition, the side of the workpiece 97 can be set as the second characteristic portion. The second setting point P2 can be set at the corner of the side of the workpiece 97.

[0154] Reference Figure 3 and Fig. 20 The feature quantity detection unit 32 detects a first feature quantity related to the position of the first feature part and a second feature quantity related to the position of the second feature part. The calculation unit 33 calculates the relative position quantity based on the first feature quantity and the second feature quantity. The instruction generation unit 34 can generate a movement instruction for causing the robot 1 to move based on the relative position quantity in the image 65 captured by the camera 27 and the relative position quantity in the reference image.

[0155] In a robot device in which a camera is fixed, the position of the robot can also be controlled by the same control as the first control. In addition, the same control as the second control, the third control, or the fourth control can be performed using a plurality of cameras fixed to an installation surface or the like.

[0156] The rest of the structure, function and effect of the fifth robot apparatus 6 are the same as those of the first robot apparatus 5 , the second robot apparatus 8 , the third robot apparatus 9 and the fourth robot apparatus 10 , and thus will not be described again here.

[0157] The visual sensor in the above-mentioned embodiment is a two-dimensional camera, but it is not limited to this mode. The visual sensor may also be a three-dimensional camera capable of detecting the three-dimensional position of the component contained in the image. By using a three-dimensional camera as a visual sensor, the position of the first component and the position of the second component in the reference coordinate system can be detected even without preparing a base image.

[0158] In the above-mentioned embodiment, a robot device for assembling a product and a robot device for inserting a workpiece into a housing are listed, but the present invention is not limited to this method. The control device of this embodiment can be applied to a robot device for performing any operation. For example, the control device of this embodiment can be applied to a robot device for removing a workpiece from a housing, a robot device for spot welding, or a robot device for applying an adhesive.

[0159] The first member and the second member of the above-mentioned embodiment are workpieces or shells, but are not limited to this mode. The first member can be any member that can be operated by a robot device. In addition, the second member can be any member that is moved by driving a robot. In particular, the second member can also be a working tool installed on the robot. For example, in a robot device that performs spot welding, a spot welding gun is installed on the robot as a working tool. In this case, the second member is a spot welding gun. In the spot welding gun, the second characteristic part, the fourth characteristic part, and the sixth characteristic part can be set as parts that can be detected in the image captured by the camera.

[0160] In the above-mentioned embodiment, the characteristic part is set on the surface of the workpiece or the shell, but it is not limited to this mode. The characteristic part can also be set on the structural member of the robot, the structural member of the hand, or the structural member of the conveyor. For example, in the first control to the fourth control, sometimes there is no gripping offset of the workpiece relative to the hand. Or, sometimes the position of the robot is adjusted corresponding to the gripping offset of the workpiece relative to the hand. In these cases, the second characteristic part, the fourth characteristic part, and the sixth characteristic part can also be set on the hand when the hand is arranged in the camera range of the camera.

[0161] According to the present disclosure, it is possible to provide a control device for a robot device that can control the position of the robot with high accuracy using a simple method.

[0162] In each of the above-mentioned controls, the order of the steps can be appropriately changed within the range where the functions and effects are not changed.

[0163] The above-mentioned embodiments can be combined appropriately. In the above-mentioned figures, the same or equivalent parts are marked with the same symbols. In addition, the above-mentioned embodiments are examples and do not limit the invention. In addition, the embodiments include changes to the embodiments shown in the claims.

Claims

1. A control device for a robot device, wherein the robot moves a second member to adjust the relative position of the second member with respect to a first member, the control device comprising: A visual sensor for photographing the first component and the second component, wherein the visual sensor is mounted on the robot; A motion control unit that sends instructions to the robot for driving the robot; as well as an image processing unit that processes the image captured by the visual sensor, in, predetermining a first characteristic portion for detecting the position of the first member and a second characteristic portion for detecting the position of the second member, The image processing unit comprises: a feature quantity detection unit that detects a first feature quantity related to the position of a first feature part and a second feature quantity related to the position of a second feature part in the image captured by the visual sensor; a calculation unit that calculates a difference between the first feature amount and the second feature amount as a relative position amount; and a command generating unit for generating a movement command for causing the robot to move, wherein the visual sensor pre-captures a reference image, the feature quantity detection unit detects the first feature quantity and the second feature quantity in the reference image, the calculation unit pre-calculates the difference between the first feature quantity and the second feature quantity detected in the reference image as a relative position quantity in the reference image, and the reference image includes images of the first feature part and the second feature part when the second member is configured at a target position relative to the first member, The instruction generation unit generates a movement instruction for causing the robot to move so as to configure the second component to a target position relative to the first component based on the relative position amount in the image captured by the visual sensor and the relative position amount in the reference image including the image of the first characteristic part and the second characteristic part. The motion control unit changes the position of the robot based on the movement instruction. In the image captured by the visual sensor mounted on the robot, the coordinate value of the position of the second characteristic part is fixed, When repeatedly performing control of detecting the first feature quantity and the second feature quantity by the feature quantity detection unit, controlling the detecting the relative position quantity by the calculation unit, and controlling the generating unit to generate a movement command so that the position of the second member relative to the first member approaches the target position, the storage unit stores the coordinate value of the position of the second feature part when the position of the second feature part is detected in the image captured by the visual sensor for the first time, and uses the coordinate value of the position of the second feature part stored in the storage unit to calculate the relative position quantity for images captured by the visual sensor for the second time and thereafter.

2. The control device according to claim 1, wherein: The command generation unit calculates a relative position difference between a relative position in an image captured by the visual sensor and a relative position in a reference image, that is, a relative position difference. The moving direction of the robot position corresponding to the relative position difference is predetermined. A method for calculating the amount of movement of the robot position corresponding to the relative position amount difference is predetermined. The command generation unit sets a movement direction and a movement amount of the robot's position based on the difference in relative position amounts, and generates a movement command based on the movement direction and the movement amount of the robot's position.

3. The control device according to claim 1, wherein: The robot device includes a conveyor for conveying the first component. While the conveyor is conveying the first component, the following controls are repeatedly implemented: control in which the feature quantity detection unit detects the first feature quantity and the second feature quantity; control in which the calculation unit detects the relative position quantity; and control in which the command generation unit generates a movement command.

4. The control device according to claim 2, wherein: The robot device includes a conveyor for conveying the first component. While the conveyor is conveying the first component, the following controls are repeatedly implemented: control in which the feature quantity detection unit detects the first feature quantity and the second feature quantity; control in which the calculation unit detects the relative position quantity; and control in which the command generation unit generates a movement command.

5. The control device according to claim 3, wherein: predetermining the moving direction and moving speed of the first member moved by the conveyor, The command generation unit generates a movement command based on a movement direction and a movement amount of a position of the robot for causing the position of the robot to follow a position of a first member moved by the conveyor.

6. The control device according to claim 4, wherein: predetermining the moving direction and moving speed of the first member moved by the conveyor, The command generation unit generates a movement command based on a movement direction and a movement amount of a position of the robot for causing the position of the robot to follow a position of a first member moved by the conveyor.

7. The control device according to any one of claims 1 to 6, wherein: The visual sensor constitutes a first visual sensor, The control device further includes a second visual sensor for photographing the first component and the second component. A third characteristic portion for detecting the position of the first member and a fourth characteristic portion for detecting the position of the second member are predetermined. The feature quantity detection unit detects a third feature quantity related to the position of a third feature part and a fourth feature quantity related to the position of a fourth feature part in the image captured by the second visual sensor. The calculation unit calculates a difference between the third feature value and the fourth feature value as a relative position value, predetermining a relative position amount in a reference image including an image of a third characteristic portion and a fourth characteristic portion when the second member is arranged at a target position relative to the first member, The instruction generating unit generates a movement instruction for causing the robot to move so as to configure the second component to the target position relative to the first component based on the relative position amount in the image captured by the second visual sensor and the relative position amount in the reference image including the image of the third characteristic part and the fourth characteristic part, and generates a movement instruction to be sent to the action control unit based on the movement instruction generated according to the image of the first visual sensor and the movement instruction generated according to the image of the second visual sensor, In the image captured by the second visual sensor mounted on the robot, the coordinate value of the position of the fourth characteristic part is fixed, In the case of repeating the control of the feature quantity detection unit detecting the third feature quantity and the fourth feature quantity, the control of the calculation unit detecting the relative position quantity as the difference between the third feature quantity and the fourth feature quantity, and the control of the instruction generation unit generating a movement instruction so that the position of the second member relative to the first member approaches the target position, the storage unit stores the coordinate value of the position of the fourth feature part when the position of the fourth feature part is detected in the image captured by the second visual sensor for the first time, and uses the coordinate value of the position of the fourth feature part stored in the storage unit in order to calculate the relative position quantity as the difference between the third feature quantity and the fourth feature quantity for the images captured by the second visual sensor after the second time.

8. The control device according to claim 7, wherein: The first visual sensor is configured to capture a portion of the second member that is disposed on one side in a predetermined direction. The second visual sensor is configured to capture an image of a portion of the second member that is disposed on the other side in a predetermined direction. Based on the movement command generated based on the image of the first visual sensor and the movement command generated based on the image of the second visual sensor, it is determined whether the posture of the second member relative to the first member is within a predetermined determination range.

Images