Robot control method, system, article manufacturing method, program, and recording medium

Abstract

To provide a technology that is advantageous in positioning two objects in an appropriate target positional relationship in robot control.SOLUTION: A computer-implemented method for controlling a robot includes acquiring a current image showing a positional relationship between a first object held by the robot and a second object, determining whether the first object and the second object are in a reference positional relationship based on the current image, and controlling, when determined that the first object and the second object are in a reference positional relationship, the position and posture of the robot according to a trajectory of the robot set before the determination or a trajectory of the robot instructed after the acquisition of the current image such that the first object and the second object have a target positional relationship.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a robot control method and the like. [Background technology]

[0002] Robots are increasingly being introduced in various fields, including manufacturing sites. Visual servoing, which measures the change in the position of a target object as visual information and uses it as feedback information, is known as a method for controlling the position and posture of a robot.

[0003] Patent Document 1 discloses a robot control device that performs control by adding together the output of visual servo control based on visual information and the output of force control based on force detection results in different ratios depending on the distance from a target position.

[0004] Patent Document 2 discloses a control device that incorporates a force control system into a visual servo system and controls a robot based on both information about the load applied to an object and information about the positional relationship between the object and an attached part. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2015-74058 A [Patent Document 2] JP 2013-180380 A Summary of the Invention [Problem to be solved by the invention]

[0006] For example, when a robot hand is made to grasp part 1 and assemble part 1 to part 2, there are cases where the assembly work is not performed correctly if the robot is controlled by a conventional control method. Specifically, if an unintended deviation occurs in the relative positions of parts 1 and 2 while the robot hand is grasping and moving part 1, there are cases where the assembly is not performed correctly even if the robot hand is moved according to a taught trajectory.

[0007] Therefore, there was a need for a technology that would be advantageous in controlling robots to bring two objects into an appropriate target positional relationship. [Means for solving the problem]

[0008] A first aspect of the present invention is a method for controlling a robot, characterized in that a computer acquires a current image reflecting a positional relationship between a first object and a second object held by a robot, and if it determines based on the current image that the first object and the second object are in a reference positional relationship, controls the position and posture of the robot in accordance with an action of the robot set before the determination or an action of the robot instructed by a user after the current image is acquired, so that the first object and the second object are in a target positional relationship.

[0009] In addition, a second aspect of the present invention is a method for controlling a robot, comprising the steps of: setting a position and posture of a mechanical device so that a first teaching object and a second teaching object held by the mechanical device are in a target positional relationship; changing a position and posture of the mechanical device holding the first teaching object so that the first teaching object and the second teaching object are in a reference positional relationship from the target positional relationship; acquiring a reference image reflecting that the first teaching object and the second teaching object are in the reference positional relationship; acquiring a current image reflecting the positional relationship between the first object and the second object held by a robot; and, if a computer determines based on the reference image and the current image that the first object and the second object are in the reference positional relationship, controlling a position and posture of the robot holding the first object so that the first object and the second object are in the target positional relationship. Effect of the Invention

[0010] According to the present invention, since it is determined that two objects are in a reference positional relationship based on a current image, it is possible to provide a technique that is advantageous in bringing two objects into an appropriate target positional relationship. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram for explaining the configuration of a robot system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a control system provided in the robot according to the embodiment. [Diagram 3] FIG. 1 is a schematic diagram showing a working environment of a robot. [Figure 4] (a) A diagram showing the position and posture of the first workpiece W1 held by the robot hand in the initial stage of the assembly work, (b) A diagram showing the position and posture of the first workpiece W1 held by the robot hand at the approach position, and (c) A diagram showing the state after the operation of assembling the first workpiece W1 to the second workpiece W2 is completed. [Diagram 5] 4 is a flowchart showing a procedure of an assembly operation according to the embodiment. [Figure 6] 4A to 4C are diagrams showing an example of image features used to extract a target image IG and a target image feature amount Fg according to the embodiment. [Figure 7] (a) A diagram showing the positional relationship between the first workpiece W1 and the second workpiece W2 before teaching. (b) A diagram showing the positional relationship (target positional relationship) between the first workpiece W1 and the second workpiece W2 when the operation of assembling the first workpiece W1 to the second workpiece W2 is completed. (c) A diagram showing the positional relationship (reference positional relationship) between the first workpiece W1 and the second workpiece W2 when the operator manually pulls the first workpiece W1 out of the second workpiece W2. [Figure 8] 11 is a flowchart for explaining a procedure for teaching image features as visual feedback teaching data. [Figure 9] 4 is a diagram for explaining an imaging area A1 of the visual sensor when acquiring a reference image. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] A robot control method and the like according to an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example, and those skilled in the art can appropriately modify the detailed configuration without departing from the spirit of the present invention. When describing the control of the robot's operation and position and orientation, the content may include an operation to stop the robot and control to maintain the robot's position and orientation.

[0013] In the drawings referred to in the following description of the embodiments and examples, elements denoted by the same reference numerals have the same functions unless otherwise noted. When a plurality of identical elements are arranged in a drawing, the reference numerals and their descriptions may be omitted.

[0014] In addition, since the drawings may be represented diagrammatically for the convenience of illustration and explanation, the shape, size, arrangement, etc. of the elements depicted in the drawings may not strictly correspond to the actual objects.

[0015] [Embodiment 1] As the first embodiment, a control method for making a robot execute an operation of assembling a first workpiece to a second workpiece is exemplified. In the assembling operation, the first workpiece is held by the robot and moved toward the second workpiece, and the first workpiece is moved to a position where the first workpiece and the second workpiece have a reference positional relationship, that is, an approach position. In this specification, the "positional relationship" refers to a relative relationship including not only the position but also the posture. The approach position is set in advance as a positional relationship (reference positional relationship) from which the robot can assemble the held first workpiece to the second workpiece with high reliability. The operation of the robot to move the first workpiece from the approach position to the assembling completion position is taught to the control unit of the robot in advance.

[0016] In the assembly work, the control unit uses the imaging unit to obtain a current image of at least the first workpiece, and determines whether the first workpiece and the second workpiece are in a reference positional relationship. If it is determined that they are in the reference positional relationship, the control unit controls the position and posture of the robot according to a control program taught in advance, and moves the first workpiece to the assembly completion position (target positional relationship).

[0017] In order to determine whether the first workpiece and the second workpiece are in a reference positional relationship, the control unit acquires a target image (reference image) in advance as visual feedback teaching data. The reference image is an image in which the first teaching workpiece and the second teaching workpiece are arranged so as to be in a reference positional relationship, and the image is captured using the imaging unit so as to include at least the first teaching workpiece. Except for manufacturing errors, the first teaching workpiece has a substantially identical shape to the first workpiece, and the second teaching workpiece has a substantially identical shape to the second workpiece. The first teaching workpiece may be called the first teaching object, and the second teaching workpiece may be called the second teaching object. That is, in the embodiment, a reference image reflecting that the first teaching object and the second teaching object are in a reference positional relationship is acquired in advance.

[0018] By comparing the current image with the reference image, the control unit can accurately determine whether the first workpiece is in the approach position, i.e., whether the first workpiece and the second workpiece are in a reference positional relationship.

[0019] In conventional methods, during assembly work, whether or not the robot hand is in a predetermined position and posture to start the approach operation is determined using the position sensor of the robot hand or an image taken of the robot hand. Even if it can be determined that the robot hand is in the predetermined position and posture, if a so-called gripping misalignment occurs, the first workpiece will no longer be in the predetermined position and posture with respect to the second workpiece. Grip misalignment refers to the phenomenon in which the first workpiece shifts relative to the robot hand when or after the robot hand grasps the first workpiece. If the assembly work is continued in a state in which a gripping misalignment occurs, not only will the assembly not be performed correctly, but the workpieces may also collide with each other and be damaged.

[0020] In the control according to the present embodiment, a control unit that controls a robot holding a first workpiece acquires a current image that can determine the positional relationship between the first workpiece and the second workpiece, and determines whether or not the first workpiece and the second workpiece are in a preset reference positional relationship based on the current image. The reference positional relationship is set in advance before making the determination, for example, before acquiring the current image. The reference positional relationship can be set when capturing a target image. If it is determined that the first workpiece and the second workpiece are in the reference positional relationship (approach position), the position and posture of the robot are controlled so that the first workpiece and the second workpiece are in the target positional relationship (assembly completion state). Thus, in the present embodiment, the robot is made to perform the final assembly operation after confirming with an image that the first workpiece held from that position is in a positional relationship (i.e., a reference positional relationship) that allows the robot to properly assemble the second workpiece to the first workpiece from that position. Therefore, when a so-called gripping deviation occurs, there is no concern that the final assembly operation will continue unless it is confirmed that the first workpiece and the second workpiece are in the reference positional relationship. Furthermore, even if a gripping error occurs, if it is confirmed that the first workpiece and the second workpiece are in a reference positional relationship, the final assembly operation can be performed, for example, by force control.

[0021] In this embodiment, there are no particular limitations on the method of controlling the robot to move the first workpiece to the approach position. For example, the control unit may move the robot hand according to a pre-taught trajectory, or the user may move the robot hand using a pendant.

[0022] In addition, at least the first teaching workpiece is imaged in the target image (reference image). For example, it is sufficient that both the first teaching workpiece and the second teaching workpiece in the reference positional relationship are imaged in the reference image. The control unit can make a determination by comparing the positional relationship between the first teaching workpiece and the second teaching workpiece imaged in the current image with the positional relationship between the first teaching workpiece and the second teaching workpiece imaged in the target image (reference image).

[0023] Furthermore, when there is a position reference object (for example, a jig for fixing the second work) whose positional relationship is fixed with respect to the second work, the first teaching work and the position reference object may be captured in the target image (reference image). The control unit can make a determination by comparing the positional relationship between the first teaching work and the position reference object captured in the current image with the positional relationship between the first teaching work and the position reference object captured in the reference image.

[0024] The robot operation for moving the first workpiece, which has been confirmed to be in the reference positional relationship based on the current image, from the approach position to the assembly completion position is preferably performed by force control. However, if there is no problem in moving the first workpiece to the assembly completion position by position control, it may be performed by position control. For example, if a wide clearance is ensured between the first workpiece and other objects (including the second workpiece) on the path along which the first workpiece is moved to the assembly completion position, and the possibility of interference is extremely small, it may be moved by position control. Alternatively, if the first workpiece or the second workpiece is made of a flexible material and there is no problem even if slight contact occurs due to a control error, it may be moved by position control. Furthermore, if a compliance mechanism such as an RCC (Remote Center Compliance) device is provided on the robot side, it may be moved by position control. The specific details will be explained below.

[0025] (Robot system configuration) 1 is a schematic diagram for explaining the configuration of a robot system 1000 according to embodiment 1. The robot system 1000 includes a robot 100, a servo control unit 230, a control device 400, an input device 500, a display 600, and a visual sensor 800. A control system 440 is configured with the control device 400 at the center.

[0026] 2 is a block diagram showing the configuration of a control system 440 included in the robot system 1000 according to the embodiment. Each unit of the robot system 1000 will be described with reference to FIGS.

[0027] (Control unit) The control device 400 serving as a control unit is a computer that controls the entire robot system 1000 and executes various operations such as assembly work, etc. The control device 400 has a CPU 401 (Central Processing Unit) serving as a processor.

[0028] The control device 400 has, as a storage unit, for example, a read only memory (ROM) 402, a random access memory (RAM) 403, and a hard disk drive (HDD) 404. The control device 400 also has a recording disk drive 405 and interfaces 406 to 409 as a plurality of input / output interfaces (I / F).

[0029] The CPU 401 is connected to the ROM 402, the RAM 403, the HDD 404, the recording disk drive 405, and the interfaces 406 to 409 via a bus 420. The ROM 402 stores basic programs such as the BIOS. The RAM 403 is a storage device that temporarily stores various data such as the results of calculations performed by the CPU 401.

[0030] The HDD 404 is a non-transitory computer-readable recording medium that stores the processing program executed by the CPU 401, the arithmetic processing results of the CPU 401, various data acquired from the outside, etc. The HDD 404 stores a program 430 for causing the CPU 401 to execute arithmetic processing, and visual feedback teaching data 432. The CPU 401 executes each process of the article manufacturing method based on the program 430 recorded (stored) in the HDD 404. Each process may include an image processing method.

[0031] In this embodiment, the program 430 and the visual feedback teaching data 432 are stored in the HDD 404, but may be stored in other recording media. That is, the program 430 and the visual feedback teaching data 432 may be recorded in any recording media as long as the recording media is a non-transient recording medium readable by a computer. For example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, or a non-volatile memory may be used. The optical disk is, for example, a disk medium such as a Blu-ray disk, a DVD, or a CD. The non-volatile memory is, for example, a storage device such as a USB memory, a memory card, a ROM, or an SSD. Various data, programs, etc. recorded in the recording disk 431 can be read by the recording disk drive 405.

[0032] An input device 500 is connected to the interface 406 of the control device 400. The CPU 401 acquires input data (input information) from the input device 500 via the interface 406 and the bus 420. The input device 500 is a device that allows a user to operate it to input various information to the control device 400. If a teaching pendant, for example, is used as the input device 500, the operator (user) can use it to teach the robot arm 200 how to operate.

[0033] A display 600, which is an example of a display unit, is connected to the interface 407 of the control device 400. The display 600 is capable of displaying various images output by the control device 400.

[0034] An external storage device 700, which is a storage unit such as a rewritable non-volatile memory or an external HDD, can be connected to the interface 408 of the control device 400.

[0035] The interface 409 of the control device 400 is connected to a servo control unit 230 of the robot 100. The servo control unit 230 will be described later.

[0036] A visual sensor 800 serving as an imaging unit is connected to the interface 410 of the control device 400. The visual sensor 800 is an example of an imaging unit that captures an image and transmits it to the control device 400, and is, for example, a digital camera. The visual sensor 800 is a two-dimensional camera that can capture an image of a subject and obtain two-dimensional image information. Note that the visual sensor 800 is not limited to a two-dimensional camera, and may be, for example, a three-dimensional camera. The visual sensor 800 captures images at a predetermined time interval (for example, every 30 ms) under the control of the CPU 401. This allows the CPU 401 to obtain visual information, i.e., captured image data, from the visual sensor 800 at a predetermined time interval (for example, every 30 ms).

[0037] In this embodiment, the input device 500, the display 600, the visual sensor 800, the control device 400, and the servo control unit 230 constitute a control system 440.

[0038] The CPU 401 of the control device 400 can acquire angle information from the angle sensor 250 of each joint via the servo control unit 230, the interface 409, and the bus 420. Similarly, the CPU 401 can acquire force information from the force sensor 260 of each joint. The force sensor 260 may be a torque sensor that detects torque, which is one type of force information, a force sensor that detects force, which is one type of force information, or a sensor that detects both torque and force. The servo control unit 230 may divide the angle of the motor 231 detected using the angle sensor 250 by the reduction ratio of a reducer (not shown), convert it into angle information of the corresponding joint, and transmit it to the CPU 401.

[0039] The CPU 401 of the control device 400 outputs command value data corresponding to each of the joints J1 to J6 to the servo control unit 230 via the bus 420 and the interface 409 at a predetermined time interval (for example, 1 ms). The servo control unit 230 controls the drive of the motor 231 of each joint so that the angle or torque of the joints J1 to J6 follows the command value based on the command value corresponding to each of the joints J1 to J6 acquired from the control device 400. In other words, the servo control unit 230 is configured to be able to control the position or torque of each joint of the robot 100.

[0040] (robot) The robot 100 (FIG. 1) is, for example, an industrial robot, and has a robot arm 200 and a robot hand 300. In this embodiment, the robot 100 equipped with the vertically articulated robot arm 200 is illustrated as an example, but the robot is not limited to this. Any robot may be used as long as it is controlled by a control device and has a mechanism that can automatically perform, for example, movements of extension and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these movements.

[0041] The robot 100 is installed, for example, in a manufacturing line and is used to perform various operations for manufacturing an article. The operations for manufacturing an article may include, for example, a conveying operation, an assembly operation, a processing operation, or a coating operation. The processing operation may include, for example, a cutting operation, a grinding operation, a polishing operation, or a sealing operation.

[0042] An end effector according to the work content is attached to the robot arm 200. A link 216 at the tip of the robot arm 200 is a support configured to mount and support the end effector. In this embodiment, a control method for an assembly work will be described using an example in which a robot hand 300 is mounted on the robot arm 200 as the end effector. As will be described later with reference to FIG. 3, in one example of the assembly work, the robot hand 300 is caused to grip a first workpiece W1, and the robot arm 200 is operated to assemble the first workpiece W1 to a second workpiece W2.

[0043] The robot arm 200 has a base 209, links 210 to 216, and joints J1 to J6. The base 209, which is the base end (fixed end), is installed on the upper surface of the pedestal B1. The links 210 to 216 are connected in series via joints J1 to J6 in this order. Each of the joints J1 to J6 has a motor 231, an angle sensor 250, and a force sensor 260. For convenience of illustration, FIG. 2 shows the configuration of only one joint as a component of the robot arm 200. The link 210, which is the base end of the robot arm 200, is fixed to the base 209. Each of the links 211 to 216 is rotationally driven by the joints J1 to J6 that connect them. This allows the robot arm 200 to adjust the position and posture of the robot hand 300 to any position in three axial directions and any posture in three axial directions.

[0044] The position and orientation of the robot arm 200 can be expressed by a coordinate system. The coordinate system T0 in FIG. 1 is a coordinate system set on the base B1 to which the robot arm 200 is fixed. eis a coordinate system set in the robot hand 300. e represents TCP (Tool Center Position). For example, in the coordinate system T e is set in the robot hand 300. e is expressed in three-axis orthogonal coordinates consisting of the X, Y and Z axes.

[0045] Coordinate system T c is a coordinate system set at the center of the visual sensor 800, and the coordinate system T o and coordinate system T e Similarly, it is expressed in three-axis orthogonal coordinates consisting of X, Y and Z axes, and is set so that the optical axis direction of the visual sensor 800 is the Z-axis direction. Note that in this embodiment, the visual sensor 800 is described as being fixed to a predetermined position based on the coordinate system T0, for example, to a pedestal B1, but it may also be fixed to the robot arm 200 or the robot hand 300.

[0046] The servo control unit 230 controls the driving of the motor 231 included in each of the joints J1 to J6. The servo control unit 230 is electrically connected to the motor 231, angle sensor 250, and force sensor 260 included in each of the joints J1 to J6. The motor 231 is, for example, a brushless DC motor or an AC motor, and rotates the joint via a reduction gear (not shown). The angle sensor 250 is, for example, a rotary encoder, and is provided in association with the motor 231 and configured to be able to detect the rotation angle of the motor 231. The force sensor 260 is configured to be able to detect the torque acting on the joint.

[0047] The servo control unit 230 is disposed, for example, inside the base 209, but may be disposed in another position. For example, the servo control unit 230 may be disposed inside the housing of the control device 400. That is, the servo control unit 230 may be a part of the configuration of the control device 400.

[0048] (work) Next, an operation performed by the robot 100 in this embodiment will be described. Fig. 3 is a schematic diagram showing the working environment of the robot 100. In this embodiment, the control method will be described by taking as an example an operation of assembling a first workpiece W1 held by the robot hand 300 to a second workpiece W2 fixed to a workpiece fixing jig M1.

[0049] 3, the imaging range captured by the visual sensor 800 is illustrated as an imaging area A1 surrounded by a dashed line. The imaging area A1 is set so that at least the first workpiece W1 held by the robot hand 300 is included in the angle of view when the robot 100 is in the approach position, but specifically, it can be set in several ways.

[0050] For example, the imaging area A1 can be set so that both the first workpiece W1 held by the robot hand 300 at the approach position and the second workpiece W2 fixed to the workpiece fixing jig M1 are included in the angle of view. When set in this way, it is possible to extract the first workpiece W1 and the second workpiece W2 from the captured image by image processing and calculate the relative positional relationship between them.

[0051] Furthermore, when the positional relationship between the second workpiece W2 and the workpiece fixing jig M1 is known in advance, the imaging area A1 can be set so that both the first workpiece W1 held by the robot hand 300 at the approach position and the workpiece fixing jig M1 are included in the angle of view. When set in this way, the first workpiece W1 and the workpiece fixing jig M1 can be extracted from the captured image by image processing to determine the relative positional relationship between them, and the relative positional relationship between the first workpiece W1 and the second workpiece W2 can be calculated.

[0052] In addition, the imaging area A1 can be set so that both the first workpiece W1 and the second workpiece W2 are always included in the angle of view from the start of the assembly work, not just when the robot hand 300 is at the approach position. This is suitable for moving the first workpiece W1 to the approach position by visual servoing.

[0053] The visual sensor 800 does not necessarily have to be installed at the position illustrated in FIG. 3, as long as it can acquire an image for determining the positional relationship between the first workpiece W1 and the second workpiece W2. The visual sensor 800 may be arranged so as to move with changes in the position and posture of the robot. For example, it may be installed on the link at the tip side of the robot arm or on the robot hand 300. When the visual sensor 800 is used as an on-hand camera, the first workpiece W1 held by the robot hand 300 moves together with the visual sensor 800, so that the first workpiece W1 may be immobile within the visual field of the visual sensor 800. When the visual sensor 800 is used as a global camera, the visual sensor 800 is fixed at the installation location of the robot arm, and the robot hand 300 and the first workpiece W1 move relative to the visual sensor 800, so that the first workpiece W1 may be movable within the visual field of the visual sensor 800. To more accurately determine the positional relationship between the first workpiece W1 and the second workpiece W2, an on-hand camera is more suitable than a global camera.

[0054] 4(a) to 4(c) are schematic diagrams for specifically explaining the assembly work exemplified in this embodiment. FIG. 4(a) shows the position and posture of the first work W1 held by the robot hand 300 in the initial stage of the assembly work. FIG. 4(b) shows the position and posture of the first work W1 held by the robot hand 300 at the approach position. In this example, the approach position is set as a position where the assembly work to the second work W2 is completed if the first work W1 is translated by a predetermined distance. The positional relationship between the first work W1 and the second work W2 when the first work W1 is at the approach position can be called the reference positional relationship. FIG. 4(c) shows a state where the operation of assembling the first work W1 to the second work W2 is completed. The positional relationship between the first work W1 and the second work W2 when the assembly is completed can be called the target positional relationship. Normally, the first workpiece W1 and the second workpiece W2 are separated from each other in the reference positional relationship, and the first workpiece W1 and the second workpiece W2 are in contact with each other in the target positional relationship.

[0055] In this embodiment, the control device 400 uses visual feedback to control the robot arm 200 to move it to the approach position, and uses an image captured by the visual sensor 800 to determine whether the first workpiece W1 and the second workpiece W2 are in the positional relationship shown in FIG. 4(b). Here, the visual feedback is a method of feeding back and controlling the position command value to the robot arm 200 so that the image feature value acquired by the visual sensor 800 becomes a desired value. In the embodiment, a case where so-called visual servoing is used in which the image feature value is gradually made to follow a desired value is described, but this is not limited to this. For example, instead of gradually following, an approach trajectory may be generated in advance based on the visual information of the visual sensor 800, and the robot arm 200 may be driven along the trajectory. When the control device 400 determines that the first workpiece W1 and the second workpiece W2 are in the positional relationship shown in FIG. 4(b), the control device 400 sends an operation command to the servo control unit 230 to translate the robot hand 300 by a predetermined distance by force control according to a control program that has been taught in advance.

[0056] 5 is a flowchart showing the procedure of an assembly operation according to the embodiment. The CPU 401 executes the program 430 to perform the processes shown in the flowchart of FIG.

[0057] In step S11, the CPU 401 acquires the visual feedback teaching data 432 recorded in the HDD 404. Here, the visual feedback teaching data 432 is, for example, a target image IG (reference image) captured by the visual sensor 800 when the first workpiece W1 and the second workpiece W2 are in the state shown in FIG. 4(b). The target image IG is temporarily copied to, for example, the RAM 403. The target image IG may be captured using the first workpiece W1 and the second workpiece W2 to be used later in the actual assembly work, but as long as it is possible to recognize that the two objects have a target positional relationship, an object other than the first workpiece W1 and the second workpiece W2 to be used in the actual assembly work can be used as the teaching object. For example, a first teaching workpiece and a second teaching workpiece that are substantially similar or identical in shape to the first workpiece W1 and the second workpiece W2 can be captured. Similarly, a teaching robot arm used during teaching can be a mechanical device other than the robot arm used in the actual assembly work. The teaching robot arm can be a mechanical device of the same model as the working robot arm. However, taking into consideration individual differences between robot arms, it is preferable to teach the robot arm and obtain the target image IG using the robot arm that will actually be used in the assembly work.

[0058] In step S12, the CPU 401 calculates the target image feature amount F g FIG. 6 shows a target image IG and a target image feature amount F g1 is an example of an image feature used for extracting the edge f111 and edge f121 of the first work W1 and the second work W2 that are closest to each other, and the predetermined portion f112 and portion f122 of the first work W1 and the second work W2 from the target image IG. The predetermined portion is, for example, the end points of the edge f111 and the edge f121 of the first work W1 and the second work W2, respectively. The edge f111, the edge f121, the portion f112, and the portion f122 can be extracted by a known method. For example, the edge f111 and the edge f121 can be extracted by a line segment detection method using a Hough transform. The portion f112 and the portion f122 may be extracted based on the extraction result of the edge f111 and the edge f121.

[0059] More preferably, it is desirable to extract the edge f111 and the edge f121 and the part f112 and the part f122 by different methods. For example, a line segment detection method using a Hough transform can be used to extract the edge f111 and the edge f121, and template matching can be used to extract the part f112 and the part f122. In this case, even if an erroneous image feature is extracted due to a failure in image processing, the error can be detected by comparing the extraction results of the edge f111 and the part f112, or the extraction results of the edge f121 and the part f122. When template matching is used to extract the part f112 and the part f122, in step S12, values ​​input by the operator or values ​​recorded in the HDD 404 or the like are registered as the positions and sizes of the part f112 and the part f122. Then, the corresponding range of the target image IG may be registered as a template. The registered template is temporarily recorded in the RAM 403 or the like. In the coordinate system T c The target image feature F′ is a coordinate system on the target image IG, with the perpendicular bisector of the edge f111 as the X′ axis and the direction along the edge f111 as the Y′ axis. g is the coordinate system T c For example, the target image feature F gis composed of three values: a distance f131 between the intersection of the edge f111 and the edge f121 with the X' axis, a difference f132 between the positions of the predetermined portion f112 and the portion f122 in the Y' axis direction, and an angle f133 between the edge f111 and the edge f121. g is expressed as a three-dimensional vector consisting of distance f131, difference f132, and angle f133. g =[f131 f132 f133] T The subscript T represents the transpose of a vector or matrix.

[0060] Next, when the actual assembly work is started, in step S13, the CPU 401 acquires a current image IC by having the visual sensor 800 capture an image. The current image IC is temporarily stored in the RAM 403, for example.

[0061] In step S14, the CPU 401 extracts the target image feature quantity F from the target image IG in step S12. g The current image feature F is extracted from the current image IC in the same way. c The target image feature F g The components of fg131, fg132, fg133, and the current image feature F c The components of F are denoted as fc131, fc132, and fc133, respectively. g =[fg131 fg132 fg133] T , F c =[fc131 fc132 fc133] T At this time, the extracted current image feature FC may be displayed on the display 600.

[0062] In step S15, the CPU 401 calculates the target image feature F g and the current image feature F c The control amount q of the robot's joints J1 to J6 is v The control variable q is calculated and the robot is operated. v To calculate the current image feature F c and the target image feature F g The feature difference Fe (F is in bold) is calculated according to the following formula (1).

[0063]

number

[0064] Next, CPU 401 calculates the image Jacobian J img (J is in bold) matrix and the robot Jacobian J r (J is in bold) Calculate the image Jacobian J img is the coordinate system T set in the robot hand 300. e The small displacement of and the current image feature F c The robot Jacobian J is a 3-by-3 matrix that relates the small displacement of r is a small displacement amount of the joints J1 to J6 of the robot arm 200 and a coordinate system T e is a 3-by-6 matrix that relates the small displacement of and . The image Jacobian J is calculated according to the following equation (2): img and the robot Jacobian J r Define.

[0065]

number

[0066] where x e (x in bold) is in coordinate system T o In the coordinate system T e The three-degree-of-freedom position vector x e =[X e Y e α e ] T and α e is the coordinate system T o is the rotation angle around the Z axis. q (q is in bold) is the joint angle vector q=[q1...q6] of each joint J1 to J6 of the robot arm 200. T It is. Next, the CPU 401 calculates the control amount q of each joint J1 to J6 of the robot. v (q is in bold) is calculated. Control quantity qv is calculated, for example, according to the following formula (3).

[0067]

number

[0068] In equation (3), λ (λ is in bold) is a feedback gain expressed as a three-dimensional vector, and the subscript -1 indicates an inverse matrix, and + indicates a pseudo-inverse matrix. Note that the feature difference F e to the control quantity q v The method of calculating the control amount q is not limited to the above-mentioned method, and other known methods may be used freely. v Then, the CPU 401 performs visual feedback operation by operating the robot arm 200 via the servo control unit 230 based on the angle command value.

[0069] In step S16, the CPU 401 determines whether the position correction operation by visual feedback is completed and the reference position relationship is achieved. e If it is equal to or smaller than the predetermined value, it is determined that the reference positional relationship is established, and the process proceeds to step S17, whereas if it is greater than the predetermined value, the process returns to step S13 and performs visual feedback operation again. Here, the predetermined value is a value that allows a determination to be made with sufficient accuracy to appropriately perform the subsequent assembly work under force control, and the feature amount difference F e The same value may be used for all values ​​included in the condition, or different values ​​may be used for each determination, and the correction operation may be continued until all values ​​satisfy the condition.

[0070] In step S17, the CPU 401 operates the robot arm 200 using force control, and performs an assembly operation so that the first workpiece W1 follows the second workpiece W2 and achieves a target positional relationship. The control device 400 is set in advance by teaching the amount of movement of the robot hand 300 for transitioning from the state of FIG. 4(b) to the state of FIG. 4(c) and the amount of movement of the robot hand 300 with respect to the hand reaction force. Here, the hand reaction force is an external force generated in the robot hand 300 by contact between the first workpiece W1 and the second workpiece W2, and is obtained by converting the detection value of the force sensor 260 of each joint into the hand reaction force. The CPU 401 operates the robot arm 200 until the hand reaction force becomes a taught value, or the amount of movement of the robot hand 300 becomes a taught value, or both are satisfied. The operation performed by the robot arm 200 in step S17 is not limited to an operation previously set before performing step S16, i.e., an operation previously set by teaching, but may be an operation instructed by a user after acquiring a current image in step S13, or an operation following a trajectory instructed by a user after acquiring a current image. The operation or trajectory instructed by a user after acquiring a current image in step S13 can be set by the user (operator) manually operating the robot arm 200 via the input device 500 (so-called JOG operation).

[0071] There are two types of force control: hybrid control, which completely separates position and force and controls them individually, and impedance control, which linearly relates position and force to control them. Impedance control is a method in which the mechanical impedance characteristics of the robot's hand are set to an appropriate value in advance, and the force applied by the hand to an object is indirectly controlled through adjustment of the hand target position. Hybrid control is a method in which the position control direction of the hand and the force control direction are clearly distinguished, and a force control law is used for the force control direction. In impedance control, both the elastic term and the viscous term can be added to the calculation of the target value, but the viscous term can be ignored and only the elastic term can be used, in which case it can be called compliance control. Compliance control is treated as a type of impedance control. In this embodiment, the force control may be either hybrid control or impedance control, but it is preferable to use impedance control for the operation from the reference positional relationship to the target positional relationship. In step S17, the first work W1 is made to follow the second work W2 by force control, and the assembly operation is performed, but it may be performed by position control depending on the situation. For example, if there is a sufficient gap between the workpieces when they are assembled, if the material is soft, or if a compliance mechanism is provided on the robot side, assembly can be performed using position control. In step S17, the position and orientation of the first workpiece W1 may be maintained, and the second workpiece W2 may be moved to follow the first workpiece W1, so that the first workpiece W1 and the second workpiece W2 have a target positional relationship.

[0072] As described above, according to this embodiment, visual feedback control is performed using the distance f131, the difference f132, and the angle f133, which are image features determined by the positional relationship between the first workpiece W1 and the second workpiece W2. According to this embodiment, even if the positional relationship between the first workpiece W1 and the robot hand 300 is shifted due to a gripping error or the like, it is confirmed from the image whether the relative positions of the first workpiece W1 and the second workpiece W2 are in a predetermined reference positional relationship, and whether or not the final operation can be performed is determined.

[0073] Furthermore, by setting the approach position at a position immediately before the first workpiece W1 and the second workpiece W2 come into contact with each other and switching from visual feedback control to force control at the approach position, it is possible to prevent the visual feedback control from becoming unstable due to occlusion, etc. Furthermore, by minimizing the amount of movement of the robot hand 300 under force control, it is possible to complete the assembly operation without compromising accuracy, thereby improving the work performance of the robot.

[0074] [Embodiment 2] As the second embodiment, a modified example of a control method for causing a robot to perform an operation of assembling a first workpiece to a second workpiece will be described. Explanations of matters common to the first embodiment will be simplified or omitted. This embodiment is characterized by a method for setting an approach position and a method for acquiring a reference image (and image feature amount) as visual feedback teaching data.

[0075] As already described, when the control unit determines that the first workpiece W1 and the second workpiece W2 are in a preset reference positional relationship (i.e., the first workpiece W1 is in a predetermined approach position), it can cause the robot to execute the final movement operation in the assembly operation. In order to perform precise assembly, it is necessary to set the approach position appropriately so that the first workpiece W1 can easily enter the opening of the second workpiece W2 in the final movement operation. In this embodiment, the approach position is set by the method described below, a reference image is captured, and the control device 400 acquires visual feedback teaching data. That is, the steps shown as steps S11 and S12 in the flowchart of FIG. 5 of the first embodiment are performed in the procedure described below in this embodiment.

[0076] 7(a) to 7(c) are explanatory diagrams for explaining the procedure for acquiring visual feedback teaching data in this embodiment, and FIG. 8 is a flowchart for explaining the procedure for teaching image features as visual feedback teaching data.

[0077] 7A shows the positional relationship between the first workpiece W1 and the second workpiece W2 before teaching. The robot hand 300 is holding the first workpiece W1, but the first workpiece W1 is located away from the second workpiece W2.

[0078] In step S21, the worker manually operates the robot arm 200 (so-called JOG operation) via the input device 500 to move the first workpiece W1 from the position shown in FIG. 7(a) and assemble it to the second workpiece W2, completing the assembly. That is, the first workpiece W1 and the second workpiece W2 are arranged in a target positional relationship. FIG. 7(b) is a diagram showing the positional relationship (target positional relationship) between the first workpiece W1 and the second workpiece W2 when the worker completes the assembly of the first workpiece W1 to the second workpiece W2 by manual operation. The manual operation of the robot arm 200 is an operation to move the robot arm 200 in each of the X-axis translation direction, Y-axis translation direction, Z-axis translation direction, X-axis rotation direction, Y-axis rotation direction, and Z-axis rotation direction of the coordinate system Te set in the robot hand 300. The worker adjusts the positions in these directions or the rotation angles using the input device 500, and assembles the first workpiece W1 to the second workpiece W2.

[0079] Next, in step S22, the worker manually operates the robot arm 200 to offset the first workpiece W1 in a direction opposite to the assembly direction in the previous step, and pulls it out from the second workpiece W2. That is, the first workpiece W1 is pulled out from the second workpiece W2 by moving it linearly in the negative direction of the Z axis of the coordinate system Te. FIG. 7(c) is a diagram showing the positional relationship (reference positional relationship) between the first workpiece W1 and the second workpiece W2 when the worker manually pulls out the first workpiece W1 from the second workpiece W2. At this time, the first workpiece W1 and the second workpiece W2 are in a positional relationship in which the workpieces can be assembled by moving the first workpiece W1 linearly in the Z axis direction of the coordinate system Te. In other words, the position of the first workpiece W1 shown in FIG. 7(c) is set as the approach position, and the assembly operation is performed by the force control described above, so that the workpieces can be assembled appropriately. In this embodiment as well, the force control may be either hybrid control or impedance control, but it is preferable to use impedance control for the movement from the reference positional relationship to the target positional relationship.

[0080] FIG. 9 is a diagram for explaining the imaging area A1 of the visual sensor 800 when acquiring a reference image as visual feedback teaching data. The imaging area A1 shown by a dashed line in FIG. 9 represents the field of view of the visual sensor 800, where E1 is the end on the tip side of the first work W1 and E2 is the end on the opening side of the second work W2. In step S22, the offset amount when the first work W1 is pulled out from the second work W2 is set to a range in which the end E1 on the tip side of the first work W1 and the end E2 on the opening side of the second work W2 are within the imaging area A1. Note that the imaging area A1 may change depending on the position, attitude, angle of view, etc. of the visual sensor 800 as long as the first work W1 and the second work W2 are within the imaging area A1. In that case, for example, the imaging area A1 may be the same when capturing the target image IG (reference image) and the current image IC, or the visual information may be corrected by a known optical geometric correction method based on the internal and external parameters of the visual sensor 800.

[0081] In step S23, the worker inputs a command to acquire the target image IG to the control device 400. The control device 400 receives the command to acquire the target image IG and sends an image capturing command to the visual sensor 800 via the interface 410 and the bus 420. The control device 400 acquires an image of the imaging area A1 from the visual sensor 800 and temporarily stores it in the RAM 403 as a target image IG (reference image). The target image IG is image data (visual information) of the first workpiece W1 located at the approach position shown in FIG. 9. That is, the target image IG includes visual information of the end E1 on the tip side of the first workpiece W1 and the end E2 on the opening side of the second workpiece W2, which are in a reference positional relationship before the start of assembly.

[0082] In step S24, the control device 400 extracts image features between the end E1 on the tip side of the first workpiece W1 and the end E2 on the opening side of the second workpiece W2 from the target image IG acquired in step S23. Note that the method of extracting the image features is the same as the method described above, and therefore the description thereof will be omitted here.

[0083] Finally, in step S25, the control device 400 converts the image feature quantity extracted in step S24 into a target image feature quantity F g The target image feature amount F g is a feature value extracted from an image captured in step S23 with the first workpiece W1 and the second workpiece W2 in a state where they are in a predetermined assembly start positional relationship (reference positional relationship). g , the first workpiece W1 can be moved to the correct assembly start position (approach position). g If it is confirmed that there is no problem in carrying out the actual assembly work using the target image feature amount F g can be stored in the HDD 404 and reused.

[0084] According to the above-described assembly start position teaching method of the embodiment, the target image IG and the target image feature amount F at the accurate assembly start position are obtained by offsetting the position where assembly has been completed once. g can be easily obtained.

[0085] In step S21, the case where the worker jogs the robot arm 200 via the input device 500 to assemble the first workpiece W1 to the second workpiece W2 has been described, but the worker may directly touch and operate the robot by direct teaching. Direct teaching is a teaching method that gives the robot arm 200 flexibility against external forces by impedance control or the like, and makes the robot follow the reaction forces from the worker and the workpiece. In other words, not only does it make the operation of the robot arm 200 intuitive, but it also reduces the load on the workpiece until it reaches the workpiece assembly completion position shown in FIG. 7(b), and it eliminates bias in the workpiece, so that it is possible to teach a more accurate position.

[0086] The control device 400 can acquire an execution program for the final operation to be performed in step S17 (FIG. 5) during the assembly work by performing a reverse playback process of the movement operation from FIG. 7(b) to FIG. 7(c) performed in response to an instruction from the worker in step S22. That is, the position and posture of the robot arm can be changed during the assembly work based on the transition of the position and posture of the robot arm when the movement operation (change of the position and posture of the robot arm) is performed. More specifically, the coordinate system T e The motion (change in position and orientation) of the object is stored in time series, and the time series motion data is inverted on the time axis. e The position of each joint is stored as a coordinate calculated from the encoder value of each joint, and the change (displacement) of the coordinate is stored as a transition of the position and posture of the robot arm. Then, during the assembly work in step S17, the inverted motion data is stored in the coordinate system T e The robot arm 200 is operated based on the target trajectory of the above. This allows for more complicated assembly, such as assembling the workpiece W1 to the workpiece W2 while rotating it, in addition to assembly by linear movement. The motion data may be subjected to smoothing or thinning processing, etc., within a range that does not affect the assembly work, by analyzing the noise level and the density in the time series of the position and orientation as necessary. The operation performed by the robot arm 200 in step S17 is not limited to the operation set in advance before performing step S16, such as the operation based on the transition of the position and orientation of the robot arm in step S22. The operation performed by the robot arm 200 in step S17 may be the operation instructed by the user after the current image is acquired in step S13.

[0087] Also, although the steps S22 and S23 have been described as being executed by the operator performing input operations on the control device 400, the execution method is not limited to this. For example, if design information such as the angle of view of the visual sensor 800, the dimensions of the first workpiece W1 and the second workpiece W2, and the assembly direction is known in advance, the direction in which the first workpiece W1 is pulled out and the offset amount can also be determined in advance. In other words, steps S22 to S25 may be programmed in advance, and a series of teaching operations may be formulated so that the program execution automatically ends after step S21.

[0088] In the above embodiment, the case where both the first workpiece W1 and the second workpiece W2 are captured in the target image IG has been described, but this is not necessarily required. For example, when the positional relationship of one workpiece with respect to the visual sensor 800 is known in advance or when the change can be ignored, it is sufficient that either the first workpiece W1 or the second workpiece W2 is captured in the target image IG. Alternatively, when there is a position reference object (e.g., a jig for fixing the second workpiece W2) whose positional relationship is fixed with respect to the second workpiece W2, it is sufficient that the first workpiece W1 and the position reference object are captured in the target image. The control unit can make a judgment by comparing the positional relationship between the first workpiece and the position reference object captured in the current image with the positional relationship between the first teaching workpiece and the position reference object captured in the target image.

[0089] In this embodiment, in the assembly operation, the control unit uses the imaging unit to obtain a current image of at least the first workpiece, and judges whether the first workpiece and the second workpiece are in a reference positional relationship. If it is judged that they are in the reference positional relationship, that is, if it is judged that the first workpiece is in the approach position, the control unit controls the position and posture of the robot according to a control program taught in advance, and moves the first workpiece to the assembly completion position (target positional relationship).

[0090] In this embodiment, when the approach position is set and the target image is captured in advance, the first workpiece is moved in the opposite direction to the final operation of the assembling, the approach position is set, and the image is captured. Therefore, in this embodiment, the approach position can be set as a positional relationship that allows the robot to properly assemble the held first workpiece to the second workpiece from that positional relationship. In this embodiment, as in the first embodiment, the workpiece and the teaching workpiece do not necessarily need to use the same object as long as they are in a relationship in which various positional relationships can be recognized. Furthermore, as long as various positional relationships can be recognized and there is no discrepancy between the teaching content and the work content, the work robot arm and the teaching robot arm do not necessarily need to use the same mechanical device (robot arm).

[0091] [Other embodiments] The present invention is not limited to the above-described embodiments, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0092] In the above embodiment, the feedback control unit 450 is a part of the functions of the CPU 401, and the servo control unit 230 is configured as a device different from the CPU 401. However, the present invention is not limited to this. The CPU 401 may be configured to realize some or all of the functions of the servo control unit 230 based on the program 430.

[0093] In the above embodiment, the control device 400 and the display 600 are used to calculate the current image feature F cHowever, the present invention is not limited to this. For example, a separate electronic device equipped with a CPU and a display device such as a display may be used. The electronic device may be, for example, an information processing device such as a desktop PC (Personal Computer), a laptop PC, a tablet PC, or a smartphone. In addition, if the input device 500 is a teaching pendant having a display device, the display device may display the current image feature amount F c and the current image IC, the target image feature F g etc. may be displayed.

[0094] In the above embodiment, the robot arm 200 is a vertically articulated robot arm, but the present invention is not limited to this. The robot arm 200 may be, for example, a horizontally articulated robot arm, a parallel link robot arm, an orthogonal robot, or any other type of robot arm. In other words, the present invention can be implemented in various types of articulated robots.

[0095] In the above embodiment, the case where the robot hand 300 is attached to the robot arm 200 has been described, but the present invention is not limited to this. A holding mechanism capable of holding an object such as a workpiece may be attached to the robot arm 200 as an end effector. An example of the holding mechanism is a mechanism that holds a workpiece by suction. In addition, a tool for processing a workpiece may be attached to the robot arm 200 as an end effector.

[0096] The first workpiece W1 and the second workpiece W2 as objects are typically parts, but the present invention may be embodied in the control of a robot that performs an operation of bringing a tool as an object into contact with a part. For example, the first workpiece may be a tool attached to the robot, and the second workpiece W2 may be a part. Alternatively, the first workpiece may be a tool held by the robot, and the second workpiece W2 may be a part. Alternatively, the present invention may be embodied as control when the robot holds a screwdriver, a screw (first workpiece W1) is set in the screwdriver by a magnet, and the robot performs an operation of inserting the screw into a screw hole in the part that is the second workpiece W2. In this way, the workpiece and the tool are collectively referred to as objects, and these can be referred to as a work object or a teaching object.

[0097] The control method, control device, etc. of the present invention can be applied to the control of equipment and facilities equipped with various mechanical devices, such as production equipment with moving parts, for example, industrial robots, service robots, medical robots, machine tools operated by computer numerical control, etc.

[0098] The present invention can also be realized by a process in which a program for implementing one or more functions of the embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0099] This specification discloses at least the following: [Item 1] The computer acquires a current image that reflects a positional relationship between a first object and a second object held by the robot, and determines that the first object and the second object are in a reference positional relationship based on the current image. controlling a position and posture of the robot according to a motion of the robot set before the determination or a motion of the robot instructed by a user after the acquisition of the current image, so that the first object and the second object have a target positional relationship; A method for controlling a robot comprising the steps of: [Matter 2] Both the first object and the second object are captured in the current image. 2. A method for controlling a robot according to item 1. [Matter 3] The computer acquires an image feature amount corresponding to a positional relationship between a portion of the first object and a portion of the second object from the current image, and performs the determination based on the image feature amount. 3. A method for controlling a robot according to item 1 or 2. [Matter 4] In the control of the position and posture of the robot, the position and posture of the robot are changed according to a trajectory of the robot that is set before the determination. 4. A method for controlling a robot according to any one of claims 1 to 3. [Matter 5] When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by force control so that the first object and the second object are in the target positional relationship. 5. A method for controlling a robot according to any one of claims 1 to 4. [Matter 6] When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by position control so that the first object and the second object are in the target positional relationship. 5. A method for controlling a robot according to any one of claims 1 to 4. [Matter 7] In the reference positional relationship, the first object and the second object are spaced apart from each other, In the target positional relationship, the first object and the second object are in contact with each other. 7. A method for controlling a robot according to any one of claims 1 to 6. [Matter 8] an imaging unit for capturing the current image is disposed so as to move in accordance with a change in the position and posture of the robot; 3. A method for controlling a robot according to item 2. [Matter 9] controlling a position and posture of the robot by visual servoing so that the first object and the second object are in the reference positional relationship; 9. A method for controlling a robot according to any one of items 1 to 8. [Matter 10] the computer acquires a reference image that reflects that a first teaching object and a second teaching object held by the robot are in the reference positional relationship, and determines whether the first object and the second object are in the reference positional relationship by using the current image and the reference image; 10. A method for controlling a robot according to any one of claims 1 to 9. [Matter 11] setting a position and orientation of the mechanical device so that a first teaching object and a second teaching object held by the mechanical device have a target positional relationship; changing a position and posture of the mechanical device holding the first teaching object so that the first teaching object and the second teaching object change from the target positional relationship to a reference positional relationship; acquiring a reference image that reflects that the first teaching object and the second teaching object are in the reference positional relationship; The computer acquires a current image reflecting a positional relationship between a first object and a second object held by the robot, and determines that the first object and the second object are in the reference positional relationship based on the reference image and the current image. controlling a position and posture of the robot holding the first object so that the first object and the second object are in the target positional relationship; A method for controlling a robot comprising the steps of: [Matter 12] In the control of the position and posture of the robot, the position and posture of the robot are changed in accordance with a motion of the robot set before the determination or a motion of the robot instructed by a user after the acquisition of the current image. 12. A method for controlling a robot according to item 11. [Matter 13] In the control of the position and posture of the robot, the position and posture of the robot are changed based on a transition of the position and posture of the mechanical device when the position and posture of the mechanical device is changed. 13. A method for controlling a robot according to item 11 or 12. [Matter 14] The second teaching object is imaged in the reference image, The second object is captured in the current image. 14. A method for controlling a robot according to any one of items 11 to 13. [Matter 15] When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by force control so that the first object and the second object are in the target positional relationship. 15. A method for controlling a robot according to any one of items 11 to 14. [Matter 16] When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by position control so that the first object and the second object are in the target positional relationship. 15. A method for controlling a robot according to any one of items 11 to 14. [Matter 17] In the reference positional relationship, the first teaching object and the second teaching object are spaced apart from each other, In the target positional relationship, the first teaching object and the second teaching object are in contact with each other. 17. A method for controlling a robot according to any one of items 11 to 16. [Matter 18] an imaging unit for capturing the current image is disposed so as to move in accordance with a change in the position and posture of the robot; 18. A method for controlling a robot according to any one of items 11 to 17. [Matter 19] controlling a position and posture of the robot by visual servoing so that the first object and the second object are in the reference positional relationship; 19. A method for controlling a robot according to any one of items 11 to 18. [Matter 20] The mechanical device is the robot. 20. A method for controlling a robot according to any one of items 11 to 19. [Matter 21] A method for controlling a robot according to any one of claims 1 to 20, comprising the computer. A system characterized by: [Matter 22] The robot; A control unit for controlling the robot; Equipped with 22. The system according to item 21, characterized in that [Matter 23] The robot is an articulated robot, At least one joint of the articulated robot is provided with a force sensor. 23. The system according to item 21 or 22, [Matter 24] An imaging unit for capturing the current image, 24. The system according to any one of claims 21 to 23. [Matter 25] 21. Controlling the robot by the robot control method according to any one of items 1 to 20 to assemble the first object to the second object. A method for producing an article comprising the steps of: [Matter 26] 21. A program for causing a computer to execute the robot control method according to any one of items 1 to 20. [Matter 27] 27. A computer-readable recording medium storing the program according to item 26.

[0100] The present invention is not limited to the above-described embodiment, and many modifications are possible within the technical concept of the present invention. Moreover, the effects described in the embodiment are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiment. Moreover, the described embodiment can be appropriately modified within the scope of the technical concept. For example, a plurality of embodiments can be combined. Moreover, some matters of at least one embodiment can be deleted or replaced. Moreover, new matters can be added to at least one embodiment. The contents disclosed in this specification include not only what is explicitly described in this specification, but also all matters that can be understood from this specification and the drawings attached to this specification. Moreover, the contents disclosed in this specification include the complement of the individual concepts described in this specification. In other words, for example, if there is a description to the effect that "A is B" in this specification, even if the description to the effect that "A is not B" is omitted, it can be said that this specification discloses "A is not B". This is because when "A is B" is described, it is assumed that the case where "A is not B" is taken into consideration. [Explanation of symbols]

[0101] 100···Robot / 200···Robot arm / 210~216···Link / 300···Robot hand / 400···Control device / 401···CPU / 406~409···Interface / 500···Input device / 600···Display / 700···External storage device / 800···Visual sensor / 1000···Robot system / J1~J6···Joints / W1···First workpiece / W2···Second workpiece

Claims

1. The computer acquires a current image reflecting a positional relationship between a first object and a second object held by the robot, and determines based on the current image that the first object and the second object are in a reference positional relationship. controlling a position and posture of the robot according to a motion of the robot set before the determination or a motion of the robot instructed by a user after the acquisition of the current image, so that the first object and the second object have a target positional relationship; A method for controlling a robot comprising the steps of:

2. Both the first object and the second object are captured in the current image.

2. The method for controlling a robot according to claim 1.

3. The computer acquires an image feature amount according to a positional relationship between a portion of the first object and a portion of the second object from the current image, and performs the determination based on the image feature amount.

2. The method for controlling a robot according to claim 1.

4. In the control of the position and posture of the robot, the position and posture of the robot are changed according to a trajectory of the robot that is set before the determination.

2. The method for controlling a robot according to claim 1.

5. When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by force control so that the first object and the second object are in the target positional relationship.

2. The method for controlling a robot according to claim 1.

6. When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by position control so that the first object and the second object are in the target positional relationship.

2. The method for controlling a robot according to claim 1.

7. In the reference positional relationship, the first object and the second object are spaced apart from each other, In the target positional relationship, the first object and the second object are in contact with each other.

2. The method for controlling a robot according to claim 1.

8. an imaging unit for capturing the current image is disposed so as to move in accordance with a change in the position and posture of the robot; 3. The method for controlling a robot according to claim 2.

9. controlling a position and posture of the robot by visual servoing so that the first object and the second object are in the reference positional relationship; 2. The method for controlling a robot according to claim 1.

10. the computer acquires a reference image that reflects that a first teaching object and a second teaching object held by the robot are in the reference positional relationship, and determines whether the first object and the second object are in the reference positional relationship by using the current image and the reference image; 2. The method for controlling a robot according to claim 1.

11. setting a position and orientation of the mechanical device so that a first teaching object and a second teaching object held by the mechanical device have a target positional relationship; changing a position and posture of the mechanical device holding the first teaching object so that the first teaching object and the second teaching object change from the target positional relationship to a reference positional relationship; acquiring a reference image that reflects that the first teaching object and the second teaching object are in the reference positional relationship; The computer acquires a current image reflecting a positional relationship between a first object and a second object held by a robot, and determines that the first object and the second object are in the reference positional relationship based on the reference image and the current image. controlling a position and posture of the robot holding the first object so that the first object and the second object are in the target positional relationship; A method for controlling a robot comprising the steps of:

12. In the control of the position and posture of the robot, the position and posture of the robot are changed in accordance with a motion of the robot set before the determination or a motion of the robot instructed by a user after the acquisition of the current image. The method for controlling a robot according to claim 11.

13. In the control of the position and posture of the robot, the position and posture of the robot are changed based on a transition of the position and posture of the mechanical device when the position and posture of the mechanical device is changed. The method for controlling a robot according to claim 11.

14. The second teaching object is imaged in the reference image, The second object is captured in the current image. The method for controlling a robot according to claim 11.

15. When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by force control so that the first object and the second object are in the target positional relationship. The method for controlling a robot according to claim 11.

16. When the computer determines that the first object and the second object are in the reference positional relationship, the computer controls the robot by position control so that the first object and the second object are in the target positional relationship. The method for controlling a robot according to claim 11.

17. In the reference positional relationship, the first teaching object and the second teaching object are spaced apart from each other, In the target positional relationship, the first teaching object and the second teaching object are in contact with each other. The method for controlling a robot according to claim 11.

18. an imaging unit for capturing the current image is disposed so as to move in accordance with a change in the position and posture of the robot; The method for controlling a robot according to claim 11.

19. controlling a position and posture of the robot by visual servoing so that the first object and the second object are in the reference positional relationship; The method for controlling a robot according to claim 11.

20. The mechanical device is the robot. The method for controlling a robot according to claim 11.

21. A method for controlling a robot according to any one of claims 1 to 20, comprising the computer. A system characterized by:

22. The robot; A control unit for controlling the robot; Equipped with 22. The system of claim 21 .

23. The robot is an articulated robot, A force sensor is provided in at least one joint of the articulated robot.

22. The system of claim 21 .

24. An imaging unit for capturing the current image, 22. The system of claim 21 .

25. 21. The robot is controlled by the method for controlling a robot according to claim 1 to 20, so as to assemble the first object to the second object. A method for producing an article.

26. A program for causing a computer to execute the robot control method according to any one of claims 1 to 20.

27. A computer-readable recording medium storing the program according to claim 26.

Images