Robot Control System

Abstract

To provide a robot control system that enables a robot to perform working by visual feedback control.SOLUTION: A robot control system 11, which controls a robot 20 that grips a work-piece 50 with a robot hand 24, comprises a three-dimensional visual sensor 30 that obtains three-dimensional information about the work-piece, sensor moving means that moves the three-dimensional visual sensor, and a control device 40. The control device calculates a distance between a gripping position on the work-piece and the robot hand on the basis of the three-dimensional information; feedback-controls a position of the robot hand so that the distance between the gripping position and the robot hand reduces; determines whether or not the gripping position in a field of view of the three-dimensional visual sensor; and when determining that the gripping position is out of the field of view, searches the gripping position by widening the field of view of the three-dimensional visual sensor or moving the sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a robot control system that uses a three-dimensional visual sensor to perform feedback control of a robot. [Background technology]

[0002] Robots are being used to automate tasks such as gripping a workpiece and connecting it to a predetermined position. In a typical robot control system, to move the robot's hand to a target object, position information of the object is acquired using a visual sensor such as a camera, and this position information is then converted from the camera coordinate system to a world coordinate system, and then converted to the robot coordinate system before being transmitted to the robot. However, determining the transformation matrix between the camera coordinate system and the world coordinate system requires camera calibration, which is an extremely time-consuming and costly process.

[0003] In response to this, a technology called visual feedback control has been developed in recent years. This control method involves feedback-controlling a robot based on feature quantities obtained from images captured by a visual sensor. For example, by using the distance between an object and the robot's hand as a feature quantity and feedback-controlling the robot's movement to reduce this distance, the hand can be made to reach the object without converting the coordinates of the object's position.

[0004] Patent Document 1 describes a robot control device that performs visual feedback control of the position and orientation of a robot arm in a robot system that grips a workpiece. The robot control device described in Patent Document 1 first photographs the workpiece with a CCD camera attached to the robot arm while the robot arm is positioned in advance at a taught target position relative to the workpiece, and stores the feature quantities in the obtained image as target data in a visual recognition device. Then, the error between the current data and the target data during the operation is calculated, and the position and orientation of the robot arm are repeatedly controlled until the error converges to zero. The feature quantities are calculated by extracting multiple feature points set on the workpiece from the image and using the visual recognition device based on their positions and orientations to recognize the position and orientation of the workpiece.

[0005] Patent Document 2 describes a control device that performs visual feedback control on the arm and hand of a life support robot that assists in the care of elderly people and the like, and that moves the object to be grasped to a target position after the hand of the robot arm grasps the object to be grasped. In order to solve the problem that the object to be grasped itself obstructs the camera's field of view just before and just after grasping the object to be grasped and the target position cannot be identified, Patent Document 2 describes that a front camera is provided in the grasping section of the hand and side cameras are also installed on the sides, and just before or just after the hand grasps the object to be grasped, the camera that forms the basis of the visual feedback control is switched from the front camera to the side camera. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-211381 [Patent Document 2] Japanese Patent Application Laid-Open No. 2011-235386 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the control device described in Patent Document 1 has the problem that it cannot be applied to work on linear objects or the like that are easily deformed and the positional relationships between feature points are not constant, because the feature amount to be used as target data differs for each work. Also, the control device described in Patent Document 2 cannot complete the work if the object to be grasped or the target position moves and goes out of the camera's field of view.

[0008] The present invention has been made in consideration of the above, and aims to provide a robot control system that causes a robot to perform work using visual feedback control, that is applicable even when the workpiece is easily deformed, and that can continue working even when the work object is out of the field of view of the three-dimensional visual sensor. [Means for solving the problem]

[0009] A robot control system of the present invention is a system for controlling a robot that moves a first target, which moves in conjunction with the movement of a robot hand, to a second target that is stationary or moves independently of the movement of the robot hand, and includes a 3D visual sensor that acquires 3D information of the second target, a sensor moving means that moves the 3D visual sensor, and a control device. The control device calculates the distance between the second target and the first target based on the 3D information, feedback-controls the position of the robot hand so as to reduce the distance between the second target and the first target, determines whether the second target is within the field of view of the 3D visual sensor, and if it is determined that the second target is outside the field of view, searches for the second target by widening or moving the field of view of the 3D visual sensor, and controls the sensor moving means so that the second target is included in the field of view of the 3D visual sensor.

[0010] In a first aspect of the robot control system of the present invention, the first object is the robot hand itself, and the second object is a task target on a workpiece. That is, a first aspect of the robot control system is a system for controlling a robot that moves a robot hand to a work target on a workpiece that is stationary or moves independently of the movement of the robot hand, and includes a 3D visual sensor that acquires 3D information of the work target, a sensor moving means that moves the 3D visual sensor, and a control device. The control device calculates the distance between the work target and the robot hand based on the 3D information, feedback controls the position of the robot hand so as to reduce the distance between the work target and the robot hand, determines whether the work target is within the field of view of the 3D visual sensor, and if it is determined that the work target is outside the field of view, searches for the work target by widening or moving the field of view of the 3D visual sensor, and controls the sensor moving means so that the work target is included in the field of view of the 3D visual sensor.

[0011] Here, work broadly means any action performed by a robot hand on a work target, and includes, for example, grasping the work target by clamping or suction with the robot hand, and performing various processes on the work target such as cutting, grinding, discharge, and soldering.

[0012] In the robot control system of the first aspect described above, preferably, the control device further calculates a working direction for the robot hand to work on the work target based on the three-dimensional information, and feedback-controls the orientation of the robot hand so as to reduce the deviation between the working direction and the orientation of the robot hand.

[0013] In the robot control system of the first aspect described above, preferably, the control device further calculates a working posture for the robot hand to work on the work target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the working posture and the posture of the robot hand.

[0014] In a second aspect of the robot control system of the present invention, the first object is a specific portion of a workpiece gripped by the robot hand, and the second object is a connection target that connects the specific portion. That is, a robot control system of a second aspect is a system for controlling a robot that moves a specific portion of a workpiece held by a robot hand to a connection target that connects the specific portion and that is stationary or moves independently of the movement of the robot hand, and includes a 3D visual sensor that acquires 3D information of the connection target, a sensor moving means that moves the 3D visual sensor, and a control device. The control device calculates the distance between the connection target and the specific portion based on the 3D information, feedback-controls the position of the robot hand so as to reduce the distance between the connection target and the specific portion, determines whether the connection target is within the field of view of the 3D visual sensor, and if it is determined that the connection target is outside the field of view, searches for the connection target by widening or moving the field of view of the 3D visual sensor, and controls the sensor moving means so that the connection target is included in the field of view of the 3D visual sensor.

[0015] Here, connection broadly means moving a specific part of the work to align it with a connection target, and includes, for example, connecting a connector (specific part) on the work to another mating connector (connection target), or inserting the tip (specific part) of a linear or strip-shaped work into a hole or slit (connection target).

[0016] In the robot control system of the second aspect described above, preferably, the control device further calculates a connection direction for connecting the specific part to the connection target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the connection direction and the orientation of the specific part.

[0017] In the robot control system of the second aspect described above, preferably, the control device further calculates a connection posture for connecting the specific part to the connection target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the connection posture and the posture of the specific part.

[0018] In a third aspect of the robot control system of the present invention, the first object is a workpiece grasped by the robot hand, the second object is a placement target onto which the workpiece is to be placed, and the control device further calculates a placement posture for placing the workpiece on the placement target based on the three-dimensional information, and feedback-controls the posture of the robot hand so as to reduce the deviation between the placement posture and the posture of the workpiece. That is, a robot control system of a third aspect is a system for controlling a robot that moves a workpiece grasped by a robot hand to a placement target on which the workpiece is to be placed, the placement target being stationary or moving independently of the movement of the robot hand, and includes a 3D visual sensor that acquires 3D information of the placement target, a sensor moving means that moves the 3D visual sensor, and a control device. The control device calculates the distance between the placement target and the workpiece based on the 3D information, calculates the orientation of the placement target, feedback-controls the position of the robot hand to reduce the deviation between the distance between the placement target and the workpiece and the orientation of the placement target and the orientation of the workpiece, determines whether the placement target is within the field of view of the 3D visual sensor, and if it is determined that the placement target is outside the field of view, searches for the placement target by widening or moving the field of view of the 3D visual sensor, and controls the sensor moving means to include the placement target in the field of view of the 3D visual sensor.

[0019] Here, the placement target is an area large enough to place the workpiece, and placement means placing the workpiece within the placement target. [Effects of the Invention]

[0020] According to the robot control system of the present invention, when a robot performs work using visual feedback control, it can be applied even to workpieces that are easily deformed, and the work can continue even if the work object moves out of the field of view of the three-dimensional visual sensor. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram illustrating a device configuration of a robot control system according to a first embodiment. [Figure 2] FIG. 2 is a process flow diagram of a workpiece gripping operation using the robot control system of the first embodiment. [Figure 3] 10A and 10B are diagrams for explaining the gripping position and gripping posture of a linear workpiece; [Figure 4] FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a second embodiment. [Figure 5] FIG. 10 is a process flow diagram of a workpiece gripping operation using a robot control system according to a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a third embodiment. [Figure 7] FIG. 11 is a process flow diagram of a connection operation using a robot control system according to the third embodiment. [Figure 8] FIG. 10 is a diagram for explaining the connection attitude of the connection target. [Figure 9] FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a fourth embodiment. [Figure 10] FIG. 10 is a diagram illustrating a device configuration of a robot control system according to a fifth embodiment. [Figure 11] FIG. 10 is a process flow diagram of a placement operation using the robot control system of the fifth or sixth embodiment. [Figure 12] FIG. 13 is a diagram illustrating a device configuration of a robot control system according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] A first embodiment of a robot control system of the present invention will be described with reference to Figures 1 to 3. In this embodiment, in an operation in which a workpiece is grasped by a robot hand, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand.

[0023] The robot control system 11 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 11. The robot control system 11 causes the robot hand 24 to grasp a workpiece, that is, a cable 50, at a grasping position 52. The grasping position 52 is the task target for this grasping operation. The position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30. Note that in this specification, the robot hand may be simply referred to as the "hand." Furthermore, the position of the hand refers to the position of the tool center point (TCP) of the hand, unless otherwise specified.

[0024] In this embodiment, a cable 50 is described as an example of a workpiece; however, the type of workpiece grasped by the hand 24 is not particularly limited. The workpiece may be a linear object other than a cable, or a strip-shaped object other than a linear object, such as a flat cable. The workpiece may also be a harness with a connector, an electronic component, food, stationery, or the like. The robot control system 11 of this embodiment can be applied to flexible and easily deformed workpieces, and is therefore particularly advantageous when the workpiece is a linear or strip-shaped object. The form in which the cable 50 is supplied is not particularly limited. The cable 50 may be placed on a tray or the like, or may be housed in a cylindrical container with one end protruding from the container. The cable 50 may be stationary, or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

[0025] In addition, although the present embodiment will be described taking the example of a cable gripping operation, the operation performed by the hand 24 on the work target is not limited to gripping. For example, by selecting an appropriate end effector as the hand, various processes such as cutting, grinding, discharge, soldering, etc. may be performed on the work target.

[0026] The robot 20 is an articulated robot and includes an arm 21 consisting of multiple links 22 and joints 23, and a hand 24 attached to the end of the arm. The type of hand 24 is not particularly limited, and a hand suitable for grasping the workpiece can be used depending on the type of workpiece. For example, the hand may hold the workpiece between a pair of fingers, or may grasp the workpiece by vacuum suction with a nozzle. In this embodiment, a stereo camera 30, which is a three-dimensional visual sensor, is fixed to the hand 24, and the robot 20 also serves as a sensor moving means for changing the position and orientation of the stereo camera.

[0027] The stereo camera 30 is a three-dimensional visual sensor that acquires three-dimensional information of the cable 50. Here, the three-dimensional information of the cable is information that enables the three-dimensional coordinates of the gripping position on the cable to be derived. Specifically, the three-dimensional information acquired by the stereo camera 30 is two images captured from different viewpoints. Hereinafter, the images acquired by the stereo camera 30 will be referred to as "stereo images" or simply "images."

[0028] In this embodiment, the stereo camera 30 is fixed to the hand 24. The stereo camera is preferably installed at the base end of the hand, facing the tip of the hand. However, since the relative positions of the stereo camera and the hand are known from design drawings, etc., it is not necessary for the hand to be within the field of view of the stereo camera.

[0029] The type of 3D visual sensor is not limited to a stereo camera, and is not particularly limited as long as it can acquire images containing 3D information about the workpiece. Examples of 3D visual sensors include active stereo systems, such as light-section methods in which one of the stereo cameras is replaced with a projector, and ToF systems. A stereo camera is preferably used as the 3D visual sensor. A stereo camera is suitable for accurately measuring close-range objects in 3D. When the hand 24 grasps the cable 50, as the stereo camera 30 approaches the cable 50, the 3D coordinates of the grasping position 52 can be determined with higher accuracy. Note that the stereo camera may be a system in which three or more cameras simultaneously capture three or more images.

[0030] The control device 40 performs various calculations and controls the entire robot control system 11. The control device 40 performs various calculations, such as image processing of images acquired by the stereo camera 30 to determine the three-dimensional coordinates of the gripping position 52 on the cable 50 and calculate the distance between the gripping position 52 and the hand 24. The control device 40 feedback-controls the position of the hand by feedback-controlling the robot 20 to move the hand 24 so as to reduce the distance between the gripping position 52 and the hand 24. The control device 40 also determines whether the gripping position is within the image acquired by the stereo camera 30. If the gripping position 52 is outside the field of view of the stereo camera 30 and is not captured in the image, the control device 40 searches for the gripping position by changing the position and / or posture of the stereo camera 30, and controls the operation of the sensor movement means, or in this embodiment, the robot 20, so that the gripping position is included in the field of view of the stereo camera. Details of the calculations and control performed by the control device will be described later.

[0031] There are no particular limitations on the physical configuration of the control device 40. The control device 40 may be physically configured as a single device, or may be configured as two or more separate parts. Furthermore, the control device 40 may be configured integrally with a robot control unit (not shown) that controls the operations of the joints 23 and hands 24 of the robot 20.

[0032] Next, the operation of gripping the cable 50 with the hand 24 using the robot control system 11 of this embodiment will be described along the process flow of FIG.

[0033] (S11) Stereo camera 30 captures an image of cable 50 to obtain three-dimensional information about the cable. This three-dimensional information serves as the basis for control device 40 to calculate the three-dimensional coordinates of gripping position 52 on cable 50, so stereo camera 30 captures an image of at least the range in which the coordinates of the gripping position can be calculated. For example, if the gripping position is indicated by a mark, the range in which the coordinates of the gripping position can be calculated is sufficient to include the mark. If the gripping position is determined by the distance from some landmark such as a mark or the tip of the cable, the range in which the coordinates of the gripping position can be calculated is the portion from the landmark to the gripping position. In FIG. 3, gripping position 52 is set at a distance d from tip 51 of cable 50.

[0034] Furthermore, if the orientation X of the hand 24 when gripping the cable 50 is limited, the stereo camera 30 captures an image of at least a range in which it is possible to calculate the gripping direction A for the hand to grip the gripping position 52. For example, in the case of gripping the cable from the side, the range in which it is possible to calculate the gripping direction A is an area that includes the gripping position and is wide enough to allow calculation of the tangent direction T of the cable at the gripping position 52. Once the tangent direction T is determined, it is possible to calculate the gripping direction A with respect to the cable.

[0035] Furthermore, when the hand 24 is not isotropic around its direction X and grips the cable 50 with two fingers as in the present embodiment, it is necessary to prevent the fingers from interfering with the cable when the tool center point TCP is aligned with the gripping position 52. Therefore, when the hand grips the cable, not only the hand direction X but also the rotation angle θ around X is limited. In such a case, the stereo camera 30 captures images of at least a range in which the gripping posture for the hand to grip the gripping position 52 can be calculated. The gripping posture is the posture required for the hand to grip the cable. The hand posture is expressed by the hand direction X and the rotation angle θ around X, and the gripping posture is expressed by the gripping direction A and the rotation angle α around A of the hand gripping the gripping position. Similar to the range in which the gripping direction A can be calculated, the range in which the gripping posture can be calculated is, for example, an area including the gripping position when gripping the cable from the side, and is wide enough to allow the tangential direction T of the cable at the gripping position 52 to be calculated.

[0036] (S12) The control device 40 determines whether or not the gripping position 52 is included in the stereo image acquired by the stereo camera 30. If the gripping position is not within the field of view of the stereo camera 30 and is not included in the image, the control device 40 searches for the gripping position. The search step (S17) will be described later. Note that if there is no landmark near the gripping position and it is not possible to determine whether or not the gripping position is included in the image without calculating the three-dimensional coordinates of the gripping position, this determination step (S12) is performed after the next step (S13). If the three-dimensional coordinates of the gripping position 52 can be calculated in the next step (S13), it is determined that the gripping position was included in the image, and if the three-dimensional coordinates of the gripping position 52 cannot be calculated, it can be determined that the gripping position was not included in the image.

[0037] (S13) The control device 40 processes the stereo images to recognize the three-dimensional shape of the cable 50 and calculates the three-dimensional coordinates of the gripping position 52 and the gripping orientation. These values may be calculated within a certain tolerance range. The three-dimensional coordinates and orientation calculated by the control device 40 are all values in the camera coordinate system. Recognition of the three-dimensional shape of the cable and calculation of the three-dimensional coordinates of the gripping position 52 and the gripping orientation can be performed using known methods. For example, a method for calculating the gripping position and gripping orientation when the workpiece is a strip-shaped object is disclosed in Japanese Patent Application Laid-Open No. 2020-037147 by the present applicant.

[0038] (S14) The control device 40 calculates the distance between the gripping position 52 and the hand 24, and the deviation between the gripping posture and the posture of the hand. The position and posture of the hand 24 may be calculated based on the image if the hand is within the field of view of the stereo camera 30 and appears in the image. However, in this embodiment, the stereo camera 30 is fixed to the hand 24, and therefore the relative positional relationship between the stereo camera and the hand is known. Therefore, even if the hand does not appear in the image, the position and posture of the hand in the camera coordinate system can be known.

[0039] (S15) The control device 40 determines whether the coordinates of the gripping position 52 and the position of the hand 24, and the gripping posture and posture of the hand 24 match. If the gripping position and the position of the hand, and the gripping posture and posture of the hand match, the hand 24 grasps the gripping position (step S18). Note that if an allowable range is given for the gripping posture, the two are considered to match if the posture of the hand is within that range. The same applies to the three-dimensional coordinates of the gripping position.

[0040] (S16) If the gripping position 52 and the position of the hand 24, or the gripping posture and the posture of the hand 24, do not match, the control device 40 instructs the robot 20 to feedback control the position and posture of the hand 24 so as to reduce the distance between the gripping position and the hand, and the deviation between the gripping posture and the posture of the hand 24. During this process, the stereo camera 30 fixed to the hand 24 also approaches the gripping position, and the accuracy of calculating the three-dimensional coordinates of the gripping position, etc., gradually improves.

[0041] (S17) If it is determined in step S12 that the image acquired by the stereo camera 30 does not include the gripping position 52, the field of view of the stereo camera is widened or moved to search for a gripping position outside the field of view of the stereo camera.

[0042] One method for widening the field of view of the stereo camera is to move the hand 24 to which the stereo camera is fixed away from the cable 50. Once the gripping position 52 is found, steps S11 and after are repeated. Alternatively, the direction of the stereo camera 30 may be changed to move the gripping position closer to the center of the field of view, and then steps S11 and after may be repeated.

[0043] Another method for widening the field of view of the stereo camera is to zoom out the stereo camera 30 if the focal length of the stereo camera 30 is variable. Once the gripping position 52 is found, the orientation of the stereo camera 30 is changed to bring the gripping position closer to the center of the field of view, and then steps S11 and subsequent steps are repeated. The focal length of the stereo camera is returned to its original position after the gripping position is brought closer to the center of the field of view, or at some stage during the process of repeating steps S11 and subsequent steps.

[0044] One method for moving the field of view of the stereo camera is to change the posture of the hand 24 to which the stereo camera is fixed, for example, to move the field of view so that the center of the line of sight describes a spiral. Once the gripping position 52 is found, the orientation of the stereo camera 30 is left as it was when the gripping position 52 was found, or preferably, the gripping position is moved toward the center of the field of view, and then step S11 and subsequent steps are repeated.

[0045] As described above, the control device 40 controls the robot 20, which is the sensor moving means, or controls the robot 20 and the stereo camera 30, to search for the gripping position 52. Note that in this step (S17), visual feedback control can be used to move the gripping position to the center of the field of view of the stereo camera. Also, in this step (S17), it is sufficient to know whether the gripping position 52 is within the field of view of the stereo camera 30, and the three-dimensional coordinates of the gripping position are not used. Therefore, this step can also be performed using only one image from the stereo images.

[0046] By repeating the above steps, as described above, if it is determined in step S15 that the gripping position 52 and the position of the hand 24, and the gripping posture and the posture of the hand, are consistent, the gripping position 52 is gripped by the hand 24 (S18), and the work is completed.

[0047] Note that if the posture of the hand when gripping a workpiece is not an issue, there is no need to calculate the grip posture, calculate the deviation between the grip posture and the hand posture, or determine whether they match. Each of the above steps can be performed by focusing only on the grip position and the position of the hand 24. Similarly, if the rotation angle θ of the hand when gripping a workpiece is not an issue, each of the above steps can be performed by focusing only on the grip position, grip direction A, hand position, and hand orientation X. Furthermore, correction of the hand orientation X and rotation angle θ may be performed independently of correction of the hand position at some stage during the repetition of the above steps. The fact that it is sufficient to focus only on the hand position and orientation X when the posture and rotation angle of the hand are not an issue is the same in the second to fourth embodiments described below. The fact that correction of the hand posture can be performed independently of correction of the position is the same in the second to sixth embodiments described below.

[0048] A second embodiment of a robot control system of the present invention will be described with reference to Figures 4-5 and 3. In this embodiment, in an operation in which a workpiece is grasped by a robot hand, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor installed away from the robot. In the following, differences from the first embodiment will be described, and explanations of similarities to the first embodiment will be omitted.

[0049] The robot control system 12 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 installed away from the robot, a sensor moving means 32 for moving the stereo camera, and a control device 40 for overall control of the robot control system 12. The robot control system 12 causes the robot hand 24 to grasp a workpiece, a cable 50, at a grasping position 52. During this grasping operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

[0050] The stereo camera 30 acquires three-dimensional information of the cable 50, and also acquires three-dimensional information of the hand 24 in this embodiment.

[0051] The sensor moving means 32 changes the position and orientation of the stereo camera 30, which is a three-dimensional visual sensor. The structure of the sensor moving means may be an articulated robot similar to that of the robot 20, but since it is only necessary to change the position and orientation of the stereo camera within a predetermined range depending on the task, it can have a simpler structure than the robot 20. For example, when the cable 50 does not move, it is only necessary to change the distance between the stereo camera pointed at the cable and the cable in accordance with the movement of the hand 24, so a linear actuator that extends and retracts a cylinder can be used. When the cable 50 moves, even if an arm with joints is used, it is often possible to use fewer joints than the robot 20.

[0052] The control device 40 processes the images acquired by the stereo camera 30, calculates the three-dimensional coordinates of the position of the hand 24 together with the three-dimensional coordinates of the gripping position 52 on the cable 50, and determines whether the gripping position 52 and the hand 24 are included in the stereo images acquired by the stereo camera 30. Details of the calculations and controls performed by the control device will be described later.

[0053] Next, the operation of gripping the cable 50 with the hand 24 using the robot control system 12 of this embodiment will be described along the process flow of FIG.

[0054] (S21) The cable 50 and the hand 24 are imaged by the stereo camera 30 to obtain three-dimensional information about the cable and the hand.

[0055] (S22) The control device 40 determines whether the gripping position 52 and the hand 24 are included in the stereo image acquired by the stereo camera 30. If either the gripping position or the hand is not within the field of view of the stereo camera 30 and is not included in the image, the control device 40 searches for the gripping position or the hand in a search step (S27).

[0056] (S23) The control device 40 processes the stereo image, recognizes the three-dimensional shape of the cable 50, and calculates the three-dimensional coordinates of the gripping position 52 and the gripping posture, and at the same time, recognizes the three-dimensional shape of the hand 24 and calculates the position and posture of the hand.

[0057] (S24) The control device 40 calculates the distance between the gripping position 52 and the hand 24, and the deviation between the gripping posture and the posture of the hand.

[0058] (S25) The control device 40 determines whether or not the coordinates of the gripping position 52 and the position of the hand 24 match, and whether or not the gripping posture and the posture of the hand 24 match. If the gripping position and the position of the hand, and the gripping posture and the posture of the hand match, the hand 24 grips the gripping position (step S28).

[0059] (S26) If the gripping position 52 and the position of the hand 24, or the gripping posture and the posture of the hand 24, do not match, the control device 40 instructs the robot 20 to feedback control the position and posture of the hand 24 so as to reduce the difference between the distance between the gripping position and the hand, and between the gripping posture and the posture of the hand 24. The control device 40 also instructs the sensor moving means 32 to control the position and posture of the stereo camera 30. Specifically, when the gripping position 52 or the hand 24 is on the periphery of the image, the field of view of the stereo camera is moved so as to move both toward the center of the image. Furthermore, as the hand approaches the cable, the distance between the stereo camera and the gripping position is reduced to improve the measurement accuracy of the gripping position and the position and posture of the hand.

[0060] (S27) If it is determined in step S22 that the gripping position 52 or the hand 24 is not included in the image acquired by the stereo camera 30, the gripping position is searched for. The gripping position can be searched for by widening or moving the field of view of the stereo camera 30, as in the first embodiment. Specifically, the field of view of the stereo camera is widened by moving the stereo camera away from the cable 50, or by controlling the stereo camera to zoom out if the focal length of the stereo camera 30 is variable, and a gripping position outside the field of view of the stereo camera is searched for. Alternatively, the field of view of the stereo camera is moved to search for a gripping position outside the field of view of the stereo camera. If the gripping position 52 or the hand 24 is found, the gripping position and the hand are brought into the field of view of the stereo camera, and then step S21 and subsequent steps are repeated.

[0061] By repeating the above steps, as described above, if it is determined in step S25 that the gripping position 52 and the position of the hand 24, and the gripping posture and the posture of the hand, match, the gripping position 52 is gripped by the hand 24 (S28), and the work is completed.

[0062] A third embodiment of a robot control system of the present invention will be described with reference to Figures 6 to 8. In this embodiment, in the task of connecting a characteristic portion on a workpiece gripped by a robot hand to a predetermined connection target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand. Below, differences from the first embodiment will be described, and explanations of similarities to the first embodiment will be omitted.

[0063] The robot control system 13 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 13. The robot control system 13 connects a plug 61 (specific part) attached to the end of a cable 60 (workpiece) held by the robot hand 24 to a socket 63 (connection target) on a circuit board 62. During this connection operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

[0064] In this embodiment, the cable 60 is described as an example of the workpiece, the plug 61 as a specific part, and the socket 63 as an example of the connection target, but these types are not particularly limited. The robot control system 13 of this embodiment can also be applied to connection tasks such as passing the tip of a cable or tube through an opening or slit in a processing device or the like, fitting a predetermined part of various components into a predetermined position inside the device, or fixing a predetermined part of a cover or exterior panel to a predetermined position on the device body or the like. The socket 63 may be stationary or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

[0065] In this embodiment, when the workpiece does not deform (including when the robot hand 24 directly grips the plug 61), the relative positional relationship between the stereo camera 30 fixed to the hand 24 and the plug 61 is known if the workpiece is gripped in a specific position and orientation. On the other hand, when the workpiece is flexible, easily deformed, or the gripping position or orientation of the workpiece is not fixed, the relative positional relationship between the stereo camera and the plug is not constant and is unknown. The following describes a case where the relative positional relationship between the stereo camera and the plug is unknown.

[0066] The stereo camera 30 acquires three-dimensional information of the socket 63 and the plug 61. The robot 20 of this embodiment also serves as a sensor moving means for changing the position and posture of the stereo camera.

[0067] The control device 40 performs image processing of images acquired by the stereo camera 30, various calculations, feedback control of the robot 20, search for the socket 63, etc. The details of the calculations and controls performed by the control device will be described later.

[0068] Next, the connection work using the robot control system 13 of this embodiment will be described along the process flow of FIG.

[0069] (S31) Images of the socket 63 and the plug 61 are captured by the stereo camera 30 to obtain three-dimensional information on their respective positions. Furthermore, if the orientation Y of the plug when connecting the plug to the socket and the rotation angle φ about Y are limited, the stereo camera 30 captures images within a range that allows calculation of at least the connection direction B for connecting the plug to the socket and the rotation angle β about B. Hereinafter, the connection direction B and the rotation angle β required to connect the plug to the socket will be collectively referred to as the "connection posture."

[0070] (S32) The control device 40 determines whether the socket 63 and the plug 61 are included in the stereo image acquired by the stereo camera 30, and if the socket or the plug is not within the field of view of the stereo camera 30 and is not included in the image, the control device 40 performs a search process (S37).

[0071] (S33) Control device 40 processes the stereo image to recognize the three-dimensional shapes of socket 63 and plug 61, and calculates the three-dimensional coordinates of the socket position and the connection attitude, as well as the three-dimensional coordinates of the plug position and the attitude.

[0072] (S34) The control device 40 calculates the distance between the socket 63 and the plug 61, and the deviation between the connected posture and the posture of the plug.

[0073] (S35) The control device 40 determines whether the positions of the socket 63 and the plug 61 and the connection posture and the plug posture match. If the positions of the socket and the plug and the connection posture and the plug posture match, the plug is connected to the socket (step S38).

[0074] (S36) If the positions of the socket 63 and plug 61 or the connection attitude and the plug attitude do not match, the control device 40 instructs the robot 20 to feedback control the hand 24 so as to reduce the distance between the socket and plug and the discrepancy between the connection attitude and the plug attitude. During this process, the stereo camera 30 fixed to the hand 24 also approaches the socket, so that the accuracy of calculation of the three-dimensional coordinates of the socket etc. gradually improves.

[0075] (S37) If the determination result in step S32 indicates that the socket 63 or plug 61 is to be searched for, the field of view of the stereo camera is widened or moved, as in the first embodiment, to search for the socket or plug that is out of the field of view of the stereo camera.

[0076] By repeating the above steps, if it is determined in step S35 that the positions of the socket 63 and the plug 61 and the connection posture and the plug posture match, as described above, the plug is connected to the socket (S38) and the work is completed.

[0077] In addition, if the relative positional relationship between the stereo camera 30 and the plug 61 is known, it is not necessary to image the plug in step S31, determine whether the plug is included in the image in step S32, calculate the position and orientation of the plug in step S33, and search for the plug in step S37.

[0078] A fourth embodiment of the robot control system of the present invention will be described with reference to Figures 8 and 9. In this embodiment, in the task of connecting a characteristic portion on a workpiece gripped by a robot hand to a predetermined connection target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor installed away from the robot. Below, differences from the third embodiment will be described, and explanations of similarities to the third embodiment will be omitted.

[0079] The robot control system 14 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 installed away from the robot, a sensor moving means 32 for moving the stereo camera, and a control device 40 for overall control of the robot control system 14. The robot control system 14 connects a plug 61 (specific part) attached to the end of a cable 60 (workpiece) held by the robot hand 24 to a socket 63 (connection target) on a circuit board 62. During this connection operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

[0080] In this embodiment, the stereo camera 30 is installed away from the robot, so the relative positional relationship between the stereo camera 30 and the plug 61 is unknown regardless of the type of workpiece. Therefore, the stereo camera 30 acquires three-dimensional information about the socket 63 and the plug 61, and the control device 40 calculates the three-dimensional coordinates of the socket and plug positions based on the images acquired by the stereo camera 30. The control device 40 also determines whether the socket and plug are present in the image, and searches for the socket or plug if they are not visible in the image. The sensor moving means 32 changes the position and orientation of the stereo camera 30, which is a three-dimensional visual sensor. The structure of the sensor moving means 32 is the same as that described in the second embodiment.

[0081] The connection operation method by the robot control system 14 of this embodiment is the same as that of the third embodiment, in which the relative positional relationship between the stereo camera 30 and the plug 61 is unknown. However, in this embodiment, the stereo camera 30 is installed away from the robot, so in step S36 of FIG. 7, the hand 24 is moved and the control device 40 instructs the sensor moving means 32 to control the position and orientation of the stereo camera 30. Specifically, when the socket 63 or the plug 61 is on the periphery of the image, the field of view of the stereo camera is moved so as to move both toward the center of the image. Furthermore, as the hand approaches the socket, the distance between the stereo camera and the socket is reduced to improve the measurement accuracy of the position and orientation of the socket and plug.

[0082] A fifth embodiment of a robot control system of the present invention will be described with reference to Figures 10 and 11. In this embodiment, in the task of placing a workpiece grasped by a robot hand at a predetermined placement target, the movement of the robot hand is feedback-controlled based on information acquired by a three-dimensional visual sensor fixed to the robot hand. Below, only the parts that differ from the first and third embodiments will be described, and a description of the parts that are the same as the first or third embodiments will be omitted.

[0083] The robot control system 15 of this embodiment includes a robot 20 equipped with a robot hand 24, a stereo camera 30 fixed to the robot hand, and a control device 40 that controls the entire robot control system 15. The robot control system 15 places a cable 70 held by the robot hand 24 on a tray 72. During this placement operation, the position and orientation of the robot hand 24 are feedback-controlled by the control device 40 based on three-dimensional information acquired by the stereo camera 30.

[0084] In this embodiment, the cable 70 is described as an example of a workpiece and the tray 72 as an example of a placement target, but these types are not particularly limited. The robot control system 15 of this embodiment can be applied to tasks such as placing various types of workpieces in predetermined locations or storing them in predetermined containers. The tray 72 may be stationary, or may be placed on a conveyor belt 55 or the like and move independently of the operation of the hand 24.

[0085] In this embodiment, as in the third embodiment, if the workpiece does not deform, the relative positional relationship between the stereo camera 30 fixed to the hand 24 and the workpiece is known. On the other hand, if the workpiece is flexible or easily deformed, the relative positional relationship between the stereo camera and the workpiece is not constant and is unknown. The following describes a case where the relative positional relationship between the stereo camera and the workpiece is unknown.

[0086] The stereo camera 30 acquires three-dimensional information of the tray 72 and the cable 70. The robot 20 also serves as a sensor moving means for changing the position and posture of the stereo camera. The control device 40 performs image processing of the images acquired by the stereo camera 30, various calculations, feedback control of the robot 20, searching for the tray 72, etc. The details of the calculations and controls performed by the control device will be described later.

[0087] Next, the connection work using the robot control system 15 of this embodiment will be described along the process flow of FIG.

[0088] (S41) The tray 72 and the cable 70 are imaged by the stereo camera 30 to obtain three-dimensional information that enables the position and orientation of each to be calculated.

[0089] (S42) The control device 40 determines whether the tray 72 and the cable 70 are included in the stereo image acquired by the stereo camera 30, and if the tray or the cable is not within the field of view of the stereo camera 30 and is not included in the image, the control device 40 performs a search process (S47).

[0090] (S43) The control device 40 processes the stereo images to recognize the three-dimensional shapes of the tray 72 and the cable 70, and calculates the position and orientation of the tray and the position and orientation of the cable. Here, the position and orientation of the tray can be represented by the center position and orientation of the tray surface, and the position and orientation of the cable can be represented by the center position and orientation of a rectangle or rectangular parallelepiped that contains the cable and is large enough to fit inside the tray.

[0091] (S44) The control device 40 calculates the distance between the tray 72 and the cable 70 and the deviation in posture.

[0092] (S45) The control device 40 determines whether or not the position and posture of the tray 72 and the cable 70 match. If the position and posture of the tray and the cable match, the cable is placed on the tray (step S48).

[0093] (S46) If the positions or postures of the tray 72 and the cable 70 do not match, the control device 40 instructs the robot 20 to perform feedback control of the hand 24 so as to reduce the distance between the tray and the cable and the deviation in posture.

[0094] (S47) If the determination result in step S42 indicates that a tray 72 or a cable 70 is to be searched for, the field of view of the stereo camera is widened or moved, as in the first embodiment, to search for a tray or a cable that is out of the field of view of the stereo camera.

[0095] By repeating the above steps, if it is determined in step S45 that the positions and connections of the tray 72 and the cable 70 match, as described above, the cable is placed on the tray (S48) and the work is completed.

[0096] The sixth embodiment of the robot control system 16 of the present invention feedback controls the movement of the robot hand based on information acquired by a three-dimensional visual sensor installed away from the robot when placing a workpiece grasped by the robot hand at a predetermined placement target.

[0097] 12, the robot control system 16 of this embodiment differs from the fifth embodiment in that the stereo camera 30 is installed away from the robot 20 and attached to sensor moving means 32 independent of the robot, but the other configurations are the same as those of the fifth embodiment. Also, the cable arrangement work method using the robot control system 16 of this embodiment is carried out according to the process flow shown in FIG. 11, and differs from the fifth embodiment in that the movement of the stereo camera is controlled by the sensor moving means 32 rather than the robot 20, but the other configurations are the same as those of the fifth embodiment.

[0098] The present invention is not limited to the above-described embodiments and examples, and various modifications are possible within the scope of the technical concept thereof. [Explanation of symbols]

[0099] 11~16 Robot Control System 20. Robot 21 Arm 22 Links 23 Joints 24 Robot Hand 25 Tool Center Points 30 Stereo camera (3D visual sensor) 32 Sensor movement means 40 Control device 50 Cable (Work) 51 Cable end 52 Gripping position (work target) 55 Conveyor Belt 60 Cable (Work) 61 Plug (specific part) 62 PCB 63 Socket (Connection Target) 70 Cable (Work) 72 Tray (Placement Target) A Gripping direction B Connection direction d Distance from the cable tip to the gripping position T: Tangential direction of the cable at the gripping position X Hand Direction Y-plug orientation α Rotation angle of gripping posture β Rotation angle of connection posture θ Rotation angle of the robot hand φ Plug rotation angle

Claims

1. A system for controlling a robot that moves a robot hand to a gripping position of a flexible cable that is stationary or moves independently of the movement of the robot hand, comprising: a stereo camera that captures an image of the gripping position to acquire a stereo image; a sensor moving means for moving the stereo camera; a control device; the robot hand is capable of grasping the cable by pinching it, The control device Calculating three-dimensional coordinates of the gripping position in a camera coordinate system based on the stereo images, calculating a distance between the gripping position and the robot hand, and feedback-controlling the position of the robot hand so as to reduce the distance between the gripping position and the robot hand; calculating a tangential direction of the cable at the gripping position, a gripping direction intersecting the tangential direction, and a rotation angle around the gripping direction at which the robot hand can pinch the cable in the camera coordinate system based on the stereo images, thereby calculating a working posture for the robot hand to grip the gripping position, and feedback-controlling the posture of the robot hand so as to reduce a deviation between the working posture and the posture of the robot hand; determining whether the gripping position is within the field of view of the stereo camera, and if it is determined that the gripping position is outside the field of view, searching for the gripping position by widening or moving the field of view of the stereo camera, and controlling the sensor moving means so that the gripping position is included in the field of view of the stereo camera; Robot control system.

2. the stereo camera is fixed to the robot hand, the sensor moving means is the robot; The robot control system of claim 1 .

3. The stereo camera is installed at a distance from the robot. The robot control system of claim 1 .

Images