Robot control system

Abstract

To provide a robot control system that determines whether or not a target object is moved during when a robot hand approaches the target object, so as to enable handling the target object appropriately.SOLUTION: When gripping a workpiece by a robot hand 14, a robot control system according to the present invention, compares actual image information (actual image information of the workpiece obtained by imaging the workpiece by an imaging device 13 at a predetermined position closer to the workpiece from a position at a predetermined distance from the workpiece) with estimated image information (estimated image information of the workpiece which is estimated to be obtained by imaging the workpiece by the imaging device 13 at the predetermined position on the assumption that the workpiece is not moved until a robot arm 12 moves together with the imaging device 13 to the predetermined position), and, when the estimated image information and the actual image information coincide as a result of the comparison, executes an operation for gripping the workpiece using object information including distance information held in an object information holding unit 22a.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a robot control system. In particular, the present invention relates to an improvement in robot control when a robot handles an object (for example, when gripping an object or processing an object).

Background Art

[0002] Conventionally, as a robot control system for gripping and transferring an object using a robot, for example, those disclosed in Patent Document 1 and Patent Document 2 are known. The robot control systems disclosed in these patent documents grip an object while measuring the distance to the object (while recognizing the position of the object) by imaging the object with an imaging device provided at the tip of a robot arm (for example, by gripping with a robot hand provided at the tip of the robot arm).

[0003] In particular, Patent Document 1 discloses that when a feature portion of an object cannot be specified by pattern matching in an image obtained by an imaging device, the robot arm is operated so that the relative position or relative posture between the object, the imaging device, and the lighting device changes, the object is imaged again, and pattern matching is performed again on the image thus obtained.

[0004] Further, Patent Document 2 discloses analyzing a sensing result by an imaging device (sensor unit) provided on a robot arm, specifying an object to be gripped in the current gripping operation, controlling the operation of the robot arm so as to grip the object to be gripped in the current gripping operation, analyzing the sensing result, specifying a sensing position for the next gripping operation, and updating sensing position information.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] However, none of the above patent documents consider the situation where the object moves while the robot arm is approaching the object. As an imaging device provided at the tip of the robot arm, a 3D camera is generally adopted. However, the range within which this 3D camera can measure the distance to the object is limited. That is, a 3D camera that measures the distance to the object using triangulation cannot measure the distance to the object when the distance to the object becomes shorter than a predetermined distance because the parallax becomes too large. Hereinafter, the range of the distance to the object within which the distance to the object can be measured will be referred to as the measurable range. This measurable range exists not only in passive imaging devices but also in active imaging devices.

[0007] When the object does not move while the robot arm is approaching the object, it is possible to grasp the object well by using the information on the distance to the object (information on the position of the object) measured within the measurable range, the information on the shape of the object, etc. However, if the object moves when the imaging device is outside the measurable range while the robot arm is approaching the object, it becomes impossible to grasp the object by using the information on the distance to the object measured within the measurable range, and in some cases, there is also a risk that the robot hand may collide with the object and cause an adverse effect on the object.

[0008] Such problems may occur not only in collaborative robots typified by robots that grip and transfer objects but also in industrial robots. Further, such problems may occur not only when an object is gripped by a robot hand but also in various operations of handling the object, such as processing the object.

[0009] The present invention has been made in view of such points, and an object thereof is to provide a robot control system that determines whether or not the object has moved while the robot arm is approaching the object and enables the robot to perform an operation according to the determination.

Means for Solving the Problem

[0010] The solution means of the present invention for achieving the above object is premised on a robot control system that captures an object by an imaging device provided on a robot arm of the robot and controls the robot when handling the object based on the image information of the object obtained by the imaging. And, This robot control system has a measurable range in which the imaging device can measure the distance to the object, and a non-measurable range in which the distance to the object is shorter than the measurable range and the distance to the object cannot be measured. Also, this robot control system includes an object information holding unit, a predicted image information generation unit, a comparison unit, and a control switching unit. The object information holding unit stores the imaging device at a predetermined distance from the object. Located within the measurable range It holds object information including the distance information to the object obtained based on the image information obtained by imaging with the imaging device in the set state. The estimated image information generation unit, when assuming that the object has not moved until the robot arm moves together with the imaging device from a position existing at the predetermined distance from the object to a predetermined position approaching the object, generates the estimated image information of the object that is estimated to be obtained by imaging with the imaging device at the predetermined position based on the object information held in the object information holding unit. The comparison unit compares the actual image information of the object obtained by imaging with the imaging device at the predetermined position with the estimated image information generated by the estimated image information generation unit. When the robot handles the object, the control switching unit, as a comparison result of the comparison unit, when the actual image information and the estimated image information match, performs an operation of handling the object using the distance information held in the object information holding unit, while when the actual image information and the estimated image information do not match, does not perform the operation of handling the object using the distance information held in the object information holding unit. When the imaging device intermittently captures images while approaching the object, in the comparison of the real image information and the estimated image information in the comparison unit when the imaging is performed, on the condition that the real image information and the estimated image information of the object match and the imaging device is located within the measurable range, it includes an image information update unit that updates the object information held in the object information holding unit. Furthermore, the image information update unit When a plurality of image information is obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the respective image information is synthesized, and the object information including the distance information to the object obtained based on the synthesized image information is held in the object information holding unit. Yes.

[0011] The "estimated image information" mentioned here is a composite image (image information obtained by synthesizing a plurality of image information) described later or one or more sets of feature points sufficient to specify the position of the object. Further, the "the actual image information and the estimated image information match" mentioned here is a concept including both the case where these information (features of the information) match and the case where these information are regarded as matching.

[0012] Due to this specific matter, when the robot handles an object, the real image information (the real image information of the object obtained by imaging with an imaging device at a predetermined position approaching the object from a position existing at a predetermined distance from the object) and the estimated image information (when it is assumed that the object has not moved until the robot arm moves to the predetermined position together with the imaging device, the estimated image information (the above-mentioned feature point set) of the object that is estimated to be obtained by imaging with the imaging device at the predetermined position) are compared. As a result of the comparison, when the estimated image information and the real image information match or are considered to match (in other words, when the features of the estimated image information and the features of the real image information match or are considered to match), the object has not moved, and even if the real image information is information that cannot obtain the distance information to the object (even if it is information obtained in a state outside the measurable range), the distance information (the distance information held in the object information holding unit) can be used to appropriately handle the object. Therefore, an operation of handling the object using this distance information is performed. On the other hand, when the estimated image information and the real image information do not match (including cases where they are considered not to match), there is a possibility that the object has moved, and there is a possibility that the object cannot be appropriately handled even if the distance information is used. Therefore, the operation of handling the object using the distance information is not performed. As a result, it becomes possible to appropriately handle the object while determining whether the object has moved during the process of the robot arm approaching the object. Also, for example, when there are defects in a part of each image information and the defective parts are different from each other, the images (image information) of the defective parts are synthesized so as to be complemented by other images (images of non-defective parts). As a result, high-precision image information is generated. By causing the object information holding unit to hold the object information including the distance information to the object obtained based on this image information, the distance to the object can be recognized with high precision, and the accuracy of the operation of handling the object can be increased. Another solution means of the present invention for achieving the above object is, in addition to the object information holding unit, the estimated image information generation unit, the comparison unit, and the control switching unit described above, as a comparison result of the comparison unit, when the real image information and the estimated image information match, a viewpoint change command unit for causing the imaging device to image the object from a direction different from the imaging direction of the imaging device that obtained the real image information. For example, when gripping an object, the gripping position is preferably the center position or the center of gravity position of the object. For that purpose, it is necessary to grasp the overall shape of the object. Therefore, by causing the imaging device to image the object from a direction different from the imaging direction of the imaging device that obtained the real image information, the overall shape of the object can be grasped, and an operation of handling the object (for example, an operation of gripping the center position or the center of gravity position of the object) can be appropriately performed. Another solution means of the present invention for achieving the above object is based on a robot control system that controls the robot when imaging an object by an imaging device provided on a robot arm of the robot Object and performing handling of the object based on the image information of the object obtained by the imaging. And this robot control system includes an object information holding unit, an estimated image information generation unit, a comparison unit, and a control switching unit. Also, the imaging device has a measurable range in which the distance to the object can be measured, and a non-measurable range in which the distance to the object is shorter than the measurable range and the distance to the object cannot be measured. The object information holding unit is obtained by imaging by the imaging device in a state where the imaging device exists at a predetermined distance from the object Located within the measurable range Including the distance information to the object obtained based on the image information ​It holds object information. When it is assumed that the object has not moved during the movement of the robot arm together with the imaging device from a position at the predetermined distance from the object to a predetermined position approaching the object, the estimated image information generation unit estimates the object's estimated image information that is presumed to be obtained by imaging with the imaging device at the predetermined position, based on the object information held in the object information holding unit, and the information on the relative movement direction, movement amount, and posture change of the imaging device with respect to the object information when the imaging device approaches the object up to the predetermined position. The comparison unit compares the actual image information of the object obtained by imaging with the imaging device at the predetermined position with the estimated image information generated by the estimated image information generation unit. When the robot handles the object, the control switching unit, as a comparison result of the comparison unit, if the actual image information and the estimated image information match, performs an operation of handling the object using the distance information held in the object information holding unit; on the other hand, if the actual image information and the estimated image information do not match, it does not perform the operation of handling the object using the distance information held in the object information holding unit. Furthermore, when the imaging device intermittently captures images while approaching the object, in the comparison of the real image information and the estimated image information in the comparison unit when the imaging is performed, on the condition that the real image information and the estimated image information of the object match and the imaging device is located within the measurable range, it includes an image information update unit that updates the object information held in the object information holding unit.

[0013] More specifically, when the operation of handling the object is not performed by the control switching unit, the robot control system discards the object information held in the object information holding unit, and re-performs the operation of holding the object information including the distance information in the object information holding unit by performing imaging with the imaging device set at a position at a predetermined distance from the object.

[0014] That is, when there is a possibility that the object has moved, the robot control system determines that there may be a possibility that the object cannot be properly handled even if the distance information held in the object information holding unit is used, and for the object after this movement, newly obtains object information including distance information and holds it in the object information holding unit. Thereby, it becomes possible to properly handle the object even after the object has moved.

[0015] In addition, when a plurality of pieces of image information are obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the robot control system preferably includes an image information update unit that causes the object information holding unit to hold the object information including the distance information to the object obtained based on the image information obtained at the position where the distance from the object is short.

[0016] The image (image information) obtained by imaging with the imaging device can be obtained as clearer image information as the distance from the object is shorter. For this reason, by causing the object information holding unit to hold the object information including the distance information to the object obtained based on the image information obtained at the position where the distance from the object is short, the distance to the object can be recognized with high accuracy, and the accuracy of the operation of handling the object can be increased.

[0017] In addition, when a plurality of pieces of image information are obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the robot control system may include an image information update unit that selects highly accurate image information and causes the object information holding unit to hold the object information including the distance information to the object obtained based on the image information.

[0018] Also in this configuration, the distance to the object can be recognized with high accuracy, and the accuracy of the operation of handling the object can be increased.

Advantages of the Invention

[0023] In the present invention, when the robot handles an object, the estimated image information and the actual image information are compared, and as a result of the comparison, when the estimated image information and the actual image information match, the operation of handling the object is performed using the distance information held in the object information holding unit. For this reason, it becomes possible to appropriately handle the object while determining whether or not the object has moved while the robot arm is approaching the object.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0025] Hereinafter, embodiments of the present invention will be described with reference to the drawings. This embodiment describes a case where the robot control system according to the present invention is applied as a system that controls a robot hand provided at the tip of a robot arm in a collaborative robot to grip an object (hereinafter, may also be referred to as a workpiece). Note that the application form of the robot control system according to the present invention is not limited to this.

[0026] -Configuration of the Robot Unit- FIG. 1 is a diagram for explaining the schematic configuration of the robot unit 1 to which the robot control system 20 (see FIG. 2) according to the present embodiment is applied and the object imaging operation. This robot unit 1 includes a robot 11 that performs an operation of gripping and transferring a workpiece W placed on a workbench (not shown) and an imaging device 13 attached to the tip of a robot arm 12 of the robot 11. The robot unit 1 is configured to save space by attaching the imaging device 13 to the tip of the robot arm 12.

[0027] The robot unit 1 is applied to a cell production line. For example, a workpiece W placed at a predetermined position (pick position) on a first workbench is transferred by the robot 11 from the first workbench to a predetermined position (place position) on a second workbench, and is applied when an operator performs an assembly operation of the workpiece W on this second workbench.

[0028] The robot 11 includes a robot arm 12 composed of a multi-axis arm and a robot hand 14 as an end effector provided at the tip of the robot arm 12. The robot arm 12 is provided to move the robot hand 14 to a predetermined position, and the robot hand 14 is provided to grip the workpiece W.

[0029] The imaging device 13 attached to the tip of the robot arm 12 (near the attachment position of the robot hand 14) is composed of a three-dimensional camera such as an RGB-D camera, for example, and is for imaging the workpiece W placed on the workbench, and outputs information (image information) of the captured image to a control device 21 (see FIG. 2) of the robot control system 20. As will be described later, the image information captured by this imaging device 13 recognizes the position (position obtained based on the distance to the workpiece W) and shape of the workpiece W, and is used as information for gripping the workpiece W.

[0030] Since the imaging device 13 is a three-dimensional camera, as described above, the range within which the distance to the workpiece W can be measured is limited to the measurable range. For this reason, if the workpiece W moves while the robot arm 12 is approaching the workpiece W to grip it with the robot hand 14 and the imaging device 13 is at a position outside the measurable range, it becomes impossible to grip the workpiece W using the distance information to the workpiece W measured within the measurable range.

[0031] In view of this problem, in the robot control system 20 according to the present embodiment, it is determined whether or not the workpiece W has moved while the robot arm 12 is approaching the workpiece W, and the robot 11 is controlled accordingly. Hereinafter, the robot control system 20 will be specifically described.

[0032] FIG. 2 is a block diagram showing a control system of the robot control system 20 according to the present embodiment. As shown in this FIG. 2, as a control system of the robot control system 20, a control device 21 for controlling the robot unit 1 is provided. This control device 21 is composed of a computer including an arithmetic unit (a processor such as a CPU) 22, a storage unit (such as a ROM) 23, and an input / output unit 24.

[0033] The storage unit 23 stores an operation program and the like for controlling the robot unit 1.

[0034] The arithmetic unit 22 calculates control command information for controlling the robot unit 1 by executing arithmetic processing based on a program (operation program) and the like stored in the storage unit 23.

[0035] The input / output unit 24 can receive image information obtained by imaging by the imaging device 13 and transmit control command information and the like to the robot unit 1.

[0036] The arithmetic unit 22 includes, as functional units realized by the program, an object information holding unit 22a, an estimated image information generation unit 22b, a comparison unit 22c, an image information update unit 22d, a viewpoint change command unit 22e, and a control switching unit 22f. Hereinafter, an outline of the functions of each of these units will be described.

[0037] The object information holding unit 22a holds information (object information) of the work W including distance information to the work W obtained based on the image information obtained by imaging with the imaging device 13 in a state where the imaging device 13 is positioned at a predetermined distance from the work W.

[0038] Specifically, the object information holding unit 22a controls the operation of the robot arm 12 (controls by transmitting control command information to the robot unit 1) so that the imaging device 13 is positioned within the measurable range, and acquires the image information obtained by imaging the work W with the imaging device 13 at that position via the input / output unit 24. Hereinafter, the imaging operation at this position at the start of the gripping operation of the work W will be referred to as the initial imaging operation. Since the measurable range of the imaging device 13 is recognized in advance, the posture of the robot arm 12 in which the work W is positioned within the field of view of the imaging device 13 and the distance to the work W is within the measurable range can also be set in advance. By transmitting control command information from the control device 21 to the robot unit 1 so as to obtain this posture, the operation of the robot arm 12 is controlled so that the imaging device 13 is positioned within the measurable range.

[0039] Then, the imaging device 13 is made to perform an imaging operation (initial imaging operation), the distance to the work W is calculated from the image information obtained by the initial imaging operation using well-known triangulation, and the distance information is held by the object information holding unit 22a. Also, information on the posture of the work W and position information of each point on the outer edge of the work W and the like are also held together from the image information of the work W.

[0040] FIG. 1 shows the object imaging operation (initial imaging operation) when the imaging device 13 is located within the measurable range. Further, FIG. 3 shows an example of an image of the work W captured by the imaging device 13 in the state shown in FIG. 1. In this image, information such as the distance to the work W and the shape of the work W can be acquired, and this information (object information) will be held in the object information holding unit 22a. Also, this object information holding unit 22a is configured to hold (update) the object information in the same manner as described above even when the imaging device 13 intermittently performs imaging multiple times while approaching the work W. This update operation will be described later.

[0041] The estimated image information generation unit 22b generates estimated image information (the above-described feature point set) of the work W that is estimated to be obtained by imaging with the imaging device 13 at the predetermined position, assuming that the work W has not moved during the period from a position at a predetermined distance from the work W to a predetermined position approaching the work W while the robot arm 12 moves together with the imaging device 13.

[0042] Specifically, based on the information on the movement amount of the imaging device 13 when approaching the work W with respect to the position of the imaging device 13 during the initial imaging operation, image information of the work W is generated as estimated image information assuming that the work W has not moved. For example, since the position information of each point on the outer edge etc. of the work W is held, when assuming that the work W has not moved, the movement direction and movement amount of the position of each point when approaching the work W (the relative movement direction and movement amount of each point obtained by the initial imaging operation) can be calculated from the control command information. Therefore, it is possible to generate the estimated image information when approaching the work W assuming that the work W has not moved.

[0043] The comparison unit 22c compares the actual image information obtained by imaging with the imaging device 13 at the predetermined position approaching the workpiece W and the estimated image information generated by the estimated image information generation unit 22b. When the workpiece W is not moving, the estimated image information and the actual image information are substantially the same image (substantially matching images). On the other hand, when the workpiece W has moved, the estimated image information and the actual image information are different images. Therefore, by comparing these image informations, it is possible to determine whether the workpiece W has moved while approaching the workpiece W (while the robot arm 12 is moving together with the imaging device 13 from a position having a predetermined distance from the workpiece W to the predetermined position approaching the workpiece W).

[0044] FIG. 4 shows the object imaging operation (object imaging operation after changing the robot posture) by the imaging device 13 at the predetermined position approaching the workpiece W. The actual image information is to be acquired by this object imaging operation.

[0045] FIG. 5 is a diagram for explaining the image comparison operation. The image on the right side in FIG. 5 is the estimated image information generated by the estimated image information generation unit 22b, and the image on the left side in FIG. 5 shows the image of the workpiece W (actual image information) imaged by the imaging device 13 in the state shown in FIG. 4. As shown in FIG. 5, when the estimated image information and the actual image information are substantially the same image, it can be determined that the workpiece W is not moving.

[0046] FIG. 6 shows an image when the workpiece W moves. The image on the right side in FIG. 6 is the estimated image information generated by the estimated image information generation unit 22b, and the image on the left side in FIG. 6 shows the image (actual image information) of the workpiece W captured by the imaging device 13 in the state shown in FIG. 4. As shown in FIG. 6, when the estimated image information and the actual image information do not match (including the case where they are considered not to match), it can be determined that the workpiece W has moved. This determination as to whether the workpiece W has moved can be made based on the image information obtained when the predetermined position approaching the workpiece W is within the measurable range of the imaging device 13 and also based on the image information obtained when the position is outside the measurable range of the imaging device 13.

[0047] In the comparison process in the comparison unit 22c, when the estimated image information and the actual image information are substantially the same image information, that is, when it is determined that the workpiece W has not moved, the image information update unit 22d determines whether to update the image information held in the object information holding unit 22a, and updates the held image information as necessary.

[0048] Specifically, the predetermined position approaching the workpiece W may be a position where the imaging device 13 is within the measurable range or a position where the imaging device 13 is outside the measurable range. Since the measurable range of the imaging device 13 is recognized in advance as described above, the posture of the robot arm 12 can be grasped from the control command information transmitted to the robot unit 1, and thereby the position of the imaging device 13 can also be specified. Therefore, it is possible to determine whether the imaging device 13 is within the measurable range.

[0049] When the estimated image information and the actual image information are substantially the same image and the imaging device 13 is within the measurable range, the image information update unit 22d determines whether to update the image information held in the object information holding unit 22a.

[0050] Specifically, the previous image information captured by the imaging device 13 (for example, the image information obtained by the initial imaging operation) and the current image information (for example, the image information obtained by the imaging operation at a position closer to the workpiece W than the position where the initial imaging operation was performed) are compared. That is, when a plurality of image information is obtained by imaging with the imaging device 13 at each position where the distances from the workpiece W are different from each other and distance information can be acquired, these image information are compared. Then, among these image information, the object information including the distance information to the workpiece W obtained based on the image information obtained at the position where the distance from the workpiece W is short is held (updated) in the object information holding unit 22a. This is because generally, the image (image information) obtained by imaging with the imaging device 13 can be obtained as clearer image information as the distance from the workpiece W is shorter. That is, by causing the object information holding unit 22a to hold the object information including the distance information to the workpiece W obtained based on the image information obtained at the position where the distance from the workpiece W is short, the distance to the workpiece W can be recognized with high accuracy.

[0051] Further, as the image information updating unit 22d, the previous image information and the current image information captured by the imaging device 13 may be compared, high-precision image information may be selected, and the object information including the distance information to the workpiece W obtained based on the image information may be caused to be held in the object information holding unit 22a. For example, the previous image information and the current image information are compared, clear image information (image information in focus, image information without information loss caused by light reflection, etc.) is selected, and the object information including the distance information to the workpiece W obtained based on the image information is caused to be held in the object information holding unit 22a. Also by this, the distance to the workpiece W can be recognized with high accuracy.

[0052] Further, as the image information update unit 22d, the previous image information and the current image information captured by the imaging device 13 are compared. If there are defects in some parts of the respective image information (for example, defects in image information caused by light reflection, etc.) and the defective parts are different from each other, the images of the defective parts may be complemented by other images (images of parts without defects). That is, the previous image information and the current image information may be combined to generate new image information, and the object information including the distance information to the work W obtained based on this image information may be held in the object information holding unit 22a. FIG. 7 is a diagram for explaining the synthesis process of two images. The lower left image in FIG. 7 is the previous image information (for example, the image information obtained by the initial imaging operation), and the upper left image is the current image information (for example, the image information obtained by the imaging operation at a position closer to the work W than the position where the initial imaging operation was performed). Although not shown in the figure, there are defects in some parts of these image informations. By synthesizing these images, the images of the defective parts are complemented by other images, and the right image (synthesized image) in FIG. 7 is generated. As a result, highly accurate image information is generated, and by holding the object information including the distance information to the work W obtained based on this image information in the object information holding unit 22a, the distance to the work W can be recognized with high accuracy.

[0053] When the imaging device 13 intermittently performs imaging a plurality of times while approaching the work W, the update of the image information held in the object information holding unit 22a by the above-described image information update unit 22d is performed as follows: If all of these multiple imaging operations are performed in a state where the imaging device 13 is located within the measurable range, each time imaging is performed, it is determined whether to update the image information held in the object information holding unit 22a, and the held image information is updated as necessary.

[0054] When the viewpoint change instruction unit 22e determines as a comparison result of the comparison unit 22c that the estimated image information and the actual image information are substantially the same and the workpiece W is not moving, the viewpoint change instruction unit 22e has a function of transmitting control command information for causing the imaging device 13 to image the workpiece W from a direction different from the imaging direction of the imaging device 13 from which the actual image information was obtained to the robot unit 1. That is, when gripping the workpiece W with the robot hand 14, it is preferable to grip the center position and the center of gravity position of the workpiece W in order to stably obtain the gripping state. For that purpose, it is necessary to grasp the overall shape of the workpiece W. The viewpoint change instruction unit 22e is a functional unit for grasping the overall shape of the workpiece W by setting the imaging direction of the workpiece W by the imaging device 13 to a direction different from the above-described direction. FIG. 8 is a diagram showing an example of an image of the workpiece W obtained by the first different viewpoint imaging operation. What is shown in this FIG. 8 is an image obtained by causing the imaging device 13 to wrap around to the head side of the workpiece (bolt) W and imaging the workpiece W. Further, FIG. 9 is a diagram showing an example of an image of the workpiece W obtained by the second different viewpoint imaging operation. What is shown in this FIG. 9 is an image obtained by causing the imaging device 13 to wrap around to the threaded portion side of the workpiece (bolt) W and imaging the workpiece W. In this way, the workpiece W is imaged from multiple aspects to grasp the overall shape, so that the center position and the center of gravity position of the workpiece W can be gripped.

[0055] When the robot hand 14 grips the workpiece W, the control switching unit 22f, as a comparison result of the comparison unit 22c, implements an operation of gripping the workpiece W using the distance information held in the object information holding unit 22a when the estimated image information and the actual image information match (including cases where they are considered to be substantially the same). That is, when the actual image information captured at a position outside the measurable range of the imaging device 13 and the estimated image information match, although distance information cannot be obtained from this actual image information, since it is determined that the workpiece W has not moved, it is possible to grip the workpiece W using the distance information held in the object information holding unit 22a. Therefore, an operation of gripping the workpiece W using the distance information held in the object information holding unit 22a is performed. More specifically, in the case of this embodiment, the workpiece W placed at a predetermined position on the first workbench is gripped by the robot hand 14 and transferred to a predetermined position on the second workbench by the operation of the robot arm 12.

[0056] On the other hand, when the comparison result of the comparison unit 22c is that the estimated image information and the actual image information do not match (including cases where they are considered not to match), the control switching unit 22f does not implement an operation of gripping the workpiece W using the distance information held in the object information holding unit 22a. In this case, the object information held in the object information holding unit 22a is discarded, the above-described initial imaging operation is performed again, and an operation of holding the object information including the distance information in the object information holding unit 22a is performed.

[0057] -Object Gripping Operation- Next, the object gripping operation by the robot control system 20 configured as described above will be explained. FIG. 10 is a flowchart showing the processing procedure of the object gripping operation. This flowchart is repeatedly executed each time a gripping request for the workpiece W (for example, a request to transfer the workpiece W from the first workbench to the second workbench) occurs.

[0058] When a gripping requirement for the workpiece W occurs, first, an initial imaging operation is performed in step ST1. That is, the operation of the robot arm 12 is controlled so that the imaging device 13 is positioned within the measurable range, and image information is obtained by imaging the workpiece W with the imaging device 13 at that position. In step ST2, the object information including the distance information to the workpiece W obtained based on this image information is transmitted to the object information holding unit 22a via the input / output unit 24 and held in the object information holding unit 22a.

[0059] In step ST3, the posture of the robot arm 12 is changed so as to approach the workpiece W. Then, in step ST4, assuming that the workpiece W has not moved while the robot arm 12 moves with the imaging device 13 to a predetermined position of the imaging device 13 after the posture change, estimated image information that is estimated to be obtained by imaging with the imaging device 13 at the predetermined position is generated.

[0060] In step ST5, imaging with the imaging device 13 at the predetermined position is executed, and the actual image information obtained by the imaging is acquired.

[0061] In step ST6, the estimated image information and the actual image information are compared. If these images are different from each other (the images are different from each other because the workpiece W has moved) and a NO determination is made in step ST7, the process proceeds to step ST8, where the object information held in the object information holding unit 22a is discarded and the process returns. As a result, the above-described initial imaging operation is performed again, and the operation of holding the object information including the distance information of the workpiece W after movement in the object information holding unit 22a is performed.

[0062] If the estimated image information and the actual image information match and a YES determination is made in step ST7, the process proceeds to step ST9 to determine whether the current position of the imaging device 13 is within the measurable range. If the current position of the imaging device 13 is within the measurable range and a YES determination is made in step ST9, then in step ST10, after performing the image information update process by the aforementioned image information update unit 22d, the process returns to step ST3. In this way, while imaging is being performed by the imaging device 13 in a state where the position of the imaging device 13 is within the measurable range, the object information held in the object information holding unit 22a is sequentially updated, and object information including highly accurate distance information will be held in the object information holding unit 22a.

[0063] On the other hand, if a NO determination is made in step ST9 because the current position of the imaging device 13 is outside the measurable range, then in step ST11, an image from a different viewpoint is acquired by setting the imaging direction of the work W by the imaging device 13 to a direction different from the aforementioned direction. For example, the aforementioned first different viewpoint imaging operation (the operation of acquiring an image of the work W shown in FIG. 8) or the aforementioned second different viewpoint imaging operation (the operation of acquiring an image of the work W shown in FIG. 9) is performed. This makes it possible to grasp the overall shape of the work W and to grip the center position and the centroid position of the work W.

[0064] Thereafter, in step ST12, the object information including the distance information held in the object information holding unit 22a is read out, and a gripping operation of the work W is executed using this object information, and the transfer operation of the work W is started.

[0065] In step ST13, it is determined whether the transfer operation of the work W has ended. If the transfer operation of the work W has ended and a YES determination is made in step ST13, then the robot arm 12 returns to the initial position and is retracted. The above operations are repeated.

[0066] -Effects of the Embodiment- As described above, in the present embodiment, when the robot hand 14 grips the workpiece W, the real image information (the real image information of the workpiece W obtained by imaging with the imaging device 13 at a predetermined position approaching the workpiece W from a position having a predetermined distance from the workpiece W) and the estimated image information (when it is assumed that the workpiece W has not moved until the robot arm 12 moves together with the imaging device 13 to the predetermined position, the estimated image information of the workpiece W that is estimated to be obtained by imaging with the imaging device 13 at the predetermined position) are compared, and as a result of the comparison, when the estimated image information and the real image information match or are considered to match, the operation of gripping the workpiece W is performed using the object information including the distance information held in the object information holding unit 22a. For this reason, it is possible to appropriately grip the workpiece W while determining whether or not the workpiece W has moved while the robot arm 12 is approaching the workpiece W.

[0067] -Other Embodiments- Note that the present invention is not limited to the above-described embodiment, and all modifications and applications included in the scope of the claims and the scope equivalent thereto are possible.

[0068] For example, in the above-described embodiment, the case where the present invention is applied as the robot control system 20 for controlling the collaborative robot has been described. The present invention is not limited to this, and it can also be applied to industrial robots. Further, in the above-described embodiment, the case of performing control to grip the workpiece W with the robot hand 14 has been described as an example. The present invention is not limited to this, and it can also be applied to various operations for handling the workpiece W, such as processing the workpiece W.

[0069] Also, in the above-described embodiment, the imaging device 13 is of a passive type. The present invention is not limited to this, and the imaging device 13 may be of an active type.

[0070] Further, in the above-described embodiment, the viewpoint change command unit 22e is provided as a functional unit of the robot control system 20, and when the estimated image information and the actual image information match, the imaging device 13 captures an image of the workpiece W from different directions. The present invention is not limited to this, and a configuration in which the viewpoint change command unit 22e is not provided, that is, a configuration in which the imaging device 13 does not capture an image of the workpiece W from different directions may be employed.

[0071] Further, as the comparison operation by the comparison unit 22c, the composite image (new image information obtained by combining the previous image information and the current image information: see FIG. 7) and the estimated image information may be compared. In this case, it is necessary to make the position of the workpiece W in the composite image coincide with the position of the workpiece W when generating the estimated image information (the position of the workpiece W assuming that the workpiece W has not moved). Therefore, the "actual image information of the object" as used in the present invention is a concept that includes the composite image generated from a plurality of pieces of actual image information.

Industrial Applicability

[0072] The present invention is applicable to a robot control system that holds an object while measuring the distance to the object by imaging the object with an imaging device provided at the tip of a robot arm.

Explanation of Signs

[0073] 11 Robot 12 Robot Arm 13 Imaging Device 20 Robot Control System 22a Object Information Holding Unit 22b Estimated Image Information Generation Unit 22c Comparison Unit 22d Image Information Update Unit 22e Viewpoint Change Command Unit 22f Control Switching Unit W Workpiece (Object)

Claims

1. A robot control system for controlling a robot when imaging an object by an imaging device provided on a robot arm of the robot and handling the object based on image information of the object obtained by the imaging, comprising: The imaging device has a measurable range that is a range in which the distance to the object can be measured, and a non-measurable range that is a range in which the distance to the object is shorter than the measurable range and the distance to the object cannot be measured. An object information holding unit that holds object information including distance information to the object obtained based on image information obtained by imaging with the imaging device in a state where the imaging device is located in the measurable range at a predetermined distance from the object. When it is assumed that the object has not moved until the robot arm moves together with the imaging device from a position at the predetermined distance from the object to a predetermined position closer to the object, estimated image information of the object estimated to be obtained by imaging with the imaging device at the predetermined position is generated based on the object information held in the object information holding unit. An estimated image information generation unit. A comparison unit that compares the actual image information of the object obtained by imaging with the imaging device at the predetermined position with the estimated image information generated by the estimated image information generation unit. When the robot handles the object, as a comparison result of the comparison unit, when the actual image information and the estimated image information match, an operation of handling the object is performed using the distance information held in the object information holding unit. On the other hand, when the actual image information and the estimated image information do not match, a control switching unit that does not perform an operation of handling the object using the distance information held in the object information holding unit. When the imaging device intermittently images while approaching the object, in the comparison of the actual image information and the estimated image information in the comparison unit when the imaging is performed, the actual image information and the estimated image information of the object match. And an image information update unit that updates the object information held in the object information holding unit on the condition that the imaging device is located within the measurable range. Furthermore, when a plurality of pieces of image information are obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the image information updating unit synthesizes each piece of image information and causes the object information holding unit to hold object information including the distance information to the object obtained based on the synthesized image information. A robot control system characterized by this.

2. A robot control system for controlling a robot when the robot handles an object based on image information of the object obtained by imaging the object with an imaging device provided on a robot arm of the robot, The imaging device has a measurable range that is a range in which the distance to the object can be measured, and a non-measurable range that is a range in which the distance to the object is shorter than the measurable range and the distance to the object cannot be measured. An object information holding unit that holds object information including distance information to the object obtained based on image information obtained by imaging with the imaging device in a state where the imaging device is located in the measurable range at a predetermined distance from the object; When it is assumed that the object has not moved until the robot arm moves together with the imaging device from a position at the predetermined distance from the object to a predetermined position closer to the object, the estimated image information of the object that is estimated to be obtained by imaging with the imaging device at the predetermined position, An estimated image information generation unit that generates the estimated image information based on the object information held in the object information holding unit; A comparison unit that compares the actual image information of the object obtained by imaging with the imaging device at the predetermined position and the estimated image information generated by the estimated image information generation unit; When the robot handles the object, as a comparison result of the comparison unit, when the actual image information and the estimated image information match, an operation of handling the object using the distance information held in the object information holding unit is performed. On the other hand, when the actual image information and the estimated image information do not match, a control switching unit that does not perform an operation of handling the object using the distance information held in the object information holding unit. When the imaging device intermittently performs imaging while approaching the object, in the comparison of the actual image information and the estimated image information in the comparison unit when the imaging is performed, on the condition that the actual image information and the estimated image information of the object match and the imaging device is located within the measurable range, an image information update unit that updates the object information held in the object information holding unit. A robot control system characterized by comprising a viewpoint change command unit that causes the imaging device to image the object from a direction different from the imaging direction of the imaging device when the actual image information and the estimated image information match as a comparison result of the comparison unit.

3. A robot control system for controlling a robot when the robot images an object with an imaging device provided on a robot arm of the robot and handles the object based on the image information of the object obtained by the imaging. The imaging device has a measurable range that is a range in which the distance to the object can be measured, and a non-measurable range that is a range in which the distance to the object is shorter than the measurable range and the distance to the object cannot be measured. An object information holding unit that holds object information including distance information to the object obtained based on image information obtained by imaging with the imaging device in a state where the imaging device is located in the measurable range at a predetermined distance from the object. An estimated image information generation unit that generates estimated image information of the object that is estimated to be obtained by imaging with the imaging device at the predetermined position, assuming that the object has not moved during the movement of the robot arm together with the imaging device from a position at the predetermined distance from the object to a predetermined position closer to the object. The estimated image information is generated based on the object information held in the object information holding unit, and information on the relative movement direction, movement amount, and posture change of the imaging device with respect to the object information when the imaging device approaches the object to the predetermined position. A comparison unit that compares the actual image information of the object obtained by imaging with the imaging device at the predetermined position and the estimated image information generated by the estimated image information generation unit. When the robot handles the object, as a comparison result of the comparison unit, if the real image information and the estimated image information match, an operation of handling the object is performed using the distance information held in the object information holding unit. On the other hand, if the real image information and the estimated image information do not match, a control switching unit that does not perform the operation of handling the object using the distance information held in the object information holding unit, When the imaging device intermittently performs imaging while approaching the object, in the comparison of the real image information and the estimated image information in the comparison unit when the imaging is performed, the real image information and the estimated image information of the object match, and an image information updating unit that updates the object information held in the object information holding unit on the condition that the imaging device is located within the measurable range, A robot control system characterized by comprising the same.

4. In the robot control system according to claim 1, 2, or 3, When the operation of handling the object is not performed by the control switching unit, the object information held in the object information holding unit is discarded, and imaging is performed with the imaging device at a position having a predetermined distance from the object, and the object information including the distance information is held again in the object information holding unit. A robot control system characterized by having a configuration for performing the operation,

5. In the robot control system according to claim 2 or 3, When a plurality of pieces of image information are obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the image information updating unit holds, in the object information holding unit, the object information including the distance information to the object obtained based on the image information obtained at the position where the distance from the object is short. A robot control system characterized by the above,

6. In the robot control system according to claim 2 or 3, When a plurality of pieces of image information are obtained by imaging with the imaging device at each position where the distances from the object are different from each other and the distance information can be acquired, the image information updating unit selects high-precision image information, and holds, in the object information holding unit, the object information including the distance information to the object obtained based on the selected image information. A robot control system characterized by the above,

Images