Control device for robot device that acquires three-dimensional position information and robot device

Abstract

This control device for a robot device comprises: an imaging control unit that changes exposure conditions of a two-dimensional camera; and a position information generation unit that generates three-dimensional position information of an object on the basis of a two-dimensional image captured by the two-dimensional camera. The control device further comprises a synthesis unit that synthesizes a plurality of pieces of three-dimensional position information. While the robot is operating, the imaging control unit captures two-dimensional images at predetermined intervals under predetermined exposure conditions. When the robot stops, the imaging control unit changes the exposure conditions and captures a plurality of two-dimensional images, and the synthesis unit synthesizes a plurality of pieces of three-dimensional position information.

Description

[Technical Field]

[0001] The present invention relates to a control device for a robot device that acquires three-dimensional position information, and to a robot device. [Background technology]

[0002] In the prior art, a visual sensor is known that captures an image of an object and detects three-dimensional position information of the surface of the object (for example, Japanese Patent Application Laid-Open No. 63-251192). A two-dimensional camera can be used in such a visual sensor that detects three-dimensional position. For example, a stereo camera that detects three-dimensional position information of an object based on the parallax between two two-dimensional cameras, or a visual sensor that detects the three-dimensional position of an object based on a phase shift method are known.

[0003] In recent years, there has been known a device that calculates the positional deviation of a workpiece when the robot grasps the workpiece with its hand or the positional deviation of a workpiece fixed to a stand based on the three-dimensional position of the workpiece acquired by a visual sensor. The visual sensor can be attached to the tip of the robot's arm, for example (see, for example, JP 2020-38074 A).

[0004] When a visual sensor is attached to a robot, images can be captured by the visual sensor while the robot is being driven. By capturing images while the robot is being driven, it is possible to quickly capture images of an object from various directions. For example, when simulating a robot device, it may be necessary to obtain in advance three-dimensional position information of surrounding objects such as devices, stands, or shelves around the robot. To obtain three-dimensional position information of surrounding objects, images can be captured while the robot is being driven.

[0005] However, depending on the light intensity, light direction, and surface shape of the object, characteristic parts such as the outline of the object may not be captured accurately. For example, blown-out highlights or blocked-up shadows may occur in the two-dimensional image. In such cases, it is known to combine multiple images to obtain an image with a wide dynamic range (for example, JP 2002-334326 A). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 63-251192 [Patent Document 2] Japanese Patent Application Publication No. 2020-38074 [Patent Document 3] Japanese Patent Application Laid-Open No. 2002-334326 Summary of the Invention [Problem to be solved by the invention]

[0007] When capturing images using a 2D camera of a visual sensor while driving a robot, if the 2D image is overexposed or underexposed, it becomes impossible to accurately obtain 3D position information of the target object. The operator can stop the robot every time it moves a short distance and adjust the exposure conditions of the 2D camera. Alternatively, multiple exposure conditions can be set in advance and images can be captured repeatedly under the multiple exposure conditions every time the robot moves a short distance. However, these methods require the robot to be stopped every time an image is captured, which can be time-consuming. [Means for solving the problem]

[0008] A control device for a robot device according to an embodiment of the present disclosure includes a robot and a 3D visual sensor including a 2D camera that captures an image of an object. The control device includes a motion detection unit that detects the motion state of the robot and an imaging control unit that changes the exposure conditions of the 2D camera. The control device includes a position information generation unit that generates 3D position information of the object based on 2D images captured by the 2D camera. The control device includes a synthesis unit that performs control to synthesize multiple 2D images or control to synthesize multiple pieces of 3D position information. While the motion detection unit is detecting the motion of the robot, the imaging control unit captures 2D images at predetermined intervals under predetermined exposure conditions. When the motion detection unit detects that the robot has stopped, the imaging control unit changes the exposure conditions and captures multiple 2D images, and the synthesis unit synthesizes the multiple 2D images or multiple pieces of 3D position information.

[0009] A robotic device according to an embodiment of the present disclosure includes the above-described control device, a three-dimensional visual sensor including a two-dimensional camera that captures an image of an object, and a robot. [Effects of the Invention]

[0010] According to one aspect of the present disclosure, it is possible to provide a control device for a robot device and a robot device that can acquire three-dimensional position information of an object in a short time. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a perspective view of a first robot device according to an embodiment. [Figure 2] FIG. 2 is a block diagram of a first robotic device equipped with a first control device. [Figure 3] FIG. 1 is a schematic diagram of a visual sensor according to an embodiment. [Figure 4] 10 is a perspective view of the mount and the visual sensor when the position and attitude of the visual sensor are changed to capture an image. FIG. [Figure 5] FIG. 1 is a perspective view illustrating three-dimensional points obtained based on the output of a visual sensor. [Figure 6]5 is a flowchart of a first control for acquiring three-dimensional position information of an object in the embodiment. [Figure 7] FIG. 3 is a block diagram of a processing unit of a second control device of a first robot device according to the embodiment. [Figure 8] 10 is a flowchart of a second control for acquiring three-dimensional position information of an object in the embodiment. [Figure 9] FIG. 4 is a side view of a second robot device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] A control device for a robot device and a robot device according to an embodiment will be described with reference to Figures 1 to 9. The robot device according to the present embodiment includes a robot and a three-dimensional visual sensor that acquires three-dimensional position information regarding the surface of an object.

[0013] Fig. 1 is a perspective view of a first robot device according to the present embodiment. Fig. 2 is a block diagram of the first robot device according to the present embodiment. With reference to Figs. 1 and 2, the first robot device 3 includes a hand 5 as a work tool for gripping a workpiece, and a robot 1 that moves the hand 5. The robot device 3 includes a first control device 2 that controls the robot 1 and the hand 5. The robot device 3 includes a visual sensor 30 that is a three-dimensional visual sensor that acquires information about the surface of an object.

[0014] The robot 1 of this embodiment is an articulated robot including multiple joints 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 to which the hand 5 is fixed. The robot 1 of this embodiment has six drive shafts, but is not limited to this form. Any robot capable of moving a work tool can be used.

[0015] Furthermore, the work tool attached to the robot 1 is not limited to the hand 5, and any work tool can be used depending on the work to be performed by the robot device 3. For example, a work tool for welding or a work tool for applying a sealant can be used.

[0016] In the first robot device 3, the visual sensor 30 is attached to the robot 1. The visual sensor 30 is fixed to the flange 16 via a support member 68. The visual sensor 30 in this embodiment is supported by the robot 1 so that its position and posture change together with the hand 5.

[0017] The robot 1 of this embodiment includes a robot driving device 21 that drives components such as the upper arm 11. The robot driving device 21 includes a plurality of driving motors for driving the upper arm 11, the lower arm 12, the rotating base 13, and the wrist 15. The hand 5 includes a hand driving device 22 that drives the hand 5. The hand driving device 22 of this embodiment drives the hand 5 by air pressure. The hand driving device 22 includes a pump, an electromagnetic valve, and the like for driving the fingers of the hand 5.

[0018] The control device 2 includes an arithmetic processing device 24 (computer) including a CPU (Central Processing Unit) as a processor. The arithmetic processing device 24 has a RAM (Random Access Memory) and a ROM (Read Only Memory), which are connected to the CPU via a bus. The robot device 3 drives the robot 1 and the hand 5 based on an operation program 41. The robot device 3 of this embodiment has a function of automatically transporting a workpiece.

[0019] The arithmetic processing unit 24 of the control device 2 includes a storage unit 42 that stores information related to the control of the robot device 3. The storage unit 42 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. An operation program 41, which is created in advance to perform the operation of the robot 1, is input to the control device 2. The operation program 41 is stored in the storage unit 42.

[0020] The arithmetic processing device 24 includes an operation control unit 43 that controls the operation of the robot. The operation control unit 43 sends operation commands to the robot driving unit 44 to drive the robot 1 based on the operation program 41. The robot driving unit 44 includes an electrical circuit that drives the drive motor. The robot driving unit 44 supplies electricity to the robot driving device 21 based on the operation commands. The operation control unit 43 also sends operation commands to the hand driving unit 45 to drive the hand driving device 22. The hand driving unit 45 includes an electrical circuit that drives a pump or the like. The hand driving unit 45 supplies electricity to the hand driving device 22 based on the operation commands.

[0021] The operation control unit 43 corresponds to a processor that operates in accordance with the operation program 41. The processor reads the operation program 41 and performs the control defined in the operation program 41, thereby functioning as the operation control unit 43.

[0022] The robot 1 includes a state detector for detecting the position and posture of the robot 1. In this embodiment, the state detector includes a position detector 23 attached to the drive motor of each drive axis of the robot drive device 21. The position detector 23 is configured by, for example, an encoder. The position and posture of the robot 1 are detected based on the output of the position detector 23.

[0023] The control device 2 includes a teaching operation panel 49 as an operation panel with which an operator manually operates the robot device 3. The teaching operation panel 49 includes an input unit 49a for inputting information about the robot 1, the hand 5, and the visual sensor 30. The input unit 49a is composed of operating members such as a keyboard, a dial, and a button. The teaching operation panel 49 includes a display unit 49b for displaying information about the control of the robot device 3. The display unit 49b is composed of a display panel such as a liquid crystal display panel or an organic EL (Electro Luminescence) panel.

[0024] In the robot device 3 of this embodiment, a robot coordinate system 71 is set that remains unchanged even when the position and posture of the robot 1 change. In the example shown in FIG. 1, the origin of the robot coordinate system 71 is located on the base 14 of the robot 1. The robot coordinate system 71 is also referred to as a world coordinate system or a reference coordinate system. The position of the origin of the robot coordinate system 71 is fixed, and the orientation of the coordinate axes is fixed. Even if the position and posture of the robot 1 change, the position and posture of the robot coordinate system 71 do not change.

[0025] In the robot device 3, a flange coordinate system 72 is set on the surface of the flange 16 to which the hand 5 is fixed. The origin of the flange coordinate system 72 is located at the center of rotation of the flange 16. In this example, the rotation axis of the flange 16 is set to the Z axis of the flange coordinate system 72. The flange coordinate system 72 is also referred to as a hand coordinate system. The position of the robot 1 in this embodiment corresponds to the position of the origin of the flange coordinate system 72 in the robot coordinate system 71. Furthermore, the posture of the robot 1 corresponds to the posture of the flange coordinate system 72 with respect to the robot coordinate system 71.

[0026] A tool coordinate system may be arranged on the work tool. The position of the origin of the tool coordinate system in the robot coordinate system 71 may be taken as the position of the robot, and the attitude of the tool coordinate system relative to the robot coordinate system 71 may be taken as the attitude of the robot.

[0027] FIG. 3 shows a schematic diagram of a visual sensor according to the present embodiment. Referring to FIGS. 2 and 3, visual sensor 30 according to the present embodiment is a three-dimensional visual sensor (three-dimensional camera) capable of acquiring three-dimensional position information of the surface of an object. Visual sensor 30 according to the present embodiment is a stereo camera including a first camera 31 and a second camera 32. Each of cameras 31 and 32 is a two-dimensional camera capable of capturing two-dimensional images. The two cameras 31 and 32 are disposed apart from each other. The relative positions of the two cameras 31 and 32 are predetermined. Visual sensor 30 according to the present embodiment includes a projector 33 that projects a pattern of light, such as a striped pattern, onto a workpiece. Cameras 31 and 32 and projector 33 are disposed inside a housing 34.

[0028] Referring to FIG. 1, in the robot device 3, a sensor coordinate system 73 is set relative to the visual sensor 30. The sensor coordinate system 73 is a coordinate system whose origin is fixed at an arbitrary position on the visual sensor 30. The position and orientation of the sensor coordinate system 73 change together with the visual sensor 30. In this embodiment, the sensor coordinate system 73 is set so that the Z axis is parallel to the optical axis of the camera included in the visual sensor 30. The sensor coordinate system 73 moves and rotates together with the flange coordinate system 72. The position of the origin of the sensor coordinate system 73 in the flange coordinate system 72 and the orientation of the sensor coordinate system 73 relative to the flange coordinate system 72 are determined in advance.

[0029] 2, the arithmetic processing device 24 of the control device 2 in this embodiment includes a processing unit 51 that performs arithmetic processing. The processing unit 51 includes a manual control unit 52 that generates operation commands for manually driving the robot 1 and the hand 5 in response to an operator's operation of the teaching operation panel 49. The manual control unit 52 sends the operation commands for the robot 1 and the hand 5 to the operation control unit 43. The operator can change the position and posture of the robot in a desired direction by operating the input unit 49a of the teaching operation panel 49.

[0030] For example, pressing a button on the input unit 49a can perform a jog operation, in which the robot moves or rotates in the direction of the coordinate axis of the coordinate system corresponding to the button. In the jog operation, the robot moves while the button is pressed. Note that the operation control unit 43 may also have the functions of the manual control unit 52.

[0031] The processing unit 51 includes a motion detection unit 53 that detects the motion state of the robot 1. The motion detection unit 53 detects the motion state of the robot 1, for example, based on the output of the position detector 23. Alternatively, the motion detection unit 53 may obtain a motion command transmitted from the motion control unit 43 and detect the motion state of the robot 1. The motion detection unit 53 of this embodiment can detect whether the robot 1 is moving or is stopped.

[0032] The processing unit 51 includes a position information generation unit 54 that generates three-dimensional position information of the object based on a two-dimensional image acquired from the visual sensor 30. As will be described later, the three-dimensional position information can be, for example, a range image or a three-dimensional map representing the surface of the object. The position information generation unit 54 has a function of converting the position information of the object's surface acquired in the sensor coordinate system 73 into position information of the object's surface expressed in the robot coordinate system 71. For example, the position information generation unit 54 has a function of converting the position (coordinate value) of a three-dimensional point in the sensor coordinate system 73 into the position (coordinate value) of a three-dimensional point in the robot coordinate system 71 based on the position and posture of the robot 1.

[0033] The processing unit 51 includes an imaging control unit 59 that controls imaging by the visual sensor 30. The imaging control unit 59 controls the timing of imaging by the visual sensor 30. The imaging control unit 59 changes the exposure conditions of the two-dimensional camera included in the visual sensor 30. The exposure conditions in this embodiment include the exposure time (shutter speed) of the two-dimensional camera.

[0034] The robot device may also be equipped with a lighting device for illuminating an object. The lighting device may be fixed around the object. Alternatively, the lighting device may be fixed to the robot 1 and move along with the movement of the robot 1. The imaging control unit 59 may be configured to adjust the brightness of the lighting. In this case, at least one of the exposure time of the two-dimensional camera and the light intensity of the lighting device may be used as the exposure condition.

[0035] The processing unit 51 includes an information processing unit 55 that processes the three-dimensional position information generated by the position information generation unit 54. Alternatively, the information processing unit 55 processes two-dimensional images acquired by the two-dimensional camera of the visual sensor 30. The information processing unit 55 includes a synthesis unit 57 that synthesizes two-dimensional images or three-dimensional position information. The synthesis unit 57 synthesizes multiple two-dimensional images captured by the two-dimensional camera so as to correct defects in the two-dimensional images, or the synthesis unit 57 synthesizes multiple pieces of three-dimensional position information generated by the position information generation unit 54 so as to correct defects in the three-dimensional position information.

[0036] The information processing unit 55 includes a determination unit 58 that determines whether or not there is a defect in the two-dimensional image or the three-dimensional position information. When determining a two-dimensional image, the determination unit 58 in this embodiment determines whether or not there is a blown-out highlight or crushed shadow defect in the two-dimensional image. For example, when all pixels within an area of ​​a predetermined size have pixel values ​​of blown-out highlight or crushed shadow, the determination unit 58 determines that the area is defective.

[0037] On the other hand, when determining the three-dimensional position information, the determination unit 58 determines whether there is a defect in which some distance information is missing. For example, the determination unit 58 determines whether distance information is missing for each pixel. If distance information is missing for some pixels, the determination unit 58 determines that there is a defect in the three-dimensional position information.

[0038] The processing unit 51 corresponds to a processor that operates in accordance with the operation program 41. The manual control unit 52, the operation detection unit 53, the position information generation unit 54, the information processing unit 55, the synthesis unit 57, the determination unit 58, and the imaging control unit 59 included in the processing unit 51 correspond to the processor that operates in accordance with the operation program 41. The processor reads the operation program 41 and performs the control defined in the operation program 41, thereby functioning as each unit.

[0039] Before carrying out an actual workpiece transport operation, the robot device 3 in this embodiment acquires three-dimensional position information of surrounding objects such as conveyors or robots, fences, and stands arranged around the robot 1. In other words, it acquires three-dimensional information of surrounding objects arranged around the robot 1. Here, the acquisition of three-dimensional position information of the surface of the stand 65 for fixing the workpiece will be described.

[0040] The three-dimensional position information of objects around the robot 1 is obtained, for example, before performing an offline simulation of the robot device. The display unit of the simulation device can display a stereoscopic image based on the three-dimensional position information of the surrounding objects. While viewing the image of the surrounding objects, the worker can generate a movement path for the robot in actual work so that the robot, hand, and workpiece do not come into contact with the surrounding objects. Alternatively, the simulation device may have a function for automatically generating a movement path. In this case, the worker specifies the start and end points of the robot's movement based on the three-dimensional position information of the surrounding objects. The simulation device can then automatically generate a movement path for the robot so that the robot does not interfere with the surrounding objects.

[0041] Alternatively, the control device of the robot device can store three-dimensional position information of surrounding objects arranged around the robot device. The control device can determine whether the robot device will come into contact with a surrounding object. For example, when an operator operates a teaching pendant to manually drive the robot, the control device can determine whether the robot device will come into contact with a surrounding object. The control device can execute control to prevent the robot device from driving if it is determined that the robot device will come into contact with a surrounding object. Alternatively, the control device can execute control to slow down or stop the robot when the robot device approaches a surrounding object. Furthermore, the control device can display a warning to stop the robot on the display unit of the display pendant.

[0042] Fig. 4 shows a perspective view of the visual sensor and the pedestal when acquiring three-dimensional position information of the surface of the pedestal. With reference to Figs. 2 and 4, in the first control in this embodiment, the worker manually drives the robot 1. The worker operates the input unit 49a of the teaching operation panel 49. The manual control unit 52 changes the position and posture of the robot in response to the operation of the input unit 49a.

[0043] In the example shown in FIG. 4 , the worker places the visual sensor 30 at position P30a by jogging. Then, as indicated by arrows 96a and 96b, the worker changes the position and orientation of the visual sensor 30 to positions P30b and P30c. The imaging control unit 59 captures images with the cameras 31 and 32 at predetermined intervals while the visual sensor 30 is moving. Three-dimensional points can be generated on surfaces facing the visual sensor 30 within the imaging area 79 of the visual sensor 30. The worker changes the position and orientation of the robot 1 so that the cameras 31 and 32 capture images of all surfaces for which three-dimensional position information is required. Here, images are captured at various positions and orientations to capture all surfaces of the pedestal 65 in order to generate three-dimensional position information for all visible surfaces of the pedestal 65.

[0044] The position information generation unit 54 sets a three-dimensional point on the surface of the object included in the image based on the two-dimensional images acquired by the first camera 31 and the second camera 32. The position information generation unit 54 calculates the distance from the visual sensor 30 to the three-dimensional point set on the surface of the object based on the parallax between the image captured by the first camera 31 and the image captured by the second camera 32. The three-dimensional point can be set for each pixel of the imaging element, for example. Furthermore, the position information generation unit 54 calculates the coordinate value of the position of the three-dimensional point in the sensor coordinate system 73 based on the distance from the visual sensor 30.

[0045] FIG. 5 shows a perspective view of a point cloud of 3D points generated by the position information generation unit. FIG. 5 is a perspective view of the 3D points arranged in a three-dimensional space. In FIG. 5, the outline of the mount 65 is shown by a dashed line. 3D points 70 are arranged on the surface of an object facing the visual sensor 30. The position information generation unit 54 sets the 3D points 70 on the surface of an object included inside the imaging area 79. Here, a large number of 3D points 70 are arranged on the surface of the mount 65.

[0046] The position information generation unit 54 can display the three-dimensional position information of the object's surface in a perspective view of a point cloud of three-dimensional points, as shown in FIG. 5. The position information generation unit 54 can also generate the three-dimensional position information of the object's surface in the form of a range image or a three-dimensional map. A range image represents the position information of the object's surface using a two-dimensional image. In a range image, the darkness or color of each pixel represents the distance from the visual sensor 30 to a three-dimensional point. On the other hand, a three-dimensional map represents the position information of the object's surface using a set of coordinate values ​​(x, y, z) of three-dimensional points on the object's surface. The coordinate values ​​can be expressed in any coordinate system, such as the sensor coordinate system 73 or the robot coordinate system 71.

[0047] In this embodiment, a distance image is mainly used as an example of 3D position information of the surface of an object. The position information generator 54 in this embodiment generates a distance image in which the color intensity is changed according to the distance from the visual sensor 30 to the 3D point 70.

[0048] In this embodiment, the position information generator 54 is disposed in the processing unit 51 of the arithmetic processing device 24, but this is not limiting. The position information generator may be disposed inside the visual sensor. That is, the visual sensor may include an arithmetic processing device including a processor such as a CPU, and the processor of the arithmetic processing device of the visual sensor may function as the position information generator. In this case, the visual sensor outputs a 3D map, a range image, or the like.

[0049] The processing unit 51 of this embodiment is configured to perform automatic imaging control that starts or stops imaging by the visual sensor 30 depending on the driving state of the robot 1. The automatic imaging control of this embodiment includes normal imaging control and synthesis control.

[0050] The motion detection unit 53 detects whether the robot 1 is moving. Normal imaging control is performed while the motion detection unit 53 is detecting the motion of the robot 1. In normal imaging control, the imaging control unit 59 captures two-dimensional images at predetermined intervals using the cameras 31 and 32. The imaging control unit 59 captures images while the position and posture of the visual sensor 30 are being changed. The imaging control unit 59 also captures two-dimensional images under the same predetermined exposure conditions.

[0051] The imaging control unit 59 can capture images at predetermined time intervals. For example, the imaging control unit 59 captures images using the cameras 31, 32 at time intervals ranging from 300 msec to 500 msec. The predetermined intervals are not limited to time intervals, and may be intervals corresponding to the movement distance of a predetermined part. For example, the movement distance of the origin position of the flange coordinate system of the robot may be used. Alternatively, the intervals may be intervals corresponding to the rotation angle of a predetermined part. The predetermined intervals do not have to be constant. The position information generation unit 54 generates three-dimensional position information at predetermined intervals.

[0052] The exposure conditions in normal imaging control can be any exposure conditions that allow for capturing an image without blurring. For example, predetermined fixed exposure conditions can be used. It is preferable to shorten the exposure time so that the 2D image does not blur. To achieve this, for example, it is preferable to maximize the light intensity of the lighting device and set a fast shutter speed. The operator maximizes the light intensity of the lighting device and moves the camera so as to capture a medium-bright portion of the object. Then, the minimum exposure time that allows for capturing a 2D image can be set. In other words, the maximum shutter speed can be determined according to the brightness of the lighting device.

[0053] In this embodiment, since the object is imaged from various positions and directions, defects such as blown-out highlights or crushed shadows may occur in the 2D image depending on the position of the lighting device, the orientation of the visual sensor, the material of the object's surface, and the shape of the object's surface. 3D position information cannot be created for areas where the 2D image has defects. Therefore, the information processing unit 55 of this embodiment performs synthesis control to compensate for defects in the 2D image or 3D position information.

[0054] The motion detection unit 53 detects whether the robot 1 is moving. When the motion detection unit 53 detects that the robot 1 has stopped, synthesis control is performed. In synthesis control, the imaging control unit 59 automatically changes the exposure conditions to capture multiple 2D images. For example, the imaging control unit 59 captures multiple 2D images while changing the exposure time from 0.1 msec to 100 msec in increments of 10. Alternatively, the imaging control unit 59 may change the light intensity of the lighting device. For example, the imaging control unit 59 captures multiple images with the cameras 31 and 32 while increasing the light intensity by 10% from 0% (off) to 100% (fully on). This imaging is performed by both the first camera 31 and the second camera 32.

[0055] Next, the position information generating unit 54 generates multiple pieces of 3D position information from the 2D images under the multiple exposure conditions. Then, the combining unit 57 combines the multiple pieces of 3D position information to generate one piece of 3D position information. Alternatively, the combining unit 57 combines 2D images under the multiple exposure conditions for each image from the cameras 31 and 32. The position information generating unit 54 may generate 3D position information based on the two combined 2D images for each of the cameras 31 and 32.

[0056] In this example, position information generating unit 54 generates a distance image corresponding to each exposure condition based on a two-dimensional image corresponding to each exposure condition. Position information generating unit 54 generates multiple distance images. Then, combining unit 57 combines the multiple distance images.

[0057] As a method for combining distance images, for pixels in one distance image that lack three-dimensional positions (distance information), the three-dimensional positions of pixels in a distance image captured under a different exposure condition can be supplemented. For example, the combining unit 57 selects one distance image captured under a medium exposure condition. The combining unit 57 identifies pixels that do not have distance information. Then, for pixels that do not have distance information, the combining unit 57 can obtain distance information from a distance image captured under a different exposure condition. Alternatively, the combining unit 57 deletes pixels that lack distance information in multiple distance images. The combining unit 57 may calculate an average value of the distance information of multiple distance images for all pixels that have distance information.

[0058] In addition, when the three-dimensional position information is a three-dimensional map, the coordinate values ​​of each three-dimensional point can be interpolated or averaged. In this way, the synthesis unit can synthesize multiple pieces of three-dimensional position information to generate synthesized position information, which is the three-dimensional position information after synthesis.

[0059] FIG. 6 shows a flowchart of the first control in this embodiment. In the first control, the operator manually drives the robot 1 by jogging, and normal imaging control is performed. The display unit 49b of the teaching operation panel 49 displays a distance image. The operator then determines whether or not there is a defect in the distance image. If there is a defect in the distance image, the operator manually stops the robot 1, and synthesis control is performed.

[0060] In step 81, the operator operates the teaching pendant 49 to drive the robot 1 to a position and posture for starting imaging. For example, referring to Fig. 4, the manual control unit 52 drives the robot 1 in response to the operator's operation so that the visual sensor 30 is located at position P30a. The visual sensor 30 is located at position P30a directly above the platform 65, which is the target object.

[0061] In step 82, automatic imaging control is started. The worker operates the input unit 49a to send a command to start automatic imaging control to the imaging control unit 59 and the motion detection unit 53. In step 83, the worker manually drives the robot 1. For example, as shown by arrow 96a, the position and orientation of the visual sensor 30 are changed from position P31a to position P30b.

[0062] In step 84, the motion detection unit 53 detects the movement of the robot 1. The imaging control unit 59 performs normal imaging control to capture two-dimensional images at predetermined intervals and under predetermined exposure conditions. The position information generation unit 54 generates three-dimensional position information based on the two-dimensional images captured by the cameras 31 and 32. Here, the position information generation unit 54 generates a distance image. The display unit 49b of the teaching pendant 49 displays the distance image each time it is generated.

[0063] While driving the robot 1, the worker views the distance image displayed on the display unit 49b. The worker determines whether there is a defect in the distance information. No distance information is generated in areas where there is a defect in the two-dimensional image. For example, the position of a three-dimensional point is not calculated in the sensor coordinate system 73. In the distance image, defective pixels that lack distance information are displayed in a predetermined darkness. For example, if the distance image is created so that the color darkness changes depending on the distance, defective pixels are displayed in the darkest color. The worker can easily determine whether there is a defect in the distance information by looking at the distance image.

[0064] If the operator determines that there are no defects in the distance image, he or she continues to manually drive the robot. In step 85, the motion detection unit 53 determines that the robot 1 has not stopped. Control returns to step 83. In steps 83 and 84, driving of the robot 1 and generation of the distance image continue.

[0065] On the other hand, if the worker determines that there is a defect in the range image while the robot 1 is being driven, he or she manually stops the robot 1. That is, he or she interrupts the jogging operation of the robot 1 and stops the movement of the visual sensor 30. In step 85, the motion detection unit 53 detects that the robot 1 has stopped. Normal imaging control is stopped by the worker's operation. Control then proceeds to step 88.

[0066] In step 88, the imaging control unit 59 determines whether or not the driving of the robot 1 has ended. When the imaging of the object from various directions is completed, the driving of the robot 1 ends. In this case, the operator operates the teaching operation panel 49 to send a command to the processing unit 51 to end the driving of the robot 1. If the driving of the robot 1 has not ended, the control proceeds to step 86.

[0067] In step 86, the imaging control unit 59 starts synthesis control. The imaging control unit 59 changes the exposure conditions and captures multiple 2D images. Here, the position information generation unit 54 generates a distance image corresponding to the exposure conditions each time an image is captured by the 2D camera. In step 87, the synthesis unit 57 of the information processing unit 55 generates a synthetic distance image as 3D position information from the multiple distance images. This control generates a synthetic distance image without any defects. The display unit 49b displays the synthetic image generated by the synthesis unit 57. The worker can check the image displayed on the display unit 49b. After this, control returns to step 83. The worker resumes manual operation of the robot 1. When the operation detection unit 53 detects that operation of the robot 1 has resumed, the synthesis control stops and normal imaging control starts.

[0068] In step 88, when the manual driving of the robot 1 is completed, control proceeds to step 89. That is, when imaging from various directions is completed, control proceeds to step 89. In step 89, the operator operates the input unit 49a of the teaching operation panel 49 to send a command to end the automatic imaging control to the imaging control unit 59 and the motion detection unit 53, and this control ends.

[0069] The operator can change the position and orientation of the visual sensor 30 so that the entire desired surface of the pedestal 65 is imaged. Then, images can be captured from various angles to obtain three-dimensional position information of the surface of the pedestal 65. The position information generation unit 54 can convert the three-dimensional position information obtained in the sensor coordinate system 73 into three-dimensional position information in the robot coordinate system 71. The processing unit 51 can store the position and orientation of the visual sensor 30 and the three-dimensional position information in the storage unit 42. Furthermore, the processing unit 51 can generate three-dimensional position information (see FIG. 5) on the surface of the pedestal 65 by integrating or averaging multiple pieces of three-dimensional position information obtained at multiple positions and orientations of the robot 1.

[0070] In the above embodiment, the synthesis unit 57 synthesizes a plurality of distance images to generate a synthetic distance image, but this is not limiting. For example, the synthesis unit 57 may synthesize a plurality of two-dimensional images captured by the first camera 31 and the second camera 32. Any synthesis method that provides a large dynamic range can be used as a method for synthesizing the two-dimensional images.

[0071] For example, the composition unit 57 can compose two-dimensional images using the HDR (High Dynamic Range) method. By combining multiple two-dimensional images with different exposure conditions, a two-dimensional composite image with a wide dynamic range can be obtained. The composition unit 57 generates a composite two-dimensional image by combining the two-dimensional images from each of the cameras 31 and 32. Thereafter, the position information generation unit 54 may generate three-dimensional position information such as a distance image based on the composite two-dimensional image from the first camera 31 and the composite two-dimensional image from the second camera 32.

[0072] In this way, if there are no defects in the distance image, the worker can automatically repeat capturing images and generating distance images until the robot's operation is terminated. If the worker finds a defect in the distance image, the worker stops the operation of the robot 1. This operation causes the processing unit to capture multiple images with different exposure conditions and automatically synthesize three-dimensional position information or a two-dimensional image. Then, it is possible to generate composite position information by synthesizing the three-dimensional position information or three-dimensional position information generated from a composite two-dimensional image by synthesizing the two-dimensional images. The worker can generate three-dimensional position information of a desired part of the object by operating the robot, without performing operations such as capturing multiple images or generating a composite image. This makes it easy to generate three-dimensional position information of the object.

[0073] Furthermore, when there is a defect in the 2D image or 3D position information, the time required to stop the robot can be shortened. As a result, 3D position information of surrounding objects can be generated in a short time. Alternatively, 3D position information of surrounding objects can be generated easily.

[0074] Furthermore, if the worker finds defects remaining in the composite distance image when viewing the composite distance image displayed on display unit 49b, the worker may input a command to change the composition method into the teaching operation panel. Processing unit 51 can change the composition method and display the composite distance image on display unit 49b. For example, processing unit 51 can change from a method of composition of distance images to a method of composition of two-dimensional images. Then, when a composite distance image with fewer defects is obtained, the worker may resume driving the robot. This control makes it possible to more reliably obtain three-dimensional position information with fewer defects.

[0075] When manually driving a robot, the visual sensor may be temporarily stopped in order to change the direction of movement of the visual sensor. For example, referring to FIG. 4, the visual sensor 30 may be moved from position P30a to position P30b and then returned to position P30a. The visual sensor 30 temporarily stops at position P30b in order to change the direction of movement. In this case, synthesis control is performed in the automatic imaging control. Synthesis position information is generated even if it is not necessary. However, since there is no problem in using synthesis position information as three-dimensional position information, synthesis control may be performed.

[0076] 2, the information processing unit 55 in this embodiment includes a determination unit 58 that determines whether or not there is a defect in the two-dimensional image or the three-dimensional position information. Therefore, instead of an operator determining whether or not there is a defect in the three-dimensional position information, the determination unit 58 can automatically make the determination.

[0077] The determination unit 58 determines whether or not there are pixels for which there is no distance information in the three-dimensional position information generated under normal imaging control. If there are pixels for which there is no distance information, the determination unit 58 can determine that there is a defect in the three-dimensional position information. The determination unit 58 can make a determination for each piece of three-dimensional position information that is generated.

[0078] Alternatively, the determination unit 58 can determine whether or not there is a defect in the two-dimensional image captured by the cameras 31 and 32. For example, the determination unit 58 determines, as a defective pixel, a pixel with a white pixel value outside a predetermined determination range or a pixel with a black pixel value outside a predetermined determination range. If all pixels within an area of ​​a predetermined size have defective pixel values, the determination unit 58 determines that blown-out highlights or crushed shadows have occurred in that area. In other words, the determination unit 58 determines that a defect exists in the two-dimensional image.

[0079] If the determination unit 58 determines that there is a defect in the three-dimensional position information or the two-dimensional image, the manual control unit 52 sends a command to the operation control unit 43 to stop driving the robot 1, regardless of the operator's operation of the teaching operation panel 49. The operation control unit 43 stops the robot 1. This control detects that the robot 1 has stopped, and the imaging control unit 59 performs synthesis control. The imaging control unit 59 automatically changes the exposure conditions to capture multiple two-dimensional images. Then, the synthesis unit 57 can synthesize the two-dimensional images or the three-dimensional position information.

[0080] In this way, the determination unit 58 may automatically determine the presence of a defect in the three-dimensional position information or two-dimensional image and forcibly stop the robot 1. After the synthesis unit 57 finishes synthesizing the two-dimensional image or three-dimensional position information, the manual control unit 52 releases the forced stop of the robot 1. The manual control unit 52 resumes driving the robot 1 in response to the operator's operation of the teaching operation panel 49. That is, the operator can perform a jog operation by operating the input unit 49a of the teaching operation panel 49. The operation detection unit 53 detects the driving of the robot 1, and normal image capture control is resumed.

[0081] FIG. 7 shows a block diagram of the processing unit of the second control device in the first robot device in this embodiment. The second control device automatically drives the robot to obtain three-dimensional position information of an object. The worker creates in advance an operation program for automatically driving the robot 1. Then, while the robot is driven according to the operation program, images are captured by the visual sensor 30 under automatic imaging control to obtain three-dimensional position information of surrounding objects.

[0082] The processing unit 61 includes an automatic control unit 62 that generates operation commands to automatically drive the robot 1 in accordance with a predetermined operation program. The automatic control unit 62 sends the operation commands to the operation control unit 43. The operation control unit 43 drives the robot based on the operation commands from the automatic control unit 62. The operation control unit 43 may also have the functions of the automatic control unit 62. The remaining configuration of the processing unit 61 is the same as the configuration of the processing unit 51 of the first control device 2 (see FIG. 2).

[0083] FIG. 8 shows a flowchart of the control of the second control device in this embodiment. The second control device performs second control to automatically drive the robot 1 to capture images. The operator creates in advance an operation program that defines the movement path along which the robot 1 will move. For example, as shown in FIG. 4, an operation program is created that defines the movement path along which the visual sensor 30 passes through positions P30a, P30b, and P30c.

[0084] The robot's movement path included in the operation program is preferably a path that allows the visual sensor 30 to capture images of the object at various positions and postures so that three-dimensional position information of the desired surface of the object can be obtained.

[0085] In step 82, the operator starts automatic imaging control by operating the teaching operation panel 49. In step 92, the operator starts automatic driving of the robot by operating the teaching operation panel 49. The automatic control unit 62 sends an operation command for the robot 1 to the operation control unit 43 in accordance with the operation program. The position and posture of the robot 1 change, and the position and posture of the visual sensor 30 also change.

[0086] In step 84, the motion detection unit 53 detects the movement of the robot 1, and the imaging control unit 59 performs normal imaging control. Under normal imaging control, a two-dimensional image of the object is captured and three-dimensional position information is generated. In this example, a distance image is generated.

[0087] In step 93, while the robot 1 is being driven by the operation command from the automatic control unit 62, the determination unit 58 determines whether or not a defect exists in the two-dimensional image or the three-dimensional position information. In this example, it determines whether or not a defect exists in the distance image. If no defect exists in the two-dimensional image or the three-dimensional position information in step 93, control proceeds to step 95.

[0088] In step 95, the imaging control unit 59 determines whether or not the driving of the robot 1 has finished. That is, it determines whether or not the robot 1 has been driven to the end point of the movement path in the movement program. If, in step 95, the driving of the robot 1 has not finished, control returns to step 92. In this way, it is possible to generate three-dimensional position information of the surface of the target object while automatically driving the robot 1 based on a previously created movement program.

[0089] On the other hand, if the determination unit 58 determines in step 93 that a defect exists in the two-dimensional image or the three-dimensional position information, control proceeds to step 94. In step 94, the automatic control unit 62 issues a command to stop the robot 1. The motion detection unit 53 detects that the robot 1 has stopped, and the imaging control unit 59 performs synthesis control. In steps 86 and 87, the synthesis unit 57 synthesizes the two-dimensional image or the three-dimensional position information. In this example, the synthesis unit 57 generates a synthetic range image from multiple range images as the three-dimensional position information with the defect corrected.

[0090] After this, the automatic control unit 62 cancels the command to stop the robot 1. Then, the control returns to step 92. Since the robot 1 is driven automatically, the synthesis control stops and the normal imaging control starts.

[0091] In step 95, when the driving of the robot 1 is completed, the control proceeds to step 89. In step 89, the operator operates the input unit 49a of the teaching operation panel 49 to send a command to end the automatic imaging control to the processing unit 61, and this control ends.

[0092] In this way, the second control of the second control device can automatically generate three-dimensional position information of the target object in order to automatically drive the robot 1. Note that the operation program may contain a command statement to start the automatic imaging control and a command statement to stop the automatic imaging control. This control allows the automatic imaging control to be started and ended automatically in accordance with the operation program.

[0093] The other configurations, operations, and effects of the second control device are similar to those of the first control device, and therefore, description thereof will not be repeated here.

[0094] In the normal imaging control of this embodiment, a two-dimensional image is captured while driving the robot 1. For this reason, if the speed at which the robot 1 is driven is high, the two-dimensional image may be blurred. Therefore, in the normal imaging control, a control can be implemented to change the exposure time in advance according to the driving speed of the robot 1.

[0095] During the period when the motion detection unit 53 is detecting the motion of the robot 1, the imaging control unit 59 can determine whether or not the speed at which the robot 1 moves with respect to the exposure time of the cameras 31 and 32 exceeds a predetermined judgment value. When the speed at which the robot 1 moves with respect to the exposure time of the cameras 31 and 32 exceeds the predetermined judgment value, the imaging control unit 59 can perform control to shorten the exposure time of the cameras 31 and 32.

[0096] For example, the imaging control unit 59 calculates a variable by multiplying the movement speed (unit: mm / sec) of the visual sensor 30 by the exposure time (unit: sec) of the cameras 31 and 32. This variable corresponds to the distance the visual sensor 30 moves during exposure. If this variable exceeds a predetermined judgment value, the imaging control unit 59 determines that the speed at which the robot 1 moves relative to the exposure time of the cameras 31 and 32 exceeds the predetermined judgment value. Then, the imaging control unit 59 performs control to shorten the current exposure time. For example, the imaging control unit 59 performs control to reduce the exposure time at a predetermined rate. Alternatively, the operator can create in advance a table of exposure times corresponding to the values ​​of the above variables. The imaging control unit 59 may perform control to shorten the exposure time based on this table.

[0097] In the first robot device 3, the visual sensor 30 is attached to the robot 1. The objects for which three-dimensional position information is to be acquired are objects placed around the robot 1. By employing this configuration, the three-dimensional position information of the objects placed around the robot 1 can be easily acquired by changing the position and posture of the robot.

[0098] 9 shows a schematic side view of a second robot device according to the present embodiment. In the second robot device 6, the position and orientation of the visual sensor 30 are fixed. The visual sensor 30 is fixed to the installation surface of the robot 1 by a support member 69. The robot 1 is equipped with a hand 5 as a work tool.

[0099] In the second robot device 6, the hand 5 corresponds to the object for which three-dimensional position information is to be acquired. The control device of the second robot device 6 captures images of the surface of the hand 5 at various positions and postures of the hand 5. The control device generates three-dimensional position information of the surface of the hand 5 based on the output of the visual sensor 30.

[0100] The configuration of the control device of the robot device is the same as the first control device 2 or the second control device in the first robot device (see FIGS. 2 and 7). When an operator manually drives the robot 1, the processing unit 51 shown in FIG. 2 can be used. When the robot 1 is automatically driven by an operation program, the processing unit 61 shown in FIG. 7 can be used.

[0101] In the second robot device 6, the robot 1 is driven manually or automatically as indicated by arrows 97a and 97b. For example, the hand 5 is moved to positions P5a, P5b, and P5c. The control device performs automatic imaging control while the robot 1 is being automatically controlled. That is, while the robot 1 is being driven, normal imaging control is performed, and when the robot 1 stops, synthesis control can be performed. In either the manual or automatic case, an image of the hand 5 is captured as an object to be imaged by the visual sensor 30. The position and posture of the robot 1 are changed so that all surfaces requiring three-dimensional position information of the hand 5 are imaged.

[0102] By driving the robot 1 and capturing images of the hand 5 from various directions, it is possible to obtain three-dimensional position information of the surface of the hand 5. Furthermore, based on the position and posture of the robot 1 when each piece of three-dimensional position information is obtained, it is possible to obtain position information of the surface of the hand 5 relative to the position and posture of the robot 1. For example, it is possible to calculate the position of a three-dimensional point set on the surface of the hand 5 in the flange coordinate system 72.

[0103] An operator can perform offline simulation using the three-dimensional position information of the hand 5. For example, the operator generates a motion path of the robot 1 so that the hand 5 does not come into contact with surrounding objects. Alternatively, a simulation device that automatically generates a motion path of the robot can automatically generate a motion path of the robot 1 so that the hand 5 does not come into contact with surrounding objects.

[0104] Other configurations, actions, and effects of the second robotic device are similar to those of the first robotic device, and therefore will not be described repeatedly here.

[0105] The 3D visual sensor in this embodiment is a stereo camera, but is not limited to this. Any visual sensor, including a 2D camera, can be used as the visual sensor. For example, a visual sensor that detects the 3D position of an object using a phase shift method can be used. Alternatively, a sensor that calculates the position of a 3D point on the surface of an object by acquiring a 2D image of the object whose distance to the object is known can be used as the 3D visual sensor.

[0106] In each of the above-described controls, the order of steps can be changed as appropriate within the scope that does not change the functions and actions.

[0107] The above-described embodiments can be combined as appropriate. In each of the above-described drawings, the same or equivalent parts are designated by the same reference numerals. Note that the above-described embodiments are merely examples and do not limit the invention. Furthermore, the embodiments include modifications of the embodiments as set forth in the claims. [Explanation of symbols]

[0108] 1. Robot 2. Control device 3,6 Robotic Devices 5 hands P5a,P5b,P5c position 23 Position detector 24 Processing unit 30 Visual Sensor P30a,P30b,P30c position 31,32 Camera 43 Motion control section 49 Teaching control panel 49a Input section 52 Manual control unit 53 Motion detection unit 54 Location information generation section 57 Synthesis section 58 Judgment section 59 Imaging control unit 62 Automatic control unit 65 Mounting stand 70 3D points

Claims

1. A control device for a robot device including a three-dimensional visual sensor including a two-dimensional camera that captures an image of an object, and a robot, a motion detection unit that detects the motion state of the robot; an imaging control unit that changes the exposure conditions of the two-dimensional camera; a position information generating unit that generates three-dimensional position information of the object based on a two-dimensional image captured by a two-dimensional camera; a synthesis unit that performs control to synthesize a plurality of two-dimensional images or control to synthesize a plurality of pieces of three-dimensional position information, During a period in which the motion detection unit detects the motion of the robot, the imaging control unit captures two-dimensional images at predetermined intervals under predetermined exposure conditions; A control device in which, when the motion detection unit detects that the robot has stopped, the imaging control unit changes exposure conditions to capture multiple two-dimensional images, and the synthesis unit synthesizes the multiple two-dimensional images or multiple pieces of three-dimensional position information.

2. a manual control unit that generates an operation command to manually drive the robot in response to an operation by a worker; The control device according to claim 1 , wherein the operation detection unit detects the stop of the robot when an operator manually stops the robot.

3. a motion control unit that controls the motion of the robot; a determination unit that determines whether or not there is a defect in the two-dimensional image or the three-dimensional position information, The control device according to claim 1 , wherein the operation control unit stops the robot when the determination unit determines that the defect exists in the two-dimensional image or the three-dimensional position information.

4. a motion control unit that controls the motion of the robot; an automatic control unit that generates operation commands to automatically drive the robot according to a predetermined operation program; a determination unit that determines whether or not there is a defect in the two-dimensional image or the three-dimensional position information, 2. The control device according to claim 1, wherein the operation control unit stops the robot when the determination unit determines that the defect exists in the two-dimensional image or the three-dimensional position information during a period in which the robot is driven by an operation command from the automatic control unit.

5. 3. The control device according to claim 1, wherein, during a period in which the motion detection unit is detecting the motion of the robot, the imaging control unit performs control to shorten the exposure time of the two-dimensional camera if the speed at which the robot moves with respect to the exposure time of the two-dimensional camera exceeds a predetermined judgment value.

6. The control device according to claim 1 , wherein the exposure condition is at least one of an exposure time of a two-dimensional camera and a light amount of an illumination device.

7. The control device according to claim 1 ; a three-dimensional visual sensor including a two-dimensional camera for capturing an image of an object; A robotic device comprising: a robot.

8. the three-dimensional visual sensor is attached to a robot; The robot device according to claim 7 , wherein the object is an object disposed around the robot.

9. equipped with a work tool attached to the robot, the three-dimensional visual sensor is fixed at a predetermined position by a support member; The robotic device according to claim 7 , wherein the object is the work tool.

Images