Control device for robot device for acquiring three-dimensional position information, and robot device

Abstract

A control device for a robot device includes: an imaging control unit that changes an exposure condition of a two-dimensional camera; and a position information generation unit that generates three-dimensional position information of the object on the basis of the two-dimensional image captured by the two-dimensional camera. The control device includes a synthesis unit that synthesizes a plurality of pieces of three-dimensional position information. During the operation of the robot, the imaging control unit captures two-dimensional images at predetermined intervals under predetermined exposure conditions. When the robot is stopped, the imaging control unit changes the exposure conditions and captures a plurality of two-dimensional images, and the synthesis unit synthesizes a plurality of pieces of three-dimensional position information.

Description

Technical Field

[0001] The present invention relates to a control device for a robot device and a robot device for obtaining three-dimensional position information. Background Art

[0002] In the prior art, there is known a visual sensor that captures an object and detects three-dimensional position information of the surface of the object (for example, Japanese Patent Publication No. 63-251192). In such a visual sensor that detects a three-dimensional position, a two-dimensional camera can be used. For example, there is known a stereo camera that detects the three-dimensional position information of an object by the parallax of two two-dimensional cameras, or a visual sensor that detects the three-dimensional position of an object by a phase shift method.

[0003] In recent years, there is known a device that calculates the offset of the workpiece position when a robot grips the workpiece with a manipulator or the offset of the workpiece position fixed to a platform based on the three-dimensional position of the workpiece obtained by a visual sensor. The visual sensor can be installed at the front end of the robot arm (for example, Japanese Patent Publication No. 2020-38074).

[0004] When a visual sensor is installed on a robot, the visual sensor can be used to capture images while the robot is being driven. By capturing images while the robot is being driven, objects can be captured quickly from all directions. For example, when simulating a robot device, it is sometimes desirable to obtain three-dimensional position information of surrounding objects such as devices, stands, or racks around the robot in advance. When obtaining the three-dimensional position information of the surrounding objects, images can be captured while the robot is being driven.

[0005] However, sometimes, due to light intensity, light direction, and the shape of the surface of the object, it is not possible to capture the characteristic parts such as the outline of the object with high accuracy. For example, overexposure or underexposure may occur in a two-dimensional image. In such a case, it is known to synthesize multiple images to obtain an image with a large dynamic range (for example, Japanese Patent Application Publication No. 2002-334326).

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 63-251192

[0009] Patent Document 2: Japanese Patent Application Publication No. 2020-38074

[0010] Patent Document 3: Japanese Patent Application Publication No. 2002-334326 Summary of the invention

[0011] Problems to be solved by the invention

[0012] When the robot is driven while taking an image using a two-dimensional camera of a visual sensor, if overexposure or underexposure occurs in the two-dimensional image, the three-dimensional position information of the object cannot be accurately obtained. The operator can stop the robot every time the robot is slightly moved to adjust the exposure conditions of the two-dimensional camera. Alternatively, multiple exposure conditions can be set in advance, and the image can be repeatedly taken under multiple exposure conditions every time the robot is slightly moved. However, in these methods, the robot needs to be stopped every time an image is taken, and there is a problem that the operation takes a long time.

[0013] Means for solving problems

[0014] The control device of the robot device of the disclosed method has a three-dimensional visual sensor and a robot, and the three-dimensional visual sensor includes a two-dimensional camera for photographing an object. The control device has: a motion detection unit, which detects the motion state of the robot; and a shooting control unit, which changes the exposure conditions of the two-dimensional camera. The control device has: a position information generation unit, which generates three-dimensional position information of the object based on the two-dimensional image photographed by the two-dimensional camera. The control device has: a synthesis unit, which implements control to synthesize multiple two-dimensional images or control to synthesize multiple three-dimensional position information. During the period when the motion detection unit detects the motion of the robot, the shooting control unit shoots the two-dimensional image at a predetermined interval under a predetermined exposure condition. When the motion detection unit detects the stop of the robot, the shooting control unit changes the exposure conditions to shoot multiple two-dimensional images, and the synthesis unit synthesizes the multiple two-dimensional images or multiple three-dimensional position information.

[0015] A robot device according to the disclosed embodiment includes the above-mentioned control device, a three-dimensional visual sensor including a two-dimensional camera for capturing an image of an object, and a robot.

[0016] Effects of the Invention

[0017] According to one aspect of the present disclosure, it is possible to provide a control device for a robot device and a robot device that can acquire three-dimensional position information of an object in a short time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a perspective view of the first robot device in the embodiment.

[0019] Figure 2 is a block diagram of a first robotic device having a first control device.

[0020] Figure 3 This is a schematic diagram of a visual sensor in an embodiment.

[0021] Figure 4This is a perspective view of the stand and the vision sensor when the position and posture of the vision sensor are changed to perform imaging.

[0022] Figure 5 This is a stereogram that explains the three-dimensional points obtained based on the output of the vision sensor.

[0023] Figure 6 This is a flowchart of the first control for acquiring the three-dimensional position information of the object in the embodiment.

[0024] Figure 7 This is a block diagram of a processing unit of a second control device of a first robot apparatus in an embodiment.

[0025] Figure 8 This is a flowchart of the second control for acquiring the three-dimensional position information of the object in the embodiment.

[0026] Fig. 9 It is a side view of the second robot apparatus in the embodiment. DETAILED DESCRIPTION

[0027] Reference Figures 1 to 9 The robot device of the present embodiment includes a robot and a three-dimensional vision sensor for acquiring three-dimensional position information related to the surface of an object.

[0028] Figure 1 It is a perspective view of the first robot device in this embodiment. Figure 2 is a block diagram of the first robot device in this embodiment. Figure 1 and Figure 2 The first robot device 3 includes a manipulator 5 as a working tool for holding a workpiece and a robot 1 for moving the manipulator 5. The robot device 3 includes a first control device 2 for controlling the robot 1 and the manipulator 5. The robot device 3 includes a three-dimensional visual sensor, namely a visual sensor 30, for acquiring information related to the surface of an object.

[0029] The robot 1 of this embodiment is a multi-joint robot including a plurality of joints 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported on a revolving base 13. The revolving base 13 is supported on a base 14. The robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 for fixing the manipulator 5. The robot 1 of this embodiment has six drive axes, but is not limited to this mode. The robot can adopt any robot that can move a working tool.

[0030] The working tool attached to the robot 1 is not limited to the manipulator 5, and any working tool corresponding to the work performed by the robot device 3 may be used. For example, a working tool for welding or a working tool for applying a sealing material may be used.

[0031] In the first robot device 3, the vision sensor 30 is attached to the robot 1. The vision sensor 30 is fixed to the flange 16 via a support member 68. The vision sensor 30 of the present embodiment is supported by the robot 1 so that the position and posture thereof change together with the manipulator 5.

[0032] The robot 1 of this embodiment includes a robot driving device 21 for driving components such as the upper arm 11. The robot driving device 21 includes a plurality of driving motors for driving the upper arm 11, the lower arm 12, the rotating base 13, and the wrist 15. The manipulator 5 includes a manipulator driving device 22 for driving the manipulator 5. The manipulator driving device 22 of this embodiment drives the manipulator 5 by air pressure. The manipulator driving device 22 includes a pump and an electromagnetic valve for driving the fingers of the manipulator 5.

[0033] The control device 2 has a processing unit 24 (computer) including a CPU (Central Processing Unit) as a processor. The processing unit 24 has a RAM (Random Access Memory) and a ROM (Read Only Memory) connected to the CPU via a bus. The robot device 3 drives the robot 1 and the manipulator 5 according to the action program 41. The robot device 3 of this embodiment has a function of automatically transporting a workpiece.

[0034] The operation processing device 24 of the control device 2 includes a storage unit 42 for storing information related to the control of the robot device 3. The storage unit 42 can be composed of a non-temporary storage medium capable of storing information. For example, the storage unit 42 can be composed of a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. An action program 41 prepared in advance for performing the action of the robot 1 is input to the control device 2. The action program 41 is stored in the storage unit 42.

[0035] The operation processing device 24 includes: an action control unit 43 for controlling the action of the robot. The action control unit 43 sends an action command for driving the robot 1 to the robot driving unit 44 according to the action program 41. The robot driving unit 44 includes: a circuit for driving a drive motor. The robot driving unit 44 supplies power to the robot driving device 21 according to the action command. In addition, the action control unit 43 sends an action command for driving the robot driving device 22 to the manipulator driving unit 45. The manipulator driving unit 45 includes a circuit for driving a pump, etc. The manipulator driving unit 45 supplies power to the manipulator driving device 22 according to the action command.

[0036] The motion control unit 43 corresponds to a processor driven according to the motion program 41. The processor reads the motion program 41 and executes the control specified in the motion program 41, thereby functioning as the motion control unit 43.

[0037] The robot 1 includes a state detector for detecting the position and posture of the robot 1. The state detector in this embodiment includes a position detector 23 of a drive motor installed in each drive shaft of the robot drive device 21. The position detector 23 is composed of an encoder, for example. The position and posture of the robot 1 are detected based on the output of the position detector 23.

[0038] The control device 2 includes a teaching operation panel 49 as an operation panel for the operator to manually operate the robot device 3. The teaching operation panel 49 includes an input unit 49a for inputting information related to the robot 1, the manipulator 5, and the visual sensor 30. The input unit 49a is composed of operation components such as a keyboard, a dial, and buttons. The teaching operation panel 49 includes a display unit 49b for displaying information related to the control of the robot device 3. The display unit 49b is composed of a display panel such as a liquid crystal display panel or an organic EL (ElectroLuminescence) panel.

[0039] The robot device 3 of this embodiment sets a robot coordinate system 71 that does not change when the position and posture of the robot 1 change. Figure 1 In the example shown, the origin of the robot coordinate system 71 is arranged on the base 14 of the robot 1. The robot coordinate system 71 is also called a world coordinate system or a reference coordinate system. The position of the origin of the robot coordinate system 71 is fixed, and the orientation of the coordinate axis is fixed. Even if the position and posture of the robot 1 change, the position and posture of the robot coordinate system 71 do not change.

[0040] In the robot device 3, a flange coordinate system 72 is set for the surface of the flange 16 to which the manipulator 5 is fixed. The origin of the flange coordinate system 72 is arranged at the rotation center of the flange 16. In the example here, the rotation axis of the flange 16 is set as the Z axis of the flange coordinate system 72. The flange coordinate system 72 is also called a fingertip coordinate system. The position of the robot 1 of this embodiment corresponds to the position of the origin of the flange coordinate system 72 in the robot coordinate system 71. In addition, the posture of the robot 1 corresponds to the posture of the flange coordinate system 72 relative to the robot coordinate system 71.

[0041] Furthermore, a tool coordinate system may be arranged for the working tool. Also, the origin position of the tool coordinate system in the robot coordinate system 71 may be regarded as the position of the robot, and the posture of the tool coordinate system relative to the robot coordinate system 71 may be regarded as the posture of the robot.

[0042] Figure 3 A schematic diagram showing a visual sensor according to this embodiment. Figure 2 and Figure 3 The visual sensor 30 of this embodiment is a three-dimensional visual sensor (three-dimensional camera) that can obtain three-dimensional position information of the surface of an object. The visual sensor 30 of this embodiment is a stereo camera including a first camera 31 and a second camera 32. Each camera 31, 32 is a two-dimensional camera that can capture a two-dimensional image. The two cameras 31, 32 are configured to be separated from each other. The relative positions of the two cameras 31, 32 are predetermined. The visual sensor 30 of this embodiment includes: a projector 33 that projects pattern light such as a stripe pattern toward a workpiece. The cameras 31, 32 and the projector 33 are arranged inside a housing 34.

[0043] Reference Figure 1 In the robot device 3, a sensor coordinate system 73 is set for the visual sensor 30. The sensor coordinate system 73 is a coordinate system whose origin is fixed at an arbitrary position of the visual sensor 30. The position and posture of the sensor coordinate system 73 change together with the visual sensor 30. The sensor coordinate system 73 of the present embodiment is set so that the Z axis is parallel to the optical axis of the camera included in the visual sensor 30. The sensor coordinate system 73 moves or rotates together with the flange coordinate system 72. In addition, the origin position of the sensor coordinate system 73 in the flange coordinate system 72 and the posture of the sensor coordinate system 73 relative to the flange coordinate system 72 are predetermined.

[0044] Reference Figure 2The operation processing device 24 of the control device 2 in this embodiment includes a processing unit 51 that performs operation processing. The processing unit 51 includes a manual control unit 52 that generates an operation instruction for manually driving the robot 1 and the manipulator 5 according to the operator's operation on the teaching operation panel 49. The manual control unit 52 sends the operation instruction of the robot 1 and the manipulator 5 to the operation control unit 43. The operator can change the position and posture of the robot in a desired direction by operating the input unit 49a of the teaching operation panel 49.

[0045] For example, if a button of the input unit 49a is pressed, a jog operation can be performed in which the position of the robot moves in the direction of the coordinate axis of the coordinate system corresponding to the button or the robot rotates in the direction of the coordinate axis of the coordinate system corresponding to the button. In the jog operation, the robot is driven while the button is pressed. In addition, the motion control unit 43 may also have the function of the manual control unit 52.

[0046] The processing unit 51 includes a motion detection unit 53 for detecting the motion state of the robot 1. The motion detection unit 53 detects the motion state of the robot 1 based on the output of the position detector 23, for example. Alternatively, the motion detection unit 53 may obtain a motion instruction issued from the motion control unit 43 to detect the motion state of the robot 1. The motion detection unit 53 of this embodiment can detect whether the robot 1 is driving or the robot 1 is stopped.

[0047] The processing unit 51 includes a position information generating unit 54 that generates three-dimensional position information of an object based on a two-dimensional image obtained from the visual sensor 30. As the three-dimensional position information, a distance image or a three-dimensional map representing the surface of the object can be exemplified as described later. The position information generating unit 54 has the following function: converting the position information of the surface of the object obtained by the sensor coordinate system 73 into the position information of the surface of the object represented by the robot coordinate system 71. For example, the position information generating unit 54 has the following function: converting the position (coordinate value) of a three-dimensional point in the sensor coordinate system 73 into the position (coordinate value) of the three-dimensional point in the robot coordinate system 71 according to the position and posture of the robot 1.

[0048] The processing unit 51 includes a photographing control unit 59 that controls photographing of the visual sensor 30. The photographing control unit 59 controls the photographing period of the visual sensor 30. The photographing control unit 59 changes the exposure conditions of the two-dimensional camera included in the visual sensor 30. The exposure conditions in this embodiment include the exposure time (shutter speed) of the two-dimensional camera.

[0049] In addition, the robot device may also have a lighting device for illuminating the object. The lighting device can be fixed around the object. Alternatively, the lighting device can be fixed to the robot 1 and move together with the driving of the robot 1. The shooting control unit 59 can be configured to adjust the lighting brightness. At this time, as the exposure condition, at least one of the exposure time of the two-dimensional camera and the light amount of the lighting device can be used.

[0050] The processing unit 51 includes an information processing unit 55 that processes the three-dimensional position information generated by the position information generating unit 54. Alternatively, the information processing unit 55 processes a two-dimensional image obtained by the two-dimensional camera of the visual sensor 30. The information processing unit 55 includes a synthesizing unit 57 that synthesizes the two-dimensional image or the three-dimensional position information. The synthesizing unit 57 synthesizes a plurality of two-dimensional images captured by the two-dimensional camera in a manner that corrects defects in the two-dimensional image, or the synthesizing unit 57 synthesizes a plurality of three-dimensional position information generated by the position information generating unit 54 in a manner that corrects defects in the three-dimensional position information.

[0051] The information processing unit 55 includes a determination unit 58 for determining whether there is a defect in the two-dimensional image or the three-dimensional position information. The determination unit 58 in this embodiment determines whether there is an overexposure or underexposure defect in the two-dimensional image when determining the two-dimensional image. For example, when all pixels within an area of ​​a predetermined size have overexposure or underexposure pixel values, the determination unit 58 determines that the area is a defect.

[0052] On the other hand, when determining the three-dimensional position information, the determination unit 58 determines whether there is a defect in which a part of the distance information is missing. For example, the determination unit 58 determines whether the distance information is missing for each pixel. When the distance information is missing in a part of the pixels, the determination unit 58 determines that there is a defect in the three-dimensional position information.

[0053] The processing unit 51 described above corresponds to a processor driven according to the action program 41. The manual control unit 52, the action detection unit 53, the position information generation unit 54, the information processing unit 55, the synthesis unit 57, the determination unit 58, and the imaging control unit 59 included in the processing unit 51 correspond to the processor driven according to the action program 41. The processor reads the action program 41 and implements the control determined in the action program 41, thereby functioning as each unit.

[0054] The robot device 3 in this embodiment obtains the three-dimensional position information of the surrounding objects such as the conveyor or robot, fences, and platforms arranged around the robot 1 before carrying out the operation of conveying the actual workpiece. In other words, the three-dimensional information of the surrounding objects arranged around the robot 1 is obtained. Here, the acquisition of the three-dimensional position information of the surface of the platform 65 for fixing the workpiece is described.

[0055] The three-dimensional position information of objects around the robot 1 is obtained, for example, before implementing the offline simulation of the robot device. The display unit of the simulation device can display a stereoscopic image based on the three-dimensional position information of the surrounding objects. The operator can generate the robot's motion path in actual operation in a manner that the robot, the manipulator, and the workpiece do not contact the surrounding objects while observing the image of the surrounding objects. Alternatively, sometimes the simulation device has a function of automatically generating a motion path. In this case, the operator specifies the starting point and the end point of the robot's motion based on the three-dimensional position information of the surrounding objects. In addition, the simulation device can automatically generate the robot's motion path so that the robot does not interfere with the surrounding objects.

[0056] Alternatively, the control device of the robot device can pre-store the three-dimensional position information of the surrounding objects arranged around the robot device. The control device can determine whether the robot device is in contact with the surrounding objects. For example, when the operator operates the teaching operating panel to manually drive the robot, the control device can determine whether the robot device is in contact with the surrounding objects. When the control device determines that the robot device is in contact with the surrounding objects, it implements control to prevent the robot from being driven. Alternatively, it can control the robot to slow down or stop when the robot device approaches the surrounding objects. In addition, the control device can display a warning such as stopping the robot on the display part of the display operating panel.

[0057] Figure 4 A perspective view showing the visual sensor and the stage when acquiring three-dimensional position information on the stage surface. Figure 2 and Figure 4 In the first control of the present embodiment, the operator manually drives the robot 1. The operator operates the input unit 49a of the teaching operation panel 49. The manual control unit 52 changes the position and posture of the robot according to the operation of the input unit 49a.

[0058] exist Figure 4 In the example shown, the operator places the visual sensor 30 at position P30a by jogging. Then, as shown by arrows 96a and 96b, the position and posture of the visual sensor 30 are changed to position P30b and position P30c. The shooting control unit 59 uses the cameras 31 and 32 to shoot images at predetermined intervals while the visual sensor 30 moves. In the shooting area 79 of the visual sensor 30, three-dimensional points can be generated on the surface opposite to the visual sensor 30. The operator changes the position and posture of the robot 1 so that the cameras 31 and 32 can shoot all surfaces that require three-dimensional position information. Here, in order to generate three-dimensional position information on all visible surfaces of the platform 65, images are shot at various positions and postures so that the entire surface of the platform 65 can be shot.

[0059] The position information generating unit 54 sets a three-dimensional point on the surface of the object included in the two-dimensional image obtained by the first camera 31 and the second camera 32. The position information generating unit 54 calculates the distance from the visual sensor 30 to the three-dimensional point set on the surface of the object based on the parallax between the image captured by the first camera 31 and the image captured by the second camera 32. For example, the three-dimensional point can be set according to the pixel of the imaging element. In addition, the position information generating unit 54 calculates the coordinate value of the three-dimensional point position in the sensor coordinate system 73 based on the distance from the visual sensor 30.

[0060] Figure 5 A stereogram showing a three-dimensional point group generated by the position information generating unit. Figure 5 It is a stereogram when three-dimensional points are arranged in three-dimensional space. Figure 5 , the outline of the stand 65 is indicated by a dotted line. The three-dimensional point 70 is arranged on the surface of the object facing the visual sensor 30. The position information generating unit 54 sets the three-dimensional point 70 on the surface of the object included in the imaging area 79. Here, a plurality of three-dimensional points 70 are arranged on the surface of the stand 65.

[0061] The position information generating unit 54 can Figure 5 In this way, the three-dimensional position information of the surface of the object is represented by a stereogram of a three-dimensional point group. In addition, the position information generating unit 54 can generate the three-dimensional position information of the surface of the object in the form of a distance image or a three-dimensional mapping. The distance image represents the position information of the surface of the object through a two-dimensional image. In the distance image, the distance from the visual sensor 30 to the three-dimensional point is represented by the concentration or color of each pixel. On the other hand, the three-dimensional mapping represents the position information of the surface of the object through a set of coordinate values ​​(x, y, z) of the three-dimensional points on the surface of the object. The coordinate values ​​at this time can be represented by any coordinate system such as the sensor coordinate system 73 or the robot coordinate system 71.

[0062] In this embodiment, a distance image is mainly used as an example of three-dimensional position information of the object surface. The position information generating unit 54 of this embodiment generates a distance image in which the color density changes according to the distance from the visual sensor 30 to the three-dimensional point 70 .

[0063] In addition, the position information generating unit 54 of the present embodiment is configured in the processing unit 51 of the operation processing device 24, but is not limited to this mode. The position information generating unit may also be configured inside the visual sensor. That is, the visual sensor may include an operation processing device including a processor such as a CPU, and the processor of the operation processing device of the visual sensor may function as the position information generating unit. In this case, a three-dimensional map or a distance image is output from the visual sensor.

[0064] The processing unit 51 of the present embodiment is configured to perform automatic imaging control for performing imaging with the vision sensor 30 or stopping imaging according to the driving state of the robot 1. The automatic imaging control of the present embodiment includes normal imaging control and synthesis control.

[0065] The motion detection unit 53 detects whether the robot 1 is moving. The normal shooting control is implemented while the motion detection unit 53 detects the movement of the robot 1. In the normal shooting control, the shooting control unit 59 uses the cameras 31 and 32 to shoot two-dimensional images at predetermined intervals. The shooting control unit 59 shoots images while changing the position and posture of the visual sensor 30. In addition, the shooting control unit 59 shoots two-dimensional images under the same exposure conditions that are predetermined.

[0066] The shooting control unit 59 can shoot images at predetermined time intervals. For example, the shooting control unit 59 uses the cameras 31 and 32 to shoot images at time intervals within a range of 300 msec to 500 msec. The predetermined interval is not limited to the time interval, and may also be an interval of a moving distance of a specified part. For example, the moving distance of the origin position of the flange coordinate system of the robot may be used. Alternatively, it may be an interval of a rotation angle of a specified part. In addition, the predetermined interval may not be a fixed interval. The position information generating unit 54 generates three-dimensional position information at predetermined intervals.

[0067] The exposure conditions in the general shooting control can adopt any exposure conditions that can shoot an image without blur. For example, a predetermined fixed exposure condition can be adopted. The exposure time is preferably short so that the two-dimensional image is not blurred. Therefore, for example, it is preferable to maximize the light intensity of the lighting device and set a faster shutter speed. The operator maximizes the light intensity of the lighting device and moves the camera to shoot a portion of the object with medium brightness. In addition, the minimum exposure time that can shoot a two-dimensional image can be set. That is, the maximum shutter speed can be determined according to the brightness of the lighting device.

[0068] However, in this embodiment, in order to capture the object from various positions and directions, defects such as overexposure or underexposure may occur in the two-dimensional image depending on the position of the lighting device, the orientation of the visual sensor, the material of the surface of the object, and the shape of the surface of the object. For the defective portion in the two-dimensional image, three-dimensional position information cannot be generated. Therefore, the information processing unit 55 of this embodiment implements synthesis control to supplement the defects of the two-dimensional image or the three-dimensional position information.

[0069] The motion detection unit 53 detects whether the robot 1 is moving. When the motion detection unit 53 detects that the robot 1 has stopped, synthesis control is implemented. In the synthesis control, the shooting control unit 59 automatically changes the exposure conditions to shoot a plurality of two-dimensional images. The shooting control unit 59 shoots a plurality of two-dimensional images while changing the exposure time from 0.1 msec to 100 msec by 10 times each time, for example. Alternatively, the shooting control unit 59 may also change the light intensity of the lighting device. For example, the shooting control unit 59 shoots a plurality of images with the cameras 31 and 32 while increasing the light intensity by 10% each time from 0% (off) to 100% (fully lit). The shooting is performed by both the first camera 31 and the second camera 32.

[0070] Next, the position information generating unit 54 generates a plurality of three-dimensional position information based on the two-dimensional images of the plurality of exposure conditions. Furthermore, the synthesizing unit 57 synthesizes the plurality of three-dimensional position information to generate one three-dimensional position information. Alternatively, the synthesizing unit 57 synthesizes the two-dimensional images of the plurality of exposure conditions according to the images of the respective cameras 31 and 32. The position information generating unit 54 may also generate the three-dimensional position information based on the two two-dimensional images synthesized by the cameras 31 and 32.

[0071] In this example, the position information generating unit 54 generates a distance image corresponding to each exposure condition based on the two-dimensional image corresponding to each exposure condition. The position information generating unit 54 generates a plurality of distance images. Furthermore, the synthesizing unit 57 synthesizes the plurality of distance images.

[0072] As a method for synthesizing distance images, the three-dimensional positions of pixels of distance images captured under other exposure conditions may be supplemented for pixels whose three-dimensional positions (distance information) are missing in a distance image. For example, the synthesis unit 57 selects a distance image with medium exposure conditions. The synthesis unit 57 determines pixels for which distance information does not exist. Furthermore, the synthesis unit 57 is able to obtain distance information of distance images obtained under other exposure conditions for pixels for which distance information does not exist. Alternatively, the synthesis unit 57 deletes pixels for which distance information is missing in a plurality of distance images. The synthesis unit 57 may also calculate the average value of distance information of a plurality of distance images for all pixels for which distance information exists.

[0073] Furthermore, when the three-dimensional position information is a three-dimensional map, the coordinate values ​​of each three-dimensional point can be supplemented or averaged. In this way, the synthesis unit can synthesize a plurality of three-dimensional position information to generate synthesized position information as synthesized three-dimensional position information.

[0074] Figure 6Flow chart showing the first control in this embodiment. In the first control, the operator manually drives the robot 1 by jogging operation to implement normal shooting control. The display unit 49b of the teaching operation panel 49 displays the distance image. And the operator determines whether there is a defect in the distance image. If there is a defect in the distance image, if the operator manually stops the robot 1, the synthesis control is implemented.

[0075] In step 81, the operator drives the robot 1 by operating the teaching operation panel 49 to achieve a position and posture for starting imaging. Figure 4 The manual control unit 52 drives the robot 1 according to the operator's operation so as to place the visual sensor 30 at the position P30a. The visual sensor 30 is placed at the position P30a directly above the platform 65 as the object.

[0076] In step 82, automatic imaging control is started. The operator sends a command to start automatic imaging control to the imaging control unit 59 and the motion detection unit 53 by operating the input unit 49a. In step 83, the operator manually drives the robot 1. For example, as shown by arrow 96a, the position and posture of the visual sensor 30 are changed from position P31a to position P30b.

[0077] In step 84, the motion detection unit 53 detects the driving of the robot 1. The imaging control unit 59 implements normal imaging control for imaging a two-dimensional image at a predetermined interval and under a predetermined exposure condition. The position information generation unit 54 generates three-dimensional position information based on the two-dimensional image captured by the cameras 31 and 32. Here, the position information generation unit 54 generates a distance image. The display unit 49b of the teaching operation panel 49 displays the distance image every time the distance image is generated.

[0078] While driving the robot 1, the operator observes the distance image displayed on the display unit 49b. The operator determines whether there is a defect in the distance information. No distance information is generated for a defective portion in the two-dimensional image. For example, the position of the three-dimensional point is not calculated in the sensor coordinate system 73. In the distance image, defective pixels for which distance information is missing are displayed at a specified concentration. For example, when the distance image is produced in such a way that the color concentration changes according to the distance, the defective pixels are displayed in the darkest color. The operator can easily determine whether the distance information is missing by observing the distance image.

[0079] When the operator determines that there is no defect in the distance image, the operator continues to manually drive the robot. In step 85, the motion detection unit 53 determines that the robot 1 has not stopped. The control returns to step 83. In steps 83 and 84, the driving of the robot 1 and the generation of the distance image are continued.

[0080] On the other hand, when the operator determines that there is a defect in the distance image during the driving of the robot 1, the operator manually stops the robot 1. That is, the inching operation of the robot 1 is interrupted and the movement of the visual sensor 30 is stopped. In step 85, the motion detection unit 53 detects the stop of the robot 1. The normal imaging control is stopped by the operator's operation. Control is transferred to step 88.

[0081] In step 88, the imaging control unit 59 determines whether the driving of the robot 1 is completed. When the imaging of the object from all directions is completed, the driving of the robot 1 is terminated. At this time, the operator sends a command to terminate the driving of the robot 1 to the processing unit 51 by operating the teaching operation panel 49. If the driving of the robot 1 is not completed, the control is transferred to step 86.

[0082] In step 86, the shooting control unit 59 starts synthesis control. The shooting control unit 59 changes the exposure conditions to shoot multiple two-dimensional images. Here, the position information generating unit 54 generates a distance image corresponding to the exposure conditions each time a shooting in the two-dimensional camera is performed. In step 87, the synthesis unit 57 of the information processing unit 55 generates a synthetic distance image as three-dimensional position information based on the multiple distance images. Through this control, a synthetic distance image without defects is generated. The display unit 49b displays the synthetic image generated by the synthesis unit 57. The operator can confirm the image displayed on the display unit 49b. After that, the control returns to step 83. The operator restarts to manually drive the robot. When the action detection unit 53 detects that the drive of the robot 1 is restarted, the synthesis control is stopped and the normal shooting control is started.

[0083] In step 88, when the manual driving of the robot 1 is completed, the control is transferred to step 89. That is, when the imaging from various directions is completed, the control is transferred to step 89. In step 89, the operator sends a command to end the automatic imaging control to the imaging control unit 59 and the motion detection unit 53 by operating the input unit 49a of the teaching operation panel 49, and the control ends.

[0084] The operator can change the position and posture of the visual sensor 30 so as to capture the desired entire surface of the platform 65. Furthermore, the platform 65 can be photographed from various angles to obtain three-dimensional position information of the surface. The position information generating unit 54 can convert the three-dimensional position information obtained in the sensor coordinate system 73 into three-dimensional position information of the robot coordinate system 71. The processing unit 51 can store the position and posture of the visual sensor 30 and the three-dimensional position information in the storage unit 42. Furthermore, the processing unit 51 can generate three-dimensional position information of the surface of the platform 65 by integrating or averaging a plurality of three-dimensional position information obtained at a plurality of positions and postures of the robot 1 (see Figure 5 ).

[0085] In addition, in the above-mentioned embodiment, the synthesis unit 57 synthesizes a plurality of distance images to generate a synthesized distance image, but the method is not limited to this. For example, the synthesis unit 57 may synthesize a plurality of two-dimensional images captured by the first camera 31 and the second camera 32. As a method of synthesizing two-dimensional images, any synthesis method with a large dynamic range may be used.

[0086] For example, the synthesis unit 57 can synthesize a two-dimensional image by using the HDR (High Dynamic Range) method. By synthesizing a plurality of two-dimensional images with changed exposure conditions, a two-dimensional synthetic image with a large dynamic range can be obtained. The synthesis unit 57 synthesizes the two-dimensional images according to the cameras 31 and 32 to generate a synthetic two-dimensional image. After that, the position information generation unit 54 can generate three-dimensional position information such as a distance image based on the synthetic two-dimensional image of the first camera 31 and the synthetic two-dimensional image of the second camera 32.

[0087] In this way, when there are no defects in the distance image, the operator can automatically repeat the shooting and the generation of the distance image until the driving of the robot is completed. When the operator finds a defect in the distance image, the driving of the robot 1 is stopped. Through this operation, the processing unit shoots multiple images with changed exposure conditions and automatically synthesizes three-dimensional position information or two-dimensional images. In addition, it is possible to generate synthetic position information synthesized with three-dimensional position information or three-dimensional position information generated from a synthetic two-dimensional image synthesized with a two-dimensional image. Even if the operator does not perform operations such as shooting multiple images or generating synthetic images, the three-dimensional position information of the desired part of the object can be generated by driving the robot. Therefore, the three-dimensional position information of the object can be easily generated.

[0088] In addition, when there is a defect in the two-dimensional image or the three-dimensional position information, the time to stop the robot can be shortened. As a result, the three-dimensional position information of the surrounding objects can be generated in a short time. Alternatively, the three-dimensional position information of the surrounding objects can be easily generated.

[0089] In addition, when the operator observes the synthetic distance image displayed on the display unit 49b, if there are defects in the synthetic distance image, the operator can also input an instruction to change the synthesis method to the teaching operation panel. The processing unit 51 can change the synthesis method and display the synthetic distance image on the display unit 49b. For example, the processing unit 51 can change the method of synthesizing the distance image to the method of synthesizing the two-dimensional image. In addition, the operator can also restart the driving of the robot when a synthetic distance image with fewer defects is obtained. Through this control, three-dimensional position information with fewer defects can be obtained more reliably.

[0090] In addition, when the robot is driven manually, the vision sensor may be temporarily stopped in order to change the moving direction of the vision sensor. Figure 4Sometimes, the visual sensor 30 is moved from the position P30a to the position P30b and then returned to the position P30a. The visual sensor 30 temporarily stops at the position P30b in order to change the moving direction. At this time, the synthesis control is implemented in the automatic shooting control. Even if the synthesis position information is not needed, the synthesis position information is generated. However, as the three-dimensional position information, there will be no problem even if the synthesis position information is used, so the synthesis control can also be implemented.

[0091] However, refer to Figure 2 The information processing unit 55 in this embodiment includes a determination unit 58 for determining whether there is a defect in the two-dimensional image or the three-dimensional position information. Therefore, the determination unit 58 can automatically determine whether there is a defect in the three-dimensional position information instead of the operator.

[0092] The determination unit 58 determines whether there are pixels without distance information in the three-dimensional position information generated by the normal shooting control. If there are pixels without distance information, the determination unit 58 can determine that there is a defect in the three-dimensional position information. The determination unit 58 can make the determination according to the three-dimensional position information generated.

[0093] Alternatively, the determination unit 58 can determine whether there is a defect in the two-dimensional image captured by the cameras 31 and 32. For example, the determination unit 58 determines that a pixel with a white pixel value that deviates from a predetermined determination range or a pixel with a black pixel value that deviates from a predetermined determination range is a defective pixel. When all pixels within an area of ​​a predetermined size are defective pixel values, the determination unit 58 determines that overexposure or underexposure has occurred in the area. That is, the determination unit 58 determines that there is a defect in the two-dimensional image.

[0094] When the determination unit 58 determines that there is a defect in the three-dimensional position information or the two-dimensional image, the manual control unit 52 sends a command to stop the driving of the robot 1 to the motion control unit 43 regardless of the operation of the teaching operation panel 49 by the operator. The motion control unit 43 stops the robot 1. Through this control, the stop of the robot 1 is detected, and the shooting control unit 59 performs synthesis control. The shooting control unit 59 automatically changes the exposure conditions to shoot a plurality of two-dimensional images. In addition, the synthesis unit 57 can synthesize the two-dimensional image or the three-dimensional position information.

[0095] In this way, the determination unit 58 can automatically determine that there is a defect in the three-dimensional position information or the two-dimensional image, and forcibly stop the robot 1. After the synthesis unit 57 completes the synthesis of the two-dimensional image or the three-dimensional position information, the manual control unit 52 releases the forced stop of the robot 1. The manual control unit 52 restarts the driving of the robot 1 according to the operation of the teaching operation panel 49 by the operator. That is, the operator can perform a jog operation by operating the input unit 49a of the teaching operation panel 49. The motion detection unit 53 detects the driving of the robot 1 and restarts the normal shooting control.

[0096] Figure 7 A block diagram showing a processing unit of a second control device in a first robot device in this embodiment. The second control device automatically drives the robot to obtain three-dimensional position information of an object. An operator prepares an action program for automatically driving the robot 1 in advance. Then, the robot is driven according to the action program, and an image is captured by the visual sensor 30 through automatic imaging control to obtain three-dimensional position information of surrounding objects.

[0097] The processing unit 61 includes an automatic control unit 62 that generates an action command for automatically driving the robot 1 according to a predetermined action program. The automatic control unit 62 sends the action command to the action control unit 43. The action control unit 43 drives the robot according to the action command from the automatic control unit 62. In addition, the action control unit 43 may also have the function of the automatic control unit 62. The structure of the other processing units 61 is the same as that of the processing unit 51 of the first control device 2 (see Figure 2 ).

[0098] Figure 8 The flowchart of the control of the second control device in this embodiment is shown. The second control device implements the second control to automatically drive the robot 1 and perform imaging. The operator pre-creates an action program that specifies the action path for driving the robot 1. For example Figure 4 As shown, an operation program is created that defines an operation path for the visual sensor 30 to pass through the position P30a, the position P30b, and the position P30c.

[0099] Preferably, the motion path of the robot included in the motion program is a path that can capture an object at various positions and postures of the vision sensor 30 so as to obtain three-dimensional position information of a desired surface of the object.

[0100] In step 82, the operator starts automatic shooting control by operating the teaching operation panel 49. In step 92, the operator starts automatic driving of the robot by operating the teaching operation panel 49. The automatic control unit 62 sends the motion command of the robot 1 to the motion control unit 43 according to the motion program. The position and posture of the robot 1 change, and the position and posture of the visual sensor 30 change.

[0101] In step 84, the motion detection unit 53 detects the driving of the robot 1, and the imaging control unit 59 performs normal imaging control. The normal imaging control captures a two-dimensional image of the object to generate three-dimensional position information. In this example, a distance image is generated.

[0102] In step 93, while the robot 1 is driven by the motion command of the automatic control unit 62, the determination unit 58 determines whether there is a defect in the two-dimensional image or the three-dimensional position information. In this example, it is determined whether there is a defect in the distance image. In step 93, if there is no defect in the two-dimensional image or the three-dimensional position information, control is transferred to step 95.

[0103] In step 95, the imaging control unit 59 determines whether the driving of the robot 1 is completed. In other words, it determines whether the robot 1 is driven to the end point of the motion path in the motion program. In step 95, if the driving of the robot 1 is not completed, the control returns to step 92. In this way, the three-dimensional position information of the surface of the object can be generated while the robot 1 is automatically driven according to the pre-made motion program.

[0104] On the other hand, in step 93, when the determination unit 58 determines that there is a defect in the two-dimensional image or the three-dimensional position information, control is transferred to step 94. In step 94, the automatic control unit 62 sends a command to stop the robot 1. The motion detection unit 53 detects the stop of the robot 1, and the imaging control unit 59 performs synthesis control. In steps 86 and 87, the synthesis unit 57 synthesizes the two-dimensional image or the three-dimensional position information. In this example, the synthesis unit 57 generates a synthesized distance image based on a plurality of distance images as three-dimensional position information with the defect corrected.

[0105] Thereafter, the automatic control unit 62 cancels the command to stop the robot 1. Then, the control returns to step 92. In order to automatically drive the robot 1, the synthesis control is stopped and the normal imaging control is started.

[0106] When the driving of the robot 1 ends in step 95, the control is transferred to step 89. In step 89, the operator sends a command to end the automatic imaging control to the processing unit 61 by operating the input unit 49a of the teaching operation panel 49, and the control ends.

[0107] In this way, in the second control of the second control device, the three-dimensional position information of the object can be automatically generated in order to automatically drive the robot 1. In addition, the command statement for starting the automatic shooting control and the command statement for stopping the automatic shooting control can also be recorded in the action program. Through this control, the automatic shooting control can be automatically started or ended according to the action program.

[0108] The other structures, functions and effects of the second control device are the same as those of the first control device, and therefore, they will not be described again here.

[0109] However, in the normal shooting control of this embodiment, the two-dimensional image is shot while driving the robot 1. Therefore, when the driving speed of the robot 1 is high, the two-dimensional image may be blurred. Therefore, in the normal shooting control, a control of changing the exposure time in advance according to the driving speed of the robot 1 may be implemented.

[0110] While the motion detection unit 53 is detecting the motion of the robot 1, the imaging control unit 59 can determine whether the speed at which the robot 1 is driven exceeds a predetermined determination value relative to the exposure time of the cameras 31 and 32. When the speed at which the robot 1 is driven exceeds the predetermined determination value relative to the exposure time of the cameras 31 and 32, the imaging control unit 59 can implement control to shorten the exposure time of the cameras 31 and 32.

[0111] For example, the shooting control unit 59 calculates a variable obtained by multiplying the moving speed (unit: mm / sec) of the visual sensor 30 by the exposure time (unit: sec) of the cameras 31 and 32. This variable is equivalent to the distance that the visual sensor 30 moves during exposure. When the variable exceeds a predetermined judgment value, the shooting control unit 59 determines that the speed at which the robot 1 is driven exceeds a predetermined judgment value relative to the exposure time of the cameras 31 and 32. Then, the shooting control unit 59 implements control to shorten the current exposure time. For example, the shooting control unit 59 implements control to reduce the exposure time at a predetermined ratio. Alternatively, the operator can prepare a table of exposure time relative to the above-mentioned variable value in advance. The shooting control unit 59 can implement control to shorten the exposure time based on the table.

[0112] In the first robot device 3, the visual sensor 30 is mounted on the robot 1. The object for obtaining the three-dimensional position information is an object arranged around the robot 1. By adopting this structure, the three-dimensional position information of the object arranged around the robot 1 can be easily obtained by changing the position and posture of the robot.

[0113] Fig. 9 A schematic side view of a second robot device according to the present embodiment is shown. In the second robot device 6, the position and posture of the visual sensor 30 are fixed. The visual sensor 30 is fixed to the installation surface of the robot 1 via a support member 69. The robot 1 is equipped with a manipulator 5 as a working tool.

[0114] In the second robot 6, the manipulator 5 corresponds to the object for which the three-dimensional position information is obtained. The control device of the second robot 6 images the surface of the manipulator 5 at various positions and postures of the manipulator 5. The control device generates the three-dimensional position information of the surface of the manipulator 5 based on the output of the visual sensor 30.

[0115] The structure of the control device of the robot device is similar to the first control device 2 or the second control device (see Figure 2 as well as Figure 7 When the operator manually drives the robot 1, Figure 2 When the robot 1 is automatically driven by the action program, it is possible to use Figure 7 The processing unit 61 is shown.

[0116] In the second robot device 6, the robot 1 is driven manually or automatically as shown by arrows 97a and 97b. For example, the manipulator 5 is moved to position P5a, position P5b, and position P5c. The control device implements automatic imaging control while automatically controlling the driving of the robot 1. That is, it is possible to implement normal imaging control while the robot 1 is being driven, and implement synthesis control when the robot 1 is stopped. In both the manual and automatic cases, the image of the manipulator 5 is captured as the object captured by the visual sensor 30. The position and posture of the robot 1 are changed so that the entire surface requiring the three-dimensional position information of the manipulator 5 is captured.

[0117] By driving the robot 1 and photographing the manipulator 5 from various directions, it is possible to obtain three-dimensional position information of the surface of the manipulator 5. Furthermore, based on the position and posture of the robot 1 when each three-dimensional position information is obtained, it is possible to obtain position information of the surface of the manipulator 5 relative to the position and posture of the robot 1. For example, the position of a three-dimensional point on the surface of the manipulator 5 set in the flange coordinate system 72 can be calculated.

[0118] The operator can perform an offline simulation using the three-dimensional position information of the manipulator 5. For example, the motion path of the robot 1 is generated in a manner that the manipulator 5 does not contact surrounding objects. Alternatively, in a simulation device that automatically generates the motion path of the robot, the motion path of the robot 1 can be automatically generated so that the manipulator 5 does not contact surrounding objects.

[0119] The other structures, functions and effects of the second robot device are the same as those of the first robot device, and therefore, they will not be described again here.

[0120] The three-dimensional visual sensor of this embodiment is a stereo camera, but is not limited to this method. As a visual sensor, any visual sensor including a two-dimensional camera can be used. For example, a visual sensor that detects the three-dimensional position of an object by a phase shift method can be used. Alternatively, as a three-dimensional visual sensor, a sensor that calculates the three-dimensional point position on the surface of an object by obtaining a two-dimensional image of an object whose distance to the object is known can be used.

[0121] In each of the above-mentioned controls, the order of the steps can be appropriately changed within the range where the functions and effects are not changed.

[0122] The above-mentioned embodiments can be combined appropriately. In the above-mentioned figures, the same or equivalent parts are marked with the same symbols. In addition, the above-mentioned embodiments are examples and do not limit the invention. In addition, in the embodiments, changes of the embodiments shown in the claims are included.

[0123] Explanation of symbols

[0124] 1. Robot

[0125] 2 Control device

[0126] 3.6 Robotic Device

[0127] 5. Robot

[0128] P5a, P5b, P5c positions

[0129] 23 Position detector

[0130] 24. Processing device

[0131] 30 Vision Sensor

[0132] P30a, P30b, P30c position

[0133] 31, 32 Camera

[0134] 43 Motion Control Unit

[0135] 49 Teaching operation panel

[0136] 49a Input section

[0137] 52 Manual control unit

[0138] 53 Motion detection unit

[0139] 54 Position information generation unit

[0140] 57 Synthesis Department

[0141] 58 Judgment Department

[0142] 59 Shooting Control Unit

[0143] 62 Automatic Control Department

[0144] 65 racks

[0145] 70 3D points.

Claims

1. A control device for a robot device, the robot device having a three-dimensional visual sensor and a robot, the three-dimensional visual sensor including a two-dimensional camera for photographing an object, characterized in that: The control device has: An action detection unit, which detects the action state of the robot; A shooting control unit that changes the exposure conditions of the two-dimensional camera; a position information generating unit for generating three-dimensional position information of the object based on the two-dimensional image captured by the two-dimensional camera; as well as a synthesis unit that implements control for synthesizing a plurality of two-dimensional images or controls for synthesizing a plurality of three-dimensional position information, The imaging control unit captures a two-dimensional image at a predetermined interval under a predetermined exposure condition while the motion detection unit detects the motion of the robot. When the motion detection unit detects that the robot has stopped, the imaging control unit changes exposure conditions to capture a plurality of two-dimensional images, and the synthesis unit synthesizes the plurality of two-dimensional images or a plurality of three-dimensional position information.

2. The control device according to claim 1, characterized in that: The control device includes a manual control unit that generates an operation instruction for manually driving the robot according to an operation of an operator. When an operator manually stops the robot, the motion detection unit detects the stop of the robot.

3. The control device according to claim 1 or 2, characterized in that: The control device has: a motion control unit that controls the motion of the robot; and a determination unit for determining whether there is a defect in the two-dimensional image or the three-dimensional position information, When the determination unit determines that the defect exists in the two-dimensional image or the three-dimensional position information, the motion control unit stops the robot.

4. The control device according to claim 1, characterized in that: The control device has: A motion control unit, which controls the motion of the robot; an automatic control unit that generates an operation instruction for automatically driving the robot according to a predetermined operation program; and a determination unit for determining whether there is a defect in the two-dimensional image or the three-dimensional position information, While the robot is being driven by the motion command of the automatic control unit, when the determination unit determines that the defect exists in the two-dimensional image or the three-dimensional position information, the motion control unit stops the robot.

5. The control device according to any one of claims 1 to 4, characterized in that: When the speed of driving the robot exceeds a predetermined determination value relative to the exposure time of the two-dimensional camera while the motion detection unit is detecting the motion of the robot, the imaging control unit controls to shorten the exposure time of the two-dimensional camera.

6. The control device according to any one of claims 1 to 5, characterized in that: The exposure condition is at least one of an exposure time of the two-dimensional camera and a light amount of the lighting device.

7. A robot device, characterized in that: have: The control device according to claim 1; A three-dimensional vision sensor including a two-dimensional camera that photographs an object; and robot.

8. The robot device according to claim 7, characterized in that: The three-dimensional visual sensor is installed on the robot. The target object is an object arranged around the robot.

9. The robot device according to claim 7, characterized in that: The robot device comprises: a working tool mounted on the robot, The three-dimensional vision sensor is fixed at a predetermined position by a supporting component. The object is the working tool.

Images