Three-dimensional measuring apparatus for generating position information of an object surface based on images captured by a plurality of cameras

Abstract

Provided is a three-dimensional measuring device that improves the accuracy of position information of an object surface. A control device (2) has a processing section (51) that processes a first image captured by a first camera (31) and a second image captured by a second camera (32). The processing section (51) controls block matching. A first determination value related to a score of block matching and a second determination value larger than the first determination value are determined in advance. The processing section (51) generates a first distance image using the first determination value and a second distance image using the second determination value. The processing section (51) generates a composite distance image using distance information of pixels included in the first distance image in a region corresponding to a contour of a workpiece and using distance information of pixels included in the second distance image in a region other than the region corresponding to the contour.

Description

Technical Field

[0001] This invention relates to a three-dimensional measuring device for generating positional information of an object's surface based on images captured by multiple cameras. Background Technology

[0002] In conventional technology, there are known 3D measuring devices that use visual sensors to capture images and detect the position information of an object's surface based on the obtained images. As a visual sensor for detecting 3D position, a stereo camera comprising two 2D cameras is known. In the processing of images captured by the stereo camera, the distance from the stereo camera to the object is calculated based on the parallax of the object's position in the images captured by one camera and the other camera. Furthermore, the 3D position of a measuring point set on the object's surface can be calculated based on the distance to the object and the positions of the two cameras.

[0003] In the processing of images from stereo cameras, it is necessary to detect the position of an object in an image captured by another camera in a manner corresponding to the position of an object contained in an image captured by one camera. Therefore, a control method is known that sets a block of a predetermined size in one image and searches for the position corresponding to that block in another image (e.g., Japanese Patent Application Publication No. 2001-82927). Such a method is called block matching because it searches for each block individually.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2001-82927 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] In stereo camera image processing, when performing block matching, a first image is captured using one camera and a second image is captured using the other camera. It is determined whether a selected block in the first image corresponds to a selected block in the second image. Each block contains multiple pixels. If a block in the second image corresponds to a block in the first image, disparity is calculated for the pixels contained in that block.

[0009] In control using block matching to calculate position information of a surface such as a distance image, it is sometimes difficult to accurately measure the profile of a small step of an object, a slight gap between objects, or a fine shape of an object. For example, in the case of generating a distance image using a stereo camera, a small step of an object or a gap between objects, or the like sometimes disappears. Or, sometimes accurate distance information cannot be obtained in a small step of an object or a gap between objects, or the like. Thus, in a stereo camera, there is a problem that it is difficult to accurately obtain position information of an object surface in a portion where a change in shape in an image is fine.

[0010] Means for solving the problem

[0011] The three-dimensional measurement apparatus of the first aspect of the present disclosure includes a vision sensor including a first camera that captures a first image and a second camera that captures a second image, and a profile detection sensor that detects a profile of an object. The three-dimensional measurement apparatus includes a processing unit that detects position information of a surface of the object based on the first image and the second image. The processing unit includes a profile detection unit that detects the profile of the object based on an output of the profile detection sensor. The processing unit includes a block search unit that sets a selected block composed of a plurality of pixels including a selected pixel selected from among the pixels in the first image, and searches for a specific block corresponding to the selected block in the second image. The processing unit includes a calculation unit that calculates distance information of the pixel based on a parallax between a position of the selected pixel in the selected block and a position of a specific pixel corresponding to the selected pixel in the specific block. The processing unit includes a generation unit that generates the position information of the surface of the object including the distance information of the plurality of pixels. The processing unit includes a synthesis unit that synthesizes the position information of the surface of the object generated under different conditions. The block search unit calculates a cumulative value obtained by accumulating a size of a difference or a square value of the difference between a value of a pixel included in the selected block and a value of a pixel included in the specific block. The block search unit determines that the pixel included in the specific block corresponds to the pixel included in the selected block when the cumulative value is less than a predetermined determination value. The block search unit sets information indicating that the selected pixel is an invalid pixel as the distance information of the selected pixel when the cumulative value exceeds the predetermined determination value or when the specific block corresponding to the selected block is not detected. A first determination value related to the cumulative value and a second determination value related to the cumulative value that is larger than the first determination value are determined in advance. The block search unit, the calculation unit, and the generation unit generate position information of a first surface using the first image, the second image, and the first determination value, and generate position information of a second surface using the first image, the second image, and the second determination value. The synthesis unit sets a region corresponding to the profile based on the profile detected by the profile detection unit, adopts distance information of a pixel included in the position information of the first surface for a pixel included in the region corresponding to the profile, adopts distance information of a pixel included in the position information of the second surface for a pixel included in a region other than the region corresponding to the profile, and thereby generates position information of a surface obtained by synthesizing the position information of the first surface and the position information of the second surface.

[0012] The three-dimensional measurement device according to the second aspect of the present disclosure includes a vision sensor including a first camera that captures a first image and a second camera that captures a second image, and a profile detection sensor that detects a profile of an object. The three-dimensional measurement device includes a processing unit that detects position information of a surface of the object based on the first image and the second image. The processing unit includes a profile detection unit that detects the profile of the object based on an output of the profile detection sensor. The processing unit includes a block search unit that sets a selected block composed of a plurality of pixels including a selected pixel in the first image, and searches for a specific block corresponding to the selected block in the second image. The processing unit includes a calculation unit that generates distance information of the pixel based on a parallax between a position of the selected pixel in the selected block and a position of a specific pixel corresponding to the selected pixel in the specific block. The processing unit includes a generation unit that generates the position information of the surface of the object including the distance information of the plurality of pixels. The processing unit includes a synthesis unit that synthesizes the position information of the surface of the object generated under different conditions. A first selected block and a second selected block including more pixels than the first selected block are determined in advance. The block search unit, the calculation unit, and the generation unit generate position information of a first surface using the first image, the second image, and the first selected block, and generate position information of a second surface using the first image, the second image, and the second selected block. The synthesis unit sets a region corresponding to the profile based on the profile detected by the profile detection unit, sets distance information of a pixel included in the position information of the first surface for a pixel included in the region corresponding to the profile, sets distance information of a pixel included in the position information of the second surface for a pixel included in a region other than the region corresponding to the profile, and thereby generates position information of a surface obtained by synthesizing the position information of the first surface and the position information of the second surface.

[0013] Effects of the Invention

[0014] According to the aspect of the present disclosure, it is possible to provide a three-dimensional measurement device in which the accuracy of position information of a surface of an object is improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a perspective view of a robot device according to an embodiment.

[0016] Figure 2 is a block diagram of the robot device.

[0017] Figure 3 is a schematic view of a vision sensor according to an embodiment.

[0018] Figure 4 is a view of a selected block in a first image and a search region and a search block in a second image.

[0019] Figure 5is a diagram of a selected block and a search area for explaining a first procedure of control to search for a specific block corresponding to the selected block.

[0020] Figure 6 is a diagram of a selected block and a search area for explaining a second procedure of control to search for a specific block corresponding to the selected block.

[0021] Figure 7 is a perspective view of a vision sensor and a workpiece for explaining an example of photographing the workpiece by the vision sensor.

[0022] Figure 8 is an example of a distance image obtained by photographing the workpiece.

[0023] Figure 9 is a 2-dimensional image photographed by a first camera of a robot device.

[0024] Figure 10 is a flowchart of a method of setting a contrast threshold for detecting a contour from a 2-dimensional image.

[0025] Figure 11 is a flowchart of a first control in an embodiment.

[0026] Figure 12 is an enlarged view of an image for explaining a region corresponding to a contour.

[0027] Figure 13 is a flowchart of control to generate a synthetic distance image.

[0028] Figure 14 is a distance image generated by the first control.

[0029] Figure 15 is a distance image generated by control of a comparative example.

[0030] Figure 16 is a flowchart of a method of setting a decision value of a first score.

[0031] Figure 17 is a flowchart of a second control in an embodiment.

[0032] Figure 18 is a flowchart of a method of setting a block size of a first selected block.

[0033] Figure 19 is a distance image generated by the second control. DETAILED DESCRIPTION

[0034] REFERENCE Figures 1 to 19A three-dimensional measurement device in the embodiment will be described. The three-dimensional measurement device of the embodiment has a vision sensor including a first camera that captures a first image and a second camera that captures a second image. The three-dimensional measurement device generates position information of an object surface based on the first image and the second image. In particular, the position information including information of a measurement point set in three dimensions on the object surface is generated.

[0035] Figure 1 is a perspective view of a robot device of the embodiment. Figure 2 is a block diagram of the robot device of the embodiment. Refer to Figure 1 and Figure 2 The robot device 3 has a robot 1 that moves a robot hand 5 that grips workpieces 61, 62, and the robot hand 5. The robot device 3 has a control device 2 that controls the robot device 3. The robot device 3 has a vision sensor 30 that generates position information of a measurement point in three dimensions corresponding to a surface of the workpieces 61, 62 as an object.

[0036] The workpieces 61, 62 of the embodiment are rectangular parallelepiped corrugated boxes. The robot hand 5 is an end effector that grips or releases the workpieces 61, 62. The robot hand 5 of the embodiment is a suction robot hand that grips a surface of the workpiece 61, 62 by suction. As an end effector mounted on the robot 1, it is not limited to this mode, and any work tool corresponding to a work performed by the robot device 3 can be adopted. For example, as an end effector, a work tool for performing welding or a work tool that applies a sealing material to a surface of a workpiece, or the like can be adopted. That is, the three-dimensional measurement device of the embodiment can be applied to a robot device that performs any work.

[0037] The robot 1 of the embodiment is a multi-joint robot including a plurality of joint sections 18. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a rotary base 13. The rotary base 13 is supported by a base 14. The robot 1 includes a wrist section 15 connected to an end of the upper arm 11. The wrist section 15 includes a flange 16 that fixes the robot hand 5. The constituent components of the robot 1 are formed to rotate around a predetermined driving axis. As a robot, it is not limited to this mode, and any robot that can move a work tool can be adopted.

[0038] Figure 3Fig. 1 is a schematic diagram showing the camera of the embodiment. The vision sensor 30 of the embodiment is a stereo camera including a first camera 31 and a second camera 32. The cameras 31, 32 are 2-dimensional cameras capable of capturing 2-dimensional images. As the cameras 31, 32, any cameras provided with a shooting element such as a CCD (Charge-Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor can be used. The two cameras 31, 32 are arranged separately from each other. The relative positions of the two cameras 31, 32 are determined in advance. The two cameras 31, 32 of the embodiment are arranged so that the optical axes of the respective cameras 31, 32 are parallel to each other. The vision sensor 30 of the embodiment includes a projector 33 that projects a pattern light such as a stripe pattern toward the workpieces 61, 62. The cameras 31, 32 and the projector 33 are arranged inside a housing 34.

[0039] Referring to Figure 1 and Figure 2 , the vision sensor 30 is supported by a support member 66. The position of the vision sensor 30 of the embodiment is fixed. The vision sensor 30 is arranged at a position from which the workpieces 61, 62 can be captured.

[0040] The three-dimensional measurement device can generate position information of the surface of the object in the form of a range image or a three-dimensional map by processing the image acquired by the vision sensor 30. The range image is an image that represents the position information of the surface of the object by an image. In the range image, the position of the surface of the object or the distance from the vision sensor 30 is represented by the density or color of each pixel. On the other hand, the three-dimensional map represents the position information of the surface of the object by a set of coordinate values (x, y, z) of the measured points of the surface of the object corresponding to the pixels. In the embodiment, the position information of the surface of the object is described taking the range image as an example.

[0041] The robot 1 of the embodiment includes a robot drive device 21 that drives the constituent members such as the upper arm 11. The robot drive device 21 includes a plurality of drive motors for driving the upper arm 11, the lower arm 12, the rotary base 13, and the wrist 15. The manipulator 5 includes a manipulator drive device 22 that drives the manipulator 5. The manipulator drive device 22 of the embodiment drives the manipulator 5 by air pressure. The manipulator drive device 22 includes a pump and a solenoid valve or the like for depressurizing the internal space of the suction pad.

[0042] The control device 2 controls the robot 1 and the robot hand 5. The control device 2 has an arithmetic processing device (computer) including a CPU (Central Processing Unit) as a processor. The arithmetic processing device has a RAM (Random Access Memory) and a ROM (Read Only Memory) and the like which are connected to each other via a bus. The robot device 3 of the present embodiment automatically carries the workpieces 61, 62 based on the motion program 41. The robot drive device 21 and the robot hand drive device 22 are controlled by the control device 2.

[0043] The control device 2 includes a storage section 42 which stores information related to the control of the robot device 3. The storage section 42 can be constituted by a volatile memory, a non-volatile memory, or a storage medium such as a hard disk which can store information. The motion program 41 which is generated in advance for the motion of the robot 1 is input to the control device 2. The motion program 41 is stored in the storage section 42.

[0044] The control device 2 includes a motion control section 43 which transmits a motion instruction. The motion control section 43 transmits a motion instruction for driving the robot 1 to a robot drive section 44 based on the motion program 41. The robot drive section 44 includes a circuit which drives a drive motor. The robot drive section 44 supplies power to the robot drive device 21 based on the motion instruction. In addition, the motion control section 43 transmits a motion instruction for driving the robot hand drive device 22 to a robot hand drive section 45. The robot hand drive section 45 includes a circuit which drives a pump and the like. The robot hand drive section 45 supplies power to the pump and the like based on the motion instruction.

[0045] The motion control section 43 corresponds to a processor which drives in accordance with the motion program 41. The processor reads in the motion program 41 and implements the control prescribed in the motion program 41, thereby functioning as the motion control section 43.

[0046] The robot 1 includes a state detector for detecting the position and posture of the robot 1. The state detector in the present embodiment includes a position detector 23 which is attached to a drive motor of each drive shaft of the robot drive device 21. The position and posture of the robot 1 are detected from the output of the position detector 23. As the state detector, it is not limited to the position detector attached to the drive motor, and any detector which can detect the position and posture of the robot 1 can be employed.

[0047] The control device 2 includes a teaching operation panel 49 as an operation panel by which the robot device 3 is manually operated by an operator. The teaching operation panel 49 includes an input section 49a that inputs information related to the robot 1, the robot hand 5, and the vision sensor 30. The input section 49a is constituted by components such as a keyboard and a dial. The teaching operation panel 49 includes a display section 49b that displays information related to the control of the robot device 3. The display section 49b is constituted by a display panel such as a liquid crystal display panel.

[0048] In the robot device 3 of the present embodiment, a world coordinate system 71 that does not move when the position and posture of the robot 1 change is set. In the world coordinate system 71, the position of the robot 1 is expressed. Figure 1 In the example shown, the origin of the world coordinate system 71 is disposed at the base 14 of the robot 1. The world coordinate system 71 is also referred to as a reference coordinate system. In the world coordinate system 71, the position of the origin is fixed, and the orientation of the coordinate axes is fixed. Even if the position and posture of the robot 1 change, the position and posture of the world coordinate system 71 do not change.

[0049] In addition, in the robot device 3, a tool coordinate system 72 having an origin set at an arbitrary position of a work tool is set. The position and posture of the tool coordinate system 72 change together with the robot hand 5. In the present embodiment, the origin of the tool coordinate system 72 is set to the tool tip point. Also, in the robot device 3, a camera coordinate system 73 is set for the vision sensor 30. The camera coordinate system 73 is a coordinate system in which the origin is fixed at the vision sensor 30.

[0050] If the position and posture of the robot 1 change, the position and posture of the origin of the tool coordinate system 72 change. For example, the position of the robot 1 corresponds to the position of the tool tip point (the position of the origin of the tool coordinate system 72). In addition, the posture of the robot 1 corresponds to the posture of the tool coordinate system 72 with respect to the world coordinate system 71.

[0051] The robot device 3 of the present embodiment includes a three-dimensional measuring device that detects the workpieces 61, 62. In the present embodiment, the control device 2 functions as the three-dimensional measuring device. The three-dimensional measuring device includes the vision sensor 30, a profile detection sensor that detects the profile of the workpieces 61, 62, and a processing section 51 that detects position information of the surface of the workpieces 61, 62 based on the first image captured by the first camera 31 and the second image captured by the second camera 32.

[0052] The profile detection sensor can employ any sensor capable of detecting the profile of a step portion, a recess portion, a protrusion portion, and an outer edge of an object in a 2-dimensional image. In the present embodiment, the first camera 31 of the vision sensor 30 functions as the profile detection sensor. Based on the first image captured by the first camera 31, the profile of the surface of the workpiece 61, 62 is detected. According to this configuration, no profile detection sensor needs to be provided other than the vision sensor 30, and the structure of the three-dimensional measuring device can be simplified. The second camera 32 can also be used as the profile detection sensor. Alternatively, a 2-dimensional camera or the like other than the vision sensor 30 can be provided as the profile detection sensor in the robot device.

[0053] The processing section 51 includes a profile detection section 52 that detects the profile of an object based on the output of the profile detection sensor. The processing section 51 includes a block search section 53 that sets a selected block composed of a plurality of pixels in the first image and searches for a specific block corresponding to the selected block in the second image. The processing section 51 includes a calculation section 54 that calculates distance information of a pixel based on the parallax between the position of a selected pixel in the selected block and the position of a specific pixel corresponding to the selected pixel in the specific block. The processing section 51 includes a generation section 55 that generates position information of the surface of an object including distance information of a plurality of pixels. The processing section 51 includes a synthesis section 56 that synthesizes position information of the surface of an object generated under mutually different conditions.

[0054] In addition, the processing section 51 includes a photographing control section 57 that sends an instruction to the vision sensor 30 to capture an image. The processing section 51 includes a motion instruction section 58 that generates a motion instruction to drive the robot 1 based on the position information of the surface of an object after synthesis.

[0055] The processing section 51 described above corresponds to a processor driven in accordance with the motion program 41. In particular, the profile detection section 52, the block search section 53, the calculation section 54, the generation section 55, and the synthesis section 56 each correspond to a processor driven in accordance with the motion program 41. In addition, the photographing control section 57 and the motion instruction section 58 correspond to a processor driven in accordance with the motion program 41. The processor reads in the motion program 41 and implements the control specified in the motion program 41, thereby functioning as each unit.

[0056] The robot device 3 of the present embodiment generates a distance image of the workpieces 61, 62 on the basis of the output of the vision sensor 30 before the robot 5 grips the workpieces 61, 62. The imaging control section 57 sends an instruction to the vision sensor 30 to take an image. The processing section 51 generates position information of the surfaces of the workpieces 61, 62 on the basis of the images taken by the first camera 31 and the second camera 32 of the vision sensor 30. For example, the position information of the surfaces is generated by the camera coordinate system 73. The processing section 51 can convert the position information of the surfaces represented by the camera coordinate system 73 into position information of the surfaces represented by the world coordinate system 71 in accordance with the position and posture of the camera coordinate system 73 with respect to the world coordinate system 71.

[0057] The action instruction section 58 of the processing section 51 detects the shape and position of the surface of the workpiece 61 on the basis of the position information of the surface of the workpiece 61. The action instruction section 58 sends an action instruction of the robot 1 to the action control section 43 so that the surface of the workpiece 61 can be gripped by the robot 5. The action control section 43 grips the workpiece 61 with the robot 5 after changing the position and posture of the robot 1 on the basis of the action instruction. Then, the robot 1 carries the workpiece 61 to a target position on the basis of the action program 41. Next, the robot device 3 carries the workpiece 62 after gripping the workpiece 62 on the basis of the position information of the surface of the workpiece 62. In this way, the robot device 3 can detect the positions of the workpieces 61, 62 to carry the workpieces 61, 62.

[0058] Next, the control to generate a distance image by the three-dimensional measurement device of the present embodiment will be described. The three-dimensional measurement device of the present embodiment generates distance images under a plurality of conditions different from each other and generates a synthesized distance image by synthesizing the plurality of distance images.

[0059] Figure 4 The first image taken by the first camera and the second image taken by the second camera are indicated by 77 and 78, respectively. The vision sensor 30 takes the first image 77 by the first camera 31. The vision sensor 30 takes the second image 78 by the second camera 32. The first image 77 and the second image 78 are 2-dimensional images. Each of the images 77, 78 is composed of a plurality of pixels. A screen coordinate system 74 having an origin at a predetermined point in the image is set in each of the images 77, 78. The position in the image represented by the screen coordinate system 74 corresponds to the position in the image pickup element arranged inside each of the cameras 31, 32.

[0060] Figure 5 A diagram showing a selected block and a search block for explaining the first process when block matching is performed is indicated by 80. The selected block 81 set in the first image 77 and the search region 84 set in the second image 78 are shown in an enlarged view in Figure 5 Figure 2 ,​Figure 4 as well as Figure 5 The block search unit 53 of the processing unit 51 selects one pixel from the pixels 85 contained in the first image 77 as selected pixel 85a. The block search unit 53 sets a selection block 81 that includes the selected pixel 85a and the pixels 85 surrounding the selected pixel 85a.

[0061] In the selected block 81 of this embodiment, the selected pixel 85a is surrounded by a plurality of pixels 85. Figure 5 In the selected block 81 shown, three pixels 85 are selected horizontally (in the Y-axis direction of the screen coordinate system 74) and three pixels 85 are selected vertically in the first image 77. The block size of the selected block 81 is not limited to this method and can contain any number of pixels. In addition, in this embodiment, a selected pixel 85a is arranged in the center of the selected block 81, but it is not limited to this method and the selected pixel 85a can be arranged at any position in the selected block 81.

[0062] The block search unit 53 controls the search within the second image 78 to find a block match that corresponds to a specific block 82 of the selected block 81. Each pixel 85 has a value set by quantifying information such as density, brightness, or color. The block search unit 53 searches the second image 78 for blocks whose values ​​correspond well to these pixels. Then, the block search unit 53 designates that block as the specific block 82.

[0063] The block search unit 53 sets a search area 84 in the second image 78 to detect a specific block 82 corresponding to the selected block 81. The search area 84 can be, for example, a region located at the same position as the selected block 81 and parallel to the epipolar line. The height of the search area 84 can be set to be the same as the height of the selected block 81. In this embodiment, the search area 84 is set in the second image 78 in a manner that extends in the direction of the Y-axis of the screen coordinate system 74. The block search unit 53 selects the search block 83 within the search area 84. The shape and size of the search block 83 can be set to be the same as the size and shape of the selected block 81.

[0064] exist Figure 5 In the example shown, a search block 83a is defined within the search area 84. The block search unit 53 in this embodiment performs block matching using the SAD (Sum of Absolute Difference) method. The block search unit 53 calculates a score SC1 based on the value of pixel 86 contained in search block 83a and the value of pixel 85 contained in selected block 81. The score is a cumulative value obtained by accumulating the differences in pixel values ​​between corresponding pixels in each block 81, 83a. This score is also called the matching score. Figure 5 In the example shown, the score SC1 is calculated using the following equation (1).

[0065] SC1 = |5-4| + |3-3| + |4-4| + |5-5| + |6-7| + |5-3| + |6-6| + |8-8| + |8-7| = 5... (1)

[0066] Next, as indicated by an arrow 101, the block search section 53 moves the position of the search block 83 along the search region 84. In the present embodiment, the search block 83 is shifted by a distance of the width of one pixel 86. As the distance by which the search block 83 is moved, it is not limited to this manner, but can be a distance of the width of two or more pixels.

[0067] Figure 6 A diagram showing the selected block and the search block at the time of explaining the second procedure while the block matching is being performed. Referring to Figure 5 and Figure 6 The block search section 53 sets the search block 83b by shifting the search block 83. The block search section 53 calculates the score SC2 based on the values of the pixels 85 included in the selected block 81 and the values of the pixels 86 included in the search block 83b. In the example shown in Figure 6 , the score SC2 becomes the following expression (2).

[0068] SC2 = |5-3| + |3-4| + |4-8| + |5-7| + |6-3| + |5-9| + |6-8| + |8-7| + |8-10| = 21... (2)

[0069] Thus, the block search section 53 calculates the score while shifting the position of the search block 83 along the search region 84. The block search section 53 moves the search block 83 from one end to the other end of the search region 84. The block search section 53 calculates the score at the position of each search block 83 while shifting the position of the search block 83 little by little. In the present embodiment, the smaller the score, the more it is determined that the search block 83 better corresponds to the selected block 81. In the example described above, it is determined that Figure 5 the search block 83a shown in Figure 6 corresponds to the selected block 81 better than the search block 83b shown in

[0070] The block search section 53 sets the search block 83 at the position of which the score is the lowest among the positions of the search blocks 83 as the specific block 82 corresponding to the selected block 81. Thus, the block search section 53 can set the search block 83 in which the values of the pixels 85 included in the selected block 81 are close as the specific block 82.

[0071] The block search section can also calculate a cumulative value obtained by cumulating the square values of the differences between the values of the pixels included in the selected block and the values of the pixels included in the search block, as the score. This method is called the SSD (Sum of Squared Difference) method. In this method, the smaller the value of the score, the better the correspondence of the two blocks can be judged.

[0072] Thus, the block search section 53 can search for the specific block 82 that best corresponds to the selected block 81 by performing block matching. Next, the block search section 53 compares the score calculated from the selected block 81 and the specific block 82 with a predetermined decision value. In the present embodiment, the decision value for the score is predetermined. The block search section 53 judges that the pixels 86 included in the specific block 82 correspond to the pixels 85 included in the selected block 81 if the score is smaller than the decision value. The block search section 53 selects the specific pixel that corresponds to the selected pixel 85a in the specific block 82. The search block 83a shown in FIG. 8 is set as the specific block 82 in this case. Figure 5 The pixel disposed at the center of the search block 83a becomes the specific pixel in the case where the search block 83a shown in FIG. 8 is set as the specific block 82.

[0073] On the other hand, the block search section 53 judges that the specific block 82 does not sufficiently correspond to the selected block 81 if the score is equal to or larger than the decision value. The block search section 53 sets information indicating an invalid pixel as the distance information of the selected pixel 85a. The invalid pixel refers to a pixel that does not include information such as a specific distance and a position of a specific measurement point. In addition, the block search section 53 sets information indicating an invalid pixel as the distance information of the selected pixel in the case where no specific block corresponding to the selected block is detected.

[0074] The block search section 53 can select all the pixels included in the first image 77 as the selected pixels and perform the same control as described above. The specific block 82 and the specific pixel are set for the pixels 85 included in the first image 77. Alternatively, the pixels 85 are set as invalid pixels. In addition, the block search section 53 can perform the same control as described above for the pixels disposed in a predetermined region of the first image 77.

[0075] Next, the calculating section 54 calculates the disparity between the position of the selected pixel 85a in the selected block 81 and the position of the specific pixel in the specific block 82. The calculating section 54 calculates the distance from the vision sensor 30 to the surface of the object based on the disparity. In particular, the calculating section 54 calculates the distance to the measurement point on the surface of the object corresponding to the selected pixel 85a and the specific pixel. In addition, the calculating section 54 can calculate the position of the measurement point on the surface of the object based on the calculated distance. The position of the measurement point can be calculated by the camera coordinate system 73.

[0076] The calculating section 54 sets distance information of a pixel for each pixel 85 of the first image 77. The distance information of a pixel is, for example, information of a distance from the vision sensor 30 to a measurement point or a position of a measurement point, and the like. Further, as described above, the distance information of a pixel includes information that is an invalid pixel. Further, the distance information of a pixel is information related to a position of a measurement point, and thus corresponds to distance information of a measurement point.

[0077] The generating section 55 generates position information of a surface of an object including distance information of a plurality of pixels using the distance information of each pixel. In the example here, the generating section 55 sets a density of a color of a pixel according to a distance from the vision sensor 30 to an object. By setting an image of a density corresponding to a distance for each pixel, a distance image can be generated.

[0078] Figure 7 A perspective view showing a vision sensor and a workpiece for explaining an example of a distance image is shown. In this example, a workpiece 63 is arranged on a surface of a stage 67. The workpiece 63 is arranged to be inclined with respect to the surface of the stage 67. The surface of the stage 67 extends perpendicularly with respect to optical axes of the cameras 31, 32 of the vision sensor 30. By processing images captured by the cameras 31, 32 of the vision sensor 30, distance information of measurement points set on a surface of the workpiece 63 can be detected as indicated by arrows 102, 103.

[0079] Figure 8 A distance image obtained by the vision sensor of Figure 7 is shown. Referring to Figure 7 and Figure 8 , in the example here, the distance image 91 is generated in such a manner that a color is denser as a distance from the vision sensor 30 is farther. On the surface of the workpiece 63, a color is lighter as a distance from the vision sensor 30 is closer. The surface of the stage 67 is located at a position far away, and thus a color is set to be the densest for a pixel.

[0080] In a region corresponding to an outer edge of the workpiece 63, information that is an invalid pixel is set as distance information of a pixel. An invalid pixel region 97 is formed by invalid pixels. In the example here, the distance image 91 is generated in such a manner that no color is given to an invalid pixel. Further, an invalid pixel region 97 is formed in a region outside the stage 67.

[0081] However, in the block matching performed by the block search section 53, if the determination value of the score is set to be large, many specific blocks that are not set to be invalid pixels remain. As a result, the number of pixels that contain information on a specific distance or the like increases. However, pixels that contain information on a specific distance or the like are set in portions where the reliability of the block matching is low, such as the step portion or the outer edge of the workpiece. As a result, the reliability of the distance image decreases. For example, in the outer edge, the convex portion, the concave portion, or the step portion of the workpiece, a distance image in which the profile disappears is sometimes generated.

[0082] On the other hand, if the determination value of the score is set to be small, if the values of the plurality of pixels in the selected block and the values of the plurality of pixels in the specific block are not values close to each other, the selected pixels are set to be invalid pixels. That is, portions where the reliability of the block matching is low are set to be invalid pixels. Portions such as the step portion or the outer edge of the workpiece are portions where the reliability is low, and are set to be invalid pixels. By generating an invalid pixel region from the invalid pixels, a fine profile such as the outer edge or the step portion of the workpiece can be detected. However, when the block matching is performed by making the determination value of the score small over the entire image, the number of pixels that contain information on a specific distance or the like decreases. For example, there is a problem in that the number of measurement points for detecting the position of the surface of the object decreases.

[0083] In the first control of the present embodiment, a first determination value related to the score and a second determination value larger than the first determination value are decided in advance. Then, a first distance image is generated using the first determination value, and a second distance image is generated using the second determination value. The first distance image corresponds to the position information of the first surface, and the second distance image corresponds to the position information of the second surface. The profile of the object is detected in advance based on the image of the first camera that is the profile detection sensor. Also, for pixels included in a region corresponding to the profile of the object, the distance information of the pixels included in the first distance image is adopted. For pixels included in a region other than the region corresponding to the profile, the distance information of the pixels included in the second distance image is adopted. In this way, a synthesized distance image in which the first distance image and the second distance image are synthesized is generated.

[0084] Here, the control in which the profile detection section 52 detects the profiles of the workpieces 61 and 62 will be described. Figure 9 An image in 2 dimensions captured by the first camera 31 in the robot device 3 of the present embodiment is shown. In the control in which the profiles of the workpieces 61 and 62 are detected, the image 96 in 2 dimensions is captured without using the projector 33. That is, a pattern such as a fringe pattern generated by the projector 33 is not generated on the surfaces of the workpieces 61 and 62. The image 96 includes an image of the workpiece 61 and an image of the workpiece 62. There is a slight gap between the workpiece 61 and the workpiece 62. The portion of the gap is dark. The profile detection section 52 of the processing section 51 detects the profiles of the workpieces 61 and 62 using the image 96. More specifically, the profile detection section 52 detects the positions of the pixels of the profiles in the image 96.

[0085] The outline detection section 52 of the present embodiment first processes each pixel of the image 96 using a Sobel filter. For the image processed by the Sobel filter, 2 pixels adjacent to each other are selected. Then, the luminance gradient between the 2 pixels is calculated. That is, the contrast for the 2 pixels is calculated. Then, in a case where the contrast exceeds a threshold value of the contrast decided in advance, it is determined that the pixel is an outline of an object.

[0086] Figure 10 A flowchart showing a method of setting a threshold value of a contrast for detecting an outline of an object. The operator can set the threshold value of the contrast by changing the threshold value of the contrast and observing an image of an outline actually generated.

[0087] In step 111, the operator sets the threshold value of the contrast to an arbitrary value. The threshold value of the contrast is set, for example, according to an empirical rule. Next, in step 112, the operator captures a 2-dimensional image using the first camera 31 by operating the teaching pendant 49.

[0088] In step 113, the outline detection section 52 performs processing of the Sobel filter on the image. The outline detection section 52 detects a pixel that becomes an outline based on the threshold value of the contrast. The outline detection section 52 generates, for example, an image in which the pixel that becomes the outline is given a color other than black. In step 114, the display section 49b of the teaching pendant 49 displays the image of the outline superimposed on the image 96. In step 115, the operator observes the image of the outline and determines whether or not the desired outline is included. In step 115, in a case where the desired outline is not obtained, the process returns to step 111.

[0089] In step 111, the operator sets a new threshold value of the contrast. For example, in a case where the threshold value of the contrast is large, if the luminance gradient in pixels adjacent to each other is not large, it is not detected as an outline. In a case where a clear outline is not displayed in the image of the outline, the operator can reduce the threshold value of the contrast. For example, in a case where the outline is not displayed in such a manner that the gap between the workpieces 61, 62 becomes clear, the operator can reduce the threshold value of the contrast. On the contrary, in a case where the outline is too much, the operator can increase the threshold value of the contrast. Then, the process of steps 112 to 115 is repeated.

[0090] In step 115, in a case where the desired outline is obtained, the operator can adopt the threshold value of the contrast at that time. In this way, the operator can set the threshold value of the contrast.

[0091] In addition, the contour detection in the 2-dimensional image is not limited to this method, and can be detected by any method. For example, the contour can be detected by binarizing the image obtained by the 2-dimensional camera.

[0092] Figure 11 A flowchart showing the first control of the present embodiment. Referring to Figure 2 and Figure 11 , the photographing control section 57 of the processing section 51 photographs a first image with the first camera 31 in step 131. In addition, the photographing control section 57 photographs a second image with the second camera 32.

[0093] Next, the contour detection section 52 detects the contour of the object using the first image photographed by the first camera 31 in step 132. In the present embodiment, the first image is used when detecting the contour of the object, but is not limited to this method, and the contour can be detected using the second image.

[0094] In addition, the operator sets the determination value of the first score and the determination value of the second score and stores them in the storage section 42. The determination value of the second score is set to be greater than the determination value of the first score.

[0095] In step 133, the processing section 51 generates a first distance image using the determination value of the first score decided in advance, the first image, and the second image. More specifically, the block search section 53 sets a selected block in the first image. The block search section 53 detects a specific block that corresponds most to the selected block in the second image. Here, the block search section 53 sets information that the selected pixel of the selected block is an invalid pixel as the distance information of the selected pixel in a case where the score is the determination value of the first score or more. The calculation section 54 calculates the distance information of the pixel corresponding to the measurement point on the work surface for the pixels other than the invalid pixel. The generation section 55 generates the first distance image by merging the distance information of the plurality of pixels. In step 134, the processing section 51 generates a second distance image using the determination value of the second score, the first image, and the second image similarly to step 133.

[0096] In step 135, the synthesis section 56 sets a region corresponding to the contour based on the contour of the workpieces 61, 62 detected by the contour detection section 52. More specifically, the synthesis section 56 acquires the pixels that become the contour, and sets the region corresponding to the contour around the pixels that become the contour. Then, the synthesis section 56 determines the pixels included in the region corresponding to the contour and the pixels included in the region other than the region corresponding to the contour of the workpieces 61, 62.

[0097] Figure 12 is an enlarged view of the image showing the pixels corresponding to the contour. Figure 12The illustrated image is, for example, an enlarged view of an image captured by the first camera 31. The image is composed of a plurality of pixels 87. The profile detection section 52 detects the positions of the pixels 87a that become the profile. In the example here, the pixels 87a are arranged in a straight line. A line 89 that passes through the center point 87aa of the shape of the pixels 87a corresponds to the profile line of the object. The synthesis section 56 sets a region of width d as the region corresponding to the profile with respect to the line 89. The width d can be set in advance by the operator. For example, the width d can be set to the length of several pixels.

[0098] The synthesis section 56 determines the pixels 87b included in the region corresponding to the profile and the pixels 87c included in the region other than the region corresponding to the profile. At this time, it can be determined that the pixel 87d that is arranged partly within the region corresponding to the profile is arranged within the region corresponding to the profile. Alternatively, it can be determined that the pixel 87d that is arranged partly within the region corresponding to the profile is arranged in the region other than the region corresponding to the profile.

[0099] In Figure 12 In the illustrated example, the pixels 87a that become the profile are arranged in a straight line along the direction in which the plurality of pixels 87 are arranged, but are not limited to this manner. The pixels that become the profile are sometimes arranged in a direction inclined with respect to the direction in which the plurality of pixels are arranged. Alternatively, there are cases in which the pixels that become the profile are arranged in a curved line. In such cases, the region corresponding to the profile can also be set based on the pixels that become the profile by an arbitrary method. For example, a straight line that connects the center points of the pixels that become the profile adjacent to each other is calculated. Then, a region of width d is set in a direction perpendicular with respect to the straight line that connects the center points. By performing this control with respect to each of the pixels that become the profile, the region corresponding to the profile can be set. Alternatively, a region of width d is set in a predetermined direction from the center point of each of the pixels that become the profile. Furthermore, a region of width d is set in a direction perpendicular to the predetermined direction from the center point. By performing this control with respect to each of the pixels that become the profile, the region corresponding to the profile can be set.

[0100] Referring to Figure 11 In step 136, the synthesis section 56 generates a synthesized distance image based on the first distance image, the second distance image, and the region corresponding to the profile.

[0101] Figure 13 A flowchart that represents the control of the synthesis section to generate a synthesized distance image. In step 141, the synthesis section 56 selects, for example, one pixel included in the first image captured by the first camera 31. In step 142, the synthesis section 56 determines whether the one pixel is arranged inside the region corresponding to the profile. In step 142, in the case where the one pixel is arranged inside the region corresponding to the profile, the control is transferred to step 143.

[0102] In step 143, the synthesizing section 56 adopts the distance information of the pixel included in the first distance image for the pixel included in the region corresponding to the contour. That is, the synthesizing section 56 detects the pixel corresponding to 1 pixel in the first distance image. The synthesizing section 56 sets the distance information of 1 pixel as the distance information of the pixel of the first distance image. When the distance information of the pixel of the first distance image is the information of the invalid pixel, 1 pixel is set as the invalid pixel.

[0103] In step 142, when 1 pixel is disposed outside the region corresponding to the contour, the control is shifted to step 144. In step 144, the synthesizing section 56 adopts the distance information of the pixel in the second distance image for the pixel disposed in the region outside the region corresponding to the contour. When the distance information of the pixel of the second distance image is the information of the invalid pixel, 1 pixel is set as the invalid pixel.

[0104] Next, in step 145, the synthesizing section 56 determines whether the distance information is set for all pixels. In step 145, when the distance information is not set for at least one pixel, the control is shifted to step 141. Then, in step 141, another pixel is selected, and the control of steps 142 to 145 is repeated. In this way, the distance information is set for all pixels.

[0105] In step 145, when the distance information is set for all pixels, the control is shifted to step 146. In step 146, the synthesizing section 56 generates a synthesized distance image in which the distance information of each pixel is combined.

[0106] Figure 14 An example of the synthesized distance image generated in the first control is shown. In the distance image 92, the distance information of the pixel is represented in the density of color. Further, the invalid pixel is not colored. In the gap existing between the workpieces 61 and 62, an invalid pixel region 97 composed of the invalid pixel is generated. In addition, in the region other than the edge portion of the upper surface of each workpiece 61, 62, the invalid pixel is small, and the distance information of each pixel includes the information of the specific distance or position. That is, many measurement points are detected in the region other than the edge portion of the upper surface of each workpiece 61, 62.

[0107] Figure 15An example of a distance image of the comparative example. In the distance image 93 of the comparative example, the decision value of the score when block matching is performed is set large for all pixels. For example, the distance image 93 is a second distance image generated using the decision value of the second score. In this example, distance information including specific distance or position information is set for many pixels. That is, many measurement points are detected. However, distance information including specific distance or position information is also set in the gap present between the workpiece 61 and the workpiece 62. Therefore, in the distance image 93, the gap present between the workpiece 61 and the workpiece 62 disappears. In this way, the reliability of the distance information of the pixels becomes low in the portions of the outer edges of the workpieces 61, 62 and the like.

[0108] Reference Signs Figure 14 In contrast, in the first control of the present embodiment, the decision value of the score is set small in the region corresponding to the contour, and therefore an invalid pixel region 97 composed of invalid pixels is found in the region corresponding to the outer edges of the workpieces 61, 62. The gap between the first workpiece 61 and the second workpiece 62 is clearly shown. In this way, the first workpiece 61 and the second workpiece 62 can be clearly separated.

[0109] The action instruction section 58 of the processing section 51 can detect that there are two workpieces 61, 62. In addition, many measurement points are detected in the regions of the upper surfaces of the respective workpieces 61, 62 away from the outer edges. Therefore, the action instruction section 58 can drive the robot 1 based on the distance information of the many measurement points. For example, the action instruction section 58 can drive the robot 1 so that the robot hand 5 is disposed at the center of gravity position of the upper surface of the workpiece 61.

[0110] In this way, in the first control, it is judged whether each pixel is included in the region corresponding to the contour, and the distance information of the pixel in the first distance image or the distance information of the pixel in the second distance image is set for each of the respective pixels. Therefore, in the region where the contour is present, a fine shape clear distance image can be obtained by the invalid pixels. On the other hand, in the region other than the region where the contour is present, many measurement points can be obtained because the distance information in the distance image using a large decision value of the score is adopted. A three-dimensional measurement device in which the precision of the position information of the surface of an object is improved can be provided.

[0111] The method of setting the decision value of the first score in the present embodiment will be described in detail in the Figure 16 The decision value of the first score and the decision value of the second score in the present embodiment can be set by an operator while observing the actually generated distance image. For example, the operator can set the decision value of the score by generating a distance image of a workpiece on which an actual work is performed.

[0112] In step 121, the operator sets a determination value of the first score. For example, the operator sets an arbitrary determination value of the first score based on an empirical rule. In step 122, the operator captures images with the first camera 31 and the second camera 32 by operating the teaching panel 49.

[0113] In step 123, the processing section 51 generates a distance image based on the determination value of the first score, the first image, and the second image. In step 124, the display section 49b of the teaching panel 49 displays the distance image. For example, the distance image shown in the drawing is displayed. Figure 15

[0114] In step 125, the operator determines whether a desired distance image is obtained. Here, in order to set the determination value of the first score, it is determined whether the invalid pixel region 97 is formed between the workpieces 61, 62. If the determination value of the score is made too large, the invalid pixel region 97 becomes less or disappears. On the other hand, if the determination value of the score is made too small, many invalid pixel regions 97 are generated in a region other than the outer edge region of the workpieces 61, 62. As a result, the number of pixels including specific distance information corresponding to the surfaces of the workpieces 61, 62 becomes less. That is, the number of measurement points set on the upper surfaces of the workpieces 61, 62 becomes less.

[0115] In step 125, in a case where a desired distance image is not obtained, the process returns to step 121. Then, the operator sets another determination value of the first score. The operator observes the distance image to change the determination value of the first score in a manner that the outline of the outer edge and the step portion of the workpiece or the like generates the invalid pixel region 97. For example, the operator sets the determination value of the first score to be small in a case where the invalid pixel region 97 is not generated between the workpieces 61, 62. In addition, the operator sets the determination value of the first score to be large in a case where many invalid pixel regions 97 are formed. Then, the operator repeatedly performs the process of steps 122 to 125. In step 125, in a case where a desired distance image is generated, the determination value of the first score at that time can be adopted.

[0116] The determination value of the second score can be set by the same method as Figure 16 the determination value of the first score. In step 125, the operator determines whether many pixels for which specific distance information is set exist in a region corresponding to the upper surfaces of the workpieces 61, 62 in the distance image. It is all right that no invalid pixel region is found between the workpieces 61, 62 in the distance image generated by the determination value of the second score. The determination value of the second score is set to be larger than the determination value of the first score. The processing section 51 can generate a composite distance image using the first score and the second score thus set.

[0117] ​Next, the second control of the present embodiment will be described. In the second control, a plurality of distance images are captured by changing the block size, instead of changing the determination value of the score to capture a plurality of distance images. Then, a plurality of distance images formed in a manner that the block sizes are different from each other are synthesized to generate a synthesized distance image.

[0118] Referring to Figure 4 and Figure 5 When the block size is increased so that the selected block 81 and the search block 83 at the time of performing the block matching contain many pixels, the accuracy of the calculation of the disparity is reduced because the differences of many pixels from each other are cumulated. In addition, in the case where the block size is reduced, in a region where the value of the pixel changes slowly, it is sometimes difficult to perform the block matching. For example, there is a case where many search blocks having the same score are detected.

[0119] In the second control, the operator decides a first selected block and a second selected block containing more pixels than the first selected block in advance. That is, the operator decides the selected blocks whose block sizes are different from each other. The block size of the first selected block is smaller than the block size of the second selected block. For example, as the first selected block, a block of 3 rows and 3 columns can be adopted. In addition, as the second selected block, a block of 9 rows and 9 columns can be adopted.

[0120] Figure 17 A flowchart showing the second control. Referring to Figure 2 and Figure 17 , the step 131 and the step 132 are the same as the control in the first control (refer to Figure 11 ). In the step 153, the block search section 53, the calculation section 54, and the generation section 55 generate the first distance image using the first image, the second image, and the first selected block. The block search section 53 performs the control of the block matching using the first selected block and the search block having the same block size as the first selected block.

[0121] In the step 154, the block search section 53, the calculation section 54, and the generation section 55 generate the second distance image using the first image, the second image, and the second selected block. The block search section 53 performs the control of the block matching using the second selected block and the search block having the same block size as the second selected block.

[0122] Next, in the step 135, the synthesis section 56 acquires the pixels that become the outlines of the workpieces 61, 62 detected by the outline detection section 52, as in the first control. The synthesis section 56 sets the region corresponding to the outline based on the pixels that become the outline. The synthesis section 56 determines the pixels contained in the region corresponding to the outline and the pixels contained in the region other than the region corresponding to the outline (refer to Figure 11 and Figure 12 ).

[0123] In step 156, the synthesizing section 56 generates a synthesized distance image. The synthesizing section 56 sets distance information of the pixel included in the first distance image generated by the first selected block for the pixel included in the region corresponding to the contour. On the other hand, the distance information of the pixel included in the second distance image generated by the second selected block is set for the pixel included in the region other than the region corresponding to the contour. Thus, the distance information of the pixel included in the first distance image or the distance information of the pixel included in the second distance image is set for each pixel, whereby the synthesized distance image can be generated.

[0124] Figure 18 A flowchart showing a method of setting the block size of the first selected block. The operator can set the block size of the first selected block while observing the actually generated distance image. In step 161, the operator sets the first selected block. The operator sets the first selected block with an arbitrary block size based on experience or the like. In step 162, the operator captures an image using the first camera 31 and the second camera 32 by operating the teaching panel 49. In step 163, the processing section 51 generates a distance image using the first selected block. Further, the display section 49b of the teaching panel 49 displays the distance image.

[0125] In step 165, the operator determines whether the desired distance image is obtained. In the first distance image generated by the first selected block, it is preferable that the contour of the outer edge or the step portion or the like of the workpiece is clearly displayed. For example, if the block size is increased, the contour of the workpieces 61, 62 becomes unclear. In addition, there is a case where the gap between the workpieces 61, 62 disappears. On the other hand, if the block size is made too small, the contour other than the desired contour is sometimes detected.

[0126] In step 165, in a case where the desired distance image is not obtained, the process proceeds to step 161. In step 161, the operator sets the first selected block whose block size is changed. For example, in a case where the gap between the workpieces 61, 62 is not clear, the first selected block whose block size is reduced can be set. Then, the operator repeatedly performs steps 162 to 165. In step 165, in a case where the desired distance image is obtained, the first selected block whose block size at that time can be adopted.

[0127] As to the block size of the second selected block, it can also be set by the same method. For example, the operator can set the block size of the second selected block in such a manner that the contour of the gap between the workpieces 61, 62 becomes unclear in steps 161, 165. Figure 18

[0128] Figure 19 ​The distance image captured by the second control is indicated. In the area where the outer edges of the workpieces 61, 62 are present, the distance information of the pixels contained in the first distance image generated by the first selected block having a small block size is adopted. For example, in the area where the gap between the workpiece 61 and the workpiece 62 is present, the distance information of the pixels of the first distance image is adopted. Thus, in the distance image 94, the outer edge of the workpiece 61 and the outer edge of the workpiece 62 are clearly shown. In particular, the gap between the workpiece 61 and the workpiece 62 is clearly indicated by the difference in the concentration of the pixels.

[0129] On the other hand, in the area outside the outlines of the workpieces 61, 62, the distance information of the pixels of the second distance image generated in the second block size having a large block size is adopted. In the central portion of the upper surfaces of the workpieces 61, 62, the distance information of the pixels of the second distance image is adopted. Thus, a large number of measurement points having accurate distance information are set. In addition, even in the area where the gradient of the pixel value such as the brightness or the concentration in the 2-dimensional image is small, the accurate position information of the surfaces of the workpieces 61, 62 can be detected.

[0130] The other control is the same as the first control, and thus the description thereof is not repeated here.

[0131] The vision sensor 30 of the present embodiment is fixed to the support member 66, but is not limited to this manner. The vision sensor can be arranged in a manner that can capture the workpiece. For example, the vision sensor can also be fixed to the wrist portion in a manner that moves integrally with the wrist portion of the robot. In this case, the position of the camera coordinate system in the robot can be calculated in advance. Furthermore, the position information of the object surface generated in the camera coordinate system can be converted into the world coordinate system based on the position and the posture of the robot.

[0132] The vision sensor 30 of the present embodiment contains two 2-dimensional cameras, but is not limited to this manner. The vision sensor can contain three or more 2-dimensional cameras. By the 3-dimensional sensor containing three or more cameras, even if a part of the image is not clear due to flare or the like, the position information of the object surface can be generated based on the image captured by the other cameras. In addition, the vision sensor of the present embodiment is provided with a projector, but is not limited to this manner. The vision sensor can not be provided with a projector.

[0133] In the present embodiment, the control device of the robot functions as a processing portion that processes the image of the vision sensor, but is not limited to this manner. The processing portion can also be constituted by an arithmetic processing device (computer) different from the control device of the robot. For example, a tablet terminal that functions as the processing portion can be connected to the control device of the robot.

[0134] The three-dimensional measurement apparatus of the present embodiment is configured in the robot apparatus, but is not limited to this. The three-dimensional measurement apparatus can be configured in any apparatus.

[0135] In each of the above controls, the order of the steps can be appropriately changed within a range where the functions and the effects are not changed.

[0136] The above embodiments can be appropriately combined. In the above drawings, the same reference numerals are assigned to the same or equivalent portions. The above embodiments are illustrative and do not limit the invention. In addition, in the embodiments, changes of the above embodiments are included in the scope of the protection.

[0137] Explanation of Reference Numerals

[0138] 2 Control apparatus

[0139] 23 Position detector

[0140] 30 Vision sensor

[0141] 31 First camera

[0142] 32 Second camera

[0143] 51 Processing section

[0144] 52 Contour detection section

[0145] 53 Block search section

[0146] 54 Calculation section

[0147] 55 Generation section

[0148] 56 Synthesis section

[0149] 61, 62, 63 Workpiece

[0150] 77 First image

[0151] 78 Second image

[0152] 81 Selected block

[0153] 82 Specific block

[0154] 85, 86, 87 Pixel

[0155] 85a Selected pixel

[0156] 87a, 87b, 87c Pixel

[0157] 91, 92, 93, 94 Distance image

[0158] 96 Image

[0159] 97 Invalid pixel region.

Claims

1. A three-dimensional measuring device, characterized by, Possessing: a visual sensor including a first camera that captures a first image and a second camera that captures a second image; a profile detection sensor that detects a profile of an object; a processing section that detects position information of an object surface based on the first image and the second image, the processing section includes: a profile detection section that detects the profile of the object based on an output of the profile detection sensor; a block search section that sets a selected block composed of a plurality of pixels including a selected pixel selected in the first image, and searches for a specific block corresponding to the selected block in the second image; a calculation section that calculates distance information of the pixel based on a parallax between a position of the selected pixel in the selected block and a position of a specific pixel corresponding to the selected pixel in the specific block; a generation section that generates position information of the object surface including distance information of the plurality of pixels; and a synthesis section that synthesizes position information of the object surface generated under mutually different conditions, the block search section calculates a cumulative value obtained by accumulating a size of a difference between a value of a pixel included in the selected block and a value of a pixel included in the specific block or a square value of the difference, and in a case where the cumulative value is smaller than a predetermined determination value, determines that the pixel included in the specific block corresponds to the pixel included in the selected block, and in a case where the cumulative value exceeds the predetermined determination value or in a case where the specific block corresponding to the selected block is not detected, sets information that is an invalid pixel as distance information of the selected pixel, a first determination value related to the cumulative value and a second determination value related to the cumulative value that is larger than the first determination value are predetermined, the block search section, the calculation section, and the generation section generate position information of a first surface using the first image, the second image, and the first determination value, and generate position information of a second surface using the first image, the second image, and the second determination value, the synthesis section sets a region corresponding to the profile based on the profile detected by the profile detection section, adopts distance information of a pixel included in the position information of the first surface for a pixel included in the region corresponding to the profile, adopts distance information of a pixel included in the position information of the second surface for a pixel included in a region other than the region corresponding to the profile, and thereby generates position information of a surface obtained by synthesizing the position information of the first surface and the position information of the second surface.

2. The three-dimensional measurement apparatus according to claim 1, wherein the profile detection sensor is the first camera or the second camera. ​

Images