Measuring device, measuring method, and program

Abstract

To provide a measuring device that can improve the accuracy of three-dimensional measurement. [Solution] The measuring device of the present disclosure is a measuring device that performs three-dimensional measurement and comprises a pattern projection unit that projects pattern light onto an object to be measured, a camera having a first camera and a second camera that photographs the object, and an image processing unit that processes data acquired from the camera, wherein the image processing unit derives a first evaluation result from a first patterned image acquired by the first camera and a second patterned image acquired by the second camera while the pattern light is projected, and derives a proposed solution to improve the accuracy of the three-dimensional measurement based on the first evaluation result.

Description

Technical Field

[0005]

[0001] The present disclosure relates to a measuring device, a measuring method, and a program.

Background Art

[0002] In recent years, in production lines of industrial products and the like, instead of manual assembly work, assembly work is performed by an assembly production device including a robot device or the like.

[0003] In such a production system, in order to perform measurement and inspection of work required for assembly work, a camera and an image processing device may be used. For example, when information in the depth direction is required for measurement or inspection, a stereo camera is configured by two or more cameras, and a method of performing three-dimensional measurement of an object based on the principle of triangulation is used. In this type of three-dimensional measurement, the difference (parallax) in the position of an object in a plurality of images captured by a plurality of cameras for each camera is calculated, and three-dimensional information is obtained by converting this parallax into a depth amount. Generally, obtaining the parallax is called stereo matching processing.

[0004] In stereo matching processing, if the texture is weak in an image of the work, the accuracy of the calculated parallax may decrease. Here, the texture refers to a pattern or a motif that appears due to light and shade on an image. Therefore, a method is known in which a pattern is irradiated onto the work using a pattern projection unit that irradiates light having a texture (pattern light) and stereo matching processing is performed. In this case, if white spots or black spots occur in an image of the work, the accuracy of the parallax calculated in stereo matching processing may decrease.

[0005] For example, Patent Document 1 discloses an information processing system and an information processing method capable of suppressing a decrease in the recognition accuracy of a subject.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2016-75658 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] This disclosure aims to provide a measuring device that can improve the accuracy of three-dimensional measurement. [Means for solving the problem]

[0008] The measuring device of this disclosure comprises a pattern projection unit that projects patterned light onto an object to be measured, a camera having a first camera and a second camera that photographs the object, and an image processing unit that processes data acquired from the camera, and is a measuring device that performs three-dimensional measurement, wherein the image processing unit derives a first evaluation result from a first patterned image acquired by the first camera and a second patterned image acquired by the second camera while the patterned light is projected, and derives a proposed solution to improve the accuracy of the three-dimensional measurement based on the first evaluation result. [Effects of the Invention]

[0009] The technology disclosed herein provides a measuring device that can improve the accuracy of three-dimensional measurement. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows the configuration of a three-dimensional measuring device according to the first embodiment. [Figure 2] This figure shows the configuration of a stereo camera according to the first embodiment. [Figure 3] This is a functional block diagram of the image processing apparatus according to the first embodiment. [Figure 4] This figure shows the parameters of the three-dimensional measuring device according to the first embodiment. [Figure 5] This figure shows a stereo matching method according to the first embodiment. [Figure 6] It is a schematic diagram of the device configuration according to the first embodiment. [Figure 7] It is a flowchart showing a method for estimating factors causing accuracy degradation according to the first embodiment. [Figure 8] It is a flowchart showing a method for estimating factors causing accuracy degradation according to the first embodiment. [Figure 9] It is a diagram showing an example of the display screen of the three-dimensional measurement application according to the first embodiment. [Figure 10] It is a diagram showing an example of the display screen of the three-dimensional measurement application according to the first embodiment. [Figure 11] It is a diagram showing an example of a captured image in which white discharge has occurred according to the first embodiment. [Figure 12] It is a diagram showing epipolar displacement according to the first embodiment. [Figure 13] It is a diagram showing an example of a captured image according to the second embodiment. <_{ [Figure 14] It is a diagram showing an example of the display screen of the three-dimensional measurement application according to the second embodiment. [Figure 15] It is a flowchart showing a method for estimating factors causing accuracy degradation according to the third embodiment. [Figure 16] It is a diagram showing an example of the display screen of the three-dimensional measurement application according to the third embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments for implementing the present disclosure will be described with reference to the accompanying drawings. Note that the embodiments shown below are merely examples, and for example, those skilled in the art can appropriately change details of the configuration without departing from the spirit of the present disclosure. Also, the numerical values taken up in this embodiment are reference numerical values for explaining this embodiment and do not limit the present disclosure.

[0012] (First Embodiment) FIG. 1 shows the configuration of a three-dimensional measurement system using a stereo camera and a pattern projection unit in the present embodiment. The three-dimensional measurement system is used, for example, in an inspection process for inspecting an article manufactured in a manufacturing process on a production line.

[0013] The three-dimensional measurement system in the present embodiment has a configuration in which, for example, a stereo camera 101 and a pattern projection unit 102 are connected to an image processing device 104 having the function of an image processing unit. In the following description of the present embodiment, the image processing device 104 and the stereo camera 101 will be described as separate devices, but the image processing device 104 may be incorporated in the stereo camera 101 like a so-called smart camera. In such a configuration, the wiring between the stereo camera 101 and the image processing device 104 becomes unnecessary, and the installation man-hours of the system can be significantly reduced.

[0014] The connection cable between the stereo camera 101 and the image processing device 104 constitutes a communication interface between the two, and includes a power line, a communication line for transmitting and receiving imaging data, an IO line used for communication control, etc. The communication interface can be configured based on a standard such as USB (Universal Serial Bus), for example.

[0015] The connection cable between the pattern projection unit 102 and the image processing device 104 constitutes a communication interface between the two, and includes a power line, a communication line for transmitting and receiving dimming values, an IO line used for communication control, etc. The pattern projection unit 102 incorporates an LED or a glass chart with a pattern drawn thereon, and has a function of turning on / off the pattern light 103 in response to an instruction from the image processing device 104. Also, the pattern light 103 can be dimming-controlled in response to an instruction from the image processing device 104. For example, as the dimming control, there is a method of transmitting a PWM (Pulse Width Modulation) signal from the image processing device 104 to the pattern projection unit 102.

[0016] The image processing device 104 and the display 105, which functions as a display unit, are connected by a video cable, allowing the image processing device 104 to output video and display it on the display 105. This communication interface can be configured based on a standard such as HDMI®.

[0017] As shown in Figure 2, the stereo camera 101 includes a monocular camera (first camera) 201 and a monocular camera (second camera) 202, whose respective imaging optical axes are spaced apart by an appropriate baseline length. Monocular cameras 201 and 202 each include an image sensor 203 and an image sensor 204, a lens 205 and a lens 206, and are capable of capturing a desired image. Image sensors 203 and 204 are, for example, CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) image sensors.

[0018] Image data captured by the stereo camera 101 can be transmitted to the image processing device 104 via the communication interface described above. Furthermore, the imaging parameters, which are setting information during imaging, can be controlled according to setting commands received from the image processing device 104 or other devices via the communication interface. These imaging parameters include, for example, exposure time, gain, and image size.

[0019] The image processing device 104 can be configured with hardware such as a processing unit having a CPU and a GPU, a memory unit having ROM and RAM, and an interface (I / F) unit for communicating with the outside. Note that, as will be described later, the image processing device 104 of this embodiment also includes an imaging control function for the stereo camera 101, and conceptually, it can be considered a control device that performs imaging control via image processing.

[0020] Furthermore, the image processing device 104 may be equipped with a user interface device (UI device). The user interface device may be a GUI device consisting of, for example, a display 105, a keyboard, a pointing device (mouse, joystick, jog dial, etc.). The user interface device can be used to notify the user of captured images and three-dimensional measurement results, or to set areas for evaluating factors that reduce accuracy, as described later.

[0021] Figure 3 is a functional block diagram of the image processing device 104. The image processing device 104 has the functions of a camera control unit 301, a pattern projection unit control unit 302, a three-dimensional measurement unit 303, a measurement accuracy degradation factor estimation unit 304, and a user interface unit 305. For example, each of these functional blocks may be realized by the above-mentioned calculation unit reading and executing a program stored in the memory unit.

[0022] The following describes the general functions of each functional block (301-305) of the image processing device 104. The camera control unit 301 controls the imaging operations of the monocular camera 201 and monocular camera 202 within the stereo camera 101. The imaging operation is performed, for example, as follows.

[0023] First, power is supplied to monocular cameras 201 and 202, and an initialization instruction is sent from the camera control unit 301. Once the initialization of monocular cameras 201 and 202 is complete, instructions to change the imaging parameters are sent to monocular cameras 201 and 202. Once the adjustment of the imaging parameters is complete, the camera control unit 301 sends a shooting trigger signal to monocular cameras 201 and 202, causing them to output the captured image data. The camera control unit saves the received image data to a memory unit such as RAM that can be accessed by other functional blocks.

[0024] As described above, the pattern projection unit control unit 302 has the function of controlling the ON / OFF status and dimming of the pattern projection unit 102. This pattern projection unit control is performed, for example, by transmitting a PWM (Pulse Width Modulation) signal to the pattern projection unit 102 via the IO line.

[0025] The three-dimensional measurement unit 303 performs three-dimensional measurement using images captured by the monocular cameras 201 and 202 of the stereo camera 101. The three-dimensional measurement unit 303 can calculate distance using the principle of triangulation as shown in Figure 4, using the parallax d obtained from the stereo matching process described later, and the intrinsic and extrinsic parameters obtained from the stereo camera calibration. Here, the intrinsic parameters refer to optical characteristics such as the focal length f and distortion characteristics of the lens, and the extrinsic parameters refer to relative position and orientation, including the baseline length B, which is the distance between the principal lens points of the two monocular cameras 201 and 202 in the stereo camera.

[0026] The internal and external parameters are generally calculated using a calibration method that involves photographing a calibration chart with a known shape, measuring the positions of multiple feature points within the calibration chart, and then calculating them, for example, by optimization calculation, such that the epipolar constraint condition is satisfied for these feature points. Here, the epipolar constraint is a constraint condition that corresponding points on the images captured by the two cameras of a stereo camera lie in the same plane (epipolar plane). The epipolar plane is the plane passing through the principal lens points and measurement points of the two cameras. The internal and external parameters calculated in advance for monocular cameras 201 and 202 are stored, for example, in ROM within the image processing device 104.

[0027] Next, we will explain the stereo matching process for determining the parallax d. For example, as shown in Figure 5, an image of the workpiece 504, which is the object to be measured, captured by the monocular camera 201 is set as the reference image 401, and an image captured by the monocular camera 202 is set as the corresponding image 402. A reference block 501 is set in the reference image 401, and the corresponding block, that is, the region in which the same specific part of the subject is captured, is searched for within the corresponding image 402 to determine the parallax d.

[0028] In this case, since the corresponding block 502 in the corresponding image 402 is known to lie on the epipolar line, the search is performed along the epipolar line. The epipolar line is the intersection line of the epipolar plane and the corresponding image 402. The search range 503 can be set as a parameter for three-dimensional measurement and is stored, for example, in the ROM of the image processing device 104. The distance between the reference block 501 and the corresponding block 502 on the image is the disparity d.

[0029] Block matching methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), and SGM (Semi Global Matching) are known for performing such processing. These known matching methods can also be used in this embodiment.

[0030] Here, we will explain three-dimensional measurement when stereo camera images are parallelized using internal and external parameters. Stereo parallelization is the process of correcting the lens distortion of the cameras and converting the images into those taken with two cameras positioned parallel to each other. After parallelization, the measurement result z in the Z direction (the depth direction of the image) can be expressed as z = Bf / d using the principle of triangulation. In this way, the three-dimensional measurement unit 303 has the function of performing three-dimensional measurement using the captured image.

[0031] The measurement accuracy degradation factor estimation unit 304 has the function of estimating the factors that caused the reduction in three-dimensional measurement accuracy due to image abnormalities such as overexposure in the captured image. Details of the method for estimating the factors that caused the reduction in measurement accuracy will be described later.

[0032] The user interface unit 305 has the function of controlling a user interface device that consists of, for example, a display 105, a keyboard, and a pointing device (mouse, joystick, jog dial, etc.).

[0033] Next, Figure 6 shows the measurement environment when photographing the workpiece 504 using the stereo camera 101 and the pattern projection unit 102 for three-dimensional measurement. The workpiece 504 is placed within the common field of view of the monocular cameras 201 and 202 in the stereo camera 101. The pattern projection unit 102 is positioned so that patterned light is projected onto the workpiece 504. Fluorescent lights 601 are installed on the ceiling.

[0034] Depending on the position and material of the workpiece 504, reflection of light from the pattern projection unit 102 or the fluorescent lamp 601 may cause overexposure in the captured image, reducing the accuracy of three-dimensional measurement. If image abnormalities occur due to ambient light such as the fluorescent lamp 601, adjusting the brightness of the pattern projection unit 102 alone may not be sufficient to resolve the image abnormalities. Therefore, if it is possible to distinguish and estimate whether the cause of the image abnormality is the pattern light or the ambient light, the burden on the user who wants to resolve the image abnormalities can be reduced.

[0035] The method for estimating the factors that reduce the accuracy of three-dimensional measurement will be explained with reference to Figures 7 to 9. Figure 7 is a flowchart showing the sequence from capturing an image with the stereo camera 101 to displaying the results of the estimation of the factors that reduce the accuracy of three-dimensional measurement. Figure 8 is a detailed flowchart for estimating the factors that reduce the accuracy of three-dimensional measurement. Figure 9 is an example of the screen of the three-dimensional measurement application 901 displayed on the display 105.

[0036] Step S101 is the process in which the image processing device 104 acquires images captured by the stereo camera 101 and displays them in the reference image display window 904 and the corresponding image display window 905. After the user presses the capture button 902 in the three-dimensional measurement application 901, images are acquired from the stereo camera 101. The image captured by the monocular camera 201 is displayed as the reference image in the reference image display window 904, and the image captured by the monocular camera 202 is displayed as the corresponding image in the corresponding image display window 905.

[0037] Shooting parameters such as the intensity of the pattern projection unit 102, the exposure time of the stereo camera 101, and the gain can be set in the shooting conditions window 903. The image processing device 104 sets the stereo camera 101 and the pattern projection unit 102 before performing the shooting process. If the intensity of the pattern projection unit 102 is set to 0, the image processing device 104 sends a trigger signal to the stereo camera 101 and acquires an image with the pattern projection turned off. On the other hand, if the intensity is set to a value other than 0, the image processing device 104 sends a power-on command to the pattern projection unit 102, and then sends a trigger signal to the stereo camera 101 to acquire an image.

[0038] After the shooting is complete, a power-off command is sent to the pattern projection unit 102. This allows for the acquisition of images with the pattern projection active. Furthermore, since the pattern projection unit 102 is not constantly lit, its lifespan can be extended and its temperature rise can be suppressed.

[0039] Step S102 involves performing three-dimensional measurement on the image captured in step S101 using the stereo matching process described above, and displaying the results in the three-dimensional measurement results window 908. The three-dimensional measurement process starts after the user presses the three-dimensional measurement button 906 in the three-dimensional measurement application 901.

[0040] The three-dimensional measurement conditions, such as the stereo matching algorithm, block size, search range, and minimum disparity, can be set in the three-dimensional measurement conditions window 907, and the image processing device 104 sets these conditions before performing the three-dimensional measurement. The three-dimensional measurement results obtained are then displayed in the three-dimensional measurement results window 908. The three-dimensional measurement results window 908 may display a three-dimensional point cloud in three dimensions, or it may display a so-called distance image in two dimensions, where each pixel of the reference image contains the measured value in the Z direction.

[0041] In this embodiment, the shooting button 902 and the three-dimensional measurement button 906 are separate buttons, but they may be combined into a single button, allowing steps S101 and S102 to be performed simultaneously. This reduces the number of times the user has to press a button, thereby improving work efficiency.

[0042] Step S103 is the process in which the user selects the area in the three-dimensional evaluation results measured in step S102 in which they want to evaluate the factors causing a decrease in three-dimensional measurement accuracy. As shown in Figure 10, we will explain the case in which there is an abnormal area 1002 in the three-dimensional measurement results displayed in the three-dimensional measurement results window 908 that differs from the actual workpiece shape. In this embodiment, an abnormal area 1002 is defined as an area in which point cloud data does not exist or the three-dimensional measurement results differ significantly from the actual workpiece shape, indicating a decrease in three-dimensional measurement accuracy.

[0043] If there are areas where the three-dimensional measurement accuracy is reduced, the user needs to set an evaluation area 1003 in order to estimate the cause of the reduction. The user uses the mouse or other means on the three-dimensional measurement results window 908 to select an evaluation area 1005 that includes the abnormal area 1002.

[0044] Since the three-dimensional measurement results and the reference image are viewed from the same coordinate system, if the evaluation region 1005 is determined on the three-dimensional measurement results, the evaluation region 1003 on the reference image 401 is uniquely determined. On the other hand, since the corresponding image 402 is viewed from a different coordinate system, the evaluation region on the corresponding image is not uniquely determined. Therefore, the evaluation region 1004 is also set on the corresponding image display window 905. By selecting these regions and then pressing the evaluation region determination button 909, evaluation regions 1003, 1004, and 1005 can be set. The coordinate value data of the set evaluation regions is saved in the ROM or other location within the image processing device 104.

[0045] In this embodiment, the selection was made on the three-dimensional measurement results window 908, but it may also be made on the reference image display window 904. Furthermore, although the user manually selected evaluation regions 1003, 1004, and 1005, the evaluation regions 1003, 1004, and 1005 may be automatically set by performing image processing on the three-dimensional measurement results.

[0046] For example, one method for automatic setting is to model-fit the point cloud data to a CAD model representing the actual workpiece shape when the workpiece shape is known. Areas with large differences from the CAD model are identified as abnormal regions, and the vicinity of these abnormal regions is set as the evaluation region. Alternatively, for example, when measuring a uniform plane, the distance image is binarized using distance information, areas below a threshold are identified as abnormal regions, and the vicinity is set as the evaluation region. This eliminates the need for the user to select the evaluation region, thus improving work efficiency.

[0047] If the user wishes to evaluate the entire image, step S103 may be omitted. If omitted, the entire image will be automatically set as the evaluation area.

[0048] Step S104 is the process of estimating the factors that caused the decrease in three-dimensional measurement accuracy in evaluation areas 1003, 1004, and 1005, which were set in step S103. Figure 8 shows the flowchart for estimating the factors that caused the decrease in three-dimensional measurement accuracy. Details of this flowchart will be described later.

[0049] Step S105 is the process of displaying the factors estimated in step S104 and the proposed countermeasures in the estimation results window 911. Based on the proposed countermeasures displayed in this window, the user can easily take countermeasures.

[0050] The detailed flow for estimating the factors that led to the decrease in three-dimensional measurement accuracy, which is performed in step S104, will be explained using the flowchart in Figure 8.

[0051] Step S201 involves turning on the pattern projection unit 102 and acquiring images captured by the stereo camera 101. The shooting conditions are the same as those used in step S101. In this embodiment, the images captured in this step are referred to as patterned images. Among the patterned images, the reference image 401 is referred to as the first patterned image. Similarly, among the patterned images, the corresponding image 402 is referred to as the second patterned image. In this embodiment, new patterned images were captured for estimation purposes, but the images captured in step S101 for three-dimensional measurement may also be reused.

[0052] Step S202 is the process of turning off the pattern projection unit 102 and acquiring the image captured by the stereo camera 101. The shooting conditions, other than the intensity of the pattern projection unit 102, are the same as when the image was captured in step S101. In this embodiment, the image captured in this step is referred to as a patternless image. Of the patternless images, the reference image 401 is referred to as the first patternless image. Similarly, of the patternless images, the corresponding image 402 is referred to as the second patternless image.

[0053] Step S203 involves reading the data from evaluation regions 1003 and 1004, which were set in step S103, and extracting evaluation region images from the patterned and unpatterned images captured in steps S201 and S202. In subsequent steps, image processing is performed on the extracted evaluation region images to estimate the factors that caused the decrease in three-dimensional measurement accuracy.

[0054] In step S204, a first evaluation result is derived, indicating whether the area of ​​the overexposed region in the evaluation region of the patterned image is above a threshold. If the first evaluation result is above the threshold, the process proceeds to step S205; otherwise, the process proceeds to step S206. Here, overexposure refers to a state in the captured image where the brightness is saturated.

[0055] This section describes one example of how to determine the area of ​​the overexposed region. The brightness threshold is set to a value near the brightness of the saturated state. For example, in the case of an 8-bit depth captured image, the maximum brightness is 255. However, due to fixed pattern noise in the image sensor 203, the brightness is not necessarily 255 even when pixels are overexposed. Therefore, a predetermined value, such as the value of the largest neighbor (250, for example), is set as the brightness threshold. Regions (blobs) above the brightness threshold within the evaluation area are extracted using binarization, and the area can be calculated from the number of pixels in these blobs.

[0056] Furthermore, the area threshold is set based on the size of the evaluation area. For example, if the evaluation area is 100 x 100 pixels, the area (number of pixels) of the evaluation area will be 10,000. It is possible to set the area threshold based on the ratio to this area. For example, if it is 80%, the area threshold will be 8,000. It is possible to determine whether the blob area calculated earlier is greater than or equal to the area threshold.

[0057] The brightness threshold and area threshold for the overexposed areas set in these parameters should be adjustable on the 3D measurement application 901, as these parameters will vary depending on the use case. For example, in a use case where you want to measure fine shapes in 3D, it is desirable that the area where the 3D measurement accuracy is reduced is small. In this case, reducing the brightness threshold and area threshold allows for identification even if the overexposed area is small.

[0058] In step S205, a second evaluation result is derived, indicating whether the area of ​​the overexposed region in the evaluation region of the pattern-free image is above a threshold. The method for determining the area of ​​the overexposed region is the same as in step S204. Steps S204 and S205 allow us to determine whether the cause of the decreased three-dimensional measurement accuracy is overexposure, and whether the cause of the overexposure is due to the pattern projection unit or ambient light.

[0059] The method of troubleshooting will be explained using Figure 11. Figure 11(a) is an image diagram in which overexposure occurs in the patterned image 1101 in step S204, but not in the patternless image 1102 in step S205. The occurrence of overexposure strongly suggests that the stereo camera 101 is capturing specularly reflected light. Since overexposure disappears when the pattern light 103 is turned off, it can be determined that overexposure was caused by specular reflection from the pattern light projection unit 102. In such cases, countermeasures can be taken by changing the arrangement of the pattern light projection unit 102, adjusting the light projection intensity, or shortening the exposure time of the stereo camera 101.

[0060] Figure 11(b) is an illustrative diagram showing that overexposure occurs in the patterned image 1103 in step S204, and also in the unpatterned image 1104 in step S205. Since overexposure occurs even when the pattern light 103 is turned off, it can be determined that the overexposure is caused by specular reflection of ambient light such as the fluorescent lamp 601. In such cases, countermeasures can be taken by blocking the light from the fluorescent lamp 601 with a light-shielding plate or the like, changing the position of the workpiece 504, or shortening the exposure time of the stereo camera 101.

[0061] Figure 11(c) is an illustrative diagram showing that no overexposure occurs in the patterned image 1103 in step S204, and no overexposure occurs in the unpatterned image 1104 in step S205. Since no overexposure occurs, it can be determined that there is another cause. Proceed to step S206 to estimate the other factors.

[0062] In step S206, a third evaluation result is derived, indicating whether the area of ​​the underexposed (blacked-out) region in the evaluation region of the patterned image is above a threshold. Here, underexposed refers to a state in the captured image where the brightness value is close to 0. In this case, the contrast of the pattern light 103 cannot be obtained, and stereo matching processing cannot be performed correctly. If it is not above the threshold, the process proceeds to step S207. On the other hand, if it is above the threshold, it can be identified that insufficient brightness of the pattern light 103 received by the stereo camera 101 is the cause of the decrease in three-dimensional measurement accuracy. If the light from the pattern projection unit 102 is blocked by an obstruction (occlusion), countermeasures can be taken by removing the obstruction or increasing the exposure time of the stereo camera 101.

[0063] This section describes one example of how to determine the area of ​​a blacked-out region. Even if there is no incident light on the image sensor 203, the brightness may not be zero due to dark current noise in the image sensor 203. Therefore, a predetermined value, such as a brightness value near zero (e.g., 5), is set as the brightness threshold. Regions (blobs) below the brightness threshold within the evaluation area are extracted using binarization, and the area can be calculated from the number of pixels in these blobs. The area threshold is the same as for the blown-out region, so the explanation is omitted.

[0064] Furthermore, similar to the case of overexposed areas, the brightness threshold and area threshold for underexposed areas set in these parameters will vary depending on the use case, so it is desirable that they be adjustable parameters on the 3D measurement application 901.

[0065] In step S207, a fourth evaluation result is derived, indicating whether the contrast of the pattern is above a threshold in the evaluation region of the patterned image. Contrast is the difference between the bright and dark areas in the captured image. For example, if the brightness of the bright area is dmax and the brightness of the dark area is dmin, the contrast can be calculated as (dmax-dmin) / (dmax+dmin).

[0066] As mentioned above, low contrast (weak texture) degrades the performance of stereo matching processing. Therefore, if the contrast is below the threshold, it can be identified that low contrast is the cause of the decrease in three-dimensional measurement accuracy. In such cases, it is presumed that the pattern light 103 is outside the focus range of the stereo camera 101. Therefore, countermeasures can be taken by adjusting the arrangement of the pattern light projection unit 102 so that the pattern light 103 is in focus, or by adjusting the arrangement of the stereo camera 101 so that the camera is in focus.

[0067] On the other hand, if the value is above the threshold, it means that contrast is obtained, so it can be determined that there is another factor causing the decrease in three-dimensional measurement accuracy. One possible cause is that the parameters of the stereo matching process are inappropriate. For example, if the search range and minimum parallax settings are inappropriate, the actual height at which the workpiece is placed may be outside the search range. In this way, adjustment of the parameters of the stereo matching process may be a possible solution. Thus, the suggested countermeasures in the estimation result window 911 specifically suggest reviewing the adjustment of the parameters of the stereo matching process.

[0068] If the problem persists even after reviewing the parameter adjustments, it is possible that the internal and external parameters of the stereo camera 101 have changed since calibration. For example, if the ambient temperature at the usage location is high, the holding mechanisms of the monocular cameras 201 and 202 may undergo thermal deformation. In such cases, as shown in Figure 12, a marker plate such as a whiteboard with black circles drawn on it is photographed separately, and the center positions of the black circles in the reference image 1201 and the corresponding image 1202 are measured. If there is no calibration error, the deviation of these center positions (epipolar deviation) will be close to 0px. On the other hand, if a calibration error occurs, the epipolar deviation will be large. Thus, it is possible to determine whether the epipolar deviation is within the acceptable range. The acceptable range depends on the required accuracy, but 1px or less is desirable.

[0069] As described above, the flow from S201 to S207 makes it possible to estimate the factors that caused the decrease in three-dimensional measurement accuracy and to present countermeasures to the user.

[0070] (Second embodiment) In the first embodiment described above, in step S103, the user set evaluation regions 1003 and 1004 for the reference image and the corresponding image, respectively. However, it can be difficult to find the same region as evaluation region 1003 of the reference image in the corresponding image.

[0071] For example, as shown in Figure 13, when pattern light 103 is projected onto a uniform plane and photographed, it is extremely difficult for a user to visually locate the same position in the reference image 1301 and the corresponding image 1302. Thus, there was a problem in that it was difficult to set the evaluation area at the same location in the reference image 1301 and the corresponding image 1302. Therefore, in this embodiment, a method is described in which, once the evaluation area is set in one image, the evaluation area of ​​the other image is automatically set.

[0072] In the following section, we will illustrate and explain the parts of the flow that differ from the first embodiment. Furthermore, we will assume that the parts similar to the first embodiment have the same configuration and operation as described above, and will omit their detailed explanation.

[0073] Figure 14 shows an example of the display screen of the three-dimensional measurement application 1401 of this embodiment. In this example, it is assumed that a pattern light 103 is projected onto a uniform plane and photographed. The three-dimensional measurement application 1401 has multiple input units. The user can set the evaluation area and input the ideal distance using the user interface device described above. In this embodiment, we will describe the case in which the user sets the evaluation area 1404 on the reference image window 1402 and the evaluation area 1405 in the corresponding image window 1403 is set automatically. As described above, the evaluation area 1406 of the three-dimensional measurement result has the same coordinate values ​​as the evaluation area 1404 of the reference image window 1402, so the method for setting the evaluation area 1406 will be omitted from the explanation.

[0074] First, the user sets the evaluation area 1404 on the reference image window 1402, as in the first embodiment. Next, the user enters the ideal distance to the evaluation area 1404 in the ideal distance window 1408. Here, the ideal distance is the distance that could be correctly measured by the stereo camera 101. The user can estimate this ideal distance from measured values ​​in the vicinity of the evaluation area 1404.

[0075] For example, in this embodiment, since a uniform plane is being measured, the ideal distance will be the same as the distance value of the correctly measured area near the evaluation area. It is desirable that the user can easily find out the distance value near the evaluation area by displaying the distance value of the location dragged with the mouse in the three-dimensional measurement results window in the distance measurement results window 1409. Alternatively, if the shape of the workpiece is known, it is also possible to estimate the ideal distance from the shape of the workpiece.

[0076] Next, we will explain how to calculate the evaluation area 1405 on the corresponding image window 1403. As mentioned above, the measurement result z can be expressed as z = Bf / d, where B is the baseline length, f is the focal length, and d is the parallax. In other words, if the ideal distance zi is known, the ideal parallax di can be calculated as di = Bf / zi.

[0077] If the column direction value at the center of the evaluation region 1404 in the reference image window 1402 is denoted as cl, then the column direction value cr at the center of the evaluation region 1405 in the corresponding image window 1403 can be calculated as cr = cl - di. Due to the epipolar constraint, the row direction value is equal in the reference image and the corresponding image.

[0078] The user can determine if they are at the desired position by checking the evaluation area 1405 in the corresponding image window 1403. If the position of the evaluation area 1405 changes in conjunction with the change in the value of the ideal distance window 1408, the user can fine-tune the position while checking the position of the evaluation area 1405. After confirming that they are at the desired position, the evaluation area confirmation button 909 is pressed to set evaluation areas 1404, 1405, and 1406. In this way, the evaluation area 1405 in the corresponding image window 1403 can be set automatically.

[0079] In this embodiment, the evaluation region 1405 of the corresponding image window 1403 was automatically determined, but the user may manually set the evaluation region 1405 of the corresponding image window 1403 and automatically set the evaluation region 1404 of the reference image window 1402 using a similar method. In other words, the evaluation region used to derive the first evaluation result can be derived from the evaluation region of either the first patterned image or the second patterned image, and the distance from the stereo camera 101 to the workpiece 504.

[0080] In this embodiment, the ideal distance was entered into the ideal distance window 1408, but the ideal distance may also be automatically set by inferring it from the distance values ​​near the evaluation area. This eliminates the need for the user to enter the ideal distance, thus improving usability.

[0081] (Third embodiment) In the first and second embodiments described above, the proposed countermeasures are only displayed in the estimated results window 911 in step S105, and the user has to manually implement the countermeasures, which presents the problem of being time-consuming. Therefore, in this embodiment, a method for automatically implementing countermeasures when they can be expected to be implemented automatically will be described.

[0082] In the following sections, we will illustrate and explain the parts of the flow that differ from the first and second embodiments. Furthermore, we will assume that the parts similar to those in the first and second embodiments have the same configuration and operation as described above, and will omit their detailed explanation.

[0083] Figure 15 is a flowchart of the process in this embodiment, from capturing an image with the stereo camera 101 to estimating the factors causing the decrease in three-dimensional measurement accuracy and implementing countermeasures. Steps S301 to S303 are the same as steps S101 to S103 in Embodiment 1, so their explanation will be omitted.

[0084] Step S304 is a process in which the factors causing the decrease in three-dimensional measurement accuracy in the evaluation area set in step S303 are estimated, and countermeasures to be taken automatically and countermeasures to be taken manually are estimated. Countermeasures to be taken automatically are those that can be taken only by adjusting the parameters of the software installed in the image processing device 104.

[0085] Let's consider an example of an automated countermeasure. First, let's consider the case where overexposure occurs, reducing the accuracy of the three-dimensional measurement. In this case, shortening the exposure time of the stereo camera 101 may improve the situation. Conversely, if underexposure occurs, increasing the exposure time of the stereo camera 101 may improve the situation.

[0086] Let's describe an example of a measure that requires human intervention. For example, blocking ambient light can be done by installing a light-blocking plate. This is a measure that cannot be controlled by the image processing device 104.

[0087] In this embodiment, only the adjustment of exposure time has been described, but measures such as HDR (High Dynamic Range) synthesis of images taken with multiple exposure times may also be implemented. In this way, measures can be taken against both overexposure and underexposure. If the dimming of the pattern projection unit 102 can be controlled, it may be controlled by dimming. Increasing the exposure time increases the shooting time, but by controlling the dimming, measures can be taken without changing the shooting time.

[0088] Step S305 is a process in which the estimation results window 1603 displays the factors that reduced the three-dimensional measurement accuracy estimated in step S304, the countermeasures that can be performed automatically, and the countermeasures that require human intervention. Here, it is preferable that the countermeasures that can be performed automatically and the countermeasures that require human intervention are displayed separately. Also, if there were multiple countermeasures that could be performed automatically in step S304, the countermeasure should be selectable here.

[0089] Step S306 is the process of implementing the countermeasures described in the countermeasures section of the estimated results window when the countermeasures implementation button 1602 is pressed. After implementing the countermeasures, the image is photographed again and three-dimensional measurement is performed, and the results after the countermeasures are displayed in the reference image display window 904, the corresponding image display window 905, and the three-dimensional measurement results window 908 of the three-dimensional measurement application 1601.

[0090] Next, an example of countermeasures will be explained. Let's consider the case where overexposure occurs and the accuracy of three-dimensional measurement decreases. In this case, the image processing device 104 shortens the exposure time of the stereo camera 101 and performs the same processing as steps S204 to S207 of Embodiment 1 for each captured image to identify whether the image within the evaluation area is suitable for three-dimensional measurement. An image is considered suitable for three-dimensional measurement if the result in steps S204, S206, and S207 is No.

[0091] Once the range of exposure times suitable for three-dimensional measurement is determined, an appropriate exposure time is set within that range. For example, if an exposure time between 10ms and 100ms is identified as suitable for three-dimensional measurement, the exposure time is set to the intermediate value of 55ms. The exposure time after the adjustment is displayed in the shooting conditions window 903.

[0092] Step S307 is the process of determining whether the countermeasures implemented in step S306 are sufficient, based on the results displayed in the three-dimensional measurement results window 908. If the countermeasures are insufficient, the process proceeds to step S308. The determination can be made by the user visually inspecting the three-dimensional measurement results. Alternatively, if the user performs post-processing on the three-dimensional measurement results, the determination can be made by executing the post-processing program and determining whether sufficient accuracy can be obtained from the post-processing.

[0093] Step S308 is the process in which the user implements the countermeasures displayed in the estimated results window 1603 if the problem is not resolved by the countermeasures already implemented. After implementing the countermeasures, the user takes another image and performs a 3D measurement, continuing to implement countermeasures until the problem is resolved.

[0094] Embodiments of this disclosure also include a control program capable of executing the measurement method described above, and a recording medium readable by a computer that stores the control program. As a recording medium for supplying the control program, for example, ROM, disks, external storage devices, etc., may be used. To give specific examples, as a computer-readable non-temporary recording medium, flexible disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory such as USB memory, SSDs, etc., can be used.

[0095] Furthermore, this disclosure can also be implemented by supplying a program that implements one or more functions of the embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0096] Furthermore, these effects are merely a list of the most preferred effects arising from the technology of this disclosure, and the effects of the technology of this disclosure are not limited to those described above.

[0097] This embodiment includes the following configuration.

[0098] (Item 1) A measuring device for performing three-dimensional measurement, comprising: a pattern projection unit that projects patterned light onto an object to be measured; a camera having a first camera and a second camera that photographs the object; and an image processing unit that processes data acquired from the camera, The aforementioned image processing unit, With the patterned light projected, the first evaluation result is derived from the first patterned image acquired by the first camera and the second patterned image acquired by the second camera. Based on the first evaluation results, a plan of action to improve the accuracy of the three-dimensional measurement is derived. A measuring device characterized by the following features.

[0099] (Item 2) The image processing unit derives a second evaluation result from the first patternless image acquired by the first camera and the second patternless image acquired by the second camera, while the pattern light is not being projected. The measuring device according to item 1, characterized in that it derives the proposed countermeasures using the first evaluation result and the second evaluation result.

[0100] (Item 3) The measuring device according to item 2, characterized in that, based on the first evaluation result and the second evaluation result, it derives whether the proposed countermeasure is a countermeasure against the pattern light of the pattern projection unit or a countermeasure against ambient light in the environment in which the measuring device is installed.

[0101] (Item 4) The measuring device according to any one of items 1 to 3, characterized in that the first evaluation result is derived as the area of ​​a blob having a brightness value higher than a predetermined value.

[0102] (Item 5) The measuring device according to any one of items 1 to 4, characterized in that the second evaluation result is derived as the area of ​​a blob having a brightness value higher than a predetermined value.

[0103] (Item 6) The measuring device according to any one of items 1 to 5, characterized in that the image processing unit derives a third evaluation result as the area of ​​a blob having a brightness value lower than a predetermined value from the first patterned image and the second patterned image.

[0104] (Item 7) The measuring device according to item 6, characterized in that, based on the third evaluation result, it derives whether or not the proposed countermeasure relates to the brightness of the pattern light received by the camera.

[0105] (Item 8) The measuring device according to item 7, characterized in that the image processing unit derives a fourth evaluation result, which is the contrast, which is the difference between bright and dark areas, from the first patterned image and the second patterned image.

[0106] (Item 9) The measuring device according to item 8, characterized in that, based on the fourth evaluation result, it derives whether or not the proposed countermeasure relates to the focus of the camera with respect to the pattern light.

[0107] (Item 10) The measuring device according to any one of items 1 to 9, characterized in that the image processing unit derives an evaluation region used to derive the first evaluation result from the evaluation region of either the first patterned image or the second patterned image, and the distance from the camera to the object.

[0108] (Item 11) It further has a display unit, The measuring device according to item 10, characterized in that the image processing unit causes the display unit to display an input unit for inputting either of the evaluation areas and the distance.

[0109] (Item 12) It further has a display unit, The measuring device according to any one of items 1 to 11, characterized in that the image processing unit displays the proposed countermeasures on the display unit.

[0110] (Item 13) The measuring device according to item 12, characterized in that the display unit displays separately the proposed countermeasures as those that do not require manpower to implement and those that do require manpower to implement.

[0111] (Item 14) The measuring device according to any one of items 1 to 13, characterized in that the camera incorporates the image processing unit.

[0112] (Item 15) A measuring device for performing three-dimensional measurement, comprising a control unit that controls a pattern projection unit that projects patterned light onto an object to be measured, and an image processing unit that processes data acquired from a camera, The aforementioned image processing unit, With the control unit projecting patterned light onto the object, the first evaluation result is derived from the first patterned image acquired by the first camera of the camera and the second patterned image acquired by the second camera of the camera. Based on the first evaluation results, a plan of action to improve the accuracy of the three-dimensional measurement is derived. A measuring device characterized by the following features.

[0113] (Item 16) A measurement method for a measuring apparatus that performs three-dimensional measurement, comprising: a pattern projection unit that projects patterned light onto an object to be measured; a camera having a first camera and a second camera that photographs the object; and an image processing unit that processes data acquired from the camera, With the pattern light projected, the image processing unit derives a first evaluation result from the first patterned image acquired by the first camera and the second patterned image acquired by the second camera. Based on the first evaluation result, the process involves deriving proposed measures to improve the accuracy of the three-dimensional measurement. A measurement method characterized by the following.

[0114] (Item 17) A program to cause a computer to execute the measurement method described in item 16.

[0115] (Item 18) A computer-readable recording medium containing the program described in item 17.

[0116] (Item 19) The manufacturing process of goods, An inspection step is performed in which the articles manufactured by the manufacturing process are inspected using a measuring device described in any one of items 1 to 15. A method for manufacturing an article characterized by the following: [Explanation of Symbols]

[0117] 101 Stereo Camera 102 Pattern projection unit 103 Light Patterns 104 Image Processing Device 201, 202 Monocular Camera 504 Work 601 Fluorescent lamp Images with patterns 1101, 1103, 1105 1102, 1104, 1106 No pattern image

Claims

1. A measuring device for performing three-dimensional measurement, comprising: a pattern projection unit that projects patterned light onto an object to be measured; a camera having a first camera and a second camera that photographs the object; and an image processing unit that processes data acquired from the camera, The aforementioned image processing unit, With the patterned light projected, the first evaluation result is derived from the first patterned image acquired by the first camera and the second patterned image acquired by the second camera. Based on the first evaluation results, a plan of action to improve the accuracy of the three-dimensional measurement is derived. A measuring device characterized by the following features.

2. The image processing unit derives a second evaluation result from the first patternless image acquired by the first camera and the second patternless image acquired by the second camera, while the pattern light is not being projected. The measuring device according to claim 1, characterized in that it derives the proposed countermeasures using the first evaluation result and the second evaluation result.

3. The measuring device according to claim 2, characterized in that, based on the first evaluation result and the second evaluation result, it is determined whether the proposed countermeasure is a countermeasure against the pattern light of the pattern projection unit or a countermeasure against ambient light in the environment in which the measuring device is installed.

4. The measuring device according to claim 1, characterized in that the first evaluation result is derived as the area of ​​a blob having a brightness value higher than a predetermined value.

5. The measuring device according to claim 2, characterized in that the second evaluation result is derived as the area of ​​a blob having a brightness value higher than a predetermined value.

6. The measuring device according to claim 1, characterized in that the image processing unit derives a third evaluation result as the area of ​​a blob having a brightness value lower than a predetermined value from the first patterned image and the second patterned image.

7. The measuring device according to claim 6, characterized in that, based on the third evaluation result, it derives whether or not the proposed countermeasure relates to the brightness of the pattern light received by the camera.

8. The measuring device according to claim 7, characterized in that the image processing unit derives a fourth evaluation result, which is the contrast, which is the difference between bright and dark areas, from the first patterned image and the second patterned image.

9. The measuring device according to claim 8, characterized in that, based on the fourth evaluation result, it derives whether or not the proposed countermeasure relates to the focus of the camera with respect to the pattern light.

10. The measuring device according to claim 1, characterized in that the image processing unit derives an evaluation region used to derive the first evaluation result from the evaluation region of either the first patterned image or the second patterned image, and the distance from the camera to the object.

11. It further has a display unit, The measuring device according to claim 10, characterized in that the image processing unit causes the display unit to display an input unit for inputting either of the evaluation areas and the distance.

12. It further has a display unit, The measuring device according to claim 1, characterized in that the image processing unit causes the proposed countermeasures to be displayed on the display unit.

13. The measuring device according to claim 12, characterized in that the display unit displays, as a set of countermeasures, separately, a set of countermeasures that does not require manpower to implement and a set of countermeasures that does require manpower to implement.

14. The measuring device according to claim 1, characterized in that the camera incorporates the image processing unit.

15. A measuring device for performing three-dimensional measurement, comprising a control unit that controls a pattern projection unit that projects patterned light onto an object to be measured, and an image processing unit that processes data acquired from a camera, The aforementioned image processing unit, With the control unit projecting patterned light onto the object, the first evaluation result is derived from the first patterned image acquired by the first camera of the camera and the second patterned image acquired by the second camera of the camera. Based on the first evaluation results, a plan of action to improve the accuracy of the three-dimensional measurement is derived. A measuring device characterized by the following features.

16. A measurement method for a measuring apparatus that performs three-dimensional measurement, comprising: a pattern projection unit that projects patterned light onto an object to be measured; a camera having a first camera and a second camera that photographs the object; and an image processing unit that processes data acquired from the camera, With the pattern light projected, the image processing unit derives a first evaluation result from the first patterned image acquired by the first camera and the second patterned image acquired by the second camera. Based on the first evaluation result, the process involves deriving proposed measures to improve the accuracy of the three-dimensional measurement. A measurement method characterized by the following.

17. A program for causing a computer to execute the measurement method described in claim 16.

18. A computer-readable recording medium storing the program described in claim 17.

19. The manufacturing process of goods, An inspection step is performed in which the article manufactured by the manufacturing step is inspected using the measuring device described in any one of claims 1 to 15. A method for manufacturing an article, characterized by the following:

Images