Three-dimensional measurement device, three-dimensional measurement method, program, system, and article manufacturing method

Abstract

To provide a three-dimensional measurement device in which a calculation amount is reduced in the calculation of a distance value to an object.SOLUTION: A three-dimensional measurement device comprises a projection section that projects a pattern on an object, a plurality of imaging sections that image the object from mutually different viewpoints, and a processing section that calculates a distance value to the object by performing association using an image group obtained by imaging the object by the plurality of imaging sections. The processing section integrates information based on a first image group captured by projecting the pattern on the object and information based on a second image group captured without projecting the pattern on the object with an image or an evaluation value used for association so that a feature on the image in each of the image groups remains, and calculates the distance value using the integrated image or evaluation value.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a three-dimensional measuring device, a three-dimensional measuring method, a program, a system, and a method for manufacturing an article. [Background technology]

[0002] Traditionally, in factory production lines, robots equipped with 3D measurement devices have been used to assemble and inspect products. A conventional 3D measurement device is known to use stereo measurement, which searches for corresponding points between images taken with two cameras with different viewpoints and measures distance points based on the principle of triangulation using the parallax information of the corresponding points.

[0003] In Patent Document 1, a system is provided with two cameras and a projector that projects one pattern, and a distance value is measured by the principle of triangulation based on the correspondence between the dot pattern on the image captured by one camera and the dot pattern of the projector by projecting a dot pattern onto the measurement object.The system then calculates the measured distance value and a distance value measured stereoscopically from images captured by two different cameras without projecting a dot pattern onto the measurement object, and performs weighted voting based on the reliability of the distance value to calculate the final distance value.

[0004] In Patent Document 2, the reliability of distance values ​​measured stereoscopically from images captured by two different cameras by projecting a pattern in a configuration similar to that of Patent Document 1 and distance values ​​measured stereoscopically from images captured by two different cameras without projecting a pattern is calculated. After this calculation, low-reliability distance values ​​from active measurement are replaced with distance values ​​from passive measurement. Note that hereinafter, the above-mentioned stereo measurement from images captured by two different cameras by projecting a pattern will be referred to as active measurement, and stereo measurement from images captured by two different cameras without projecting a pattern will be referred to as passive measurement.

[0005] By combining active and passive measurements as described above, stereo measurements are possible even for measurement objects with little texture, enabling more accurate measurements to be achieved. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Pat. No. 10,636,155 [Patent Document 2] JP 2001-264033 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the conventional method described above, it is necessary to calculate distance points for both active and passive measurements. In general, the process of calculating distance points requires a large amount of calculation, which causes a problem that it takes a long time to calculate the distance value to the measurement target. In assembly and inspection applications on factory production lines, it is required to output distance points at high speed in order to shorten takt time as much as possible.

[0008] Therefore, an object of the present invention is to provide a three-dimensional measuring device that reduces the amount of calculation required to calculate the distance to an object. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, one aspect of the present invention provides a three-dimensional measuring device that includes a projection unit that projects a pattern onto an object, multiple imaging units that image the object from different viewpoints, and a processing unit that calculates a distance value to the object by performing matching using a group of images of the object imaged by the multiple imaging units, wherein the processing unit integrates information based on a first group of images imaged by projecting a pattern onto the object and information based on a second group of images imaged without projecting a pattern onto the object into images or an evaluation value used for matching so that image features in each image group remain, and calculates a distance value using the integrated images or evaluation value. Effect of the Invention

[0010] According to the present invention, it is possible to reduce the amount of calculation required to calculate the distance to an object. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a three-dimensional measuring device of the present invention. [Diagram 2] 4 is a flowchart showing an operation process of the three-dimensional measurement apparatus according to the first embodiment of the present invention. [Diagram 3] 3A to 3C are diagrams showing examples of an active image and a passive image captured by a first imaging unit and a second imaging unit in the first embodiment of the present invention. [Figure 4] 11 is a flowchart showing a distance value calculation process in a processing unit in Example 1 of Embodiment 1 of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of an integrated image generated using an active image and a passive image in the first embodiment of the present invention. [Figure 6] FIG. 11 is a diagram illustrating an example of the configuration of a first imaging unit in Example 2 of Embodiment 1 of the present invention. [Figure 7] 11 is a flowchart showing an operation process of a three-dimensional measuring apparatus in Example 2 of Embodiment 1 of the present invention. [Figure 8] FIG. 10 is a diagram showing a distance value calculation processing flow in a processing unit according to the second embodiment of the present invention. [Figure 9] FIG. 1 is a diagram showing an example of a control system including a gripping device equipped with a three-dimensional measuring device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiment for carrying out the present invention will be described in detail with reference to the attached drawings. The embodiment described below is one example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment.

[0013] <Embodiment 1> In the first embodiment, an active image and a passive image, which are images captured by one of the multiple image capturing units (the same image capturing unit), are integrated to generate (create) a new image as image data used for searching for corresponding points in stereo measurement. Here, when generating the new image, the image is generated so that features observed in each of the active image and the passive image, which are clues for matching, remain. Then, a distance value to the measurement target is calculated based on the newly generated image.

[0014] As a result, in a technique that combines active measurement and passive measurement, the amount of calculation required for distance value calculation can be reduced compared to the conventional technique. Below, a three-dimensional measuring device 100 that calculates the distance value to the measurement object using the above-mentioned method will be described. Note that active measurement refers to projecting a pattern onto the measurement object and performing stereo measurement from images captured by two different cameras, while passive measurement refers to stereo measurement from images captured by two different cameras without projecting a pattern onto the measurement object.

[0015] An active image is an image in which a pattern projected onto the surface of the measurement target is observed, whereas a passive image is an image in which no pattern is projected onto the surface of the measurement target, and the three-dimensional structure and surface texture of the measurement target are observed.

[0016] Fig. 1 is a diagram illustrating a three-dimensional measuring device 100 according to embodiment 1. As shown in Fig. 1, the three-dimensional measuring device 100 according to embodiment 1 is composed of a control unit 101, a projection unit 102, a first imaging unit 103, a second imaging unit 104, and a processing unit 105. Fig. 1 also illustrates an example of a measurement object 106 measured by the three-dimensional measuring device 100 according to embodiment 1.

[0017] The control unit 101 includes a CPU, a memory (storage unit), etc., and is configured by at least one computer. The control unit 101 controls various operations and the start and end of operations of the projection unit 102, the first imaging unit 103, the second imaging unit 104, and the processing unit 105 so that the three-dimensional measuring device 100 outputs a distance value of the measurement target object 106. Specifically, the control unit 101 sends a pattern light projection start command and a pattern light projection end command to the projection unit 102, and sends an imaging start command and an imaging end command to the first imaging unit 103 and the second imaging unit 104. The control unit 101 sends a distance value calculation start command and the like to the processing unit 105.

[0018] When the projection unit 102 receives a pattern light projection start command from the control unit 101, it projects the pattern light onto the surface of the measurement object 106 until it receives a pattern light projection end command from the control unit 101. The pattern light projected at this time desirably has a non-periodic intensity pattern based on uniform random numbers or normal random numbers. Examples of the light source for the pattern light include, but are not limited to, a laser or an LED. Any light source may be used as long as the wavelength of the projected light is included in a wavelength band to which the first imaging unit 103 and the second imaging unit 104 are sensitive, and the pattern on the surface of the measurement object 106 can be observed in the active image.

[0019] The first imaging unit 103 and the second imaging unit 104 are disposed at least at different positions or angles with respect to the measurement object 106, and the first imaging unit 103 and the second imaging unit 104 image the measurement object 106 from different viewpoints. When the first imaging unit 103 and the second imaging unit 104 receive an imaging start command from the control unit 101, they start imaging the measurement object 106. Then, they capture images in each line of sight direction until they receive an imaging end command from the control unit 101, and output the acquired image data to the processing unit 105. At this time, the first imaging unit 103 and the second imaging unit 104 are controlled so that the timing of image capture of the first imaging unit and the second imaging unit is synchronized with each other.

[0020] When the pattern light projected from the projection unit 102 is near-infrared light, the active image and the passive image have different wavelength bands received as reflected light from the measurement target 106. For this reason, a phenomenon (chromatic aberration) is likely to occur in which an image is observed blurred only in one of the active and passive images at the same pixel position.

[0021] There are several methods for reducing the effect of chromatic aberration. For example, it is effective to install chromatic aberration correction lenses inside the first imaging unit 103 and the second imaging unit 104. Another method for reducing the effect of chromatic aberration is to adjust the focus to different positions each time an active image is captured and a passive image is captured. When using these methods, special members and mechanisms are required, so the entire device tends to become expensive. Another method for reducing the effect of chromatic aberration is to correct the effect in the processing unit 105 using calibration parameters for active measurement and passive measurement, which will be described in detail later.

[0022] When the focus positions of the first imaging unit 103 and the second imaging unit 104 are fixed regardless of whether active or passive image capture is performed, the higher the spatial frequency, the more susceptible to the effect of defocus. Therefore, taking into consideration the effect of defocus, the focus may be adjusted to be on one of the active image and the passive image before being fixed. Specifically, the spatial frequency originating from the pattern on the surface of the measurement object 106 observed in the active image and the spatial frequency originating from the feature on the measurement object 106 observed in the passive image are identified and compared. After that, the focus position is adjusted and fixed so that the image with the higher spatial frequency is focused on.

[0023] The first imaging unit 103 and the second imaging unit 104 need to have sensitivity to the wavelength band of the pattern light projected from the projection unit 102 and the wavelength band of visible light.

[0024] The processing unit 105 receives each image data as an image group of the measurement object 106 captured from the first imaging unit 103 and the second imaging unit 104, and performs distance value calculation processing based on each received image data upon receiving a distance calculation start command from the control unit 101. Specifically, the processing unit 105 performs processing to calculate a distance value to the measurement object 106 by performing correspondence using each image data of the measurement object 106 captured by the first imaging unit 103 and the second imaging unit 104. The processing of calculating a distance value to the measurement object 106 in the processing unit 105 will be described in detail later.

[0025] Furthermore, the processing unit 105 holds calibration parameters of the first imaging unit 103 and the second imaging unit 104 as data necessary for performing distance value calculation processing. Here, the calibration parameters include parameters representing the relative position and orientation of the first imaging unit 103 and the second imaging unit 104, and parameters representing the image center, focal length, and distortion aberration of each imaging unit. Note that, when the wavelength of the pattern light projected from the projection unit 102 during active image capture is near-infrared light, the processing unit 105 may hold calibration parameters for active measurement and calibration parameters for passive measurement.

[0026] Example 1 In Example 1 of the first embodiment, a method for capturing an active image and a passive image separately will be described below.

[0027] 2 is a flowchart showing operations (processing) up to calculating the distance value of the measurement target object 106 by the three-dimensional measuring device 100 in Example 1 of the embodiment 1. Note that each operation (processing) shown in the flowchart in FIG. 2 is controlled by the control unit 101 executing a computer program.

[0028] First, in step S1101, an image capture start command is sent from the control unit 101 to the first image capture unit 103 and the second image capture unit 104. When the image capture start command is sent to each image capture unit, the first image capture unit 103 and the second image capture unit 104 capture an ambient light image in which no pattern light is projected onto the surface of the measurement target object 106 from each viewpoint. The ambient light image is used to remove ambient light noise from the active image in the noise removal process described below, and the exposure time during image capture needs to match the exposure time during active image capture. This step is necessary when the exposure times for active image capture and passive image capture are different, but does not need to be performed when images are captured with the same exposure time.

[0029] Next, in step S1102, the control unit 101 sends a pattern light projection start command to the projection unit 102, and the projection unit 102 projects the pattern light onto the measurement object 106 (projection step).

[0030] Next, in step S1103, the control unit 101 sends an imaging start command to the first imaging unit 103 and the second imaging unit 104. When the imaging start command is sent to each imaging unit performing imaging, the first imaging unit 103 and the second imaging unit 104 capture the scene in which the pattern light is projected onto the surface of the measurement object 106 from each viewpoint (imaging process). This imaging process is performed until an imaging end command is sent from the control unit 101. This step makes it possible to obtain a group of images (first image group) captured by each imaging unit by projecting the pattern light onto the measurement object 106.

[0031] 3A and 3B are diagrams showing examples of an active image and a passive image acquired by the first imaging unit 103 and the second imaging unit 104 in the first embodiment of the present invention. Fig. 3(A) is a diagram showing an example of an active image acquired by the first imaging unit 103 and the second imaging unit 104. Fig. 3(B) is a diagram showing an example of a passive image acquired by the first imaging unit 103 and the second imaging unit 104.

[0032] 3A is a conceptual diagram of an active image captured by the first imaging unit 103, and a second active image 111 is a conceptual diagram of an active image captured by the second imaging unit 104. In the first active image 110 and the second active image 111 shown in FIG. 3A, a pattern observed on the surface of the measurement object 106 is used as a clue for calculating a parallax value in a calculation process of a distance value to the measurement object 106, which will be described later. In this step, image data of each active image (first image group) captured and acquired by the first imaging unit 103 and the second imaging unit 104 is sent to the processing unit 105.

[0033] Next, in step S1104, the control unit 101 sends a pattern light projection end command to the projection unit 102, and the projection unit 102 ends the projection of the pattern light onto the measurement object 106.

[0034] Next, in step S1105, the control unit 101 sends an image capture start command to the first imaging unit 103 and the second imaging unit 104. When the image capture start command is sent to each imaging unit that performs imaging, the first imaging unit 103 and the second imaging unit 104 capture a scene where no pattern light is projected onto the surface of the measurement object 106 from each viewpoint (imaging process). This imaging process is performed until an image capture end command is sent from the control unit 101. At this time, the first imaging unit 103 and the second imaging unit 104 receive visible light reflected from the measurement object 106. This step makes it possible to obtain a group of images (second group of images) captured by each imaging unit without projecting pattern light onto the measurement object 106.

[0035] 3B is a conceptual diagram of a passive image captured by the first imaging unit 103, and the second passive image 113 is a conceptual diagram of a passive image captured by the second imaging unit 104. Features such as steps, edges, and surface textures derived from the three-dimensional structure of the measurement object 106 observed in the first passive image 112 and the second passive image 113 are used as clues for calculating a parallax value in the calculation process of the distance value to the measurement object 106. In this step, the image data of each passive image (second image group) captured and acquired by the first imaging unit 103 and the second imaging unit 104 is sent to the processing unit 105.

[0036] The operational flow from step S1101 to step S1105 described so far is a flow for capturing an image for ambient light noise removal, an active image, and a passive image in this order, but this order may be changed as desired. For example, when capturing an image in the order of a passive image, an active image, and an image for ambient light noise removal, various processes are executed in the order of steps S1105, S1102, S1103, S1104, and S1101.

[0037] Next, in step S1106, the control unit 101 sends a distance calculation start command to the processing unit 105, and the processing unit 105 calculates a distance value using each image data of the active image and each image data of the passive image, and the calibration parameters (processing step).

[0038] Fig. 4 is a flowchart showing a specific operation of the process for calculating the distance value of the measurement object 106, which is performed by the processing unit 105. That is, Fig. 4 is a flowchart for explaining the details of the process of S1106 shown in Fig. 2. Note that each operation (process) shown in the flowchart in Fig. 4 is controlled by the control unit 101 executing a computer program.

[0039] First, in step S1111, noise removal processing is performed on the first active image 110 and the second active image 111. The noise to be removed in step S1111 refers to ambient light noise contained when the first active image 110 and the second active image 111 are captured, and speckle noise generated when pattern light is reflected by the measurement object 106.

[0040] A method for removing ambient light noise is to subtract the luminance value originating from the active image from the luminance value originating from the ambient light image captured in step S1101 described above, for the luminance values ​​of the same pixel positions in an active image captured by the same imaging unit and in the ambient light image captured in step S1101 described above. An effective method for removing speckle noise is to perform smoothing processing using a smoothing filter such as a Gaussian filter. In this case, a higher noise removal effect can be expected by optimizing the kernel size of the smoothing filter according to the observed pattern, for example by setting the kernel size of the smoothing filter to approximately the dot size of the pattern observed in the first active image 110 and the second active image 111.

[0041] In this step, there is no strict restriction on the order in which the ambient light noise removal and the speckle noise removal are performed, and the speckle noise removal may be performed first, followed by the ambient light noise removal. However, it is preferable to perform the ambient light noise removal first, followed by the speckle noise removal. The ambient light noise removal in this step has been described on the premise that the exposure times in the active image capture and the passive image capture are different, but if the exposure times are the same, the passive image may be used as the ambient light image.

[0042] Next, in step S1112, aberrations occurring in the first passive image 112, the second passive image 113, and the first active image 110 and the second active image 111 after noise removal are corrected (distortion correction). Calibration parameters of the first imaging unit 103 and the second imaging unit 104 are used for the aberration correction. When the wavelengths of light received by the first imaging unit 103 and the second imaging unit 104 differ between when capturing an active image and when capturing a passive image, it is preferable to use different calibration parameters for active measurement and passive measurement in order to reduce the influence of chromatic aberration. Specifically, the influence of chromatic aberration can be reduced by using calibration parameters for active measurement for aberration correction in the active image and calibration parameters for passive measurement for aberration correction in the passive image.

[0043] Next, in step S1113, features in each of the first active image 110 and the second active image 111, and the first passive image 112 and the second passive image 113 are detected. As described above, features in the active image are patterns projected onto the surface of the measurement object 106. Features in the passive image are the three-dimensional structure and surface texture of the measurement object 106. These features are considered to be observed in the image near areas with large brightness gradients. Therefore, an edge detection process (edge ​​extraction process) is performed on the first active image 110 and the second active image 111, and the first passive image 112 and the second passive image 113, using an edge detection filter such as a Sobel filter to detect (extract) edges.

[0044] Thereafter, for example, by binarizing and performing expansion processing, a feature extraction image can be generated in which the features of the active image and the passive image, for example, the features of the measurement object 106, are extracted. By using the feature extraction image, it becomes possible to perform appropriate weighting addition for each pixel position in step S1114. As another example of a method for generating a feature extraction image in this step, a smoothing filter such as a Gaussian filter may be used after detecting edges. In this way, in steps S1111 to S1113, individual image processing (correction processing) is performed on each image in accordance with the characteristics of each image. Then, the processing in this step completes preparations for image generation in the subsequent step S1114.

[0045] Next, in step S1114, an image (third image) to be used for matching for parallax calculation is newly generated using the active image (first image), the passive image (second image), and the feature extraction image that have been subjected to aberration correction and noise removal. Specifically, an image (third image) is generated based on an active image (first image) captured by one imaging unit from each active image (first image group) and a passive image (second image) captured by the one imaging unit from each passive image (second image group).

[0046] In addition, when generating a new image, the feature extraction image is also used as described above. Here, the newly generated image (third image) is an integrated image generated by integrating (combining) the active image and the passive image so as to retain their respective features. In other words, the newly generated integrated image is an image in which the pattern projected onto the surface of the measurement object 106, as well as the three-dimensional structure and surface texture of the measurement object 106, can be clearly observed.

[0047] In this case, a method for generating a new image is to perform weighted addition of luminance values ​​of corresponding pixel positions in an active image and a passive image captured by the same image capture unit. The weighting coefficient is determined by using the feature extraction image generated in step S1113. For example, a predetermined threshold is set in advance, and the weighting coefficient is determined depending on whether the luminance value in the feature extraction image is equal to or greater than the predetermined threshold.

[0048] For example, if the luminance value is equal to or greater than a predetermined threshold only in the feature extraction image generated from the passive image, the weighting coefficient of the luminance value derived from the passive image is increased and weighted addition is performed. Similarly, if the luminance value is equal to or greater than a predetermined threshold only in the feature extraction image generated from the active image, the weighting coefficient of the luminance value derived from the active image is increased and weighted addition is performed. If the luminance value is equal to or greater than the predetermined threshold for both feature extraction images, or if the luminance value is equal to or greater than the predetermined threshold for both feature extraction images, the same weighting coefficient is set and weighted addition is performed.

[0049] Fig. 5 is a diagram showing an example of an integrated image that is an image generated using an active image and a passive image in embodiment 1. Fig. 5(A) is a conceptual diagram showing a first integrated image 114 newly generated from a first active image 110 and a first passive image 112. Fig. 5(B) is a conceptual diagram showing a second integrated image 115 newly generated from a second active image 111 and a second passive image 113.

[0050] In the active image shown in Fig. 3(A), the pattern on the surface of the measurement object 106 can be clearly observed, but the step, which is a feature derived from the measurement object, is unclear, making it difficult to correctly match the area near the step. In the passive image shown in Fig. 3(B), the step, which is a feature derived from the measurement object 106, can be clearly observed, but there are no features on the surface of the measurement object 106, making it difficult to correctly match this area.

[0051] On the other hand, in the integrated images (first integrated image 114, second integrated image 115) shown in Fig. 5, the features derived from the active image and the passive image remain, so that the features in both images can be clearly observed. In the subsequent processing, the distance value to the measurement target object 106 is calculated using the integrated image. By using the integrated image, it is possible to reduce the amount of calculation required for distance value calculation compared to the conventional method in which the active image and the passive image are used independently to calculate the parallax value.

[0052] Next, in step S1115, an evaluation value is calculated for each combination of pixels of interest in the first integrated image 114 and the second integrated image 115. Examples of evaluation formulas for calculating the evaluation value include SAD (Sum of Absolute Difference) and SSD (Sum of Squared Difference). Other examples include ZNCC (Zero Means Normalized Cross Correlation). Note that the evaluation formula used in this step is not limited to these, and any formula can be used as long as it can evaluate the similarity between features in the vicinity of the pixel position of interest in the first integrated image 114 and features in the vicinity of the pixel position of interest in the second integrated image 115.

[0053] Next, in step S1116, the disparity value is calculated using the evaluation value calculated in step S1115. For example, when the disparity value is calculated based on the first integrated image 114, the epipolar line corresponding to the pixel position of interest Pi on the first integrated image 114 exists in the second integrated image 115. In addition, in step S1115, the evaluation values ​​(Eij1, Eij2, Eij3, . . ., Eijn) between each pixel position (Pj1, Pj2, Pj3, . . ., Pjn) on the epipolar line of the second integrated image 115 and Pi have already been calculated. In this step, the combination with the highest degree of agreement among these evaluation values ​​is selected, and the corresponding disparity value is calculated as the disparity value for Pi. The same process is performed while changing the pixel position of interest.

[0054] Next, in step S1117, the parallax value is converted into a distance value. The distance value can be calculated from the parallax value using the calibration parameters of the first imaging unit 103 and the second imaging unit 104. Note that since the active image and the passive image are integrated in step S1114, each piece of information may remain after integration. As a result, the distance value calculated in this step may differ from both the distance value calculated from only the active image and the distance value calculated from only the passive image.

[0055] Although Example 1 in the first embodiment has been described above, Example 1 is not limited to the above-mentioned method and various modifications are possible. For example, it is possible to perform the subsequent processes without performing all or part of the processes in steps S1111 and S1112. In step S1113, it is also possible to generate a feature extraction image only for the active image or passive image and use it to determine the weighting coefficient in step S1114.

[0056] For example, when step S1113 is not performed, a method of setting a preset weighting factor uniformly for all pixels may be used in step S1114. As a method of setting the weighting factor, for example, a method of comparing the average luminance in a region where a pattern is projected on the surface of the measurement object in the active image with the average luminance in a region where a feature derived from the measurement object is observed in the passive image may be used. When this method is used, the ratio after the comparison is set as the weighting factor.

[0057] Example 2 In Example 2 of the first embodiment, a method will be described in which the measurement target 106 is imaged only once, and the image is temporarily divided into an active image and a passive image based on one acquired image data. In Example 2, it is assumed that the wavelength of the pattern light projected by the projection unit 102 is near infrared. It is also assumed that the first imaging unit 103 and the second imaging unit 104 can respectively acquire an active image and a passive image by one imaging.

[0058] FIG. 6 is a diagram showing an example of the configuration of the first imaging unit 103 in the second embodiment. Note that FIG. 6 shows only the configuration of the first imaging unit 103, but the configuration of the second imaging unit 104 is similar. In the three-dimensional measuring device 100 of the second embodiment shown in FIG. 6, the reflected light from the measurement target 106 is split into two optical paths by the spectroscope 120. Thereafter, the visible light component of one of the split reflected lights is cut by the visible light cut filter 121, and the near-infrared light receiving unit 122 receives the transmitted near-infrared light. The near-infrared light component of the other split reflected light is cut by the near-infrared light cut filter 123, and the visible light receiving unit 124 receives the transmitted visible light. An active image is generated based on the near-infrared light received by the near-infrared light receiving unit 122, and a passive image is generated based on the visible light received by the visible light receiving unit 124.

[0059] Fig. 7 is a flowchart showing operations (processing) up to calculating the distance value of the measurement target object 106 by the three-dimensional measuring device 100 in Example 2 of the first embodiment. Note that the processing contents of steps S1201, S1203, and S1204 in Fig. 7 are similar to the processing contents of steps S1101, S1103, and S1105 in Fig. 2, respectively, and therefore a description thereof will be omitted. Note that each operation (processing) shown in the flowchart in Fig. 7 is controlled by the control unit 101 executing a computer program.

[0060] In step S1202, the control unit 101 sends an imaging start command to the first imaging unit 103 and the second imaging unit 104, and the first imaging unit 103 and the second imaging unit 104 capture an image of a scene in which the pattern light is projected onto the surface of the measurement object 106 from each viewpoint. This imaging process is performed until an imaging end command is sent from the control unit 101. In the second embodiment, as described above, the first imaging unit 103 and the second imaging unit 104 have the configuration shown in FIG. 6, and therefore an active image and a passive image can be acquired simultaneously by imaging in step S1202.

[0061] As an example of the configuration of the first imaging unit 103 and the second imaging unit in the second embodiment, a single sensor having sensitivity to both wavelength bands of visible light and near-infrared light, such as an RGB-IR sensor, may be used as the configuration other than the above. In that case, the spectrometer 120, the visible light cut filter 121, and the near-infrared light cut filter 123 shown in FIG. 6 are not necessary.

[0062] As described above, according to the three-dimensional measuring device 100 of the first embodiment, when calculating the distance value to the measurement object 106, the distance value is calculated using an integrated image generated based on the active image and the passive image. This makes it possible to reduce the amount of calculation required to calculate the distance value compared to calculating the parallax value by using the active image and the passive image independently.

[0063] <Embodiment 2> In the second embodiment, the process of generating an image for matching (third image) performed in the first embodiment is not performed, and an evaluation value is calculated by a method different from that of the first embodiment. That is, in the second embodiment, a method of integrating features derived from an active image and features derived from a passive image into an evaluation value used for matching without generating a new image and calculating a distance value to the measurement target object 106 will be described. Specifically, an evaluation value used for matching is calculated based on an active image (first image) captured by one imaging unit from each active image (first image group) and a passive image (second image) captured by the one imaging unit from each passive image (second image group). According to the method in the second embodiment, a process of generating an image for matching is not required, so that high speed and memory saving can be realized. Furthermore, in the second embodiment, an evaluation value can be calculated according to the characteristics of each feature in the active image and the passive image.

[0064] Incidentally, since the device configuration of the three-dimensional measuring device 100 of the second embodiment is the same as that of the three-dimensional measuring device 100 of the first embodiment, a description of the device configuration of the second embodiment will be omitted, and the following will describe the parts that differ from the first embodiment.

[0065] Example 1 Hereinafter, a method for calculating a distance value to the measurement object 106 using the three-dimensional measuring device 100 in the second embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing operations (processing) up to calculating a distance value to the measurement object 106 in Example 1 of the second embodiment. Note that the processing contents of steps S2111, S2112, S2113, S2115, and S2116 in Fig. 8 are similar to the processing contents of steps S1111, S1112, S1113, S1116, and S1117 in Fig. 4, respectively, and therefore will not be described. Each operation (processing) shown in the flowchart in Fig. 8 is controlled by the control unit 101 executing a computer program.

[0066] 8, an evaluation value is calculated by directly referring to the luminance values ​​of the active image after aberration correction in step S2112 and the passive image whose features have been detected from the image in step S2113. Here, when calculating the evaluation value, the luminance of pixel (x, y) of the active image in the first imaging unit 103 of the second embodiment is defined as shown in Equation 1 below.

number

[0067]

number

[0068]

number

[0069]

number

[0070]

number

[0071] Example 2 Example 2 of the second embodiment differs from Example 1 of the second embodiment in that a method is used in which window sizes are set individually when calculating evaluation values ​​for an active image and a passive image in step S2114 of FIG. 8. For example, the window size is appropriately changed according to the density of the projection pattern in the active image and the size of features such as the texture of the image in the passive image. In this way, the accuracy of the correspondence is improved by individually setting the range in which the correspondence is performed according to the characteristics of each active image and each passive image. Here, when SAD is used as the evaluation formula, the evaluation value in the active image is calculated by SAD. active The evaluation value in the passive image is expressed as SAD. passive Then, the integrated evaluation value Cost is expressed by the following formulas (4) to (6).

number

[0072] Here, the evaluation value SAD in the active image active represents the evaluation value of the features derived from the projection pattern. passive represents the evaluation value of the features derived from the contour and texture of the measurement object 106. k is the ratio for integrating them and is selected in the range of 0≦k≦1. The evaluation value SAD in the active image active The window size for calculating the SAD score in passive images is xa,ya. passiveThe window sizes xp and yp for calculating are set to different values ​​and calculation is performed. Then, the calculation of the subsequent steps is performed based on the integrated evaluation value, and the distance value to the measurement object 106 is calculated.

[0073] Example 3 Example 3 of the second embodiment differs from Examples 1 and 2 of the second embodiment in that different evaluation scales are used for the active image and the passive image, which are suited to the characteristics of the features of each image. That is, in Example 3 of the third embodiment, evaluation values ​​are calculated using different evaluation value calculation methods suited to the characteristics of each active image and each passive image. For example, in an active image, the brightness and contrast of the pattern on the image may differ even if the corresponding positional relationship is correct, depending on the intensity of the projected pattern, the material and the attitude of the measurement target. In such a case, a method of evaluating a value normalized by brightness (for example, ZNCC, etc.) is effective.

[0074] On the other hand, in passive images, the brightness and contrast of features originating from the measurement target may not change much even when the corresponding positional relationship is correct. In such cases, a method for evaluating the absolute value of the brightness difference (e.g., SAD, etc.) is effective. In such cases, the evaluation value in the active image is compared with ZNCC active The evaluation value SAD in passive images is passive Then, the calculated evaluation value Cost is expressed by the following formula (7).

number

[0075] As described above, by using the methods of the examples of the second embodiment, the features derived from the active image and the features derived from the passive image can be integrated into an evaluation value used for matching without generating a new image, and the distance value to the measurement object 106 can be calculated. As a result, the three-dimensional measuring device 100 of the second embodiment does not require processing to generate an image, and therefore can achieve higher speeds and memory savings than the three-dimensional measuring device 100 of the first embodiment.

[0076] <Example of article manufacturing method> The above-mentioned three-dimensional measuring device 100 can be used in a state where it is supported by a certain support member. In this embodiment, as an example, a control system that is attached to a robot arm 200 (gripping device) and used as shown in FIG. 9 will be described. Although not shown in FIG. 9, the three-dimensional measuring device 100 has a first imaging unit 103 and a second imaging unit 104, and uses the first imaging unit 103 and the second imaging unit 104 to capture an image of an object W placed on a support table T in the same manner as in each of the above-mentioned embodiments, and acquires an image (image data). Then, the control unit 101 of the three-dimensional measuring device 100, or the arm control unit 210 that acquires the image output from the control unit 101 of the three-dimensional measuring device 100, determines the position and posture of the object W. The arm control unit 210 sends a drive command to the robot arm 200 to control the robot arm 200 based on the information on the position and posture (measurement result) and the information on the distance value of the object W calculated by the three-dimensional measuring device 100 of each of the above-mentioned embodiments.

[0077] The robot arm 200 holds the object W with a robot hand (gripping unit) at the tip based on the distance value of the object W calculated by the three-dimensional measuring device 100 of each of the above-mentioned embodiments, and moves the object W in a translational or rotational manner (moving step). Furthermore, by assembling the object W to other parts using the robot arm 200 (performing an assembly process), a predetermined item composed of a plurality of parts, such as an electronic circuit board or a machine, can be manufactured. Furthermore, by processing (treating) the moved object W, an item can be manufactured. The arm control unit 210 has a calculation unit such as a CPU as a computer, and a storage device such as a memory in which a computer program is stored. A control unit for controlling the robot may be provided outside the arm control unit 210.

[0078] Furthermore, the measurement data measured by the three-dimensional measuring device 100 and the images obtained may be displayed on a display unit 220 such as a display. Furthermore, the object W may be grasped and moved in order to align it with other parts. The object W has a similar configuration to the measurement target 106 exemplified in FIG. 1. Furthermore, the measuring device may be configured to include a robot arm 200.

[0079] In each of the above-described embodiments, a pattern light is projected onto the measurement object 106 and an image of the measurement object 106 is captured. However, the present invention can also be applied to a case where a pattern light is not projected.

[0080] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0081] The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. The device is, for example, a three-dimensional measuring device 100. In this case, the program and the storage medium storing the program constitute the present invention. The present invention can also be realized by a circuit (for example, an ASIC) that realizes one or more functions. [Explanation of symbols]

[0082] 100 3D Measuring Device 101 Control section 102 Projection section 103 First imaging unit 104 Second imaging unit 105 Processing section 106 Measurement Object

Claims

1. A projection unit that projects a pattern onto an object, a plurality of imaging units that image the object from different viewpoints, and a processing unit that calculates the distance to the object, and the processing unit uses information based on a first image group obtained by projecting and imaging the pattern on the object and information based on a second image group obtained by imaging the object without projecting the pattern, and calculates the distance using an integrated image or an evaluation value in which the features on the image in each image group are retained. A three-dimensional measurement device characterized by this.

2. The processing unit generates a third image as the integrated image based on a first image included in the first image group imaged by one of the imaging units and a second image included in the second image group. The three-dimensional measurement device according to claim 1, characterized by this.

3. The processing unit generates the third image by weighting the features in the first image and the features in the second image. The three-dimensional measurement device according to claim 2, characterized by this.

4. The processing unit generates a feature extraction image by extracting the features from the first image and the second image, and determines a coefficient for performing the weighting based on the luminance value in the feature extraction image. The three-dimensional measurement device according to claim 3, characterized by this.

5. The processing unit generates the third image so that the pattern projected onto the surface of the object, which is the feature in the first image, and the three-dimensional structure or texture on the surface of the object, which is the feature in the second image, are retained. The three-dimensional measurement device according to claim 2, characterized by this.

6. The processing unit performs at least one of smoothing processing, image distortion correction, and image edge extraction processing in accordance with the characteristics of each image in the first image group and the second image group, and generates the third image. The three-dimensional measurement device according to claim 2, characterized by this.

7. The processing unit calculates the evaluation value using both a first image included in the first image group imaged by one of the imaging units and a second image included in the second image group. The three-dimensional measurement device according to claim 1, characterized by this.

8. The processing unit performs individual processing in accordance with the characteristics of each image in the first image group and the second image group, and calculates the evaluation value. The three-dimensional measurement device according to claim 7, characterized by this.

9. The three-dimensional measurement device according to claim 8, wherein the processing includes at least one of a smoothing process, an image distortion correction, and an image edge extraction process.

10. The three-dimensional measurement device according to claim 7, wherein the processing unit individually sets a range for performing the association according to the characteristics of each image of the first image group and the second image group, and calculates the evaluation value.

11. The three-dimensional measurement device according to claim 7, wherein the processing unit calculates the evaluation value using different methods for calculating the evaluation value according to the characteristics of each image of the first image group and the second image group.

12. A projection step of projecting a pattern onto an object, An imaging step of imaging the object from different viewpoints, And a processing step of calculating a distance to the object, In the processing step, The distance is calculated using an integrated image or an evaluation value in which information based on a first image group obtained by projecting and imaging the pattern on the object and information based on a second image group obtained by imaging the object without projecting the pattern are integrated so that features on the image in each image group remain. A three-dimensional measurement method characterized by this.

13. A program for causing a computer to execute a processing method for calculating a distance to an object, the processing method Having a processing step of calculating a distance to the object, In the processing step, the distance is calculated using an integrated image or an evaluation value in which information based on a first image group obtained by projecting and imaging a pattern on the object and information based on a second image group obtained by imaging the object without projecting the pattern are integrated so that features on the image in each image group remain. A program characterized by this.

14. A system comprising the three-dimensional measurement device according to claim 1, And a robot that grips and moves the object based on the distance of the object calculated by the three-dimensional measurement device. A system characterized by this.

15. A moving step of gripping and moving the object based on the distance of the object calculated by the three-dimensional measurement device according to claim 1, And a step of manufacturing a predetermined article by processing the object. A method for manufacturing an article characterized by this.

Images