Three-dimensional measuring device and method

Abstract

This three-dimensional measuring device comprises: a first measuring unit that acquires two-dimensional image data relating to an object; a second measuring unit that acquires first three-dimensional data relating to the object by means of a phase shift method; a third measuring unit that acquires second three-dimensional data relating to the object by means of a light sectioning method; and a processing unit that generates three-dimensional information relating to the object on the basis of the data acquired by the first measuring unit, the second measuring unit and the third measuring unit. The processing unit extracts a specific region of the object on the basis of the two-dimensional image data, and generates one piece of three-dimensional information relating to the object by performing synthesis processing for fitting one of the first three-dimensional data or the second three-dimensional data to the specific region and fitting the other to a remaining region excluding the specific area.

Description

Three-dimensional measuring device and method

[0001] The present invention relates to a three-dimensional measuring device and a three-dimensional measuring method for acquiring three-dimensional information of an object.

[0002] Phase shifting and light sectioning methods are known as methods for non-contact three-dimensional measurement of an object. The phase shifting method involves capturing an image of an object onto which stripe pattern light having different phases is sequentially projected, and obtaining three-dimensional data of the object from phase information extracted from the captured image. The light sectioning method involves scanning an object illuminated with line light, and obtaining three-dimensional data from the captured image based on the principle of triangulation. Patent Document 1 discloses a three-dimensional measuring device that combines the phase shifting and light sectioning methods.

[0003] The device in Patent Document 1 acquires three-dimensional data of an object using both the phase shift method and the light section method, and also acquires reliability information indicating the reliability of each piece of three-dimensional data.The device then generates a single piece of three-dimensional information about the object by complementing the less reliable parts of one piece of three-dimensional data with the other piece of three-dimensional data.

[0004] However, it has been found that relying on reliability information to synthesize three-dimensional data may result in inaccurate three-dimensional measurement of the object. For example, if the object being measured includes a transparent portion on its surface, the phase shift method calculates the height based on the light reflected from the bottom surface of the transparent portion. The calculated height data is highly reliable because it faithfully represents the height of the bottom surface, but does not represent the actual surface height of the transparent portion. When relying on reliability information, three-dimensional data acquired by the phase shift method may be used for the transparent portion. As a result, a problem may occur in which the accurate shape of the object cannot be measured.

[0005] International Publication No. 2020-065850

[0006] An object of the present invention is to provide a three-dimensional measuring device and method that can acquire three-dimensional information of an object with high accuracy.

[0007] A three-dimensional measuring device according to one aspect of the present invention comprises a first measuring unit that acquires two-dimensional image data of an object, a second measuring unit that acquires first three-dimensional data of the object by a phase shift method, a third measuring unit that acquires second three-dimensional data of the object by a light-section method, and a processing unit that generates three-dimensional information of the object based on the data acquired by the first measuring unit, the second measuring unit, and the third measuring unit, wherein the processing unit extracts a specific region of the object based on the two-dimensional image data, and generates one piece of three-dimensional information about the object by performing a synthesis process in which either the first three-dimensional data or the second three-dimensional data is fitted to the specific region and the other is fitted to the remaining region other than the specific region.

[0008] A three-dimensional measurement method according to another aspect of the present invention includes acquiring two-dimensional image data of an object, acquiring first three-dimensional data of the object using a phase shift method, acquiring second three-dimensional data of the object using a light section method, extracting a specific region of the object based on the two-dimensional image data, and performing a synthesis process in which either the first three-dimensional data or the second three-dimensional data is fitted to the specific region and the other is fitted to the remaining region other than the specific region, thereby generating a single piece of three-dimensional information about the object.

[0009] FIG. 1 is a block diagram showing the configuration of a three-dimensional measuring device according to an embodiment of the present invention. FIG. 2 is a cross-sectional view showing a measuring head unit provided in the three-dimensional measuring device, showing the configuration of a first measurement unit that acquires two-dimensional image data. FIG. 3 is a diagram showing the configuration of a second measurement unit that employs a phase shift method. FIG. 4 is a diagram showing the configuration of a third measurement unit that employs a light-section method. FIG. 5 is a diagram showing an example of an image acquired of a component having a mirrored surface. FIGS. 6A and 6B are graphs showing profiles of three-dimensional data acquired by the second and third measurement units for a component having a mirrored surface. FIG. 7 is a diagram showing an example of an image acquired of a component having a transparent portion. FIG. 8 is a diagram showing an example of image synthesis for a component having a mirrored surface. FIG. 9 is a diagram showing an example of image synthesis for a component having a transparent portion. FIG. 10 is a diagram showing the configuration of a mounted substrate production line to which the three-dimensional measuring device according to this embodiment is applied. FIG. 11 is a flowchart showing an example of a substrate inspection process performed by a substrate inspection machine in the mounted substrate production line. FIG. 12 is a flowchart showing an example of image synthesis process.

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The three-dimensional measuring device according to the present invention can be widely used to obtain three-dimensional information, such as height and shape recognition, of various measurement targets, including industrial products, semi-finished products, machine parts, electronic parts, food products, and agricultural products. For example, a suitable application for the three-dimensional measuring device of the present invention is to measure components mounted on a circuit board to inspect the mounting state of the components.

[0011] [Device Configuration] Fig. 1 is a block diagram showing the configuration of a three-dimensional measuring device MA according to an embodiment of the present invention. The three-dimensional measuring device MA includes a measuring head unit 2 that performs measurement operations on an object, and a control unit 6 that controls the operation of the measuring head unit 2. The measuring head unit 2 includes a first measurement unit 3, a second measurement unit 4, and a third measurement unit 5. The measuring head unit 2 is movable in the X, Y, and Z directions by a head movement mechanism 201. Fig. 1 illustrates a substrate PB on which electronic components are mounted as the measurement object. The substrate PB is loaded onto a conveyor 202 and transported to the measurement stage of the three-dimensional measuring device MA, and is then transported out after measurement.

[0012] <Regarding the Measurement Unit> The first measurement unit 3 acquires two-dimensional image data of the substrate PB. The second measurement unit 4 acquires first three-dimensional data about the substrate PB using a phase shift method. The third measurement unit 5 acquires second three-dimensional data about the substrate PB using an optical section method. The measurement head unit 2 includes a frame driven by a head movement mechanism 201. The first measurement unit 3, second measurement unit 4, and third measurement unit 5 are mounted on the frame. In this embodiment, an example is shown in which the first measurement unit 3 and the second measurement unit 4 are integrated into a single combined head 2U. The combined head 2U includes a first camera 21. The first camera 21 is shared by the first measurement unit 3 and the second measurement unit 4. The third measurement unit 5 includes a laser light source 51 and a second camera 52. Of course, the first measurement unit 3 and the second measurement unit 4 may be mounted on the frame as separate units.

[0013] 2 is a cross-sectional view showing the combined head 2U of the measurement head unit 2. The first camera 21 includes a camera body 22 and a lens barrel 23. The camera body 22 includes an image sensor 24 that captures images. The image sensor 24 is a sensor in which pixels made of photoelectric conversion elements are arranged in a matrix. A CMOS sensor, for example, can be used as the image sensor 24. The lens barrel 23 includes multiple optical lenses that form an optical image of the measurement object on the light receiving surface of the image sensor 24. A half mirror 25 that forms the measurement optical path of the first measurement unit 3 is arranged on the optical path within the lens barrel 23.

[0014] The first measurement unit 3 includes a coaxial illumination unit 31 and a multidirectional illumination unit 32 as illumination systems, and a camera body 22 as an imaging system. The coaxial illumination unit 31 irradiates a substrate PB as a measurement object with coaxial light. The coaxial illumination unit 31 includes an LED light-emitting unit and is attached to the side surface of the lens barrel 23. The coaxial illumination light L11 emitted from the coaxial illumination unit 31 is reflected by the half mirror 25 and incident on the substrate PB along the imaging optical axis of the first camera 21. The reflected light RL from the substrate PB enters the camera body 22 through the lens barrel 23 and is received by the imaging sensor 24. By using the coaxial illumination light L11 irradiated coaxially with the imaging optical axis, sufficient reflected light RL can be incident on the camera body 22 from the mirror surface that specularly reflects the illumination light.

[0015] The multi-directional illumination unit 32 includes an upper illumination unit 34, a middle illumination unit 35, and a lower illumination unit 36, all of which are composed of LED light-emitting units. The upper illumination unit 34 is attached to the lower end of the first camera 21 and irradiates the board PB with upper illumination light L12 at an illumination angle close to the vertical direction. The upper illumination light L12 is, for example, white light or a combination of white light and infrared light. Reflected light RL of the upper illumination light L12 along the imaging optical axis is also received by the imaging sensor 24.

[0016] The middle-level illumination unit 35 irradiates the substrate PB with middle-level illumination light L13 at an illumination angle that is more inclined with respect to the vertical direction than the upper-level illumination light L12. The middle-level illumination light L13 is, for example, white light. A dome reflector 331 with a hemispherical reflective surface is attached to the lower end of the first camera 21. The middle-level illumination unit 35 is disposed facing upward, and the middle-level illumination light L13 is reflected by the dome reflector 331 and irradiated onto the substrate PB.

[0017] The lower illumination unit 36 ​​irradiates the substrate PB with lower illumination light L14 at an illumination angle that is more tilted with respect to the vertical direction than the middle illumination light L13. The lower illumination light L14 is, for example, white light. A holding dish 332 having a larger diameter than the dome reflector 331 is attached to the lower end of the dome reflector 331. The lower illumination unit 36 ​​is attached to the holding dish 332 at a predetermined inclination. Reflected light RL of the middle illumination light L13 and the lower illumination light L14 along the imaging optical axis is also received by the imaging sensor 24.

[0018] The camera body 22 of the first camera 21 functions as a camera for capturing a two-dimensional image of the substrate PB in the first measurement unit 3, which acquires two-dimensional image data. When capturing a two-dimensional image by the first measurement unit 3, one or more of the coaxial illumination unit 31, upper illumination unit 34, middle illumination unit 35, and lower illumination unit 36 ​​are selected. Based on component data, etc., if a component including a mirror surface is within the imaging field of the first camera 21, at least the coaxial illumination unit 31 is turned on. As in this embodiment, by irradiating the coaxial illumination light L11, upper illumination light L12, middle illumination light L13, and lower illumination light L14, the substrate PB can be illuminated from multiple angles. Therefore, a clear two-dimensional image of the substrate PB can be captured.

[0019] FIG. 3 is a diagram showing the configuration of the second measurement unit 4 that employs the phase shift method. In FIG. 3, the configuration related to the first measurement unit 3 is omitted from the combined head 2U shown in FIG. 2. The second measurement unit 4 includes a plurality of projectors 41 as an illumination system and a camera body 22 as an imaging system. The projectors 41 are arranged at a predetermined inclination around the imaging optical axis of the first camera 21. In other words, the projection axes of the projectors 41 are inclined at a predetermined angle with respect to the imaging optical axis. For example, four or eight projectors 41 are arranged at equal distances and evenly spaced intervals in the circumferential direction surrounding the imaging optical axis.

[0020] In the phase shift method, an image is captured while changing the phase of light irradiating the object, thereby obtaining three-dimensional data of the surface of the object. A projector 41 irradiates the substrate PB with pattern light L2, such as a sine wave pattern or a stripe pattern. Reflected light RL1 of the pattern light L2 along the imaging optical axis is incident on the first camera 21. An imaging sensor 24 of the camera body 22 receives the reflected light RL1.

[0021] The projector 41 emits pattern light L2 with different phases per field of view, for example, four times. The first camera 21 captures an image each time the pattern light L2 is emitted. Based on the brightness changes in the four acquired images, the phase change resulting from the surface shape of the substrate PB is analyzed. Based on the analysis results, three-dimensional data such as the height of components on the substrate PB is obtained. Note that if there are high-reflectivity portions, such as mirror surfaces, on the surface of the substrate PB, the pattern light L2 will be dominated by reflected light RL2 that is regularly reflected by the high-reflectivity portions. Therefore, measurement accuracy tends to decrease due to insufficient reflected light RL1 entering the first camera 21.

[0022] 4 is a diagram showing the configuration of the third measurement unit 5 employing the light-section method. The third measurement unit 5 includes a laser light source 51 and a second camera 52. The laser light source 51 is a light source that generates slit-shaped light and irradiates the substrate surface PBS with a line of light L3. The second camera 52 captures reflected light RL31, RL32 of the line of light L3. The angle between the projection axis of the laser light source 51 and the vertical axis Q is θ, and the angle between the imaging optical axis of the second camera 52 and the vertical axis Q is also θ.

[0023] The laser light source 51 is a light source that generates slit-shaped light and irradiates the workpiece W to be measured with a line light L3 for three-dimensional measurement. For example, the laser light source 51 may be a light source device including a laser light source and an optical component that converts the laser light emitted by the laser light source into slit light that spreads in a fan shape. The second camera 52 includes a camera body 53 and a lens barrel 54. The camera body 53 is equipped with an image sensor 55 that receives the reflected light beams RL31 and RL32. The lens barrel 54 forms an image of the reflected light beams RL31 and RL32 on the light receiving surface of the image sensor 55.

[0024] The third measurement unit 5 scans the board surface PBS on which the component C is mounted with line light L3, while capturing images of reflected light RL31 and RL32 with the second camera 52. From the captured image, three-dimensional data of the component C, specifically the height h of the component C, is obtained. A parallax P occurs on the light receiving surface of the imaging sensor 55 between the reflected light RL31 reflected by the top surface of the component C and the reflected light RL32 reflected by the board surface PBS, which serves as the reference plane for height. Using the parallax P, the height h of the component C can be calculated using the formula h = P / 2 sin θ.

[0025] The light-section method captures reflected light RL31 and RL32, which are specular reflections of the line light L3, and therefore has the advantage of enabling accurate height measurement of high-reflectivity areas. However, the light-section method requires one second camera 52 per laser light source 51, and it is not possible to share one camera with multiple projectors 41, as in the second measurement unit 4. Therefore, installing multiple third measurement units 5 on the measurement head unit 2 is difficult in terms of layout and increases equipment costs. The light-section method also has the problem of taking a long time to measure. The inability to perform light-section measurement from multiple angles leads to the occurrence of blind spots in the measurement. In consideration of the above circumstances, this embodiment utilizes the advantages of the phase shift method and the light-section method to accurately measure three-dimensional information about the target object.

[0026] <Regarding the Control Unit> Returning to Figure 1, the control unit 6 is composed of a processor and the like that operates by loading a predetermined program, and functionally includes an axis control unit 61, an imaging control unit 62, a measurement processing unit 63, a memory unit 64, and an overall control unit 65. The axis control unit 61 controls a head movement mechanism 201 to move the measuring head unit 2 in the X, Y, and Z directions. The head movement mechanism 201 has an X-axis drive motor, a Y-axis drive motor, and a Z-axis drive motor for moving the measuring head unit 2. The axis control unit 61 controls these drive motors to move the measuring head unit 2 to the imaging positions of the first measurement unit 3, the second measurement unit 4, and the third measurement unit 5.

[0027] The imaging control unit 62 controls the first measurement unit 3, second measurement unit 4, and third measurement unit 5 mounted on the measurement head unit 2 to acquire two-dimensional image data of the substrate PB and three-dimensional data by the phase shift method and the light section method. The imaging control unit 62 includes a projection control unit 621, a laser illumination control unit 622, a 2D illumination control unit 623, a first camera control unit 624, and a second camera control unit 625.

[0028] The projection control unit 621 controls the projector 41 of the second measurement unit 4 at the timing of measurement using the phase shift method to irradiate the measurement area on the substrate PB with pattern light L2. The laser illumination control unit 622 controls the laser light source 51 of the third measurement unit 5 at the timing of measurement using the light section method to irradiate the substrate PB with line light L3. The 2D illumination control unit 623 controls the coaxial illumination unit 31 and multidirectional illumination unit 32 of the first measurement unit 3 at the timing of acquiring a two-dimensional image to irradiate the substrate PB with at least one of the coaxial illumination light L11, upper illumination light L12, middle illumination light L13, and lower illumination light L14.

[0029] The first camera control unit 624 controls the imaging operation of the first camera 21 of the combination head 2U. The first camera control unit 624 causes the first camera 21 to capture reflected light RL of illumination light L11-L14 irradiated onto the substrate PB at the timing of acquiring a two-dimensional image (FIG. 2). The first camera control unit 624 also causes the first camera 21 to capture reflected light RL1 of pattern light L2 irradiated onto the substrate PB at the timing of measurement using the phase shift method (FIG. 3). The second camera control unit 625 causes the second camera 21 to capture reflected light RL31, RL32 of line light L3 irradiated onto the substrate PB at the timing of measurement using the light section method (FIG. 4).

[0030] The measurement processing unit 63 performs processing to generate three-dimensional information of the substrate PB as the target object based on the data acquired by the first measurement unit 3, the second measurement unit 4, and the third measurement unit 5. The measurement processing unit 63 performs roughly the following processing. First, a specific region of the substrate PB is extracted within the imaging field of the first camera 21 based on the two-dimensional image data acquired by the first measurement unit 3. Next, a synthesis process is performed on the first three-dimensional data acquired by the second measurement unit 4 and the second three-dimensional data acquired by the third measurement unit 5. That is, either the first three-dimensional data or the second three-dimensional data is fitted to the specific region, and the other is fitted to the remaining region other than the specific region. By performing this synthesis process, one piece of three-dimensional information is generated for the substrate PB within the current imaging field of view.

[0031] The specific region is a region where the measurement accuracy of three-dimensional data is low by either the second measurement unit 4 using the phase shift method or the third measurement unit 5 using the light-section method, but measurement accuracy can be ensured by the other. In the specific examples described below, examples where the specific region is a mirrored portion of a component ( FIGS. 5 and 8 ) and examples where the specific region is a transparent portion of a component ( FIGS. 7 and 9 ) are shown.

[0032] The measurement processing unit 63 functionally includes an image processing unit 631, an area extraction unit 632, a height calculation unit 633, and a synthesis processing unit (processing unit) 634. The image processing unit 631 performs necessary image processing such as contrast and brightness correction processing, noise removal processing, etc. on the two-dimensional image data and three-dimensional data of the substrate PB acquired by the measurement head unit 2.

[0033] The region extraction unit 632 performs processing to extract a specific region included in the two-dimensional image based on the two-dimensional image data. The region extraction unit 632 extracts the specific region, for example, by performing processing to binarize the brightness values ​​of the pixels that make up the two-dimensional image. When a component including a mirrored surface is irradiated with coaxial illumination light L11, pixels that receive the light reflected from the mirrored surface become highly luminous. Therefore, by setting an appropriate threshold, it is possible to distinguish the highly luminous region as the specific region and the rest as the remaining region. Note that instead of the binarization processing, the specific region may be extracted by, for example, edge extraction processing.

[0034] The height calculation unit 633 calculates the height of the substrate PB within the imaging field of view based on the three-dimensional data acquired by the second measurement unit 4 and the third measurement unit 5. For the three-dimensional data from the second measurement unit 4, the height calculation unit 633 determines the height of the substrate PB by performing phase analysis on multiple images illuminated with pattern light L2 of different phases. For the three-dimensional data from the third measurement unit 5, the height calculation unit 633 determines the height of the substrate PB based on the parallax P on the light-receiving surface between reflected light RL31 reflected by the top surface of the component C and reflected light RL32 reflected by the substrate surface PBS.

[0035] The synthesis processing unit 634 synthesizes the three-dimensional data acquired by the second measurement unit 4 using the phase shift method and the third measurement unit 5 using the light-section method to generate a single piece of three-dimensional information. In the synthesis process of this embodiment, the first three-dimensional data acquired by the second measurement unit 4 is treated as base data. Synthesis is performed by fitting the second three-dimensional data acquired by the third measurement unit 5 into a specific region of this base data. The reason for this is as follows.

[0036] First, the second measurement unit 4 includes four to eight projectors 41 arranged around the imaging optical axis of the first camera 21. Therefore, blind spots that occur when lighting is provided by each individual projector 41 can be complemented by the lighting provided by the other projectors 41. In contrast, in the third measurement unit 5, it is difficult in terms of layout to mount a set of a laser light source 51 and a second camera 52 on the measurement head unit 2 so that no blind spots are created. Therefore, in the three-dimensional measuring device MA of this embodiment, the first three-dimensional data obtained by the phase shift method can be said to have an advantage in terms of basic reliability. This is the reason why the first three-dimensional data is treated as base data.

[0037] On the other hand, the phase shift method tends to have lower measurement accuracy in specific areas such as mirrored or transparent areas. In contrast, the second three-dimensional data obtained by the light-section method is obtained by detecting specularly reflected light reflected from the surface of a component on the substrate PB, regardless of the presence of mirrored or transparent areas, and therefore has high measurement accuracy in specific areas. Therefore, by using the first three-dimensional data as base data and performing a synthesis process in which the second three-dimensional data is fitted to a specific area of ​​the base data, it is possible to generate a single piece of three-dimensional information with high overall accuracy.

[0038] The storage unit 64 stores various data, setting values, etc. necessary for the operation of the three-dimensional measuring device MA. For example, storage unit 64 stores board data such as the size and type of the board PB to be measured, component data such as the type, size, and arrangement of components mounted on the board PB, data linking the components with the lighting to be used, binarization threshold values ​​used by the area extraction unit 632, etc.

[0039] Two-dimensional image data of the substrate PB to be measured, acquired in advance by the first measurement unit 3, may be stored in the memory unit 64. That is, rather than acquiring two-dimensional image data along with three-dimensional data in real time, a mass-production sample substrate, prototype substrate, or the like is imaged in advance by a camera device corresponding to the first measurement unit 3. The captured two-dimensional image data is stored in the memory unit 64. The region extraction unit 632 reads the two-dimensional image data from the memory unit 64 during three-dimensional measurement of the substrate PB and extracts a specific region. According to this aspect, the first measurement unit 3 does not need to acquire two-dimensional image data during three-dimensional measurement of the object. This simplifies the measurement process and improves the takt time.

[0040] If design data for the board PB that specifies the mounting positions of components and the like exists, the design data may be stored in the storage unit 64. The area extraction unit 632 extracts a specific area based on the design data. In this case, it is possible to omit the imaging of the board PB by the first measurement unit 3.

[0041] The overall control unit 65 comprehensively controls the operations of the axis control unit 61, imaging control unit 62, and measurement processing unit 63. The overall control unit 65 controls the axis control unit 61 so that the measuring head unit 2 performs a movement operation in accordance with a predetermined measurement sequence. The overall control unit 65 controls the imaging control unit 62 so that the first measurement unit 3, the second measurement unit 4, and the third measurement unit 5 perform an imaging operation under predetermined lighting. The overall control unit 65 controls the measurement processing unit 63 so that it derives three-dimensional information of the substrate PB based on various data acquired by the measuring head unit 2.

[0042] [Images Detected by Each Measurement Unit] Figure 5 shows an example of an image acquired of a component having a mirrored surface. Figure 5 shows an example in which three components are present in one field of view PB1 for imaging the substrate PB. The three components are a CSP (Chip Size Package) component 71 having a mirrored surface 71M on its upper surface, and a first adjacent component 72 and a second adjacent component 73 adjacent to the left and right of the CSP component 71. The CSP component 71 is a rectangular IC component with ball-shaped electrodes on its lower surface that contacts the substrate PB, and its upper surface is often a mirrored surface. The mirrored surface 71M specularly reflects illumination light.

[0043] The bottom row of Figure 5 shows, from left to right, a coaxial illumination image acquired by irradiating the field of view PB1 with coaxial illumination light L11 in the first measurement unit 3, a phase-shift image acquired by the second measurement unit 4, and a light-section image acquired by the third measurement unit 5. Referring to Figure 2, in the two-dimensional coaxial illumination image acquired by performing imaging while irradiating the field of view PB1 with coaxial illumination light L11, reflected light RL that is specularly reflected from the mirror surface 71M is incident on the first camera 21. On the other hand, in the remaining area other than the mirror surface 71M, the specularly reflected light component of the coaxial illumination light L11 is small. Therefore, in the coaxial illumination image, only the mirror surface 71M is very bright, and the remaining area is dark.

[0044] In contrast, the phase shift image does not allow the mirror surface portion 71M to be detected with high accuracy. Referring to Fig. 3, in the phase shift method, the projection optical axis of the pattern light L2 emitted from the projector 41 has an inclination angle with respect to the image capturing optical axis of the first camera 21. On the mirror surface 71M, most of the pattern light L2 is specularly reflected as reflected light RL2. Therefore, the amount of the pattern light L2 reflected by the mirror surface 71M that enters the first camera 21 is small.

[0045] FIG. 6A is a graph showing a profile of three-dimensional data for the mirrored surface 71M, created based on a phase-shifted image acquired by the second measurement unit 4. Although the mirrored surface 71M is a flat surface with a constant height, unevenness appears in the height data. As described above, this measurement error is caused by the fundamental insufficient and unstable amount of reflected light of the patterned light L2 from the mirrored surface 71M. On the other hand, for the remaining area other than the mirrored surface 71M, the patterned light L2 is irradiated from multiple projectors 41 arranged in a polygonal fashion, enabling imaging without blind spots. Therefore, highly accurate three-dimensional data can be acquired for the remaining area, in which the first adjacent component 72 and the second adjacent component 73 are reflected.

[0046] Conversely, the light-section image allows for highly accurate detection of the mirror surface 71M. Referring to FIG. 4, in the light-section method, the second camera 52 detects the specularly reflected light RL31 and RL32 of the line light L3. Therefore, the light-section method allows for highly accurate three-dimensional data to be acquired for the mirror surface 71M, where specular reflection occurs. FIG. 6B is a graph showing a profile of three-dimensional data for the mirror surface 71M, created based on the light-section image acquired by the third measurement unit 5. This graph shows that the mirror surface 71M, which is a flat surface with a constant height, is measured with high accuracy.

[0047] On the other hand, since it is difficult to set up a layout in the measurement head unit 2 that irradiates the line light L3 from multiple angles, the light section image may include data of areas that are measurement blind spots. The light section image shown in Figure 5 shows blind spots NG1 and NG2. The blind spot NG1 is an area where the reflected light RL31 and RL32 are not detected because the line light L3 is blocked by the second adjacent component 73. The blind spot NG2 is an area where abnormally bright reflected light is detected due to multiple reflections of the line light L3 in the narrow area between the first adjacent component 72 and the CSP component 71.

[0048] FIG. 7 shows an example of an image acquired of a component having a transparent portion. FIG. 7 illustrates an example in which one LED component 74 is present in one field of view PB2 for imaging the substrate PB. The LED component 74 has a transparent portion 74T that serves as the light-emitting portion. The transparent portion 74T is surrounded by a holding portion 741 made of an opaque material. The bottom row of FIG. 7 shows, from left to right, a 2D image acquired by irradiating the field of view PB2 with the required illumination light in the first measurement unit 3, a phase-shift image acquired by the second measurement unit 4, and a light-section image acquired by the third measurement unit 5. The phase-shift image and the light-section image show cross sections to show the reflection of the illumination light at the transparent portion 74T, and graphs showing the three-dimensional data profiles P1 and P2 at the cross sections are also provided.

[0049] In the 2D image, the optical image of the LED component 74 is clearly captured. That is, the outline of the LED component 74 and the shapes of the transparent portion 74T and the holder 741 are clearly captured. Regarding the illumination light, the surface of the transparent portion 74T is not necessarily a mirror finish. Therefore, the field of view PB2 may be illuminated using only the multi-directional illumination unit 32 without using the coaxial illumination light L11.

[0050] The phase-shift image does not allow accurate measurement of the height of the transparent portion 74T. The second measurement unit 4 detects reflected light RL1 that is generated when the pattern light L2, which is irradiated obliquely by the projector 41 onto the LED component 74, is reflected by the bottom surface 742 of the transparent portion 74T and travels vertically. In other words, the pattern light L2 passes through the transparent portion 74T, reaches the bottom surface 742, and is reflected there, resulting in the detection of reflected light RL1. The reflected light RL2 that is specularly reflected by the surface of the transparent portion 74T does not enter the first camera 21. Outside the transparent portion 74T, the pattern light L2 is diffusely reflected by the surface 74S.

[0051] Therefore, in the profile P1 of the phase-shift image, the surface 74S shows its true height P11, and the graph shows a depression P12 in the transparent portion 74T area. Therefore, relying on the phase-shift image would result in an incorrect height measurement of the transparent portion 74T of the LED component 74. On the other hand, the depression P12 can be considered the result of a faithful measurement of the height of the bottom surface 742. If the measurement relies on reliability information, as with the measurement device of Patent Document 1, the measurement result of the depression P12 would also be determined to be highly reliable. Therefore, conventional measurement devices may output incorrect three-dimensional information for LED components 74 having transparent portions 74T.

[0052] The light section image allows accurate measurement of the height of the transparent portion 74T. In the light section method, reflected light RL3, which is the line light L3 specularly reflected by the surface of the transparent portion 74T, is detected. The reflected light from the bottom surface 742 contains few specularly reflected components, and is therefore hardly detected. Therefore, the profile P2 of the light section image allows accurate measurement of the height of not only the surface 74S but also the surface of the transparent portion 74T.

[0053] 8 is a diagram showing an example of image synthesis for a component having a mirrored surface 71M. First, based on the coaxial illumination image of the field of view PB1, a process is performed to divide the image into a specific area AR1 corresponding to the mirrored surface 71M and a remaining area AR2 other than the specific area AR1. As described above, very bright reflected light RL reflected by the mirrored surface 71M enters the first camera 21, while relatively dark reflected light enters from other areas. Therefore, the luminance values ​​detected by each pixel of the image sensor 24 are binarized using a predetermined threshold value to create a binarized image, thereby extracting the specific area AR1 from the field of view PB1.

[0054] The phase-shifted image is a highly accurate image except for the mirrored portion 71M, i.e., the specific region AR1. Therefore, the phase-shifted image is treated as base data, and the phase-shifted image is used for the remaining region AR2. On the other hand, the light-section image is a highly accurate image of the mirrored portion 71M. Therefore, an image of the portion corresponding to the specific region AR1 is cut out from the light-section image. Then, the highly accurate portions of the phase-shifted image and the light-section image are combined to create a single composite image. That is, the image of the portion corresponding to the specific region AR1 cut out from the light-section image is combined with the specific region AR1 of the phase-shifted image of the base data. This combination process generates a composite image in which the phase-shifted image is fitted to the specific region AR1 and the phase-shifted image is fitted to the remaining region AR2. Based on this composite image, three-dimensional information can be measured with high accuracy for a part including the mirrored portion 71M.

[0055] FIG. 9 shows an example of image synthesis for a component having a transparent portion 74T. Based on a 2D image of the field of view PB2, a process is performed to divide the field of view PB2 into a specific area AR1 corresponding to the transparent portion 74T and a remaining area AR2 other than the specific area AR1. The transparent portion 74T also appears bright in the 2D image. The specific area AR1 can be extracted from the field of view PB2 by creating a binarized image by binarizing the brightness values ​​detected by each pixel of the image sensor 24 using a predetermined threshold value. Note that it may be difficult to identify the transparent portion 74T using simple binarization processing. In this case, the position of the transparent portion 74T is recognized based on the component data, and a fixed area encompassing the position and its surroundings is set as a search area. The transparent portion 74T may then be detected within the search area based on pixel values.

[0056] Because the phase-shifted image exclusively detects the bottom surface 742 of the transparent portion 74T, it is unable to detect the height that should be detected in the specific region AR1, while providing a highly accurate image for the remaining region AR2. On the other hand, because the light-section image can detect the surface of the transparent portion 74T, it provides a highly accurate image for the specific region AR1. Therefore, an image of the portion corresponding to the specific region AR1 is cut out from the light-section image. Then, the image of the portion corresponding to the specific region AR1 cut out from the light-section image is combined with the specific region AR1 of the phase-shifted image. This combination process generates a combined image in which the phase-shifted image is fitted to the specific region AR1 and the phase-shifted image is fitted to the remaining region AR2. Based on this combined image, three-dimensional information can be measured with high accuracy for a component including the transparent portion 74T.

[0057] [Example of Application to a Mounted Board Production Line] Figure 10 is a block diagram showing the configuration of a mounted board production line 1 to which the three-dimensional measuring device MA of this embodiment is applied. The mounted board production line 1 is a line for mounting electronic components and the like on printed circuit boards, and includes a board transport path TR, multiple operating devices, and a line control device 10. The board transport path TR transports boards on which electronic components are mounted from right to left in Figure 10. The multiple operating devices are arranged on the board transport path TR and perform predetermined operations on the boards. The line control device 10 comprehensively controls the multiple operating devices.

[0058] 10 shows the multiple working devices arranged in tandem from upstream to downstream in the board transport direction: a printer 11, a print inspection machine 12, a mounting machine 13, a board inspection machine 14, a reflow oven 15, and an appearance inspection machine 16. A loader 17 that carries boards into the printer 11 is located at the upstream end of the mounted board production line 1, and an unloader 18 that removes produced boards from the appearance inspection machine 16 is located at the downstream end.

[0059] The printer 11 applies solder to the pads of the printed circuit board. The print inspection machine 12 takes an image of the printed circuit board to which the solder has been applied and inspects whether the position, amount, and height of the solder are appropriate. The mounter 13 is equipped with a component mounting head and mounts required electronic components on the printed circuit board. Figure 10 shows an example in which three mounters 13 are arranged in series. That is, from the upstream side, a first mounter 13A, a second mounter 13B, and a third mounter 13C are arranged in tandem.

[0060] The board inspection machine 14 takes an image of the printed circuit board that has passed through the mounter 13, and inspects the electronic components mounted on the printed circuit board for misalignment, lead misalignment, component lift, missing mounting, soldering defects, the presence of foreign matter, etc. The three-dimensional measuring device MA of this embodiment is applied to the board inspection machine 14, and measures three-dimensional information of the printed circuit board for the above-mentioned inspections.

[0061] The reflow furnace 15 heats the printed circuit board on which electronic components are mounted to melt the solder and fix the electronic components to the printed circuit board. The visual inspection machine 16 takes an image of the printed circuit board after heat treatment in the reflow furnace 15 and, like the board inspection machine 14, inspects for misalignment of the electronic components, lead misalignment, component lift, missing mounting, soldering defects, and the presence of foreign matter. The three-dimensional measuring device MA of this embodiment can also be applied to this visual inspection machine 16. The line control device 10 controls the production work of mounted boards by comprehensively controlling the work devices included in the mounted board production line 1, which are the printing machine 11, printing inspection machine 12, mounting machine 13, board inspection machine 14, reflow furnace 15, and visual inspection machine 16.

[0062] [Board Inspection Processing] Figure 11 is a flowchart showing an example of board inspection processing performed by a board inspection machine in a mounted board production line. Here, it is assumed that the board inspection machine 14 is equipped with the three-dimensional measuring device MA of this embodiment. The conveyor 202 shown in Figure 1 and elsewhere corresponds to the conveyor for transporting boards installed on the board transport path TR. When the mounted board production line 1 is operating and component mounting on the board PB is completed by the mounter 13, the conveyor carries the board PB to an inspection stage provided in the board inspection machine 14 (step S1).

[0063] An optical section image of the loaded substrate PB is acquired by the third measurement unit 5 (step S2). The axis control unit 61 of the control unit 6 moves the measurement head unit 2 to position the third measurement unit 5 facing the substrate PB. The laser illumination control unit 622 of the imaging control unit 62 then operates the laser light source 51, irradiating the substrate PB with line light L3. The second camera control unit 625 also operates the second camera 52, capturing images of the reflected light RL31 and RL32 of the line light L3. This imaging continues until the line light L3 has scanned the entire area of ​​the substrate PB. Note that scanning with the line light L3 may be limited to a specific region or specific component of the substrate PB.

[0064] Next, the axis control unit 61 moves the measurement head unit 2, and a field of view movement is performed so that the second measurement unit 4 faces the first measurement location on the substrate PB that has been predetermined by the measurement sequence (step S3). Generally, the measurement field of view of the first camera 21 is small compared to the size of the substrate PB, so that images of the substrate PB are taken in multiple divided shots.

[0065] After the field of view is moved, a phase-shifted image is acquired by the second measurement unit 4 (step S4). Specifically, the projection control unit 621 operates the projector 41 to irradiate the measurement location on the substrate PB with patterned light L2. The first camera control unit 624 also operates the first camera 21 to capture an image of reflected light RL1 of the patterned light L2. This image capture is performed multiple times (for example, four times) by changing the phase of the patterned light L2.

[0066] Next, a two-dimensional image is acquired by the first measurement unit 3 (step S5). Specifically, the 2D illumination control unit 623 operates the coaxial illumination unit 31 and the multi-directional illumination unit 32, and at least one of the coaxial illumination light L11, upper illumination light L12, middle illumination light L13, and lower illumination light L14 is irradiated onto the measurement location on the substrate PB. The first camera control unit 624 operates the first camera 21 to capture the reflected light RL of the illumination lights L11 to L14. Note that step S5 may be performed prior to step S4.

[0067] Next, the synthesis processing unit 634 of the measurement processing unit 63 executes an image synthesis process to synthesize the phase shift image and the light section image (step S6). The height calculation unit 633 calculates height data of the substrate PB within the current measurement field of view based on the created synthesis image.

[0068] Thereafter, it is determined whether or not the imaging of phase-shifted images and two-dimensional images has been completed for all divided measurement fields of view for the loaded substrate PB (step S7). If the imaging has been completed (YES in step S7), the substrate PB is unloaded from the substrate inspection machine 14 (step S8). On the other hand, if the imaging has not been completed (NO in step S7), the process returns to step S3, the field of view of the first camera 21 is moved to the next measurement location, and the processes from step S4 onwards are repeated.

[0069] FIG. 12 is a flowchart showing an example of the image synthesis process executed by the measurement processing unit 63. First, the area extraction unit 632 performs binarization processing on the two-dimensional image acquired in step S5 (step S11). During the binarization processing, the area extraction unit 632 reads a predetermined threshold value stored in the memory unit 64 and creates a binarized image based on whether the brightness value of each pixel constituting the two-dimensional image exceeds the threshold value. Next, the area extraction unit 632 uses the binarized image to set a specific area AR1 for the measurement field of view of the current board PB (step S12). As described above, the specific area AR1 is the mirrored portion 71M or the transparent portion 74T of the component. In addition to these, if there is a measurement surface for which high-precision three-dimensional data cannot be obtained using the phase shift method but for which high-precision three-dimensional data can be obtained using the light-section method, that measurement surface may be designated as the specific area AR1. The area other than the specific area AR1 in the measurement field of view is designated as the remaining area AR2.

[0070] Next, the synthesis processor 634 performs a process of identifying the position coordinates corresponding to the specific area AR1 for each of the phase-shifted image acquired in step S4 and the light-section image acquired in step S2 (step S13). The synthesis processor 634 then performs a synthesis process in which the specific area AR1 of the light-section image is fitted to the specific area AR1 of the phase-shifted image. Specifically, the synthesis processor 634 performs a process of overwriting the pixel values ​​of the specific area AR1 of the phase-shifted image onto the pixel values ​​of the specific area AR1 of the light-section image (step S14). The pixel values ​​of the light-section image are maintained for the remaining area AR2. Based on the synthesized image created in this way, three-dimensional information (height data) of the current measurement field of view is derived.

[0071] According to the three-dimensional measuring device MA described above, a specific area AR1 of the substrate PB to be measured is extracted based on two-dimensional image data. The specific area AR1 is an area for which accurate three-dimensional data cannot be obtained using the phase shift method or the light-section method. In the example given above, the specific area AR1 is exemplified by the mirrored portion 71M and the transparent portion 74T, for which accurate three-dimensional data cannot be obtained using the phase shift method. Then, by fitting the three-dimensional data with higher accuracy to the specific area AR1 and the remaining area AR2, respectively, a highly accurate composite image is created overall. Therefore, accurate three-dimensional information of the substrate PB can be derived based on the composite image.

[0072] [Inventions Included in the Above-Described Embodiments] The above-described embodiments include the following inventions.

[0073] A three-dimensional measuring device according to one aspect of the present invention comprises a first measuring unit that acquires two-dimensional image data of an object, a second measuring unit that acquires first three-dimensional data of the object by a phase shift method, a third measuring unit that acquires second three-dimensional data of the object by a light-section method, and a processing unit that generates three-dimensional information of the object based on the data acquired by the first measuring unit, the second measuring unit, and the third measuring unit, wherein the processing unit extracts a specific region of the object based on the two-dimensional image data, and generates one piece of three-dimensional information about the object by performing a synthesis process in which either the first three-dimensional data or the second three-dimensional data is fitted to the specific region and the other is fitted to the remaining region other than the specific region.

[0074] A three-dimensional measurement method according to another aspect of the present invention includes acquiring two-dimensional image data of an object, acquiring first three-dimensional data of the object using a phase shift method, acquiring second three-dimensional data of the object using a light section method, extracting a specific region of the object based on the two-dimensional image data, and performing a synthesis process in which either the first three-dimensional data or the second three-dimensional data is fitted to the specific region and the other is fitted to the remaining region other than the specific region, thereby generating a single piece of three-dimensional information about the object.

[0075] According to the above-described three-dimensional measuring device or measuring method, a specific region of an object is extracted based on two-dimensional image data. Either first three-dimensional data acquired by the phase shift method or second three-dimensional data acquired by the light-section method is fitted to this specific region. The other data is fitted to the remaining region other than the specific region. For example, based on the two-dimensional image data, a region for which accurate three-dimensional data cannot be obtained using the phase shift method or the light-section method is identified as the specific region. Then, by fitting the three-dimensional data with higher accuracy to the specific region and the remaining region, respectively, a single piece of highly accurate three-dimensional information for the entire object can be generated.

[0076] In the above-mentioned three-dimensional measuring device, the first measuring unit includes a coaxial lighting unit that irradiates the object with coaxial light and a camera that images the object, and the processing unit may extract the mirrored portion of the object as the specific area based on two-dimensional image data acquired by the camera when the object irradiated with the coaxial light is imaged.

[0077] When an object includes a mirror surface, the mirror surface can be accurately recognized based on two-dimensional image data acquired by capturing an image using coaxial light as illumination. The first three-dimensional data obtained using the phase shift method has low accuracy due to specular reflection of the projected light from the mirror surface. On the other hand, if the projected light is irradiated from multiple angles, the first three-dimensional data is highly accurate for the remaining area other than the mirror surface. In contrast, the second three-dimensional data obtained using the light-section method detects specular reflection of line light, so it is highly accurate even for the mirror surface. However, because it is difficult to set a layout for irradiating line light from multiple angles, the second three-dimensional data may include data for areas that are blind spots for measurement. Therefore, by referencing the recognition results of the mirror surface based on the two-dimensional image data and fitting the high-accuracy areas of the first and second three-dimensional data to the mirror surface and the remaining area, respectively, highly accurate three-dimensional information about an object including a mirror surface can be generated.

[0078] In the above-mentioned three-dimensional measuring device, the first measuring unit may include a camera that images the object, and the processing unit may extract a transparent portion of the object as the specific area based on two-dimensional image data obtained by the camera capturing the object.

[0079] When an object includes a transparent portion, it is easy to recognize the transparent portion based on the two-dimensional image data. The first three-dimensional data generated by the phase shift method tends to recognize the shape of the bottom surface of the transparent portion, resulting in lower accuracy. On the other hand, the second three-dimensional data generated by the light-section method detects specularly reflected light of a line of light, resulting in high accuracy even for transparent portions. Therefore, by referencing the transparent portion recognition results based on the two-dimensional image data and fitting the high-accuracy regions of the first and second three-dimensional data to the transparent portion and remaining region, respectively, high-accuracy three-dimensional information about the object including the transparent portion can be generated.

[0080] In the above-mentioned three-dimensional measuring device, it is desirable that the processing unit uses the first three-dimensional data as base data and performs a synthesis process in which the second three-dimensional data is fitted to the specific area on the first three-dimensional data.

[0081] According to this aspect, second three-dimensional data based on the light-section method, which can measure specific areas of mirrored or transparent parts with high accuracy and where the accuracy of the first three-dimensional data based on the phase shift method is reduced, is fitted to these areas. The accuracy of the remaining areas can be ensured by the first three-dimensional data. Therefore, three-dimensional information of the object can be measured with high accuracy. Furthermore, with the phase shift method, multi-angle pattern light is projected and captured without moving the imaging device during measurement, and blind spots can be filled by combining the images obtained by these projections and captures. This allows for a fast measurement cycle. Therefore, by using the first three-dimensional data as base data, the measurement cycle can be shortened.

[0082] The above-mentioned three-dimensional measuring device may further include a memory unit that stores two-dimensional image data of the object previously acquired by the first measuring unit, and the processing unit may extract the specific area based on the two-dimensional image data read from the memory unit.

[0083] According to this aspect, since the two-dimensional image data of the object is stored in advance in the storage unit, the operation of acquiring the two-dimensional image data by the first measurement unit can be omitted when performing three-dimensional measurement of the object, thereby simplifying the measurement operation.

[0084] According to the present invention as described above, it is possible to provide a three-dimensional measuring device and method that can acquire three-dimensional information of an object with high accuracy.

Claims

1. A three-dimensional measurement device comprising: a first measurement unit that acquires two-dimensional image data of an object; a second measurement unit that acquires first three-dimensional data of the object by a phase shift method; a third measurement unit that acquires second three-dimensional data of the object by an optical cutting method; and a processing unit that generates three-dimensional information of the object based on the data acquired by the first measurement unit, the second measurement unit, and the third measurement unit, wherein the processing unit extracts a specific region of the object based on the two-dimensional image data, and generates one piece of three-dimensional information about the object by performing a synthesis process of fitting either the first three-dimensional data or the second three-dimensional data to the specific region and fitting the other to the remaining region other than the specific region.

2. The three-dimensional measurement device according to claim 1, wherein the first measurement unit includes a coaxial illumination unit that irradiates the object with coaxial light and a camera that images the object, and the processing unit extracts a specular portion included in the object as the specific region based on two-dimensional image data acquired by the camera imaging the object irradiated with the coaxial light.

3. The three-dimensional measurement device according to claim 1, wherein the first measurement unit includes a camera that images the object, and the processing unit extracts a transparent portion included in the object as the specific region based on two-dimensional image data acquired by the camera imaging the object.

4. The three-dimensional measurement device according to any one of claims 1 to 3, wherein the processing unit performs a synthesis process of using the first three-dimensional data as base data and fitting the second three-dimensional data to the specific region on the first three-dimensional data.

5. The three-dimensional measurement device according to any one of claims 1 to 3, further comprising a storage unit that stores two-dimensional image data of the object acquired in advance by the first measurement unit, and the processing unit extracts the specific region based on the two-dimensional image data read from the storage unit.

6. A three-dimensional measurement method for generating one piece of three-dimensional information about an object, comprising: acquiring two-dimensional image data of the object; acquiring first three-dimensional data of the object by a phase shift method; acquiring second three-dimensional data of the object by an optical cutting method; extracting a specific region of the object based on the two-dimensional image data; and performing a synthesis process of fitting either the first three-dimensional data or the second three-dimensional data to the specific region and fitting the other to the remaining region other than the specific region.

Images