Image Processing Device

Abstract

To provide an image processing apparatus using an image data composition method that can compose a plurality of pieces of image data acquired by an imaging unit at different positions in an imaging direction accurately in a short time.SOLUTION: An inspection device 10 being an image processing apparatus has a camera unit 21 and a control unit 30. The control unit 30 changes relative positions in an imaging direction of the camera unit 21 and an inspected body 12 to acquire at least two pieces of image data picked up by the camera unit 21 at different positions in the imaging direction, and executes composite image update processing of determining, for every image data, if pixels of the image data satisfy a condition for a focused pixel, and when determining that the pixels satisfy the condition, selecting values of the pixels as values of corresponding pixels of composite image data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device that uses a method for synthesizing image data. [Background technology]

[0002] In recent years, electronic circuit boards have come to be mounted on a variety of devices, but miniaturization, thinning, etc. are always challenges for devices on which these types of electronic circuit boards are mounted, and from this point of view, there is a demand for higher density mounting of electronic circuit boards. In inspection devices that inspect the state of solder application on such electronic circuit boards, the state of mounting of electronic components, etc., a three-dimensional measurement device based on a phase shift method has been proposed that uses multiple light patterns with different periods to expand the measurement range and shorten the measurement time (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-003513 Summary of the Invention [Problem to be solved by the invention]

[0004] The above method has limitations on expanding the measurement range due to the depth of field of the imaging unit. On the other hand, it is known that multiple measurement data can be stitched together by taking measurements from multiple imaging positions and performing coordinate transformation according to their positional relationships. However, there is a problem in that simply combining measurement results calculated from multiple imaging positions often contains erroneous measurements, especially near the seams of the combined data.

[0005] The present invention has been made in consideration of such problems, and aims to provide an image processing device that uses an image data synthesis method that can synthesize multiple image data acquired by imaging units located at different positions in the imaging direction (Z direction) with high accuracy and in a short time. [Means for solving the problem]

[0006] In order to solve the above problem, an image processing device according to the present invention includes an imaging unit capable of changing a relative position of an imaging direction with respect to an object to be inspected, and a control unit, wherein the control unit: a first step of acquiring reference position image data by capturing an image with the imaging unit at a position when focused on a predetermined reference plane on the object under test, and executing height basic processing to acquire height information of the object under test for each pixel of the reference position image data; a second step of setting a position in an imaging direction different from the position when focused on the predetermined reference plane and among predetermined positions that are the furthest from the object under test as a current position; a third step of acquiring current position image data by capturing an image with the imaging unit at the current position, and executing the height basic processing for each pixel of the current position image data to acquire height information of the object under test; and a third step of calculating, for each pixel of the current position image data, brightness information for each pixel of the reference position image data and the current position image data, which are included in the height information, based on information on the distance between the object under test and the most focused position of the imaging unit for each pixel of the current position image data, It is determined whether the pixel satisfies the conditions for being in focus, and if it is determined that the pixel satisfies the conditions, The current position image data A composite image update process is performed to select the value of the pixel in question as the value of the corresponding pixel in the composite image data. a fourth step; a fifth step of repeatedly executing the third step and the fourth step, with the next position among the predetermined positions in a direction approaching the predetermined reference plane from the current position set as the current position; and a sixth step of, when it is determined in the fifth step that the third step and the fourth step have been executed at all of the predetermined positions, setting the position when the predetermined reference plane is in focus as the current position, executing the composite image update process for each pixel of the reference position image data, and when it is determined that the pixel satisfies the condition, selecting the value of the pixel in the reference position image data as the value of the corresponding pixel in the composite image data. The focal length of the imaging unit is fixed.

[0008] The image processing device according to the present invention further includes a projection unit that projects a stripe pattern onto the object to be inspected, the stripe pattern being formed by repeatedly changing the phase so as to monotonically increase or decrease, and the control unit is configured to: Reference position Image data Or, the current position image data is acquired and the basic height processing is performed. configured to acquire the height information. are It is preferable.

[0009] In the image processing device according to the present invention, the control unit performs the following in the composite image update process: Current position Image data pixel Included in the brightness information Information about the amplitude and Reference position Image data pixel Included in the brightness information It is preferable to judge whether the pixel satisfies the above condition by comparing with information about the amplitude.

[0010] In the image processing device according to the present invention, the control unit performs the following in the basic height processing: Reference position For image data, the height information is acquired so as to maximize the number of measurement points, Current positionIt is preferable to acquire the height information from the image data so as to exclude pixels that are highly suspected of having a measurement error.

[0011] The image processing device according to the present invention further comprises an illumination unit that projects illumination light onto the object to be inspected, Reference position Image data and each of the current position image data The height map includes pattern image data acquired by the imaging unit when the stripe pattern is projected by the projection unit, and two-dimensional image data acquired by the imaging unit when the illumination light is projected by the illumination unit, and the control unit acquires the height information from the pattern image data in the basic height processing, and when it is determined in the composite image update processing that a pixel satisfies the condition, it is preferable that the control unit selects the height information of the pixel in the pattern image data as the value of the corresponding pixel in the composite image data of the height map, and selects the value of the pixel in the two-dimensional image data as the value of the corresponding pixel in the composite image data of the two-dimensional image data.

[0012] In addition, in the image processing device of the present invention, it is preferable that the control unit performs height post-processing such as unmeasured pixel interpolation processing and distortion correction processing on the composite image data of the height map synthesized by the composite image update processing.

[0013] In the image processing device according to the present invention, the control unit, in the composite image update process, performs the composite image update process by detecting two or more images of the same pixel when the imaging unit is at different positions in the imaging direction. The reference position Image data Or the current location image data When it is determined that a pixel satisfies the above condition, it is preferable to select the value of the pixel in the image data that is located farthest from the object to be inspected by the imaging unit as the value of the corresponding pixel in the composite image data. [Effects of the Invention]

[0018] According to the present invention, it is possible to provide an image processing device that uses an image data synthesis method that can synthesize multiple image data acquired by imaging units located at different positions in the imaging direction (Z direction) with high accuracy and in a short time. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is an explanatory diagram for explaining the configuration of an inspection device including an image processing device. [Figure 2] FIG. 10 is an explanatory diagram illustrating the relationship between the phase of a reference plane and a measured phase. [Figure 3] FIG. 2 is an explanatory diagram for explaining the imaging position of a camera unit. [Figure 4] 10 is a flowchart showing the flow of a method for combining image data. [Figure 5] FIG. 10 is an explanatory diagram for explaining the correspondence between the range of condition (1) at each imaging position and composite image data. DETAILED DESCRIPTION OF THE INVENTION

[0020] A preferred embodiment of the present invention will be described below with reference to the drawings. First, the configuration of an inspection device 10 according to this embodiment will be described with reference to FIG. 1. This inspection device 10 is a device that inspects an object 12 under inspection using image data (two-dimensional image data or pattern image data) of the object 12 under inspection obtained by imaging the object 12 under inspection. Therefore, this inspection device 10 also functions as an image processing device. The object 12 under inspection is, for example, an electronic circuit board on which components are mounted and solder is applied.

[0021] The inspection device 10 includes an inspection table 14 for holding the object under inspection 12, an imaging unit 20 for illuminating and imaging the object under inspection 12, an XY stage 16 for moving the imaging unit 20 relative to the inspection table 14, and a control unit 30 for controlling the operation of the imaging unit 20 and the XY stage 16 and inspecting the object under inspection 12. For ease of explanation, as shown in Fig. 1, the surface of the inspection table 14 on which the object under inspection is placed is defined as the XY plane, and the direction perpendicular to the surface (i.e., the imaging direction by the camera unit 21 constituting the imaging unit 20 (the optical axis direction of the optical system of the camera unit 21)) is defined as the Z direction.

[0022] The imaging unit 20 is attached to a moving table (not shown) of the XY stage 16 and can be moved in both the X and Y directions by the XY stage 16. The XY stage 16 is, for example, a so-called H-shaped XY stage. The XY stage 16 includes a Y drive unit that moves the moving table in the Y direction along a Y-direction guide extending in the Y direction, and two X-direction guides and an X drive unit that support the Y-direction guide at both ends and are configured to move the moving table and the Y-direction guide in the X direction. The XY stage 16 also includes a Z-movement mechanism that moves the imaging unit 20 in the Z direction. It may also include a rotation mechanism that rotates the imaging unit 20. The inspection device 10 may further include an XY stage that moves the inspection table 14. In this case, the XY stage 16 that moves the imaging unit 20 may be omitted. A linear motor or a ball screw may be used for the X drive unit and the Y drive unit. Instead of moving the imaging unit 20 in the Z direction, the XY stage 16 may be configured to move in the Z direction.

[0023] The imaging unit 20 includes a camera unit 21 that captures an image from a direction perpendicular (Z direction) to the inspection surface (substrate surface) of the object under inspection 12, an illumination unit 22, and a projection unit 23. In the inspection device 10 according to this embodiment, the camera unit 21, the illumination unit 22, and the projection unit 23 may be configured as an integrated imaging unit 20. In this integrated imaging unit 20, the relative positions of the camera unit 21, the illumination unit 22, and the projection unit 23 may be fixed, or each unit may be configured to be relatively movable. Alternatively, the camera unit 21, the illumination unit 22, and the projection unit 23 may be separate units configured to be independently movable.

[0024] The camera unit 21 includes an imaging element that generates a two-dimensional image of the object, and an optical system (e.g., a lens) for focusing the image on the imaging element. The camera unit 21 is, for example, a CCD camera. The maximum field of view of the camera unit 21 may be smaller than the area on the inspection table 14 where the object under test 12 is placed. In this case, the camera unit 21 captures an image of the entire object under test 12 by dividing the image into multiple partial images. The control unit 30 controls the XY stage 16 so that the camera unit 21 moves to the next imaging position each time the camera unit 21 captures a partial image. The control unit 30 synthesizes the partial images to generate an entire image of the object under test 12.

[0025] The camera unit 21 may include an imaging element that generates a one-dimensional image instead of the two-dimensional imaging element. In this case, the entire image of the inspection object 12 can be acquired by scanning the inspection object 12 with the camera unit 21.

[0026] The illumination unit 22 is configured to project illumination light onto the surface of the inspection object 12 for imaging by the camera unit 21. The illumination unit 22 includes one or more light sources that emit light of a wavelength or wavelength range selected from the wavelength range detectable by the imaging element of the camera unit 21. The illumination light is not limited to visible light, and ultraviolet light, X-rays, etc. may also be used. When multiple light sources are provided, each light source is configured to project light of a different wavelength (e.g., red, blue, and green) onto the surface of the inspection object 12 at a different projection angle.

[0027] The object under inspection 12 illuminated by the lighting unit 22 is imaged by the camera unit 21. The inspection device 10 determines the presence or absence of defects on the board of the object under inspection 12 (for example, whether components are present and properly arranged, and whether the solder application state is good or bad) based on image data of the object under inspection 12 illuminated and imaged by the lighting unit 22 (this image data is referred to as "two-dimensional image data") and a height map, which will be described later.

[0028] In the inspection device 10 according to this embodiment, the illumination unit 22 is a side illumination source that projects illumination light obliquely onto the inspection surface of the object under inspection 12, and in this embodiment, includes an upper light source 22a, a middle light source 22b, and a lower light source 22c. In the inspection device 10 according to this embodiment, the side illumination sources 22a, 22b, and 22c are each a ring illumination source that surrounds the optical axis of the camera unit 21 and is configured to project illumination light obliquely onto the inspection surface of the object under inspection 12. Each of these side illumination sources 22a, 22b, and 22c may be configured with multiple light sources arranged in a circular ring shape. Furthermore, the upper light source 22a, the middle light source 22b, and the lower light source 22c, which are side illumination sources, are each configured to project illumination light at different angles onto the inspection surface.

[0029] The projection unit 23 projects a pattern onto the inspection surface of the object under test 12. The object under test 12 onto which the pattern is projected is imaged by the camera unit 21. The inspection device 10 creates a height map of the inspection surface of the object under test based on image data of the imaged object under test 12 (this image data is referred to as "pattern image data"). Here, the height map is data that contains height information of the object under test for each pixel of the pattern image data. The control unit 30 detects local mismatches between the projected pattern and the pattern image data, and obtains height information for that portion based on the local mismatches. In other words, changes in the imaged pattern relative to the projected pattern correspond to changes in height on the inspection surface.

[0030] The projection pattern is preferably a one-dimensional stripe pattern in which bright and dark lines alternate periodically. The projection unit 23 is arranged to project the stripe pattern obliquely onto the inspection surface of the inspection object 12. Discontinuities in height on the inspection surface of the inspection object 12 appear as pattern shifts in the stripe pattern image. Therefore, the height difference can be determined from the amount of pattern shift. For example, the control unit 30 creates a height map using the PMP (Phase Measurement Profilometry) method, which uses a stripe pattern whose brightness changes according to a sine curve. In the PMP method, the amount of shift in the stripe pattern corresponds to the phase difference of the sine curve.

[0031] The projection unit 23 includes a pattern forming device, a light source device for illuminating the pattern forming device, and an optical system for projecting a pattern onto the inspection surface of the object under test 12. The pattern forming device may be, for example, a variable patterning device capable of dynamically generating a desired pattern, such as a liquid crystal display, or a fixed patterning device in which a pattern is fixedly formed on a substrate, such as a glass plate. When the pattern forming device is a fixed patterning device, it is preferable to provide a movement mechanism for moving the fixed patterning device or an adjustment mechanism in the optical system for pattern projection, thereby making it possible to change the projection position of the pattern. Furthermore, the projection unit 23 may be configured to be able to switch between multiple fixed patterning devices having different patterns.

[0032] A plurality of projection units 23 may be provided around the camera unit 21. The plurality of projection units 23 are arranged so as to project patterns onto the inspection object 12 from different projection directions. In this way, it is possible to reduce the area on the inspection surface that is shaded due to height differences and onto which the pattern is not projected.

[0033] 1, which controls the entire device, is realized as hardware by the CPU, memory, and other LSIs of any computer, and as software by programs loaded into memory, but the diagram shows functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination of both.

[0034] FIG. 1 shows an example of the configuration of the control unit 30. The control unit 30 includes an inspection control unit 31 and a memory 35, which is a storage unit. The inspection control unit 31 includes an image processing unit 32, an inspection data processing unit 33, and an inspection unit 34. The image processing unit 32 includes an image capture processing unit 32a, a height measurement unit 32b, and an image synthesis unit 32c. The inspection device 10 also includes an input unit 36 ​​for receiving input from a user or another device, and an output unit 37 for outputting information related to the inspection. The input unit 36 ​​and the output unit 37 are each connected to the control unit 30. The input unit 36 ​​includes, for example, input means such as a mouse or keyboard for receiving input from a user, and communication means for communicating with other devices. The output unit 37 includes known output means such as a display or printer.

[0035] The inspection control unit 31 is configured to execute various control processes for inspection based on input from the input unit 36 ​​and inspection-related information stored in the memory 35. The inspection-related information includes two-dimensional image data of the object 12 under inspection, a height map of the object 12 under inspection (calculated from pattern image data), and substrate inspection data. Prior to inspection, the inspection data processing unit 33 creates substrate inspection data using the two-dimensional image data and height map of the object 12 under inspection that is guaranteed to pass all inspection items. The inspection unit 34 executes inspection based on the created substrate inspection data and the two-dimensional image data and height map of the object 12 under inspection to be inspected.

[0036] Board inspection data is inspection data created for each type of board. Board inspection data is a collection of inspection data for the components placed on the board, their positions, and each solder applied to the board. Inspection data for each component or solder includes the inspection items required for that component or solder, the inspection window, which is the inspection area on the image for each inspection item, and the inspection criteria that serve as the basis for judging the placement and pass / fail for each inspection item. One or more inspection windows are set for each inspection item. For example, in an inspection item that judges the pass / fail of the solder application state, typically, the same number of inspection windows as the number of solder application areas on the component are set in an arrangement that corresponds to the arrangement of the solder application areas. Furthermore, for inspection items that use images that have undergone specified image processing on the image of the object to be inspected, the details of that image processing are also included in the inspection data.

[0037] The inspection data processing unit 33 sets each item of inspection data to suit the board in the board inspection data creation process. For example, the inspection data processing unit 33 automatically sets the position and size of each inspection window for each inspection item so that it matches the solder layout of the board. The inspection data processing unit 33 may also accept user input for some items of the inspection data. For example, the inspection data processing unit 33 may also accept tuning of the inspection criteria by the user. The inspection criteria may also be set using height information.

[0038] The inspection control unit 31 performs imaging processing of the object under inspection 12 using the image processing unit 32 as preprocessing for creating substrate inspection data. The object under inspection 12 used is one that has passed all inspection items. As described above, the imaging processing is performed by controlling the relative movement of the imaging unit 20 and the inspection table 14 while illuminating the object under inspection 12 using the illumination unit 22, and sequentially capturing partial images of the object under inspection 12. Multiple partial images are captured so that the entire object under inspection 12 is covered. The inspection control unit 31 combines these multiple partial images to generate full-surface substrate image data (two-dimensional image data) that includes the entire inspection surface of the object under inspection 12. The inspection control unit 31 stores the full-surface substrate image data in the memory 35.

[0039] Furthermore, as a preprocessing step for creating a height map, the inspection control unit 31 controls the relative movement of the imaging unit 20 and the inspection table 14 while projecting a pattern onto the inspection object 12 using the projection unit 23, using the imaging processing unit 32a of the image processing unit 32, to sequentially capture divided pattern images of the inspection object 12. The projected pattern is preferably a stripe pattern whose brightness varies according to a sine curve based on the PMP method. The inspection control unit 31 combines the captured divided images to generate pattern image data for the entire inspection surface of the inspection object 12. The inspection control unit 31 stores the pattern image data in the memory 35. Note that pattern image data may be generated for only a portion of the inspection surface rather than the entire surface. As will be described later, the image data captured by the imaging processing unit 32a is combined by the height measurement unit 32b and the image combining unit 32c.

[0040] The height measurement unit 32b creates a height map of the entire inspection surface of the object under inspection 12 based on the image of the pattern in the pattern image data. The height measurement unit 32b first calculates the local phase difference between the pattern image data and the reference pattern image data for the entire image data, thereby calculating a phase difference map of the inspection surface of the object under inspection 12. Here, the "reference pattern image data" refers to image data in which a pattern is projected onto a reference plane by the projection unit 23 (i.e., image data in which a pattern generated by a pattern forming device built into the projection unit 23 is projected onto the reference plane). The height measurement unit 32b creates a height map of the object under inspection 12 based on a reference plane serving as a reference for height measurement and the phase difference map. The reference plane is, for example, the surface of the electronic circuit board to be inspected. The reference plane does not necessarily have to be flat, but may be a curved surface that reflects deformation such as warpage of the board.

[0041] Specifically, the height measurement unit 32b calculates the phase difference of the stripe pattern between each pixel of the pattern image data and the corresponding pixel of the reference pattern image data. The height measurement unit 32b converts the phase difference into height information. This is because the distance from the projection unit 23 varies depending on the position on the inspection surface, so even if the stripe width of the reference pattern is constant, the stripe width changes from one end of the pattern projection area on the inspection surface to the other. The height measurement unit 32b obtains height information from the reference plane based on the converted height information and the reference plane, and creates a height map of the inspection object 12.

[0042] The image processing unit 32 of the inspection control unit 31 may create an image of the object under test 12 having a height distribution by associating height information contained in the height map of the object under test 12 with each pixel of the two-dimensional image data of the object under test 12. The image processing unit 32 may also perform a three-dimensional modeling display of the object under test 12 based on the object under test image data with height distribution. The image processing unit 32 may also superimpose the height distribution on the two-dimensional object under test image data and display it on the output unit 37. For example, the object under test image data may be displayed in different colors depending on the height distribution.

[0043] A method for acquiring height information by projecting a pattern using the projection unit 23 in such an inspection device 10 will be described below.

[0044] First, we will explain how height information is obtained using the PMP method with a stripe pattern. Focusing on one measurement point (pixel) in the stripe pattern projection area, when the stripe pattern is projected while shifting the phase spatially (in other words, when the stripe pattern is scanned in the direction in which the stripes repeat), the brightness of that measurement point (pixel) fluctuates periodically. Although the average brightness differs for each measurement point (pixel) depending on the surface characteristics of the measurement point (pixel), such as color and reflectance, a periodic brightness fluctuation corresponding to the stripe pattern occurs at every measurement point (pixel). Therefore, the phase is calculated from the periodic brightness fluctuation, and the phase difference from the initial phase provides height information.

[0045] When acquiring height information using the PMP method using the characteristics of such a stripe pattern, it is necessary in principle to capture the stripe pattern at least three times, typically four times, with the phase shifted. Since the stripe pattern is a sine wave, the brightness fluctuation at each measurement point (pixel) also becomes a sine wave. Since the stripe pitch is known, the sine wave representing the brightness fluctuation can be identified by determining the average luminance (brightness), amplitude, and initial phase.

[0046] By obtaining luminance (brightness) measurements of measurement points (pixels) from at least three image data frames captured with the phase start position of the stripe pattern shifted, three variables—the average luminance (brightness), amplitude, and initial phase—can be determined. Here, let I0, I1, I2, and I3 be the luminances of a pixel when a stripe pattern is captured four times with the phase start position shifted by 90 degrees (π / 2 radians). If the stripe pattern is a sine wave, the luminance In can be expressed by the following equation (a): In equation (a), A represents the amplitude dependent on the brightness of the object under test 12, B represents the brightness offset dependent on the camera unit 21 and the ambient brightness, φ represents the desired phase, and π / 2 represents the phase shift (amount of shift). Note that n = 0, 1, 2, or 3. However, because errors are generally introduced due to factors such as sensor noise, the least-squares method is often used to calculate the phase φ, amplitude A, and offset B. Let E represent the error obtained by the least-squares method.

[0047] In = A×sin(φ+n×π / 2)+B (a)

[0048] Height information is calculated by finding the phase φ, amplitude A, and offset B of each pixel from the pattern image data. For example, when one stripe pattern is imaged four times with the phase start position shifted by 90 degrees (π / 2 radians), it is known that the phase φ of the measurement point corresponding to each pixel can be expressed as the following equation (b).

[0049] tan(φ) = (I3-I1) / (I2-I0) (b)

[0050] If the pitch of the stripe pattern (the length of one period) is P and the initial phase is 0, the amount of deviation of the stripe pattern can be calculated as P × (φ / 2π). Using the projection angle of the pattern, height information at that position can be obtained from the amount of pattern deviation.

[0051] 2, (a) shows the phase obtained from pattern image data captured by camera unit 21 after projecting a stripe pattern onto reference plane S using projection unit 23. (c) shows the phase obtained from pattern image data captured by camera unit 21 after projecting the same stripe pattern onto reference plane S as shown in (b). (d) shows the phase obtained from pattern image data captured by projecting the same stripe pattern onto reference plane S with object O placed thereon. Height information for object O can be calculated from the amount of phase shift (phase difference = measured phase - phase of reference plane). This phase difference can be calculated for each pixel in the image captured by camera unit 21, and height information for the position corresponding to that pixel can be calculated by multiplying the phase difference calculated for each pixel by a coefficient. This coefficient can be calculated simply by, for example, tan (projection angle) × stripe pitch / 2π.

[0052] Here, the stripe pattern has a sinusoidal brightness that changes monotonically (monotonically increasing or decreasing) and is repeatedly formed. Note that in one period (pitch), as the phase changes from 0 to 2π (radians), the brightness monotonically increases from the darkest state (e.g., brightness I=0) to the brightest state (e.g., brightness I=255) (although the relationship between phase and brightness in the pattern may be reversed, i.e., monotonically decreasing).

[0053] As explained using Figure 2, when a reference plane S is imaged using a stripe pattern and an object O (corresponding to the object under test 12) placed on the reference plane S is also imaged using the stripe pattern, the phase difference Δφ between the phase φ0 of the reference plane S obtained based on equation (b) and the phase φ obtained with the object O placed thereon can be calculated using the following equation (c).

[0054] Δφ = φ - φ0 (c)

[0055] From the above, height information H can be obtained by the following equation (d) based on the phase difference Δφ calculated by equation (c) from the phases φ0, φ (calculated by equation (b) when four images are taken with a 90° shift) obtained from a reference plane and at least three pattern image data images captured by placing an object O (inspection object 12) on this reference plane and projecting a stripe pattern with the phase start position shifted.

[0056] H = P×(Δφ / 2π) (d)

[0057] As described above, the inspection device 10 according to this embodiment is configured to inspect the object under inspection 12 using a height map calculated from the two-dimensional image data and the pattern image data, but because the camera of the camera unit 21 has a predetermined depth of field (depth of focus), for example, when the camera unit 21 is focused on the vicinity of the surface of the substrate, which is the object under inspection 12, it is possible that the top surface of a tall component will be captured as out-of-focus image data. Therefore, when such image data is used for inspection, it may be impossible to read characters written on the top surface of a tall component (such as a capacitor) in the two-dimensional image data, and it may also be impossible to calculate accurate height information in the pattern image data because the image of the stripe pattern on the top surface of the tall component is blurred.

[0058] Therefore, the inspection device 10 according to this embodiment is configured to acquire at least two pieces of image data captured by the camera unit 21 at different positions in the Z direction (imaging direction), extract relatively in-focus pixels from each piece of image data, and combine them to obtain image data that is in focus across the entire field of view, even when components are located at different heights. Here, a case where image data acquired when the camera unit 21 is focused on a reference plane (a plane that serves as a reference for calculating height information, as described above) and image data acquired by the camera unit 21 at a different position in the Z direction by moving the camera unit 21 in the Z direction are combined is described. Below, a method for combining image data in the inspection device 10, which is an image processing device according to this embodiment, is described. Here, a case where the camera unit 21 is moved in the Z direction is described. However, as long as the relative positions of the camera unit 21 and the object under test 12 in the Z direction change, the object under test 12 may also be moved in the Z direction. As shown in FIG. 3(a), a predetermined surface of the object under test 12 (e.g., the top surface of an electronic circuit board) is defined as the "reference surface." Here, FIG. 3(a) shows a case where the reference surface of the object under test 12 coincides with the reference plane.

[0059] In the following description, the position of the camera unit 21 in the Z direction is expressed in a coordinate system with Z=0 as the reference, which is the position of the camera unit 21 when focused on the reference plane, as shown in FIG. 3(a). Specifically, P(i), which indicates the position of the camera unit 21 in the Z direction, is expressed as i=0 when the camera unit 21 is focused on the reference plane, and as i=1, 2, 3, ... in order of increasing distance from the reference plane. In the following description, an example will be taken in which four pieces of image data are acquired at different positions in the Z direction. For example, assuming that the depth of field of the camera unit 21 is 4 mm, image data is acquired at a position "P(0)=0" of the camera unit 21 when focused on the reference plane, and three pieces of image data are acquired at positions shifted 4 mm upward (in the Z direction) from the position P(0)=0. The respective positions are represented as "P(1) = 4," "P(2) = 8," and "P(3) = 12" (units of mm are omitted). Furthermore, when the camera unit 21 is focused on the reference plane, that is, when the camera unit 21 is located at Z = 0, the distance from the tip of the camera unit 21 to the reference plane is defined as the focal distance. Furthermore, the position away from the tip of the camera unit 21 in the direction of the object under test 12 is defined as the most focused position FP.

[0060] The number of image data obtained by shifting the position of camera unit 21 in the Z direction is not limited to four, but may be two or more. The width by which camera unit 21 is shifted in the Z direction is not limited to 4 mm, but may be set to any value taking into consideration, for example, the height of the components mounted on inspection object 12 and the depth of field of camera unit 21. Furthermore, the position of camera unit 21 in the Z direction may not only be changed at equal intervals, but also the intervals at which each image data is acquired may be selected arbitrarily.

[0061] It is assumed that the position in the Z direction at which camera unit 21 acquires image data is set in advance in control unit 30 via input unit 36 ​​or the like. For example, the position of the reference plane and the height of the inspection target (the height of the components mounted on the board) are input, and the positions and intervals in the Z direction at which image data is acquired are set. It is also assumed that camera unit 21 has a fixed focal length.

[0062] A method for combining image data in the inspection device 10, which is an image processing device according to this embodiment, will be described with reference to Fig. 4. This image data combining method is configured to perform a combining process using image data acquired at a position farthest from the inspected object 12 (reference surface) among image data acquired at different positions in the Z direction.

[0063] 4(a), when the control unit 30 determines that the process of acquiring image data of the object under inspection 12 has started, the control unit 30 controls the image capturing processing unit 32a to move the camera unit 21 to a predetermined position (position in the X and Y directions) above the object under inspection 12, and then adjusts the position of the camera unit 21 in the Z direction so that the camera unit 21 is at position i=0, i.e., P(0), and acquires image data of the object under inspection 12 (step S100). When the camera unit 21 is moved to the position P(0), the camera unit 21 is focused on the vicinity of the reference plane of the object under inspection 12. In step S100, as described above, in order to acquire a height map of the object under inspection 12, the projection unit 23 projects a stripe pattern onto the object under inspection 12 and the camera unit 21 acquires pattern image data. At this time, at least three (preferably four) pieces of pattern image data are acquired by shifting the phase of the stripe pattern. Furthermore, when there are multiple projection units 23, pattern image data is acquired by projecting stripe patterns onto the inspection object 12 from each projection unit 23 while shifting the phase as described above. Furthermore, when acquiring two-dimensional image data, the side illumination sources 22a, 22b, and 22c of the illumination unit 22 are sequentially turned on to illuminate the inspection object 12, and an image is taken by the camera unit 21.

[0064] When the control unit 30 has finished capturing the image of the object under test 12, it sets parameters for removing erroneous measurement values ​​from height information calculated based on pattern image data captured when the camera unit 21 is at position i=0, i.e., at position P(0) (referred to as "pattern image data of P(0)") (step S102). As will be described later, the synthesis process of the image data captured when the camera unit 21 is at position i=0 (at P(0)) is executed last. In this final synthesis process, the above parameters are set to values ​​that maximize the number of measurement points so that as many pixels of the image data at P(0) as possible are selected as pixels of the synthesized image data. Then, the basic height process S200 is executed using these parameters (step S104).

[0065] When the control unit 30 starts the basic height processing S200 shown in FIG. 4(b), the height measurement unit 32b performs phase calculation processing for each pixel (within the field of view) in the pattern image data based on the above-mentioned formulas (a) to (d) from three or four pieces of pattern image data captured with the phase of the stripe pattern shifted (step 202). If multiple projection units 23 are included, the phase calculation processing is performed for each projection unit 23 (for each projection). Next, if multiple projection units 23 are included, the control unit 30 performs projection synthesis processing to synthesize the respective phase calculation results and calculate height information for each pixel (step S204). Finally, noise filtering processing using a median filter or the like is performed on the height information for each pixel (step S206), and the execution of the basic height processing S200 for the pattern image of P(0) is terminated.

[0066] The reason why image data is acquired and height information is calculated from pattern image data initially when the camera unit 21 is at position P(0) (when the camera unit 21 is focused near the reference plane) is that, as will be explained below, the amplitude and height information at P(0) are used in condition (2) used to determine the synthesis process.

[0067] 4(a), when the execution of the basic height process S200 is completed in step S104, the control unit 30 causes the imaging processing unit 32a to move the position of the camera unit 21 in the Z direction to the position farthest from the object under test 12 among the set imaging positions as the next imaging position (step S106). Here, the camera unit 21 is moved to the position P(3)=12 when i=3.

[0068] When the control unit 30 moves the camera unit 21 to the next imaging position in the Z direction, similarly to the case of P(0), the imaging processing unit 32a acquires pattern image data and two-dimensional image data at that imaging position, and further sets parameters for removing erroneous measurement values ​​from height information calculated from the pattern image data (step S108). Furthermore, the control unit 30 causes the height measurement unit 32b to execute the basic height processing S200 on the pattern image data captured in step S108 (step S110). Here, because the camera unit 21 is at a position other than i=0 (in midair), the parameters for removing erroneous measurement values ​​from the height information are set to exclude points that are highly suspected of being measurement errors (exclude erroneous measurement points in midair).

[0069] Once the height information is calculated from the pattern image data captured at the current imaging position (here, position P(3)), the control unit 30 causes the image synthesis unit 32c to execute a synthetic image update process using the image data at the current imaging position (step S112). Here, from among the pixels of the image data at the current imaging position, an image that is in focus (in-focus pixel) is selected and used as the pixel value of the synthetic image data. Specifically, for each pixel, it is determined whether or not to select a pixel of the image data at the current imaging position as synthetic image data based on the height information of that pixel at the current imaging position, using the conditions of the following two equations (conditions (1) and (2)):

[0070] P(i)-DL≦H(X,Y,i)+P(i)≦P(i+1)−DH (1) A(X,Y,i) ≧ A(X,Y,0)×WA (2) however, P(i): Current imaging position P(i+1): The next imaging position that is farther away from the reference plane than the current imaging position H(X,Y,i): Calculated distance at the current imaging position when the position of the pixel to be judged is (X,Y) DL, DH: constant A(X,Y,i): Amplitude at the current imaging position when the position of the pixel to be judged is (X,Y) A(X,Y,0): Amplitude at the imaging position i=0 when the position of the pixel to be judged is (X,Y) WA: Weight coefficient

[0071] Condition (1) determines whether the position (calculated distance) of the object under test 12 corresponding to the pixel to be determined is near the focal point of the camera unit 21 at the current imaging position. Here, the calculated distance H(X, Y, i) is the distance in the Z direction from the most focused position FP at P(i) to the object under test 12 at the pixel to be determined, and is height information of the pixel at position (X, Y) obtained by the basic height processing S200, and is expressed in the coordinate system of the camera unit 21 described above. For example, when P(0) = 0, if the object under test 12 corresponding to the pixel to be determined is on the reference plane, H(X, Y, i) = 0. Note that, as shown in FIG. 3(a), when the camera unit 21 moves in the Z direction, the most focused position FP also moves in the Z direction. 3(b), DL and DH are constants for determining the range in which focused image data can be obtained by the camera unit 21, based on the current imaging position P(i) of the camera unit 21 and the adjacent imaging position P(i+1) that is farther from the reference plane than the current imaging position. For example, if the camera unit 21 is positioned at 4 mm intervals, DL = 1.5 mm and DH = 1.0 mm can be set. Of course, these constants DL and DH can be determined based on the specifications of the camera unit 21 (depth of field, etc.) and the interval by which the camera unit 21 is shifted in the Z direction. Note that it is generally desirable that DL > DH. Furthermore, when the imaging position is the farthest from the object under test 12 (reference plane) (here, P(3) = 12), the value of P(i+1) (here, P(4)) is preset, and this value is used to determine condition (1) (e.g., P(4) = 16). From the above, a pixel that satisfies condition (1) means that the object 12 to be inspected is within a range where focused image data (clear image data) can be obtained by the camera unit 21 at that position (P(i)).

[0072] Here, instead of setting DL and DH in condition (1) as constants, they can also be variables as shown below. Note that FUNC indicates a predetermined function, DL(i) is a variable at imaging position P(i) and is determined by function FUNC with parameters of the next imaging position P(i-1) and the current imaging position P(i), and DH(i) is a variable at imaging position i and is determined by function FUNC with parameters of the current imaging position P(i) and the previous imaging position P(i+1).

[0073] DL(i)=FUNC(P(i-1),P(i)) DH(i)=FUNC(P(i),P(i+1))

[0074] Furthermore, condition (2) determines whether a pixel has a high degree of focus based on the amplitudes A(X, Y, i) and A(X, Y, 0) obtained by the above-described formula (a) when calculating the phase using pattern image data. This condition (2) compares the focus degrees to extract pixels with higher focus degrees, and determines that the pixel is in focus when its evaluation value is equal to or greater than a threshold value. Specifically, since a higher amplitude value indicates a higher degree of focus, the threshold value is determined based on the amplitude at P(0) (i = 0). For the pixel to be determined (pixel at position (X, Y)), the pixel to be determined can be determined to be in focus when the amplitude A(X, Y, i) at the current imaging position is sufficiently larger than the amplitude A(X, Y, 0) at i = 0. The value of weight WA is preferably determined based on the depth of field of the camera unit 21.

[0075] As described above, at the imaging position P(i), conditions (1) and (2) are judged for each pixel, and for pixels that simultaneously satisfy conditions (1) and (2), the value of the pixel at the current imaging position is selected as the pixel of the composite image. Specifically, when a pixel at position (X, Y) satisfies conditions (1) and (2), the height information (H(X, Y, i) + P(i)) calculated from the pattern image data of P(i) for that pixel is set as the value of the pixel at the same position in the composite image data of the height map, and the value (brightness, etc.) of that pixel in the two-dimensional image data captured at the position P(i) is set as the value of the pixel at the same position in the composite image data of the two-dimensional image data. By making judgments using conditions (1) and (2) in this way, it is possible to prevent erroneous measurements, especially near the seams of the composite.

[0076] When the image capturing processing unit 32a has performed the above-described composition processing for all pixels, the control unit 30 determines whether the image capturing processing has been completed for all of the predetermined image capturing positions in the Z direction (step S114). If it is determined that the image capturing processing has not been completed for all of the image capturing positions (step S114: No), the control unit 30 causes the image capturing processing unit 32a to set the position of the camera unit 21 to the next image capturing position closer to the reference plane (step S116), and repeats the processing of steps S108 to S114 described above at that image capturing position. For example, if the current image capturing position is P(3)=12, the camera unit 21 is set to the next image capturing position, P(2)=8, and if the current image capturing position is P(2)=8, the camera unit 21 is set to the next image capturing position, P(1)=4.

[0077] On the other hand, if it is determined that the imaging process has been completed at all imaging positions (step S114: Yes), the control unit 30 causes the image composition unit 32c to perform composite image update processing of the image data (image data of P(0)=0) acquired in step S100 when the camera unit 21 is focused near the reference plane (step S118). Here, since condition (2) is always satisfied for the image data of P(0), the determination is made based on whether condition (1) is satisfied. That is, in step S118, for pixels for which a value for composite image data has not yet been selected and which satisfy condition (1), the pixel value of the image data at the imaging position of P(0) (height information in the height map and pixel value of the two-dimensional image data) is selected as the pixel value of the composite image data.

[0078] Note that, in the repeated processing of steps S108 to S114, even if a certain pixel satisfies conditions (1) and (2), if an imaging position farther from the object under test 12 than the imaging position already satisfies conditions (1) and (2) and the value of the pixel has been selected as the value of the composite image data, the value that has already been selected takes precedence. That is, for a pixel that satisfies conditions (1) and (2) at multiple imaging positions in the Z direction, the value of the pixel at the imaging position farthest from the object under test 12 is selected. The light irradiated by the projection unit 23 is reflected by the object under test 12 and then captured. As the camera unit 21 moves farther from the substrate surface, the amount of light reflected from the object under test 12 decreases. This leads to a reduction in noise due to unnecessary light, particularly at out-of-focus pixels, and as a result, erroneous measurement values ​​included in the height information are reduced. Therefore, for a pixel that satisfies conditions (1) and (2) at multiple imaging positions in the Z direction, the value of the pixel at the imaging position farthest from the object under test 12 is selected.

[0079] 5 shows the relationship between the most focused position FP of the camera unit 21, the image data captured at that position, and the composite image data when the object under inspection 12 is a portion where components 12a, 12b, and 12c are mounted on a board 12d in a certain field of view. Specifically, the upper rows of FIGS. 5(a) to 5(d) show the relationship between the most focused position FP of the camera unit 21 and the components 12a to 12d, and the lower rows show, in shaded areas, the portions (pixels) that may be selected as composite image data. FIG. 5(a) shows the case where the camera unit 21 is at P(3)=12. Since the top surface of component 12a satisfies condition (1), if condition (2) is also satisfied, the top surface of component 12a in the image data at P(3) is selected as composite image data, as shown in FIG. 5(e). On the other hand, the top surface of component 12b satisfies condition (1) (assuming that condition (2) is also satisfied) when camera unit 21 is at P(2)=8 as shown in FIG. 5(b) and when camera unit 21 is at P(1)=4 as shown in FIG. 5(c). In this case, the top surface of component 12b in the image data at P(2), which is farther from the object under inspection 12, is selected as the composite image data, as shown in FIG. 5(e). Furthermore, the top surface of component 12c satisfies condition (1) when camera unit 21 is at P(1)=4 as shown in FIG. 5(c). Therefore, if condition (2) is also satisfied at the same time, the top surface of component 12c in the image data at P(1) is selected as the composite image data, as shown in FIG. 5(e). Similarly, the top surface (reference surface) of substrate 12d satisfies condition (1) when camera unit 21 is at P(0)=0, as shown in Figure 5(d), and therefore if condition (2) is also satisfied at the same time, the top surface of substrate 12d in the image data of P(0) is selected as the composite image data, as shown in Figure 5(e) (see Figure 5(e)).

[0080] Returning to FIG. 4, finally, the control unit 30 causes the height measurement unit 32b to execute height post-processing S300 (step S120). When the height measurement unit 32b of the control unit 30 starts height post-processing S300 shown in FIG. 4(c), it executes a process to interpolate unmeasured pixels in the composite image data, such as pixels whose values ​​were not selected in the composite image update process described above (step S302), and then executes a process to correct distortion in the composite image data (step S304), terminating height post-processing S300 and terminating the synthesis process. If height post-processing is executed on the image data before synthesis (a height map calculated from pattern image data acquired at each imaging position), interpolated or corrected pixels may be selected as pixel values ​​in the composite image data, potentially reducing the accuracy of the image data. Therefore, it is desirable to execute height post-processing, such as unmeasured pixel interpolation and distortion correction, on the composite image data after the synthesis process for image data at all imaging positions has been completed.

[0081] As shown in FIG. 4 , in the image data synthesis method according to this embodiment, height basic processing and composite image update processing are performed each time the camera unit 21 is moved in the Z direction to acquire image data, except for image data captured when the camera unit 21 is focused near the reference plane (at position P(0) where i = 0). In FIG. 4 , the imaging processing and the synthesis processing (height basic processing and composite image update processing) of the image data acquired in the imaging processing are described as a series of steps. However, while imaging processing at the next imaging position is being performed, height basic processing and composite image update processing can be performed in parallel on the image data acquired at the previous imaging position. As described above, in the imaging processing, multiple pattern image data with shifted stripe pattern phases must be acquired for each of the multiple projection units 23, and multiple 2D image data must be acquired by sequentially lighting the side illumination sources 22 a, 22 b, and 22 c of the illumination unit 22. Therefore, by performing the height basic processing and composite image update processing at the previous imaging position in parallel while performing the imaging processing at the next imaging position, the overall processing time can be shortened, and as a result, the inspection time can be shortened. In this case, in Figure 4, after performing the imaging process and basic height process at position P(0), the camera unit 21 is moved to P(3), which is the farthest from the object under test 12, and the camera unit 21 is moved from this imaging position in a direction approaching the object under test 12 to perform the imaging process, basic height process, and composite image update process, and finally the composite image update process of the image data of P(0) is performed.By performing the imaging process, height basic processing, and composite image update processing in this order, if a pixel value of the composite image data has already been selected in the height basic processing and composite image update processing at an imaging position prior to the imaging position at which the height basic processing and composite image update processing are currently being performed, the pixel value of that pixel will not be selected as the value of the corresponding pixel of the composite image data even if the image data at the imaging position at which the height basic processing and composite image update processing are currently being performed satisfies conditions (1) and (2), thereby complying with the above-mentioned judgment criterion that "if pixels at the same position in image data at two or more imaging positions satisfy conditions (1) and (2), the pixel value of the image data at the imaging position farthest from the subject 12 among these image data will be selected as the pixel value of the composite image data."

[0082] Note that the above-mentioned processing is performed for one field of view, and when the inspected object 12 is divided into multiple fields of view and imaged (inspected), the camera unit 21 is moved in the XY direction and the above-mentioned processing is performed for each field of view.

[0083] The inspection device 10 according to this embodiment performs the above-described inspection based on the composite image data (height map and two-dimensional image data) acquired by the image processing unit 32 of the inspection control unit 31 of the control unit 30.

[0084] (First variation of the conditions for selecting pixels) In the above-described composite image update process, the determination is made based on the amplitude of the target pixel under condition (2), but the determination may be made based on the following condition (3) instead of condition (2). That is, the above-described conditions (1) and (3) may be used to determine whether or not to select a pixel of image data at the current imaging position as composite image data based on height information of that pixel at the current imaging position. Note that contrast C is calculated as C=A / B using amplitude A and offset B obtained from the above-described formula (a).

[0085] C(X,Y,i) ≧ C(X,Y,0)×WC (3) however, C(X,Y,i): Contrast at the current imaging position when the pixel to be judged is located at (X,Y) C(X,Y,0): Contrast at i=0 when the target pixel position is (X,Y) WC: Weight Count

[0086] Condition (3) determines whether a pixel has a high degree of focus based on the contrasts C(X,Y,i) and C(X,Y,0) obtained based on the above-described formula (a) when calculating the phase using pattern image data. Condition (3) compares the focus degrees to extract pixels with higher focus degrees, and determines that the pixel is in focus when its evaluation value is equal to or greater than a threshold value. Specifically, since a higher contrast value indicates a higher degree of focus, the pixel being determined (pixel at position (X,Y)) can be determined to be in focus when the contrast C(X,Y,i) at the current imaging position is sufficiently greater than the contrast C(X,Y,0) when i=0. The value of the weight WC is preferably determined based on the depth of field of the camera unit 21.

[0087] (Modification of height measurement method) In the above description, height information is calculated from three or four pieces of pattern image data obtained by projecting one stripe pattern from the projection unit 23 with different phases. However, to improve the accuracy of the height information, height information can also be acquired using at least two stripe patterns with different periods. For example, in the imaging process of steps S100 and S108 described above, the projection unit 23 projects a first pattern with a long period while changing its phase to acquire three or four pieces of pattern image data, and projects a second pattern with a short period while changing its phase to acquire three or four pieces of pattern image data. If there are multiple projection units 23, each projection unit 23 acquires pattern image data with different phases of the first and second patterns. Then, rough height information (rough height) is acquired using the first pattern with a long period (i.e., thick stripes), and precise height information is acquired using the second pattern with a short period (thin stripes). As described above, the PMP method acquires height information based on phase. Therefore, if there is a large height gap (height difference) and the stripes are shifted by more than one period, height information cannot be uniquely identified. By acquiring height information using the first pattern in combination, it becomes possible to uniquely identify height information even if the stripes are shifted by one period or more when the second pattern is projected.

[0088] The luminance In obtained from the three or four pattern image data obtained by projecting the first pattern with the projection unit 23 while changing the phase, and the luminance In obtained from the three or four pattern image data obtained by projecting the second pattern with the projection unit 23 while changing the phase, both satisfy the relationship shown in the above-mentioned formula (a). Moreover, the relationship between these luminance In and the phase φ satisfies the relationship shown in the above-mentioned formula (b).

[0089] Here, when the phase of the long-period first pattern is φw and the phase of the reference plane is φw0, the phase difference Δφw has the relationship of the following equation (c1) from the above equation (c).

[0090] Δφw = φw - φw0 (c1)

[0091] Similarly, when the phase of the short-period second pattern is φf and the phase of the reference plane is φf0, the phase difference Δφf has the relationship of the following equation (c2) from the above equation (c).

[0092] Δφf = φf - φf0 (c2)

[0093] From these phase differences Δφw and Δφf, detailed height information H can be obtained using the following equation (d'): where M is the number of periods of the second short-period pattern relative to one period of the first long-period pattern, and Round() is a function that rounds off to the nearest integer.

[0094] H = Round((M×Δφw-Δφf) / 2π)×2π+Δφf (d′)

[0095] In addition, in the case of a configuration in which height information is calculated using two stripe patterns, one with a long period and one with a short period, it is desirable to make a judgment based on the amplitude ratio as shown in the following condition (2') instead of the judgment based on the amplitude as shown in the above-mentioned condition (2). In other words, it is desirable to determine whether or not to select a pixel of image data at the current imaging position as composite image data based on the above-mentioned conditions (1) and (2').

[0096] AR(X,Y,i) ≦ AR(X,Y,0)×WAR (2′) however, AR(X,Y,i): Amplitude ratio at the current imaging position when the position of the pixel to be judged is (X,Y) AR(X,Y,0): Amplitude ratio at i=0 when the pixel position to be judged is (X,Y) WAR: Weighted Count

[0097] The contrast ratio AR (parameters (X, Y, i) are omitted) of a pixel at position (X, Y) at imaging position P(i) is calculated based on the following formula (e): where Aw is the amplitude obtained by formula (a) from three or four pattern image data obtained by projecting a first pattern with a long period, and Af is the amplitude obtained by formula (a) from three or four pattern images obtained by projecting a second pattern with a short period.

[0098] AR = (Aw-Af) / Af (e)

[0099] As with condition (2), condition (2') compares the focus degrees to extract pixels with higher focus degrees, and determines that a pixel is in focus when its evaluation value is equal to or less than a threshold. Specifically, since the smaller the amplitude ratio, the higher the focus degree. Therefore, the pixel to be determined (pixel at position (X, Y)) can be determined to be in focus when the amplitude ratio AR(X, Y, i) at the current imaging position is sufficiently smaller than the amplitude ratio AR(X, Y, 0) when the camera unit 21 is focused near the reference plane (when i = 0). Note that the value of the weight WAR is preferably determined based on the depth of field of the camera unit 21.

[0100] Alternatively, instead of condition (2'), the following condition (3') may be used for the determination. That is, whether or not to select pixels of image data at the current imaging position as composite image data may be determined based on the above-described conditions (1) and (3').

[0101] CR(X,Y,i) ≦ CR(X,Y,0)×WCR (3′) however, CR(X,Y,i): Contrast ratio at the current imaging position when the pixel position to be judged is (X,Y) CR(X,Y,0): Contrast ratio at i=0 when the pixel position is (X,Y) WCR: Weight coefficient

[0102] Condition (3') determines whether a pixel is in focus based on the contrast ratio obtained based on the above-mentioned formula (a) when calculating the phase using pattern image data. This condition (3') compares the focus degrees to extract pixels with higher focus degrees, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or less than a threshold. Specifically, since the smaller the contrast ratio value, the higher the focus degree, it can be determined that the pixel to be determined is in focus when the contrast ratio CR(X,Y,i) at the current imaging position is sufficiently smaller than the contrast ratio CR(X,Y,0) when the camera unit 21 is focused near the reference plane (when i=0).

[0103] The contrast ratio CR (parameters (X, Y, i) are omitted) is obtained from the contrast Cw obtained from three or four pattern image data obtained by projecting a long-period first pattern and the contrast Cf obtained from three or four pattern images obtained by projecting a short-period second pattern using the following formula (f): where Aw is the amplitude obtained from the first pattern, Af is the amplitude obtained from the second pattern, Bw is the offset obtained from the first pattern, and Bf is the offset obtained from the second pattern, and these values ​​are obtained from the above formula (a).

[0104] CR = (Cw-Cf) / Cf (f) however, Cw = Aw / Bw (g) Cf = Af / Bf (h)

[0105] In this way, in the above-described composite image update process S112, even if a pixel that simultaneously satisfies conditions (1) and (3') is selected, the focused pixel can be set as the pixel value of the composite image data.

[0106] (Second Modification of the Condition for Selecting Pixels) In the above-described image data compositing method, in the composite image update process S112, when there is one type of stripe pattern for acquiring height information, the determination of whether to select a pixel of the image data captured at the current imaging position as a pixel of the composite image data is configured to select a pixel that simultaneously satisfies conditions (1) and (2), or conditions (1) and (3), and when there are two types of stripe patterns, long period and short period, the determination is configured to simultaneously satisfy conditions (1) and (2'), or conditions (1) and (3'). However, it is also possible to configure the image data captured at the current imaging position to select a pixel as a pixel of the composite image data when condition (1) is satisfied, without determining conditions (2), (2'), (3), or (3'). This is because condition (1) alone can determine whether a pixel of the image data at the current imaging position is an in-focus pixel, further shortening the processing time. In addition, when configured in this manner, the imaging process at the imaging position P(0) when the focus is near the reference plane may be performed last (in other words, the imaging process, basic height process, and composite image update process may be performed from an imaging position far from the object under test 12).

[0107] Furthermore, when the imaging process, basic height process, and composite image update process are performed from an imaging position far from the object under test 12, the image data when the camera unit 21 is focused near the reference plane (P(0)) is acquired last, so instead of the above-mentioned condition (2), the judgment may be made based on the following condition (2a).

[0108] A(X,Y,i) ≧ TH (2a) however, TH: Threshold value determined in advance from the amplitude A

[0109] Furthermore, when the above-mentioned condition (2') is used for the judgment, the following condition (2a') may be used instead of the condition (2').

[0110] AR(X,Y,i) ≦ TH (2a′) however, TH: Threshold value determined in advance from the amplitude ratio AR

[0111] Furthermore, when the above-mentioned condition (3) is used for the judgment, the following condition (3a) may be used for the judgment instead of this condition (3).

[0112] C(X,Y,i) ≧ TH (3a) however, TH: Predetermined threshold from contrast C

[0113] Furthermore, when the above-mentioned condition (3') is used for the judgment, the following condition (3a') may be used for the judgment instead of this condition (3').

[0114] CR(X,Y,i) ≦ TH (3a′) however, TH: Threshold value determined in advance from the contrast ratio CR

[0115] Even if conditions (2a), (2a'), (3a), and (3a') are used for the judgment instead of conditions (2), (2') or conditions (3), (3'), it is possible to extract pixels with a higher degree of focus from among pixels that satisfy condition (1).

[0116] (Main features and effects) The main features and effects of the image data synthesis method according to this embodiment are summarized below.

[0117] First, in the image data synthesis method according to this embodiment, for one field of view, composite image data is obtained by synthesizing multiple pieces of image data captured by moving the camera unit 21 to different positions in the Z direction (imaging direction), but the synthesis process is performed on image data captured at an imaging position of the camera unit 21 that is far from the object under test 12 (from the reference plane). With this configuration, as the camera unit 21 moves farther away from the object under test 12, the amount of reflected light from the object under test 12 decreases, and noise due to unnecessary light in out-of-focus pixels in particular decreases, thereby reducing erroneous measurement values ​​included in height information.

[0118] Second, in the image data synthesis method according to this embodiment, in-focus pixels are selected from the image data at each imaging position and used as pixel values ​​for the synthesized image data. Criteria for determining in-focus pixels include height information for each pixel measured by the PMP method, amplitude, and offset in the PMP method. The methods include selecting pixels that simultaneously satisfy the above-described conditions (1) and (2), or pixels that simultaneously satisfy conditions (1) and (3), selecting pixels that only satisfy condition (1), selecting pixels that simultaneously satisfy conditions (1) and (2′), or conditions (1) and (3′), and selecting pixels that simultaneously satisfy condition (1) and any of conditions (2a), (2a′), (3a), and (3a′). This configuration allows for accurate selection of in-focus pixels through simple calculations. As described in the first feature, among pixels that satisfy the in-focus condition in the image data at different imaging positions, the value of the imaging position farthest from the object under test 12 (from the reference plane) is selected.

[0119] Third, in the image data synthesis method according to this embodiment, when height information is acquired from pattern image data using the PMP method, the above-described synthesis process is performed on the pattern image data, and then the synthesized image data is subjected to interpolation of unmeasured pixels and distortion correction (height post-processing). If interpolation or correction is performed on a height map calculated from pattern image data captured at each imaging position, even the interpolated or corrected pixel values ​​(unreliable data) may be selected as pixel values ​​of the synthesized image data, which may result in a decrease in the accuracy of the image data. However, by performing interpolation and correction on the synthesized image data (height map), interpolation and correction are performed on focused image data, making it possible to acquire highly accurate height information.

[0120] Fourth, in the image data synthesis method according to this embodiment, the parameter for removing erroneous measurement values ​​is set to a value that maximizes the number of measurement points on the reference surface (substrate surface) when the camera unit 21 is focused near the reference surface (when P(0)), and is set to a value that removes points that are highly suspected of being measurement errors when the camera unit 21 is not focused near the reference surface (when P(1) to P(3)). By configuring in this way, it is possible to acquire focused synthetic image data, improving the accuracy of height information.

[0121] Fifth, the image data synthesis method according to this embodiment is configured to perform image data imaging processing in ascending order of the Z-direction position of the camera unit 21 from the object under test 12, and to execute height basic processing and composite image update processing for each imaging. With this configuration, while imaging processing is being executed at the next imaging position, height basic processing and composite image update processing can be executed in parallel using image data captured at the previous imaging position, thereby shortening the processing time.

[0122] With the above configuration, it is possible to acquire focused image data even for an object to be inspected 12 having tall components mounted on a substrate, thereby improving the accuracy of the height information of the height map calculated from the pattern image data and obtaining clear two-dimensional image data.

[0123] It goes without saying that the above embodiments do not limit the invention described in the claims, and that not all of the combinations of the characteristic features described in the embodiments are necessarily essential features of the solution. [Explanation of symbols]

[0124] 10 Inspection equipment (image processing equipment) 12 Test subject 21 Camera unit (imaging unit) 22 Lighting unit (lighting section) 23 Projection unit (projection section) 30 Control unit (control section)

Claims

1. an imaging unit capable of changing a relative position in an imaging direction with respect to the object under inspection; A control unit, The control unit is a first step of performing a basic height process of acquiring reference position image data by capturing an image with the imaging unit at a position when the imaging unit is focused on a predetermined reference plane on the object to be inspected, and acquiring height information of the object to be inspected for each pixel of the reference position image data; a second step of setting, as a current position, a position in an imaging direction different from a position when the focus is on the predetermined reference plane and among predetermined positions, the farthest position from the object to be inspected; a third step of acquiring current position image data by capturing an image with the imaging unit at the current position, and executing the height basic processing for each pixel of the current position image data to acquire height information of the test object; a fourth step of executing a composite image update process for determining whether or not a condition for a pixel being in focus is satisfied for each pixel of the current position image data based on luminance information for each pixel of the reference position image data and the current position image data, which are included in the height information, and information on the distance between the test object and the most focused position of the imaging unit for each pixel of the current position image data, and selecting a value of the pixel in the current position image data as a value of a corresponding pixel in the composite image data when it is determined that the pixel satisfies the condition; a fifth step of repeatedly executing the third step and the fourth step, with a next position in a direction approaching the predetermined reference plane from the current position as a current position, among the predetermined positions; a sixth step of setting a position when the predetermined position is focused on the predetermined reference plane as a current position when it is determined in the fifth step that the third step and the fourth step have been performed at all of the predetermined positions, and of executing the composite image update process for each pixel of the reference position image data, and selecting the value of the pixel of the reference position image data as the value of the corresponding pixel of the composite image data when it is determined that the pixel satisfies the condition. Image processing device.

2. a projection unit that projects a stripe pattern, in which a pattern whose phase changes so as to monotonically increase or decrease is repeatedly formed, onto the object under inspection; The control unit is The reference position image data or the current position image data acquired by projecting the stripe pattern by the projection unit is acquired, and the height information is acquired by the height basic processing. The image processing device according to claim 1 .

3. The control unit is In the composite image update process, information regarding amplitude included in the luminance information of the pixel of the current position image data is compared with information regarding amplitude included in the luminance information of the pixel of the reference position image data to determine whether the pixel satisfies the condition. The image processing device according to claim 2 .

4. The control unit is In the height basic processing, The height information is acquired for the reference position image data so as to maximize the number of measurement points; The height information is acquired so as to exclude pixels that are highly suspected of having measurement errors from the current position image data. The image processing device according to claim 3 .

5. The inspection apparatus further includes an illumination unit that projects illumination light onto the inspection object, each of the reference position image data and the current position image data includes pattern image data acquired by the imaging unit after the stripe pattern is projected by the projection unit, and two-dimensional image data acquired by the imaging unit after the illumination light is projected by the illumination unit; The control unit is In the basic height processing, the height information is obtained from the pattern image data; In the composite image update process, when it is determined that the pixel satisfies the condition, the height information of the pixel in the pattern image data is selected as the value of the corresponding pixel in the composite image data of the height map, and the value of the pixel in the two-dimensional image data is selected as the value of the corresponding pixel in the composite image data of the two-dimensional image data. The image processing device according to any one of claims 2 to 4.

6. The control unit is The height post-processing such as non-measured pixel interpolation processing and distortion correction processing is performed on the composite image data of the height map synthesized by the synthetic image update processing. The image processing device according to claim 5 .

7. The control unit is In the composite image update process, when it is determined that the same pixel satisfies the condition in two or more of the reference position image data or the current position image data when the imaging unit is at different positions in the imaging direction, the value of the pixel in the image data in which the imaging unit is located at the farthest position from the subject is selected as the value of the corresponding pixel in the composite image data. The image processing device according to any one of claims 1 to 6.

Images