Image processing apparatus

Abstract

To provide an image processing apparatus using an image data composition method that can compose a plurality of pieces of image data acquired by an imaging unit at different positions in an imaging direction accurately in a short time.SOLUTION: An inspection device 10 being an image processing apparatus has a camera unit 21 and a control unit 30. The control unit 30 changes relative positions in an imaging direction of the camera unit 21 and an inspected object 12 to acquire at least two pieces of image data picked up by the camera unit 21 at different positions in the imaging direction, and executes composite image update processing of determining if pixels of the image data satisfy a predetermined condition and selecting values of the pixels determined to satisfy the predetermined condition as values of corresponding pixels of composite image data.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus using an image data synthesis method.

Background Art

[0002] In recent years, electronic circuit boards have been mounted on various devices. However, in devices on which this type of electronic circuit board is mounted, miniaturization, thinning, etc. have always been issues. From this point, it is required to increase the mounting density of electronic circuit boards. In an inspection apparatus for inspecting the soldering state of such an electronic circuit board, the mounting state of electronic components, etc., a three-dimensional measurement apparatus based on a phase shift method has been proposed that uses a plurality of optical patterns with different periods to expand the measurement range and shorten the measurement time (for example, see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above method, due to the depth of field of the imaging unit, there is a limit to the expansion of the measurement range. On the other hand, it is known that a plurality of measurement data can be joined by measuring from a plurality of imaging positions and performing coordinate conversion according to the positional relationship. However, there is a problem that if the measurement results calculated from a plurality of imaging positions are simply synthesized, a large number of measurement errors are included, especially near the joint of the synthesis.

[0005] The present invention has been made in view of such problems, and an object thereof is to provide an image processing apparatus using a method for synthesizing a plurality of pieces of image data acquired by imaging units at different positions in the imaging direction (Z direction) with high accuracy and in a short time.

Means for Solving the Problems

[0006] In order to solve the above problems, the present invention The first aspect The image processing apparatus according to the present invention includes an imaging unit capable of changing the relative position in the imaging direction with respect to the subject, and a control unit. The control unit changes the relative position in the imaging direction between the imaging unit and the subject, and acquires the image data of the subject imaged by the imaging unit at different positions in the imaging direction, which is the first image data when the imaging unit is in focus on the reference plane of the subject, and at least one piece of second image data when the imaging unit is in focus on the side closer to the imaging unit than the reference plane. Each of the first image data and the second image data has two-dimensional image data of the subject and pattern image data of the subject onto which a periodically changing stripe pattern is projected. From each of the pattern image data of the first image data and the pattern image data of the second image data, an amplitude, an offset value, and a phase of luminance values calculated for each pixel by a phase shift method, and a height map including height information calculated from the phase are created. Select one by one from the second image data in the order from the position of the imaging unit when the second image data is acquired being far from the reference plane as comparison target image data, and the first image data The height map And the comparison target image data The height map Compare the values of the pixels at the same position to determine whether the value of the pixel of the comparison target image data satisfies the condition of the in-focus pixel, When the height information of the pixels of the comparison target image data is within a predetermined range including the best focus position of the imaging unit when the subject acquired the comparison target image data, and the amplitude of the pixels of the comparison target image data is equal to or greater than a value obtained by multiplying the amplitude of the pixels at the same position in the first image data by a predetermined weighting factor, it is determined that the condition is satisfied. The comparison target image data The two-dimensional image data Execute a composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data, use the first image data as the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition, The height map Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data. When the height information of the comparison target image data is within a predetermined range including the best focus position of the imaging unit when the subject acquired the comparison target image data, it is determined that the condition is satisfied. The comparison target image data The two-dimensional image data Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data.

[0007] In order to solve the above problems, the present invention The second aspectThe image processing apparatus according to the present invention includes an imaging unit capable of changing the relative position in the imaging direction with respect to a subject, and a control unit. The control unit changes the relative position in the imaging direction between the imaging unit and the subject, and acquires image data of the subject captured by the imaging unit at different positions in the imaging direction, including first image data when the imaging unit is in focus on a reference plane of the subject, and at least one piece of second image data when the imaging unit is in focus on the side closer to the imaging unit than the reference plane. Each of the first image data and the second image data has two-dimensional image data of the subject and pattern image data of the subject onto which a periodically changing stripe pattern is projected. From each of the pattern image data of the first image data and the pattern image data of the second image data, an amplitude, an offset value, and a phase of luminance values calculated for each pixel by a phase shift method, and a height map including height information calculated from the phase are created. Select, one by one in descending order of the position of the imaging unit from the reference plane when the second image data is acquired from the second image data, the second image data as comparison target image data, and the first image data The height map and the comparison target image data The height map Compare the values of pixels at the same position, and determine whether the value of the pixel of the comparison target image data satisfies the condition of a focused pixel. When the height information of the pixels of the comparison target image data is within a predetermined range including the best focus position of the imaging unit when the subject acquired the comparison target image data, and the contrast calculated from the amplitude and the offset value of the pixels of the comparison target image data is equal to or greater than a value obtained by multiplying the contrast calculated from the amplitude and the offset value of the pixels at the same position in the first image data by a predetermined weighting factor, it is determined that the condition is satisfied. the comparison target image data The two-dimensional image data Execute a composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data, use the first image data as the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition. The height map Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data. When it is determined that the condition is satisfied when the height information of the pixels of the comparison target image data is within a predetermined range including the optimal focus position of the imaging unit when the subject acquired the comparison target image data, the comparison target image data of the two-dimensional image data Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data.

[0008] To solve the above problems, the present invention of the third aspect The image processing apparatus according to the present invention includes an imaging unit capable of changing the relative position in the imaging direction with respect to a subject, and a control unit. The control unit changes the relative position in the imaging direction between the imaging unit and the subject, and acquires image data of the subject captured by the imaging unit at different positions in the imaging direction, including first image data when the imaging unit is in focus on a reference plane of the subject, and at least one piece of second image data when the imaging unit is in focus on the side closer to the imaging unit than the reference plane. Each of the first image data and the second image data includes two-dimensional image data of the subject, first pattern image data of the subject onto which a first stripe pattern that changes in a first period is projected, and second pattern image data of the subject onto which a second stripe pattern that changes in a second period different from the first period is projected. For each of the first image data and the second image data, a first amplitude and a first phase of a first luminance value calculated for each pixel from the first pattern image data by a phase shift method, a second amplitude and a second phase of a second luminance value calculated for each pixel from the second pattern image data by a phase shift method, and height information calculated for each pixel from the first phase and the second phase by a phase shift method are included to create a height map. Select one by one the second image data in the order of the position of the imaging unit when the second image data is acquired from the second image data being farther from the reference plane as comparison target image data, and the first image data of the height map and the comparison target image data of the height map Compare the values of the pixels at the same positions of the first image data and the comparison target image data to determine whether the values of the pixels of the comparison target image data satisfy the condition of being in focus, When it is determined that the condition is satisfied when the height information of the pixels of the comparison target image data is within a predetermined range including the optimal focus position of the imaging unit when the subject acquired the comparison target image data, and the amplitude ratio, which is the ratio of the first amplitude and the second amplitude of the pixels of the comparison target image data, is equal to or greater than a value obtained by multiplying the amplitude ratio, which is the ratio of the first amplitude and the second amplitude of the pixels at the same position of the first image data, by a predetermined weight count. Execute a composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data, use the first image data as the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition, of the two-dimensional image data Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data, use the first image data as the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition, of the height map Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data. When it is determined that the condition is satisfied when the height information of each pixel of the comparison target image data is within a predetermined range including the optimal focus position of the imaging unit when the subject acquired the comparison target image data. the comparison target image data of the two-dimensional image data Execute the composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data.

[0009] To solve the above problems, the present invention of the fourth aspect The image processing apparatus according to the present invention includes an imaging unit capable of changing the relative position in the imaging direction with respect to the subject, and a control unit. The control unit changes the relative position in the imaging direction between the imaging unit and the subject, and obtains the image data of the subject imaged by the imaging unit at different positions in the imaging direction, including the first image data when the imaging unit is in focus on the reference plane of the subject, and at least one second image data when the imaging unit is in focus on the imaging unit side from the reference plane, Each of the first image data and the second image data includes two-dimensional image data of the subject, first pattern image data of the subject onto which a first stripe pattern that changes in a first period is projected, and second pattern image data of the subject onto which a second stripe pattern that changes in a second period different from the first period is projected. For each of the first image data and the second image data, a first amplitude, a first offset value, and a first phase of first luminance values calculated for each pixel by a phase shift method from the first pattern image data, a second amplitude, a second offset value, and a second phase of second luminance values calculated for each pixel by the phase shift method from the second pattern image data, and height information calculated for each pixel by the phase shift method from the first phase and the second phase are used to create a height map. Select one by one the second image data in the order of the position of the imaging unit when the second image data is acquired from the second image data being farther from the reference plane as comparison target image data, and the first image data of the height map and the comparison target image data of the height map Compare the values of the pixels at the same positions of the first image data and the comparison target image data to determine whether the values of the pixels of the comparison target image data satisfy the condition of being in focus, When the contrast ratio, which is the ratio of a first contrast calculated from the first amplitude and the first offset value of the pixel of the comparison target image data to a second contrast calculated from the second amplitude and the second offset value, is equal to or greater than a value obtained by multiplying the contrast ratio of the first contrast calculated from the first amplitude and the first offset value of the pixel at the same position in the first image data to the second contrast calculated from the second amplitude and the second offset value by a predetermined weight count, and the height information of the pixel of the comparison target image data is within a predetermined range including the optimal focusing position of the imaging unit when the subject acquired the second image data, it is determined that the condition is satisfied. the comparison target image data of the two-dimensional image data Execute a composite image update process of selecting the value of the pixel of the comparison target image data as the value of the corresponding pixel of the composite image data, use the first image data as the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition, of the height map Determine whether the pixel value satisfies the condition, When the height information of the pixel of the comparison target image data is within a predetermined range including the optimal focusing position of the imaging unit when the subject acquired the second image data, it is determined that the condition is satisfied. the comparison target image data of the two-dimensional image data execute the composite image update process of selecting the pixel value of the pixel as the corresponding pixel value of the composite image data.

[0010] However, of the image processing apparatus according to the first to fourth aspects in the composite image update process, when it is determined that the pixels at the same position in two or more comparison target image data satisfy the condition, among the comparison target image data, the pixel value of the comparison target image data captured when the imaging unit is at the position farthest from the subject is selected as the corresponding pixel value of the composite image data. Note that the focal length of the imaging unit is fixed.

[0011] In the image processing apparatus according to the present invention, when the imaging unit is in focus on the reference plane of the subject, the control unit the first acquires the image data, and then preferably acquires the image data from the position where the imaging unit is farther from the subject. the second It is preferable to acquire the image data.

[0012] In the image processing apparatus according to the present invention, it is preferable that the control unit the second executes the composite image update process every time the image data is acquired.

Effect of the Invention

[0013] According to the present invention, it is possible to provide an image processing apparatus using a method for synthesizing image data that can accurately synthesize a plurality of image data acquired by imaging units at different positions in the imaging direction (Z direction) in a short time.

Brief Description of the Drawings

[0014]

FIG. 1

FIG. 2

FIG. 3

FIG. 4

FIG. 5

Embodiments for Carrying Out the Invention

[0015] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. First, the configuration of the inspection apparatus 10 according to the present embodiment will be described with reference to FIG. 1. This inspection apparatus 10 is an apparatus that inspects the object 12 to be inspected using the image data (2D image data or pattern image data) of the object 12 to be inspected obtained by imaging the object 12 to be inspected. Therefore, this inspection apparatus 10 also has a function as an image processing apparatus. The object 12 to be inspected is, for example, an electronic circuit board on which components are mounted and solder is applied.

[0016] The inspection apparatus 10 includes an inspection table 14 for holding the object 12 to be inspected, an imaging unit 20 for illuminating and imaging the object 12 to be inspected, an XY stage 16 for relatively moving the imaging unit 20 with respect to the inspection table 14, and a control unit 30 for controlling the operations of the imaging unit 20 and the XY stage 16 and performing the inspection of the object 12 to be inspected. For convenience of explanation, as shown in FIG. 1, the object placement surface of the inspection table 14 is defined as the XY plane, and the direction perpendicular to the placement surface (that is, the imaging direction by the camera unit 21 constituting the imaging unit 20 (the optical axis direction of the optical system of the camera unit 21)) is defined as the Z direction.

[0017] The imaging unit 20 is attached to a moving table (not shown) of the XY stage 16 and is movable in the X direction and the Y direction by the XY stage 16. The XY stage 16 is, for example, a so-called H-type XY stage. Thus, the XY stage 16 includes a Y drive unit that moves the moving table in the Y direction along a Y guide extending in the Y direction, and two X guides and an X drive unit that support the Y guide at both ends thereof and are configured to be movable in the X direction together with the moving table and the Y guide. The XY stage 16 further includes a Z movement mechanism that moves the imaging unit 20 in the Z direction. Further, the imaging unit 20 may further include a rotation mechanism that rotates the imaging unit 20. The inspection apparatus 10 may further include an XY stage that enables the inspection table 14 to move. In this case, the XY stage 16 that moves the imaging unit 20 may be omitted. Further, a linear motor or a ball screw can be used for the X drive unit and the Y drive unit. Further, instead of moving the imaging unit 20 in the Z direction, the XY stage 16 may be configured to move in the Z direction.

[0018] The imaging unit 20 includes a camera unit 21 that images from a direction perpendicular (Z direction) to the inspection surface (substrate surface) of the object to be inspected 12, an illumination unit 22, and a projection unit 23. In the inspection apparatus 10 according to the present embodiment, the camera unit 21, the illumination unit 22, and the projection unit 23 may be configured as an integrated imaging unit 20. In this integrated imaging unit 20, the relative positions of the camera unit 21, the illumination unit 22, and the projection unit 23 may be fixed, or each unit may be configured to be relatively movable. Further, the camera unit 21, the illumination unit 22, and the projection unit 23 may be separated and configured to be movable separately.

[0019] The camera unit 21 includes an image sensor that generates a two-dimensional image of the object, and an optical system (for example, a lens) that forms an image on the image sensor. This camera unit 21 is, for example, a CCD camera. The maximum field of view of the camera unit 21 may be smaller than the inspection object placement area of the inspection table 14. In this case, the camera unit 21 images the entire inspection object 12 by dividing it into a plurality of partial images. The control unit 30 controls the XY stage 16 so that the camera unit 21 is moved to the next imaging position each time the camera unit 21 images a partial image. The control unit 30 synthesizes the partial images to generate an entire image of the inspection object 12.

[0020] Note that the camera unit 21 may be provided with an image sensor that generates a one-dimensional image instead of a two-dimensional image sensor. In this case, by scanning the inspection object 12 with the camera unit 21, an entire image of the inspection object 12 can be obtained.

[0021] The illumination unit 22 is configured to project illumination light for imaging by the camera unit 21 onto the surface of the inspection object 12. The illumination unit 22 includes one or more light sources that emit light having a wavelength or wavelength range selected from a wavelength range detectable by the image sensor of the camera unit 21. The illumination light is not limited to visible light, and ultraviolet light, X-rays, etc. may be used. When a plurality of light sources are provided, each light source is configured to project light of different wavelengths (for example, red, blue, and green) onto the surface of the inspection object 12 at different light projection angles.

[0022] The inspection object 12 illuminated by the illumination unit 22 is imaged by the camera unit 21. The inspection apparatus 10 determines the presence or absence of defects on the substrate of the inspection object 12 (for example, the presence or absence of components, whether the arrangement is appropriate, and the quality of the solder application state) based on the image data of the inspection object 12 illuminated and imaged by the illumination unit 22 (this image data is referred to as "two-dimensional image data") and the height map described later.

[0023] In the inspection apparatus 10 according to the present embodiment, the illumination unit 22 is a side illumination source that projects illumination light from an oblique direction onto the inspection surface of the inspection object 12. In the present embodiment, it includes an upper light source 22a, a middle light source 22b, and a lower light source 22c. Note that in the inspection apparatus 10 according to the present embodiment, the side illumination sources 22a, 22b, and 22c are each a ring illumination source, configured to surround the optical axis of the camera unit 21 and project illumination light obliquely onto the inspection surface of the inspection object 12. Each of these side illumination sources 22a, 22b, and 22c may be configured by arranging a plurality of light sources in an annular shape. Further, the upper light source 22a, the middle light source 22b, and the lower light source 22c, which are side illumination sources, are each configured to project illumination light at different angles with respect to the inspection surface.

[0024] The projection unit 23 projects a pattern onto the inspection surface of the inspection object 12. The inspection object 12 onto which the pattern has been projected is imaged by the camera unit 21. The inspection apparatus 10 creates a height map of the inspection surface of the inspection object based on the image data of the inspected object 12 (this image data is referred to as "pattern image data"). Here, the height map is data having height information of the inspection object for each pixel of the pattern image data. The control unit 30 detects a local discrepancy between the projected pattern and the pattern image data, and acquires the height information of that part based on the local discrepancy. That is, the change in the imaged pattern with respect to the projected pattern corresponds to the change in height on the inspection surface.

[0025] The projection pattern is preferably a one-dimensional striped pattern in which bright lines and dark lines are alternately and periodically repeated. The projection unit 23 is arranged to project the striped pattern from an oblique direction onto the inspection surface of the object to be inspected 12. The discontinuity in height on the inspection surface of the object to be inspected 12 appears as a pattern shift in the striped pattern image. Therefore, the height difference can be obtained from the amount of pattern shift. For example, the control unit 30 creates a height map by the PMP (Phase Measurement Profilometry) method using a striped pattern whose brightness changes according to a sine curve. In the PMP method, the amount of pattern shift corresponds to the phase difference of the sine curve.

[0026] The projection unit 23 includes a pattern forming device, a light source device for illuminating the pattern forming device, and an optical system for projecting the pattern onto the inspection surface of the object to be inspected 12. The pattern forming device may be, for example, a variable patterning device capable of dynamically generating a desired pattern such as a liquid crystal display, or a fixed patterning device in which a pattern is fixedly formed on a substrate such as a glass plate. When the pattern forming device is a fixed patterning device, it is preferable to make the projection position of the pattern variable by providing a moving mechanism for moving the fixed patterning device or by providing an adjustment mechanism in the optical system for pattern projection. Further, the projection unit 23 may be configured to be able to switch between a plurality of fixed patterning devices having different patterns.

[0027] A plurality of projection units 23 may be provided around the camera unit 21. The plurality of projection units 23 are arranged to project patterns onto the object to be inspected 12 from different projection directions. In this way, it is possible to reduce the area where a shadow is formed due to the height difference on the inspection surface and the pattern is not projected.

[0028] The control unit 30 shown in Fig. 1 comprehensively controls the entire device. In terms of hardware, it is realized by the CPU, memory, and other LSIs of an arbitrary computer, and in terms of software, it is realized by a program loaded into the memory, etc. Here, however, functional blocks realized by their cooperation are depicted. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

[0029] Fig. 1 shows an example of the configuration of the control unit 30. The control unit 30 includes an inspection control unit 31 and a memory 35 which is a storage unit. The inspection control unit 31 includes an image processing unit 32, an inspection data processing unit 33, and an inspection unit 34. Further, the image processing unit 32 includes an imaging processing unit 32a, a height measurement unit 32b, and an image synthesis unit 32c. Also, the inspection device 10 is provided with an input unit 36 for receiving inputs from a user or other devices, and an output unit 37 for outputting information related to the inspection. The input unit 36 and the output unit 37 are each connected to the control unit 30. The input unit 36 includes, for example, input means such as a mouse and a keyboard for receiving inputs from the user, and communication means for communicating with other devices. The output unit 37 includes known output means such as a display and a printer.

[0030] The inspection control unit 31 is configured to execute various control processes for inspection based on the input from the input unit 36 and the inspection-related information stored in the memory 35. The inspection-related information includes two-dimensional image data of the object to be inspected 12, a height map of the object to be inspected 12 (calculated from pattern image data), and substrate inspection data. Prior to the inspection, the inspection data processing unit 33 creates substrate inspection data using the two-dimensional image data and the height map of the object to be inspected 12 for which it is guaranteed that all inspection items are passed. The inspection unit 34 executes an inspection based on the created substrate inspection data, the two-dimensional image data, and the height map of the object to be inspected 12 to be inspected.

[0031] The substrate inspection data is inspection data created for each type of substrate. The substrate inspection data is an aggregate of inspection data for each component arranged on the substrate, its position, and the solder applied to the substrate. The inspection data for each component and solder includes the inspection items required for that component and solder, the inspection window which is the inspection area on the image for each inspection item, and the inspection criteria which serve as the basis for arrangement and pass / fail determination for each inspection item. One or more inspection windows are set for each inspection item. For example, in the inspection item for determining the quality of the solder application state, usually, the same number of inspection windows as the number of solder application areas of the component are set in an arrangement corresponding to the arrangement of the solder application areas. Also, for inspection items that use an image obtained by performing predetermined image processing on the inspection object image, the content of the image processing is also included in the inspection data.

[0032] As a substrate inspection data creation process, the inspection data processing unit 33 sets each item of the inspection data according to the substrate. For example, the inspection data processing unit 33 automatically sets the position and size of each inspection window for each inspection item so as to conform to the solder layout of the substrate. The inspection data processing unit 33 may accept user input for some items of the inspection data. For example, the inspection data processing unit 33 may accept tuning of the inspection criteria by the user. The inspection criteria may be set using height information.

[0033] As a pre - process for creating substrate inspection data, the inspection control unit 31 executes an imaging process of the inspection object 12 by the image processing unit 32. For this inspection object 12, those that pass all inspection items are used. As described above, the imaging process is performed by controlling the relative movement of the imaging unit 20 and the inspection table 14 while illuminating the inspection object 12 with the illumination unit 22, and sequentially imaging partial images of the inspection object 12. A plurality of partial images are imaged so as to cover the entire inspection object 12. The inspection control unit 31 synthesizes these plurality of partial images and generates substrate full - surface image data (2 - dimensional image data) including the entire inspection surface of the inspection object 12. The inspection control unit 31 stores the substrate full - surface image data in the memory 35.

[0034] Also, as preprocessing for height map creation, the inspection control unit 31 controls the relative movement of the imaging unit 20 and the inspection table 14 while the projection unit 23 projects a pattern onto the inspection object 12 by the imaging processing unit 32a of the image processing unit 32, divides the pattern image of the inspection object 12, and sequentially captures images. The pattern to be projected is preferably a stripe pattern whose brightness changes according to a sine curve based on the PMP method. The inspection control unit 31 synthesizes the captured divided images and generates pattern image data for the entire inspection surface of the inspection object 12. The inspection control unit 31 stores the pattern image data in the memory 35. Note that the pattern image data may be generated for only a part of the inspection surface instead of the entire surface. Also, as will be described later, the image data captured by the imaging processing unit 32a is subjected to synthesis processing by the height measurement unit 32b and the image synthesis unit 32c.

[0035] The height measurement unit 32b creates a height map for the entire inspection surface of the inspection object 12 based on the image of the pattern in the pattern image data. First, the height measurement unit 32b obtains a phase difference map of the inspection surface of the inspection object 12 by obtaining the local phase difference between the pattern image data and the reference pattern image data for the entire image data. Here, the "reference pattern image data" is image data in which a pattern is projected onto a reference plane by the projection unit 23 (that is, image data in which a pattern generated by the pattern forming device built in the projection unit 23 is projected onto the reference plane). The height measurement unit 32b creates a height map of the inspection object 12 based on the reference plane serving as a reference for height measurement and the phase difference map. The reference plane is, for example, the surface of the printed circuit board to be inspected. The reference plane does not necessarily have to be a plane and may be a curved surface that reflects deformations such as warping of the substrate.

[0036] Specifically, the height measurement unit 32b obtains the phase difference of the stripe pattern between each pixel of the pattern image data and the pixel of the reference pattern image data corresponding to the pixel. The height measurement unit 32b converts the phase difference into height information. This is because the distance from the projection unit 23 varies depending on the position on the inspection surface, so even if the stripe width of the reference pattern is constant, the stripe width changes from one end to the other end of the pattern projection area on the inspection surface. The height measurement unit 32b obtains the height information from the reference plane based on the converted height information and the reference plane, and creates a height map of the object under inspection 12.

[0037] The image processing unit 32 of the inspection control unit 31 may create an object under inspection image having a height distribution by associating the height information included in the height map of the object under inspection 12 with each pixel of the two-dimensional image data of the object under inspection 12. Further, the image processing unit 32 may perform three-dimensional modeling display of the object under inspection 12 based on the object under inspection image data with height distribution. Further, the image processing unit 32 may superimpose the height distribution on the two-dimensional object under inspection image data and display it on the output unit 37. For example, the object under inspection image data may be color-coded and displayed according to the height distribution.

[0038] Hereinafter, in such an inspection apparatus 10, a method of projecting a pattern by the projection unit 23 to acquire height information will be described.

[0039] First, a method of acquiring height information by the PMP method using a stripe pattern will be described. Focusing on one measurement point (pixel) in the stripe pattern projection area, when the stripe pattern is projected while spatially shifting the phase (in other words, when the stripe pattern is scanned in the direction in which the stripes repeat), the brightness of the measurement point (pixel) varies periodically. Although the average brightness differs for each measurement point (pixel) according to the surface characteristics of the measurement point (pixel) such as color and reflectivity, periodic brightness variation corresponding to the stripe pattern occurs at any measurement point (pixel). Therefore, the phase is calculated from the periodic brightness variation, and the phase difference from the initial phase gives the height information.

[0040] When obtaining height information by the PMP method using the characteristics of such stripe patterns, in principle, it is necessary to shift the phase of the stripe pattern and image at least three times, typically four times. Corresponding to the stripe pattern being a sine wave, the brightness variation at each measurement point (pixel) is also a sine wave. Since the pitch of the stripes is known, if the average value, amplitude, and initial phase of the luminance (brightness) are made clear, the sine wave representing the brightness variation is specified.

[0041] By obtaining the measured values of the luminance (brightness) of the measurement points (pixels) from at least three pattern image data captured with the starting position of the phase of the stripe pattern shifted, three variables, namely the average value, amplitude, and initial phase of the luminance (brightness), can be determined. Here, if the luminance of a certain pixel when one stripe pattern is imaged four times with the starting position of the phase shifted by 90 degrees (π / 2 radians) each time is denoted as I0, I1, I2, and I3 respectively, the luminance In when the stripe pattern is a sine wave is expressed by the following formula (a). In this formula (a), A represents the amplitude depending on the brightness of the object 12, B represents the brightness offset depending on the camera unit 21 and the ambient luminance, φ represents the phase to be obtained, and π / 2 represents the phase shift amount (shift amount). Note that n = 0, 1, 2, 3. However, generally, errors are included due to factors such as sensor noise, so the least squares method is often required when calculating the phase φ, amplitude A, and offset B. Also, let the error obtained by the least squares method be E.

[0042] In = A×sin(φ + n×π / 2) + B (a)

[0043] The calculation of the height information is to obtain the phase φ, amplitude A, and offset B of each pixel (picture element) from the pattern image data. For example, it is known that the phase φ of the measurement point corresponding to each pixel when one stripe pattern is imaged four times with the starting position of the phase shifted by 90 degrees (π / 2 radians) each time is expressed as the following formula (b).

[0044] tan(φ) = (I3 - I1) / (I2 - I0) (b)

[0045] Assuming that the pitch (length of one cycle) of the stripe pattern is P and the initial phase is 0, the amount of stripe pattern shift is obtained by P×(φ / 2π). Using the projection angle of the pattern, height information at that position can be acquired from the amount of pattern shift.

[0046] In Fig. 2, as shown in (a), the phase obtained from the pattern image data obtained by projecting the stripe pattern by the projection unit 23 onto the reference plane S and imaging with the camera unit 21 is shown in (c). As shown in (b), when the phase obtained from the pattern image data obtained by projecting and imaging the same stripe pattern with an object O placed on this reference plane S is shown in (d), the height information of the object O can be calculated from the amount of phase shift (phase difference = measured phase - phase of the reference plane) between these phases. This phase difference can be calculated for each pixel of the image captured by the camera unit 21, and by multiplying the coefficient to the phase difference obtained for each pixel, the height information at the position corresponding to that pixel is calculated. This coefficient is obtained, for example, simply by tan(projection angle)×stripe pitch / 2π.

[0047] Here, the stripe pattern is formed by repeating a pattern whose brightness is sinusoidal and monotonically changing (monotonically increasing or monotonically decreasing). Note that when the phase changes from 0 to 2π (radians) in one cycle (pitch), the case where the luminance monotonically increases from the darkest state (for example, luminance I = 0) to the brightest state (for example, luminance I = 255) is shown (the relationship between the phase and luminance in the pattern may be reversed, that is, monotonically decreasing).

[0048] As described with reference to Fig. 2, when imaging the reference plane S using the stripe pattern and also imaging the object O (corresponding to the inspection object 12) placed on the reference plane S using the stripe pattern, the phase difference Δφ between the phase φ0 of the reference plane S obtained based on formula (b) and the phase φ obtained in the state where the object O is placed is obtained by the following formula (c).

[0049] Δφ = φ - φ0 (c)

[0050] From the above, based on the phase difference Δφ calculated by Equation (c) from the reference plane and at least three pattern image data obtained by placing the object O (the inspection object 12) on this reference plane and projecting and imaging the stripe pattern while shifting the start position of the phase (when imaging four times with a 90° shift each, it is calculated by Equation (b)), the height information H can be obtained by the following Equation (d).

[0051] H = P×(Δφ / 2π) (d)

[0052] As described above, the inspection apparatus 10 according to the present embodiment is configured to inspect the inspection object 12 using the height map calculated from the two-dimensional image data and the pattern image data. However, since the camera of the camera unit 21 has a predetermined depth of field (depth of focus), for example, when the camera unit 21 is focused near the surface of the substrate that is the inspection object 12, there is a possibility that out-of-focus image data of the upper surface of a tall component may be captured. Therefore, when inspecting using such image data, in the two-dimensional image data, there may be a case where the characters written on the upper surface of a tall component (such as a capacitor) cannot be read, and in the pattern image data, the image of the stripe pattern on the upper surface of a tall component may be blurred, and there may be a case where accurate height information cannot be calculated.

[0053] Therefore, in the inspection apparatus 10 according to the present embodiment, at least two pieces of image data captured by the camera unit 21 at different positions in the Z direction (imaging direction) are acquired, and pixels that are relatively in focus are extracted and synthesized from each piece of image data, so that even if components at different heights are arranged, image data that is in focus across the entire field of view can be obtained. Here, the case of performing synthesis using the image data acquired when the camera unit 21 is in focus on the reference plane (the plane that serves as a reference when calculating height information as described above) and the image data acquired by the camera unit 21 at different positions in the Z direction by moving the camera unit 21 in the Z direction will be described. Hereinafter, a method for synthesizing image data in the inspection apparatus 10, which is an image processing apparatus according to the present embodiment, will be described. Here, the case of moving the position of the camera unit 21 in the Z direction will be described, but it is only necessary that the relative position in the Z direction between the camera unit 21 and the inspection object 12 changes, and the inspection object 12 may be moved in the Z direction. As shown in FIG. 3(a), a predetermined surface of the inspection object 12 (for example, in the case of an electronic circuit board, the upper surface of the board) is set as the "reference surface". Here, FIG. 3(a) shows the case where the reference surface of the inspection object 12 coincides with the reference plane.

[0054] Also, in the following description, the position of the camera unit 21 in the Z direction is represented in a coordinate system with Z = 0 as the reference, as the position of the camera unit 21 when it is in focus on the reference plane, as shown in Fig. 3(a). Specifically, P(i) indicating the position of the camera unit 21 in the Z direction is expressed with i = 0 when the camera unit 21 is in focus on the reference plane, and i = 1, 2, 3... in the order that the camera unit 21 moves away from the reference plane. In the following description, an example of obtaining four pieces of image data at different positions in the Z direction is taken. For example, assuming that the depth of field of the camera unit 21 is 4 mm, image data is obtained at the position of the camera unit 21 when it is in focus on the reference plane "P(0) = 0", and further, a case of obtaining three pieces of image data at positions shifted 4 mm upward (in the Z direction) from the position of P(0) = 0 will be described. Each position is represented as "P(1) = 4", "P(2) = 8", "P(3) = 12" (the unit of mm is omitted). Also, when the camera unit 21 is in focus on the reference plane, that is, when the camera unit 21 is at Z = 0, the distance from the tip of this camera unit 21 to the reference plane is defined as the focal length. Further, a position that is the focal length away from the tip of the camera unit 21 in the direction of the inspection object 12 is defined as the optimal focus position FP.

[0055] Note that the number of pieces of image data with the position of the camera unit 21 shifted in the Z direction is not limited to four, and two or more pieces are sufficient. Also, the width of shifting the camera unit 21 in the Z direction is not limited to 4 mm, and can be set to any value in consideration of, for example, the height of the components mounted on the inspection object 12 and the depth of field of the camera unit 21. Further, not only can the position of the camera unit 21 in the Z direction be changed at equal intervals, but the interval for obtaining each piece of image data can also be arbitrarily selected.

[0056] Also, it is assumed that the position at which the camera unit 21 acquires image data in the Z direction is set in advance in the control unit 30 from the input unit 36 or the like. For example, the position of the reference plane and the height of the inspection target (the height of the components mounted on the substrate) are input, and the position and interval in the Z direction at which image data is acquired are set. Also, it is assumed that the focal length of the camera unit 21 is fixed.

[0057] Using FIG. 4, a method for synthesizing image data in the inspection apparatus 10 which is an image processing apparatus according to the present embodiment will be described. In this method for synthesizing image data, among the image data acquired at different positions in the Z direction, the synthesis process is configured to be performed from the image data acquired at the position farthest from the inspection object 12 (reference plane).

[0058] As shown in FIG. 4(a), when the control unit 30 determines that the process of acquiring the image data of the inspection object 12 has started, the imaging processing unit 32a moves the camera unit 21 to a predetermined position (position in the XY direction) above the inspection object 12, and further adjusts the position of the camera unit 21 in the Z direction so that the camera unit 21 is at the position of i = 0, that is, the camera unit 21 is located at P(0), and acquires the image data of the inspection object 12 (step S100). When the camera unit 21 is moved to the position of P(0), the camera unit 21 will be in focus near the reference plane of the inspection object 12. In step S100, as described above, in order to acquire the height map of the inspection object 12, the projection unit 23 projects a stripe pattern onto the inspection object 12 and the camera unit 21 acquires pattern image data. At this time, the phase of the stripe pattern is shifted to acquire at least three (four if possible) pieces of pattern image data. Also, when there are a plurality of projection units 23, stripe patterns are projected onto the inspection object 12 from each projection unit 23 while shifting the phase as described above to acquire pattern image data. Further, when acquiring two-dimensional image data, the side illumination sources 22a, 22b, 22c of the illumination unit 22 are sequentially lit to illuminate the inspection object 12, and the camera unit 21 performs imaging.

[0059] When the imaging of the subject 12 is completed, the control unit 30 sets a parameter for removing an error measurement value from the height information calculated based on the pattern image data (referred to as "pattern image data of P(0)") captured when the camera unit 21 is at the position where i = 0, that is, at the position of P(0)) (step S102). As will be described later, the synthesis process of the image data when the camera unit 21 is at the position where i = 0 (at the time of P(0)) is executed last. In this final synthesis process, in order to select the pixels of the image data at the time of P(0) as the pixels of the synthesized image data as much as possible, the above parameter is set to a value that maximizes the number of measurement points. Then, the height basic process S200 is executed using this parameter (step S104).

[0060] When the control unit 30 starts the height basic process S200 shown in FIG. 4(b), the height measurement unit 32b performs a phase calculation process for each pixel in the pattern image data (within the field of view) based on the above-described formulas (a) to (d) from three or four pieces of pattern image data captured by shifting the phase of the stripe pattern (step 202). When there are a plurality of projection units 23, the phase calculation process is executed for each projection unit 23 (separately for each projection). Next, when there are a plurality of projection units 23, the control unit 30 executes a projection synthesis process for synthesizing the respective phase calculation results, and calculates the height information for each pixel (step S204). Finally, noise filtering processing such as a median filter is executed on the height information for each pixel (step S206), and the execution of the height basic process S200 for the pattern image of P(0) is terminated.

[0061] The reason for first acquiring the image data when the camera unit 21 is at the position of P(0) (when the camera unit 21 is in focus near the reference plane) and calculating the height information from the pattern image data is to use the amplitude and height information at the time of P(0) and the like in the condition (2) used for the determination of the synthesis process, as will be described later.

[0062] Returning to FIG. 4(a), in step S104, when the execution of the height basic process S200 ends, the control unit 30 causes the imaging processing unit 32a to move the Z-direction position of the camera unit 21 to the position farthest from the inspection object 12 among the set imaging positions as the next imaging position (step S106). Here, the camera unit 21 is moved to the position of P(3)=12 when i = 3.

[0063] When the control unit 30 moves the camera unit 21 to the next imaging position in the Z direction, similar to the case of P(0), the imaging processing unit 32a acquires pattern image data and two-dimensional image data at that imaging position, and further sets a parameter for removing an erroneous measurement value from the height information calculated from the pattern image data (step S108). Further, the control unit 30 causes the height measurement unit 32b to execute the height basic process S200 on the pattern image data imaged in step S108 (step S110). Here, since the camera unit 21 is at a position other than i = 0 (hollow), the parameter for removing an erroneous measurement value from the height information is set to exclude points with a high suspicion of measurement error (excluding erroneous measurement points in the hollow).

[0064] When the height information based on the pattern image data imaged at the current imaging position (here, the position of P(3)) is calculated, the control unit 30 causes the image synthesis unit 32c to execute a composite image update process using the image data of the current imaging position (step S112). Here, among the pixels of the image data of the current imaging position, the in-focus image (in-focus pixel) is selected as the value of the pixel of the composite image data. Specifically, for each pixel, based on the height information of that pixel at the current imaging position, whether to select the pixel of the image data of the current imaging position as the composite image data is determined according to the following two conditions (conditions (1) and (2)) for each pixel.

[0065] P(i)-DL≦H(X,Y,i)+P(i)≦P(i + 1)―DH (1) A(X,Y,i) ≧ A(X,Y,0)×WA (2) However, P(i): Current imaging position P(i + 1): Next imaging position farther from the reference plane than the current imaging position H(X, Y, i): Calculated distance at the current imaging position when the position of the pixel to be determined is (X, Y) DL, DH: Constants A(X, Y, i): Amplitude at the current imaging position when the position of the pixel to be determined is (X, Y) A(X, Y, 0): Amplitude at the imaging position when i = 0 when the position of the pixel to be determined is (X, Y) WA: Weight count

[0066] Condition (1) determines whether the position (calculated distance) of the subject 12 corresponding to the pixel to be determined is near the focus of the camera unit 21 at the current imaging position. Here, the calculated distance H(X, Y, i) is the distance in the Z direction from the optimal focus position FP at P(i) to the subject 12 at the pixel to be determined, and is the height information of the pixel at the position (X, Y) obtained by the basic height process S200, and is represented in the coordinate system of the camera unit 21 described above. For example, when P(0) = 0, H(X, Y, i) = 0 when the subject 12 corresponding to the pixel to be determined is on the reference plane. As shown in FIG. 3(a), when the camera unit 21 moves in the Z direction, the optimal focus position FP also moves in the Z direction. Also, as shown in FIG. 3(b), DL and DH are constants for determining the range in which image data in focus can be obtained by the camera unit 21 based on the current imaging position P(i) of the camera unit 21 and the adjacent imaging position P(i + 1) farther from the reference plane than the current imaging position. For example, when the position of the camera unit 21 is set at intervals of 4 mm, DL = 1.5 mm and DH = 1.0 mm can be set. Of course, these constants DL and DH can be determined by the specifications of the camera unit 21 (depth of field, etc.) and the interval by which the camera unit 21 is shifted in the Z direction. Generally, it is desirable that DL > DH. Also, when at the imaging position farthest from the subject 12 (reference plane) (here, P(3) = 12), the value of P(i + 1) (here, P(4)) is set in advance, and condition (1) is determined using this value (for example, set P(4) = 16). From the above, the pixels that satisfy condition (1) mean that the subject 12 is within the range where image data (clear image data) in focus can be obtained by the camera unit 21 at that position (P(i)).

[0067] Here, instead of making DL and DH in condition (1) constants, they can be variables as shown below. Note that FUNC represents a predetermined function, DL(i) is a variable at the imaging position P(i), which is determined by a function FUNC that takes the next imaging position P(i - 1) and the current imaging position P(i) as parameters, and DH(i) is a variable at the imaging position i, which is determined by a function FUNC that takes the current imaging position P(i) and the previous imaging position P(i + 1) as parameters.

[0068] DL(i)=FUNC(P(i - 1),P(i)) DH(i)=FUNC(P(i),P(i + 1))

[0069] Also, condition (2) is to determine whether a pixel has a large defocus value based on the amplitudes A(X, Y, i) and A(X, Y, 0) obtained by the above formula (a) when calculating the phase using the pattern image data. This condition (2) compares the defocus values to extract pixels with a larger defocus value, and determines that it is in focus when the evaluation value of the pixel is greater than or equal to the threshold. Specifically, since a larger amplitude value indicates a larger defocus value, a threshold is determined based on the amplitude at P(0) (when i = 0), and at the pixel to be determined (the pixel at position (X, Y)), when the amplitude A(X, Y, i) at the current imaging position is sufficiently larger than the amplitude A(X, Y, 0) at i = 0, it can be determined that the pixel to be determined is an in-focus pixel. Note that it is desirable to determine the value of the weight WA based on the depth of field of the camera unit 21.

[0070] From the above, at the imaging position P(i), for each pixel, the conditions (1) and (2) are judged. For the pixels that simultaneously satisfy the conditions (1) and (2), the value of the pixel at the current imaging position is selected as the pixel of the composite image. Specifically, when the pixel at the position (X, Y) satisfies the conditions (1) and (2), the height information (H(X, Y, i) + P(i)) calculated from the pattern image data of P(i) of the pixel is set as the value of the pixel at the same position in the composite image data of the height map, and the value (such as luminance) of the pixel of the two-dimensional image data captured at the position of P(i) is set as the value of the pixel at the same position in the composite image data of the two-dimensional image data. By judging using such conditions (1) and (2), it is possible to prevent mismeasurement particularly near the seam of the synthesis.

[0071] When the imaging processing unit 32a has performed the above composite processing for all pixels, the control unit 30 determines whether the imaging processing has been completed at all the previously determined imaging positions in the Z direction (step S114). If it is determined that the imaging processing has not been completed at all the imaging positions (step S114: No), the control unit 30 sets the position of the camera unit 21 to the next imaging position approaching the reference plane by the imaging processing unit 32a (step S116), and at that imaging position, the processing of steps S108 to S114 described above is repeated. For example, when the current imaging position is P(3) = 12, the camera unit 21 is set to P(2) = 8 which is the next imaging position, and when the current imaging position is P(2) = 8, the camera unit 21 is set to P(1) = 4 which is the next imaging position.

[0072] On the other hand, when it is determined that the imaging process has been completed at all imaging positions (step S114: Yes), the control unit 30 performs a composite image update process on the image data (image data when P(0) = 0) obtained in step S100, where the camera unit 21 is in focus near the reference plane, by the image composite unit 32c (step S118). Here, in the image data of P(0), since condition (2) is always satisfied, a determination is made based on whether condition (1) is satisfied. That is, in step S118, for pixels where the value of the composite image data has not yet been selected and that satisfy condition (1), the pixel value (height information in the height map and the pixel value of the two-dimensional image data) of the image data at the imaging position of P(0) is selected as the pixel value of the composite image data.

[0073] In addition, in the repeated process of steps S108 to S114, even if a certain pixel satisfies condition (1) and condition (2), if it has already satisfied condition (1) and condition (2) at an imaging position farther from the test object 12 than that imaging position and the value of that pixel has been selected as the value of the composite image data, the already selected value is prioritized. That is, for pixels that satisfy condition (1) and condition (2) at a plurality of imaging positions in the Z direction, among those imaging positions, the pixel value at the imaging position farthest from the test object 12 is selected. The light irradiated by the projection unit 23 is imaged after being reflected by the test object 12. As the camera unit 21 moves farther from the substrate surface, the amount of reflected light from the test object 12 decreases. This leads to a reduction in noise caused by unnecessary light, especially in pixels that are not in focus. As a result, the mismeasurement values included in the height information are reduced. Therefore, for pixels that satisfy condition (1) and condition (2) at a plurality of imaging positions in the Z direction, among those imaging positions, the pixel value at the imaging position farthest from the test object 12 is selected.

[0074] FIG. 5 shows the relationship between the best focus position FP of the camera unit 21, the image data captured at that position, and the composite image data when the subject 12 is a portion on the substrate 12d where components 12a, 12b, and 12c are mounted, in a certain field of view. Specifically, the upper rows of FIGS. 5(a) to 5(d) show the relationship between the best focus position FP of the camera unit 21 and the components 12a to 12d, and the lower rows show, in hatching, the portions (pixels) that may be selected as the composite image data. FIG. 5(a) shows the case when the camera unit 21 is at P(3)=12. Since the upper surface of the component 12a satisfies condition (1), if condition (2) is also satisfied at the same time, as shown in FIG. 5(e), the upper surface of the component 12a in the image data at P(3) is selected as the composite image data. On the other hand, as shown in FIG. 5(b), the upper surface of the component 12b satisfies condition (1) when the camera unit 21 is at P(2)=8, and also satisfies condition (1) when the camera unit 21 is at P(1)=4 as shown in FIG. 5(c) (assuming that condition (2) is also satisfied). In this case, as shown in FIG. 5(e), the upper surface of the component 12b in the image data at P(2), which is farther from the subject 12, is selected as the composite image data. Also, as shown in FIG. 5(c), the upper surface of the component 12c satisfies condition (1) when the camera unit 21 is at P(1)=4. Therefore, if condition (2) is satisfied at the same time, as shown in FIG. 5(e), the upper surface of the component 12c in the image data at P(1) is selected as the composite image data. Similarly, as shown in FIG. 5(d), the upper surface (reference surface) of the substrate 12d satisfies condition (1) when the camera unit 21 is at P(0)=0. Therefore, if condition (2) is satisfied at the same time, as shown in FIG. 5(e), the upper surface of the substrate 12d in the image data at P(0) is selected as the composite image data (see FIG. 5(e)).

[0075] Returning to FIG. 4, finally, the control unit 30 executes height post-processing S300 by the height measurement unit 32b (step S120). When the height measurement unit 32b of the control unit 30 starts the height post-processing S300 shown in FIG. 4(c), it executes a process of interpolating unmeasured pixels such as pixels whose values were not selected in the above-described composite image update process in the composite image data (step S302), further performs a process of correcting the distortion of the composite image data (step S304), ends the height post-processing S300, and further ends the composite process. If the height post-processing is executed on the image data before composition (height map calculated from the pattern image data acquired at each imaging position), there is a possibility that the interpolated values and corrected pixels will be selected as the pixel values of the composite image data, and the accuracy of the image data may deteriorate. Therefore, it is desirable that the height post-processing such as the unmeasured pixel interpolation process and the distortion correction process be executed on the composite image data after the composite process of the image data at all imaging positions is completed.

[0076] As shown in Fig. 4, in the method for synthesizing image data according to the present embodiment, except for the image data captured when the camera unit 21 is in focus near the reference plane (at the position P(0) where i = 0), every time the position of the camera unit 21 in the Z direction is moved to acquire image data, the height basic process and the composite image update process are executed. At this time, in Fig. 4, the imaging process and the composite process of the image data acquired by the imaging process (height basic process and composite image update process) are described as a series of processes. However, when the imaging process at the next imaging position is being executed, the height basic process and the composite image update process for the image data acquired at the previous imaging position can be executed in parallel. As described above, in the imaging process, acquisition of a plurality of pattern image data with the phase of the stripe pattern shifted is executed for each of the plurality of projection units 23. Further, since the side illumination sources 22a, 22b, 22c of the illumination unit 22 must be sequentially lit to acquire a plurality of two-dimensional image data, while these imaging processes are being performed at the next imaging position, by executing the height basic process and the composite image update process for the previous imaging position in parallel, the overall processing time can be shortened, and as a result, the inspection time can be shortened. At this time, in Fig. 4, after the imaging process and the height basic process are executed at the position P(0), the camera unit 21 is moved to P(3) which is the farthest from the object to be inspected 12, and the camera unit 21 is moved in the direction approaching the object to be inspected 12 from this imaging position to execute the imaging process, the height basic process, and the composite image update process. Finally, the composite image update process for the image data at P(0) is executed.By performing the imaging process, the height basic process, and the composite image update process in this order, when the pixel values of the composite image data have already been selected in the height basic process and the composite image update process at an imaging position before the imaging position where the current height basic process and the composite image update process are being executed, even if the image data at the imaging position where the current height basic process and the composite image update process are being performed satisfies conditions (1) and (2), the pixel values are not selected as the corresponding pixel values of the composite image data. By performing such processing, it is possible to conform to the above-mentioned determination criterion that "when pixels at the same position in the image data of two or more imaging positions satisfy conditions (1) and (2), among these image data, the pixel values of the image data at the imaging position farthest from the subject 12 are selected as the pixel values of the composite image data."

[0077] Note that the above-described processing is executed for one field of view. When the subject 12 is imaged (examined) by dividing it into a plurality of fields of view, the camera unit 21 is moved in the XY directions, and the above processing is executed in each field of view.

[0078] The inspection apparatus 10 according to the present embodiment executes the above-described inspection based on the composite image data (height map and two-dimensional image data) acquired by the image processing unit 32 included in the inspection control unit 31 of the control unit 30.

[0079] (First Modification Example of Conditions for Selecting Pixels) In the above-described composite image update process, in condition (2), the determination is made based on the amplitude of the target pixel. However, instead of this condition (2), the determination may be made according to condition (3) shown below. That is, the pixel of the image data at the current imaging position may be configured to be selected as the composite image data or not based on the height information at the current imaging position of the pixel according to the above-described conditions (1) and (3). Note that the contrast C is calculated as C = A / B using the amplitude A and the offset B obtained from the above-described formula (a).

[0080] C(X,Y,i) ≧ C(X,Y,0)×WC (3) However, C(X,Y,i): The contrast at the current imaging position when the position of the pixel to be determined is (X,Y) C(X,Y,0): The contrast at i = 0 when the position of the pixel to be determined is (X,Y) WC: Weight count

[0081] Condition (3) is used to determine whether a pixel has a large depth of focus based on the contrasts C(X,Y,i) and C(X,Y,0) obtained based on the above-mentioned formula (a) when calculating the phase using the pattern image data. This condition (3) extracts a pixel with a larger depth of focus by comparing the depths of focus, and determines that it is in focus when the evaluation value of the pixel is equal to or greater than the threshold value. Specifically, since a larger contrast value indicates a larger depth of focus, in the pixel to be determined (the pixel at position (X,Y)), when the contrast C(X,Y,i) at the current imaging position is sufficiently larger than the contrast C(X,Y,0) at i = 0, it can be determined that the pixel to be determined is an in-focus pixel. It is desirable that the value of the weight WC be determined based on the depth of field of the camera unit 21.

[0082] (Modification example of the height measurement method) In the above description, the case where height information is calculated from three or four pattern image data obtained by projecting one stripe pattern from the projection unit 23 while changing the phase has been described. However, in order to improve the accuracy of the height information, it is also possible to configure to obtain the height information using at least two stripe patterns having different periods. For example, in the imaging processes of steps S100 and S108 described above, the projection unit 23 projects a first pattern with a long period while changing the phase to obtain three or four pattern image data, and also projects a second pattern with a short period while changing the phase to obtain three or four pattern image data. When there are a plurality of projection units 23, pattern image data with the phases of the first pattern and the second pattern changed are obtained by each projection unit 23. Then, rough height information (rough height) is obtained from the first pattern with a long period (that is, thick stripes), and precise height information is obtained with the second pattern with a short period (thin stripes). As described above, since the PMP method obtains height information based on the phase, if the height gap (height difference) is large and the stripes are shifted by more than one period, the height information cannot be uniquely specified. By obtaining the height information using the first pattern in combination, it becomes possible to uniquely specify the height information even when the stripes are shifted by more than one period when the second pattern is projected.

[0083] The luminance In obtained from three or four pattern image data obtained by projecting the first pattern with the phase changed by the projection unit 23 also has the relationship of the above formula (a) with the luminance In obtained from three or four pattern image data obtained by projecting the second pattern with the phase changed. Also, the relationship between these luminances In and the phase φ has the relationship of the above formula (b).

[0084] Here, when the phase by the first pattern with a long period is φw and the phase of the reference plane is φw0, the phase difference Δφw has the relationship of the following formula (c1) from the above formula (c).

[0085] Δφw = φw - φw0 (c1)

[0086] Similarly, when the phase due to the short-period second pattern is φf and the phase of the reference plane is φf0, the phase difference Δφf has the relationship of the following equation (c2) from the above-mentioned equation (c).

[0087] Δφf = φf - φf0 (c2)

[0088] From these phase differences Δφw and Δφf, the detailed height information H can be obtained by the following equation (d′). Here, M is the number of periods of the short-period second pattern with respect to one period of the long-period first pattern, and Round() is a function that rounds off the digits after the decimal point.

[0089] H = Round((M×Δφw - Δφf) / 2π)×2π + Δφf (d′)

[0090] In the case of a configuration that calculates height information using two stripe patterns of long period and short period, instead of the determination based on the amplitude shown in the above-mentioned condition (2), it is desirable to perform determination based on the amplitude ratio as shown in the following condition (2′). That is, it is desirable to determine whether to select the pixel of the image data at the current imaging position as the composite image data according to the above-mentioned condition (1) and condition (2′).

[0091] AR(X,Y,i) ≦ AR(X,Y,0)×WAR (2′) However,[[]] AR(X,Y,i): Amplitude ratio at the current imaging position when the position of the pixel to be determined is (X,Y) AR(X,Y,0): Amplitude ratio at i = 0 when the position of the pixel to be determined is (X,Y) WAR: Weight count

[0092] Note that the contrast ratio AR (the parameters (X, Y, i) are omitted) of the pixel at the position (X, Y) at the imaging position P(i) is calculated based on the following formula (e). Here, Aw is the amplitude obtained from the formula (a) from three or four pattern image data obtained by projecting the first pattern of the long period, and Af is the amplitude obtained from the formula (a) from three or four pattern images obtained by projecting the second pattern of the short period.

[0093] AR =(Aw-Af) / Af (e)

[0094] Similar to the case of condition (2), this condition (2′) extracts pixels with a larger defocus value by comparing the defocus values, and determines that it is in focus when the evaluation value of the pixel is less than or equal to the threshold. Specifically, since the smaller the value of the amplitude ratio, the larger the defocus value, at the pixel to be determined (the pixel at the position (X, Y)), when the amplitude ratio AR(X, Y, i) at the current imaging position is sufficiently smaller than the amplitude ratio AR(X, Y, 0) when the camera unit 21 is in focus near the reference plane (when i = 0), it can be determined that the pixel to be determined is a focused pixel. Note that the value of the weight WAR is preferably determined based on the depth of field of the camera unit 21.

[0095] Alternatively, instead of condition (2′), it may be determined by the following condition (3′). That is, according to the above-mentioned conditions (1) and (3′), it may be determined whether to select the pixel of the image data at the current imaging position as the composite image data.

[0096] CR(X,Y,i) ≦ CR(X,Y,0)×WCR (3′) However, CR(X,Y,i): The contrast ratio at the current imaging position when the position of the pixel to be determined is (X, Y) CR(X,Y,0): The contrast ratio when i = 0 when the position of the pixel to be determined is (X, Y) WCR: Weight count

[0097] Condition (3′) is used to determine whether a pixel is in focus based on the contrast ratio obtained based on the above-described formula (a) when calculating the phase using the pattern image data. This condition (3′) compares the focus levels to extract pixels with a higher focus level, and determines that the pixel is in focus when the evaluation value of the pixel is equal to or less than a threshold value. Specifically, since a smaller value of the contrast ratio indicates a higher focus level, when the contrast ratio CR(X, Y, i) at the current imaging position is sufficiently smaller than the contrast ratio CR(X, Y, 0) when the camera unit 21 is in focus near the reference plane (when i = 0) for the pixel to be determined, it can be determined that the pixel to be determined is in focus.

[0098] Note that the contrast ratio CR (the (X, Y, i) of the parameters is omitted) is obtained from the contrast Cw obtained from three or four pattern image data obtained by projecting the first pattern with a long period and the contrast Cf obtained from three or four pattern images obtained by projecting the second pattern with a short period according to the following formula (f). Here, Aw is the amplitude obtained from the first pattern, Af is the amplitude obtained from the second pattern, Bw is the offset obtained from the first pattern, and Bf is the offset obtained from the second pattern, and these values are obtained from the above-described formula (a).

[0099] CR = (Cw - Cf) / Cf (f) However, Cw = Aw / Bw (g) Cf = Af / Bf (h)

[0100] As described above, in the above-described composite image update process S112, even if pixels that simultaneously satisfy condition (1) and condition (3′) are selected, the pixels in focus can be used as the pixel values of the composite image data.

[0101] (Second modification example of the condition for selecting pixels) In the above-mentioned image data synthesis method, in the synthetic image update process S112, when there is one type of stripe pattern for acquiring height information, a pixel of the image data captured at the current imaging position is selected as a pixel of the synthetic image data by simultaneously satisfying conditions (1) and (2), or conditions (1) and (3), and when there are two types of stripe patterns, long period and short period, a pixel of the image data captured at the current imaging position is selected as a pixel of the synthetic image data by simultaneously satisfying conditions (1) and (2'), or conditions (1) and (3'). However, it is also possible to select a pixel of the synthetic image data when it satisfies condition (1) in the image data captured at the current imaging position without judging conditions (2), (2'), (3), and (3'). This is because it is possible to judge whether a pixel of the image data at the current imaging position is a focused pixel based on condition (1) alone, and the processing time can be further shortened. In addition, in the case of such a configuration, the imaging process at the imaging position P(0) when the focus is on the vicinity of the reference plane may be performed last (in other words, the imaging process, the basic height process, and the composite image update process may be performed from an imaging position far from the object under test 12).

[0102] In addition, when the imaging process, basic height process, and composite image update process are performed from an imaging position far from the subject 12, the image data when the camera unit 21 is focused near the reference plane (P(0)) is acquired last, so instead of the above-mentioned condition (2), the judgment may be made based on the following condition (2a).

[0103] A(X,Y,i) ≧ TH (2a) however, TH: Threshold value determined in advance from the amplitude A

[0104] In addition, when the above-mentioned condition (2') is used for the judgment, the judgment may be made based on the following condition (2a') instead of this condition (2').

[0105] AR(X,Y,i) ≦ TH (2a′) however, TH: Threshold value determined in advance from the amplitude ratio AR

[0106] Also, when making a determination using the above-described condition (3), instead of this condition (3), it may be determined using the following condition (3a).

[0107] C(X,Y,i) ≧ TH (3a) However,[[]] TH: A threshold value predetermined from the contrast C

[0108] Also, when making a determination using the above-described condition (3'), instead of this condition (3'), it may be determined using the following condition (3a').

[0109] CR(X,Y,i) ≦ TH (3a')[[]] However,[[]] TH: A threshold value predetermined from the contrast ratio CR

[0110] Even if the determination is made using conditions (2a), (2a'), (3a), and (3a') instead of conditions (2), (2'), or conditions (3), (3'), it is possible to extract pixels with a larger defocus degree among the pixels satisfying condition (1).

[0111] (Main Features and Effects)[[]] The main features and effects of the method for synthesizing image data in this embodiment are summarized below.[[]]

[0112] First, in the method for synthesizing image data according to this embodiment, for one field of view, a synthesized image data is obtained by synthesizing a plurality of image data captured by moving the camera unit 21 to different positions in the Z direction (imaging direction), and the synthesis process is configured to be executed from the image data in which the imaging position of the camera unit 21 is far from the inspection object 12 (from the reference plane). With such a configuration, as the camera unit 21 moves farther from the inspection object 12, the amount of reflected light from the inspection object 12 decreases, and in particular, noise due to unnecessary light in pixels that are not in focus decreases, so that measurement error values included in the height information are reduced.[[]]

[0113] Second, in the method for synthesizing image data according to the present embodiment, in the image data at each imaging position, pixels that are in focus are selected as the values of the pixels in the synthesized image data. As criteria for determining in-focus pixels, height information of each pixel measured by the PMP method, or the amplitude and offset in the PMP method are used, and pixels that simultaneously satisfy the above-described conditions (1) and (2), or pixels that simultaneously satisfy conditions (1) and (3) are selected; a method of selecting pixels that satisfy only condition (1); a method of selecting pixels that simultaneously satisfy condition (1) and condition (2′) or condition (1) and condition (3′); and a method of selecting pixels that simultaneously satisfy condition (1) and any one of conditions (2a), (2a′), (3a), and (3a′). With this configuration, in-focus pixels can be accurately selected with simple arithmetic processing. Note that, for pixels that satisfy the in-focus conditions in the image data at different imaging positions, as described in the first feature, the value at the imaging position farthest from the test object 12 (from the reference plane) is selected.

[0114] Third, in the method for synthesizing image data according to the present embodiment, when acquiring height information from the pattern image data by the PMP method, the above-described synthesis process is executed on the pattern image data, and interpolation processing for non-measured pixels and distortion correction processing (height post-processing) are executed on the synthesized image data. If interpolation processing and correction processing are performed on the height map calculated from the pattern image data captured at each imaging position, the values of the interpolated and corrected pixels (unreliable data) may be selected as the values of the pixels in the synthesized image data, which may deteriorate the accuracy of the image data. However, by performing interpolation processing and correction processing on the synthesized image data (height map), interpolation and correction are performed on the in-focus image data, so that high-accuracy height information can be obtained.

[0115] Fourth, in the method for synthesizing image data according to the present embodiment, the parameter for removing measurement errors is set to a value that maximizes the number of measurement points on the reference plane (substrate surface) when the camera unit 21 is in focus near the reference plane (when P(0)), and when the camera unit 21 is not in focus near the reference plane (when P(1) to P(3)), it is set to a value that removes points with a high suspicion of measurement error. With such a configuration, in-focus synthesized image data can be obtained, and the accuracy of the height information can be improved.

[0116] Fifth, in the method for synthesizing image data according to the present embodiment, the imaging process of the image data is performed in the order from the position of the camera unit 21 in the Z direction being far from the inspection object 12, and the height basic process and the synthesized image update process are executed for each imaging. With such a configuration, when the imaging process is being executed at the next imaging position, the height basic process and the synthesized image update process can be executed in parallel using the image data captured at the previous imaging position, and the processing time can be shortened.

[0117] According to the above configuration, even for the inspection object 12 on which components with height are mounted on the substrate, in-focus image data can be obtained. Therefore, the accuracy of the height information of the height map calculated from the pattern image data is improved, and clear two-dimensional image data can be obtained.

[0118] It should be noted that the above embodiments do not limit the invention described in the claims, and it goes without saying that not all combinations of the characteristic matters described in the embodiments are essential matters of the solution means.

Explanation of Reference Numerals

[0119] 10 Inspection apparatus (image processing apparatus) 12 Inspection object 21 Camera unit (imaging unit) 22 Lighting unit (lighting unit) 23 Projection unit (projection unit) 30 Control unit (control unit)

Claims

1. An imaging unit capable of changing the relative position in the imaging direction with respect to the subject, and a control unit, wherein the control unit changes the relative position in the imaging direction between the imaging unit and the subject, and obtains first image data which is image data of the subject imaged by the imaging unit at positions with different imaging directions and when the imaging unit is in focus on a reference plane which is a predetermined plane of the subject, and at least one piece of second image data when the imaging unit is in focus closer to the imaging unit side than the reference plane, each of the first image data and the second image data has two-dimensional image data of the subject and pattern image data of the subject onto which a periodically changing stripe pattern is projected, and creates a height map including the amplitude and phase of the luminance value calculated for each pixel by the phase shift method, and height information calculated from the amplitude, from each of the pattern image data of the first image data and the pattern image data of the second image data, selects one by one the second image data in the order from the position of the imaging unit when the second image data was obtained being far from the reference plane as comparison target image data, compares the values of the pixels at the same positions in the height map of the first image data and the height map of the comparison target image data to determine whether the value of the pixel of the comparison target image data satisfies the condition of a pixel in focus, and when the height information of the pixel of the comparison target image data is within a predetermined range including the most in-focus position of the imaging unit when the subject obtained the comparison target image data, and the amplitude of the pixel of the comparison target image data is equal to or greater than the value obtained by multiplying the amplitude of the pixel at the same position in the first image data by a predetermined weight count, determines that the condition is satisfied, and executes a composite image update process of selecting the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data, Using the first image data as comparison target image data, determine whether the pixel values of the height map of the comparison target image data satisfy the condition. When the imaging unit is within a predetermined range including the optimal focus position when the subject acquired the comparison target image data, as determined from the height information of the comparison target image data, determine that the condition is satisfied, and execute the composite image update process of selecting the pixel values of the two-dimensional image data of the comparison target image data as the corresponding pixel values of the composite image data. Image processing apparatus. However, in the composite image update process, when it is determined that the condition is satisfied for pixels at the same position in two or more pieces of comparison target image data, among the comparison target image data, select the pixel value of the comparison target image data captured when the imaging unit is at the position farthest from the reference plane as the corresponding pixel value of the composite image data.

2. An imaging unit capable of changing the relative position in the imaging direction with respect to the subject, A control unit, and having The control unit Changes the relative position in the imaging direction between the imaging unit and the subject, and is image data of the subject captured by the imaging unit at different positions in the imaging direction. The first image data when the imaging unit is in focus on a reference plane which is a predetermined plane of the subject, and at least one second image data when the imaging unit is in focus closer to the imaging unit side than the reference plane are acquired. Each of the first image data and the second image data has two-dimensional image data of the subject and pattern image data of the subject onto which a periodically changing stripe pattern is projected. From each of the pattern image data of the first image data and the pattern image data of the second image data, an amplitude, an offset value, and a phase of luminance values calculated for each pixel by the phase shift method, and a height map including height information calculated from the phase are created. Select one by one the second image data in the order from the position of the imaging unit when the second image data was acquired being far from the reference plane as comparison target image data, compare the values of the pixels at the same positions in the height map of the first image data and the height map of the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition of being a focused pixel. From the height information of the pixel of the comparison target image data, when the subject is within a predetermined range including the most in-focus position of the imaging unit when the comparison target image data was acquired, and the contrast calculated from the amplitude and the offset value of the pixel of the comparison target image data is equal to or greater than the value obtained by multiplying the contrast calculated from the amplitude and the offset value of the pixel at the same position of the first image data by a predetermined weight count, it is determined that the condition is satisfied, and a composite image update process is executed to select the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data. Use the first image data as comparison target image data, determine whether the value of the pixel in the height map of the comparison target image data satisfies the condition, and when the subject is within a predetermined range including the most in-focus position of the imaging unit when the comparison target image data was acquired from the height information of the pixel of the comparison target image data, it is determined that the condition is satisfied, and the composite image update process is executed to select the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data. Image processing apparatus. However, in the composite image update process, when it is determined that the condition is satisfied for pixels at the same position in two or more pieces of comparison target image data, among the comparison target image data, select the value of the pixel of the comparison target image data captured when the imaging unit is at the position farthest from the reference plane as the value of the corresponding pixel of the composite image data.

3. An imaging unit capable of changing the relative position in the imaging direction with respect to the subject, A control unit, and The control unit is Image data of the subject captured by the imaging unit at positions with different imaging directions by changing the relative position in the imaging direction between the imaging unit and the subject, the first image data when the imaging unit is in focus on a reference plane which is a predetermined plane of the subject, and at least one second image data when the imaging unit is in focus closer to the imaging unit side than the reference plane are acquired. Each of the first image data and the second image data includes two-dimensional image data of the subject, first pattern image data of the subject onto which a first stripe pattern changing in a first period is projected, and second pattern image data of the subject onto which a second stripe pattern changing in a second period different from the first period is projected. For each of the first image data and the second image data, a first amplitude and a first phase of first luminance values calculated pixel by pixel from the first pattern image data by a phase shift method, a second amplitude and a second phase of second luminance values calculated pixel by pixel from the second pattern image data by the phase shift method, and height information calculated pixel by pixel from the first phase and the second phase by the phase shift method are included to create a height map. The second image data is selected one by one in the order from the position of the imaging unit when the second image data was acquired being far from the reference plane as comparison target image data. The values of pixels at the same positions in the height map of the first image data and the height map of the comparison target image data are compared to determine whether the value of the pixel of the comparison target image data satisfies the condition of a pixel in focus. From the height information of the pixel of the comparison target image data, within a predetermined range including the best focus position of the imaging unit when the subject acquired the comparison target image data, and when an amplitude ratio which is the ratio of the first amplitude and the second amplitude of the pixel of the comparison target image data is equal to or greater than a value obtained by multiplying the amplitude ratio which is the ratio of the first amplitude and the second amplitude of the pixel at the same position in the first image data by a predetermined weighting factor, it is determined that the condition is satisfied, and a composite image update process is executed to select the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data. Using the first image data as comparison target image data, determine whether the pixel values of the height map of the comparison target image data satisfy the condition. When, from the height information of each pixel of the comparison target image data, the subject is within a predetermined range including the optimal focus position of the imaging unit when the comparison target image data was acquired, determine that the condition is satisfied, and execute the composite image update process of selecting the pixel value of the two-dimensional image data of the comparison target image data as the corresponding pixel value of the composite image data. An image processing apparatus. However, in the composite image update process, when it is determined that the condition is satisfied for pixels at the same position in two or more comparison target image data, among the comparison target image data, select the pixel value of the comparison target image data captured when the imaging unit is at the position farthest from the reference plane as the corresponding pixel value of the composite image data.

4. An imaging unit capable of changing the relative position in the imaging direction with respect to the subject, a control unit, and has the control unit changes the relative position in the imaging direction between the imaging unit and the subject, and acquires first image data of the subject imaged by the imaging unit at positions with different imaging directions, where the imaging unit is in focus on a reference plane that is a predetermined plane of the subject, and at least one second image data when the imaging unit is in focus closer to the imaging unit side than the reference plane. Each of the first image data and the second image data has two-dimensional image data of the subject, as well as first pattern image data of the subject onto which a first stripe pattern that changes in a first period is projected and second pattern image data of the subject onto which a second stripe pattern that changes in a second period different from the first period is projected. For each of the first image data and the second image data, create a height map including the first amplitude, the first offset value, and the first phase of the first luminance value calculated for each pixel by the phase shift method from the first pattern image data, the second amplitude, the second offset value, and the second phase of the second luminance value calculated for each pixel by the phase shift method from the second pattern image data, and the height information calculated for each pixel by the phase shift method from the first phase and the second phase. Select one by one the second image data in the order of the position of the imaging unit being far from the reference plane when the second image data is acquired from the second image data as comparison target image data, compare the values of the pixels at the same positions in the height map of the first image data and the height map of the comparison target image data, and determine whether the value of the pixel of the comparison target image data satisfies the condition of being a focused pixel. From the height information of the pixel of the comparison target image data, the subject is within a predetermined range including the most in-focus position of the imaging unit when the second image data is acquired, and the contrast ratio, which is the ratio of the first contrast calculated from the first amplitude and the first offset value of the pixel of the comparison target image data to the second contrast calculated from the second amplitude and the second offset value, is the ratio of the first contrast calculated from the first amplitude and the first offset value of the pixel at the same position of the first image data to the second contrast calculated from the second amplitude and the second offset value When it is equal to or greater than the value obtained by multiplying the contrast ratio by a predetermined weight count, it is determined that the condition is satisfied, and a composite image update process is executed to select the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data. Use the first image data as comparison target image data, determine whether the value of the pixel in the height map of the comparison target image data satisfies the condition, and from the height information of the pixel of the comparison target image data, when the subject is within a predetermined range including the most in-focus position of the imaging unit when the second image data is acquired, it is determined that the condition is satisfied, and the composite image update process is executed to select the value of the pixel of the two-dimensional image data of the comparison target image data as the value of the corresponding pixel of the composite image data. Image processing apparatus. However, in the composite image update process, when it is determined that the condition is satisfied for the pixels at the same positions in two or more pieces of comparison target image data, among the comparison target image data, select the value of the pixel of the comparison target image data captured when the imaging unit is at the position farthest from the reference plane as the value of the corresponding pixel of the composite image data.

5. The control unit acquires the first image data when the imaging unit is in focus on the reference plane of the subject, and then acquires the second image data from the position of the imaging unit that is farther away from the subject. The image processing apparatus according to any one of claims 1 to 4.

6. The control unit Each time the second image data is acquired, the composite image update process is executed. The image processing apparatus according to claim 5.

Images