Image processor, method for processing image, and three-dimensional shape measuring device

Abstract

To provide an image processor, a method for processing an image, and a three-dimensional shape measuring device which can accurately acquire three-dimensional shape data of a measurement target object.SOLUTION: An information processor 100 acquires a plurality of images of a measurement target object Ob which an imaging device 10 imaged with different focal positions and corrects blurring of the acquired images on the basis of a point spread function for each focal position of the image formation optical system 14 of the imaging device 10. The image processor 100 calculates the evaluation value of the focus degree of each of a plurality of pixels after the blurring is corrected, and generates the three-dimensional shape data of the measurement target object Ob on the basis of the calculated evaluation degree of the focus degree for each pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and a three-dimensional shape measuring apparatus, and more particularly to an image processing apparatus, an image processing method, and a three-dimensional shape measuring apparatus that process multiple images of a measurement target captured at different focal positions. [Background technology]

[0002] The focus variation method (focus shift method) is a known method for measuring three-dimensional shapes using an optical microscope (for example, Patent Documents 1 and 2). The focus variation method involves capturing multiple images at different focal positions and measuring the three-dimensional shape of the object from the degree of focus of each pixel. Specifically, an evaluation value of the degree of focus is calculated for each pixel from the obtained images, and the focal position where the calculated evaluation value is maximized is identified. This allows the height of the object at the position corresponding to each pixel to be determined, thereby measuring its three-dimensional shape. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2013-20140 [Patent Document 2] International Publication No. 2009 / 096422 [Overview of the project] [Problems that the invention aims to solve]

[0004] Generally, the focus variation method has the advantage of being able to realize low-cost measuring equipment because it does not require special modifications to the optical system compared to other measurement methods such as white light interferometry.

[0005] On the other hand, the focus variation method has a problem in that it cannot accurately measure areas with low reflectivity when areas with high reflectivity and areas with low reflectivity coexist within the field of view. This occurs because the contrast of the blur boundary of the high-reflectivity area becomes stronger than the contrast of the surface of the low-reflectivity area. Figure 10 shows an example of an image captured when areas with high reflectivity and areas with low reflectivity coexist within the field of view. The figure shows an example of imaging a cone-shaped object under incident illumination while changing the focal position. The figure also shows an example of an image sliced ​​in the xz plane to make it easier to understand the effect of blur due to the focal position. In the figure, the triangular line L represents the original shape of the object being measured. When imaging a cone-shaped object under incident illumination, the vertex shows strong reflection against the inclined surface with low reflectivity. As a result, as shown in the elliptical region C in Figure 10, the blur of the vertex with strong reflection overwhelms the surface contrast of the original shape, making accurate measurement impossible.

[0006] One way to address this problem is to use a polarizing filter (a method that reduces direct reflection and acquires scattered light). However, using a polarizing filter has the disadvantage of significantly reducing the amount of light and increasing the exposure time. Longer exposure times lead to longer measurement times. Furthermore, the use of polarizing filters also leads to an increase in the number of components.

[0007] This invention has been made in view of these circumstances, and aims to provide an image processing device, an image processing method, and a three-dimensional shape measuring device that can accurately acquire three-dimensional shape data of an object to be measured. [Means for solving the problem]

[0008] To solve the above problems, an image processing apparatus according to a first aspect of the present invention is an image processing apparatus that processes a plurality of images of a measurement target captured at different focal positions, comprising: an image acquisition unit that acquires a plurality of images of the measurement target; an image processing unit that processes the plurality of images and corrects image blur based on the point spread function for each focal position of the optical system of the imaging device that captured the plurality of images; and a three-dimensional shape data generation unit that calculates an evaluation value of the degree of focus for each pixel of the plurality of images after blur correction, and generates three-dimensional shape data of the measurement target based on the calculated evaluation value of the degree of focus for each pixel.

[0009] An image processing apparatus according to a second aspect of the present invention, in the image processing apparatus according to the first aspect, applies a frequency domain restoration filter based on a point spreading function to the frequency domain image obtained by Fourier transforming the image to correct the frequency domain image, and then performs an inverse Fourier transform on the corrected frequency domain image to correct the blur of the image.

[0010] An image processing apparatus according to a third aspect of the present invention, in the image processing apparatus according to the second aspect, calculates an inverse frequency domain filter from a frequency domain point spreading function obtained by Fourier transforming the point spreading function, and sets the calculated inverse frequency domain filter as the frequency domain restoration filter.

[0011] In the fourth aspect of the present invention, the image processing apparatus of the second aspect is configured such that a frequency-domain Wiener filter calculated from a frequency-domain point spreading function obtained by performing a Fourier transform on the point spreading function is set as the frequency-domain restoration filter.

[0012] An image processing apparatus according to a fifth aspect of the present invention, in the image processing apparatus according to the fourth aspect, wherein the Wiener filter is calculated by the following formula: PSF f *(ω x ,ω y ,ω z ) / {PSF f (ω x ,ω y ,ω z)PSF f *(ω x ,ω y ,ω z )+SN} However, PSF(x, y, z) is the voxel data of the point spread function, PSF f (ω x ,ω y ,ω z ) is the mapping obtained by Fourier-transforming PSF(x, y, z), PSF f *(ω x ,ω y ,ω z ) is PSF f (ω x ,ω y ,ω z )'s complex conjugate, and SN is a constant.

[0013] In the image processing apparatus according to the sixth aspect of the present invention, in the image processing apparatus of the fifth aspect, the constant SN is experimentally set.

[0014] In the image processing apparatus according to the seventh aspect of the present invention, in the image processing apparatus of any one of the first to sixth aspects, the image processing unit processes the voxel data of the measurement object based on the voxel data of the point spread function to correct the blur of a plurality of images.

[0015] In the image processing apparatus according to the eighth aspect of the present invention, in the image processing apparatus of the seventh aspect, when the size of the voxel data of the point spread function is smaller than the size of the voxel data of the measurement object, the voxel data of the point spread function is zero-padded to adjust it to the same size.

[0016] In the image processing apparatus according to the ninth aspect of the present invention, in the image processing apparatus of any one of the first to eighth aspects, the point spread function is obtained by setting a bright point on the optical axis of the optical system and relatively moving the imaging device with respect to the bright point to perform imaging.

[0017] In the image processing apparatus according to the tenth aspect of the present invention, in the image processing apparatus of the ninth aspect, the diameter of the bright point is set to be not greater than the optical resolution of the imaging device via the optical system.

[0018] An image processing apparatus according to the 11th aspect of the present invention is an image processing apparatus according to the 9th or 10th aspect, wherein the bright spots are composed of pinhole illumination or fluorescent beads.

[0019] An image processing apparatus according to the twelfth aspect of the present invention, in an image processing apparatus according to any one of the first to eleventh aspects, the image processing unit acquires temperature information when the object to be measured is imaged, and corrects the blur of multiple images based on a point spread function corresponding to the acquired temperature.

[0020] In the thirteenth aspect of the present invention, an image processing apparatus according to any one of the first to twelfth aspects of the present invention captures multiple images of an object to be measured such that the maximum and minimum brightness values ​​of the object to be measured fall within the dynamic range that can be captured by the imaging device.

[0021] A three-dimensional shape measuring apparatus according to a fourteenth aspect of the present invention comprises an imaging device that moves relative to an object to be measured set on the optical axis and captures a plurality of images with different focal positions, and an image processing apparatus according to any one of the first to thirteenth aspects that processes the plurality of images of the object to be measured captured by the imaging device.

[0022] An image processing method according to a 15th aspect of the present invention is an image processing method for processing multiple images of a measurement target captured at different focal positions, wherein multiple images of the measurement target are acquired, the blur of the multiple images is corrected based on the point spread function for each focal position of the optical system of the imaging device that captured the multiple images, an evaluation value of the degree of focus is calculated for each pixel of the multiple images after blur correction, and three-dimensional shape data of the measurement target is generated based on the calculated evaluation value of the degree of focus for each pixel. [Effects of the Invention]

[0023] According to the present invention, three-dimensional shape data of the object to be measured can be acquired with high accuracy. [Brief explanation of the drawing]

[0024] [Figure 1] This is a diagram showing the schematic configuration of a three-dimensional shape measuring device. [Figure 2] This figure shows an example of the hardware configuration of an image processing device. [Figure 3] This is a block diagram of the main functions of the image processing device. [Figure 4] This is a conceptual diagram of imaging via an imaging optical system. [Figure 5] This figure shows an example of a method for obtaining the point spread function of an imaging optical system. [Figure 6] This flowchart shows the measurement procedure using a three-dimensional shape measuring device. [Figure 7] This figure shows an example of image restoration processing using the Wiener filter. [Figure 8] This is a magnified view of a portion of Figure 7. [Figure 9] This is a magnified view of a portion of an image that has been restored using an inverse filter. [Figure 10] This figure shows an example of an image captured when areas with high reflectivity and areas with low reflectivity coexist within the field of view. [Modes for carrying out the invention]

[0025] Preferred embodiments of the present invention will be described below with reference to the attached drawings.

[0026] [Three-dimensional shape measuring device] Figure 1 shows a schematic configuration of a three-dimensional shape measuring device.

[0027] As shown in the figure, the three-dimensional shape measuring device 1 of this embodiment includes an imaging device 10 and an image processing device 100.

[0028] [Imaging device] The imaging device 10 is composed of a so-called optical microscope and captures images of the object Ob while changing the focal position. As shown in Figure 1, the imaging device 10 of this embodiment includes a stage 11, an illumination light source 12, an illumination optical system 13, an imaging optical system 14, a camera 15, a focus drive unit 16, an operation unit 17, and a control unit 18, etc.

[0029] Stage 11 has a mounting surface 11a on which the object to be measured Ob is placed. The mounting surface 11a is a flat surface and constitutes the reference surface for measurement. Specifically, the X and Y axes are set on the mounting surface 11a. The X and Y axes are set as two mutually orthogonal axes.

[0030] The illumination light source 12 can be a halogen lamp, metal halide lamp, mercury lamp, xenon lamp, light-emitting diode (LED), or the like.

[0031] The illumination optical system 13 includes an illumination lens 20, a beam splitter 21, an aperture diaphragm 22, and an objective lens 23. Light emitted from the illumination light source 12 (illumination light) is irradiated onto the object to be measured Ob placed on the stage 11 via the illumination lens 20, beam splitter 21, aperture diaphragm 22, and objective lens 23. The illumination light can also be guided from the illumination light source 12 to the illumination optical system 13 using a ride guide.

[0032] The imaging optical system 14 comprises an objective lens 23, an aperture diaphragm 22, a beam splitter 21, and an imaging lens 24. The beam splitter 21, aperture diaphragm 22, and objective lens 23 are shared with the illumination optical system 13. Light reflected from the object Ob is incident on the camera 15 via the objective lens 23, aperture diaphragm 22, beam splitter 21, and imaging lens 24.

[0033] Camera 15 has an image sensor and electronically captures an image of the object Ob to be measured. For example, a CMOS image sensor (complementary metal oxide semiconductor image sensor) or a CCD image sensor (charge-coupled device image sensor) can be used as the image sensor. The image captured by camera 15 is output to the image processing device 100.

[0034] The focus drive unit 16 moves the illumination light source 12, illumination optical system 13, imaging optical system 14, and camera 15 integrally along the imaging optical axis to vary the focal position. The imaging optical axis is set perpendicular to the mounting surface 11a of the stage 11. Therefore, the illumination light source 12, illumination optical system 13, imaging optical system 14, and camera 15 move perpendicular to the mounting surface 11a of the stage 11. The axis parallel to the imaging optical axis is defined as the Z-axis. The focus drive unit 16 includes a guide mechanism and a drive mechanism. The guide mechanism guides the unit (optical head) including the illumination light source 12, illumination optical system 13, imaging optical system 14, and camera 15 along the Z-axis. The drive mechanism consists of, for example, a lead screw mechanism and a motor that drives the lead screw mechanism.

[0035] The control unit 17 has various operation buttons for operating the imaging device 10 and accepts user input. The various operation buttons include a touch panel.

[0036] The control unit 18 provides overall control over the operation of the imaging device 10. Specifically, it performs light emission control for the illumination light source 12, image capture control for the camera 15, and drive control (feed control) for the focus drive unit 16. The control unit 18 is composed of a computer. In other words, the computer functions as the control unit 18 by executing a predetermined program.

[0037] The imaging device 10, configured as described above, images the object Ob placed on the stage 11 while changing the focal position at a constant pitch. Specifically, the focus drive unit 16 moves the optical head along the Z-axis at a constant pitch, and images the object Ob at each position. The captured images are output sequentially to the image processing device 100.

[0038] [Image processing device] Figure 2 shows an example of the hardware configuration of an image processing device.

[0039] The image processing device 100 is composed of a so-called computer and includes a processor 101, main memory 102, auxiliary storage 103, input device 104, output device 105, and input / output interface 106, etc.

[0040] The processor 101 is composed of components such as a CPU (central processing unit) and a GPU (graphics processing unit).

[0041] The main memory 102 is composed of, for example, DRAM (dynamic random access memory), SRAM (static random access memory), etc.

[0042] The auxiliary storage device 103 is composed of, for example, an SSD (solid state drive), an HDD (hard disk drive), etc. The auxiliary storage device 103 stores programs executed by the processor 101 and various data.

[0043] The processor 101 reads the program stored in the auxiliary storage device 103, loads it into the main memory device 102, and executes it, thereby realizing various functions as described later.

[0044] The input device 104 is a device for the user to input instructions to the image processing device 100, and consists of, for example, a mouse, keyboard, or touch panel.

[0045] The output device 105 is a device that outputs (including display) measurement results, etc., and is composed of, for example, a liquid crystal display (LCD), an organic electroluminescent display (OELD), etc.

[0046] The input / output interface 106 is an interface for connecting input / output devices, and is configured as, for example, USB (universal serial bus). The camera 15 of the imaging device 10 is connected to the image processing device 100 via the input / output interface 106.

[0047] Figure 3 is a block diagram of the main functions of the image processing device.

[0048] As shown in the figure, the image processing device 100 has the functions of an image acquisition unit 100A, a preprocessing unit 100B, and a three-dimensional shape data generation unit 100C. The functions of each unit are realized by the processor 101 executing a predetermined program (image processing program).

[0049] The image acquisition unit 100A acquires image data of the object to be measured Ob captured by the imaging device 10. This image data is image data of the object to be measured Ob captured while changing the focal position (so-called multi-focus image data). The acquired series of image data is recorded in the auxiliary storage device 103.

[0050] The preprocessing unit 100B performs image processing to correct image blur as a preprocessing step for a series of image data. Details of the image processing will be described later. In this embodiment, the preprocessing unit 100B is an example of an image processing unit. The preprocessed image data is recorded in the auxiliary storage device 103.

[0051] The three-dimensional shape data generation unit 100C applies the focus variation method algorithm to the preprocessed image data to generate three-dimensional shape data of the object Ob. Here, three-dimensional shape data refers to a data set in which the height information of the object represented by each pixel is assigned to every pixel that makes up the image of the object. By acquiring three-dimensional shape data, it becomes possible to display the object Ob in three dimensions using computer graphics. The focus variation method itself is a well-known technique, so a detailed explanation will be omitted, but the three-dimensional shape data is generated from the multifocal image of the object Ob in roughly the following steps. First, differential calculation is performed on each pixel of the obtained image to calculate the evaluation value of the degree of focus. The degree of focus is a numerical value that indicates the degree to which the image is in focus. Next, for each pixel that makes up the image of the object, the focal position where the degree of focus is maximized is identified. From the information of the identified focal position, the height information of the object Ob at the position corresponding to each pixel is obtained. For example, suppose that the focal position where the evaluation value of the degree of focus is maximized for the pixel located at coordinates (xn,yn) is Pn. In this case, the height zn associated with the focal position Pn is acquired as height information for the pixel located at coordinates (xn,yn). By acquiring height information for all pixels, three-dimensional shape data of the object Ob can be obtained. Normally, imaging is performed by changing the focal position at a constant pitch from a reference point (origin). The order of imaging can also be assigned to the height information. The obtained three-dimensional shape data is associated with the image data obtained by imaging and recorded in the auxiliary storage device 103.

[0052] [Pre-processing] As described above, the focus variation method has a problem in that when areas with high reflectivity and areas with low reflectivity coexist within the field of view, the contrast at the blur boundary of the high-reflectivity areas becomes stronger than the contrast of the surface of the low-reflectivity areas, making it impossible to accurately obtain the height of the low-reflectivity areas.

[0053] Therefore, in this embodiment, the effect of blurring in high-reflectivity areas is reduced by applying predetermined image processing to the image obtained by imaging. Specifically, image restoration processing is performed using a restoration filter based on the point spread function (PSF) of the imaging optical system 14 of the imaging device 10. The point spread function is also called the point image distribution function or blur function.

[0054] First, we will explain the image restoration process using a restoration filter.

[0055] Figure 4 is a conceptual diagram of imaging via an imaging optical system. This figure illustrates how a measurement target in space is imaged using a lens. To facilitate understanding of the effect of blurring due to focal position, the figure shows an example of an image sliced ​​in the xz plane. An image sliced ​​in the xz plane is generated by extracting a specific sequence of pixels parallel to the x-axis from each image. The example shown in the figure illustrates the case where a sequence of pixels passing through the center of the image and parallel to the x-axis is extracted from each image. The measurement target is a frustum shape.

[0056] Let IO(x,y) be the image of the ideal object to be measured (original image), and PSF(x,y) be the point spread function of the imaging optical system 14.

[0057] The captured image I(x,y) can be expressed as I(x,y) = IO(x,y) * PSF(x,y) ... Equation (1). Here, "*" represents a convolution operation.

[0058] Image restoration is the process of reconstructing the original image IO(x,y) of the object being measured, Ob, which is unaffected by blur, from the captured image I(x,y), given that the point spread function PSF(x,y) is known. This operation is called deconvolution.

[0059] When equation (1) is subjected to a two-dimensional Fourier transform, I f (ω x ,ω y )=IO f (ω x ,ωy )PSF f (ω x ,ω y ) ...This becomes equation (2). Note that I f (ω x ,ω y ) represents the Fourier transform of I(x,y). IO f (ω x ,ω y ) represents the Fourier transform of IO(x,y). PSF f (ω x ,ω y ) represents the Fourier transform of PSF(x,y). (ω x ,ω y ) is a spatial frequency coordinate.

[0060] Since the point spreading function PSF(x,y) is known, the point spreading function PSF in the frequency domain is f (ω x ,ω y ) is also known. f (ω x ,ω y ) to PSF f (ω x ,ω y Dividing by ) gives IC f (ω x ,ω y )≡I f (ω x ,ω y ) / PSF f (ω x ,ω y )=IO f (ω x ,ω y )...Equation (3) is obtained, and the frequency domain restored image IC f (ω x ,ω y ) is obtained. By performing an inverse Fourier transform on this, the reconstructed image IC(x,y) in spatial coordinates is obtained. 1 / PSF f (ω x ,ω y This is called an inverse filter.

[0061] The image processing apparatus 100 of this embodiment uses an inverse filter 1 / PSF f (ω x ,ωy Use ) as a restoration filter and perform image restoration processing.

[0062] Inverse filter 1 / PSF f (ω x , ω y ) is set for each focal position. The information of the set inverse filter 1 / P(u, v) is acquired in advance and recorded in the auxiliary storage device 103. The preprocessing unit 100B reads the information of the inverse filter 1 / PSF f (ω x , ω y ) recorded in the auxiliary storage device 103 and performs image restoration processing. The inverse filter 1 / PSF f (ω x , ω y ) The generation method will be described later.

[0063] The preprocessing unit 100B performs image restoration processing in the following procedure.

[0064] Here, the case of processing the image data I(x, y) of the image captured at the focal position Pn will be described.

[0065] First, the image data I(x, y) of the processing target is Fourier-transformed and mapped to the frequency domain.

[0066] I(x, y) → I f (ω x , ω y ) If(ω x , ω y ) is the image data in the frequency domain.

[0067] Next, obtain the information of the inverse filter 1 / PSF in the frequency domain corresponding to the focal position Pn f (ω x , ω y ).

[0068] Next, the inverse filter 1 / PSF in the frequency domain is applied to the image data I f (ω x , ω y ) in the frequency domain. f (ωx , ω y ) is multiplied, and the IC f (ω x , ω y ) is calculated.

[0069] IC f (ω x , ω y ) = I f (ω x , ω y ) × [1 / PSF f (ω x , ω y )] The calculated IC f (ω x , ω y ) is subjected to inverse Fourier transform, the absolute value of each pixel is obtained, and the restored image IC(x, y) is obtained.

[0070] The preprocessing unit 100B performs image restoration processing on the image data of each image obtained by imaging, and corrects blurring. The image data after blurring correction is recorded in the auxiliary storage device 103.

[0071] [Method for obtaining point spread function] FIG. 5 is a diagram showing an example of a method for obtaining the point spread function of an imaging optical system.

[0072] This figure shows an example of the case where the point spread function of the imaging optical system 14 is measured using the pinhole illumination 200.

[0073] As shown in this figure, a bright spot is generated at the position of the reference plane by the pinhole illumination 200, and the image of the bright spot is imaged while changing the focal position. The bright spot is set on the optical axis of the imaging optical system 14. Thereby, the point spread function for each focal position is obtained. The point spread function for each focal position constitutes the voxel data PSF(x, y, z) of the point spread function. Here, z is the position (height) in the z-axis direction corresponding to the focal position.

[0074] The pinhole illumination 200 comprises a light source 201, a focusing lens 202, and a pinhole plate 203. The pinhole plate 203 has a pinhole 204. In the pinhole illumination 200, light from the light source 201 is focused by the focusing lens 202, and the focused light is emitted from the pinhole 204 provided in the pinhole plate 203. The light emitted from the pinhole 204 constitutes a bright spot. The pinhole 204 is positioned on a reference plane and on the optical axis of the imaging optical system 14. Furthermore, the diameter of the pinhole 204 is set to be less than or equal to the optical resolution of the imaging device 10 via the imaging optical system 14.

[0075] Bright spots can also be generated by other methods. For example, bright spots can be generated using fluorescent beads. The method using fluorescent beads involves exciting tiny fluorescent beads with external illumination and imaging their fluorescence. In this case as well, the diameter of the fluorescent beads is set to be less than or equal to the optical resolution of the imaging device 10 via the imaging optical system 14.

[0076] The point spread function obtained by the above method is a function specific to the imaging optical system, so it does not need to be reacquired unless the camera or optical system changes. However, in environments where the temperature changes significantly, it is desirable to acquire the function according to the temperature. In this case, the image reconstruction process acquires temperature information at the time the object to be measured was imaged and performs image reconstruction based on the point spread function corresponding to the temperature. That is, the image reconstruction process is performed using a reconstruction filter (frequency domain reconstruction filter) corresponding to the temperature.

[0077] [Generating a restoration filter] When using the inverse filter as the reconstruction filter, the point spreading function PSF(x,y) is Fourier transformed to obtain the PSF f (ω x ,ω y ) is calculated, and then its reciprocal 1 / PSF is calculated. f (ω x ,ω y ) is calculated and an inverse filter is generated.

[0078] In the above embodiment, the inverse filter in the frequency domain is calculated and stored in advance. However, it is also possible to calculate and acquire the inverse filter in the frequency domain during the image restoration process.

[0079] Furthermore, if the size of the image to be processed and the size of the point spreading function used are different, zero padding can be applied to the point spreading function to make the sizes of both match.

[0080] [Measurement operation] Figure 6 is a flowchart showing the measurement procedure using a three-dimensional shape measuring device.

[0081] First, the imaging device 10 captures an image of the object to be measured Ob (step S1). Specifically, the object to be measured Ob is placed on the stage 11, and the camera 15 captures an image of the object to be measured Ob placed on the stage 11 while changing the focal position at a constant pitch. The image data of the images obtained by the capture is sequentially output to the image processing device 100. The image processing device 100 acquires the image data output from the imaging device 10 and records it in the auxiliary storage device 103.

[0082] Next, as a preprocessing step, image restoration is performed on the image data obtained from the imaging (Step S2).

[0083] As described above, the image restoration process performs a Fourier transform on the image data I(x,y) to be processed, and then performs an If(ω x ,ω y ) calculates the If(ω x ,ω y ) inverse filter 1 / PSF in the frequency domain f (ω x ,ω y Multiply by )) and IC f (ω x ,ω y ) is calculated. Then, the calculated IC f (ω x ,ω yThe image is then subjected to an inverse Fourier transform to obtain the absolute value of each pixel, thereby obtaining the reconstructed image IC(x,y). This corrects the blur of the captured image. The image data after blur correction is recorded in the auxiliary storage device 103.

[0084] Next, three-dimensional shape data is generated from the image data after image restoration processing using the focus variation method (step S3). That is, for multiple images after blur correction, an evaluation value of the degree of focus is calculated for each pixel, and three-dimensional shape data of the object Ob is generated based on the calculated pixel-by-pixel evaluation value of the degree of focus. Here, since the evaluation value of the degree of focus is calculated from the blur-corrected image, it can be calculated with high accuracy. In other words, even if there are areas with high reflectivity in the image, it is possible to suppress the contrast at the blur boundary of those high-reflectivity areas from becoming stronger than the surface contrast of low-reflectivity areas. As a result, the evaluation value of the degree of focus can be calculated with high accuracy for each image. Furthermore, as a result, three-dimensional shape data can be generated with high accuracy.

[0085] [Differentiation] [Image restoration process using voxel data] It is also possible to configure the system to acquire image data as voxel data and perform image restoration processing in a batch. This process is performed using the following procedure.

[0086] Let I(x,y,z) be the voxel data of the image obtained from imaging. Also, let (N) be the number of pixels at this time. x ,N y ), the number of slices (the number of height-direction positions where imaging is performed) is N z Let's assume that.

[0087] Furthermore, the voxel data of the point spreading function is defined as PSF(x,y,z). The voxel data of the point spreading function, PSF(x,y,z), is acquired by imaging bright spots set on a reference plane while changing the focal position at a constant pitch.

[0088] If the size of the image voxel data I(x,y,z) differs from the size of the point spread function voxel data PSF(x,y,z), zero-padding is applied to the point spread function voxel data PSF(x,y,z) to make the sizes of both the image and the point spread function match.

[0089] The three-dimensional Fourier transform maps I(x,y,z) to the frequency domain.

[0090] I(x,y,z)→I f (ω x ,ω y, ω z ) The PSF(x,y,z) is mapped to the frequency domain using the three-dimensional Fourier transform.

[0091] PSF(x,y,z) → PSF f (ω x ,ω y ,ω z ) The frequency domain point spreading function (PSF) after the Fourier transform. f (ω x ,ω y ,ω z ) From this, the inverse filter 1 / PSF in the frequency domain f (ω x ,ω y ,ω z ) generates.

[0092] The generated frequency domain inverse filter 1 / PSF f (ω x ,ω y ,ω z ) Voxel data of a frequency domain image I f (ω x ,ω y, ω z Applies to IC f (ω x ,ω y ,ω z Calculate ).

[0093] I C f (ω x ,ω y ,ω z )=If (ω x ,ω y ,ω z )×[1 / PSF f (ω x ,ω y ,ω z )] Calculated IC f (ω x ,ω y ,ω z The image is then subjected to an inverse Fourier transform to obtain the absolute value of each pixel, thereby obtaining the reconstructed image IC(x,y,z).

[0094] Thus, it is also possible to configure the system to acquire image data as voxel data and perform image restoration processing in a batch.

[0095] [Image restoration process using the Wiener filter] The convolution of the point spreading function essentially acts as a low-pass filter, so its inverse filter becomes a high-pass filter. Therefore, applying a simple inverse filter to an image of the object being measured increases noise in the high-frequency range, negatively impacting image processing in the focus variation method. In other words, the focus variation method cannot distinguish between noise and reflections from the object being measured, resulting in measurement errors.

[0096] Therefore, in this example, we set the Wiener filter as the reconstruction filter. The Wiener filter minimizes the squared error between the original image and the reconstructed image.

[0097] The following describes image restoration using the Wiener filter. This explanation uses the case of processing voxel data as an example. Furthermore, it is assumed that the point spread function PSF(x,y,z) has been obtained beforehand.

[0098] First, image data I(x,y,z) is obtained from an image of the object Ob to be measured. As described above, the image data is obtained as voxel data. The number of pixels at this time is (N x ,N y ), number of slices Nz Let's assume that.

[0099] Next, we determine whether the size of the image data I(x,y,z) matches the size of the point spread function PSF(x,y,z). If the sizes are different, we zero-padding the point spread function PSF(x,y,z) to make the sizes match.

[0100] Next, the image data I(x,y,z) is subjected to a three-dimensional Fourier transform and mapped to the frequency domain.

[0101] I(x,y,z)→I f (ω x ,ω y ,ω z ) Next, the point spreading function PSF(x,y,z) is subjected to a three-dimensional Fourier transform and mapped to the frequency domain.

[0102] PSF(x,y,z) → PSF f (ω x ,ω y ,ω z ) Next, the Wiener filter is calculated using the following formula.

[0103] PSF f *(ω x ,ω y ,ω z ) / {PSF f (ω x ,ω y ,ω z )PSF f *(ω x ,ω y ,ω z )+SN} Here, "PSF f *(ω x ,ω y ,ω z )」 is PSF f (ω x ,ω y ,ω zThe complex conjugate of ) and "SN" is a real number obtained from the signal-to-noise ratio of the image. In practice, a constant is set for "SN". However, since it is strongly dependent on the camera gain, it is preferable to adjust the optimal value experimentally. Here, "experimental" refers to a concept that includes both actual experiments and computer simulations.

[0104] Next, the calculated Wiener filter is I f (ω x ,ω y ,ω z Applies to IC f (ω x ,ω y ,ω z Calculate ).

[0105] I C f (ω x ,ω y ,ω z )=I f (ω x ,ω y ,ω z )[PSF f *(ω x ,ω y ,ω z ) / {PSF f (ω x ,ω y ,ω z )PSF f *(ω x ,ω y ,ω z )+SN}] The calculated IC f (ω x ,ω y ,ω z ) is inverse Fourier transformed, and the complex image IC c We obtain (x, y, z). Then, complex image IC c The absolute value of each pixel is obtained from (x, y, z) to form the final reconstructed image IC. r Get (x, y, z).

[0106] Thus, by using a Wiener filter as the restoration filter, noise, especially noise present in the high-frequency range, can be reduced, allowing for the restoration of higher-quality images. This enables more accurate measurements.

[0107] Figure 7 shows an example of image restoration processing using a Wiener filter. Note that this figure shows an example of an image sliced ​​in the xz plane to make it easier to understand the effect of blurring due to the focal point. The object being measured is a frustum.

[0108] As shown in the figure, it can be confirmed that blurring in areas with high reflectivity is reduced by performing image restoration processing using a Wiener filter.

[0109] Figure 8 is a magnified view of a portion of Figure 7 (the area of ​​frame f). Figure 9 is a magnified view of a portion of the image after image restoration processing using an inverse filter.

[0110] As shown in Figure 9, when a simple inverse filter is used, it can be seen that high-frequency noise increases in dark areas. When such noise is present, the focus variation method cannot distinguish between the noise and the reflection of the object being measured, especially reflections at low reflectivity points, and this appears as a measurement error.

[0111] On the other hand, as shown in Figure 8, it can be confirmed that using a Wiener filter eliminates the noise problem. Therefore, highly accurate measurements become possible even in areas with low reflectivity (in this example, an inclined surface).

[0112] [Imaging conditions] When imaging an object to be measured, the image should be captured in such a way that the brightness does not saturate in any of the images. That is, the maximum and minimum brightness values ​​of the object to be measured should fall within the dynamic range that the imaging device can capture. Therefore, it is preferable to use a camera with a wide dynamic range (a so-called wide dynamic range camera) in the imaging device.

[0113] [Imaging device] In the above embodiment, the optical head is moved relative to the object to be measured to capture multiple images with different focal positions. However, the optical head may be fixed, and the object to be measured may be moved to capture images. Alternatively, both may be moved to capture images. In other words, the movement to change the focal position may be relative.

[0114] [Image processing device] In the above embodiment, a case was described in which the image processing device is configured as a separate computer from the imaging device, but the functions of the image processing device can also be integrated into the imaging device. [Explanation of Symbols]

[0115] 1...Three-dimensional shape measuring device, 10...Imaging device, 11...Stage, 11a...Mounting surface, 12...Illumination light source, 13...Illumination optical system, 14...Imaging optical system, 15...Camera, 16...Focus drive unit, 17...Operation unit, 18...Control unit, 20...Illumination lens, 21...Beam splitter, 22...Aperture diaphragm, 23...Objective lens, 24...Imaging lens, 100...Image processing device, 100A...Image acquisition unit, 100B...Preprocessing unit, 100C...Three-dimensional shape data generation unit, 101...Processor, 102...Main memory, 103...Auxiliary memory, 104...Input device, 105...Output device, 106...Input / output interface, 200...Pinhole illumination, 201...Light source, 202...Concentrating lens, 203...Pinhole plate, 204...Pinhole, Ob...Measurement target

Claims

1. An image processing device that processes a plurality of images of a measurement object captured by changing a focal position, an image acquisition unit that acquires a plurality of images of the measurement object; an image processing unit that processes the plurality of images based on a point spread function for each focal position of an optical system of an imaging device that captured the plurality of images, and corrects blur in the images; a three-dimensional shape data generation unit that calculates an evaluation value of a focus degree for each pixel of the plurality of images after blur correction, and generates three-dimensional shape data of the measurement object based on the calculated evaluation value of the focus degree for each pixel; An image processing device comprising:

2. the image processing unit applies a frequency domain restoration filter based on the point spread function to the frequency domain image obtained by Fourier transforming the image, thereby correcting the frequency domain image, and performs an inverse Fourier transform on the corrected frequency domain image to correct blurring of the image. The image processing device according to claim 1 .

3. calculating an inverse filter in the frequency domain from the point spread function in the frequency domain obtained by Fourier transforming the point spread function, and setting the calculated inverse filter in the frequency domain as the restoration filter in the frequency domain; The image processing device according to claim 2 .

4. a frequency domain Wiener filter calculated from the frequency domain point spread function obtained by Fourier transforming the point spread function, and the frequency domain Wiener filter is set as the frequency domain restoration filter; The image processing device according to claim 2 .

5. The Wiener filter is calculated by the following formula: PSF f *(oh x ,oh y ,oh z ) / {PSF f (oh x ,oh y ,oh z )PSF f *(oh x ,oh y ,oh z )+SN} however, PSF(x,y,z) is the voxel data of the point spread function. PSF f (ω x ,ω y ,ω z ) is the Fourier transform of PSF(x,y,z), PSF f *(ω x ,ω y ,ω z ) is a PSF f (ω x ,ω y ,ω z ) complex conjugate, SN is a constant, The image processing device according to claim 4 .

6. The constant SN is experimentally set. The image processing device according to claim 5 .

7. the image processing unit processes voxel data of the measurement object based on the voxel data of the point spread function to correct blurring of the plurality of images. The image processing device according to claim 1 .

8. When a size of the voxel data of the point spread function is smaller than a size of the voxel data of the measurement object, the image processing unit adjusts the voxel data of the point spread function to the same size by padding with zeros. The image processing device according to claim 7 .

9. The point spread function is acquired by setting a bright spot on the optical axis of the optical system and capturing an image by moving the imaging device relative to the bright spot. The image processing device according to claim 1 .

10. The diameter of the bright spot is set to be equal to or less than the optical resolution of the imaging device via the optical system. The image processing device according to claim 9 .

11. The bright spot is composed of pinhole illumination or fluorescent beads. The image processing device according to claim 9 or 10.

12. the image processing unit acquires information on temperatures at the time when the measurement object is imaged, and corrects blurring of the plurality of images based on the point spread function corresponding to the acquired temperatures. The image processing device according to claim 1 .

13. The plurality of images of the measurement object are captured so that the maximum and minimum values ​​of luminance of the measurement object fall within a dynamic range that can be captured by the imaging device. The image processing device according to claim 1 .

14. an imaging device that moves relatively to a measurement object set on an optical axis and captures a plurality of images with different focal positions; an image processing device according to claim 1 , wherein the image processing device processes a plurality of images of the measurement object captured by the imaging device; A three-dimensional shape measuring device equipped with the device.

15. An image processing method for processing a plurality of images of a measurement object captured by changing a focus position, comprising: acquiring a plurality of images of the measurement object; correcting blur in the plurality of images based on a point spread function for each focal position of an optical system of an imaging device that captured the plurality of images; calculating an evaluation value of a focus degree for each pixel of the plurality of images after blur correction, and generating three-dimensional shape data of the measurement object based on the calculated evaluation value of the focus degree for each pixel; Image processing methods.

Images