Digital hologram reconstruction system and method

Abstract

A method for reconstructing a digital hologram of a surface having at least one three-dimensional feature thereon, comprising acquiring a digital hologram of the surface, reconstructing a wavefront based on the digital hologram, generating a phase map of at least a portion of the surface based on the wavefront, the phase map having an inherent phase ambiguity, acquiring at least one additional image of the surface, obtaining height data for the three-dimensional feature from the at least one additional image of the surface to a first precision, resolving the phase ambiguity based on the height data, and deriving a height of the at least one three-dimensional feature to a second precision finer than the first precision based on the phase map after the inherent phase ambiguity has been resolved.

Description

[Technical Field]

[0001] The present invention relates generally to digital holography, and more particularly to the reconstruction of digital holograms. [Background technology]

[0002] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 050806, filed July 12, 2020, the entire contents of which are incorporated herein by reference.

[0003] Various digital hologram reconstruction systems and methods are known in the art. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Publication No. 2010 / 002950 [Non-patent literature]

[0005] [Non-Patent Document 1] 'Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software', DC Ghiglia and MD Pritt, Wiley, New York (1998) Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention seeks to provide a novel system and method for resolving phase ambiguity in digital hologram reconstruction. [Means for solving the problem]

[0007] According to an embodiment of the present invention, there is provided a method for reconstructing a digital hologram of a surface having at least one three-dimensional feature thereon, comprising acquiring a digital hologram of the surface, reconstructing a wavefront based on the digital hologram, generating a phase map of at least a portion of the surface based on the wavefront, the phase map having an inherent phase ambiguity, acquiring at least one additional image of the surface, obtaining height data for the three-dimensional feature from the at least one additional image of the surface to a first precision, resolving the phase ambiguity based on the height data, and deriving a height of the at least one three-dimensional feature to a second precision finer than the first precision based on the phase map after the inherent phase ambiguity has been resolved.

[0008] According to one embodiment of the invention, during acquisition of at least one additional image of the surface, the wavefront is digitally propagated through a series of depths within the surface, and a series of digital wavefronts corresponding to the series of depths are acquired.

[0009] The height data is acquired based on the series of digital wavefronts using a DFF algorithm.

[0010] According to another embodiment of the present invention, acquiring at least one additional image of the surface utilizes an AI network to generate a series of incoherent light images corresponding to a series of digital wavefronts of the surface, and acquiring the series of digital wavefronts includes digitally propagating the wavefront through a series of depths within the surface and acquiring a series of digital wavefronts corresponding to the series of depths.

[0011] The height data is acquired using a DFF algorithm based on the series of non-coherent light images.

[0012] According to yet another embodiment of the present invention, during acquisition of the at least one additional image of the surface, at least one non-coherent illumination image of the surface is acquired.

[0013] The height data is acquired automatically using an AI network based on segmentation and classification of the at least one three-dimensional feature.

[0014] The first accuracy is set to be within the range of 1 to 5 μm.

[0015] In addition, the second precision is set within a range of 1 to 100 nm or 50 to 1000 nm.

[0016] When obtaining the digital hologram, a digital microscopic hologram is obtained.

[0017] Furthermore, in accordance with another embodiment of the present invention, there is provided a system for reconstructing a digital hologram of a surface having at least one three-dimensional feature thereon, the system comprising: a digital holographic image acquisition subsystem operable to acquire a digital hologram of the surface; a wavefront reconstructor operable to reconstruct a wavefront based on the digital hologram; a phase map generator operable to receive the wavefront and to generate, based on the wavefront, a phase map of at least a portion of the surface, the phase map having an inherent phase ambiguity; and a phase map generator operable to acquire at least one additional image of the surface. an additional image acquisition or image processing subsystem operable to obtain height data for the three-dimensional feature from the at least one additional image of the surface to a first precision; an image analyzer operable to obtain height data for the three-dimensional feature from the at least one additional image of the surface to a first precision; a phase ambiguity resolver operable to resolve phase ambiguities inherent in the phase map based on the height data; and a height calculator operable to derive a height of the at least one three-dimensional feature to a second precision finer than the first precision based on the phase map after the inherent phase ambiguities have been resolved.

[0018] According to an embodiment of the present invention, the image processing subsystem can be operated to digitally propagate the wavefront through a series of depths within the surface and to acquire a series of digital wavefronts corresponding to the series of depths.

[0019] The image analyzer can be operated to utilize a DFF algorithm to obtain the height data based on the series of digital wavefronts.

[0020] According to another embodiment of the present invention, the image processing subsystem includes an AI network operable to generate a series of incoherent light images corresponding to a series of digital wavefronts of the surface, wherein the series of digital wavefronts are acquired by the image processing subsystem digitally propagating the wavefront through a series of depths within the surface and acquiring a series of digital wavefronts corresponding to the series of depths.

[0021] The image analyzer can be operated to derive the height data based on the series of non-coherent light images using a DFF algorithm.

[0022] According to yet another embodiment of the present invention, the additional image acquisition subsystem includes an incoherent illuminator operable to illuminate the surface with incoherent light and a camera operable to acquire the at least one additional image of the surface.

[0023] The image analyzer may include an AI network operable to automatically acquire height data based on the segmentation and classification of the at least one three-dimensional feature.

[0024] The first accuracy is set to be within the range of 0.5 to 5 μm.

[0025] In addition, the second precision is set within a range of 1 to 100 nm or 50 to 1000 nm.

[0026] The digital holographic image acquisition subsystem is referred to as a digital holographic microscopic image acquisition subsystem.

[0027] The present invention will be more fully understood from the following detailed description taken in conjunction with the following drawings. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a schematic high-level flowchart illustrating steps of digital hologram reconstruction according to an embodiment of the present invention. [Figure 2] 10 is a schematic high-level flowchart illustrating steps of digital hologram reconstruction according to another embodiment of the present invention. [Figure 3] 10 is a schematic high-level flowchart illustrating steps for digital hologram reconstruction according to yet another embodiment of the present invention. [Figure 4] 4 is a schematic flowchart illustrating the training of an AI network used for digital hologram reconstruction according to any of the embodiments of FIGS. 2 and 3. [Figure 5] 10 is a schematic high-level flowchart illustrating the steps of digital hologram reconstruction according to a further embodiment of the present invention; [Figure 6] 10 is a schematic high-level flowchart illustrating steps of digital hologram reconstruction according to a still further embodiment of the present invention; [Figure 7] 6 is a schematic flow chart illustrating the training of an AI network used for digital hologram reconstruction according to the embodiment of FIG. 5. [Figure 8] 7 is a schematic flow chart illustrating the training of an AI network used for digital hologram reconstruction according to the embodiment of FIG. 6. [Figure 9] FIG. 9 is a diagram showing an outline of an image obtained by training an AI network according to FIGS. 7 and 8. [Figure 10] 9 is a schematic, partially pictorial, partially block diagram of a digital hologram acquisition and reconstruction system constructed and operable in accordance with an embodiment of the present invention to perform the steps included in any of FIGS. 1-8. FIG. [Figure 11] 11 is a schematic flow chart illustrating a possible implementation of the system of FIG. 10 in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0029] Reference is first made to FIG. 1, a simplified high-level flowchart illustrating the steps of digital hologram reconstruction in accordance with an embodiment of the present invention.

[0030] As shown in FIG. 1 , the digital hologram reconstruction process 100 can begin with a first image acquisition step 102 in which a digital holography (DH) image acquisition system records a digital hologram. A digital hologram, also known as an interferogram, is a digital record of the interference pattern resulting from the interaction of coherent light reflected from a three-dimensional imaged object with a reference light. The interferogram is recorded by an electronic sensor and has a number of pixels. As an example, the DH system recording the digital hologram can be a digital holographic microscopy system (DHM), as shown in more detail with reference to FIG. 10 and described below. The interferogram can be acquired using one, two, or more wavelengths of coherent light. Using more than one wavelength can reduce, if not eliminate, phase ambiguity in the phase information recorded in the interferogram.

[0031] A wavefront is then reconstructed based on the interferogram, as seen in the second wavefront reconstruction step 104. The wavefront is not a physical entity, but rather a numerical entity commonly referred to as a complex amplitude, in which phase and amplitude information for each pixel in the interferogram is encoded. The wavefront is typically reconstructed computationally, such as by computational functions incorporated in an image processing module, as shown in more detail with reference to FIG. 10 and described below. Possible reconstruction algorithms in digital holographic microscopy include Fresnel transform algorithms, angular spectrum algorithms, and convolution algorithms, all of which are based on the fast Fourier transform (FFT).

[0032] As seen in a third wavefront processing step 106, the wavefront acquired in step 104 can be appropriately processed to generate a phase map of the imaged object. The phase map corresponds to the phase information encoded in the pixel-by-pixel complex amplitudes, from which the imaged object can be numerically reconstructed. The pixel-by-pixel phase information can be obtained from the wavefront by calculating the angle of the phasor (phase vector) encoded in the pixel-by-pixel complex amplitudes.

[0033] The phase map generated in step 106 typically contains phase discontinuity artifacts that do not represent the true shape of the three-dimensional imaged object. These discontinuities at multiples of 2π arise due to wrapping of the calculated phase within the interval (-π,π). In other words, the phase map is ambiguous modulo (2π). Therefore, the phase map can be referred to as an ambiguous phase map. The number of discontinuities can be significantly reduced by using two or more wavelengths for holographic recording; in this case, the 2π ambiguity applies only to the phase differences between the multiple wavelengths. Whether one or more wavelengths are used for holographic recording, this ambiguity in the phase map introduces a corresponding ambiguity in the imaged object dimensions to be derived based on the phase map; this is because a phase shift in the phase map can correspond to a shift of any integer multiple of the wavelength. Therefore, to accurately numerically reconstruct the structure of the imaged object based on the phase map, such ambiguities must be resolved and the true phase shifts must be ascertained without ambiguity.

[0034] A particular feature of the present invention is that phase ambiguities in a phase map are resolved in accordance with embodiments of the present invention, thereby enabling the generation of an unambiguous phase map, and thus, highly accurate and precise numerical reconstruction of the height dimension of an imaged object. The unambiguous phase map generated in accordance with embodiments of the present invention may be a full phase map used for numerical reconstruction of the entire imaged object, or a partial phase map used for numerical reconstruction of one or more features of the imaged object, thereby enabling the derivation of the height of the one or more features. This can be achieved in accordance with the present invention by resolving the phase ambiguity using height information derived from at least one additional image of the imaged object. The additional image may be either an actual, physically acquired image or a virtual, information-processing-derived image representation. Height information derived from at least one additional image of the imaged object typically has lower accuracy and poorer axial resolution along the imaging optical axis (e.g., optical axis A in FIG. 10 ) than the object-related height information that is ultimately to be derived based on the DH image. However, the height information derived from the at least one additional image is sufficiently accurate that it can be used to resolve phase ambiguities in the phase map by helping to identify, for each pixel of the phase map, the wavelength multiple, or synthetic wavelength multiple in multi-wavelength holography, that corresponds to that phase shift.

[0035] The present invention may be particularly useful in DH inspection of various types of electronic circuit boards, such as silicon wafers, PCBs, and FPDs. Such electronic circuit boards typically have three-dimensional (3D) features formed thereon, sometimes in several layers, that must be inspected during or after the substrate's manufacture to ensure that the features are properly formed. According to embodiments of the present invention, the 3D features can be measured quickly and very accurately without ambiguity using a DH inspection system.

[0036] According to one embodiment of the present invention, in addition to the wavefront reconstructed in step 104, the at least one additional image can be an image stack acquired by digitally propagating the wavefront acquired in step 104 through a series of depths within the imaged object. That is, by digitally propagating the single wavefront acquired in step 104 (or either of the two reconstructed wavefronts in the case of dual wavelengths), image information for different depths within the imaged object can be acquired. The depth of the imaged object can be defined as the dimension of the object along the z-direction, where the z-direction is perpendicular to the x- and y-axes that define the object or surface plane. The z-direction is illustrated in FIG. 10 . The image with the best focus in the z-direction image stack can then be identified for each pixel. The object height near each pixel can then be derived using the pixel-specific best focus distance. The derived object height can then be used to resolve ambiguities in the phase map. This embodiment of the present invention is illustrated in FIG. 1 .

[0037] According to another embodiment of the present invention, in addition to the wavefront reconstructed in step 104, the at least one additional image can be a computationally derived stack of incoherent light image equivalents, corresponding to an image stack obtained by digitally propagating the wavefront acquired in step 104 through a series of depths within the imaged object. By digitally propagating the single wavefront acquired in step 104, image information can be obtained for different depths within the imaged object. The depth of the imaged object can be defined as the dimension along the z-direction of the object, where the z-direction is perpendicular to the x- and y-axes defining the object or surface plane. The z-direction is illustrated in FIG. 10 . An artificial intelligence (AI) network can then be trained to perform computations equivalent to optical cross-modality image transformation. The AI ​​network can be operated on the coherent light image stack acquired by digitally propagating the wavefront through various object depths to generate a stack of corresponding images similar to those acquired with incoherent light (also known as “white” light). The image with the best focus in the z-direction AI-generated incoherent light image stack can then be identified for each pixel. The best focus distance for each pixel can then be used to derive the object height near each pixel. The derived object height can then be used to resolve ambiguities in the phase map. This embodiment of the present invention is shown in Figure 2.

[0038] According to yet another embodiment of the present invention, the embodiment of the present invention shown in Figure 2 can be augmented with an actual physical recording of an incoherent light image of the imaged object in addition to a stack of AI-generated equivalent incoherent light images. The incoherent light images can be used to improve resolution of ambiguities in the phase map by assisting in spatial filtering and smoothing of the derived height values ​​for pixel neighborhoods. This embodiment of the present invention is shown in Figure 3.

[0039] According to another embodiment of the present invention, in addition to the reconstructed wavefront in step 104, the at least one additional image can be an incoherent light image of the imaged object. For example, the imaged object can be a surface having at least one three-dimensional feature thereon. Segmenting the incoherent light image of the imaged object can identify the presence and boundaries of discontinuities in the surface. These discontinuities can correspond to three-dimensional features intentionally created on the surface as desired, or they can correspond to surface discontinuities caused by defects in the surface. If the surface is generally flat, features on the surface that deviate from the generally flat surface topography by more than a predetermined threshold can be considered surface discontinuities. These discontinuities can be referred to as bumps, but it should be understood that they can be protruding or recessed from the surface and can have any type of shape, such as curved, rounded, or straight edges.

[0040] According to this embodiment of the invention, the segmented additional incoherent light images can be appropriately processed by an AI network to classify the bump heights. The derived bump heights can then be used to resolve ambiguities in the phase map. This embodiment of the invention is illustrated in Figures 5 and 6. This embodiment of the invention is particularly useful for imaging surfaces on which multiple discontinuities may exist.

[0041] As will be appreciated, as used herein, the term "height" can refer to both positive and negative heights, in the sense that a given feature appearing within a given pixel may be a protruding feature having a positive height relative to neighboring features, or a recessed feature having a negative height relative to neighboring features. Note that in practical applications to electronic circuit board inspection, absolute heights derived in accordance with the teachings of the present invention will ultimately be given relative to some other feature on the board. That is, in the case of a wafer bump, the wafer bump height will be measured relative to other neighboring bumps or relative to a reference feature located near the side of the bump, such as a metal layer that may be referred to as under-bump metallization (UBM).

[0042] The approach of the present invention avoids the use of conventional phase unwrapping techniques for phase ambiguity resolution, which are typically very slow, computationally intensive, and highly sensitive to system conditions, and therefore often produce poor results. An example of such a conventional phase unwrapping technique can be found in "Phase ambiguity resolution: a generalized method for resolving phase ambiguities," by Chris Bauer, "Phase ambiguity resolution: a generalized method for resolving phase ambiguities," IEEE Transactions on Signal Processing, Vol. 1, No. 1, pp. 111-114, 2003, the contents of which are incorporated herein by reference.

[0043] Furthermore, such conventional algorithmic phase unwrapping techniques rely on surface continuity to perform phase unwrapping. When the imaged object or surface is severely discontinuous and the height difference between neighboring imaged points exceeds the range of unambiguous phase shifts, i.e., 2π, such conventional techniques fail to accurately resolve the phase ambiguity. In contrast, embodiments of the present invention can resolve ambiguities in the phase map even in the presence of severe surface discontinuities with phase jumps greater than 2π; this is because the present invention does not rely on surface continuity to resolve the phase ambiguity.

[0044] According to various embodiments of the present invention, phase ambiguities can be quickly resolved using lower-precision or coarser additional data, provided that the additional height data itself has a coverage extending throughout the entire object depth and is sufficiently accurate to be useful in resolving the phase ambiguity. The additional data can have a coverage extending several times the unambiguous DH range, thereby helping to extend the total effective measurement range of the system without loss of accuracy in the derived height data. That is, the height information ultimately provided by the present invention maintains the high precision and resolution of the DH image data without loss of accuracy, even though the height information is obtained with the aid of possibly less accurate height information.

[0045] It should be noted that the various embodiments of the invention described herein have a finite range of validity, outside of which the quality of the additional height data deteriorates. Furthermore, the embodiments described with reference to Figures 5 and 6 may also exhibit inherent ambiguities, such as ambiguity between being above or below the sharp focus plane.

[0046] Theoretically, the unambiguous phase shift range of a multi-wavelength DH can be extended infinitely by making the wavelength difference between the multiple wavelengths approach zero, but this comes at the expense of a loss in the accuracy of the DH data.This invention does not cause such a loss.

[0047] Furthermore, in certain embodiments of the present invention, the DH imaging has a long range due to the DH image propagation through the depth of the imaged object.

[0048] Additionally, certain embodiments of the present invention use a combination of phase information derivable from DH images and height information derivable from AI-processed images, allowing the present invention to be partially, but not entirely, within the AI ​​domain and results not based solely on AI-generated images, thereby reducing the risk of false outputs that would occur if height information were derived solely from AI-generated images.

[0049] Returning to FIG. 1 , after wavefront reconstruction in step 104, the wavefront can be digitally propagated in step 108 to generate multiple 2D images corresponding to multiple focal depths along the z-direction within the imaged object. Each of the multiple images can have multiple pixels. The multiple images form a stack of coherent light images at a range of depths within the imaged object. Note that the term “image” is used in a more general sense herein because these calculated wavefronts are actually complex arrays, as opposed to traditional digital image representations, which consist of encoded real numbers. For subsequent processing steps, such as DFF algorithms, the complex wavefront can be converted to a real image, for example, by taking coefficient values ​​at each pixel. This digital propagation of the single wavefront reconstructed in step 104 can obtain image information for different depths within the imaged object. The depth of the imaged object can be defined as the dimension of the object along the z-direction, where the z-direction is perpendicular to the x- and y-axes defining the object surface.

[0050] For example, if the object being imaged is a surface with three-dimensional features thereon, and the object has a total z-dimension of 10-100 μm, a stack containing a series of approximately 20 images can be reconstructed at z-dimension intervals of 0.5-5 μm. More generally, the image stack output in step 108 can contain any number of images separated by any suitable depth range interval, as long as the height measurements derivable from the image stack are more accurate than the phase unambiguity range and can therefore be used to resolve phase ambiguities in the phase map output in step 106. As seen in image processing step 110, the image stack along the z-dimension can be appropriately processed to find and select the best-focused image from the image stack on a pixel-by-pixel basis. For example, the image stack output in step 108 can be processed using a depth-of-focus (DFF) algorithm, also known as passive autofocus, to identify the best-focused image on a pixel-by-pixel basis. Autofocus algorithms used in DHM vary in conjunction with acceleration techniques and sharpness metrics used to speed up convergence to an optimal image frame around each pixel. These algorithms may differ somewhat from DFF algorithms used in non-coherent imaging, but are tailored to the behavior of coherent images at particularly high spatial frequencies.

[0051] Based on the pixel-by-pixel best focus depth, the height of the imaged three-dimensional feature appearing within each pixel can be found. For example, if the imaged object is a surface having at least one three-dimensional feature thereon, the height of the three-dimensional feature appearing within each pixel can be found. The height of the three-dimensional feature can be defined as the dimension of the three-dimensional feature extending in a direction perpendicular to a plane defined by the surface on which the three-dimensional feature is formed. The height of the object, e.g., the height of the three-dimensional feature on the imaged surface, can be found with a first accuracy according to this technique. As an example, the accuracy with which pixel-by-pixel heights are found using a DFF algorithm can be 1 to 5 μm. This accuracy is relatively low and may be unacceptably poor for practical height measurements. However, as described in more detail below, this accuracy is sufficiently high for assisting in resolving phase ambiguities in phase maps.

[0052] As can be seen in the further step 111, the output of the DFF processed 3D data in step 110 can be followed by spatial filtering and smoothing. As will be appreciated by those skilled in the art, the DFF object height derived in step 110 is itself often redundant and noisy, and spatial filtering and smoothing are therefore beneficial. For example, a PSF adaptive filter may be used for this step. As can be appreciated, the smoothing and filtering step 111, while providing improved results, is not essential and may be omitted if the spatial resolution quality achieved in step 110 is considered to be good enough.

[0053] As can be seen in phase ambiguity resolution step 112, the low precision processed DFF data has been smoothed and filtered by step 111 and can be used to resolve phase ambiguities in the phase map generated in step 106.

[0054] Even if the precision of the DFF or other processed data is lower, pixel-by-pixel discrimination to resolve phase ambiguities is performed based on the extended range, and the extended range data is used to unwrap the phase data by identifying how many times the ambiguity range wavelength the measured phase shift corresponds to. In other words, it is determined which integer multiple of 2π is correct. As a result, a high-precision height image is obtained in which the ambiguity is no longer inherent. While the additional images used to help resolve the phase ambiguity are typically of lower precision (first precision), this final high-precision map is typically of higher precision (second precision).

[0055] For example, a given phase shift may correspond to a phase difference of π / 2, π / 2±2π, π / 2±4π, etc., all of which differ by an integer multiple of 2π. Such a phase shift can be referred to as an ambiguous phase shift, because it also ambiguizes the height of the object at the pixel where the phase shift is measured, i.e., the height derived based on the phase shift.

[0056] This can be illustrated in the context of a dual-wavelength DH laser emitting at wavelengths of 0.520 μm and 0.532 μm. The ambiguity range, also known as the synthetic wavelength, based on these wavelengths is 23 μm, given by (0.52 × 0.532) / (0.532 − 0.52), resulting in a 2π phase shift between the two neighboring wavelengths over the optical path length. If a cross-incidence interferometer is used, the ambiguous π / 2 wavefront phase difference at a given pixel can be appropriately converted into a corresponding ambiguous sequence of object height candidates, roughly given by 2.9, 2.9 ± 11.5, 2.9 ± 23 μm, etc., all differing by integer multiples of 11.5 μm, corresponding to one synthetic wavelength path difference of the incident light. Although each candidate height may be accurate down to about 1-100 nm for single-wavelength interferometry and about 50-1000 nm for dual-wavelength interferometry, depending on the system noise level, the system in step 106 cannot confirm which of the various likely candidate heights is the correct height. However, in step 112, the system can use extended-range coarse-precision data for the object height at a given pixel to resolve the ambiguity; this is because the extended-range coarse-precision height data can be used to confirm which of several ambiguous-range candidate heights corresponds to that coarse-precision height.

[0057] To further illustrate this last point, suppose that at a given pixel with a π / 2 phase shift, the DFF algorithm in step 110 reports a height of 28.7 μm with, say, a ±5 μm uncertainty, after smoothing in step 111. Comparison with the series of likely candidate heights output by step 106 yields 2.9 + 23 = 25.9 μm as the closest value, while all other likely candidate heights fall outside the DFF uncertainty range. Because the DH data is more precise, the system reports 25.9 μm as the best-accurate height estimate for the given pixel.

[0058] Reference is now made to FIG. 2, a simplified high-level flowchart illustrating the steps of digital hologram reconstruction in accordance with an embodiment of the present invention.

[0059] As shown in FIG. 2 , the digital hologram reconstruction process 200 can begin with a first image acquisition step 202 in which a digital holographic image acquisition (DH) system records a digital hologram. A digital hologram, also known as an interferogram, is a digital recording of an interference pattern resulting from the interference of coherent light reflected from a three-dimensional imaged object with a reference light. The interferogram is recorded by an electronic sensor and has a number of pixels. As an example, the DH system recording the digital hologram can be a digital holographic microscopy system (DHM), as shown in more detail with reference to FIG. 10 and described below. The interferogram can be acquired using one, two, or more wavelengths of coherent light. Using more than one wavelength can reduce, if not eliminate, phase ambiguity in the phase information inherent in the interferogram.

[0060] A wavefront is then reconstructed based on the interferogram, as seen in the second wavefront reconstruction step 204. The wavefront is not a physical entity, but rather a numerical entity commonly referred to as a complex amplitude, in which phase and amplitude information for each pixel in the interferogram is encoded. The wavefront is typically reconstructed computationally, such as by computational functions incorporated in an image processing module, as shown in more detail with reference to FIG. 10 and described below. Possible reconstruction algorithms in digital holographic microscopy include Fresnel transform algorithms, angular spectrum algorithms, and convolution algorithms, all of which are based on the fast Fourier transform (FFT).

[0061] As seen in a third wavefront processing step 206, the wavefront acquired in step 204 can be appropriately processed to generate a phase map of the imaged object. The phase map corresponds to the phase information encoded in the pixel-wise complex amplitudes, from which the imaged object can be numerically reconstructed. The pixel-wise phase information can be obtained from the wavefront by calculating the phasor angle encoded in the pixel-wise complex amplitudes.

[0062] The phase map generated by step 206 typically contains phase discontinuity artifacts that do not represent the true shape of the three-dimensional imaged object. These discontinuities at multiples of 2π arise due to the wrapping of the calculated phase within the interval range (-π,π). In other words, the phase map is ambiguous modulo (2π). Therefore, the phase map can be referred to as an ambiguous phase map.

[0063] Using two or more wavelengths for holographic recording can greatly reduce the number of discontinuities because the 2π ambiguity then applies only to the phase difference between the wavelengths. This ambiguity in the phase map leads to a corresponding ambiguity in the imaged object dimensions derived from the phase map because a phase shift in the phase map can correspond to a shift of any integer multiple of the wavelength. Therefore, to accurately numerically reconstruct the structure of the imaged object based on the phase map, the ambiguity must be resolved and the true phase shift must be determined without ambiguity.

[0064] To resolve phase ambiguities, the wavefront may be digitally propagated in step 208 to generate multiple images corresponding to multiple focal depths along the z-direction within the imaged object. Each of the multiple images may have multiple pixels. The multiple images form a stack of coherent light images at a range of depths within the imaged object. That is, the single wavefront acquired in step 204 is digitally propagated to obtain image information for different depths within the imaged object. The depth of the imaged object may be defined as the dimension of the object along the z-direction, where the z-direction is perpendicular to the x- and y-axes that define the object or surface plane.

[0065] For example, if the object being imaged is a surface having three-dimensional features thereon, and the object has a total z-dimension of 10-100 μm, a stack containing a series of approximately 20 images can be reconstructed at intervals along the z-dimension of 0.5-5 μm. More generally, the image stack output in step 208 can include any number of images separated by any suitable depth range interval, provided that height measurements derivable from the image stack are more accurate than the phase unambiguity range and can therefore be used to resolve phase ambiguities in the phase map output in step 206.

[0066] As seen in AI network processing step 209, the image stack output in step 208 is then converted into a stack of equivalent non-coherent light 2D images at the same set of depths. This conversion can be performed by a trained AI network, e.g., a CNN. Further details regarding the training and construction of such a network are provided below with reference to FIG. 4.

[0067] As seen in image processing step 210, the incoherent light image stack can be further processed along the z-direction to find the best-focused image on a pixel-by-pixel basis, and the best-focused image can be selected from the image stack. As an example, the image stack output in step 209 can be appropriately processed with a DFF algorithm to identify the best-focused image for each pixel. Autofocus algorithms used in DHM vary in conjunction with acceleration techniques and sharpness metrics used to speed up convergence to the best image frame around each pixel. The equivalent incoherent image stack available in step 209 can be used to benefit the present invention by enabling the adoption of DFF algorithms that are more standard and used in incoherent imaging. Alternatively, other algorithmic techniques besides DFF that can find the best-focused image for each pixel can be used, such as depth of focus (DFD). Finding pixel-wise best focus in step 210 based on a non-coherent light image as described herein, rather than based on a coherent light image as described in step 110 of Figure 1, has been found to provide better results, believed to be due to the better and more consistent through-focus contrast variation behavior of the non-coherent light image compared to the DH coherent light image.

[0068] Based on the pixel-by-pixel best focus depth, the height of the imaged object appearing within each pixel can be found. For example, if the imaged object is a surface having at least one three-dimensional feature thereon, the height of the three-dimensional feature on the surface appearing within each pixel can be found. The height of the object, e.g., the height of the three-dimensional feature on the imaged surface, can be found with a first accuracy according to this technique. As an example, the accuracy with which the pixel-by-pixel height is found using a DFF algorithm can be 1-5 μm. This accuracy is relatively low and may be unacceptably poor for practical height measurements. However, this accuracy is sufficiently high to aid in resolving phase ambiguities in the phase map, as described in more detail below.

[0069] As seen in further step 211, the DFF-processed 3D data output in step 210 can be followed by spatial filtering and smoothing. As those skilled in the art will appreciate, spatial filtering and smoothing are beneficial because the DFF object heights derived in step 210 are often redundant and noisy. For example, a PSF-adaptive filter can be used for this step. The 2D white-light equivalent images acquired in step 209 can be used to assist step 211. These images can be used to identify and segment desired 3D features prior to smoothing. This allows for more efficient use of processing resources and faster processing. Another advantage of using these 2D white-light equivalent images is that the smoothing algorithm in step 211 prevents actual sharp feature discontinuities from being artificially blurred. While providing improved results, the smoothing and filtering step 211 is not essential and may be omitted if the spatial resolution quality achieved in step 210 is deemed sufficiently good.

[0070] As seen in phase ambiguity resolution step 212, the smoothed and filtered low precision processed DFF data from step 211 can then be used to resolve phase ambiguities in the phase map generated in step 206.

[0071] A pixel-by-pixel determination to resolve the phase ambiguity is performed based on the low-precision DFF or other processed data, which is then used to resolve the ambiguity in the phase data by identifying the ambiguity range wavelength corresponding to the measured phase shift. In other words, which integer multiple of 2π is correct is identified. As a result, a high-precision height image is obtained that no longer contains ambiguities. While the additional images used to assist in resolving the phase ambiguity are of lower first-precision precision, this final high-precision map is typically of higher second-precision precision.

[0072] For example, a given phase shift may correspond to a phase difference of π / 2, π / 2±2π, π / 2±4π, etc., all of which differ by an integer multiple of 2π. Such phase shifts can be referred to as ambiguous phase shifts because they also ambiguize the object height at the pixel where the phase shift is measured, i.e., the height derived based on the phase shift. However, in step 212, the ambiguity can be resolved using low-precision data regarding the object height at the given pixel; this is because the low-precision height data can be used to determine which of several ambiguity-range height candidates corresponds to the low-precision height. As a result, a high-precision height image can be obtained that no longer contains ambiguity. While the additional images used to help resolve the phase ambiguity are of a lower first precision, this final high-precision image is typically of a higher second precision.

[0073] According to one potential embodiment of the present invention, a process 300 can be implemented as shown in FIG. 3 ; it is generally similar in relevant aspects to process 200 of FIG. 2 , except that process 300 includes an additional image recording step 302, which is not performed in process 200. Image recording step 302 involves recording a non-coherent illumination image or video frame of the imaged item in addition to recording an interferogram of the imaged item. Image recording step 302 can be performed, for example, by illuminating the object to be imaged with light having a broad spectral range, commonly referred to herein as white light, although it may include wavelengths outside the visible range, and capturing an image of the illuminated object with a camera or video camera capable of recording non-coherent light images, such as those described below with reference to FIG. 10 . The image can be a bright-field (BF) image or a dark-field (DF) image. As can be seen, the AI ​​network generated image provided in step 210 is a virtual, information processing generated image, whereas the incoherent light image generated by image recording step 302 is an actual captured image.

[0074] 3, the incoherent light image acquired in step 302 can be used in step 211 to assist in spatial filtering and smoothing. Using the incoherent light image in the spatial filtering and smoothing step 211 may be advantageous compared to using only DH data in this step because the incoherent light image provides better contrast and less noise than the DH image data. The less accurate processed DFF data, smoothed and filtered in step 211, can be used to resolve phase ambiguities in the phase map, as seen in step 212.

[0075] Reference is now made to FIG. 4, a schematic flow chart illustrating the training of an AI network utilized for digital hologram reconstruction in accordance with the embodiment of FIGS.

[0076] As described above with reference to Figure 2, an equivalent non-coherent light image stack can be generated by an AI network based on a corresponding stack of coherent light images in step 210. Such a network must be trained prior to its use in process 200 or 300 to enable the network to perform image conversion between coherent and non-coherent light images. The training of such a network is now described with reference to Figure 4.

[0077] As shown in FIG. 4, the training process 400 can begin with a first training step 402 in which the DH system records an interferogram of a 3D object. The interferogram is recorded by an electronic sensor and has a number of pixels. By way of example, the DH system recording the digital hologram can be a digital holographic microscopy system (DHM), as shown in more detail with reference to FIG. 10 and described below. The interferogram can be acquired at one, two, or more wavelengths. Using more than one wavelength can reduce, if not eliminate, phase ambiguity in the phase information inherent in the interferogram.

[0078] A wavefront is then reconstructed based on the interferogram, as seen in the second training step 404. The wavefront is not a physical entity, but rather a numerical event in which the phase and amplitude information at each pixel in the interferogram is reconstructed. The wavefront is typically reconstructed computationally, such as by computational functions incorporated within an image processing module, as shown in more detail with reference to FIG. 10 and described below.

[0079] As seen in a third training step 406, the wavefront acquired in step 404 can be digitally propagated to generate multiple images corresponding to multiple focal depths along the z-direction within the imaged object. Each of the multiple images can have multiple pixels. The multiple images form a stack of coherent light images at a range of depths within the imaged object. The depth of the imaged object can be defined as the dimension of the object along the z-direction, where the z-direction is perpendicular to the x- and y-axes that define the object or surface plane.

[0080] As seen in a fourth training step 408, a non-coherent light imaging system records non-coherent illumination images at a field of view (FOV) that is the same as the FOV in the coherent image stack output in step 406. For example, the non-coherent light imaging system may be a white light microscopic imaging system, as shown with reference to Figure 10 and described in more detail below. The output of step 408 is a stack of non-coherent light images at FOVs of the imaged object, along the z-direction through the object.

[0081] As seen in the fifth training step 410, the AI ​​network can be operated to receive coherent light image stacks and non-coherent light image stacks of the imaged object with the same FOV output by steps 406 and 408, respectively, and is trained to perform inter-imaging modality conversion, thereby converting coherent light images into equivalent non-coherent light images.

[0082] As an example, the AI ​​network can be trained on approximately 10,000 examples, divided into a training set and a validation set. The examples in the validation set are not seen by the network during training. The training set can consist of approximately 80% of the examples, and the validation set can consist of approximately 20%. Network training can be stopped based on at least one of the following criteria: no further improvement in error (MSE) reduction is achieved with further training; overfitting occurs; and the maximum number of training epochs has been reached. A wide variety of networks can be suitable, including autoencoders, Unets, and residual networks. Suitable networks typically incorporate convolutional blocks, data reduction layers, and activation / normalization methods.

[0083] Reference is now made to FIG. 5, a schematic high-level flowchart illustrating the steps of digital hologram reconstruction according to a further embodiment of the present invention.

[0084] As shown in FIG. 5 , the digital hologram reconstruction process 500 can begin with a first image acquisition step 502 in which a digital holography (DH) image acquisition system records a digital hologram. A digital hologram, also known as an interferogram, is a digital recording of an interference pattern resulting from the interference of coherent light reflected by an imaged three-dimensional object with a reference light. The interferogram is recorded by an electronic sensor and has a number of pixels. As an example, the DH system recording the digital hologram can be a digital holographic microscopy system (DHM), as shown in more detail with reference to FIG. 10 and described below. The interferogram can be acquired using coherent light at one, two, or more wavelengths. Using more than one wavelength can reduce, if not eliminate, phase ambiguity in the phase information inherent in the interferogram.

[0085] A wavefront is then reconstructed based on the interferogram, as seen in a second wavefront reconstruction step 504. The wavefront is not a physical entity, but rather a numerical entity commonly referred to as a complex amplitude, in which phase and amplitude information for each pixel in the interferogram is encoded. The wavefront is typically reconstructed computationally, such as by computational functions incorporated in an image processing module, as shown in more detail with reference to FIG. 10 and described below. Possible reconstruction algorithms in digital holographic microscopy include Fresnel transform algorithms, angular spectrum algorithms, and convolution algorithms, all of which are based on the fast Fourier transform (FFT).

[0086] As seen in a third wavefront processing step 506, the wavefront acquired in step 504 can be appropriately processed to generate a phase map of the imaged object. The phase map corresponds to the phase information encoded in the pixel-wise complex amplitudes, and based on the phase information, the imaged object can be numerically reconstructed. The pixel-wise phase information can be obtained from the wavefront by calculating the phasor angle encoded in the pixel-wise complex amplitudes.

[0087] The phase map generated by step 506 typically contains phase discontinuity artifacts that do not represent the true shape of the three-dimensional imaged object. These discontinuities at multiples of 2π arise due to the wrapping of the calculated phase within the interval range (-π,π). In other words, the phase map is ambiguous modulo (2π). Therefore, the phase map can be referred to as an ambiguous phase map.

[0088] Using two or more wavelengths for holographic recording can greatly reduce the number of discontinuities because the 2π ambiguity then applies only to the phase difference between the wavelengths. This ambiguity in the phase map leads to a corresponding ambiguity in the imaged object dimensions derived from the phase map because a phase shift in the phase map can correspond to a shift of any integer multiple of the wavelength. Therefore, to accurately numerically reconstruct the structure of the imaged object based on the phase map, the ambiguity must be resolved and the true phase shift must be determined without ambiguity.

[0089] To resolve phase ambiguities in the phase map generated by step 506, process 500 can incorporate obtaining height data for the imaged object to a first accuracy from at least one additional image of the imaged object. As seen in step 508, the at least one additional image can be a video frame recorded by a non-coherent light image acquisition system. The additional image output by step 508 can also be a BF and / or DF image acquired under non-coherent illumination conditions.

[0090] As seen in step 510, the incoherent light image output in step 508 can be segmented. For example, the imaged object may be a surface having at least one three-dimensional feature thereon. In this case, segmenting the incoherent light image of the imaged object can identify the presence and boundaries of discontinuities in the surface. These discontinuities may correspond to three-dimensional features intentionally created on the surface, as desired, or they may correspond to surface discontinuities caused by defects in the surface. These discontinuities may be referred to as bumps, but it should be understood that they may be protruding or recessed from the surface and may have any type of shape, such as curved, rounded, or straight edges.

[0091] The incoherent light image may be segmented using any suitable image segmentation technique. The output of step 510 may be a list identifying the presence and location of bumps in the image. Such a list may be referred to as a segmented bump list. It should be understood that this list does not identify the heights of the bumps, only their presence and location.

[0092] As seen in step 511, the individual heights of the identified bumps in the segmented bump list output by step 510 are then classified, for example, by an AI network. Training the AI ​​network, i.e., training it to receive the segmented bump list and automatically classify the heights of the bumps in that list, is described in more detail below with reference to FIGS. 7 and 8. Classifying bump heights using an AI network is possible because such classification techniques are fast, automated, and highly efficient. However, using an AI network to classify bump heights is not essential, and in some applications, other techniques, such as defocus depth, can be used to classify bump heights. This may be appropriate, for example, if an insufficient amount of training data is available to train an AI network.

[0093] The output of step 511 is a classified bump list of bump heights. The AI ​​network may be operable to classify bump heights with a relatively low first accuracy. For example, bump heights may be classified to an accuracy of about 7 μm, about 5 μm, or down to about 3 μm. This accuracy would be unacceptably poor for classifying discontinuities in the imaged surface. However, the accuracy is sufficiently fine that it may be useful for resolving phase ambiguities in phase maps generated based on the DH data, as described in more detail below.

[0094] According to one potential embodiment of the present invention, as shown in Figure 6 as process 600, both the segmentation performed in step 510 and the bump height classification performed in step 511 can be augmented with the use of DH data generated in step 506. In this embodiment, the segmentation and bump classification are not based solely on the incoherent light image acquired in step 508, but rather the segmentation and classification can be aided by the use of the DH data. This would be advantageous, as DH data is particularly well suited for surface profiling and segmentation.

[0095] As seen in phase ambiguity resolution step 512, the low precision bump height classification data can then be used to resolve phase ambiguities in the phase map generated in step 506.

[0096] The inputs to phase ambiguity resolution step 512 are the DH phase map with the underlying phase ambiguity, generated by process step 506 of Figure 5 or Figure 6, and the segmented, height-sorted bump list, generated by process step 511 of Figure 5 or Figure 6. In step 512, an ambiguity resolution decision is applied to the DH phase map based on the segmented, height-sorted bump list.

[0097] For example, a given phase shift due to a bump may correspond to a phase difference of π / 2, π / 2+2π, π / 2+4π, etc. Such phase shifts may be referred to as ambiguous phase shifts because the height of the bump at the pixel where the phase shift is measured, i.e., the height derived based on the phase shift, is also ambiguous. However, in step 512, the ambiguity can be resolved using low-precision data regarding the bump height; this low-precision height data can be used to determine which of several ambiguous range height candidates in the phase map corresponds to the low-precision height associated with the classified bump list. That is, the bump height can be derived based on the phase map without ambiguity, with a second precision that is finer than the precision of the low-precision classification.

[0098] Reference is now made to Figures 7 and 8, which are simplified, separate flow charts illustrating the training of an AI network for use in digital hologram reconstruction in accordance with the embodiments of Figures 5 and 6, respectively.

[0099] As discussed above with reference to Figures 5 and 6, the bump heights for inclusion in the segmented bump list can be generated by an AI network in step 511. Such a network must be trained prior to use in process 500 or 600 to enable it to perform bump height classification automatically. Training such a network is now described with reference to Figures 7 and 8.

[0100] 7 and 8, as seen in step 702, an incoherent light image acquisition system can be operated to record an image stack of a surface having at least one three-dimensional feature, hereinafter referred to as a bump, but which may have any topology. The stack can be recorded at a range of focal lengths relative to the imaged surface, for example by using an automated stage to adjust the distance of the image acquisition device from the imaged surface.

[0101] The images are then segmented with respect to bumps, as seen in step 704. Such image segmentation may be performed using any suitable image segmentation technique. The output of training step 704 is a list indicating the presence and location of bumps in each of the images comprising the image stack acquired in step 702.

[0102] Turning specifically to FIG. 8 , as seen in step 705, the DH system can optionally and additionally record an interferogram of the imaged surface over the same FOV as that captured in the incoherent image acquired in step 702. This step 705 is appropriate for training the AI ​​network solely for use in the embodiment of FIG. 6 in which the DH data is used to assist in image segmentation and classification. Step 705 can be performed by physically capturing an interferogram of the imaged surface over the same FOV as that captured in the incoherent image acquired in step 702. Alternatively, step 705 can be performed by capturing a single interferogram of the imaged object, reconstructing the wavefront, and digitally propagating the wavefront to generate multiple images corresponding to multiple focal depths along the z-direction within the imaged object. Each of the multiple images can have multiple pixels. The multiple images form a stack of coherent light images at a range of depths within the imaged object, corresponding to the incoherent light image acquired in step 702. The depth of the imaged object can be defined as the dimension of the object along the z-direction, where the z-direction is perpendicular to the x and y axes that define the object plane.

[0103] Returning to both Figures 7 and 8, as seen in step 706, the AI ​​network can be operated to receive the segmented bump list output in step 704, and in the case of Figure 8, can also be operated to receive DH data used to assist in image segmentation and classification. In addition, the AI ​​network is also provided with through-focus height position data from the encoder, representing ground truth for the true bump heights. The ground truth height position data from the encoder is associated with images taken at known distance steps along the z-direction. Based on these images, a best-focus image for each bump can be found, for example, by DFF or by a human operator. The AI ​​network can then be trained in step 706 to classify the bump heights in the segmented bump list based on the appearance of the bumps in the incoherent light images. Because the through-focus motion is controlled and precise, images of the bump at multiple precisely known out-of-focus positions can be used by the AI ​​network. That is, the network can be appropriately trained to provide a bump height estimator based on bump appearance in a single arbitrarily focused video frame, as shown in FIGS.

[0104] As an example, the AI ​​network can be trained on approximately 10,000 examples, divided into a training set and a validation set. Examples in the validation set are not seen by the network during training. The training set can consist of approximately 80% of the examples, and the validation set can consist of approximately 20%. Network training can be stopped based on at least one of the following criteria: no further improvement in error (MSE) reduction is achieved with further training; overfitting occurs; and the maximum number of training epochs has been reached. A wide variety of networks can be suitable. Generally, suitable networks incorporate convolutional blocks, data reduction layers, and activation / normalization methods.

[0105] An example image stack of a bump on an imaged surface captured at a range of focal lengths is shown in FIG. 9. A set of bump images at various depths along the z-direction, actually acquired by an incoherent light imager, is shown in first image set 900. The distance between images along the z-direction is 5 μm in this example. Such a set of images can be generated by training step 702 of FIG. 7 or FIG. 8. A corresponding set of images with the same FOV as that shown in image set 900, acquired by a DH imaging system where images at various depths are generated by wavefront propagation along the z-direction, is shown in second image set 902. Such a set of images can be generated by step 705 of FIG. 8.

[0106] It should be noted that the primary difference between the various embodiments described above lies in their potentially better adaptability to various application scenarios with a given amount of system resources, rather than their theoretical performance parameters. Adaptability in this case refers to the ability to generate successful 3D profiling with less noise and signal drop-off artifacts. Indeed, the system of FIG. 2 requires training of an AI network, but may offer improvements over that of FIG. 1 in some applications. The system of FIG. 3 further improves at the expense of an auxiliary non-coherent illuminator. The systems of FIGS. 5 and 6 can run faster because the wavefront is not propagated, although the AI ​​training process may be more complex and 3D profiling performance may be degraded.

[0107] Reference is now made to FIG. 10, a schematic, partially depicted, partially block diagram of a digital hologram acquisition and reconstruction system constructed and operative in accordance with an embodiment of the present invention to perform the steps involved in the processes of any of FIGS. 1-8.

[0108] As can be seen in Figure 10, a digital hologram reconstruction system 1000 is provided. System 1000 includes a digital holography (DH) image acquisition subsystem 1002 operable to acquire a digital hologram of an object, which in this illustrative example is a surface 1004 having at least one three-dimensional feature 1006 formed thereon. Surface 1004 may be an electronic device, such as a wafer, PCB, or FPD, for which system 1000 is particularly well suited for reconstructing the DH.

[0109] The DH image acquisition subsystem 1002 may include a coherent light source of at least one wavelength, embodied in this illustrative example as a fiber-coupled laser 1010. The laser output of the fiber-coupled laser 1010 is distributed to a first reference arm or fiber 1012 and a second sample arm or fiber 1014. The laser light traveling in the reference fiber 1012 acts as a reference beam. The laser light traveling in the sample fiber 1014 impinges on the surface 1004 and generates an interference pattern, or interferogram, by interference with the reference beam. One example of a fiber-coupled laser useful in the system 1000 is the IBEAM-SMART-488-S-HP, commercially available from Toptica of Gräfelfing, Germany, which operates at wavelengths of 488, 520, and 532 nm. Although a fiber coupled laser source is shown in FIG. 10, the system 1000 can also operate with a more conventional free space laser source, as depicted as a DHM free space input 1016 .

[0110] The laser light from the reference fiber 1012 passes through a first collimator 1020, a delay line 1022, and a beam splitter 1024. A portion of the reference laser light is reflected by the beam splitter 1024 toward a camera 1026. The camera 1026 can capture DH images and / or incoherent light images. The functions of the camera 1026 can be shared among multiple cameras, such that the camera 1026 includes a DH camera and a separate white light camera. Alternatively, the functions can be combined into a single camera. One example of a camera useful in the system 1000 is the UI-3880CP-M-GL Rev.2, commercially available from IDS, Obersulm, Germany.

[0111] The laser light exiting the sample fiber 1014 passes through a second collimator 1030, a beam splitter 1032, a condenser 1034, and another beam splitter 1036. At the beam splitter 1036, the laser light is reflected through a microscope objective 1038 toward the surface 1004.

[0112] Light reflected by surface 1004 propagates back through microscope objective 1038, from where it travels through beamsplitter 1036, through tube lens 1040, to beamsplitter 1024, and to camera 1026. Light reflected from surface 1004 travels along optical axis A, along which the height of surface 1004 is measured by system 1000. That is, light diffracted by surface 1004 interferes with a reference laser beam that is not impinging on surface 1004, and the resulting interference pattern, or interferogram, is captured by camera 1026.

[0113] The interferogram acquired by the camera 1026 may be provided to an image processing module 1050. The image processing module 1050 may include a wavefront reconstructor unit 1052, a phase map generator unit 1054, an image analyzer unit 1056, a phase ambiguity resolver unit 1058, and a feature height calculator unit 1060.

[0114] The wavefront reconstructor unit 1052 may be operable to reconstruct a wavefront based on a digital hologram acquired by the camera 1026. The wavefront may be reconstructed by techniques such as Fourier and convolution reconstruction. The phase map generator unit 1054 may be operable to receive the wavefront generated by the wavefront reconstructor unit 1052 and to generate a phase map of the surface 1004 having the features 1006 thereon based on the wavefront. As explained above, such phase maps typically contain discontinuity artifacts that cause phase ambiguities due to phase wrapping, which can then attribute phase shifts to one wavelength multiple of a candidate range. Ambiguities in the phase map can be reduced, if not eliminated, by operating the system 1002 at multiple wavelengths, e.g., two or more wavelengths.

[0115] To resolve such ambiguities and thus enable unambiguous derivation of the height of three-dimensional features 1006 on surface 1004 that are responsible for the phase shift encoded in the phase map, system 1000 can include an additional imaging modality 1070. The additional imaging modality 1070 can be embodied as an incoherent light illuminator 1070. Light from the incoherent light illuminator 1070 can propagate toward beamsplitter 1032, where it can be reflected toward condenser 1034, and by beamsplitter 1036 through microscope objective 1038 toward surface 1004. Light reflected by surface 1004 propagates back toward microscope objective 1038, from where it travels via beamsplitter 1036 through tube lens 1040 to beamsplitter 1024 and camera 1026, where a white-light image is recorded.

[0116] The image analyzer 1056 may be operable to obtain height data for the three-dimensional feature 1006 from at least one additional image of the surface 1004. The at least one additional image may include at least one image acquired by an additional imaging modality 1070 according to the process outlined above with reference to Figures 2, 3, or 6. Alternatively, the at least one additional image may not be an image acquired by the additional imaging modality 1070, but may be an additional DH image acquired by propagating a wavefront derived by the wavefront reconstructor unit 1052 through a series of depths within the surface 1004, as described with reference to Figure 1.

[0117] Whatever the specific type of additional image used, height data obtained from the additional image is provided to a phase ambiguity resolver 1058. The phase ambiguity resolver 1058 may operate to resolve phase ambiguities in the phase map output by the phase map generator 1054 based on the height data provided by the image analyzer 1056. The height data provided by the image analyzer 1056 may be of a relatively poor first accuracy, but may be of sufficiently fine accuracy to be usable in resolving ambiguities in the phase map.

[0118] The feature height calculator 1060 may be operable to derive a height of at least one three-dimensional feature 1006 based on the phase map and after resolving phase ambiguities in the phase map. The height of the three-dimensional feature 1006 derived based on the phase map is derived with a second precision that is finer than the first precision of the height data output by the image analyzer 1056.

[0119] Image processing module 1050 typically includes at least one programmable processor programmed with software and / or firmware to perform the functions described herein, as well as suitable digital and / or analog interfaces for connection to other components of system 1000. Alternatively or additionally, image processing module 1050 includes hardwired and / or programmable hardware logic circuitry that performs at least some of the functions of image processing module 1050. For simplicity, image processing module 1050 is depicted in FIG. 10 as a single, monolithic functional block; however, in practice, image processing module 1050 may include multiple interconnected control units and suitable interfaces for receiving and outputting signals as depicted in the figures and described in the text. Program code or instructions for causing image processing module 1050 to perform the methods and functions disclosed herein may be stored in a readable storage medium, such as the memory within image processing module 1050 or other memory.

[0120] FIG. 11 is a schematic flow chart illustrating a possible implementation of the system of FIG. 10 in accordance with an embodiment of the present invention.

[0121] As shown in FIG. 11 , the implementation of the system of FIG. 10 can be expressed in the form of process 1100. Process 1100 represents a potential implementation of the system of the present invention when used as an inspection and / or automated repair system. Such an inspection system can be useful, for example, in inspecting electronic devices during or after their manufacture, thereby identifying the presence of defects and / or assessing whether the device features are formed according to predetermined conditions. One example application is an automated system for optically inspecting and repairing printed circuit boards (PCBs) for open or short circuit defects. A 3D profiling system can be used to close the automated repair loop. However, as can be appreciated, the method illustrated in chart 1100 is not limited to electronic device inspection applications and can also be used in a variety of other contexts, such as 3D profiling of thick biological specimens.

[0122] As seen in a first step 1102, a DH image or interferogram is acquired of a device having a surface with at least one 3D feature thereon. For example, the DH image can be a digital holographic microscopy image, such as those acquired by the DH image acquisition subsystem 1002 shown in FIG. 10 .

[0123] As seen in a second step 1104, phase data is obtained from the interferogram. The phase data can be obtained by reconstructing a wavefront based on the interferogram and then processing the wavefront to generate a phase map. The wavefront can be reconstructed intelligently, for example, by a wavefront reconstructor unit 1052 of FIG. 10 . A phase map can be generated based on the reconstructed wavefront, for example, by a phase map generator unit 1054 of FIG. 10 . The phase map can include amplitude and phase information for each pixel of the interferogram. The phase map typically has inherent phase ambiguities due to phase wrapping. These phase ambiguities must be resolved in order to unambiguously interpret the phase map and derive unambiguous data regarding the height of features on the surface that cause the measured phase shift.

[0124] As seen in a third step 1106, the phase ambiguity can be resolved using height data obtained from a corresponding incoherent light image of the surface or from a DHM image generated by propagating a wavefront through various depths relative to the surface. The height data can be referred to as coarse height data, meaning that the height data has a lower accuracy than that of height data obtainable directly from the interferogram. However, such height data accuracy is sufficient to resolve the phase ambiguity. The height data can be obtained, for example, by the image analyzer unit 1056 in FIG. 10. Based on the height data, the phase ambiguity can be resolved, for example, by the phase ambiguity resolver unit 1058 in FIG. 10. The phase ambiguity can be resolved according to the process of FIG. 1, the process of FIG. 2, the process of FIG. 3, or the process of FIG. 5 or FIG. 6, as described above.

[0125] As seen in the fourth step 1108, after any ambiguities in the phase data have been resolved, the phase data can be used to derive the height of a feature on the imaged surface, for example, the feature height can be found by the feature height calculator unit 1060 of FIG.

[0126] As seen in fifth step 1110, process 1100 may then check whether the derived feature height is within a predetermined acceptable range or threshold. If so, in sixth step 1112, a human-sensible output may be provided indicating that the height of the feature on the imaged surface is acceptable, and the apparatus may proceed unhindered as seen in seventh step 1114. Providing a human-sensible output in sixth step 1110 is optional and may not be necessary in some cases.

[0127] If the feature height is found to be outside of its predetermined acceptable range or threshold, an output indicating this can be provided, as seen in eighth step 1116. Appropriate corrective action can then be taken, as seen in ninth step 1118. Appropriate corrective action may include sending the device for reprocessing to correct the feature formation. In some cases, the device may be discarded because the time and effort required for correction is not worth it. If the device is reprocessed, the reprocessed device may be re-imaged similarly to that in first step 1102. Providing a human-sensible output in eighth step 1116 is optional and may not be necessary in some cases.

[0128] Those skilled in the art will appreciate that the present invention is not limited to what has been particularly shown and described above, and that the scope of the present invention includes both combinations and subcombinations of the features described above and modifications thereof that do not fall within the prior art.

Claims

1. A digital hologram reconstruction method, comprising: obtaining a digital hologram of a surface having at least one three-dimensional feature thereon; Reconstructing a wavefront based on the digital hologram; generating a phase map of at least a portion of the surface based on the wavefront, wherein the phase map has an inherent phase ambiguity; acquiring at least one additional image of the surface; obtaining height data for the three-dimensional features from the at least one additional image of the surface, wherein the height data is obtained with a first precision; resolving the phase ambiguity based on the height data; and deriving a height of the at least one three-dimensional feature based on the phase map after the resolution of the phase ambiguity inherent therein, wherein the height is derived with a second precision that is finer than the first precision; said acquiring at least one additional image of said surface using an AI network to generate a series of incoherent light images corresponding to a series of digital wavefronts of said surface, said series of digital wavefronts being acquired by digitally propagating said wavefronts through a series of depths within said surface and acquiring said series of digital wavefronts corresponding to said series of depths; The AI ​​network is an AI network that has been trained to convert a coherent light image into an equivalent non-coherent light image using white light measurements. method.

2. 2. The method of claim 1, wherein said acquiring at least one additional image of said surface comprises digitally propagating said wavefront through a series of depths within said surface and acquiring a series of digital wavefronts corresponding to said series of depths.

3. 3. The method of claim 2, wherein said height data acquisition utilizes a DFF algorithm to acquire height data based on said series of digital wavefronts.

4. 2. The method of claim 1, wherein acquiring the height data utilizes a DFF algorithm to acquire height data based on the series of non-coherent light images.

5. 2. The method of claim 1, wherein acquiring the height data utilizes the AI ​​network to automatically acquire height data based on segmentation and classification of the at least one three-dimensional feature.

6. 2. The method of claim 1, wherein the first accuracy is in the range of 1 to 5 μm.

7. The method of claim 6, wherein the second precision is in the range of 1 to 100 nm or 50 to 1000 nm.

8. 10. The method of claim 1, wherein said acquiring said digital hologram comprises acquiring a digital microscopic hologram.

9. A digital hologram reconstruction system, comprising: a digital holographic image acquisition subsystem operable to acquire a digital hologram of a surface having at least one three-dimensional feature thereon; a wavefront reconstructor operable to reconstruct a wavefront based on the digital hologram; a phase map generator operable to receive the wavefront and to generate a phase map of at least a portion of the surface based on the wavefront, the phase map having an inherent phase ambiguity; an additional image acquisition or image processing subsystem operable to acquire at least one additional image of the surface; an image analyzer operable to obtain height data relating to the three-dimensional feature from the at least one additional image of the surface, the image analyzer obtaining the height data with a first accuracy; a phase ambiguity resolver operable to resolve the phase ambiguity in the phase map based on the height data; a height calculator operable to derive a height of the at least one three-dimensional feature based on the phase map after the resolution of the phase ambiguity inherent therein, the height calculator deriving the height of the three-dimensional feature with a second precision that is finer than the first precision; and Equipped with acquiring the at least one additional image of the surface by utilizing an AI network to generate a series of incoherent light images corresponding to a series of digital wavefronts of the surface, the series of digital wavefronts being acquired by digitally propagating the wavefront through a series of depths within the surface and acquiring the series of digital wavefronts corresponding to the series of depths; The AI ​​network is a system that is trained to convert a coherent light image into an equivalent non-coherent light image using white light measurements.

10. 10. The system of claim 9, wherein the image processing subsystem is operable to digitally propagate the wavefront through a series of depths within the surface and to acquire a series of digital wavefronts corresponding to the series of depths.

11. 11. The system of claim 10, wherein the image analyzer is operable to obtain the height data based on the series of digital wavefronts using a DFF algorithm.

12. 10. The system of claim 9, wherein the image analyzer is operable to derive the height data based on the series of non-coherent light images using a DFF algorithm.

13. 10. The system of claim 9, wherein the additional image acquisition subsystem comprises: an incoherent illuminator operable to illuminate the surface with incoherent light; and a camera operable to acquire the at least one additional image of the surface, wherein the additional images acquired by the camera are used to assist in spatial filtering and smoothing of the series of incoherent light images generated using the AI ​​network.

14. 14. The system of claim 13, wherein the image analyzer comprises the AI ​​network operable to automatically obtain height data based on the segmentation and classification of the at least one three-dimensional feature.

15. 10. The system of claim 9, wherein the first accuracy is in the range of 0.5 to 5 μm.

16. 16. The system of claim 15, wherein the second accuracy is in the range of 1 to 100 nm or 50 to 1000 nm.

17. 10. The system of claim 9, wherein the digital holographic image acquisition subsystem is a digital holographic microscopic image acquisition subsystem.

Images