System and method for 3D imaging of samples using machine learning algorithms

Abstract

A method for 3D imaging of a sample using a machine learning algorithm is disclosed. The method uses a multimodality focus stack 101. The multimodality focus stack 101 consists of multiple images acquired at two or more distances between the sample and the forward focal plane, where at least one image is acquired using a first modality and additional images are acquired using additional modalities. The modalities may have different illumination angles and may optionally have different spectral distributions. The method may include receiving training images of the sample including multiple training focus stacks and ground truth 3D data (depth maps) for each training focus stack, and training a machine learning algorithm 118 based on the training images and ground truth data. Subsequently, the method receives product images of the sample including the focus stacks from an optical assembly 102 and generates a 3D depth map.

Description

Technical Field

[0001] The present invention generally relates to three-dimensional (3D) imaging, and more specifically, to a system and method for performing 3D imaging of a sample based on a multimodality focus stack using a machine learning algorithm, where the machine learning algorithm is trained using a set of multimodality focus stacks.

Background Art

[0002] Increasing demand for electronic circuits with ever-smaller device features continues to drive the need for improved 3D imaging technology. Optical inspection systems may require 3D imaging to measure circuit features (e.g., conductors, drill holes, etc.) in three dimensions and identify or verify "3D defects" (e.g., dish-down, i.e., small areas where copper has thinned). Optical shaping systems may be used to improve yields and reduce scrap in the manufacturing process of printed circuit boards. Such optical shaping systems can utilize 3D imaging when repairing defects. For example, 3D imaging may be used in the repair process when depositing copper on the surface of a printed circuit board. As another example, 3D imaging may be used to measure damage to a sample to ensure that the laser does not accidentally penetrate the laminate when removing excess copper (e.g., using laser ablation) after the repair process.

[0003] Current focus-based 3D imaging techniques are generally cost-effective, but due to the need for pixel-by-pixel focus quality determination based on calculations in the vicinity of each pixel, the spatial accuracy and precision are limited compared to the imaging resolution. Other 3D imaging techniques, such as those based on interferometry, are more expensive and some may require additional optical components, which can complicate the system and reduce system throughput.

Prior Art Documents

Patent Documents

[0004] [Patent Document 1] U.S. Patent Application Publication No. 2014 / 086475 [Patent Document 2] U.S. Patent Application Publication No. 2022 / 120664 [Patent Document 3] International Publication No. 2022 / 013850 [Patent Document 4] U.S. Patent Application Publication No. 2021 / 360140 [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, it is advantageous to provide systems and methods that improve upon the shortcomings of the methods identified above. [Means for solving the problem]

[0006] Optical systems according to one or more embodiments of the present disclosure are disclosed. In an embodiment, the optical system comprises an optical assembly configured to illuminate one or more portions of a sample using two or more illumination modalities, including at least a first illumination modality and a second illumination modality, wherein the first illumination modality includes a first set of illumination angles, and the second illumination modality includes a second set of illumination angles, wherein at least the second set of illumination angles is at least partially different from the first set of illumination angles. In an embodiment, the optical assembly is configured to acquire a multimodality focus stack, wherein the multimodality focus stack includes a plurality of images acquired at two or more distances between the sample and the forward focus, wherein at least a first of the plurality of images is acquired using the first illumination modality, and at least additional images are acquired using the second illumination modality. In an embodiment, the optical system further comprises an image processing subsystem communicatively coupled to the optical assembly, wherein the image processing subsystem includes one or more processors configured to execute a set of program instructions stored in memory. In one embodiment, the set of program instructions is configured to cause one or more processors to receive multiple training images, each of which includes multiple training multimodality focus stacks. In another embodiment, the set of program instructions is configured to cause one or more processors to receive 3D ground truth data for each of the multiple training multimodality focus stacks. In yet another embodiment, the set of program instructions is configured to cause one or more processors to train a machine learning algorithm based on the multiple training images and the received 3D ground truth data. In yet another embodiment, the set of program instructions is configured to cause one or more processors to receive multimodality focus stacks of samples from an optical assembly.In one embodiment, the set of program instructions is configured to cause one or more processors to generate a depth map of a sample using a trained machine learning algorithm and a received multimodality focus stack.

[0007] Image processing systems according to one or more embodiments of the present disclosure are disclosed. In an embodiment, the system comprises one or more processors configured to execute a set of program instructions stored in memory. In an embodiment, the set of program instructions is configured to cause one or more processors to receive a plurality of training images, the one or more training images comprising a plurality of training multimodality focus stacks. In an embodiment, the set of program instructions is configured to cause one or more processors to receive 3D ground truth data for each of the plurality of training multimodality focus stacks. In an embodiment, the set of program instructions is configured to cause one or more processors to train a machine learning algorithm based on the plurality of training images and the received 3D ground truth data. In an embodiment, the set of program instructions is configured to cause one or more processors to receive a multimodality focus stack of a sample from an optical assembly, the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modalities, including at least a first illumination modality and a second illumination modality, the first illumination modality including a first set of illumination angles, the second illumination modality including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodality focus stack including a plurality of images acquired at two or more distances between the sample and the forward focus, at least a first of the plurality of images acquired using the first illumination modality, and at least additional images acquired using the second illumination modality. In an embodiment, the set of program instructions is configured to cause one or more processors to generate a depth map of the sample using a trained machine learning algorithm and the received multimodality focus stack.

[0008] Methods according to one or more embodiments of the present disclosure are disclosed. In an embodiment, the method comprises receiving a plurality of training images, one or more of which comprise a plurality of training multimodality focus stacks. In an embodiment, the method comprises receiving 3D ground truth data for each of the plurality of training multimodality focus stacks. In an embodiment, the method comprises training a machine learning algorithm based on the plurality of training images and the received ground truth 3D data. In an embodiment, the method includes receiving a multimodality focus stack of a sample from an optical assembly, the optical assembly configured to illuminate one or more portions of the sample using two or more illumination modalities, including at least a first illumination modality and a second illumination modality, the first illumination modality including a first set of illumination angles, the second illumination modality including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, and the multimodality focus stack including a plurality of images acquired at two or more distances between the sample and the forward focus, at least a first of the plurality of images acquired using the first illumination modality, and at least additional images acquired using the second illumination modality. In an embodiment, the method includes generating a depth map of the sample using a trained machine learning algorithm and the received multimodality focus stack.

[0009] It will be understood that both the above general description and the following detailed description are illustrative and descriptive only and do not necessarily limit the invention as described in the claims. The accompanying drawings incorporated herein and forming part thereof illustrate embodiments of the invention and, together with the general description, are helpful in illustrating the principles of the invention.

[0010] Many of the advantages of this disclosure can be better understood by those skilled in the art by referring to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1A] A block diagram of an optical system according to one or more embodiments of this disclosure is shown. [Figure 1B] A simplified schematic top view of an optical assembly of an optical system according to one or more embodiments of the present disclosure is shown. [Figure 1C] A simplified schematic side view of an optical assembly of an optical system according to one or more embodiments of the present disclosure is shown. [Figure 1D] A simplified schematic top view of an optical assembly of an optical system according to one or more embodiments of the present disclosure is shown. [Figure 1E] A simplified schematic side view of an optical assembly of an optical system according to one or more embodiments of the present disclosure is shown. [Figure 2A] The following is a process flow diagram illustrating a method for training a machine learning algorithm for a system according to one or more embodiments of the present disclosure. [Figure 2B] The following is a process flow diagram illustrating a method for generating one or more depth maps using a system-trained machine learning algorithm according to one or more embodiments of the present disclosure. [Figure 3] The following process flow diagrams illustrate a method for obtaining one or more focus stacks using one or more components of a system according to one or more embodiments of the present disclosure. [Figure 4A] The images show focus stacks of samples according to one or more embodiments of this disclosure. [Figure 4B] The images show focus stacks of samples according to one or more embodiments of this disclosure. [Figure 5] A white light interferometry depth map of a sample according to one or more embodiments of this disclosure is shown. [Modes for carrying out the invention]

[0012] The present disclosure has been particularly shown and described with respect to several embodiments and specific features thereof. The embodiments described herein are to be construed as illustrative and not restrictive. It should be readily understood by those skilled in the art that various modifications and changes in form and detail can be made without departing from the spirit and scope of the present disclosure. Here, reference is made in detail to the subject matter of the invention as disclosed, shown in the accompanying drawings.

[0013] As features have become increasingly miniaturized and the demand for thinner electronic circuits continues to grow, the need for improved 3D imaging technology has also been continuously increasing. In the manufacturing process of printed circuit boards, an optical inspection system may be used for process control and yield management. Such a system can utilize 3D imaging for defect identification and / or verification, as well as for measuring the features of circuits in 3D. An optical shaping system may be used to eliminate scrap and improve the yield of printed circuit boards. Such a system can utilize 3D imaging when repairing defects. The optical shaping system is outlined in U.S. Patent No. 8,290,239, issued on October 16, 2012, and U.S. Patent Publication No. 2013 / 0037526, published on February 14, 2013, the entire contents of which are incorporated herein by reference.

[0014] Current 3D imaging techniques using depth from focus (DFF) capture a single image at each object distance from the focal plane of the optical system. From such a set of images at different object distances, the DFF algorithm determines, at each imaged point of the object, the position where the neighborhood of that point is in best focus. However, one drawback of "standard" DFF is that determining the focus quality of individual pixels requires calculations based on the neighborhood of each pixel, so the spatial accuracy and precision are limited compared to the imaging resolution.

[0015] White light interferometry (WLI) techniques, as well as other 3D imaging techniques such as multi-view stereo and triangulation or phase-shift methods, provide higher spatial accuracy and precision, but are costly and require additional optical components. Further, other imaging techniques such as photometric stereo (PS) typically have a high computational load (and thus are time-consuming) and require a detailed and accurate reflection model of the material composing the imaged object to obtain accurate surface reconstruction.

[0016] Therefore, it is advantageous to provide a system and method that improve the drawbacks of the techniques identified above.

[0017] Embodiments of the present disclosure are directed to a system and method for generating one or more depth maps of one or more portions of a sample by capturing a set of images at at least two different illumination angles in a set of distances between the sample and a front focal plane and using a machine learning algorithm to obtain a depth map from this data. For example, the system can include an optical assembly configured to acquire a set of images at at least two different illumination angles in a set of distances between the sample and a front focal plane. As described above herein, in a “standard” DFF, a single image is acquired for each distance from the object, whereas in the system of the present disclosure, a multi-modal focus stack of the sample can be acquired under different illumination modalities. In this regard, the system can acquire reflection information under different illumination directions for each pixel on the sample. Such information can be used to improve the spatial resolution, accuracy, and precision of the resulting depth map. In the present disclosure, a “depth map” and its variations can be defined as data consisting of depth values (z) as a function of positions (x, y) in a vertical plane (e.g., at each of the imaging pixels).

[0018] Unlike PS, which requires a reflection model as described above in this specification, the systems and methods of this disclosure do not depend on a material reflection model; rather, they use a machine learning approach to train the system with respect to the type of sample being imaged. Furthermore, the use of a machine learning approach can reduce the number of images required for 3D reconstruction when generating one or more depth maps.

[0019] Furthermore, the system may include an image processing subsystem configured to use a machine learning algorithm trained with focus stacks and ground truth 3D data from multiple illuminations (e.g., acquired by WLI or other imaging techniques). After training, the machine learning algorithm may be configured to generate one or more depth maps of one or more parts of a sample based on images collected from the optical assembly. In some embodiments, the machine learning algorithm may be configured to generate one or more depth maps of one or more parts of a sample with a depth resolution of approximately 0.1 to 2 microns (μm).

[0020] Figure 1A shows a schematic block diagram of an optical system 100 according to one or more embodiments of the present disclosure. In particular, Figure 1A shows a system 100 for generating depth maps of one or more parts of a sample using machine learning techniques.

[0021] In one embodiment, the system 100 includes an optical assembly 102. The optical assembly may include, but is not limited to, one or more imaging devices 104, one or more illumination devices 106, one or more controllers including one or more processors and memory, a chassis, and the like.

[0022] The system 100 may also include an image processing subsystem 108 that is communicatively coupled to the optical assembly 102. The image processing subsystem 108 may include, but is not limited to, one or more processors 112 and one or more controllers including memory 114.

[0023] The optical assembly 102 and / or the image processing subsystem 108 can be communicated to the user interface device 116.

[0024] Figures 1B to 1E show simplified schematic diagrams of the optical assembly 102 of the optical system 100 according to one or more embodiments of the present disclosure. In particular, Figure 1B shows a top view of the single-ring illumination device 106 of the optical assembly 102 according to one or more embodiments of the present disclosure. In particular, Figure 1C shows a side view of the single-ring optical assembly 102 of the optical system 100 according to one or more embodiments of the present disclosure. In particular, Figure 1D shows a top view of the double-ring optical assembly 102 of the optical system 100 according to one or more embodiments of the present disclosure. In particular, Figure 1E shows a side view of the double-ring optical assembly 102 of the optical system 100 according to one or more embodiments of the present disclosure.

[0025] In an embodiment, the optical assembly 102 is configured to acquire one or more multimodality focus stacks of sample 120. For example, the optical assembly 102 may be configured to illuminate sample 120 using two or more illumination modalities. For example, the two or more illumination modalities may include at least a first illumination modality and a second illumination modality, where the first modality includes a first set of illumination angles and the second modality includes a second set of illumination angles. As another example, the first modality may include a first spectral distribution and a second spectral distribution, where the second spectral distribution is at least partially different from the first spectral distribution. In this view, the multimodality focus stack may include a plurality of images acquired at two or more distances between the forward focal plane and sample 120, where at least one image is acquired using the first modality and additional images are acquired using the second modality, which are at least partially different from the first and second modalities.

[0026] The optical assembly 102 can be positioned relative to the sample 120 mounted on the stage assembly 122. In embodiments, the sample may be heterogeneous, formed from two or more materials, where the first material is at least different from the second material. For example, the heterogeneous sample may include a printed circuit board (PCB). The two or more materials may include, but are not limited to, copper, laminate, etc.

[0027] One or more imaging devices 104 of the optical assembly 102 can be configured to acquire one or more images of a sample 120 mounted on the stage assembly 122. In this disclosure, the term "one or more imaging devices 104" means one or more imaging devices including an imaging sensor (e.g., a camera) and one or more optical elements (e.g., one or more lenses), unless otherwise specified.

[0028] One or more imaging devices 104 can include any type of imaging device suitable for acquiring one or more two-dimensional (2D) images of samples 120. For example, one or more imaging devices 104 can include one or more cameras configured for image acquisition. For example, one or more imaging devices 104 can include one or more high-speed cameras configured for high-speed image acquisition (e.g., exceeding a standard video rate of 24-25 frames / second (e.g., 90 frames / second)).

[0029] In one embodiment, one or more imaging devices 104 share a common axis with the sample 120 mounted on the stage assembly 122 when acquiring one or more images. For example, one or more imaging devices 104 can be positioned above at least a portion of the sample 120 mounted on the stage assembly 122 when acquiring one or more images.

[0030] In an embodiment, the optical assembly 102 further includes one or more actuator assemblies 107. One or more actuator assemblies 107 can be configured to adjust the focal length of the optical assembly 102 with respect to the sample 120 (i.e., the distance between the sample and the forward focal plane). For example, one or more actuator assemblies can be configured to adjust the focal length of one or more imaging devices 104 by acting on the optical assembly 102 at least along the z-axis (common axis). As another example, one or more actuator assemblies 107 can be configured to adjust the focal length of the sample 120 by acting on it at least along the z-axis (common axis). For example, one or more actuator assemblies 107 can be configured to adjust the focal length of one or more imaging devices 104, thereby adjusting the focal length between one or more imaging devices 104 and the sample 120. In this view, one or more imaging devices 104 can be configured to acquire two or more images of the sample at two or more sets of focal lengths.

[0031] It should be noted that one or more actuator assemblies 107 may include at least any type of actuator device suitable for adjusting the focal length of the optical assembly 102 relative to the sample 120. For example, one or more actuator assemblies 107 may include, but are not limited to, one or more linear actuator devices. Furthermore, it should be noted that the system 100 may be configured to adjust the focal length of the optical assembly 102 relative to the sample 120 via any mechanism. Therefore, the above discussion should not be construed as limiting the scope of the disclosure. For example, the optical assembly 102 may include a variable focus lens configured to adjust the focal length of the optical assembly 102 relative to the sample 120. As another example, one or more components of the optical assembly 102 may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120. For example, one or more imaging sensors may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120.

[0032] In an embodiment, one or more imaging devices 104 can be configured to acquire two or more images of a sample 120 in a set of distances while one or more actuator assemblies are operating. For example, two or more images of sample 120 can be acquired via one or more actuator assemblies during the motion of an optical assembly 102.

[0033] In one embodiment, one or more imaging devices 104 can be configured to acquire two or more images of the sample 120 at a set of focal lengths while one or more actuator assemblies are at a fixed focal length. For example, two or more images of the sample 120 can be acquired when one or more actuator assemblies 107 are at one or more predetermined focal lengths (or distances from the sample) after adjusting the focal length of one or more imaging devices 104.

[0034] In an embodiment, one or more illumination devices 106 of the optical assembly 102 may be configured to generate one or more illumination beams 105. One or more illumination devices 106 may include any illumination devices suitable for generating one or more illumination beams 105. For example, one or more illumination devices 106 may include one or more stroboscopic illumination devices. For example, one or more stroboscopic illumination devices may include, but are not limited to, one or more light-emitting diodes (LEDs) (e.g., red LEDs). In this view, one or more stroboscopic illumination devices may be configured to generate short bursts of illumination to prevent blurring caused by movement / vibration (e.g., in a plane along or perpendicular to a common axis). This enables rapid acquisition by acquiring an image while the distance between the sample and the forward focal plane is changing (e.g., by moving the imaging device 104).

[0035] The optical assembly 102 may further include one or more optical elements 126 configured to direct one or more illumination beams onto the surface of the sample 120. For example, the optical assembly 102 may include, but is not limited to, one or more mirrors, one or more lenses, one or more polarizers, one or more beam splitters, one or more optical fibers, etc. Figures 1B to 1E are provided for illustrative purposes only and should not be construed as limiting the scope of this disclosure. For example, the optical assembly 102 may include any type of optical element and optical elements in any configuration.

[0036] In one embodiment, one or more illumination devices 106 may include a set of illumination devices 106 positioned at a set of illumination angles relative to the sample 120. For example, one or more illumination devices may include a set of illumination devices 106 that include one or more high-elevation illumination devices (e.g., positioned at a high elevation angle relative to the stage / sample) and one or more low-elevation illumination devices (e.g., positioned at a lower elevation angle than the high-elevation devices relative to the stage / sample).

[0037] It should be noted that the optical assembly 102 may include any number and configuration of illumination devices 106 suitable for illuminating the sample 120 at a specified illumination angle (e.g., in several non-collinear illumination directions). Figures 1B–1E are provided for illustrative purposes only and should not be construed as limiting the scope of this disclosure.

[0038] Each set of illumination devices 106 can be positioned near the sample 120 mounted on the stage assembly 122. For example, each set of illumination devices 106a to 106d can be positioned adjacent to (or to the side of) the sample 120 mounted on the stage assembly.

[0039] Referring to Figures 1B to 1E, the lighting devices 106 can be evenly distributed in the azimuthal direction, thereby allowing the lighting devices 106 to cover the range of illumination directions of sample 120 almost uniformly. For example, as shown in Figures 1B to 1C, the lighting devices 106 can be evenly distributed in a single ring configuration. As one example, as shown in Figure 1B, the set of lighting devices 106 can include six lighting devices separated by 60 degrees. As another example, as shown in Figures 1D to 1E, the lighting devices 106 can be evenly distributed in a double ring configuration. In one example, as shown in Figure 1D, the set of lighting devices includes ten lighting devices, with the central ring containing four lighting devices separated by 90 degrees and the outer ring containing six lighting devices separated by 60 degrees.

[0040] Figure 1B shows an optical assembly 102 containing seven illumination devices configured to illuminate a sample at seven different illumination angles. However, it should be noted that the optical assembly 102 can include any number and configuration of illumination devices configured to illuminate a sample at any azimuth and / or elevation angle. For example, the set of illumination devices could include six ring-shaped illumination devices separated by 60 degrees. Furthermore, it should be noted that the optical assembly 102 can be configured to illuminate a sample with any combination of illumination devices at one or more different illumination angles.

[0041] System 100 may further include one or more pre-fabricated reference objects (or calibration targets). For example, one or more pre-fabricated reference objects may include one or more fixed reference objects. Alternatively, one or more pre-fabricated reference objects may include one or more removable reference objects.

[0042] One or more reference objects may include one or more three-dimensional (3D) structures with known (or previously measured) depth maps. For example, one or more known depth maps can be used to verify one or more components of an optical assembly. Another example is that one or more known depth maps can be used to calibrate one or more components of an optical assembly. Another example is that one or more known depth maps can be used to retrain a machine learning algorithm. Another example is that one or more known depth maps can be used to adjust one or more inputs to a machine learning algorithm. For example, one or more depth maps can be used to correct the distance of each image, which can be adjusted after calibration.

[0043] In one embodiment, the image processing subsystem 108 is configured to generate one or more depth maps of samples 120 based on a set of multimodality focus stacks (including one or more 2D images) acquired from the optical assembly 102. As will be discussed further below, for example, the image processing subsystem 108 may be configured to generate one or more depth maps of samples 120 corresponding to two or more 2D images of the focus stack using a machine learning algorithm 118 stored in the memory 114 of the controller 110 of the image processing subsystem 108.

[0044] Figure 2A shows a process flow diagram 200 illustrating a method for training a machine learning algorithm 118 of system 100 according to one or more embodiments of the present disclosure. Note that the process flow diagram 200 can be considered a conceptual flowchart showing the steps performed by / within one or more processors 112 of controller 110.

[0045] In step 202, a set of training multimodality focus stacks can be obtained. For example, the controller 110 of the image processing subsystem 108 can be configured to obtain a set of multimodality focus stacks 101 from the optical assembly 102 that will be used to train the machine learning algorithm 118.

[0046] In an embodiment, each multimodality focus stack 101 may include one or more images acquired at two or more illumination angles at at least some of the distances between the sample and the forward focal plane, or at different illumination angles at different distances, for each of the set of distances between the sample and the forward focal plane. For example, one or more imaging devices 104 of the optical assembly 102 may be configured to acquire one or more images at a set of distances between the sample and the forward focal plane while one or more illumination devices 106 illuminate the sample 120 at one or more different illumination angles.

[0047] In the optional step 203, additional inputs to the machine learning algorithm may be computed. One or more additional inputs may include, but are not limited to, depth maps obtained from a single modality focus stack using a non-learning-based algorithm, one or more post-processed images, one or more additional parameters, etc. For example, the controller 110 may be configured to compute one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired multi-modality focus stack.

[0048] Figure 3 shows a process flow diagram 300 illustrating a method for obtaining one or more focus stacks using one or more components of system 100 according to one or more embodiments of the present disclosure. Note that the flow diagram 300 can be considered a conceptual flowchart showing steps performed by / within one or more processors 112 of controller 110.

[0049] In step 302, a first image can be acquired with a first illumination modality having a first illumination angle (more generally, a specific range of illumination angles, which may partially overlap). For example, one or more imaging devices 104 can be configured to acquire a first image at a first distance between the sample and the forward focal plane, while at least the first illumination device illuminates the sample at at least a first illumination angle. In this disclosure, “illumination modality” and its variations can be defined as a set of one or more illumination angles with the same or different spectral distributions for each angle.

[0050] In the optional step 303, the focal length may be adjusted. For example, when acquiring images during motion, the focal length may be adjusted between acquisitions of additional images with additional modalities.

[0051] In the optional step 304, a second image may be acquired with a second illumination modality having a second illumination angle. For example, one or more imaging devices 104 may be configured to acquire a second image at a first distance between the sample and the forward focal plane while the second illumination device illuminates the substrate at least a second illumination angle.

[0052] In the optional step 305, the focal length may be adjusted. For example, when acquiring images during motion, the focal length may be adjusted between acquisitions of additional images with additional modalities.

[0053] Figures 4A and 4B show exemplary focus stack images according to one or more embodiments of the present disclosure. Note that both sets of images 400a–400d and 420a–420d in Figures 4A and 4B may be part of a single focus stack that may include additional images taken at additional focal lengths and illumination angles.

[0054] Referring to Figure 4A, a first set 400 of focus-stacked images may include a plurality of images 400a to 400d of sample 120 (e.g., PCB) acquired at a first exemplary focal length. For example, the first set of focus-stacked images may include at least a first image 400a acquired at a first exemplary focal length and a first azimuth illumination direction 402a, where image 400a is focused on the copper traces of the PCB. As another example, the first set of focus-stacked images may include at least a second image 400b acquired at a first exemplary focal length and a second azimuth illumination direction 402b, where image 400b is focused on the copper traces of the PCB. As yet another example, the first set of focus-stacked images may include at least a third image 400c acquired at a first exemplary focal length and a third azimuth illumination direction 402c, where image 400c is focused on the copper traces of the PCB. As another example, the first set of focus stacked images includes at least a fourth image 400d acquired at a first exemplary focal length and a fourth azimuth illumination direction 402d, wherein image 400d is focused on the copper traces of the PCB. Referring to Figure 4B, the second set of focus stacked images 420 may include several images 420a-420d of sample 120 (e.g., PCB) acquired at a second exemplary focal length (different from the first exemplary focal length). For example, the second set of focus images may include at least a first image 420a acquired at a second exemplary focal length and a first azimuth illumination direction 422a, wherein image 420a is focused on the laminate of the PCB. As yet another example, the second set of focus images may include at least a second image 420b acquired at a second exemplary focal length and a second azimuth illumination direction 422b, wherein image 420b is focused on the laminate of the PCB. As another example, the second set of focal images may include at least a third image 420c acquired at a second exemplary focal length and a third azimuth illumination direction 422c, wherein image 420c is focused on the PCB lamination.As another example, a second set of focused images includes at least a fourth image 420d acquired at a second exemplary focal length and a fourth azimuth illumination direction 422d, wherein image 420d is focused on the PCB lamination.

[0055] Returning to Figure 3, in the optional step 306, one or more additional images may be acquired by one or more additional illumination modalities. For example, one or more imaging devices 104 may be configured to acquire a third image at a first distance between the sample and the forward focal plane while at least a third illumination device illuminates the sample at at least a third illumination angle. As another example, one or more imaging devices 104 may be configured to acquire a fourth image at a first distance between the sample and the forward focal plane while at least a fourth illumination device illuminates the sample at at least a fourth illumination angle. As yet another example, one or more imaging devices 104 may be configured to acquire one or more images at a first distance between the sample and the forward focal plane using one or more additional illumination spectra (e.g., different from those used for at least one of steps 302-306). Note that the illumination spectrum may induce fluorescence in at least one of two or more materials of the sample 120. For example, if the sample 120 is a PCB, the illumination spectrum may induce fluorescence in the laminate material of the PCB.

[0056] In step 308, the distance between the sample and the forward focal plane of the optical assembly relative to the sample can be adjusted. For example, the working assembly can be configured to adjust the focal height of the optical assembly 102 relative to the sample 120.

[0057] Next, it should be noted that one or more steps of Method 300 (e.g., steps 302-306) can be repeated for each of the one or more adjusted focal heights. For example, one or more steps 302-306 can be repeated one or more times based on a predetermined height measurement range (e.g., the distance between the optical assembly and the sample). For example, the steps can be repeated every 0.5 μm to cover a distance of 25 μm on both sides between the front focal plane of the optical assembly and the sample.

[0058] It should be noted that the optical assembly 102 can be configured to perform a continuous scan of the focal length. To keep the effective distance range between the sample and the forward focal plane sufficiently small relative to the depth of field of the optical assembly and to avoid blurring of height information, for example, the distance between the sample and the forward focal plane can be continuously adjusted during the scan, and one or more illumination devices can preferably strobe. One or more imaging devices can preferably operate to acquire an image of the sample while illuminating it with a set of illumination modalities provided by one or more illumination devices.

[0059] Returning to Figure 2A, in step 204, 3D ground truth data can be obtained for each focus stack in the set of multimodality focus stacks (obtained in step 202). For example, the controller 110 of the image processing subsystem 108 can be configured to retrieve the 3D ground truth data for each focus stack from a database stored in memory 114. Alternatively, the controller 110 of the image processing subsystem 108 can be configured to retrieve the 3D ground truth data for each focus stack from a remote storage location (not shown).

[0060] In one embodiment, as shown in Figure 5, the 3D ground truth data may include one or more white light interferometry (WLI) depth maps. For example, the WLI depth map 500 can be acquired using WLI and used as the ground truth for a multimodality focus stack. For example, the WLI depth map 500 can be acquired for the corresponding region of each multimodality focus stack (as shown in Figures 4A to 4B, etc.) and used as the ground truth.

[0061] In step 206, the machine learning algorithm can be trained based on the acquired set of training multimodality focus stacks 101 and the received 3D ground truth data. For example, the controller 110 can be configured to train the machine learning algorithm based on the acquired set of training multimodality focus stacks 101 and the received 3D ground truth data. For example, the controller 110 can be configured to train the machine learning algorithm based on the acquired set of training multimodality focus stacks 101 and the corresponding received WLI depth map.

[0062] The controller 110 can be configured to train machine learning algorithms using any technique known to those skilled in the art, including, but not limited to, supervised learning. For example, in the context of supervised learning, the training images may include a set of multimodality focus stacks used to train the machine learning algorithm. In this regard, the controller 110 can receive the training multimodality focus stacks and ground truth data. Thus, the training focus stacks and ground truth 3D data (e.g., WLI depth maps) can be used as inputs for training the machine learning algorithm.

[0063] It should be further noted that the machine learning algorithms trained in step 206 may include, but are not limited to, any type of machine learning algorithm and / or deep learning technique or algorithm known in the art, including convolutional neural networks (CNNs), generative adversarial networks (GANs), modular neural networks, transformers, etc. In this regard, as will be further detailed herein, the machine learning algorithms may include any algorithm or predictive model configured to generate one or more depth maps of samples.

[0064] Furthermore, it should be noted that the above training can also be performed on an external image processing system that is not part of the optical system 100. Therefore, (as will be described later with respect to step 216,) the trained machine learning algorithm can be provided to the controller 110 of the optical system 100 and used to generate depth maps.

[0065] In step 208, the trained machine learning algorithm can be stored. For example, the controller 110 can be further configured to store the training focus stack, ground truth 3D data, and the trained machine learning algorithm 118 in memory 112.

[0066] Figure 2B shows a process flow diagram 210 illustrating a method for generating one or more depth maps using a trained machine learning algorithm 118 of system 100 according to one or more embodiments of the present disclosure. Note that the process flow diagram 210 can be considered a conceptual flowchart showing the steps performed by / within one or more processors 112 of controller 110.

[0067] In step 212, a set of product multimodality focus stacks can be obtained. For example, the controller 110 may be configured to obtain a set of product multimodality focus stacks of a product sample from the optical assembly 102. As used herein, the terms “product image” or “product focus stack” can be used to refer to an image from which one or more depth maps are generated. Thus, a “product image” can be distinguished from a “training image,” which may be considered an image used as input to train a machine learning algorithm used in the process described.

[0068] It should be noted that any considerations regarding the acquisition of training images (shown in flowchart 300 of Figure 3) can be considered applicable to the acquisition of product images unless otherwise specified herein. Therefore, at least a first product image can be acquired at a first focal length using at least a first illumination angle. Furthermore, at least a second product image can be acquired at the first focal length using at least one additional illumination angle. Optionally, at least one additional image can be acquired at the first focal length using one or more illumination wavelengths. The additional images are then acquired at one or more distances / illumination angles.

[0069] In the optional step 213, additional inputs to the machine learning algorithm may be computed. One or more additional inputs may include, but are not limited to, depth maps obtained from a single modality focus stack using a non-learning-based algorithm, one or more post-processed images, one or more additional parameters, etc. For example, the controller 110 may be configured to compute one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired product multimodality focus stack.

[0070] In step 214, the machine learning algorithm can receive the acquired set of product multimodality focus stacks (and optionally any additional computed inputs / parameters calculated in step 213). For example, the optical assembly 102 can be configured to provide the acquired focus stacks to the image processing subsystem 108.

[0071] In step 216, one or more depth maps can be generated using a machine learning algorithm based on the acquired focus stack 103. For example, the image processing subsystem 108 can be configured to generate one or more depth maps of a product sample based on the acquired set 103 of product multimodality focus stacks via a machine learning algorithm. For example, the machine learning algorithm 118 can be configured to generate one or more depth maps in one or more repair stages of an optical molding system based on the acquired product image.

[0072] While embodiments of this disclosure cover optical inspection and / or molding systems, it is assumed that system 100 may include any optical system known in the art. For example, embodiments of this disclosure may cover optical manufacturing systems.

[0073] Each embodiment of the above-described method is further intended to include any other step of any other method described herein. In addition, each embodiment of the above-described method can be carried out by any of the systems described herein.

[0074] Referring again to Figure 1A, one or more processors 112 may include any processing elements known in the art. In this sense, one or more processors 112 may include any microprocessor-type device configured to execute algorithms and / or instructions (e.g., graphics processing unit (GPU), computer processing unit (CPU), etc.). In embodiments, one or more processors 112 may consist of a desktop computer, a mainframe computer system, a workstation, an image computer, a parallel processor, or any other computer system (e.g., a network computer) configured to run a program configured to operate system 100, as described throughout this disclosure. It is further recognized that the term “processor” can be broadly defined to include any device having one or more processing elements that execute program instructions from a non-temporary memory medium 114. Accordingly, the above description should be interpreted as merely an example and not a limitation to the invention.

[0075] The memory medium 114 may include any storage medium known in the art that is suitable for storing program instructions and data executable by one or more associated processors 112. In a non-limiting example, the memory medium 114 may include a non-temporary memory medium. In a further non-limiting example, the memory medium 114 may include, but is not limited to, read-only memory, random-access memory, magnetic or optical memory devices (e.g., disks), magnetic tape, solid-state drives, etc. It should be further noted that the memory 114 may be housed within a common controller housing having one or more processors 112. In an alternative embodiment, the memory 114 may be located remotely from the physical location of one or more processors 112 and the controller. For example, one or more processors 112 of the controller may have access to remote memory (e.g., a server) accessible via a network (e.g., the Internet, an intranet, etc.).

[0076] Those skilled in the art will recognize that the components, actions, devices, objects, and accompanying discussions described herein are used as examples for conceptual clarity, and that various structural modifications are intended. Therefore, when used herein, the specific examples and accompanying discussions described are intended to represent their more general class. In general, the use of any particular example is intended to represent its class, and the exclusion of specific components, actions, devices, and objects should not be interpreted as limitation.

[0077] With regard to the use of substantially any plural and / or singular terms herein, those skilled in the art can convert from plural to singular and / or singular to plural as appropriate to the context and / or use. Various singular / plural substitutions are not explicitly stated herein for the sake of clarity.

[0078] The subject matter described herein refers to different components that are included in or connected to other components. It should be understood that the architectures described herein are merely examples, and in practice, numerous other configurations can be implemented to achieve the same function. Conceptually, any arrangement of components that achieve the same function is effectively “associated” in such a way that the desired function is achieved. Therefore, two components combined herein to achieve a particular function can be recognized as “associated” with each other, independently of the architecture or intermediate components, in such a way that the desired function is achieved. Similarly, any two components thus associated can also be considered “connected” or “joined” to realize a desired function, and any two components that can be associated in such a way can also be considered “joinable” with each other to achieve a desired function. Specific examples of joinable components include, but are not limited to, physically joinable and / or physically interacting components and / or wirelessly interactable and / or wirelessly interacting components and / or logically interacting and / or logically interactable components.

[0079] Furthermore, it should be understood that the present invention is defined by the appended claims. In general, it will be understood by those skilled in the art that the terms used herein and especially in the appended claims (e.g., in the text of the appended claims) are intended to be "open" terms (for example, the term "includes" should be interpreted as "includes but not limited to," the term "has" should be interpreted as "has at least," and the term "includes" should be interpreted as "includes but not limited to," etc.). It will further be understood by those skilled in the art that if a specific number of claims to be introduced is intended, such intention will be explicitly stated in that claim, and if there is no such statement, such intention does not exist. For example, for the sake of understanding, the following appended claims may include introducing the claims using the introductory phrases "at least one" and "one or more." However, the use of such phrases should not be interpreted as meaning that the introduction of a claim by the indefinite article "a" or "an" limits any particular claim containing such introduced claim to an invention containing only one such claim. The same applies to the use of clear articles used to introduce the description of a claim, even when the same claim includes introductory phrases such as "one or more" or "at least one" and indefinite articles such as "a" or "an" (for example, "a" and / or "an" should typically be interpreted as meaning "at least one" or "one or more"). Furthermore, even when the specific number of items described in the introduced claim is explicitly listed, a person skilled in the art will recognize that such a description should typically be interpreted as meaning at least the number listed (for example, a bare list of "two lists" without other modifiers typically means at least two lists, or two or more lists).Furthermore, in cases where a conventional expression similar to "at least one of A, B, and C, etc." is used, such configurations are generally intended in a way that a person skilled in the art would understand the conventional expression to be (for example, "a system having at least one of A, B, and C" is not limited to but includes systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or systems having both A, B, and C). In cases where a conventional expression similar to "at least one of A, B, or C, etc." is used, such configurations are generally intended in a way that a person skilled in the art would understand the conventional expression to be (for example, "a system having at least one of A, B, or C" is not limited to but includes systems having only A, only B, only C, both A and B, both A and C, and / or systems having both A, B, and C). It will be further understood by those skilled in the art that virtually any separate word and / or phrase presenting two or more alternative terms should be understood, wherever it may be in the description, claims, or drawings, as construing the possibility of including one of those terms, either of those terms, or both of those terms. For example, the phrase “A or B” would be understood to include the possibilities of “A” or “B” or “A and B.”

[0080] The present disclosure and its numerous accompanying advantages are to be understood from the foregoing description, and it will be clear that various modifications can be made to the form, structure, and arrangement of the components without departing from the disclosed subject matter or sacrificing any of the advantages of the disclosed materials. The described forms are merely illustrative, and the following claims are intended to imply and encompass the aforementioned modifications. Furthermore, it should be understood that the present invention is defined by the appended claims.

Claims

1. An optical system, An optical assembly configured to illuminate one or more portions of one or more samples using two or more illumination modalities, wherein the two or more illumination modalities include at least a first illumination modality and a second illumination modality, the first illumination modality includes a first set of illumination angles, the second illumination modality includes a second set of illumination angles, the second set of illumination angles differs at least partially from the first set of illumination angles. The optical assembly is configured to acquire a multimodality focus stack, the multimodality focus stack comprising a plurality of images acquired at two or more distances between one or more samples and the forward focal plane, wherein at least a first image of the plurality of images is acquired using the first illumination modality, and at least additional images are acquired using the second illumination modality, An image processing subsystem communicatively coupled to the optical assembly, the image processing subsystem includes one or more processors configured to execute a set of program instructions stored in memory, the set of program instructions is configured to execute the one or more processors Receiving multiple training images, wherein the multiple training images include multiple training multimodality focus stacks. For each of the aforementioned training multimodality focus stacks, ground truth 3D data is received. Training a machine learning algorithm based on the aforementioned multiple training images and the received ground truth 3D data, Receiving the multimodality focus stack of the sample from the optical assembly, Using the trained machine learning algorithm and the multimodality focus stack of the received samples, a depth map of the samples is generated. A system characterized by being configured to perform the following.

2. A system according to claim 1, wherein the first illumination modality includes a first spectral distribution, the second illumination modality includes a second spectral distribution, and the second spectral distribution is at least partially different from the first spectral distribution.

3. The system according to claim 1, wherein the optical assembly is One or more lighting devices, wherein each of the one or more lighting devices is configured to illuminate one or more samples at multiple lighting angles, At least one imaging device positioned in a fixed direction relative to one or more samples, wherein the at least one imaging device is configured to acquire at least a first image of the one or more samples at a first illumination angle among the plurality of illumination angles, and to acquire at least a second image of the one or more samples at a second illumination angle among the plurality of illumination angles, A system characterized by comprising the following features.

4. The system according to claim 3, The system further comprises one or more actuator assemblies configured to adjust at least the distance between one or more samples and the forward focal plane of the optical assembly, wherein the at least one imaging device can be configured to acquire one or more images at two or more distances between the one or more samples and the forward focal plane.

5. The system according to claim 4, wherein the distance between one or more samples and the forward focal plane of the optical assembly is adjusted, A system characterized by comprising at least one of adjusting the position of the optical assembly or adjusting the position of one or more samples.

6. The system according to claim 3, wherein one or more processors are further configured to adjust the position of the at least one imaging device, or to adjust one or more optical properties of the at least one imaging device or the optical assembly.

7. The system according to claim 1, wherein the one or more processors are A system further configured to receive one or more additional inputs to the machine learning algorithm, wherein the one or more additional inputs include one of the following: one or more depth maps obtained from the multimodality focus stack using a non-learning-based algorithm, one or more post-processed images from the multimodality focus stack, or one or more additional parameters.

8. The system according to claim 1, wherein the ground truth data includes one or more depth maps obtained by white light interferometry.

9. The system according to claim 1, wherein the machine learning algorithm is A system characterized by comprising at least one of the following: a convolutional neural network, a generative adversarial network, a modular neural network, or a transformer.

10. The system according to claim 1, A system further comprising one or more pre-fabricated reference objects, wherein the one or more pre-fabricated reference objects include one or more three-dimensional structures having one or more known depth maps, and the one or more known depth maps are configured to be used for at least one of the following: verification of the optical assembly, calibration of the optical assembly, correction of the multimodality focus stack, or retraining of the machine learning algorithm.

11. The system according to claim 1, wherein the sample includes a printed circuit board.

12. The system according to claim 1, wherein the optical system includes an automated optical inspection system.

13. The system according to claim 1, wherein the optical system includes an automated optical molding system.

14. An image processing system, wherein the image processing system is Includes one or more processors configured to execute a set of program instructions stored in memory, wherein the set of program instructions is configured on the one or more processors Receiving multiple training images of one or more portions of one or more samples, wherein the multiple training images include multiple training multimodality focus stacks. For each of the aforementioned training multimodality focus stacks, ground truth 3D data is received. Training a machine learning algorithm based on the aforementioned multiple training images and the received ground truth 3D data, Receiving a multimodality focus stack of a sample from an optical assembly, wherein the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modalities, the two or more illumination modalities include at least a first illumination modality and a second illumination modality, the first illumination modality includes a first set of illumination angles, the second illumination modality includes a second set of illumination angles, at least the second set of illumination angles differs at least partially from the first set of illumination angles, the multimodality focus stack includes a plurality of images acquired at two or more distances between the sample and the forward focal plane, at least a first of the plurality of images acquired using the first illumination modality, and at least additional images acquired using the second illumination modality, Using the trained machine learning algorithm and the received multimodality focus stack, generate a depth map of the sample. A system characterized by being configured to perform the following action.

15. A system according to claim 14, wherein the first illumination modality includes a first spectral distribution, the second illumination modality includes a second spectral distribution, and the second spectral distribution is at least partially different from the first spectral distribution.

16. The system according to claim 14, The system further comprises an optical assembly configured to acquire the multimodality focus stack.

17. The system according to claim 16, wherein the optical assembly is One or more lighting devices, wherein the one or more lighting devices are configured to illuminate the sample at multiple lighting angles, At least one imaging device positioned in a fixed direction relative to the sample, wherein the at least one imaging device is configured to acquire at least a first image of the sample at a first illumination angle among the plurality of illumination angles, and to acquire at least a second image of the sample at a second illumination angle among the plurality of illumination angles, A system characterized by comprising the following features.

18. The system according to claim 17, wherein the optical assembly is The system further comprises one or more actuator assemblies configured to adjust at least the distance between the sample and the forward focal plane of the optical assembly, wherein the at least one imaging device can be configured to acquire two or more images at the two or more distances between the sample and the forward focal plane.

19. The system according to claim 18, wherein the distance between the sample and the forward focal plane of the optical assembly is adjusted, A system characterized by including at least one of adjusting the position of the optical assembly or adjusting the position of the sample.

20. A system according to claim 17, wherein one or more processors are further configured to adjust the position of the at least one imaging device or to adjust one or more optical properties of the at least one imaging device or the optical assembly.

21. The system according to claim 14, wherein the one or more processors are A system further configured to receive one or more additional inputs to the machine learning algorithm, wherein the one or more additional inputs include one of the following: one or more depth maps obtained from the multimodality focus stack using a non-learning-based algorithm, one or more post-processed images from the multimodality focus stack, or one or more additional parameters.

22. The system according to claim 14, wherein the ground truth data includes one or more depth maps obtained by white light interferometry.

23. The system according to claim 14, wherein the machine learning algorithm is A system characterized by comprising at least one of the following: a convolutional neural network, a generative adversarial network, a modular neural network, or a transformer.

24. The system according to claim 14, wherein the sample includes a printed circuit board.

25. It is a method, Receiving multiple training images of one or more portions of one or more samples, wherein the multiple training images include multiple training multimodality focus stacks. For each of the aforementioned training multimodality focus stacks, ground truth 3D data is received. Training a machine learning algorithm based on the aforementioned multiple training images and the received ground truth 3D data, Receiving a multimodality focus stack of one or more samples from an optical assembly, wherein the optical assembly is configured to illuminate the one or more samples using two or more illumination modalities, the two or more illumination modalities comprising at least a first illumination modality and a second illumination modality, the first illumination modality comprising a first set of illumination angles, the second illumination modality comprising a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodality focus stack comprising a plurality of images acquired at two or more distances between the sample and the forward focal plane, at least a first image of the plurality of images being acquired using the first illumination modality, and at least additional images being acquired using the second illumination modality, Using the trained machine learning algorithm and the received multimodality focus stack, generate depth maps of one or more samples. A method characterized by including the following.

Images