System and method for three-dimensional imaging of sample using machine learning algorithm

Abstract

A method for 3D imaging of a sample using a machine learning algorithm is disclosed. The method uses a multi-modal focus stack consisting of a plurality of images acquired at two or more distances between the sample and a front focal plane, where at least one image is acquired using a first modality and an additional image is acquired using an additional modality. The modalities may have different illumination angles and optionally have different spectral distributions. The method may include receiving a training image of a sample including a plurality of training focus stacks and live 3D data (depth map) for each training focus stack; and training a machine learning algorithm based on the training image and the live data. The method then receives a product image of the sample, the product image including a stack of focuses from the optical assembly; and generating a 3D depth map.

Description

Technical Field

[0001] The present invention generally relates to three-dimensional (3D) imaging, and more specifically, to a system and method for 3D imaging of a sample based on multimodal focal stacking using a machine learning algorithm, wherein the machine learning algorithm is trained using the set of multimodal focal stackings. Background Art

[0002] As the demand for electronic circuits with increasingly smaller device features continues to increase, the need for improved 3D imaging technology continues to grow. Optical inspection systems may require 3D imaging to measure circuit features (such as conductors, drill bits, and the like) in three dimensions and identify or verify "3D defects" (such as undercuts, i.e., small areas of thin copper). Optical reshaping systems can be used in printed circuit board manufacturing processes to increase yield and eliminate debris. Such optical reshaping systems can utilize 3D imaging when repairing defects. For example, 3D imaging can be used during the repair process when copper is deposited onto the surface of the printed circuit board. By way of another example, 3D imaging can be used to measure sample damage after the repair process to ensure that when removing excess copper (e.g., using laser ablation), the laser does not inadvertently penetrate the laminate.

[0003] Current focus-based 3D printing technologies are generally cost-effective but offer limited spatial accuracy and precision compared to imaging resolution, as determining the focus quality of individual pixels requires calculations based on the neighborhood of each pixel. Other 3D imaging technologies (e.g., based on interferometry) are more expensive and some may require additional optical components, which complicates the system and may reduce system throughput.

[0004] Therefore, it would be advantageous to provide a system and method that remedies the shortcomings of the above-mentioned methods. Summary of the Invention

[0005] According to one or more embodiments of the present disclosure, an optical system is disclosed. In one embodiment, the optical system includes an optical assembly configured to illuminate one or more portions of a sample using two or more illumination modalities, including at least a first illumination modality and a second illumination modality, wherein the first illumination modality includes a first set of illumination angles and the second illumination modality includes a second set of illumination angles, wherein at least the second set of illumination angles differs at least in part from the first set of illumination angles. In one embodiment, the optical assembly is configured to acquire a multimodal focal stack, the multimodal focal stack comprising a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images is acquired using the first illumination modality and at least an additional image is acquired using the second illumination modality. In one embodiment, the optical system further includes an image processing subsystem communicatively coupled to the optical assembly, wherein the image processing subsystem includes one or more processors configured to execute a set of program instructions stored in a memory. In one embodiment, the set of program instructions is configured to cause the one or more processors to receive a plurality of training images, wherein the one or more training images include a plurality of training multimodal focal stacks. In an embodiment, the set of program instructions is configured to cause the one or more processors to receive three-dimensional live data for each of the plurality of training multimodal focal stackings. In an embodiment, the set of program instructions is configured to cause the one or more processors to train a machine learning algorithm based on the plurality of training images and the received three-dimensional live data. In an embodiment, the set of program instructions is configured to cause the one or more processors to receive the multimodal focal stacking of the sample from the optical assembly. In an embodiment, the set of program instructions is configured to cause the one or more processors to generate a depth map of the sample using the trained machine learning algorithm and the received multimodal focal stackings.

[0006] According to one or more embodiments of the present disclosure, an image processing system is disclosed. In one embodiment, the system includes one or more processors configured to execute a set of program instructions stored in a memory. In one embodiment, the set of program instructions is configured to cause the one or more processors to receive a plurality of training images, wherein the one or more training images include a plurality of training multimodal focal stacks. In one embodiment, the set of program instructions is configured to cause the one or more processors to receive three-dimensional live data for each of the plurality of training multimodal focal stacks. In one embodiment, the set of program instructions is configured to cause the one or more processors to train a machine learning algorithm based on the plurality of training images and the received three-dimensional live data. In an embodiment, the set of program instructions is configured to cause the one or more processors to receive a multimodal focal stack of a sample from an optical assembly, wherein the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modalities including at least a first illumination modality and a second illumination modality, wherein the first illumination modality includes a first set of illumination angles and the second illumination modality includes a second set of illumination angles, wherein at least the second set of illumination angles is at least partially different from the first set of illumination angles, wherein the multimodal focal stack includes a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images was acquired using the first illumination modality and at least an additional image was acquired using the second illumination modality. In an embodiment, the set of program instructions is configured to cause the one or more processors to generate a depth map of the sample using the trained machine learning algorithm and the received multimodal focal stack.

[0007] According to one or more embodiments of the present disclosure, a method is disclosed. In one embodiment, the method includes receiving a plurality of training images, wherein the one or more training images include a plurality of training multimodal focal stacks. In one embodiment, the method includes receiving three-dimensional live data for each of the plurality of training multimodal focal stacks. In one embodiment, the method includes training a machine learning algorithm based on the plurality of training images and the received live three-dimensional data. In one embodiment, the method includes receiving a multimodal focal stack of a sample from an optical assembly, wherein the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modalities including at least a first illumination modality and a second illumination modality, wherein the first illumination modality includes a first set of illumination angles and the second illumination modality includes a second set of illumination angles, wherein at least the second set of illumination angles differs at least in part from the first set of illumination angles, wherein the multimodal focal stack includes a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images was acquired using the first illumination modality and at least an additional image was acquired using the second illumination modality. In embodiments, the method comprises generating a depth map of the sample using the trained machine learning algorithm and the received multimodal focal stack.

[0008] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and do not necessarily limit the invention as claimed. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description, serve to explain the principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Those skilled in the art may better understand the numerous advantages of the present disclosure with reference to the accompanying drawings.

[0010] Figure 1A A block diagram illustrating an optical system according to one or more embodiments of the present disclosure.

[0011] Figure 1B A simplified schematic top view illustrating the optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0012] Figure 1C A simplified schematic side view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0013] Figure 1D A simplified schematic top view illustrating the optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0014] Figure 1E A simplified schematic side view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0015] Figure 2A A process flow diagram is illustrated depicting a method of training a machine learning algorithm of a system according to one or more embodiments of the present disclosure.

[0016] Figure 2B A process flow diagram is illustrated depicting a method of generating one or more depth maps using a trained machine learning algorithm of the system, according to one or more embodiments of the present disclosure.

[0017] Figure 3 A process flow diagram is illustrated depicting a method of acquiring one or more focus stacks using one or more components of a system according to one or more embodiments of the present disclosure.

[0018] Figure 4A and 4B Illustrated are focus stacked images of a sample according to one or more embodiments of the present disclosure.

[0019] Figure 5 A white light interferometry depth map of a sample is illustrated according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] The present disclosure has been particularly shown and described with respect to certain embodiments and specific features thereof. The embodiments described herein are to be considered illustrative rather than restrictive. It will be readily apparent to those skilled in the art that various changes and modifications in form and details may be made without departing from the spirit and scope of the present disclosure. Reference will now be made in detail to the disclosed subject matter, which is illustrated in the accompanying drawings.

[0021] As the demand for electronic circuits with increasingly smaller features and reduced thickness continues to increase, the need for improved 3D imaging technology is also growing. Optical inspection systems can be used for process control and yield management in printed circuit board manufacturing. Such systems can utilize 3D imaging to identify and / or verify defects and measure circuit features in 3D. Optical reshaping systems can be used to eliminate debris and increase printed circuit board yield. Such systems can utilize 3D imaging when repairing defects. Optical reshaping systems are generally discussed in U.S. Patent No. 8,290,239, issued on October 16, 2012, and U.S. Patent Publication No. 2013 / 0037526, published on February 14, 2013, both of which are incorporated herein by reference in their entirety.

[0022] Current 3D imaging techniques using depth of focus (DFF) capture a single image at each object distance from the focal plane of the optical system. From this set of images at different object distances, the DFF algorithm determines, for each imaged point in the object, where the point's neighborhood is in optimal focus. However, one drawback of "standard" DFF is that it offers limited spatial accuracy and precision compared to the imaging resolution, since the determination of the focus quality of an individual pixel requires calculations based on each pixel's neighborhood.

[0023] Other 3D imaging techniques, such as white light interferometry (WLI), multi-view stereo, and triangulation or phase shifting methods, offer higher spatial accuracy and precision, but are expensive and require additional optical components. Furthermore, other imaging techniques, such as photometric stereo (PS), are typically computationally intensive (and therefore time-consuming) and require a detailed and accurate reflectance model of the material comprising the imaged object to obtain accurate surface reconstruction.

[0024] Therefore, it would be advantageous to provide a system and method that remedies the shortcomings of the above-mentioned methods.

[0025] Embodiments of the present disclosure relate to a system and method for generating one or more depth maps of one or more portions of a sample by capturing a set of images at a set of distances between the sample and a front focal plane at at least two different illumination angles and using a machine learning algorithm to obtain a depth map from this data. For example, the system may include an optical assembly configured to acquire a set of images at a set of distances between the sample and the front focal plane at at least two different illumination angles. As previously described herein, unlike "standard" DFF, in which a single image is captured at each distance from the object, the system of the present disclosure can acquire multimodal focal stacks of the sample under different illumination modalities. In this regard, the system can acquire reflectance information for each pixel on the sample under different illumination directions. This information can be used to increase the spatial resolution, precision, and accuracy of the obtained depth map. For the purposes of this disclosure, a "depth map" and its variations may be defined as data consisting of depth values (z) as a function of position (x, y) in a vertical plane (e.g., at each imaging pixel).

[0026] As previously described herein, unlike PS, which requires a reflectance model, the systems and methods of the present disclosure do not rely on a reflectance model of the material, but instead use machine learning methods to train the system on the type of sample to be imaged. Furthermore, the use of machine learning methods can reduce the number of images required for 3D reconstruction when generating one or more depth maps.

[0027] In addition, the system may include an image processing subsystem configured to use a machine learning algorithm trained using focal stacking with multiple illuminations and live 3D data (e.g., obtained using WLI or other imaging techniques). Once trained, the machine learning algorithm may be configured to generate one or more depth maps of one or more portions of the sample based on the collected images from the optical assembly. In some embodiments, the machine learning algorithm may be configured to generate one or more depth maps of one or more portions of the sample at a depth resolution of between approximately 0.1 micrometers (μm) and 2 micrometers.

[0028] Figure 1A A simplified block diagram illustrating an optical system 100 according to one or more embodiments of the present disclosure is shown. Specifically, Figure 1A A system 100 is described for generating a depth map of one or more portions of a sample using machine learning techniques.

[0029] In an embodiment, system 100 includes an optical assembly 102. The optical assembly may include, but is not limited to, one or more imaging devices 104, one or more illumination devices 106, one or more controllers including one or more processors and memory, a chassis, and the like.

[0030] The system 100 may additionally include an image processing subsystem 108 communicatively coupled to the optical assembly 102. The image processing subsystem 108 may include, but is not limited to, one or more controllers including one or more processors 112 and a memory 114.

[0031] Optical assembly 102 and / or image processing subsystem 108 may be communicatively coupled to user interface device 116 .

[0032] Figures 1B to 1E A simplified schematic diagram illustrating an optical assembly 102 of an optical system 100 according to one or more embodiments of the present disclosure is shown. Specifically, Figure 1B A top view of a single ring of illumination devices 106 of optical assembly 102 is depicted in accordance with one or more embodiments of the present disclosure. Specifically, Figure 1C A side view of a single ring optical assembly 102 of an optical system 100 is depicted in accordance with one or more embodiments of the present disclosure. Specifically, Figure 1D A top view of a dual-ring optical assembly 102 of an optical system 100 is depicted in accordance with one or more embodiments of the present disclosure. Specifically, Figure 1E Depicted is a side view of a dual-ring optical assembly 102 of an optical system 100 , in accordance with one or more embodiments of the present disclosure.

[0033] In one embodiment, optical assembly 102 is configured to acquire one or more multimodal focal stacks of sample 120. For example, optical assembly 102 may be configured to illuminate sample 120 using two or more illumination modalities. For example, the two or more illumination modalities may include at least a first illumination modality and a second illumination modality, wherein the first modality may include a first set of illumination angles and the second modality includes a second set of illumination angles. By way of another example, the first modality may include a first spectral distribution and a second spectral distribution, wherein the second spectral distribution may be at least partially different from the first. In this regard, the multimodal focal stack may include a plurality of images acquired at two or more distances between the front focal plane and sample 120, wherein at least one image is acquired using the first modality and an additional image is acquired using the second modality, wherein the first modality and the second modality are at least partially different.

[0034] Optical assembly 102 can be positioned relative to sample 120 mounted on stage assembly 122. In embodiments, the sample can be homogenous, formed from two or more materials, wherein a first material is different from at least a second material. For example, the homogenous sample can include a printed circuit board (PCB). The two or more materials can include, but are not limited to, copper, laminates, or the like.

[0035] One or more imaging devices 104 of optical assembly 102 may be configured to acquire one or more images of sample 120 mounted on stage assembly 122. For purposes of this disclosure, the term "one or more imaging devices 104" refers to one or more imaging devices including an imaging sensor (e.g., a camera) and one or more optical elements (e.g., one or more lenses), unless otherwise indicated herein.

[0036] The one or more imaging devices 104 may include any type of imaging device suitable for acquiring one or more two-dimensional (2D) images of the sample 120. For example, the one or more imaging devices 104 may include one or more cameras configured for image acquisition. For example, the one or more imaging devices 104 may include one or more high-speed cameras configured for rapid image acquisition, e.g., greater than a standard video rate of 24 to 25 frames per second (e.g., 90 frames per second).

[0037] In an embodiment, one or more imaging devices 104 share a common axis with sample 120 mounted on stage assembly 122 when acquiring one or more images. For example, one or more imaging devices 104 may be positioned over at least a portion of sample 120 mounted on stage assembly 122 when acquiring one or more images.

[0038] In one embodiment, the optical assembly 102 further includes one or more actuation assemblies 107. The one or more actuation assemblies 107 can be configured to adjust the focal length of the optical assembly 102 relative to the sample 120 (i.e., the distance between the sample and the front focal plane). For example, the one or more actuation assemblies can be configured to adjust the focal length of the one or more imaging devices 104 by actuating the optical assembly 102 at least along the z-axis (common axis). By way of another example, the one or more actuation assemblies 107 can be configured to adjust the focal length by actuating the sample 120 at least along the z-axis (common axis). For example, the one or more actuation assemblies 107 can be configured to adjust the focal length of the one or more imaging devices 104 so that the focal length between the one or more imaging devices 104 and the sample 120 can be adjusted. In this regard, the one or more imaging devices 104 can be configured to acquire two or more images of the sample at a set of two or more focal lengths.

[0039] It should be noted that the one or more actuation assemblies 107 may include any type of actuation device suitable for adjusting at least the focal length of the optical assembly 102 relative to the sample 120. For example, the one or more actuation assemblies 107 may include, but are not limited to, one or more linear actuator devices. Furthermore, it should be noted herein that the system 100 may be configured to adjust the focal length of the optical assembly 102 relative to the sample 120 via any mechanism. Thus, the above discussion should not be construed as limiting the scope of the present disclosure. For example, the optical assembly 102 may include a variable focus lens configured to adjust the focal length of the optical assembly 102 relative to the sample 120. By way of another example, one or more components of the optical assembly 102 may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120. For example, one or more imaging sensors may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120.

[0040] In one embodiment, the one or more imaging devices 104 may be configured to acquire two or more images of the sample 120 at a set of distances when one or more actuation assemblies are activated. For example, two or more images of the sample 120 may be acquired during movement of the optical assembly 102 through the one or more actuation assemblies.

[0041] In one embodiment, the one or more imaging devices 104 may be configured to acquire two or more images of the sample 120 at a set of focal lengths when the one or more actuation assemblies are located at a fixed focal length. For example, the two or more images of the sample 120 may be acquired after the one or more actuation assemblies 107 have adjusted the focus of the one or more imaging devices 104 and are located at one or more predetermined focal lengths (or distances from the sample).

[0042] In one embodiment, the one or more illuminators 106 of the optical assembly 102 may be configured to generate one or more illumination beams 105. The one or more illuminators 106 may include any suitable illuminator for generating the one or more illumination beams 105. For example, the one or more illuminators 106 may include one or more flash illuminators. For example, the one or more flash illuminators may include, but are not limited to, one or more light emitting diodes (LEDs) (e.g., red LEDs) or the like. In this regard, the one or more flash illuminators may be configured to generate illumination for a short period of time to avoid blurring caused by movement / vibration (e.g., along a common axis or in a plane perpendicular thereto). This enables rapid acquisition by capturing images as the distance between the sample and the front focal plane changes (e.g., by moving the imaging device 104).

[0043] The optical assembly 102 may further include one or more optical elements 126 configured to direct one or more illumination beams to the surface of the sample 120. For example, the optical assembly 102 may include, but is not limited to, one or more mirrors, one or more lenses, one or more polarizers, one or more beam splitters, one or more optical fibers, and the like. It should be noted that Figures 1B to 1E This is for illustrative purposes only and should not be construed as limiting the scope of the present disclosure.For example, the optical assembly 102 can include any type of optical elements and any configuration of optical elements.

[0044] In embodiments, the one or more illuminators 106 may include a set of illuminators 106 positioned at a set of illumination angles relative to the sample 120. For example, the one or more illuminators may include a set of illuminators 106 including one or more high-altitude illuminators (e.g., positioned at a high-altitude angle relative to the stage / sample) and one or more low-altitude illuminators (e.g., positioned at a lower altitude angle relative to the stage / sample than the high-altitude illuminators).

[0045] It should be noted that the optical assembly 120 may include any number and configuration of illumination devices 106 suitable for illuminating the sample 120 at a particular illumination angle (eg, at several non-colinear illumination directions). Figures 1B to 1E It is for illustrative purposes only and should not be construed as limiting the scope of the present disclosure.

[0046] Each set of illumination devices 106 can be positioned proximate to a sample 120 mounted on a stage assembly 122. For example, each set of illumination devices 106a-106d can be positioned adjacent to (or proximate to) a sample 120 mounted on a stage assembly.

[0047] refer to Figures 1B to 1E , the lighting devices 106 may be evenly distributed in azimuth so that the lighting devices 106 can substantially evenly cover the range of illumination directions of the sample 120. For example, Figures 1B to 1CAs shown in FIG, the lighting devices 106 can be arranged in a single ring at equal angular intervals. In one example, as shown in FIG. Figure 1B As shown in FIG, the set of lighting devices 106 may include six lighting devices separated by 60 degrees. By way of another example, Figures 1D to 1E As shown in FIG, the lighting devices 106 can be evenly distributed in a double ring configuration. In one example, as shown in FIG. Figure 1D As shown in , the set of lighting devices may include ten lighting devices, with a center ring including four lighting devices separated by 90 degrees and an outer ring including six lighting devices separated by 60 degrees.

[0048] although Figure 1B An optical assembly 102 is depicted that includes seven illuminators configured to illuminate a sample at seven different illumination angles, but it should be noted that the optical assembly 102 can include any number and configuration of illuminators configured to illuminate a sample at any azimuthal and / or altitude illumination angle. For example, the set of illuminators can include six illuminators spaced 60 degrees apart in a ring. Furthermore, it should be noted that the optical assembly 102 can be configured to illuminate a sample at one or more of the different illumination angles using any combination of illuminators.

[0049] The system 100 may further include one or more prefabricated reference objects (or calibration targets). For example, the one or more prefabricated reference objects may include one or more fixed reference objects. By way of another example, the one or more prefabricated reference objects may include one or more removable reference objects.

[0050] The one or more reference objects may include one or more three-dimensional (3D) structures having known (or previously measured) depth maps. For example, the one or more known depth maps may be used to verify one or more components of an optical assembly. By another example, the one or more known depth maps may be used to calibrate one or more components of an optical assembly. By another example, the one or more known depth maps may be used to retrain a machine learning algorithm. By another example, the one or more known depth maps may be used to adjust one or more inputs to a machine learning algorithm. For example, the one or more depth maps may be used to correct the distance of each image, which may be adjusted after calibration.

[0051] In an embodiment, the image processing subsystem 108 is configured to generate one or more depth maps of the sample 120 based on a set of multimodal focal stacks (including one or more 2D images) acquired from the optical assembly 102. For example, the image processing subsystem 108 may be configured to use a machine learning algorithm 118 stored in the memory 114 of the controller 110 of the image processing subsystem 108 to generate one or more depth maps of the sample 120 corresponding to the two or more 2D images of the focal stacks, as will be discussed further herein.

[0052] Figure 2AA process flow diagram 200 is illustrated depicting a method for training the machine learning algorithm 118 of the system 100 according to one or more embodiments of the present disclosure. It should be noted that the process flow diagram 200 can be viewed as a conceptual flow diagram illustrating steps performed by / within the one or more processors 112 of the controller 110.

[0053] In step 202 , a set of training multimodal focus stacks may be acquired. For example, the controller 110 of the image processing subsystem 108 may be configured to acquire a set of multimodal focus stacks 101 from the optical assembly 102 for training the machine learning algorithm 118 .

[0054] In embodiments, each multimodal focal stack 101 may include one or more acquired images at each of a set of distances between the sample and the front focal plane and having two or more illumination angles in at least some of those distances, or having different illumination angles at different distances. For example, the one or more imaging devices 104 of the optical assembly 102 may be configured to acquire one or more images at a set of distances between the sample and the front focal plane while one or more of the illumination devices 106 illuminate the sample 120 at one or more different illumination angles.

[0055] In optional step 203, additional inputs to the machine learning algorithm may be calculated. The one or more additional inputs may include, but are not limited to, a depth map obtained from a single-modality focal stack using a non-learning-based algorithm, one or more processed images, one or more additional parameters, and the like. For example, the controller 110 may be configured to calculate the one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired multi-modal focal stack.

[0056] Figure 3 A process flow diagram 300 is illustrated depicting a method for acquiring one or more focus stacks using one or more components of the system 100 in accordance with one or more embodiments of the present disclosure. It should be noted that the flow diagram 300 may be viewed as a conceptual flow diagram illustrating steps performed by / within the one or more processors 112 of the controller 110.

[0057] In step 302, a first image may be acquired having a first illumination modality with a first illumination angle (or more generally, a particular range of illumination angles, which may partially overlap). For example, one or more imaging devices 104 may be configured to acquire the first image at a first distance between the sample and the front focal plane when at least a first illumination device illuminates the sample at at least the first illumination angle. For the purposes of this disclosure, an "illumination modality" and variations thereof may be defined as a set of one or more illumination angles having the same or different spectral distributions for each angle.

[0058] In optional step 303, the focus may be adjusted. For example, when acquiring images while in motion, the focus may be adjusted between acquiring additional images in additional modalities.

[0059] In optional step 304, a second image may be acquired having a second illumination modality with a second illumination angle. For example, the one or more imaging devices 104 may be configured to acquire the second image at a first distance between the sample and the front focal plane when at least the second illumination modality illuminates the substrate at at least the second illumination angle.

[0060] In optional step 305, the focus may be adjusted. For example, when acquiring images while in motion, the focus may be adjusted between acquiring additional images in additional modalities.

[0061] Figure 4A and 4B An example focus stacked image is illustrated according to one or more embodiments of the present disclosure. It should be noted that Figure 4A and 4B The two sets of images 400a-400d and 420a-420d may be part of a single focus stack, which may include additional images taken at additional focal lengths and illumination angles.

[0062] refer to Figure 4A , a first set of focus-stacked images 400 may include a plurality of images 400a through 400d of a sample 120 (e.g., a PCB) captured at a first example focal length. For example, the first set of focus-stacked images may include at least a first image 400a captured at the first example focal length and with a first azimuthal illumination direction 402a, wherein image 400a is focused on a copper trace of the PCB. By another example, the first set of focus-stacked images may include at least a second image 400b captured at the first example focal length and with a second azimuthal illumination direction 402b, wherein image 400b is focused on a copper trace of the PCB. By another example, the first set of focus-stacked images may include at least a third image 400c captured at the first example focal length and with a third azimuthal illumination direction 402c, wherein image 400c is focused on a copper trace of the PCB. By another example, the first set of focus-stacked images may include at least a fourth image 400d captured at the first example focal length and with a fourth azimuthal illumination direction 402d, wherein image 400d is focused on a copper trace of the PCB. refer to Figure 4B, the second set of focus stacked images 420 may include a plurality of images 420a-420d of the sample 120 (e.g., a PCB) captured at a second example focal length (different from the first example focal length). For example, the second set of focus images may include at least a first image 420a captured at the second example focal length and with a first azimuth illumination direction 422a, wherein image 420a is focused on the stack of layers of the PCB. By another example, the second set of focus images may include at least a second image 420b captured at a second example focal length and with a second azimuth illumination direction 422b, wherein image 420b is focused on the stack of layers of the PCB. By another example, the second set of focus images may include at least a third image 420c captured at the second example focal length and with a third azimuth illumination direction 422c, wherein image 420c is focused on the stack of layers of the PCB. By way of another example, the second set of focus images may include at least a fourth image 420d taken at a second example focal length and with a fourth azimuthal lighting direction 422d, where image 420d is focused on a stackup of layers of the PCB.

[0063] Return Reference Figure 3 In optional step 306, one or more additional images may be acquired using one or more additional illumination modalities. For example, one or more imaging devices 104 may be configured to acquire a third image at a first distance between the sample and the front focal plane when at least a third illumination device illuminates the sample at at least a third illumination angle. By way of another example, one or more imaging devices 104 may be configured to acquire a fourth image at a first distance between the sample and the front focal plane when at least a fourth illumination device illuminates the sample at at least a fourth illumination angle. By way of another example, one or more imaging devices 104 may be configured to acquire the one or more images at the first distance between the sample and the front focal plane using one or more additional illumination spectra (e.g., different from the illumination spectra used in at least one of steps 302-306). It should be noted that the illumination spectra may induce fluorescence in at least one of the two or more materials of sample 120. For example, if sample 120 is a PCB, the illumination spectra may induce fluorescence in the PCB laminate material.

[0064] In step 308 , the distance between the sample and the front focal plane of the optical assembly relative to the sample can be adjusted. For example, the actuation assembly can be configured to adjust the focal height of the optical assembly 102 relative to the sample 120 .

[0065] It should be noted that one or more steps of method 300 (e.g., steps 302-306) may then be repeated at each of the one or more adjusted focus heights. For example, one or more of steps 302-306 may be repeated one or more times based on a predetermined height measurement range (e.g., the distance between the optical assembly and the sample). For example, the steps may be repeated every 0.5 μm to cover a distance of 25 μm on each side between the front focal plane of the optical assembly and the sample.

[0066] It should be noted that the optical assembly 102 can be configured to perform a continuous scan of the focal length. For example, during the scan, the distance between the sample and the front focal plane can be continuously adjusted and one or more illumination devices can preferably be flashed to illuminate the sample, while keeping the effective distance range between the sample and the front focal plane sufficiently small relative to the depth of field of the optical assembly during each acquisition to avoid blurring of height information. The one or more imaging devices are preferably operated to acquire images of the sample illuminated by a set of illumination modalities provided by the one or more illumination devices.

[0067] Return Reference Figure 2A , in step 204, 3D live data for each focus stack of the set of multimodal focus stacks (acquired in step 202) may be acquired. For example, the controller 110 of the image processing subsystem 108 may be configured to retrieve the 3D live data for each focus stack from a database stored in the memory 114. By way of another example, the controller 110 of the image processing subsystem 108 may be configured to retrieve the 3D live data for each focus stack from a remote storage location (not shown).

[0068] In an embodiment, Figure 5 As shown in FIG, 3D live data may include one or more white light interferometry (WLI) depth maps. For example, a WLI depth map 500 may be obtained using WLI and may be used as live for multimodal focus stacking. For example, a WLI depth map 500 may be obtained for each multimodal focus stack (e.g., Figures 4A to 4B The corresponding area on (shown in ) is obtained and used as the live view.

[0069] In step 206, a machine learning algorithm may be trained based on a set of acquired training multimodal focus stacks 101 and the received 3D live data. For example, the controller 110 may be configured to train the machine learning algorithm based on a set of acquired training multimodal focus stacks 101 and the received 3D live data. For example, the controller 110 may be configured to train the machine learning algorithm based on a set of acquired training multimodal focus stacks 101 and the received 3D live data.

[0070] The controller 110 may be configured to train the machine learning algorithm via any technique known in the art, including, but not limited to, supervised learning and the like. For example, in the context of supervised learning, the training images may comprise a set of multimodal focus stacks used to train the machine learning algorithm. In this regard, the controller 110 may receive the training multimodal focus stacks and live data. Thus, the training focus stacks and live 3D data (e.g., a WLI depth map) may be used as input to train the machine learning algorithm.

[0071] It should be further noted herein that the machine learning algorithm trained in step 206 may include any type of machine learning algorithm and / or deep learning technique or algorithm known in the art, including but not limited to convolutional neural networks (CNNs), generative adversarial networks (GANs), modular neural networks, transformers, and the like. In this regard, the machine learning algorithm may include any algorithm or predictive model configured to generate one or more depth maps of a sample, as will be discussed in further detail herein.

[0072] Furthermore, it should be noted herein that the training described may also be performed on an external image processing system that is not part of the optical system 100. Thus, the trained machine learning algorithm may be provided to the controller 110 of the optical system 100 and used to generate a depth map (as discussed below with respect to step 216).

[0073] In step 208 , the trained machine learning algorithm may be stored. For example, the controller 110 may be further configured to store the training focus stacking, the live 3D data, and the trained machine learning algorithm 118 in the memory 112 .

[0074] Figure 2B A process flow diagram 210 is illustrated depicting a method for generating one or more depth maps using the trained machine learning algorithm 118 of the system 100 in accordance with one or more embodiments of the present disclosure. It should be noted that the process flow diagram 210 may be viewed as a conceptual flow diagram illustrating steps performed by / within the one or more processors 112 of the controller 110.

[0075] In step 212, a set of product multimodal focus stacks may be acquired. For example, controller 110 may be configured to acquire a set of product multimodal focus stacks of a product sample from optical assembly 102. As used herein, the terms "product image" or "product focus stack" may be used to refer to an image for which one or more depth maps are generated. Thus, "product image" may be distinguished from "training image," which may be considered as an image used as input to train the machine learning algorithm used in the described process.

[0076] It should be noted that any discussion of obtaining training images (e.g. Figure 3Unless otherwise indicated herein, the flowchart 300 (shown in FIG. 300 ) can be considered applicable to acquiring product images. Thus, at least a first product image can be acquired at a first focal length using at least a first illumination angle. Furthermore, at least a second product image can be acquired at the first focal length using at least one or more additional illumination angles. Optionally, at least one or more additional images can be acquired at the first focal length using one or more illumination wavelengths. Subsequently, additional images can be acquired at one or more distances / illumination angles.

[0077] In optional step 213, additional inputs to the machine learning algorithm may be calculated. The one or more additional inputs may include, but are not limited to, a depth map obtained from a single-modal focus stack using a non-learning-based algorithm, one or more processed images, one or more additional parameters, and the like. For example, the controller 110 may be configured to calculate the one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired product multimodal focus stack.

[0078] In step 214, the machine learning algorithm may receive a set of acquired product multimodal focal stacks (and optionally, additional computational inputs / parameters computed in step 213). For example, the optical assembly 102 may be configured to provide the acquired focal stacks to the image processing subsystem 108.

[0079] In step 216, a machine learning algorithm may be used to generate one or more depth maps based on the acquired focal stacks 103. For example, the image processing subsystem 108, via the machine learning algorithm, may be configured to generate one or more depth maps of a product sample based on a set of acquired product multimodal focal stacks 103. For example, the machine learning algorithm 118 may be configured to generate one or more depth maps at one or more repair stages of the optical shaping system based on the acquired product images.

[0080] Although embodiments of the present disclosure relate to optical inspection and / or shaping systems, it is contemplated that system 100 may include any optical system known in the art. For example, embodiments of the present disclosure may relate to optical manufacturing systems.

[0081] It may be further contemplated that each of the embodiments of the method described above may include any other steps of any other method described herein. Additionally, each of the embodiments of the method described above may be performed by any of the systems described herein.

[0082] Reference again Figure 1A, the one or more processors 112 may include any processing element known in the art. In this sense, the one or more processors 112 may include any microprocessor-type device configured to execute algorithms and / or instructions (e.g., a graphics processing unit (GPU), a computer processing unit (CPU), and the like). In embodiments, the one or more processors 112 may be comprised of a desktop computer, a mainframe computer system, a workstation, a graphics computer, a parallel processor, or any other computer system configured to execute processes configured to operate the system 100 (e.g., a network computer), as described throughout this disclosure. It should be further recognized that the term "processor" may be broadly defined to encompass any device having one or more processing elements that execute program instructions from the non-transitory memory medium 114. Accordingly, the above description should not be construed as limiting the present invention but is merely illustrative.

[0083] The memory medium 114 may include any storage medium known in the art suitable for storing program instructions and data that can be executed by the associated one or more processors 112. By way of non-limiting example, the memory medium 114 may include a non-transitory memory medium. By way of additional non-limiting example, the memory medium 114 may include, but is not limited to, read-only memory, random access memory, magnetic or optical storage devices (e.g., magnetic disks), tape, solid-state drives, and the like. It should be further noted that the memory 114 may be housed in a common controller housing along with the one or more processors 112. In alternative embodiments, the memory 114 may be remotely located relative to the physical location of the one or more processors 112 and the controller. For example, the one or more processors 112 of the controller may access remote memory (e.g., a server) that is accessible via a network (e.g., the Internet, an intranet, and the like).

[0084] Those skilled in the art will recognize that for the sake of conceptual clarity, the components, operations, devices, objects, and accompanying discussions described herein are used as examples, and various configuration modifications are contemplated. Therefore, as used herein, the specific examples and accompanying discussions set forth herein are intended to represent their more general class. In general, the use of any specific example is intended to represent its class, and the absence of specific components, operations, devices, and objects should not be considered limiting.

[0085] With respect to the use of generally any plural and / or singular terms herein, those skilled in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application. For clarity, the various singular / plural permutations may be expressly set forth herein.

[0086] The subject matter described herein sometimes illustrates different components that are contained within or connected to other components. It should be understood that this depicted architecture is merely exemplary, and in fact, many other architectures that achieve the same functionality can be implemented. In a conceptual sense, any arrangement of components that achieve the same functionality is effectively "associated" so as to achieve the desired functionality. Therefore, any two components that are combined to achieve a specific functionality herein can be considered to be "associated" with each other so as to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two components that are so associated can be considered to be "connected" or "coupled" to each other to achieve the desired functionality, and any two components that can be so associated can also be considered to be "coupleable" to each other to achieve the desired functionality. Specific examples that can be coupled include, but are not limited to, physically matable and / or physically interactive components and / or wirelessly interactive and / or wirelessly interactive components and / or logically interactive and / or logically interactive components.

[0087] Furthermore, it should be understood that the present invention is defined by the appended claims. It should be understood by those skilled in the art that, in general, terms used herein, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be "open-ended" terms (e.g., the term "comprising" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "including" should be interpreted as "including but not limited to," and the like). It should be further understood by those skilled in the art that if a particular numbered leading claim recitation is contemplated, such intent will be explicitly recited in the claim, and in the absence of such recitation, such intent does not exist. For example, as an aid to understanding, the following appended claims may contain the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as implying that introducing a claim recitation with the indefinite article "a" limits any particular claim containing such introduced claim recitation to inventions containing only one such recitation, even when the same claim contains the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" (e.g., "a" should generally be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles introducing claim recitations. Furthermore, even if a specific number of introduced claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should generally be interpreted as meaning at least the recited number (e.g., the bare recitation of "two recitations" without other modifiers generally means at least two recitations, or two or more recitations). Furthermore, in such examples where a convention similar to “at least one of A, B, and C, and the like” is used, such construction is generally contemplated in the sense that one skilled in the art would understand the convention (e.g., “a system having at least one of A, B, and C” would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, and the like). In such examples where a convention similar to “at least one of A, B, or C, and the like” is used, such construction is generally contemplated in the sense that one skilled in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, and the like). It will be further understood by those skilled in the art that virtually any antonym and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to encompass the possibility of including one, either, or both of the terms. For example, the phrase "A or B" will be understood to include the possibility of "A" or "B" or "A and B."

[0088] It is believed that the present disclosure and its many attendant advantages will be appreciated from the foregoing description, and it will be understood that various changes may be made in the form, construction, and arrangement of components without departing from the disclosed subject matter or sacrificing all of its material advantages. The forms described are illustrative only and it is intended that such changes be encompassed and included by the appended claims. Furthermore, it should be understood that the invention is defined by the appended claims.

Claims

1. An optical system, comprising: an optical assembly configured to illuminate one or more portions of one or more samples using two or more illumination modalities, wherein the two or more illumination modalities include at least a first illumination modality and a second illumination modality, the first illumination modality including a first set of illumination angles, the second illumination modality including a second set of illumination angles, wherein the second set of illumination angles is at least partially different from the first set of illumination angles, the optical assembly being configured to acquire a multimodal focal stack comprising a plurality of images acquired at two or more distances between the one or more samples and a front focal plane, wherein at least a first image of the plurality of images is acquired using the first illumination modality and at least an additional image is acquired using the second illumination modality; and an image processing subsystem communicatively coupled to the optical assembly, the image processing subsystem comprising one or more processors configured to execute a set of program instructions stored in a memory, the set of program instructions configured to cause the one or more processors to: receiving a plurality of training images, the plurality of training images comprising a plurality of training multimodal focus stacks; receiving live three-dimensional data for each training multimodal focal stacking of the plurality of training multimodal focal stackings; training a machine learning algorithm based on the plurality of training images and the received live three-dimensional data; receiving the multimodal focal stack of a sample from the optical assembly; and A depth map of the sample is generated using the trained machine learning algorithm and the received multimodal focal stacking of the sample. 2 . The system of claim 1 , wherein the first illumination modality comprises a first spectral distribution and the second illumination modality comprises a second spectral distribution, wherein the second spectral distribution is at least partially different from the first spectral distribution.

3. The system of claim 1 , wherein the optical assembly comprises: one or more illumination devices configured to illuminate the one or more samples at a plurality of illumination angles; and At least one imaging device disposed in a fixed orientation relative to the one or more samples, the at least one imaging device configured to acquire at least a first image of the one or more samples using a first illumination angle of the plurality of illumination angles and to acquire at least a second image of the one or more samples using a second illumination angle of the plurality of illumination angles.

4. The system of claim 3, further comprising: One or more actuation assemblies configured to adjust at least a distance between the one or more samples and the front focal plane of the optical assembly, wherein the at least one imaging device is capable of being configured to acquire one or more images at two or more distances between the one or more samples and the front focal plane.

5. The system of claim 4, wherein the adjusting at least a distance between the one or more samples and the front focal plane of the optical assembly comprises at least one of: The position of the optical assembly is adjusted or the position of the one or more samples is adjusted.

6. The system of claim 3, wherein the one or more processors are further configured to one of: adjust a position of the at least one imaging device or adjust one or more optical properties of one of the at least one imaging device or the optical assembly.

7. The system of claim 1 , wherein the one or more processors are further configured to: One or more additional inputs to the machine learning algorithm are received, the one or more additional inputs comprising one of: one or more depth maps obtained from the multimodal focal stacking using a non-learning based algorithm, one or more processed images from the multimodal focal stacking, or one or more additional parameters.

8. The system of claim 1, wherein the live data comprises one or more white light interferometry depth maps.

9. The system of claim 1 , wherein the machine learning algorithm comprises at least one of: Convolutional Neural Networks, Generative Adversarial Networks, Modular Neural Networks, or Transformers.

10. The system of claim 1, further comprising: One or more prefabricated reference objects comprising one or more three-dimensional structures with one or more known depth maps configured for at least one of validation of the optical assembly, calibration of the optical assembly, correction of the multimodal focal stacking, or retraining of the machine learning algorithm.

11. The system of claim 1 , wherein the sample comprises a printed circuit board.

12. The system of claim 1, wherein the optical system comprises an automated optical inspection system.

13. The system of claim 1, wherein the optical system comprises an automated optical shaping system.

14. An image processing system, comprising: One or more processors configured to execute a set of program instructions stored in a memory, the set of program instructions configured to cause the one or more processors to: receiving a plurality of training images of one or more portions of one or more samples, the plurality of training images comprising a plurality of training multimodal focal stacks; receiving live three-dimensional data for each training multimodal focal stacking of the plurality of training multimodal focal stackings; training a machine learning algorithm based on the plurality of training images and the received live three-dimensional data; receiving a multimodal focal stack of a sample from an optical assembly configured to illuminate the one or more portions of the sample using two or more illumination modalities, wherein the two or more illumination modalities include at least a first illumination modality and a second illumination modality, the first illumination modality including a first set of illumination angles, the second illumination modality including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodal focal stack including a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images was acquired using the first illumination modality and at least an additional image was acquired using the second illumination modality; and A depth map of the sample is generated using the trained machine learning algorithm and the received multimodal focal stacks.

15. The system of claim 14, wherein the first illumination modality comprises a first spectral distribution and the second illumination modality comprises a second spectral distribution, wherein the second spectral distribution is at least partially different from the first spectral distribution.

16. The system of claim 14, further comprising: An optical assembly configured to obtain the multimodal focal stacking.

17. The system of claim 16, wherein the optical assembly comprises: one or more lighting devices configured to illuminate the sample at a plurality of lighting angles; and At least one imaging device is arranged in a fixed orientation relative to the sample, the at least one imaging device being configured to acquire at least a first image of the sample using a first illumination angle of the plurality of illumination angles and to acquire at least a second image of the sample using a second illumination angle of the plurality of illumination angles.

18. The system of claim 17, wherein the optical assembly further comprises: One or more actuation assemblies configured to adjust at least one distance between the sample and the front focal plane of the optical assembly, wherein the at least one imaging device is configurable to acquire two or more images at the two or more distances between the sample and the front focal plane.

19. The system of claim 18, wherein the adjusting at least a distance between the sample and the front focal plane of the optical assembly comprises at least one of: The position of the optical assembly is adjusted or the position of the sample is adjusted.

20. The system of claim 17, wherein the one or more processors are further configured to one of: adjust a position of the at least one imaging device or adjust one or more optical properties of one of the at least one imaging device or the optical assembly.

21. The system of claim 14, wherein the one or more processors are further configured to: One or more additional inputs to the machine learning algorithm are received, the one or more additional inputs comprising one of: one or more depth maps obtained from the multimodal focal stacking using a non-learning based algorithm, one or more processed images from the multimodal focal stacking, or one or more additional parameters.

22. The system of claim 14, wherein the live data comprises one or more white light interferometry depth maps.

23. The system of claim 14, wherein the machine learning algorithm comprises at least one of: Convolutional Neural Networks, Generative Adversarial Networks, Modular Neural Networks, or Transformers.

24. The system of claim 14, wherein the sample comprises a printed circuit board.

25. A method comprising: receiving a plurality of training images of one or more portions of one or more samples, the plurality of training images comprising a plurality of training multimodal focal stacks; receiving live three-dimensional data for each training multimodal focal stacking of the plurality of training multimodal focal stackings; training a machine learning algorithm based on the plurality of training images and the received live three-dimensional data; receiving a multimodal focal stack of the one or more samples from an optical assembly configured to illuminate the one or more samples using two or more illumination modalities, wherein the two or more illumination modalities include at least a first illumination modality and a second illumination modality, the first illumination modality including a first set of illumination angles, the second illumination modality including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodal focal stack including a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images was acquired using the first illumination modality and at least an additional image was acquired using the second illumination modality; and A depth map of the one or more samples is generated using the trained machine learning algorithm and the received multimodal focal stacks.

Images