Systems and methods for three-dimensional imaging of a sample using machine learning algorithms

Abstract

A method for 3D imaging of a sample using a machine learning algorithm is disclosed. The method uses a multi-modal focal stack, which consists of multiple images taken at two or more distances between the sample and a front focal plane, where at least one image is taken using a first modality and additional images are taken using additional modalities. The modalities can have different illumination angles and optionally different spectral distributions. The method can include receiving training images of samples including multiple training focal stacks and live 3D data (depth maps) for each training focal stack, and training a machine learning algorithm based on the training images and the live data. The method then receives a product image of a sample, which includes a focal stack from an optical assembly, and generates a 3D depth map.

Description

Technical Field

[0001] The present invention generally relates to three-dimensional (3D) imaging, and more specifically, to a system and method for 3D imaging of a sample based on a multimodal focus stack using a machine learning algorithm, wherein the machine learning algorithm is trained using the set of multimodal focus stacks. Background Technology

[0002] As the demand for electronic circuits with increasingly smaller device features continues to grow, so too does the need for improved 3D imaging technology. Optical inspection systems may require 3D imaging to measure circuit features (e.g., conductors, drill bits, and the like) in three dimensions and to identify or verify “3D defects” (e.g., sink marks, i.e., thin areas of copper). Optical shaping systems can be used in printed circuit board manufacturing processes to increase yield and eliminate debris. Such optical shaping systems can utilize 3D imaging when repairing defects. For example, 3D imaging can be used during the repair process when copper is deposited onto the surface of a printed circuit board. As another example, 3D imaging can be used after the repair process to measure sample damage to ensure that the laser does not inadvertently penetrate the stack when excess copper is removed (e.g., using laser ablation).

[0003] Current focus-based 3D printing technologies are generally cost-effective, but offer limited spatial accuracy and precision compared to imaging resolution. This is because determining the focus quality of individual pixels requires computation based on the neighborhood of each pixel. Other 3D imaging technologies (e.g., interferometry-based) are more expensive, and some may require additional optical components, which complicates the system and may reduce its processing power.

[0004] Therefore, it would be advantageous to provide a system and method that overcomes the shortcomings of the above-mentioned methods. Summary of the Invention

[0005] According to one or more embodiments of this disclosure, an optical system is disclosed. In one embodiment, the optical system includes an optical assembly configured to illuminate one or more portions of a sample using two or more illumination modes including at least a first illumination mode and a second illumination mode, wherein the first illumination mode includes a first set of illumination angles and the second illumination mode includes a second set of illumination angles, wherein at least the second set of illumination angles is at least partially different from the first set of illumination angles. In one embodiment, the optical assembly is configured to acquire a multimodal focus stack comprising multiple images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the multiple images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode. In one embodiment, the optical system further includes an image processing subsystem communicatively coupled to the optical assembly, wherein the image processing subsystem includes one or more processors configured to execute a set of program instructions stored in a memory. In one embodiment, the set of program instructions is configured to cause the one or more processors to receive multiple training images, wherein the one or more training images comprise multiple training multimodal focus stacks. In one embodiment, the set of program instructions is configured to cause the one or more processors to receive 3D live data of each of the plurality of trained multimodal focus stacks. In another embodiment, the set of program instructions is configured to cause the one or more processors to train a machine learning algorithm based on the plurality of training images and the received 3D live data. In another embodiment, the set of program instructions is configured to cause the one or more processors to receive the multimodal focus stacks of the sample from the optical assembly. In yet another embodiment, the set of program instructions is configured to cause the one or more processors to use the trained machine learning algorithm and the received multimodal focus stacks to generate a depth map of the sample.

[0006] According to one or more embodiments of this disclosure, an image processing system is disclosed. In one embodiment, the system includes one or more processors configured to execute a set of program instructions stored in a memory. In another embodiment, the set of program instructions is configured to cause the one or more processors to receive a plurality of training images, wherein the one or more training images comprise a plurality of training multimodal focus stacks. In yet another embodiment, the set of program instructions is configured to cause the one or more processors to receive three-dimensional live data of each of the plurality of training multimodal focus stacks. In yet another embodiment, the set of program instructions is configured to cause the one or more processors to train a machine learning algorithm based on the plurality of training images and the received three-dimensional live data. In one embodiment, the set of program instructions is configured to cause one or more processors to receive a multimodal focal stack of a sample from an optical assembly, wherein the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modes including at least a first illumination mode and a second illumination mode, wherein the first illumination mode includes a first set of illumination angles and the second illumination mode includes a second set of illumination angles, wherein at least the second set of illumination angles is at least partially different from the first set of illumination angles, wherein the multimodal focal stack includes multiple images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the multiple images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode. In another embodiment, the set of program instructions is configured to cause one or more processors to generate a depth map of the sample using the trained machine learning algorithm and the received multimodal focal stack.

[0007] According to one or more embodiments of this disclosure, a method is disclosed. In one embodiment, the method includes receiving a plurality of training images, wherein the one or more training images comprise a plurality of training multimodal focus stacks. In another embodiment, the method includes receiving three-dimensional live data of each of the plurality of training multimodal focus stacks. In yet another embodiment, the method includes training a machine learning algorithm based on the plurality of training images and the received live three-dimensional data. In yet another embodiment, the method includes receiving a multimodal focus stack of a sample from an optical assembly, wherein the optical assembly is configured to illuminate one or more portions of the sample using two or more illumination modes comprising at least a first illumination mode and a second illumination mode, wherein the first illumination mode comprises a first set of illumination angles and the second illumination mode comprises a second set of illumination angles, wherein at least the second set of illumination angles is at least partially different from the first set of illumination angles, wherein the multimodal focus stack comprises a plurality of images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the plurality of images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode. In one embodiment, the method includes using the trained machine learning algorithm and the received multimodal focus stack to generate a depth map of the sample.

[0008] It should be understood that both the foregoing general description and the following detailed description are exemplary and illustrative only, and are not necessarily limiting of the invention as claimed. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description, serve to explain the principles of the invention. Attached Figure Description

[0009] Those skilled in the art can better understand the many advantages of this disclosure by referring to the accompanying drawings.

[0010] Figure 1A A block diagram illustrating an optical system according to one or more embodiments of the present disclosure.

[0011] Figure 1B A simplified schematic top view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0012] Figure 1C A simplified schematic side view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0013] Figure 1D A simplified schematic top view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0014] Figure 1E A simplified schematic side view illustrating an optical assembly of an optical system according to one or more embodiments of the present disclosure.

[0015] Figure 2A A flowchart illustrating a method for training a machine learning algorithm according to one or more embodiments of the present disclosure.

[0016] Figure 2B The illustration depicts a process flowchart of a method for generating one or more depth maps using a trained machine learning algorithm of a system, according to one or more embodiments of the present disclosure.

[0017] Figure 3 The illustration depicts a process flowchart of a method for obtaining one or more focus stacks using one or more components of a system according to one or more embodiments of the present disclosure.

[0018] Figure 4A and 4B This describes a focused stacked image of a sample according to one or more embodiments of the present disclosure.

[0019] Figure 5 This describes a white light interferometric depth map of a sample according to one or more embodiments of the present disclosure. Detailed Implementation

[0020] This disclosure has been specifically shown and described with respect to certain embodiments and their specific features. The embodiments set forth herein are to be considered illustrative rather than restrictive. Those skilled in the art will readily understand that various changes and modifications in form and detail may be made without departing from the spirit and scope of this disclosure. Reference will now be made in detail to the disclosed subject matter, which is illustrated in the accompanying drawings.

[0021] With the increasing demand for electronic circuits with ever-smaller features and reduced thickness, the need for improved 3D imaging technology is also growing. Optical inspection systems can be used for process control and yield management in printed circuit board manufacturing processes. Such systems can utilize 3D imaging to identify and / or verify defects and 3D measure the features of circuits. Optical shaping systems can be used to remove debris and increase printed circuit board yield. Such systems can utilize 3D imaging when repairing defects. Optical shaping systems are substantially described in U.S. Patent No. 8,290,239, published October 16, 2012, and U.S. Patent Publication No. 2013 / 0037526, published February 14, 2013, the entire contents of which are incorporated herein by reference.

[0022] Current 3D imaging techniques using depth of focus (DFF) capture a single image at each object distance from the focal plane of the optical system. From this set of images at different object distances, the DFF algorithm determines the optimal focal position for the neighborhood of each imaging point in the object. However, a drawback of "standard" DFF is that it offers limited spatial accuracy and precision compared to imaging resolution, because determining the focus quality of individual pixels requires calculations based on the neighborhood of each pixel.

[0023] Other 3D imaging techniques, such as white light interferometry (WLI), multi-view stereo and triangulation, or phase-shifting methods, offer higher spatial accuracy and precision, but are expensive and require additional optical components. Furthermore, other imaging techniques, such as photometric stereo (PS), are typically computationally intensive (and therefore time-consuming) and require detailed and accurate reflection models of the materials composing the imaged object to obtain accurate surface reconstructions.

[0024] Therefore, it would be advantageous to provide a system and method that overcomes the shortcomings of the above-mentioned methods.

[0025] Embodiments of this disclosure relate to a system and method for generating one or more depth maps of one or more portions of a sample by capturing a set of images at a set of distances between the sample and a front focal plane at at least two different illumination angles and obtaining a depth map from this data using a machine learning algorithm. For example, the system may include an optical assembly configured to acquire a set of images at a set of distances between the sample and the front focal plane at at least two different illumination angles. As previously described herein, unlike a “standard” DFF in which a single image is captured at each distance from the object, the system of this disclosure can acquire a multimodal focal stack of the sample under different illumination modalities. In this respect, the system can acquire reflection information of each pixel on the sample under different illumination directions. This information can be used to increase the spatial resolution, precision, and accuracy of the obtained depth map. For the purposes of this disclosure, a “depth map” and its variations can be defined as data consisting of depth values ​​(z) as a function of position (x, y) in a vertical plane (e.g., at each of the imaging pixels).

[0026] As previously described herein, unlike PSs that require a reflection model, the systems and methods disclosed herein do not rely on a material reflection model. Instead, they use machine learning methods to train the system on the type of sample to be imaged. Furthermore, using machine learning methods reduces the number of images required for 3D reconstruction when generating one or more depth maps.

[0027] Furthermore, the system may include an image processing subsystem configured to use a machine learning algorithm trained on a focal stack with multiple illuminations and live 3D data (e.g., obtained using WLI or other imaging techniques). Once trained, the machine learning algorithm can be configured to generate one or more depth maps of one or more portions of the sample based on images collected from the optical assembly. In some embodiments, the machine learning algorithm can be configured to generate one or more depth maps of one or more portions of the sample at a depth resolution between about 0.1 micrometers (µm) and 2 micrometers.

[0028] Figure 1A A simplified block diagram illustrating an optical system 100 according to one or more embodiments of the present disclosure is provided. Specifically, Figure 1A This describes a system 100 for generating depth maps of one or more parts of a sample using machine learning techniques.

[0029] In one embodiment, system 100 includes an optical assembly 102. The optical assembly may include, but is not limited to, one or more imaging devices 104, one or more illumination devices 106, one or more controllers including one or more processors and memories, a rack, and the like.

[0030] System 100 may additionally include an image processing subsystem 108 communicatively coupled to the optical assembly 102. The image processing subsystem 108 may include, but is not limited to, one or more controllers including one or more processors 112 and memory 114.

[0031] The optical assembly 102 and / or the image processing subsystem 108 can be communicatively coupled to the user interface device 116.

[0032] Figures 1B to 1E A simplified schematic diagram illustrating the optical assembly 102 of an optical system 100 according to one or more embodiments of the present disclosure is provided. Specifically, Figure 1B A top view depicting a single ring of the illumination device 106 of an optical assembly 102 according to one or more embodiments of the present disclosure. Specifically, Figure 1C A side view depicting a single-ring optical assembly 102 of an optical system 100 according to one or more embodiments of the present disclosure. Specifically, Figure 1D A top view depicting a dual-ring optical assembly 102 of an optical system 100 according to one or more embodiments of the present disclosure. Specifically, Figure 1E A side view depicting a dual-ring optical assembly 102 of an optical system 100 according to one or more embodiments of the present disclosure.

[0033] In an embodiment, the optical assembly 102 is configured to acquire one or more multimodal focal stacks of the sample 120. For example, the optical assembly 102 may be configured to illuminate the sample 120 using two or more illumination modes. For example, the two or more illumination modes may include at least a first illumination mode and a second illumination mode, wherein the first mode may include a first set of illumination angles and the second mode includes a second set of illumination angles. By another example, the first mode may include a first spectral distribution and a second spectral distribution, wherein the second spectral distribution may be at least partially different from the first. In this respect, the multimodal focal stack may include multiple images acquired at two or more distances between the front focal plane and the sample 120, wherein at least one image is acquired using the first mode and additional images are acquired using the second mode, wherein the first mode and the second mode are at least partially different.

[0034] The optical assembly 102 can be positioned relative to the sample 120 mounted on the stage assembly 122. In embodiments, the sample may be homogeneous, formed of two or more materials, wherein the first material is different from at least the second material. For example, the homogeneous sample may comprise a printed circuit board (PCB). The two or more materials may comprise, but are not limited to, copper, multilayers, or the like.

[0035] One or more imaging devices 104 of the optical assembly 102 may be configured to acquire one or more images of a sample 120 mounted on the stage assembly 122. For the purposes of this disclosure, the term “one or more imaging devices 104” refers to one or more imaging devices that include an imaging sensor (e.g., a camera) and one or more optical elements (e.g., one or more lenses), unless otherwise stated herein.

[0036] One or more imaging devices 104 may comprise any type of imaging device suitable for acquiring one or more two-dimensional (2D) images of sample 120. For example, one or more imaging devices 104 may comprise one or more cameras configured for image acquisition. For example, one or more imaging devices 104 may comprise one or more high-speed cameras configured for rapid image acquisition (e.g., a standard video rate greater than 24 to 25 frames per second (e.g., 90 frames per second)).

[0037] In an embodiment, one or more imaging devices 104 share a common axis with a sample 120 mounted on a stage assembly 122 when acquiring one or more images. For example, one or more imaging devices 104 may be positioned above at least a portion of the sample 120 mounted on the stage assembly 122 when acquiring one or more images.

[0038] In an embodiment, the optical assembly 102 further includes one or more actuation assemblies 107. The one or more actuation assemblies 107 may be configured to adjust the focal length of the optical assembly 102 relative to the sample 120 (i.e., the distance between the sample and the front focal plane). For example, the one or more actuation assemblies may be configured to adjust the focal length of one or more imaging devices 104 by actuating the optical assembly 102 at least along the z-axis (common axis). By another example, the one or more actuation assemblies 107 may be configured to adjust the focal length by actuating the sample 120 at least along the z-axis (common axis). For example, the one or more actuation assemblies 107 may be configured to adjust the focal length of one or more imaging devices 104 such that the focal length between the one or more imaging devices 104 and the sample 120 can be adjusted. In this respect, the one or more imaging devices 104 may be configured to acquire two or more images of the sample at a set of two or more focal lengths.

[0039] It should be noted that one or more actuation assemblies 107 may include any type of actuation device suitable for adjusting at least the focal length of the optical assembly 102 relative to the sample 120. For example, one or more actuation assemblies 107 may include, but are not limited to, one or more linear actuator devices. Furthermore, it should be noted herein that the system 100 may be configured to adjust the focal length of the optical assembly 102 relative to the sample 120 via any mechanism. Therefore, the foregoing should not be construed as limiting the scope of this disclosure. For example, the optical assembly 102 may include a variable-focus lens configured to adjust the focal length of the optical assembly 102 relative to the sample 120. By another example, one or more components of the optical assembly 102 may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120. For example, one or more imaging sensors may be actuated to adjust the focal length of the optical assembly 102 relative to the sample 120.

[0040] In an embodiment, one or more imaging devices 104 may be configured to acquire two or more images of the sample 120 at a set distance when one or more actuation assemblies are activated. For example, two or more images of the sample 120 may be acquired during movement of the optical assembly 102 via one or more actuation assemblies.

[0041] In an embodiment, one or more imaging devices 104 may be configured to acquire two or more images of sample 120 at a set focal lengths when one or more actuation assemblies are located at a fixed focal length. For example, two or more images of sample 120 may be acquired after one or more actuation assemblies 107 have adjusted the focal length of one or more imaging devices 104 and are located at one or more predetermined focal lengths (or distances from the sample).

[0042] In embodiments, one or more illumination devices 106 of the optical assembly 102 may be configured to generate one or more illumination beams 105. The one or more illumination devices 106 may include any suitable illumination device for generating the one or more illumination beams 105. For example, the one or more illumination devices 106 may include one or more flash illumination devices. For example, the one or more flash illumination devices may include, but are not limited to, one or more light-emitting diodes (LEDs) (e.g., red LEDs) or the like. In this regard, the one or more flash illumination devices may be configured to generate illumination for a short period of time to avoid blurring caused by movement / vibration (e.g., along a common axis or in a plane perpendicular to it). This allows for rapid acquisition of images by capturing images as the distance between the sample and the front focal plane changes (e.g., by moving the imaging device 104).

[0043] The optical assembly 102 may further include one or more optical elements 126 configured to guide one or more illumination beams onto the surface of the sample 120. For example, the optical assembly 102 may include, but is not limited to, one or more mirrors, one or more lenses, one or more polarizers, one or more beam splitters, one or more optical fibers, and the like. It should be noted that... Figures 1B to 1E This is for illustrative purposes only and should not be construed as limiting the scope of this disclosure. For example, optical assembly 102 may contain any type of optical element and any configuration of optical elements.

[0044] In an embodiment, one or more illumination devices 106 may comprise a set of illumination devices 106 positioned relative to the sample 120 at a set of illumination angles. For example, one or more illumination devices may comprise a set of illumination devices 106 comprising one or more high-altitude illumination devices (e.g., positioned at a high-altitude angle relative to the stage / sample) and one or more low-altitude illumination devices (e.g., positioned at a lower altitude angle relative to the stage / sample than the high-altitude devices).

[0045] It should be noted that the optical assembly 102 may include any number and configuration of illumination devices 106 suitable for illuminating the sample 120 at a specific illumination angle (e.g., at several non-collinear illumination directions). Figures 1B to 1E This is for illustrative purposes only and should not be construed as limiting the scope of this disclosure.

[0046] Each set of lighting devices 106 can be positioned close to the sample 120 mounted on the stage assembly 122. For example, each set of lighting devices 106a to 106d can be positioned adjacent to (or close to) the sample 120 mounted on the stage assembly.

[0047] refer to Figures 1B to 1E The illumination device 106 can be evenly distributed in the azimuth angle so that it can substantially uniformly cover the illumination direction range of the sample 120. For example, as Figures 1B to 1CAs shown, the lighting fixtures 106 can be arranged in a single ring with equal angular intervals. In one example, such as Figure 1B As shown, the set of lighting devices 106 may comprise six lighting devices spaced 60 degrees apart. By another example, such as... Figures 1D to 1E As shown, the lighting fixtures 106 can be evenly distributed in a double-ring configuration. In one example, such as Figure 1D As shown, the set of lighting devices may include ten lighting devices, wherein the central ring includes four lighting devices spaced 90 degrees apart and the outer ring includes six lighting devices spaced 60 degrees apart.

[0048] although Figure 1B An optical assembly 102 is depicted comprising seven illumination devices configured to illuminate a sample at seven different illumination angles. However, it should be noted that the optical assembly 102 may comprise any number and arrangement of illumination devices configured to illuminate the sample at any azimuth and / or height illumination angle. For example, the set of illumination devices may comprise six illumination devices spaced 60 degrees apart in a ring. Furthermore, it should be noted that the optical assembly 102 may be configured to illuminate the sample at one or more of the different illumination angles using any combination of illumination devices.

[0049] System 100 may further include one or more prefabricated reference objects (or calibration targets). For example, one or more prefabricated reference objects may include one or more fixed reference objects. By another example, one or more prefabricated reference objects may include one or more removable reference objects.

[0050] One or more reference objects may contain one or more three-dimensional (3D) structures with known (or previously measured) depth maps. For example, one or more known depth maps can be used to verify one or more components of an optical assembly. By another example, one or more known depth maps can be used to calibrate one or more components of an optical assembly. By another example, one or more known depth maps can be used to retrain a machine learning algorithm. By another example, one or more known depth maps can be used to adjust one or more inputs to a machine learning algorithm. For example, one or more depth maps can be used to correct the distance in each image, which can be adjusted after calibration.

[0051] In an embodiment, the image processing subsystem 108 is configured to generate one or more depth maps of sample 120 based on a set of multimodal focus stacks (comprising one or more 2D images) acquired from the optical assembly 102. For example, the image processing subsystem 108 may be configured to use a machine learning algorithm 118 stored in the memory 114 of the controller 110 of the image processing subsystem 108 to generate one or more depth maps of sample 120 corresponding to two or more 2D images of the focus stacks, as will be further discussed herein.

[0052] Figure 2AA process flowchart 200 illustrates a method for training a machine learning algorithm 118 of a training system 100 according to one or more embodiments of the present disclosure. It should be noted that the process flowchart 200 can be considered as a conceptual flowchart illustrating steps executed by / within one or more processors 112 of the controller 110.

[0053] In step 202, a set of training multimodal focus stacks may be acquired. For example, the controller 110 of the image processing subsystem 108 may be configured to acquire a set of multimodal focus stacks 101 from the optical assembly 102 for training the machine learning algorithm 118.

[0054] In embodiments, each multimodal focal stack 101 may include one or more image acquisitions at each of a set of distances between the sample and the front focal plane, and at least some of those distances having two or more illumination angles or having different illumination angles at different distances. For example, one or more imaging devices 104 of the optical assembly 102 may be configured to acquire one or more images at a set of distances between the sample and the front focal plane when one or more of the illumination devices 106 illuminate the sample 120 at one or more different illumination angles.

[0055] In optional step 203, additional inputs to the machine learning algorithm may be computed. One or more additional inputs may include, but are not limited to, a depth map obtained from a single-modal focus stack using a non-learning-based algorithm, one or more processed images, one or more additional parameters, and the like. For example, controller 110 may be configured to compute one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired multimodal focus stack.

[0056] Figure 3 This illustration depicts a process flowchart 300 illustrating a method for obtaining one or more focus stacks using one or more components of system 100 according to one or more embodiments of the present disclosure. It should be noted that flowchart 300 can be considered as a conceptual flowchart illustrating steps executed by / within one or more processors 112 of controller 110.

[0057] In step 302, a first image can be acquired having a first illumination mode with a first illumination angle (more generally, a range of illumination angles that may partially overlap). For example, one or more imaging devices 104 may be configured to acquire a first image at a first distance between the sample and the front focal plane when at least a first illumination device illuminates the sample at at least a first illumination angle. For the purposes of this disclosure, an "illumination mode" and variations thereof may be defined as a set of one or more illumination angles having the same or different spectral distributions for each angle.

[0058] In optional step 303, the focal length can be adjusted. For example, when acquiring images in motion, the focal length can be adjusted between acquiring additional images in additional modalities.

[0059] In optional step 304, a second image having a second illumination mode with a second illumination angle can be acquired. For example, one or more imaging devices 104 may be configured to acquire a second image at a first distance between the sample and the front focal plane when at least the second illumination device illuminates the substrate at at least with a second illumination angle.

[0060] In optional step 305, the focal length can be adjusted. For example, when acquiring images in motion, the focal length can be adjusted between acquiring additional images in an additional modality.

[0061] Figure 4A and 4B This describes exemplary focus-stacked images according to one or more embodiments of the present disclosure. It should be noted that... Figure 4A and 4B The two sets of images 400a to 400d and 420a to 420d may be portions of a single focal point stack, which may include additional images taken at additional focal lengths and illumination angles.

[0062] refer to Figure 4A The first set of focused stacked images 400 may include multiple images 400a to 400d of a sample 120 (e.g., a PCB) taken at a first exemplary focal length. For example, the first set of focused stacked images may include at least a first image 400a taken at the first exemplary focal length with a first azimuth illumination direction 402a, wherein image 400a is focused on the copper lines of the PCB. By another example, the first set of focused stacked images may include at least a second image 400b taken at the first exemplary focal length with a second azimuth illumination direction 402b, wherein image 400b is focused on the copper lines of the PCB. By another example, the first set of focused stacked images may include at least a third image 400c taken at the first exemplary focal length with a third azimuth illumination direction 402c, wherein image 400c is focused on the copper lines of the PCB. By another example, the first set of focused stacked images may include at least a fourth image 400d taken at the first exemplary focal length with a fourth azimuth illumination direction 402d, wherein image 400d is focused on the copper lines of the PCB. refer to Figure 4BThe second set of focused stacked images 420 may include multiple images 420a to 420d of the sample 120 (e.g., a PCB) taken at a second exemplary focal length (different from the first exemplary focal length). For example, the second set of focused images may include at least a first image 420a taken at the second exemplary focal length with a first azimuth illumination direction 422a, wherein image 420a is focused on the stack of the PCB. By another example, the second set of focused images may include at least a second image 420b taken at the second exemplary focal length with a second azimuth illumination direction 422b, wherein image 420b is focused on the stack of the PCB. By another example, the second set of focused images may include at least a third image 420c taken at the second exemplary focal length with a third azimuth illumination direction 422c, wherein image 420c is focused on the stack of the PCB. By another example, the second set of focused images may include at least a fourth image 420d taken at the second instance focal length and in a fourth azimuth illumination direction 422d, wherein image 420d is focused on the stack-up of the PCB.

[0063] Return to reference Figure 3 In optional step 306, one or more additional images with one or more additional illumination modes may be acquired. For example, one or more imaging devices 104 may be configured to acquire a third image at a first distance between the sample and the front focal plane when the sample is illuminated by at least a third illumination device at at least a third illumination angle. By another example, one or more imaging devices 104 may be configured to acquire a fourth image at a first distance between the sample and the front focal plane when the sample is illuminated by at least a fourth illumination device at at least a fourth illumination angle. By another example, one or more imaging devices 104 may be configured to acquire one or more images at a first distance between the sample and the front focal plane using one or more additional illumination spectra (e.g., different from the illumination spectra used in at least one of steps 302 to 306). It should be noted that the illumination spectrum may induce fluorescence in at least one of two or more materials of the sample 120. For example, in the case that the sample 120 is a PCB, the illumination spectrum may induce fluorescence in the PCB stack-up material.

[0064] In step 308, the distance between the sample and the optical assembly relative to the front focal plane of the sample can be adjusted. For example, the actuation assembly can be configured to adjust the focal height of the optical assembly 102 relative to the sample 120.

[0065] It should be noted that one or more steps of method 300 (e.g., steps 302 to 306) may then be repeated at each of one or more adjusted focal heights. For example, one or more of steps 302 to 306 may be repeated one or more times based on a predetermined height measurement range (e.g., the distance between the optical assembly and the sample). For example, the steps may be repeated every 0.5 µm to cover a distance of 25 µm between the front focal plane of the optical assembly and the sample on each side.

[0066] It should be noted that the optical assembly 102 can be configured to perform continuous scanning of the focal length. For example, during scanning, the distance between the sample and the front focal plane can be continuously adjusted, and one or more illumination devices can preferably illuminate the sample by flash, while maintaining a sufficiently small effective distance range between the sample and the front focal plane relative to the depth of field of the optical assembly during each acquisition, thereby avoiding blurring of height information. One or more imaging devices are preferably operated to acquire images of the sample illuminated by a set of illumination modes provided by one or more illumination devices.

[0067] Return to reference Figure 2A In step 204, 3D live data for each focus stack of a set of multimodal focus stacks (acquired in step 202) can be acquired. For example, the controller 110 of the image processing subsystem 108 can be configured to retrieve the 3D live data for each focus stack from a database stored in memory 114. By another example, the controller 110 of the image processing subsystem 108 can be configured to retrieve the 3D live data for each focus stack from a remote storage location (not shown).

[0068] In an embodiment, such as Figure 5 As shown, the 3D live data may contain one or more white light interferometry (WLI) depth maps. For example, WLI depth map 500 can be obtained using WLI and can be used as a live view of a multimodal focal stack. For example, WLI depth map 500 can be configured for each multimodal focal stack (e.g., Figures 4A to 4B The corresponding area shown in the image is obtained and used as the live feed.

[0069] In step 206, the machine learning algorithm can be trained based on a set of acquired training multimodal focus stacks 101 and received 3D live data. For example, controller 110 can be configured to train the machine learning algorithm based on a set of acquired training multimodal focus stacks 101 and received 3D live data. For example, controller 110 can be configured to train the machine learning algorithm based on a set of acquired training multimodal focus stacks 101 and corresponding received WLI depth maps.

[0070] Controller 110 can be configured to train machine learning algorithms using any techniques known in the field, including but not limited to supervised learning and the like. For example, in the context of supervised learning, training images may contain a set of multimodal focus stacks for training the machine learning algorithm. In this regard, controller 110 may receive training multimodal focus stacks and live data. Thus, training focus stacks and live 3D data (e.g., WLI depth maps) can be used as input to train the machine learning algorithm.

[0071] It should be further noted in this document that the machine learning algorithm trained in step 206 may include any type of machine learning algorithm and / or deep learning technique or algorithm known in the relevant field, including but not limited to convolutional neural networks (CNNs), generative adversarial networks (GANs), modular neural networks, transformers, and the like. In this regard, the machine learning algorithm may include any algorithm or predictive model configured to generate one or more depth maps of the sample, as will be discussed in further detail herein.

[0072] Furthermore, it should be noted in this document that the training described can also be performed on an external image processing system that is not part of the optical system 100. Thus, the trained machine learning algorithm can be provided to the controller 110 of the optical system 100 and used to generate a depth map (as discussed below with respect to step 216).

[0073] In step 208, the trained machine learning algorithm may be stored. For example, the controller 110 may be further configured to store the training focus stack, live 3D data, and the trained machine learning algorithm 118 in memory 114.

[0074] Figure 2B A process flowchart 210 illustrates a method for generating one or more depth maps using a trained machine learning algorithm 118 of system 100 according to one or more embodiments of the present disclosure. It should be noted that process flowchart 210 can be considered as a conceptual flowchart illustrating steps executed by / within one or more processors 112 of controller 110.

[0075] In step 212, a set of product multimodal focus stacks may be acquired. For example, controller 110 may be configured to acquire a set of product multimodal focus stacks of a product sample from optical assembly 102. As used herein, the terms "product image" or "product focus stack" may be used to refer to an image for which one or more depth maps are generated. Thus, "product image" may be distinguished from "training image," which may be considered as an image used as input to train the machine learning algorithm used in the described process.

[0076] It should be noted in this article that any discussion of obtaining training images (such as...) Figure 3The flowchart shown in 300 can be considered as being applied to acquire product images, unless otherwise stated herein. Therefore, at least a first product image can be acquired at least at a first illumination angle and at a first focal length. Furthermore, at least a second product image can be acquired at at least one or more additional illumination angles and at the first focal length. Optionally, at least one or more additional images can be acquired at the first focal length using one or more illumination wavelengths. Subsequently, additional images will be acquired at one or more distances / illumination angles.

[0077] In optional step 213, additional inputs to the machine learning algorithm may be computed. One or more additional inputs may include (but are not limited to) a depth map obtained from a single-modal focus stack using a non-learning-based algorithm, one or more processed images, one or more additional parameters, and the like. For example, controller 110 may be configured to compute one or more additional inputs to the machine learning algorithm using a non-learning algorithm based on the acquired product multimodal focus stack.

[0078] In step 214, the machine learning algorithm may receive a set of acquired product multimodal focus stacks (and optionally, additional computational inputs / parameters calculated in step 213). For example, optical assembly 102 may be configured to provide the acquired focus stacks to image processing subsystem 108.

[0079] In step 216, a machine learning algorithm can be used to generate one or more depth maps based on the acquired focus stack 103. For example, the image processing subsystem 108 can be configured via a machine learning algorithm to generate one or more depth maps of a product sample based on a set of acquired product multimodal focus stacks 103. For example, the machine learning algorithm 118 can be configured to generate one or more depth maps based on acquired product images in one or more restoration stages of an optical reshaping system.

[0080] Although embodiments of this disclosure relate to optical inspection and / or shaping systems, it is contemplated that system 100 may include any optical system known in the art. For example, embodiments of this disclosure may relate to optical manufacturing systems.

[0081] Upon further consideration, each of the embodiments of the methods described above may include any other step of any other method described herein. Furthermore, each of the embodiments of the methods described above may be performed by any of the systems described herein.

[0082] Refer again Figure 1AOne or more processors 112 may comprise any processing element known in the art. In this sense, one or more processors 112 may comprise any microprocessor-type device (e.g., graphics processing unit (GPU), computer processing unit (CPU), and the like) configured to execute algorithms and / or instructions. In embodiments, one or more processors 112 may comprise a desktop computer, mainframe computer system, workstation, graphics computer, parallel processor, or any other computer system (e.g., a network computer) configured to execute processes configured to operate system 100, as described throughout this disclosure. It should be further appreciated that the term "processor" may be broadly defined to cover any device having one or more processing elements that executes program instructions from non-transitory memory media 114. Therefore, the foregoing description should not be construed as limiting the invention but is merely illustrative.

[0083] Memory media 114 may comprise any storage medium known in the art suitable for storing program instructions and data executable by one or more associated processors 112. By way of non-limiting examples, memory media 114 may comprise non-transitory memory media. By way of additional non-limiting examples, memory media 114 may comprise, but is not limited to, read-only memory, random access memory, magnetic or optical storage devices (e.g., magnetic disks), magnetic tape, solid-state drives, and the like. It should be further noted that memory 114 may be housed together with one or more processors 112 within a common controller housing. In alternative embodiments, memory 114 may be remotely located relative to the physical location of one or more processors 112 and the controller. For example, one or more processors 112 of the controller may access remote memory (e.g., a server) accessible via a network (e.g., the Internet, an intranet, and the like).

[0084] Those skilled in the art will recognize that, for clarity of concept, the component operations, devices, objects, and accompanying discussions described herein are used as examples and various configuration modifications may be considered. Therefore, as used herein, the specific examples and accompanying discussions presented are intended to represent their more general categories. In general, the use of any specific example is intended to represent its category, and the omission of specific components, operations, devices, and objects should not be considered a limitation.

[0085] Regarding any generally plural and / or singular terms used in this document, those skilled in the art can translate them from plural to singular and / or from singular to plural if appropriate for the context and / or application. For clarity, various singular / plural arrangements may be explicitly described herein.

[0086] The topics described herein sometimes illustrate different components contained within or connected to other components. It should be understood that the architectures depicted are merely exemplary, and in fact, many other architectures can be implemented to achieve the same functionality. Conceptually, any arrangement of components achieving the same functionality is effectively “associated” to achieve the desired functionality. Therefore, any two components combined herein to achieve a particular functionality can be considered “associated” with each other to achieve the desired functionality, regardless of the architecture or intermediate components. Similarly, any two such associated components can be considered “connected” or “coupled” to each other to achieve the desired functionality, and any two components that can be suchly associated can also be considered “coupleable” to each other to achieve the desired functionality. Specific examples of coupleability include, but are not limited to, physically mateable and / or physically interactive components and / or wirelessly interactive and / or logically interactive components.

[0087] Furthermore, it should be understood that the invention is defined by the appended claims. Those skilled in the art will understand that, generally, the terms used herein and especially in the appended claims (e.g., the body of the appended claims) are generally meant as “open-ended” terms (e.g., the term “comprising” should be interpreted as “comprising but not limited to,” the term “having” should be interpreted as “at least having,” the term “comprising” should be interpreted as “comprising but not limited to,” and so on). Those skilled in the art will further understand that this intention is explicitly stated in the claims if the specific numbered introductory claim is considered, and that this intention does not exist in the absence of such a statement. For example, as an aid to understanding, the appended claims may contain the introductory phrases “at least one” and “one or more” to introduce the claim statement. However, the use of such phrases should not be interpreted as implying that any particular claim containing such a predicate is limited to an invention containing only one such predicate, even when the same claim contains the predicate phrases "one or more" or "at least one" and the indefinite article "one" (e.g., "one" should generally be interpreted as meaning "at least one" or "one or more"); the same applies to the use of definite articles used to predicate the claim. Furthermore, even when a specific number of predicate claims are explicitly stated, those skilled in the art will recognize that this statement should generally be interpreted as meaning at least the number stated (e.g., the naked statement "two statements" without other modifiers generally means at least two statements, or two or more statements). Furthermore, in examples where a convention similar to "at least one of A, B, and C and the like" is used, this construction is generally anticipated in the manner that a person skilled in the art would understand the meaning of the convention (e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together and the like). In examples where a convention similar to "at least one of A, B, or C and the like" is used, this construction is generally anticipated in the manner that a person skilled in the art would understand the meaning of the convention (e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together and the like). Those skilled in the art will further understand that any actual antonyms and / or phrases presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to encompass the possibility of including one, any, or both of the terms. For example, the phrase "A or B" would be understood to include the possibility of "A" or "B" or "A and B".

[0088] It is believed that this disclosure and its many accompanying advantages will be understood from the foregoing description, and it will be appreciated that various changes in the form, construction, and arrangement of the components can be made without departing from the disclosed subject matter or without sacrificing all its material advantages. The forms described are merely illustrative and the appended claims are intended to cover and encompass such changes. Furthermore, it should be understood that the invention is defined by the appended claims.

Claims

1. An optical system, the optical system comprising: An optical assembly configured to illuminate one or more portions of one or more samples using two or more illumination modes, wherein the two or more illumination modes include at least a first illumination mode and a second illumination mode, the first illumination mode including a first set of illumination angles, and the second illumination mode including a second set of illumination angles, wherein the second set of illumination angles is at least partially different from the first set of illumination angles. The optical assembly is configured to obtain a multimodal focal stack comprising multiple images acquired at two or more distances between the one or more samples and the front focal plane, wherein at least a first image of the multiple images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode; and An image processing subsystem communicatively coupled to the optical assembly, the image processing subsystem comprising one or more processors configured to execute a set of program instructions stored in memory, the set of program instructions being configured to cause the one or more processors to: Receive multiple training images, wherein the multiple training images contain multiple training multimodal focus stacks; Receive real-time 3D data of each of the plurality of trained multimodal focus stacks; The machine learning algorithm is trained based on the multiple training images and the received real-time 3D data. The multimodal focal stack that receives the sample from the optical assembly; and The depth map of the sample is generated using the trained machine learning algorithm and the received multimodal focus stacking of the sample.

2. The system of claim 1, wherein the first illumination mode comprises a first spectral distribution and the second illumination mode comprises a second spectral distribution, wherein the second spectral distribution is at least partially different from the first spectral distribution.

3. The system of claim 1, wherein the optical assembly comprises: One or more lighting devices, the one or more lighting devices being configured to illuminate the one or more samples at multiple lighting angles; and At least one imaging device is arranged in a fixed orientation relative to the one or more samples, the at least one imaging device being configured to acquire at least a first image of the one or more samples with a first illumination angle of the plurality of illumination angles and to acquire at least a second image of the one or more samples with a second illumination angle of the plurality of illumination angles.

4. The system according to claim 3, further comprising: One or more actuation assemblies configured to adjust at least one distance between the one or more samples and the front focal plane of the optical assembly, wherein the at least one imaging device is configured to acquire one or more images at two or more distances between the one or more samples and the front focal plane.

5. The system of claim 4, wherein adjusting at least one distance between the one or more samples and the front focal plane of the optical assembly comprises at least one of the following: Adjust the position of the optical assembly or adjust the position of one or more samples.

6. The system of claim 3, wherein the one or more processors are further configured to perform one of the following: adjusting the position of the at least one imaging device or adjusting one or more optical properties of the at least one imaging device or the optical assembly.

7. The system of claim 1, wherein the one or more processors are further configured to: The machine learning algorithm receives one or more additional inputs, which include one of the following: one or more depth maps obtained from the multimodal focus stack using a non-learning-based algorithm, one or more processed images from the multimodal focus stack, or one or more additional parameters.

8. The system of claim 1, wherein the real-time three-dimensional data comprises one or more white light interferometry depth maps.

9. The system of claim 1, wherein the machine learning algorithm comprises at least one of the following: Convolutional neural networks, generative adversarial networks, modular neural networks, or converters.

10. The system of claim 1, further comprising: One or more prefabricated reference objects, the one or more prefabricated reference objects comprising one or more three-dimensional structures having one or more known depth maps, the one or more known depth maps being configured for at least one of the following: verification of the optical assembly, calibration of the optical assembly, correction of the multimodal focus stack, or retraining of the machine learning algorithm.

11. The system of claim 1, wherein the sample comprises a printed circuit board.

12. The system of claim 1, wherein the optical system comprises an automated optical inspection system.

13. The system of claim 1, wherein the optical system comprises an automated optical shaping system.

14. An image processing system, the image processing system comprising: One or more processors configured to execute a set of program instructions stored in memory, the set of program instructions being configured to cause the one or more processors to: Receive multiple training images of one or more portions of one or more samples, the multiple training images comprising multiple training multimodal focus stacks; Receive real-time 3D data of each of the plurality of trained multimodal focus stacks; The machine learning algorithm is trained based on the multiple training images and the received real-time 3D data. A multimodal focal stack of a sample is received from an optical assembly configured to illuminate one or more portions of the sample using two or more illumination modes, wherein the two or more illumination modes include at least a first illumination mode and a second illumination mode, the first illumination mode including a first set of illumination angles, the second illumination mode including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodal focal stack including multiple images acquired at two or more distances between the sample and a front focal plane, wherein at least a first image of the multiple images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode; and The trained machine learning algorithm and the received multimodal focus stack are used to generate a depth map of the sample.

15. The system of claim 14, wherein the first illumination mode comprises a first spectral distribution and the second illumination mode comprises a second spectral distribution, wherein the second spectral distribution is at least partially different from the first spectral distribution.

16. The system of claim 14, further comprising: An optical assembly configured to obtain the multimodal focus stack.

17. The system of claim 16, wherein the optical assembly comprises: One or more lighting devices, the one or more lighting devices being configured to illuminate the sample at multiple lighting angles; and At least one imaging device is arranged in a fixed orientation relative to the sample, the at least one imaging device being configured to acquire at least a first image of the sample with a first illumination angle of the plurality of illumination angles and to acquire at least a second image of the sample with a second illumination angle of the plurality of illumination angles.

18. The system of claim 17, wherein the optical assembly further comprises: One or more actuation assemblies configured to adjust at least one distance between the sample and the front focal plane of the optical assembly, wherein the at least one imaging device is configured to acquire two or more images at the two or more distances between the sample and the front focal plane.

19. The system of claim 18, wherein adjusting at least one distance between the sample and the front focal plane of the optical assembly comprises at least one of the following: Adjust the position of the optical assembly or adjust the position of the sample.

20. The system of claim 17, wherein the one or more processors are further configured to perform one of the following: adjusting the position of the at least one imaging device or adjusting one or more optical properties of the at least one imaging device or the optical assembly.

21. The system of claim 14, wherein the one or more processors are further configured to: The machine learning algorithm receives one or more additional inputs, which include one of the following: one or more depth maps obtained from the multimodal focus stack using a non-learning-based algorithm, one or more processed images from the multimodal focus stack, or one or more additional parameters.

22. The system of claim 14, wherein the real-time three-dimensional data comprises one or more white light interferometry depth maps.

23. The system of claim 14, wherein the machine learning algorithm comprises at least one of the following: Convolutional neural networks, generative adversarial networks, modular neural networks, or converters.

24. The system of claim 14, wherein the sample comprises a printed circuit board.

25. An image processing method, the method comprising: Receive multiple training images of one or more portions of one or more samples, the multiple training images comprising multiple training multimodal focus stacks; Receive real-time 3D data of each of the plurality of trained multimodal focus stacks; The machine learning algorithm is trained based on the multiple training images and the received real-time 3D data. A multimodal focal stack of one or more samples is received from an optical assembly configured to illuminate the one or more samples using two or more illumination modes, wherein the two or more illumination modes include at least a first illumination mode and a second illumination mode, the first illumination mode including a first set of illumination angles, the second illumination mode including a second set of illumination angles, at least the second set of illumination angles being at least partially different from the first set of illumination angles, the multimodal focal stack including multiple images acquired at two or more distances between the sample and the front focal plane, wherein at least a first image of the multiple images is acquired using the first illumination mode and at least an additional image is acquired using the second illumination mode; and The trained machine learning algorithm and the received multimodal focus stack are used to generate depth maps of the one or more samples.

Images