An adaptive imaging dual-wavelength reflection-type fourier-lithography system

Abstract

The application provides a self-adaptive imaging dual-wavelength reflection type Fourier superposition microscopic system, which comprises the following modules: a flexible light source module for bright field and dark field cooperation, which is used for improving imaging quality, reducing image acquisition time and improving resolution of reconstructed images; a high-speed self-adaptive optical module, which is used for real-time detection of image aberration and real-time compensation of the image aberration through a deformable mirror; a microscopic imaging module, which is used for realizing super-resolution image reconstruction and three-dimensional reconstruction of depth through images of dual-wavelength channels; an operation control platform, which is used for configuring a plurality of high-performance general-purpose GPUs and controlling software operation of other modules; and an image reconstruction algorithm module, which is used for realizing efficient and general multi-mode image 2D and 3D reconstruction. The application combines a deep learning-based FPM network which fuses an imaging physical model, significantly reduces data acquisition amount and acquisition time, optimizes reconstruction structure and reconstruction stability, improves the versatility of FPM for different imaging targets and realizes efficient and general multi-mode image reconstruction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computational imaging and deep learning, and particularly relates to a self-adaptive imaging dual-wavelength reflective Fourier ptychographic microscopy system. BACKGROUND

[0002] The development of modern optics has brought three major challenges to optical microscopy and imaging systems: (1) breaking the diffraction limit of optical imaging; (2) spatial bandwidth product bottleneck; (3) high-end optical systems are costly. Computational optical imaging is a new imaging method that realizes specific imaging functions and characteristics by jointly optimizing optical systems and signal processing, and provides new means and new ideas for breaking through many limiting factors in traditional imaging systems through the organic combination of optics and algorithms. Among them, the Fourier ptychographic microscopy (FPM) is a large-field, high-resolution, quantitative phase computational microscopy imaging technology developed in recent years. This technology integrates the concepts of phase recovery and synthetic aperture. The sample is illuminated by different angles of plane wave and imaged through a low numerical aperture objective lens, resulting in the frequency spectrum of the object being shifted to the corresponding different positions on the back focal plane of the objective lens, and some frequency components that exceed the numerical aperture of the objective lens can be transmitted to the imaging plane for imaging.

[0003] However, the existing FPM usually adopts a fixed LED light source, which has low optical efficiency and poor flexibility. In dark field illumination, long exposure time is needed to obtain images with sufficient contrast. Since FPM usually needs to collect tens or even hundreds of images, the long exposure time results in low image acquisition efficiency of the existing FPM. Like other optical systems, the image quality of FPM is also affected by the optical aberration of the system. Although FPM can obtain the system optical aberration and compensate it through algorithm in the image reconstruction process, this method has a very large amount of calculation, further increasing the calculation complexity of FPM image reconstruction, and the image reconstruction efficiency is low. At the same time, the typical FPM adopts transmission illumination, which is only suitable for thin samples with good light transmission, and cannot be applied to non-transparent samples with a certain thickness (micron level). Moreover, the illumination light source of the existing system is visible light, and if it is used for micro-nano scale rough surface, the visible light will be reflected (the object surface acts as a mirror), making it difficult for the camera to capture the information of the tiny surface features. SUMMARY

[0004] The purpose of the present application is to provide a self-adaptive imaging dual-wavelength reflective Fourier ptychographic microscopy system.

[0005] The technical scheme of the present application is as follows:

[0006] A self-adaptive imaging dual-wavelength reflective Fourier ptychographic microscopy system, the system comprising:

[0007] A flexible light source module for bright field and dark field cooperation to improve imaging quality, reduce image acquisition time and improve the resolution of reconstructed images, and support Deep Ultraviolet (DUV) light source flexible illumination;

[0008] A high-speed adaptive optical module for real-time detection of image aberration and real-time compensation of aberration by controlling a deformable mirror, reducing the complexity of image reconstruction problems and reducing the computational load of image reconstruction;

[0009] A microscopic imaging module for implementing super-resolution image reconstruction and three-dimensional reconstruction of depth through images of dual-wavelength channels;

[0010] An operation control platform for configuring multiple high-performance general-purpose GPUs to provide sufficient computing power for image reconstruction using parallel computing, while centrally controlling the software operation of other modules;

[0011] An image reconstruction algorithm module for implementing efficient and versatile multi-mode image 2D and 3D reconstruction.

[0012] Further, the flexible light source module for bright field and dark field cooperation is composed of a lifting support, a rotating mirror, a bright field light source and a dark field light source, wherein the movement of the rotating mirror and the lifting support cooperates with the bright field light source and the dark field light source to realize controllable multi-angle illumination conditions using a small number of light sources.

[0013] Further, the system uses a high-optical-efficiency DUV LED package as a light source, which is composed of a substrate, a light-emitting chip, an aspherical reflecting chip, an outer shell, a micro-nano surface light uniformity sheet and a protective glass in sequence.

[0014] Further, the high-speed adaptive optical module is composed of a wavefront sensor, a beam splitter and a deformable mirror; wherein the light path is introduced into the wavefront sensor through the beam splitter to detect the system aberration in real time, and the deformable mirror is further controlled to compensate for the aberration in real time.

[0015] Further, the microscopic imaging module is composed of a stage, an objective lens, a beam splitter and a camera.

[0016] Further, the image reconstruction algorithm includes three main parts: data synthesis, residual attention module and Transformer module;

[0017] Among them, data synthesis synthesizes the original data into an amplitude tensor and a phase tensor, reducing the computational load of the network;

[0018] The residual attention module is used for effective feature extraction of the image and suppression of negative problems caused by the increase of convolutional layers;

[0019] A transformer module is used to enhance the extraction capability of the network for multi-scale information.

[0020] Further, the residual attention module extracts image features through residual learning and channel attention mechanism, wherein the residual learning accelerates the convergence of the network, and solves the network degradation problem caused by too many network layers; and the channel attention mechanism adaptively allocates weights according to the importance of different channel features, so as to emphasize important channels and suppress unnecessary channels.

[0021] Further, the image reconstruction algorithm adopts a loss function designed in combination of the space and frequency domains.

[0022] Compared with the prior art, the present application has the following advantages:

[0023] The present application supports using a high-optical-efficiency LED package as a light source, using a flexible illumination strategy to significantly improve the reconstruction image resolution limit of the FPM, and simultaneously achieving programmable adjustment of the imaging resolution without changing the FPM objective lens. In addition, combined with the dual-wavelength difference frequency technology, the thick sample depth three-dimensional reconstruction that is difficult to achieve by the traditional FPM is solved.

[0024] For opaque or reflective samples such as wafers and semiconductor packages, a reflective microscopic imaging system combined with a flexible illumination light source is used to provide high freedom and high contrast illumination and imaging capabilities.

[0025] For the problem that the FPM needs to collect a large number of low-resolution images as the basis for reconstruction, resulting in large data acquisition and long calculation time, an adaptive optical module is added to the imaging system to compensate for the imaging aberration in the front end, thereby improving the resolution of the reconstructed image and reducing the processing time of the algorithm. In addition, combined with the FPM network based on deep learning which is fused with the imaging physical model, the data acquisition amount and acquisition time are significantly reduced, the reconstruction structure and reconstruction stability are optimized, the versatility of the FPM for different imaging targets is improved, and efficient and versatile multi-mode image reconstruction is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments of the application and together with the description, serve to explain the principles of the application. Wherever possible, the same reference numbers will be used in the different drawings to refer to the same or like elements. Such embodiments are illustrative rather than restrictive and are not intended to exhaustively enumerate all possible implementations or designs of the device or method.

[0027] Figure 1 An adaptive imaging dual-wavelength reflective Fourier ptychographic system of the present application is shown in the schematic diagram;

[0028] Figure 2A high optical efficiency DUV LED packaging schematic diagram of the application is shown.

[0029] Figure 3 A high-speed adaptive optical working principle schematic and control mode comparison schematic diagram of the application is shown.

[0030] Figure 4 A target-driven image acquisition method schematic diagram of the application is shown.

[0031] Figure 5 A dual-wavelength difference frequency schematic diagram based on a filter of the application is shown.

[0032] Figure 6 A reconstruction strategy diagram based on deep learning of the application is shown.

[0033] Figure 7 A deep learning-based FPM reconstruction network structure diagram of the application is shown.

[0034] Figure 8 A residual attention module structure schematic diagram of the application is shown.

[0035] Figure 9 A Transformer module structure schematic diagram of the application is shown.

[0036] Figure 10 A schematic diagram of the application for reconstructing micro surface defects of a silicon wafer is shown.

[0037] Figure 11 A frame diagram of an adaptive imaging dual-wavelength reflective Fourier laminar microscopic system is shown. DETAILED DESCRIPTION

[0038] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0039] The application provides an adaptive imaging dual-wavelength reflective Fourier laminar microscopic system, wherein a split type bright field and dark field light source is designed, and under the assistance of an illumination angle, height freely adjustable mechanical structure, the light source lighting mode is optimized, combined with dual-wavelength imaging technology, to realize large field of view, high resolution three-dimensional information reconstruction of non-transparent / surface reflection type samples. Specifically contains five modules as shown in Figure 1 the bright field and dark field cooperative flexible light source module (M1), the high-speed adaptive optical module (M2), the microscopic imaging module (M3), the operation control platform (M4), and the image reconstruction software (M5).

[0040] Since the imaging resolution and image quality of FPM are highly dependent on the light source module, the flexible light source with the synergy of bright field and dark field can improve the imaging quality, reduce the image acquisition time and improve the resolution of the reconstructed image, and can support deep ultraviolet light source flexible lighting to realize super-resolution detail lighting; the high-speed adaptive optical module can detect image aberration in real time, and compensate the image aberration in real time through the control of the deformable mirror, reduce the complexity of image reconstruction problem, and reduce the calculation amount of image reconstruction; the microscopic imaging module supports the low-cost microscopic imaging system using the DUV waveband, and realizes the super-resolution three-dimensional image reconstruction through the images of the dual-wavelength channel; the operation control platform is configured with multiple high-performance general-purpose GPUs, and parallel computing is used to provide sufficient computing power for image reconstruction, while the software operation of other modules is concentratedly controlled; in order to improve the versatility of FPM for different imaging targets and the stability of image reconstruction, the image reconstruction algorithm and the software module expand the physical model of reflective FPM imaging, and combine it with a deep neural network (DNN) in the image reconstruction process, to realize efficient and versatile multi-mode image (2D+3D) reconstruction.

[0041] In addition, in order to meet the requirements of microscopic imaging of different samples with surface features, the light source waveband used in the application can be freely selected and replaced. If there are micron or nanometer scale structures on the surface of the sample, a short wavelength light source such as ultraviolet light or even deep ultraviolet can be used. Under the illumination of this wavelength, more directional scattering can occur on the small surface features, thereby solving the information acquisition problem of the corresponding features, and at the same time, the resolution is improved accordingly. For the sake of simplicity, the technical details of the application are described below with the DUV light source as an example, and the light sources of other wavebands follow the same technical route.

[0042] The structure design of the application is shown in Figure 1 The high-optical-efficiency DUV LED package makes the emitted light energy concentrated and uniformly distributed, and compared with the ordinary large-angle LED, the optical efficiency of the reflective FPM system can be improved by more than ten times. For the DUV flexible lighting scheme, the application adopts a light source with a wavelength of 210-200nm.

[0043] For dark-field light source, different illumination angles of the imaging target are achieved by programmable control of the combination of lifting mechanism and two-dimensional rotating mirror, and the increase of the illumination angle of the light source can increase the numerical aperture of FPM, thereby increasing the imaging equivalent numerical aperture of the FPM system (the imaging equivalent numerical aperture is the sum of the numerical aperture of the objective and the numerical aperture of the illumination). Therefore, this flexible light source module design can realize programmable adjustment of the imaging resolution without changing the FPM objective. In the frequency spectrum, the increase of the dark-field illumination numerical aperture is equivalent to expanding the high-frequency boundary of the dark-field spectrum, thereby improving the resolution limit of the reconstructed image of FPM Figure 1 Right side).

[0044] The present application introduces a high-speed adaptive optics (Adaptive Optics, AO) module based on a wavefront sensor and a deformable mirror (Deformable Mirror, DM) in FPM to realize real-time compensation of aberration Figure 3 ). This module is placed in the relay lens group of the FPM imaging light path, and the light path is introduced into the wavefront sensor real-time detection system through a beam splitter to detect the system aberration in real time, and further compensate the aberration in real time through feedback control of the DM. The existing AO module is mainly designed for the visible light band and the near-infrared band, and the main work of the present application is to design and process the microlens array, the core component of the wavefront sensor, and to adopt an image sensor supporting the DUV band. Based on the Zernike function representation of aberration, wavefront reconstruction can be completed by solving linear equations, and then the Zernike coefficients are mapped to the parameter space of the DM to control and drive it to realize aberration compensation. This sensing adaptive optics control method is shown in the upper part of the figure. Figure 3 The present application adopts an end-to-end neural network to realize the feedback control of AO, that is, the image collected by the wavefront sensor is taken as the input of the DNN, and the DM control signal is directly output.

[0045] The present application adopts Figure 4 The imaging target driven image acquisition method shown is that the position and illumination angle of the light source are determined according to the spectral characteristics of the imaging target to be reconstructed, and the ideal position and illumination angle of the light source are realized through programmable control of the flexible light source module.

[0046] The optical imaging model of the reflective FPM can be described as follows:

[0047] Assuming that the distribution function of the imaging target is o(r)=Ae iφ(r) Each LED can be regarded as a monochromatic plane wave with a wavelength of λ, and the incident angle depends on the physical position and illumination direction of the LED. Therefore, when the lth LED is on, the wavefront emitted from the imaging target surface can be represented as:

[0048]

[0049] where ξ l is the spatial frequency vector of the LED illumination direction. In this way, a frequency shift

[0050]

[0051] Due to the finite aperture of the objective, the object wavefront is equivalent to a low-pass filtering, the degree of which depends on the objective pupil function P(u). After the imaging lens (Fourier transform), the low-resolution image collected at the imaging plane can be expressed as

[0052] y l (r) = |F -1 {P(u)F{y(r)}(u - ξ l )}| 2 #(3)

[0053] For the case of multiple LEDs being turned on simultaneously, the collected image is the intensity-weighted superposition of the images illuminated by each LED

[0054]

[0055] where the non-negative scalar c l is the relative brightness of the lth LED Figure 5 .

[0056] According to the above optical propagation model, the complex field of the imaging target is reconstructed by solving the nonlinear inverse problem. The distribution function of the imaging target and the images obtained by multiple LED multiplexed illumination are discretized, respectively, and denoted as where p < q (the reconstructed high-resolution image has a better spatial bandwidth product). In this way, the reconstruction of the super-resolution image can be expressed as an optimization problem:

[0057]

[0058] where Ω k is the index set of the kth multiple LED turning on. The discretized FPM system propagation model in the above equation is A l = F H P l F, where is the matrix representation of the pupil function with a frequency shift of ξ l . F and F H are the matrix representations of the Fourier transform and inverse transform, respectively.

[0059] Although the above model covers the propagation process of the sample complex field, in order to reconstruct it in three dimensions, the imaging target is required to meet the “thin sample” assumption (the phase gradually changes within the range of 2π). For three-dimensional reconstruction of the reflective sample imaging target, the present application proposes to additionally increase the physical conditions at the illumination light source end, such as adding an adjustable ultraviolet filter on the DUV camera or having a double DUV camera with a band-pass filter, introducing a difference frequency technology Figure 5 ), to realize three-dimensional reconstruction of depth.

[0060] In the dual-wavelength difference frequency technology, the calculation formula of the synthetic wavelength is:

[0061]

[0062] Taking λ1=210nm and λ2=215nm as an example, the synthetic wavelength is about 9μm. For a sample with a depth less than this scale, the three-dimensional image of the sample can be calculated by reconstructing two two-dimensional images and using the corresponding complex phases:

[0063]

[0064] It should be pointed out that, as in other difference frequency methods, due to the periodicity of the phase, for measuring a depth exceeding the synthetic wavelength, the 3D reconstruction is non-unique.

[0065] The present application proposes a solution based on deep learning, as shown in Figure 6 , which uses a deep neural network to reconstruct a high-resolution image from a series of low-resolution images generated by the FPM system in an end-to-end manner. This solution can reduce the acquisition of original images by taking advantage of the strong generalization of deep neural networks, avoid high complexity operations in iterations, and greatly reduce the reconstruction speed.

[0066] The FPM reconstruction network structure proposed by the present application is shown in Figure 7 , the input is a series of low-resolution images captured by the LED array light source in turn, and the output is the reconstructed high-resolution amplitude and phase. The network includes three main parts: data synthesis, residual attention module and Transformer module. Data synthesis synthesizes the FPM original data into an amplitude tensor and a phase tensor, reducing the calculation amount of the network; the residual attention module combines residual connection and attention mechanism, effectively extracts features from the image and suppresses the negative problems caused by the increase of convolutional layers; the Transformer module enhances the network's ability to extract multi-scale information by providing a larger receptive field.

[0067] Data synthesis pre-processes a series of intensity images collected by FPM, synthesizes them in Fourier space, and converts them into a two-channel complex amplitude image as input by inverse Fourier transform.

[0068] The residual attention module, as shown in Figure 8 It introduces residual learning and channel attention mechanism to extract image features, where residual learning can accelerate network convergence and solve the network degradation problem caused by too many network layers, and channel attention mechanism can adaptively allocate weights according to the importance of different channel features, so as to emphasize important channels and suppress unnecessary channels.

[0069] The Transformer module, as shown in Figure 9 It is used to expand the network receptive field and enhance the expression ability of deep features. A large receptive field is very important for the network to accurately understand the scene and image reconstruction. In order to expand the receptive field of the convolutional neural network, the general solution is to introduce a dilated convolution. However, its role is limited and may cause grid effect. Therefore, the network introduces a powerful Transformer module, where each Transformer layer has a global receptive field.

[0070] The present application collects actual wafer sample data on the existing experimental platform, then uses the traditional iterative algorithm to reconstruct the data to obtain the corresponding high-resolution complex amplitude as the true value, and finally divides the training set and test set. Specifically, first, adjust the number of LED units of the programmable LED array in the experimental platform to be equally spaced 100 and 200, then take the same 1200 samples, and the samples are pre-processed to be 512*512 pixels in size. Among them, the samples obtained by 200 LED units are further updated iteratively using the alternating projection framework Fourier spectrum, and the amplitude and phase of the high-resolution image are obtained after convergence, which are used as the true value. Among the 1200 samples obtained by 100 LED units, 900 samples are used as the training set, and 300 samples are used as the test set. In addition, in order to avoid overfitting in the training process, the present application uses rotation, flipping, translation, cropping and adding noise to perform data augmentation on the training set.

[0071] The application adopts a loss function designed in a manner of combining space and frequency domains. The L1 norm between the network output value and the true value and SSIM (Structure Similarity Index Measure) are used on the space domain. Using the L1 norm can make the network learn more detailed information, improve the sparsity of the network, avoid overfitting of the network to the training data, and also increase the robustness of the network to abnormal values such as noise. The addition of SSIM can make the network focus on the macro-structure information in the image. In the frequency domain, the high-frequency information and the low-frequency information have obvious position differences. For example, the high-frequency part corresponding to the high-resolution information is located farther from the center of the frequency spectrum, while the low-frequency information is located in the center area of the frequency spectrum. Compared with the space domain, it is easier to distinguish the high-resolution information of the sample in the frequency domain.

[0072] Embodiment 1

[0073] For a traditional reflective microscopic imaging system, the field of view and the resolution cannot be guaranteed at the same time due to the magnification requirement. However, the reflective Fourier superposition microscopic imaging system mentioned in the application can guarantee a large field of view while achieving high spatial resolution. For a reflective micro-surface sample, a DUV light source can be used to obtain more imaging information and further improve the resolution. Figure 10 The simulation schematic diagram of the silicon wafer surface defect detection microscopic imaging is shown. (a) The microscopic imaging effect of the silicon wafer surface under a low magnification objective lens is shown in the figure, (b) is an enlarged view of the field of view shown in (a), (c) is a high-resolution image obtained by using the algorithm mentioned in the application, and (d) is an enlarged view of the field of view shown in (c).

[0074] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto. Any person skilled in the art can make equivalent replacements or changes to the technical solution and the inventive concept of the application within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. An adaptive imaging dual-wavelength reflective Fourier layered microscopy system, characterized in that, The system includes: A flexible light source module that combines bright and dark fields is used to improve imaging quality, reduce image acquisition time, and increase the resolution of reconstructed images, and also supports flexible illumination from deep ultraviolet light sources. The flexible light source module that combines bright field and dark field lighting consists of a lifting bracket, a rotating mirror, a bright field light source, and a dark field light source. The movement control of the rotating mirror and the lifting bracket, in conjunction with the bright field and dark field light sources, enables the formation of controllable multi-angle lighting conditions using a few light sources. The high-speed adaptive optics module is used to detect image aberrations in real time and compensate for aberrations in real time by controlling deformable mirrors, thereby reducing the complexity of image reconstruction problems and the amount of computation required for image reconstruction. The high-speed adaptive optics module consists of a wavefront sensor, a beam splitter, and a deformable mirror. The beam splitter guides the optical path to the wavefront sensor to detect system aberrations in real time, and the deformable mirror is further controlled by feedback to compensate for aberrations in real time. The high-speed adaptive optics module also includes a microlens array, a core component of the wavefront sensor designed and fabricated for the DUV band, and an image sensor that supports the DUV band. The microscopic imaging module is used to achieve super-resolution image reconstruction and three-dimensional depth reconstruction through images from dual wavelength channels; The computing control platform is used to configure multiple high-performance general-purpose GPUs, utilize parallel computing to provide sufficient computing power for image reconstruction, and centrally control the software operation of other modules. The image reconstruction algorithm module is used to achieve efficient and versatile multi-modal 2D and 3D image reconstruction. The image reconstruction algorithm module consists of three main parts: data synthesis, residual attention module, and Transformer module; Among them, data synthesis combines the original data into an amplitude tensor and a phase tensor, reducing the computational load of the network; The residual attention module is used to effectively extract features from images and suppress the negative effects of increasing the number of convolutional layers; The Transformer module is used to enhance the network's ability to extract multi-scale information.

2. The adaptive imaging dual-wavelength reflection Fourier stacked microscopy system according to claim 1, characterized in that, The system uses a high-optical-efficiency DUV LED package as the light source. The high-optical-efficiency DUV LED package is composed of a substrate, a light-emitting chip, an aspherical reflective chip, a shell, a micro-nano surface homogenizer, and a protective glass.

3. The adaptive imaging dual-wavelength reflection Fourier stacked microscopy system according to claim 1, characterized in that, The microscopic imaging module consists of a stage, objective lens, beam splitter and camera.

4. The adaptive imaging dual-wavelength reflection Fourier stacked microscopy system according to claim 1, characterized in that, The residual attention module extracts image features through residual learning and channel attention mechanisms. Residual learning accelerates network convergence and solves the network degradation problem caused by too many network layers. The channel attention mechanism adaptively allocates weights according to the importance of different channel features, thereby emphasizing important channels and suppressing unnecessary channels.

5. The adaptive imaging dual-wavelength reflection Fourier stacked microscopy system according to claim 1, characterized in that, The image reconstruction algorithm module uses a combination of spatial and frequency domain approaches to design the loss function.

Images