Multi-magnification high-resolution semiconductor detection microscopic system sharing light path

Abstract

The invention provides a multi-magnification high-resolution semiconductor detection microscopic system with a shared light path, which relates to the field of optical microscopic detection, and comprises an optical imaging module comprising a light source, a light splitting assembly, an objective lens, a first imaging device and a second imaging device, a detected target, the objective lens, the light splitting assembly and the first imaging device form a first imaging path, and the second imaging device forms a second imaging path; the detected target, the objective lens, the light splitting assembly and the second imaging device form a second imaging path, the resolution of the imaging image of the first imaging device is greater than that of the imaging image of the second imaging device, and the light source, the light splitting assembly, the objective lens and the detected target form an illumination path; the image processing module is used for generating a dynamic reconstruction parameter based on the imaging image of the first imaging device and the imaging image of the second imaging device, and reconstructing the imaging image of the second imaging device through an image reconstruction model based on the dynamic reconstruction parameter to generate a reconstructed image, and the resolution of the reconstructed image is greater than that of the imaging image of the second imaging device.

Description

Technical Field

[0001] The present invention relates to the field of optical microscopic detection, in particular to a multi-magnification and high-resolution shared optical path semiconductor detection microscopic system. Background Art

[0002] In the field of optical microscopy, achieving high-resolution imaging, flexible switching of multiple magnifications, and ensuring a compact system structure and efficient operation have always been key technical challenges that need to be overcome.

[0003] Traditional microscopy equipment exhibits numerous inherent flaws in its design and application. Regarding light source systems, they suffer from poor stability and are susceptible to environmental factors (such as temperature fluctuations and power supply disturbances) as well as their own aging process. This leads to unpredictable variations in the output light intensity and spectral characteristics, which in turn affect image contrast and clarity. Regarding optical path structure, traditional designs are often complex, employing multiple sets of independent optical components and discrete optical path layouts. This approach results in lengthy optical paths and increases optical signal losses during propagation, including reflection, absorption, and scattering losses on the optical component surfaces. This significantly reduces the effective light flux reaching the detector, ultimately leading to a decline in image quality and making it difficult to meet the requirements of high-resolution imaging. Furthermore, the independent optical path design of traditional microscopy devices results in independent imaging systems operating independently, lacking an effective collaborative working mechanism. This not only significantly increases the overall complexity and size of the system, and raises the costs of equipment manufacturing, installation, and maintenance, but also introduces potential interference between optical signals between independent optical paths, such as stray light crosstalk and aberration coupling, further deteriorating image quality. Although shared optical path technology has emerged in recent years, by integrating the optical paths of imaging devices with different magnifications, it has achieved the sharing of some optical components, which has improved the integration of the system to a certain extent. However, the existing shared optical path technology still has obvious balancing problems between achieving high integration and maintaining high resolution and system stability. Specifically, in the pursuit of high integration, it is often difficult to take into account key performance indicators such as aberration correction and beam quality maintenance of the optical system. As a result, in high-resolution imaging applications, parameters such as imaging sharpness, signal-to-noise ratio, and spatial resolution are difficult to reach ideal levels; at the same time, the stability of the system also faces challenges, such as poor environmental adaptability and performance drift under long-term operation, which limits its widespread application in high-precision detection and scientific research.

[0004] Therefore, it is necessary to provide a multi-magnification, high-resolution, shared optical path semiconductor detection microscope system to improve the accuracy and efficiency of optical microscopy. Summary of the Invention

[0005] The present invention provides a multi-magnification, high-resolution, shared optical path semiconductor detection microscope system, comprising: an optical imaging module, comprising a light source, a spectroscopic component, an objective lens, a first imaging device and a second imaging device, wherein the detected target, the objective lens, the spectroscopic component and the first imaging device constitute a first imaging path, the detected target, the objective lens, the spectroscopic component and the second imaging device constitute a second imaging path, the resolution of the imaging image of the first imaging device is greater than the resolution of the imaging image of the second imaging device, and the light source, the spectroscopic component, the objective lens and the detected target constitute an illumination path; an image processing module, for generating dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, and reconstructing the imaging image of the second imaging device based on the dynamic reconstruction parameters through an image reconstruction model to generate a reconstructed image, wherein the resolution of the reconstructed image is greater than the imaging image of the second imaging device.

[0006] Furthermore, the image processing module generates dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, including: aligning the imaging image of the first imaging device and the imaging image of the second imaging device through a field of view alignment model; extracting a first local image corresponding to the coordinates of the imaging image of the first imaging device after the field of view alignment from the imaging image of the second imaging device after the field of view alignment; and generating dynamic reconstruction parameters based on the imaging image of the first imaging device after the field of view alignment and the first local image.

[0007] Furthermore, the field of view alignment model is:

[0008] (x HR ,y HR )=k×(x LR ,y LR )+Δ

[0009] Among them, (x HR ,y HR ) is the pixel coordinate of the imaged image of the first imaging device, (x LR ,y LR ) is the pixel coordinate of the imaging image of the second imaging device, k is the scaling factor, and Δ is the field of view offset.

[0010] Furthermore, the image processing module reconstructs the imaging image of the second imaging device based on dynamic reconstruction parameters through an image reconstruction model to generate a reconstructed image, including: dividing the imaging image of the first imaging device into multiple second local images; for each second local image, reconstructing the second local image based on dynamic reconstruction parameters through an image reconstruction model to generate a reconstructed second local image; and splicing multiple reconstructed second local images to generate a reconstructed image.

[0011] Furthermore, the image reconstruction model includes an encoder, a decoder, a skip connection and an output layer, wherein the encoder includes multiple downsampling blocks and an encoder cross-modal attention unit, the decoder includes multiple upsampling blocks and a decoder cross-modal attention unit, and the skip connection is used to connect multiple downsampling blocks and multiple upsampling blocks, wherein the dynamic reconstruction parameters include at least the dynamic weight matrix and bias vector of the upsampling block.

[0012] Furthermore, the image processing module generates dynamic reconstruction parameters based on the imaging image of the first imaging device after the field of view is aligned and the first local image, including: extracting, through a feature extractor, a first feature map corresponding to the imaging image of the first imaging device after the field of view is aligned and a second feature map corresponding to the first local image; performing channel splicing on the first feature map and the second feature map to generate a joint feature vector; generating dynamic reconstruction parameters according to the joint feature vector through a parameter generation model, wherein the loss function of the parameter generation model includes at least an L2 regularization term of the joint feature vector.

[0013] Furthermore, the downsampling block includes two consecutive 3*3 convolutional layers, a ReLU activation function layer and a maximum pooling layer; the upsampling block includes two consecutive 3*3 convolutional layers, a ReLU activation function layer, a 2*2 deconvolution layer and an attention layer.

[0014] Furthermore, the skip-layer connection includes multiple residual blocks, wherein the input of the residual block includes the output of a decoder convolution block and the output of an encoder convolution block of the encoder, and the residual block includes two convolution layers and a ReLU activation function layer.

[0015] Furthermore, the first feature map includes at least edge information, texture information and structural information; the second feature map corresponding to the first partial image includes at least overall structural information, background information and fuzzy features.

[0016] Furthermore, the training sample set used to train the image reconstruction model includes training samples simulating different lighting conditions and / or noise levels.

[0017] Compared with the prior art, the multi-magnification, high-resolution, shared optical path semiconductor detection microscope system provided by the present invention has at least the following beneficial effects:

[0018] 1. Through a shared optical path design, the system reduces the number of optical components (such as lenses and reflectors) required for independent optical paths, thereby simplifying the optical structure and reducing system complexity and cost. The shared optical path ensures more efficient utilization of the light energy emitted by the light source, reduces light energy loss caused by independent optical paths, and improves the overall optical signal utilization efficiency of the system. The shared optical path design makes the system more compact and achieves high integration, facilitating the arrangement of more functional modules within a limited space, thereby improving the system's space utilization.

[0019] 2. The image processing module reconstructs the low-resolution image from the second imaging device to generate a higher-resolution image. This intelligent resolution enhancement method effectively improves the imaging quality of the low-resolution optical path and enhances the device's overall detection capabilities. The ability to optimize parameters in real time based on the characteristics of the input image enables the system to adaptively handle a variety of complex detection scenarios, improving its robustness and accuracy. The optical consistency of the shared optical path provides physical constraints for the training of the image reconstruction model, reducing the blindness of data-driven algorithms and enabling the algorithm to more accurately learn the characteristics of the optical system, thereby generating more precise reconstructed images. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0021] Figure 1 is a schematic diagram of a module of a multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to some embodiments of this specification;

[0022] Figure 2 is a schematic structural diagram of an optical imaging module according to some embodiments of this specification;

[0023] Figure 3 is a schematic diagram of a process for generating a reconstructed image according to some embodiments of this specification;

[0024] Figure 4 is a schematic structural diagram of an image reconstruction model according to some embodiments of this specification;

[0025] Figure 5 This is a schematic diagram of a process for reconstructing an image of a second imaging device according to some embodiments of this specification. DETAILED DESCRIPTION

[0026] To more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly describes the drawings required for describing the embodiments. Obviously, the drawings described below are merely examples or embodiments of this specification. Those skilled in the art can apply this specification to other similar scenarios based on these drawings without inventive effort. Unless otherwise apparent from the context or otherwise noted, the same reference numerals in the figures represent the same structure or operation.

[0027] Figure 1 is a schematic diagram of a module of a multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to some embodiments of this specification, such as Figure 1 As shown, the multi-magnification and high-resolution shared optical path semiconductor detection microscope system may include an optical imaging module and an image processing module.

[0028] Figure 2 is a schematic structural diagram of an optical imaging module according to some embodiments of this specification, such as Figure 2 As shown, the optical imaging module includes a light source, a light splitting component (i.e. Figure 2 ), objective lens, first imaging device (including Figure 2 The optical mechanism A and camera A shown in FIG) and the second imaging device (including Figure 2 The optical mechanism B and camera B shown in the figure) are as follows: the detected target, the objective lens, the spectroscopic component and the first imaging device constitute a first imaging path; the detected target, the objective lens, the spectroscopic component and the second imaging device constitute a second imaging path; the resolution of the image formed by the first imaging device is greater than the resolution of the image formed by the second imaging device; the light source, the spectroscopic component, the objective lens and the detected target constitute an illumination path.

[0029] Specifically, the light source provides lighting to illuminate the target being inspected, using white light or monochromatic light-emitting chips, which are small in size and have stable light intensity, and are suitable for the needs of microscopic devices. The light source can also include an aperture and a lens group. The aperture accurately controls the light intensity and direction to reduce scattering; the lens group optimizes light path transmission, reduces signal loss, and ensures uniform lighting. The highly integrated design of the light source reduces the connection and transmission distance between optical components, reduces the loss and interference of light signals, and ensures the stability of the light source output. By optimizing the design of the aperture and lens group, the utilization efficiency of the light signal is improved and energy waste is reduced. The light source structure can be adjusted and optimized according to different detection requirements to adapt to a variety of microscopic imaging application scenarios. The highly integrated design makes the entire light source structure more compact, reduces the volume and weight of the device, and facilitates integration into various microscopic devices.

[0030] Beam splitting component: splits the light path into two or more paths to achieve separation of different imaging paths.

[0031] Objective lens: converges the light from the target to form a preliminary image.

[0032] The first imaging device: a high-resolution imaging device, suitable for observing small-scale details and capable of performing detailed analysis of discovered defects.

[0033] Second imaging device: A low-resolution imaging device suitable for a wide-area overview, capable of quickly scanning large areas to detect potential defects. The image generated by the first imaging device has sub-micron accuracy, but the field of view is small, covering only a local area of ​​the low-resolution optical path imaging area (for example, 40%-50% of the low-resolution image). The second imaging device provides a large field of view image (covering the entire inspection area), but with a lower resolution (micron level).

[0034] The first and second imaging devices are equipped with high-resolution cameras and precision optical components, enabling high-precision imaging at the hundred-nanometer level, meeting the requirements for precise semiconductor device inspection. Optimized optical design further enhances image clarity and contrast. The first and second imaging devices can include lens arrays composed of multiple lenses with varying magnifications (or focal lengths).

[0035] The lighting path consists of: light source → spectrometer → objective lens → target to be detected.

[0036] Function:

[0037] The light emitted by the light source is guided to the objective lens through a beam splitting component (such as a beam splitter or a mirror).

[0038] The objective lens focuses the light onto the object being inspected, illuminating the target area.

[0039] The design of the illumination path ensures that the target to be inspected is illuminated uniformly or in a specific pattern (such as structured light) for subsequent imaging.

[0040] The first imaging path (high resolution) consists of: detected target → objective lens → spectroscopic component → first imaging device.

[0041] Imaging principle:

[0042] The light reflected or transmitted by the detection target is converged by the objective lens.

[0043] The light splitting component guides part of the light to the first imaging device.

[0044] The second imaging path (low resolution) consists of: detected target → objective lens → spectroscopic component → second imaging device.

[0045] Imaging principle:

[0046] After the same beam of light from the detected target is converged by the objective lens, the beam splitting component guides the other part of the light to the second imaging device.

[0047] A beam splitter (such as a beam splitter, half-mirror, or prism) splits the light from the objective lens into two or more paths. For example, 50% of the light is reflected to the first imaging device, and 50% is transmitted to the second imaging device (or other distribution ratios). This ensures that both light paths reach their respective imaging devices simultaneously, achieving synchronized imaging.

[0048] It can be understood that by sharing the optical path, two imaging devices with different magnifications can share some optical components, such as the beam splitter component and the objective lens, thereby reducing the complexity and volume of the system.

[0049] The light intensity distribution ratio of the light splitting component will affect the signal-to-noise ratio of the images of the first imaging device and the second imaging device. Therefore, the light intensity distribution ratio must meet the following requirements:

[0050]

[0051] Among them, I HR is the light intensity of the first imaging path, I LR is the light intensity of the second imaging path, η HR and η LR are the light energy utilization rates for the first and second imaging paths, respectively. In an imaging system, signal strength is typically proportional to the light intensity I reaching the imaging device. Noise sources may include photon noise (proportional to the square root of the light intensity), readout noise, dark current noise, and so on. Therefore, higher light intensity and a stronger signal generally result in a higher signal-to-noise ratio (assuming the noise growth rate is lower than the signal growth rate). Furthermore, the light splitting ratio of the first imaging path is greater than that of the second imaging path.

[0052] The light intensity distribution ratio of the light splitting component (such as a beam splitter, a semi-transparent mirror, etc.) can be controlled by adjusting its reflectivity, transmittance or angle. As an example only, the light intensity distribution ratio of the light splitting component can be set according to human experience.

[0053] Preferably, the light intensity distribution ratio of the light splitting component can be determined based on the following process:

[0054] For each imaging task (e.g., semiconductor wafer inspection, semiconductor packaging inspection, thin film material inspection, etc.), determine the optimal light intensity distribution ratio corresponding to the imaging task;

[0055] The optimal light intensity distribution ratio corresponding to the current imaging task is retrieved as the light intensity distribution ratio of the light splitting component.

[0056] For each imaging task, the optimal light intensity distribution ratio corresponding to the imaging task can be determined according to the following method:

[0057] Determining multiple candidate light intensity distribution ratios, wherein the multiple candidate light intensity distribution ratios can be set according to manual experience;

[0058] For each candidate light intensity distribution ratio, dynamic reconstruction parameters are generated based on an image of a target under the imaging task acquired by the first imaging device and an image of a target under the second imaging device under the candidate light intensity distribution ratio, and the image of the target under the second imaging device is reconstructed by an image reconstruction model based on the dynamic reconstruction parameters to generate a reconstructed image.

[0059] Acquire a true high-resolution image of a detected target of an imaging task by a first imaging device;

[0060] For each candidate light intensity distribution ratio, the image similarity between the reconstructed image and the real high-resolution image of the detected target of the imaging task is calculated. For example, the average of the squares of the differences in the grayscale values ​​(or RGB values) of the corresponding pixels in the reconstructed image and the real high-resolution image of the detected target of the imaging task is calculated. The smaller the value, the more similar the reconstructed image and the real high-resolution image of the detected target of the imaging task are. For another example, representative features such as edges, corners, textures, etc. are extracted from the reconstructed image and the real high-resolution image of the detected target of the imaging task. The features extracted from the two images are compared through a feature matching algorithm (for example, a Brute-Force Matcher algorithm). The similarity of the two images is calculated based on indicators such as the number of matched feature points and the matching quality. For example, the ratio of the number of matched feature points to the total number of feature points can be calculated as the similarity;

[0061] The light intensity allocation ratio of the candidate with the highest image similarity is taken as the optimal light intensity allocation ratio corresponding to the imaging task.

[0062] An image processing module is used to generate dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, and reconstruct the imaging image of the second imaging device based on the dynamic reconstruction parameters through an image reconstruction model to generate a reconstructed image, wherein the resolution of the reconstructed image is greater than that of the imaging image of the second imaging device.

[0063] Figure 3 is a schematic diagram of a process for generating a reconstructed image according to some embodiments of this specification, Figure 5 is a schematic diagram of a process for reconstructing an image of a second imaging device according to some embodiments of this specification, such as Figure 3 and Figure 5As shown, the image processing module generates dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, including:

[0064] The imaging image of the first imaging device (ie Figure 5 high-resolution image (HR) in the image) and the imaged image of the second imaging device (i.e. Figure 5 Field of view alignment (using low-resolution images in the image above);

[0065] Extract the first partial image (i.e., the coordinates of the image of the first imaging device after the field of view is aligned) corresponding to the image coordinates of the image of the second imaging device after the field of view is aligned. Figure 5 LR_sub in ), wherein the pixel coordinates of the first partial image are consistent with the imaging image coordinates of the first imaging device after the field of view is aligned;

[0066] Dynamic reconstruction parameters are generated based on the imaging image of the first imaging device and the first partial image after the fields of view are aligned.

[0067] As a preference, the field of view alignment model is:

[0068] (x HR ,y HR )=k×(x LR ,y LR )+Δ

[0069] Among them, (x HR ,y HR ) is the pixel coordinate of the imaged image of the first imaging device, (x LR ,y LR ) are the pixel coordinates of the imaged image of the second imaging device, k is the scaling factor, which can be determined based on the magnification and detector pixel size of the first imaging device and the second imaging device, and Δ is the field of view offset, which can be determined through calibration experiments.

[0070] For example, the scaling factor can be calculated according to the following formula;

[0071]

[0072] Among them, β HR is the magnification of the first imaging device, S HR is the detector pixel size of the first imaging device, β LR is the magnification of the second imaging device, S LR is the detector pixel size of the second imaging device.

[0073] The field of view offset represents the spatial position deviation between the first imaging device and the second imaging device, and reflects the difference in pixel coordinates when the centers of the fields of view of the two are not completely aligned.

[0074] The factors affecting the field of view offset may include at least:

[0075] Mechanical installation error:

[0076] Deviations in the physical mounting position of the imaging device can cause field of view deviations.

[0077] Differences in optical systems:

[0078] Optical factors such as lens distortion and non-parallel optical axes may introduce additional offsets.

[0079] The field of view offset can be determined by chessboard calibration:

[0080] A checkerboard calibration plate of known size is placed in the field of view to ensure that the common field of view of the first imaging device and the second imaging device is covered.

[0081] The imaging images of the first imaging device and the second imaging device are respectively acquired.

[0082] Use a calibration object with obvious features (such as a circle or a cross mark) to calculate the offset through a feature matching algorithm.

[0083] like Figure 3 、 Figure 5 As shown, the image processing module generates dynamic reconstruction parameters based on the imaging image of the first imaging device after the field of view is aligned and the first partial image, including:

[0084] Extracting, by a feature extractor, a first feature map corresponding to the imaged image of the first imaging device after the field of view alignment and a second feature map corresponding to the first partial image, wherein the first feature map includes at least edge information, texture information, and structural information, and the second feature map corresponding to the first partial image includes at least overall structural information, background information, and blur features. The feature extractor may be the first four layers of ResNet-18;

[0085] Perform channel splicing on the first feature map and the second feature map to generate a joint feature vector;

[0086] Generate a model by parameters (i.e. Figure 5 The neural network shown in FIG5 generates dynamic reconstruction parameters according to the joint feature vector, wherein the loss function of the parameter generation model includes at least an L2 regularization term of the joint feature vector to prevent the dynamic parameters from deviating excessively from the initial distribution:

[0087]

[0088] Among them, L reg is the L2 regularization loss, θ d Dynamic reconstruction parameters are generated.

[0089] Specifically, edge information: the clarity and accuracy of the object's outline, such as the edge of the interface between different materials in a semiconductor wafer.

[0090] Texture information: complex patterns on the surface of an object, such as the fine texture on the surface of a wafer.

[0091] Structural information: Small objects or features in an image, such as tiny defects on a wafer.

[0092] Overall structural information: The general shape and layout of the target area, such as the macrostructure of a certain area on the wafer.

[0093] Background information: The environment and context of the target area, such as the material distribution around the target area on the wafer.

[0094] General features: fuzzy features of the target area, such as color, grayscale distribution, etc.

[0095] The core of the parameter generation model is based on the HyperNetwork architecture, which can be expressed in mathematical form as follows:

[0096] θ d =H(φ;I HR ,I LR_sub )

[0097] in:

[0098] θ d : A dynamically generated parameter vector that acts on a specific layer of the backbone network (such as the upsampling module).

[0099] H: Hypernetwork function, implemented by a multi-layer perceptron (MLP) or a lightweight convolutional network.

[0100] φ: Fixed parameter of the hypernetwork itself.

[0101] I HR and I LR_sub : An imaging image inputted by the first imaging device and its corresponding first partial image.

[0102] like Figure 3 、 Figure 5 As shown, the image processing module reconstructs the imaging image of the second imaging device based on the dynamic reconstruction parameters through the image reconstruction model to generate a reconstructed image, including:

[0103] dividing the imaged image of the first imaging device into a plurality of second partial images;

[0104] For each second local image, the image reconstruction model (i.e. Figure 5 The backbone network shown in FIG5 reconstructs the second partial image based on the dynamic reconstruction parameters to generate a reconstructed second partial image;

[0105] The reconstructed second partial images are stitched together to generate a reconstructed image (i.e. Figure 5 High-resolution, wide-field image shown).

[0106] Figure 4 is a schematic diagram of the structure of the image reconstruction model shown in some embodiments of this specification, such as Figure 4 As shown, the preferred image reconstruction model includes an encoder, a decoder, a skip connection and an output layer, wherein the encoder includes multiple downsampling blocks and an encoder cross-modal attention unit, the decoder includes multiple upsampling blocks and a decoder cross-modal attention unit, and the skip connection is used to connect multiple downsampling blocks and multiple upsampling blocks, wherein the dynamic reconstruction parameters include at least the dynamic weight matrix and bias vector of the upsampling block. The downsampling block includes two consecutive 3*3 convolution layers, a ReLU activation function layer and a maximum pooling layer; the upsampling block includes two consecutive 3*3 convolution layers, a ReLU activation function layer, a 2*2 deconvolution layer and an attention layer. The skip connection includes multiple residual blocks, wherein the input of the residual block includes the output of a decoder convolution block and the output of an encoder convolution block of the encoder, and the residual block includes two convolution layers and a ReLU activation function layer.

[0107] Specifically, the image reconstruction model is built based on Res-Unet. Res-UNet is designed based on UNet and residual neural network.

[0108] The UNet architecture is characterized by its U-shaped structure, consisting of an encoder (downsampling) and a decoder (upsampling). The encoder extracts high-level image features through convolution and pooling, while the decoder restores the image's spatial resolution through upsampling and convolution. It also utilizes skip-layer connections. During upsampling, the feature maps from the downsampling process are concatenated with the feature maps from the current layer, fusing feature information from different layers and preserving more detail. These skip-layer connections combine the local features of shallow networks with the global features of deep networks, enabling better capture of multi-scale image information.

[0109] The image reconstruction model incorporates an attention mechanism after the convolutional layers. Global average pooling and fully connected layers are added after the convolutional layers to learn inter-channel relationships, generate channel attention weights, and weight the original feature maps. This mechanism is added to the decoder to focus on local image details. Spatial attention weights are obtained by concatenating feature maps, performing convolution, and performing sigmoid activation, and then weighting the feature maps. This helps focus on key information and enhances the model's expressiveness.

[0110] The skip connection operation increases the depth of the network, prevents overfitting, and improves the accuracy of the model.

[0111] Cross-modal attention mechanisms include channel attention and spatial attention, enhancing the ability to express details in key areas. The combination of channel attention and spatial attention comprehensively models features and improves the model's perception of input. Channel attention focuses on enhancing the representation of key feature channels, while spatial attention highlights important areas in the image, thereby improving the signal-to-noise ratio of features. This effectively improves performance, generalization, and computational efficiency.

[0112] The channel attention mechanism is a feature recalibration technique that enhances the network's feature representation capabilities by learning the importance weights of each channel. The basic idea is that in a convolutional neural network (CNN), feature maps from different channels may contain different important information. By assigning a weight to each channel, useful information can be enhanced and useless information can be suppressed.

[0113] principle:

[0114] Global average pooling and global maximum pooling: First, global average pooling and global maximum pooling are performed on the input feature map to obtain two 1×1×C feature maps.

[0115] Multilayer Perceptron (MLP): These two feature maps are then fed into a shared multilayer perceptron (MLP) to obtain two 1×1×C feature maps. The number of neurons in the first layer of the MLP is C / r, the activation function is ReLU, and the number of neurons in the second layer is C.

[0116] Weight calculation: Finally, the result of the MLP output is added and then mapped through the Sigmoid activation function to obtain the channel attention weight matrix.

[0117] The spatial attention mechanism aims to enable the model to adaptively learn the attention weights of different regions by introducing an attention module. In this way, the model can pay more attention to important image regions and ignore unimportant regions.

[0118] principle

[0119] Channel dimension pooling: First, global maximum pooling and global average pooling are performed on the input feature map in the channel dimension to obtain two H×W×1 feature maps.

[0120] Feature concatenation: Then, the two feature maps are concatenated according to the channel, and the resulting feature map size is H×W×2.

[0121] Convolution operation: Finally, the concatenated result is convolved to obtain a feature map of size H×W×1, and then the spatial attention weight matrix is ​​obtained through the Sigmoid activation function.

[0122] Hybrid Attention Mechanism

[0123] Channel attention and spatial attention can be combined in series or parallel to enhance the model's feature expression capabilities. The CBAM used is a typical hybrid attention mechanism model. It first processes the input feature map through the channel attention module; the obtained result is then processed by the spatial attention module to finally obtain the adjusted features.

[0124] The Sigmoid function is used in the attention mechanism. The function expression of Sigmoid is: The output of the Sigmoid function ranges from 0 to 1, which is consistent with the range of probability values. In some attention mechanisms, we can interpret the output of the sigmoid function as a probability, such as the probability that a feature is focused on. This probabilistic interpretation helps the interpretability of the model and can be better combined with other probability-based modules or loss functions.

[0125] The mathematical expression for ReLU is f(x) = max(0, x). The ReLU function graph outputs 0 when the input is less than 0, and directly outputs the input value when the input is greater than or equal to 0. ReLU introduces nonlinearity into neural networks, enabling them to learn complex patterns and features. By applying nonlinear transformations to input values, neural networks can better fit various complex functional relationships.

[0126] The training sample set used to train the image reconstruction model includes training samples under simulated different lighting conditions and / or noise levels.

[0127] During the training process, slightly randomized parameters are added to different parts to improve the robustness of the model. For example, by randomly transforming the input data (such as adding noise, translation, rotation, scaling, etc.), the size and diversity of the training set are expanded, allowing the model to learn the various changing characteristics of the input data and improve its adaptability to different inputs.

[0128] The loss function of the image reconstruction model is:

[0129] L=MAE+5*MSE

[0130]

[0131] Among them, L is the loss value, MAE is the mean absolute error, MSE is the mean square error, y i is the true value, is the predicted value.

[0132] As you can understand, in the above loss functions, mean absolute error is insensitive to outliers and can provide sparse solutions, making the model more robust; mean squared error can accelerate convergence. The combination of these two benefits enhances the robustness and performance of the model. Mean squared error can quickly reduce the error and accelerate convergence when the error is large; mean absolute error can fine-tune the model when the error is small, allowing it to converge more stably to a better solution.

[0133] Before training the image reconstruction model, you can use Kaiming initialization to initialize the weights. By properly setting the initial values ​​of the weights, the signal can maintain stable amplitude during the forward and backward propagation of the neural network, thereby alleviating the problems of gradient vanishing and gradient exploding, and enabling more efficient network training. Through a scientific initialization strategy, the network can quickly converge in the right direction at the beginning of training, reducing training time and improving training efficiency.

[0134] The image reconstruction model is trained in stages. The backbone network is pre-trained using the dataset, the fixed parameter part is initialized, and the backbone network is jointly optimized in combination with the actual collected image pairs.

[0135] The image processing module is also equipped with a feedback optimization mechanism:

[0136] Defect sample library: Add defect images found during inspection to the training set and update the model regularly.

[0137] Incremental learning: Adopting Elastic Weight Consolidation (EWC) technology to avoid new data covering existing knowledge.

[0138] This multi-magnification, high-resolution, shared optical path semiconductor inspection microscope system is used to inspect wafers in semiconductor production lines. An integrated light source illuminates the wafer surface, and light from the light source is distributed through a shared optical path to two imaging devices at different magnifications. This enables both large-scale scanning of the wafer surface and narrow-scale, detailed observation, enabling rapid defect detection and analysis. The stable output of the light source ensures consistent image contrast at both magnifications, reducing false detection rates. Simultaneously, a neural network-based resolution enhancement algorithm processes images captured along the low-resolution optical path, further improving image quality and enabling more accurate identification of minute defects.

[0139] In semiconductor packaging inspection, this multi-magnification, high-resolution, shared optical path semiconductor inspection microscope system is used to perform high-resolution imaging of packaged devices. By sharing the optical path, the two magnification imaging devices can be rapidly switched to inspect internal package structures and defects, ensuring product quality. Combined with a neural network algorithm, low-resolution images of complex structural areas are optimized to improve inspection accuracy and reliability.

[0140] Finally, it should be understood that the embodiments described in this specification are intended only to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly described and illustrated in this specification.

Claims

1. A multi-magnification, high-resolution, shared optical path semiconductor inspection microscope system, characterized in that: include: An optical imaging module comprising a light source, a spectroscopic component, an objective lens, a first imaging device, and a second imaging device, wherein the detected target, the objective lens, the spectroscopic component, and the first imaging device constitute a first imaging path, and the detected target, the objective lens, the spectroscopic component, and the second imaging device constitute a second imaging path. The resolution of an image formed by the first imaging device is greater than the resolution of an image formed by the second imaging device. The light source, the spectroscopic component, the objective lens, and the detected target constitute an illumination path. An image processing module is used to generate dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, and reconstruct the imaging image of the second imaging device based on the dynamic reconstruction parameters through an image reconstruction model to generate a reconstructed image, wherein the resolution of the reconstructed image is greater than that of the imaging image of the second imaging device.

2. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 1, characterized in that: The image processing module generates dynamic reconstruction parameters based on the imaging image of the first imaging device and the imaging image of the second imaging device, including: Performing field of view alignment on the imaging image of the first imaging device and the imaging image of the second imaging device using a field of view alignment model; Extracting, from the imaged image of the second imaging device after the field of view is aligned, a first partial image corresponding to the coordinates of the imaged image of the first imaging device after the field of view is aligned; Dynamic reconstruction parameters are generated based on the imaging image of the first imaging device and the first partial image after the fields of view are aligned.

3. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 2, characterized in that: The field of view alignment model is: (x HR ,and HR )=k×(x LR ,and LR )+Δ Among them, (x HR ,y HR ) is the pixel coordinate of the imaged image of the first imaging device, (x LR ,y LR ) is the pixel coordinate of the imaging image of the second imaging device, k is the scaling factor, and Δ is the field of view offset.

4. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 2 or 3, characterized in that: The image processing module reconstructs the imaging image of the second imaging device based on the dynamic reconstruction parameters through the image reconstruction model to generate a reconstructed image, including: dividing the imaged image of the first imaging device into a plurality of second partial images; For each second partial image, reconstruct the second partial image based on the dynamic reconstruction parameters using the image reconstruction model to generate a reconstructed second partial image; The multiple reconstructed second partial images are stitched together to generate a reconstructed image.

5. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 4, characterized in that: The image reconstruction model includes an encoder, a decoder, a skip connection and an output layer, wherein the encoder includes multiple downsampling blocks and an encoder cross-modal attention unit, the decoder includes multiple upsampling blocks and a decoder cross-modal attention unit, and the skip connection is used to connect multiple downsampling blocks and multiple upsampling blocks, wherein the dynamic reconstruction parameters include at least the dynamic weight matrix and bias vector of the upsampling block.

6. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 5, characterized in that: The image processing module generates dynamic reconstruction parameters based on the imaged image of the first imaging device and the first partial image after the field of view is aligned, including: Extracting, by a feature extractor, a first feature map corresponding to the imaging image of the first imaging device after the field of view is aligned and a second feature map corresponding to the first partial image; Perform channel concatenation on the first feature map and the second feature map to generate a joint feature vector; Dynamic reconstruction parameters are generated according to the joint feature vector through a parameter generation model, wherein the loss function of the parameter generation model includes at least an L2 regularization term of the joint feature vector.

7. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 5, characterized in that: The downsampling block includes two consecutive 3*3 convolutional layers, a ReLU activation function layer and a maximum pooling layer; The upsampling block includes two consecutive 3*3 convolutional layers, a ReLU activation function layer, a 2*2 deconvolution layer and an attention layer.

8. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 7, characterized in that: The skip-layer connection includes multiple residual blocks, wherein the input of the residual block includes the output of a decoder convolution block and the output of an encoder convolution block of the encoder, and the residual block includes two convolution layers and a ReLU activation function layer.

9. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to claim 6, characterized in that: The first feature map includes at least edge information, texture information and structure information; The second feature map corresponding to the first partial image includes at least overall structural information, background information and blur features.

10. The multi-magnification, high-resolution, shared optical path semiconductor detection microscope system according to any one of claims 1 to 3, characterized in that: The training sample set used to train the image reconstruction model includes training samples under simulated different lighting conditions and / or noise levels.

Images