Method for removing motion blur of grayscale image of wafer sample based on multi-scale sensing network

Abstract

The invention discloses a motion blur removing method for a wafer sample grayscale image, and provides an end-to-end lightweight multi-scale coding and decoding neural network restoration scheme for solving the exposure integral smear problem caused by factors such as high-speed scanning of a carrier, triggering jitter, mechanical micro-vibration and a rolling shutter in wafer detection and mask plate inspection. The method comprises the following steps: acquiring or synthesizing paired'motion blurred image-clear image 'training data, and constructing grouped samples with the same background and multiple blurring degrees; a U-shaped coding-decoding network is constructed, multi-scale feature extraction and resolution recovery are realized by adopting multi-layer pixel rearrangement type down-sampling / up-sampling, and shallow texture and deep semantics are fused through jump connection; a parallel void depth separable convolution feature mixing module (multiple void rates cover smears of different scales) is introduced into a coding block and a decoding block, and meanwhile, a simple multiplication gating and a simplified channel attention mechanism are combined to suppress artifacts and stabilize the gray level; multi-scale supervision and gray scale consistency constraint loss are introduced in the decoding stage; in the reasoning stage, overlapping sliding window partitioning is adopted to adapt to the super-large resolution. According to the method, restoration can be completed without explicit estimation of a motion blurring kernel, restoration definition can be improved while gray scale consistency and fine structure integrity are kept, and the method is suitable for preorder image processing of industrial scenes such as defect detection and production line quality control and assists in improving the accuracy of subsequent tasks.

Description

Technical Field

[0001] This invention relates to the field of computer vision and industrial image restoration technology. It is a deep learning deblurring neural network that uses parallel hole depth separable convolution, simple channel attention, and simple gating mechanism to restore blurred images to clear images, maximizing the reproduction of image information to facilitate subsequent industrial inspection and other tasks. It is particularly relevant to motion blur restoration of grayscale images of wafer samples, and is also applicable to other industrial grayscale image motion blur restoration tasks. Background Technology

[0002] In wafer manufacturing and sample inspection, defect detection, CD measurement, and LER evaluation heavily rely on high-resolution grayscale images. To balance throughput and accuracy, equipment often employs stage scanning, step-by-step alignment, or continuous imaging to acquire large field-of-view images. If there is relative motion between the sample and the imaging system or sensor within the exposure integration time, motion blur occurs, manifesting as trailing and ghosting along the direction of motion, decreased detail contrast, softened line boundaries, distorted hole contours, and shifted periodic textures, thus affecting defect detectability and measurement repeatability. Essentially, this is temporal integration within the exposure window: the faster the relative motion or the longer the exposure, the more pronounced the trailing blur.

[0003] Furthermore, some industrial cameras use rolling shutter readout, meaning that pixels in each row are not exposed simultaneously. When motion or vibration is present, in addition to motion blur, temporal distortions such as tilting and bending may be introduced, resulting in spatially non-uniform degradation. In wafer sample scenarios, the motion kernel (PSF) often changes with speed fluctuations, acceleration / deceleration, mechanical resonance, and trigger jitter, and may exhibit different directions and lengths in different areas of the field of view. Simultaneously, the image contains a large amount of regular periodic texture and has high requirements for grayscale consistency. If grayscale drift, ringing, or pseudo-texture occurs during the restoration process, it will directly interfere with threshold segmentation, template matching, and CD measurement. Production lines commonly use line scan / scan imaging to cover large areas, making them more sensitive to the "motion speed - sampling readout" matching; even small synchronization errors can amplify motion-related artifacts.

[0004] Existing methods for handling motion blur can be broadly categorized into three types: hardware suppression, traditional deconvolution restoration, and deep learning-based end-to-end restoration. Hardware suppression typically reduces motion blur by shortening exposure, increasing illumination, synchronizing triggering / flickering, improving vibration reduction and control strategies, and using global shutter or specific scanning imaging schemes. However, these methods are often limited by equipment cost, illumination and noise constraints, and differences in the reflectivity of different process layers. Furthermore, they are costly to modify existing production lines, and residual blur or new imaging artifacts may still occur when stage speed fluctuations, micro-vibrations, and trigger jitter are combined. Traditional restoration methods are represented by deconvolution, typically including non-blind deconvolution methods such as Wiener filtering / regularized inverse filtering and Richardson–Lucy (RL) iteration. These methods usually require known or accurately measurable point spread function / motion kernel (PSF) and recover a clear image through model inversion. However, the motion kernel in the production line often changes over time and may be spatially non-uniform; PSF mismatch can lead to unstable restoration and artifacts such as ringing. While iterative RL methods can improve contrast, they are prone to noise amplification and poor convergence during iteration, requiring additional noise suppression / damping strategies to mitigate this noise amplification. In cases of rolling shutter speeds or spatially variable motion blur, the kernel changes at different pixel positions, making the simplified "single kernel + convolution" model less valid and further amplifying the aforementioned problems. Deep learning-based end-to-end deblurring methods can bypass explicit kernel estimation by learning a "blur-to-sharp" mapping; however, many methods lack adaptability to industrial applications, and the models used in these methods do not pay attention to certain indicators in industrial production. Furthermore, these models have high computational costs and inference expenses, and may still exhibit oversharpening and ringing artifacts when enhancing high-frequency details, which conflicts with the requirements of grayscale consistency and structural realism in industrial applications. For example, the existing invention "Method and device for identifying blurred images based on regional target detection algorithm" (CN202411700843.2) has the drawback of requiring the use of deep learning algorithm to estimate the blurred region of the image first and then Wiener filtering to deblur it, which is a cumbersome process; "A method for restoring degraded image information based on multi-sensor fusion" (CN202510211246.1) has the problem of requiring the use of multi-sensor fusion to repair the image, but the types of inspection equipment on the wafer production line are relatively fixed, and the replacement cost is high and it is not suitable.

[0005] To address the aforementioned problems, this invention discloses a method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network. This method can be used for motion blur repair of inspection images on wafer production lines and can also be extended to repair other industrial grayscale images with motion blur. This method does not require multi-device collaboration and can directly repair blurred images into clear images end-to-end, facilitating subsequent tasks. Summary of the Invention

[0006] This invention discloses a method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network. This method enables end-to-end restoration of motion-blurred grayscale images of wafer samples acquired in industrial production lines to clear grayscale images. To achieve the above objective, this invention provides a method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network, comprising the following steps:

[0007] Acquire a single-channel grayscale image of the wafer sample to be restored, captured by an industrial camera or inspection equipment, and perform normalization and size alignment preprocessing.

[0008] Construct paired or grouped supervised datasets, wherein the supervised datasets contain at least motion-blurred grayscale images and their corresponding sharp grayscale label images;

[0009] An end-to-end U-shaped encoder-decoder de-motion blurring network was constructed, which includes an input embedding layer, a multi-level encoder, a bottleneck layer, a multi-level decoder, cross-scale jump connections, and an output layer.

[0010] Parallel dilatation-depth separable convolutional feature blending, multiplication gating, and simplified channel attention mechanisms are performed in the basic blocks of the encoder and decoder.

[0011] The network was trained using multi-scale supervision and gray-scale consistency constraint loss to obtain a well-trained motion-blurred model.

[0012] The grayscale image to be restored is input into the trained model for inference and outputs the restored image. It can also be selected to use overlapping sliding window block inference and weighted fusion under ultra-high resolution conditions to obtain the whole image restoration result.

[0013] The network consists of a multi-layer encoder / decoder, an input embedding layer, a bottleneck layer, an output layer, and cross-scale skip links. Downsampling is performed by "convolution + PixelUnshuffle", and upsampling is performed by "convolution + PixelShuffle". In the decoding stage, the corresponding encoded features are concatenated in the channel dimension and then compressed through a 1×1 convolution.

[0014] The basic block includes a spatial feature hybrid sublayer and a gated feedforward sublayer. The spatial feature hybrid sublayer includes LayerNorm, 1×1 convolution, 3×3 depthwise separable convolution, parallel dilated depthwise separable convolution branch, fusion, SimpleGate, simplified channel attention mechanism, 1×1 convolution back-roll and residual connection.

[0015] The preferred void ratios for parallel void branches in the basic block are 1, 3, 7, and 9.

[0016] The multi-scale supervision generates intermediate restoration results at multiple scale outputs of the decoder. After upsampling to the original resolution, the reconstruction loss is calculated with the label and then weighted and summed. The gray-scale consistency constraint loss is used to suppress the maximum gray-scale error and high quantile error of the overall restored image.

[0017] The inference phase employs overlapping sliding window blocks, and the overlapping regions are fused using a weighted window function to reduce block boundary artifacts. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments or the prior art will be briefly described below. The present invention can be further understood in conjunction with the content of the accompanying drawings. The components shown in the drawings are not necessarily drawn to scale, but the focus is on illustrating the principles of the embodiments.

[0019] Figure 1 This is a schematic diagram illustrating the steps of a method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network, as provided by the present invention.

[0020] Figure 2 This is a schematic diagram of a multi-scale perceptual deep learning deblurring neural network structure provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of the basic encoding and decoding blocks in a multi-scale perceptual deep learning deblurring neural network provided in an embodiment of the present invention.

[0022] Figure 4 This is a comparison chart showing the restoration effect of a method for removing motion blur from grayscale images of wafer samples using a multi-scale sensing network, provided by one embodiment of the present invention, on a dataset created using data provided by the enterprise. Detailed Implementation

[0023] The following describes in detail an embodiment of the present invention, using a multi-scale sensing basic codec block to repair a wafer grayscale motion-blurred image. Examples of this embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals denote elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0024] Please see Figure 1 , Figure 2 and Figure 3 This invention provides a method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network, comprising the following steps:

[0025] S101. Obtain the grayscale image and perform preprocessing.

[0026] Specifically, the image to be processed is a single-channel grayscale image output by a wafer sample inspection device, with a grayscale depth of either 8 bits or 16 bits. First, the input image is normalized: the original grayscale values ​​are linearly mapped to... Alternatively, the image can be scaled to the network's desired dynamic range at a fixed ratio to avoid training instability caused by differences in exposure and gain between different batches. For images with dead pixels or fixed pattern noise (such as row and column stripes), lightweight preprocessing (such as dead pixel interpolation and stripe suppression) can be performed without changing the main structural information to reduce the risk that the restoration network will "amplify" noise as texture.

[0027] When the input image size is too large to be loaded into the video memory at once, the block inference parameters (block size, overlap width, fusion window function type) are pre-recorded so that overlapping sliding window inference and stitching fusion can be performed in S106.

[0028] S102. Construct a pairwise clear-fuzzy supervised dataset.

[0029] Specifically, this embodiment uses supervised learning to train the motion blur removal network. The dataset consists of paired samples of "motion-blurred image - sharp image". These paired data can be obtained in two ways and can be used in combination:

[0030] Paired acquisition: Under the same wafer sample and imaging configuration, images with varying degrees of motion blur are acquired by controlling stage scanning speed fluctuations, triggering timing jitter, or introducing controllable micro-vibrations; simultaneously, structurally clear images are acquired as labels under as consistent imaging conditions as possible. This method enables network learning to better reflect the degradation of production line distribution.

[0031] Pair composition: from clear grayscale images Departure, utilizing the motion trajectory core Simulated exposure integration process generates blur map The typical form is:

[0032]

[0033] in This represents the noise term. The synthesis method can cover multi-directional and multi-length trails, improving the network's generalization ability to different motion patterns. Training samples are usually obtained by randomly cropping patches (e.g., 256×256 or 512×512), and synchronous data augmentation (flipping, rotation, slight intensity perturbation) is performed on the input / label to ensure structural alignment and distribution diversity.

[0034] S103. Construct an end-to-end motion-de-fuzzy network.

[0035] Specifically, such as the example network framework Figure 2As shown, this embodiment adopts a U-shaped encoder-decoder structure, which includes: an input embedding layer, a multi-level encoder, a bottleneck layer (BottleNeck), a multi-level decoder, a skip connection, and an output layer.

[0036] Input embedding layer: using Convolution maps a single-channel grayscale input to a base number of channels. shallow features are obtained. .

[0037] Downsampling module: Performs scale changes before and after each level of encoding. To balance efficiency and information preservation, a combination of "convolution + PixelUnshuffle" is preferred: PixelUnshuffle rearranges spatial information to the channel dimension, achieving downsampling while reducing information loss (its essence is the rearrangement of tensor elements).

[0038] Encoder Block: Several "basic blocks" are stacked at each scale (see S104), the feature representation of the corresponding scale is extracted, and the features used for skip connections are output.

[0039] Bottleneck: Stacks basic blocks at the lowest resolution to aggregate motion blur cues for a larger receptive field.

[0040] Upsampling module (UpSample): The preferred approach is a combination of "convolution + PixelShuffle": PixelShuffle rearranges the channel information back into the spatial dimension, achieving efficient upsampling.

[0041] Decoder Block: After upsampling at each level, it is concatenated with the skip connection features of the corresponding coding scale in the channel dimension, and then... Convolution performs channel compression (reducing computation / memory usage), followed by refinement and restoration of features through several basic blocks.

[0042] Output layer: using The convolution output shows the restored result for one channel. Residual regression is commonly used in engineering: the network predicts the residuals. Final output This can improve training stability and suppress overall grayscale drift.

[0043] S104. Perform multi-scale feature extraction within the basic block.

[0044] Specifically, in this embodiment, both the encoding and decoding blocks are composed of several basic blocks connected in series. Each basic block employs a "two-stage residual sub-layer" structure: the first stage is a multi-scale spatial feature mixing sub-layer, and the second stage is a gated feedforward sub-layer. Intra-block normalization uses LayerNorm to improve training stability.

[0045] 1. Multi-scale spatial feature hybrid sublayer:

[0046] Given input features Execute in sequence:

[0047] LayerNorm: Performs layer normalization on the features to obtain... .

[0048] Conv: Performs channel linear mapping / recombination to obtain .

[0049] Depthwise Conv: Performs local spatial blending (depthwise separable convolution can perform spatial modeling with low computational cost).

[0050] Parallel dilated depthwise convolution with multiple branches: Features are fed into a multi-branch, depthwise separable convolution with a dilation rate set to... To create different receptive fields covering short and long trails; each branch is followed by PointwiseConv (pointwise convolution). Convolution (or dilated convolution) enables cross-channel combination, ultimately fusing the results of each branch. Multi-branch dilated convolution is a common and effective strategy in multi-scale feature extraction for obtaining receptive fields at different scales.

[0051] SimpleGate (multiplication gate): Divides the fused features into two channels and performs element-wise multiplication to achieve gate control.

[0052]

[0053] This gate method (SimpleGate) is a lightweight nonlinear alternative commonly used in image restoration networks.

[0054] Simplified channel attention: The gated features are subjected to global average pooling to obtain channel descriptors, which are then processed... Convolution (or equivalent linear layers) and Sigmoid are used to obtain channel weights, and the features are recalibrated to highlight effective channels related to motion blur and suppress artifact channels.

[0055] Conv: Number of channels to be recast to the target channel.

[0056] Residual connection: Connect the above output to the block input. Add them together to get the output of that sub-layer. Residual connections are used to stabilize training and preserve low-frequency grayscale information.

[0057] 2. Gated feedforward sublayer:

[0058] by For input, execute sequentially:

[0059] LayerNorm: Get .

[0060] Conv: Channel expansion (increasing expression capacity).

[0061] SimpleGate: Continue to use channel binary multiplication gating to form lightweight nonlinearity.

[0062] Conv: Channel resubmission.

[0063] Residual connection: with input Add to get block output .

[0064] By combining "parallel depth-separable convolution + gating + channel attention + dual residuals", basic blocks can cover multi-scale motion blur and suppress restoration artifacts with low computational cost, making them suitable for embedding repeated stacking of encoding / decoding scales.

[0065] S105. Train the network using a special supervised loss.

[0066] Specifically, to balance the clarity of the restored image with the stability of industrial grayscale, this embodiment uses a combined loss method of "basic reconstruction loss + structural detail constraint + multi-scale supervision + grayscale consistency constraint" for training, for example:

[0067]

[0068] Pixel-level reconstruction loss (L1 or Charbonnier) ensures overall restoration.

[0069] Multi-scale supervised loss. Intermediate results are output at multiple scales in the decoder. After upsampling to the original resolution, the reconstruction loss is calculated with the label and then weighted and summed so that each scale learns the "de-ghosting" objective, improving convergence and detail.

[0070] Gradient / edge consistency constraint (Sobel gradient L1) improves the sharpness of fine lines and edges, and reduces "blurring of boundaries".

[0071] Gray-scale consistency constraint. This constraint can limit the maximum gray-scale error / high quantile error within the entire structural area of ​​the repaired image, suppressing overall gray-scale drift and local overshoot, and reducing interference with threshold segmentation and measurement.

[0072] During training, the input / labels are randomly pruned and augmented, and iterative updates are performed using optimizers such as Adam / AdamW; learning rate decay strategies and mixed precision training can be used to improve efficiency and stability.

[0073] S106. Use the trained model to infer the output image.

[0074] During the online inference stage, the preprocessed blurred grayscale image is input into the network to obtain the restored output. If the input resolution is large, overlapping sliding window block inference is adopted: the entire image is divided into several blocks with overlapping regions for inference, and then a weighted window function is used to fuse the overlapping regions to reduce block boundary artifacts and output a clear image. This strategy can process ultra-large format wafer sample images under limited video memory conditions, meeting the needs of production line deployment.

[0075] Finally, experimental verification, as the final stage of the process of one embodiment of the present invention, verifies the feasibility and effectiveness of the method proposed in the present invention through actual experiments.

[0076] During the experimental verification phase, this invention selected other classic image deblurring methods for quantitative and qualitative experiments. In the quantitative experimental section, the comparison results of different classic methods such as Restormer and NAFNet on a dataset constructed using enterprise industrial production data are shown in the table below. The comparison metrics are: PSNR (Peak Signal-to-Noise Ratio), used to evaluate reconstruction quality, generally the higher the better; SSIM (Structure Similarity Index Measure), used to evaluate the structural similarity between the restored image and the target image, the higher the value, the more similar the structures; LPIPS (Learned Perceptual Image Patch Similarity), used by a deep learning model to measure the perceptual similarity between two images, the lower the value, the more similar the images are perceptually; MAD (Mean Absolute Differences), used to measure the average gray level difference between two images, the lower the value, the smaller the error; MaxGD (Max Gray Differences), used to evaluate the maximum gray level difference between two images, the lower the value, the smaller the error; and TIME (Time), which refers to the time required for the model to process a single image, the lower the value, the faster the processing speed. As shown in the table below, the network model of this invention has advantages in all metrics, achieving high-quality image inpainting while maintaining lightweight design; in qualitative experiments, such as Figure 4 As shown, the restoration results of the method of the present invention and other methods on a dataset constructed using enterprise industrial production data demonstrate that the method of the present invention can better repair motion-blurred images and obtain more and richer texture details, proving its advantages in the scenario of repairing grayscale motion-blurred images of industrial inspection wafer samples.

[0077] method PSNR SSIM LPIPS MAD MaxGD TIME Restormer 37.54 0.906 0.069 42 255 0.39s NAFNet 38.13 0.919 0.057 38 255 0.21s This method 39.57 0.932 0.051 22 53 0.073s

[0078] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network, characterized in that, Includes the following steps: Acquire a single-channel grayscale image of the wafer sample to be restored, captured by an industrial camera or inspection equipment, and perform normalization and size alignment preprocessing. Construct paired or grouped supervised datasets, wherein the supervised datasets contain at least motion-blurred grayscale images and their corresponding sharp grayscale label images; An end-to-end U-shaped encoder-decoder de-motion blurring network was constructed, which includes an input embedding layer, a multi-level encoder, a bottleneck layer, a multi-level decoder, cross-scale jump connections, and an output layer. Parallel dilatation-depth separable convolutional feature blending, multiplication gating, and simplified channel attention mechanisms are performed in the basic blocks of the encoder and decoder. The network was trained using multi-scale supervision and gray-scale consistency constraint loss to obtain a well-trained motion-blurred model. The grayscale image to be restored is input into the trained model for inference and outputs the restored image. It can also be selected to use overlapping sliding window block inference and weighted fusion under ultra-high resolution conditions to obtain the whole image restoration result.

2. The method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network as described in claim 1, characterized in that, The network consists of a multi-layer encoder / decoder, an input embedding layer, a bottleneck layer, an output layer, and cross-scale skip links. Downsampling is performed by "convolution + PixelUnshuffle", and upsampling is performed by "convolution + PixelShuffle". In the decoding stage, the corresponding encoded features are concatenated in the channel dimension and then compressed through a 1×1 convolution.

3. The method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network as described in claim 1, characterized in that, The basic block includes a spatial feature hybrid sublayer and a gated feedforward sublayer. The spatial feature hybrid sublayer includes LayerNorm, 1×1 convolution, 3×3 depthwise separable convolution, parallel dilated depthwise separable convolution branch, fusion, SimpleGate, simplified channel attention mechanism, 1×1 convolution back-roll and residual connection.

4. The basic blocks of the network in the method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network as described in claim 3, are characterized in that... The preferred values ​​for the void ratio of parallel void branches are 1, 3, 7, and 9.

5. The method for removing motion blur from grayscale images of wafer samples based on a multi-scale sensing network as described in claim 1, characterized in that, Multi-scale supervision generates intermediate restoration results at multiple scale outputs of the decoder. After upsampling to the original resolution, the reconstruction loss is calculated with the label and then weighted and summed. Gray-scale consistency constraint loss is used to suppress the maximum gray-scale error and high quantile error of the overall restored image.

6. The method as described in claim 1, characterized in that, During the inference phase, overlapping sliding windows are used for block segmentation, and overlapping regions are fused using weighted window functions to reduce block boundary artifacts.

Images