Method for removing out-of-focus blurring of grayscale image of wafer sample based on coding and decoding network

Abstract

The invention discloses a method for removing out-of-focus blur of a grayscale image of a wafer sample based on a coding and decoding network. The method is characterized by mainly comprising the following four steps of: firstly, obtaining, sorting and manufacturing a grey-scale map data set which meets a model input requirement by using shooting equipment such as an industrial camera and the like; adding a specific number of up-sampling modules, down-sampling modules, coding blocks and decoding blocks into each of four coding-decoding layers on a network of which the trunk structure is a U-shaped coding-decoding structure, wherein the four coding-decoding layers comprise a specific number of up-sampling modules, a specific number of down-sampling modules, a specific number of coding blocks and a specific number of decoding blocks; then, constructing a coding block and a decoding block by using depth separable convolution, a space-channel collaborative attention module and a feed-forward network; and finally, training the network model by using a task specialized training mode in combination with a task specialized loss function to obtain a trained model, and carrying out test verification. According to the method, the capability of repairing the out-of-focus blurred image of the wafer sample is improved, meanwhile, the relatively low parameter quantity and the relatively high reasoning speed are kept, the image information is repaired, and subsequent tasks can be conveniently carried out.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image restoration technology. It is a deblurring deep learning neural network that integrates directional spatial-channel attention and pixel rearrangement to restore blurred images into images with clear structures. It recovers as much image information as possible to facilitate subsequent tasks. It is suitable for defocusing and blurring restoration of wafer grayscale images, as well as restoration tasks of other industrial grayscale images. Background Technology

[0002] In wafer fabrication lines, issues such as workpiece warping, lens defocusing, exposure problems, and misalignment between the workpiece and the camera's focal plane often lead to varying degrees of blurring in grayscale images acquired during wafer fabrication, with out-of-focus blur being the most common. Out-of-focus blur weakens the microstructure and edges, resulting in information loss and affecting subsequent defect detection and measurement stability. While images acquired on new dual-pixel sensors can be effectively used for deblurring, practical limitations often result in only single-pixel images, i.e., traditional images. Therefore, deblurring methods for single-image images remain worthy of further research.

[0003] Existing deblurring methods can be broadly categorized into three types. The first is non-blind deconvolution (such as Wiener and RL methods), which requires a pre-defined or accurately estimated point spread function (PSF). However, in production line scenarios, blur is often spatially non-uniform and significantly affected by noise, making it difficult to match a fixed PSF. Furthermore, deconvolution easily amplifies noise and produces ringing artifacts. The second is traditional blind deblurring methods, which typically estimate the blur kernel before restoration. However, in wafer images with regular grids and periodic textures, gradient statistics are not sparse, kernel estimation is prone to bias, and restoration stability is insufficient. The third is deep learning end-to-end deblurring methods, which utilize deep learning networks to learn the mapping from blur to sharpness, enabling direct deblurring without prior knowledge.

[0004] Traditional methods, such as non-blind deconvolution (NDR) represented by the Richardson-Lucy (RL) algorithm and Wiener filtering, rely on a pre-given or accurately estimated point spread function (PSF). However, in industrial production line imaging, blurring often exhibits spatial inhomogeneity and is accompanied by significant noise. Fixed or parameterized PSF models struggle to accurately match the actual blur field, leading to the deconvolution process easily amplifying noise and generating ringing artifacts that were not originally present. On the other hand, traditional blind deblurring attempts to estimate both the blur kernel and the sharp image simultaneously, often utilizing the heavy-tailed gradient distribution of natural images as a prior (such as Total Variation). However, in wafer inspection scenarios, images contain a large number of regular grids and periodic textures, and their gradient statistics are not sparse. This causes kernel estimation methods relying on sparse priors to easily get trapped in local optima, resulting in insufficient stability of the restoration results.

[0005] In recent years, end-to-end deblurring methods based on convolutional neural networks (CNNs), such as DeepDeblur and SRN-Deblur, have made significant progress on general natural images by learning a nonlinear mapping from blurred to sharp. However, there are inherent limitations when applying them directly to wafer grayscale images: First, the local receptive field of ordinary two-dimensional convolutions is difficult to simultaneously cover the array-level long-range dependency and pixel-level minute defect structures in wafer images; Second, the frequent pooling or interpolation upsampling and downsampling operations in mainstream networks (such as multi-scale networks based on the U-Net architecture) can lead to irreversible loss of high-frequency detail information and may introduce interpolation artifacts; Third, such networks often use mean squared error (MSE) or L1 loss as optimization targets, tending to generate smooth results of "statistical average," and lack direct control over grayscale consistency indicators (such as maximum grayscale error, quantile grayscale error) that are of concern to industrial quality inspection, resulting in reduced contrast of the restored wafer pattern.

[0006] To address the problem of long-distance dependencies, self-attention mechanisms (such as ViT and Swin Transformer) have been introduced into the field of image restoration. However, the computational complexity of standard global self-attention increases quadratically with resolution, making it difficult to meet the real-time inference requirements of industrial-grade high-resolution grayscale images. While window-based attention reduces overhead, it weakens the ability to model globally across windows. Furthermore, existing channel attention mechanisms (such as SE-Block and Squeeze-and-Excitation) mainly focus on feature recalibration along the channel dimension, lacking explicit directional modeling for strongly anisotropic structures distributed along rows / columns in wafer images, and failing to fully utilize the geometric priors of wafer patterns. Industrial deployments also present engineering constraints: limited single-channel grayscale, limited memory budgets, and grayscale consistency requirements aligned with production line quality control thresholds (e.g., control of maximum grayscale error and high-quantile grayscale error).

[0007] In summary, existing technologies cannot simultaneously meet the comprehensive requirements of wafer production lines for single-channel grayscale data detail recovery, long-distance dependency modeling, stringent memory / computation budgets, and grayscale consistency constraints. Therefore, this invention proposes a deblurring method for wafer patterns, aiming to address the problems of artifacts, insufficient utilization of directional features, and weak grayscale deviation control in the restoration of periodic textures in the aforementioned existing technologies.

[0008] This invention discloses a lightweight end-to-end neural network for deblurring wafer grayscale images. The main network structure adopts a U-shaped encoder-decoder architecture, and directional spatial-channel collaborative attention modules are introduced at different scale layers in the encoding and decoding regions to simultaneously enhance spatial modeling and channel recalibration along rows / columns. The network achieves downsampling and upsampling through pixel unshuffle / pixel shuffle, and replaces traditional convolution stacking with depthwise separable convolution, pointwise convolution, and gated feedforward networks. This significantly compresses the number of parameters and improves inference speed while ensuring restoration quality, making it more suitable for industrial deployment. The network outputs a restored image that supports single-channel high-resolution input and has a good suppression effect on out-of-focus blur. Since it only needs to process the input image, this invention can be integrated into the front-end and back-end of a system as an independent restoration module, suitable for semiconductor production lines and other industrial grayscale inspection scenarios. Summary of the Invention

[0009] This invention aims to provide a lightweight end-to-end neural network for deblurring wafer grayscale images. While maintaining restoration quality, it reduces the number of model parameters and computational load, quickly repairs out-of-focus blur in images, enhances the texture structure recovery information in images, and reduces grayscale errors in the repaired images. It can be applied to the deployment of GPUs in industrial production lines and is not limited to data acquisition and shooting processes.

[0010] The solution proposed in this invention constructs a network using an encoder-decoder U-shaped multi-scale structure. The network structure includes:

[0011] The network consists of an input embedding layer, inter-layer upsampling and downsampling, a four-level coding layer, a four-level decoding layer, a cross-scale skip connection, and an output layer. Each coding layer contains a feature mixing module, a spatial-channel collaborative attention module, and a feedforward network (FFN) module. Scale transfer uses pixel rearrangement-based downsampling and upsampling to reduce interpolation artifacts and improve information fidelity.

[0012] The encoding layer of the U-shaped network consists of several encoding blocks, and the decoding layer consists of several decoding blocks. Skip connections are set between corresponding scales to fuse shallow and deep features. Spatial-channel collaborative attention is introduced into the basic blocks of different scale layers of the encoding and decoding layers to enhance directional spatial modeling and channel recalibration. To reduce the number of parameters and improve inference speed, the main body of the entire network uses depthwise separable convolution and pointwise convolution instead of standard convolution.

[0013] Inter-level upsampling and downsampling employ pixel rearrangement-based scale transfer to faithfully transfer information across scales. Downsampling uses a concatenation of PixelUnshuffle and convolution to map spatial displacement to the channel dimension to maintain energy consistency; upsampling uses a concatenation of PixelShuffle and convolution to achieve detail-friendly resolution restoration and reduce blurring and artifacts caused by interpolation, while progressively expanding the number of channels in the feature map to provide more information.

[0014] The encoding block in the encoding layer consists of two layers of feature mixing and one layer of spatial-channel collaborative attention, followed by a set of feedforward networks (FFNs), all of which form residual connections with the block input. Feature mixing utilizes cascaded depthwise separable convolutions and pointwise convolutions, combined with nonlinear activation and normalization, to achieve efficient fusion of local features. Spatial-channel collaborative attention employs collaborative computation of directional spatial modeling and channel recalibration.

[0015] The decoding blocks in the decoding layer include upsampling, splicing and fusion with features of the corresponding coding scale using skip connections, and a network block structure consistent with the coding blocks; upsampling uses PixelShuffle and convolution concatenation to restore spatial resolution and suppress ringing caused by deconvolution; after fusion, the reconstruction results are gradually refined through feature mixing, spatial-channel collaborative attention, and feedforward networks.

[0016] The spatial-channel collaborative attention module is used for collaborative computation of directional spatial modeling and channel recalibration. Firstly, the spatial attention unit applies one-dimensional depthwise separable convolutions in both the horizontal and vertical directions, with kernel sizes chosen from {3, 5, 7, 9} to cover different receptive fields, resulting in spatial weights along rows and columns. , Secondly, the channel attention unit generates query Q, key K, and value V by grouping windowed or downsampled features using 1×1 convolutions, calculates scaled dot product self-attention, and averages the channel weights in the spatial dimension. After being gated (sigmoid) and fused with spatial weights, the output is obtained. ,in Input features for the module, Indicates directional spatial fusion, Indicates channel gating.

[0017] Feedforward networks (FFNs) are used for nonlinear transformations and channel reconstruction. They include 1×1 convolutions to expand channels to λ times the original number of channels, 3×3 depthwise convolutions for local mixing, and multiplicative gating with channel splitting (e.g., ...). ), and 1×1 convolution is fed back to the original channel dimension; FFN and block input form residuals and are added to stabilize the training.

[0018] The output layer is used for fine-tuning and restoring the image. Preferably, several small depthwise separable convolutional blocks and spatial-channel collaborative attention modules are used to suppress color level / grayscale drift; and residual connections are used to maintain feature uniformity.

[0019] To accommodate different hardware conditions and resolution requirements, the number of network scales, basic channels, and blocks can be flexibly configured according to the deployment scenario: the number of basic channels is preferably 32 to 96, the number of coding or decoding blocks per scale is preferably 2 to 10, the number of attention heads for spatial-channel collaborative attention is preferably 4 to 8, the window size is preferably 6 to 10 or replaced by global pooling; normalization can be achieved using LayerNorm, and the activation function is preferably GELU.

[0020] The training data was created by capturing grayscale BMP format data using an industrial camera. Images with varying degrees of focus and sharpness within the same scene were acquired by moving the camera's focal plane. Scripts were then used to create pairs of supervised images to serve as training inputs and labels.

[0021] During training, the training data undergoes preprocessing such as flipping and random pruning before being fed into the network. This data is then fed in using a certain proportion of mixed block inputs, group consistency inputs, and paired inputs, along with standard paired inputs. Monitoring losses, mixed blocks Maximum grayscale error loss 95th percentile error loss Identity loss, grouped within groups Iterative training using group consistency loss is employed, specifically by employing a three-branch mixed training method and a combined loss function to reduce the grayscale error of the image. First, a mixed block input and a grouped input are constructed. The mixed block input refers to two pairs of samples cropped from the same image. Sub-blocks are combined in a four-quadrant / four-row / four-column layout. The canvas is used to randomly permutate and record two pairs of clear / blurred positions. These pairs are used to calculate the maximum grayscale error (MGD) and p95 soft thresholding constraint for paired regions within the image. An identity preservation term is applied to the "clear → clear" path to suppress unnecessary changes. Group consistency input refers to labeling samples from the same background but with different blur levels with group IDs. This is used for group consistency branch calculation, i.e., sampling within the same group. For samples with different blur levels, compare the pixel-by-pixel range of the network output at the same pixel position. Calculate softening Consistency loss; additionally, a standard "fuzzy-clear" pairwise supervision is added for overall approximation and structure preservation. Initially, standard pairwise supervision is used to improve the ability to maintain the overall structure, and later, mixed block inputs and group consistency inputs are used for a small amount of follow-up training to improve the ability to reduce grayscale errors. The total loss can be summarized as follows: Finally, the trained model is used for testing and validation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The present invention can be understood in conjunction with the accompanying drawings. The accompanying drawings described below are only some embodiments of the present invention. The modules shown in the drawings are not necessarily drawn to scale. The focus should be on the principle of the embodiments.

[0023] Figure 1 This is a schematic diagram of the overall structure and steps of a method for repairing out-of-focus and blurred images of wafers provided by the present invention.

[0024] Figure 2 This is a network structure diagram of the present invention for out-of-focus and blurred image restoration.

[0025] Figure 3 This is a schematic diagram of the spatial-channel collaborative attention module in the network structure of this invention.

[0026] Figure 4 , Figure 5 This is an image showing the effect of the present invention in repairing out-of-focus and blurred images of wafers.

[0027] Figure 6 These are comparison images showing the effects of deblurring and repairing out-of-focus and blurred images according to the present invention. Detailed Implementation

[0028] The following describes in detail an embodiment of the present invention, taking the collection of grayscale images in a wafer production line for defocusing and blur repair as an example. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar components or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0029] Please see Figure 1 This invention provides an end-to-end deep learning network for defocusing and blur restoration based on wafer grayscale images, comprising the following steps:

[0030] S001 uses an industrial camera to acquire BMP format image data.

[0031] Against the same background (wafer sample), when the wafer sample is located on the camera's focal plane, a properly exposed image with clear texture and good visual effect is captured as a label image. Then, the camera lens focal length is continuously adjusted to separate the wafer sample from the camera's focal plane. During the separation process, the camera continuously captures groups of images with different degrees of blur against the same background, forming a dataset. This method allows for the capture of several sets of data under various backgrounds. Subsequent dataset creation scripts randomly crop the original clear image and the original blurred image at the same location, obtaining several pairs of sub-images. A small portion of these sub-images is selected for testing and validation, while the remainder is used for training.

[0032] S002, build an end-to-end deep learning deblurring network.

[0033] The entire network structure is as follows Figure 2As shown, data loading, network encapsulation, loss and log evaluation, and network structure construction are implemented within a general image restoration framework. The entire network structure adopts a U-shaped encoder-decoder structure and outputs the restored value in the form of residual regression. The main body of the network consists of four encoder-decoder layers, each containing several encoder or decoder blocks. Layers are cascaded using downsampling and upsampling blocks composed of PixelUnshuffle and PixelShuffle along with convolutions. Skip connections are used between the encoder and decoder regions of each layer to keep image features flowing between different levels, enabling the network to obtain rich multi-scale information during the image restoration process.

[0034] Specifically, the network initially inputs a single-channel grayscale image. (Batch size denoted as B) is first mapped to the baseline number of channels C (e.g., C=48) via a 3×3 convolution to obtain the feature tensor. The image then enters a four-level encoder, a bottleneck, and a four-level decoder. Information of the same scale is fused across layers through skip connections. At the end, several network blocks and 3×3 convolutions are used for reshaping to obtain the residual Δ, which is then added to the input to obtain the restored image. To reduce interpolation artifacts and improve the restoration of subpixel structures, downsampling and upsampling employ pixel rearrangement family operators (PixelUnshuffle / PixelShuffle), respectively. After skip connections, channels are compressed back to the target dimension using 1×1 convolutions to ensure consistency with the decoder's desired number of channels. Each scale consists of stacked basic network blocks. Each basic block contains two key operator paths—a spatial-channel collaborative attention block and a feature mixing block (depth convolution + pointwise convolution)—that are cascaded, forming short residuals between blocks. The outermost layer of the network also has a global residual from input to output, jointly constraining grayscale consistency and detail restoration in a steady state.

[0035] Specifically, in one embodiment, the network adopts a four-layer scale: coding region layers 1 to 3: channel processes are C, 2C, and 4C; each layer contains Each decoding block has an inter-layer undersampling operator connection; the feature renormalization layer has 8 channels and contains... The decoding area consists of one decoding block; layers 3 to 1 of the decoding region: the channels from bottom to top are 4C, 2C, and C; each layer is first upsampled, then concatenated with the output of the same-scale encoded block, connected to a 1×1 convolutional compression channel, and then connected to several decoding blocks; output reshaping: a small number of decoding blocks are mapped to a single channel using 3×3 convolution and added to the input to obtain the output. The number of layers, channels, and blocks can be adjusted appropriately according to the device's computing power and target accuracy.

[0036] Downsampling (PixelUnshuffle): The number of channels is transformed layer by layer using 3×3 convolutions. (Corresponding to three layers), then the space size is adjusted using PixelUnshuffle with a step size of 2. Become Multiplying the number of channels by 4 achieves a spatial rearrangement equivalent to stride=2 while preserving local arrangement information as much as possible.

[0037] Upsampling (PixelShuffle): The number of channels is doubled using a 3×3 convolution, followed by a PixelShuffle with a stride of 2 to reduce the spatial size. Become .

[0038] Skip connections and channel compression: The features of each layer in the decoding area are concatenated with the corresponding layer in the encoding area in the channel dimension, and then compressed back to the predetermined channel by 1×1 convolution (e.g., 4C + 4C compressed back to 4C), which preserves the shallow texture while controlling the number of parameters and memory.

[0039] Specifically, such as Figure 1 Each basic coded or decoded block contains two segments: a normalization-sublayer-residual structure. The typical calculation process is as follows:

[0040] I. Normalization: Biased layer normalization or batch normalization (implementation not limited to these) is used to stabilize the amplitude of the input features. II. Feature Mixing Layer: This layer includes large-kernel depthwise convolution, pointwise convolution, and short residuals. Depthwise convolution achieves a large receptive field with minimal parameters to extract features from local structures. Pointwise convolution enables linear combination across channels and, in conjunction with activation functions, nonlinearly enhances inter-channel representation. Short residuals combine the mixing results with the block input to improve gradient flow stability. III. Spatial-Channel Cooperative Attention: This is divided into fusion of directional attention and channel attention. Directional attention first averages and aggregates features along the horizontal or vertical directions to obtain "line features" in two directions. Then, multi-scale depthwise convolutional kernels (e.g., 3 / 5 / 7 / 9) are applied in parallel in each direction, and after normalization and gating (Sigmoid), two spatial weight maps are obtained. The weights of the two directions are fused to obtain the final spatial attention, which is then weighted onto the input features. Channel attention fusion performs windowed downsampling on the weighted features from the previous step, generates lightweight Q / K / V and scales the dot product to obtain self-attention activation, and then aggregates and gates them in the channel dimension to form channel attention. This is then multiplied point-by-point with the spatial attention output to obtain the output. IV. Normalization and Feedforward Network: A gated feedforward network of "point convolution—depthwise separable convolution—channel binary gating (one branch is nonlinearly processed by GELU, the other branch is not processed, and the two branches are multiplied)—point convolution" is used to further process the local context. 5. Block-level residual: The input before entering the block is added to the output of the feedforward network to obtain the output of this block.

[0041] Specifically, the structure of spatial-channel collaborative attention in the encoding and decoding blocks is as follows: Figure 3 As shown.

[0042] like Figure 3 The operation of the attention module shown in the encoding and decoding blocks can be described in two dimensions: directional spatial attention and lightweight channel attention.

[0043] Directional spatial attention: Mean aggregation of features along the width dimension yields... Divided into four groups according to the channel Four groups were subjected to one-dimensional depthwise convolutions with different kernel sizes to obtain multi-scale lateral responses; similarly, aggregation along the height dimension yielded... After grouping, the data undergoes a one-dimensional depthwise convolution with the same kernel size as the width dimension to obtain multi-scale longitudinal responses. The four convolutional outputs from each of the horizontal and vertical paths are concatenated along the channel dimension, followed by grouping normalization and gating, and then reshaped back to their spatial weight shapes, forming... and Then, spatial weighting is performed by multiplying the input features by the two weights using pointwise multiplication, resulting in the spatial attention weighted result. This step suppresses the diffusion of non-uniform blur in the horizontal / vertical main directions by using a directional and multi-scale fusion approach, especially for the characteristics of wafer regular linear arrays and grid boundaries.

[0044] Lightweight channel attention: The input is downsampled, including but not limited to adaptive pooling, average pooling, and max pooling, and then downsampled back to channel C using a 1×1 convolution. The output tensor is denoted as... .right After normalization, pointwise convolution is used to generate... By rearranging the channel dimensions and mapping them onto the attention head, a process is formed. ; Calculate the scaled dot product attention matrix Optional attention dropout can be applied; The attention output is obtained by left multiplication and then rearranged back to the channel dimension. The mean of the attention output from the previous step is taken along the spatial dimension to obtain the channel description, and then channel gating is applied to obtain the channel weights. Finally, point-by-point multiplication. The channel self-attention results are fed back to the original resolution features.

[0045] S003, use the prepared dataset to train the constructed network.

[0046] Specifically, supervised training is carried out based on the network structure described in step S002 and the dataset obtained in step S001. The training process includes data loading and preprocessing, mixed training, loss function calculation and backpropagation, progressive scheduling of training patch size and batch size, periodic verification, and weight saving.

[0047] Specifically, grayscale errors in images are reduced by using three-branch mixed training and a combined loss function. First, a mixed block input and a grouped input are constructed. The mixed block input refers to two pairs of samples cropped from the same image. Sub-blocks are combined in a four-quadrant / four-row / four-column layout. The canvas is used to randomly permutate and record two pairs of clear / blurred positions. These pairs are used to calculate the maximum grayscale error (MGD) and p95 soft thresholding constraint for paired regions within the image. An identity preservation term is applied to the "clear → clear" path to suppress unnecessary changes. Group consistency input refers to labeling samples from the same background but with different blur levels with group IDs. This is used for group consistency branch calculation, i.e., sampling within the same group. For samples with different blur levels, compare the pixel-by-pixel range of the network output at the same pixel position. Calculate softening Consistency loss; additionally, a standard "fuzzy-clear" pairwise supervision is added for overall approximation and structure preservation. Initially, standard pairwise supervision is used to improve the ability to maintain the overall structure, and later, mixed block inputs and group consistency inputs are used for a small amount of follow-up training to improve the ability to reduce grayscale errors. The total loss can be summarized as: .

[0048] and Paired blocks within the mixed block diagram After taking the maximum and 95th percentile respectively, softplus / hinge soft penalties are applied to directly target the MGD and p95 indicators for engineering objectives;

[0049] In the same group Zhang output is calculated by pixel. Only for Inconsistent pixel weighted averaging focuses on the worst-case region, reducing the dispersion of different degradation levels within the same background;

[0050] The output of the "clear → clear" path is constrained to be approximately identical, suppressing intensity drift. Each weight is set according to the specific actual objective.

[0051] S004, using the network model trained in S003 to test the test image to obtain... Figure 4 , Figure 5 The restored image of the out-of-focus and blurred wafer sample shown is as follows. Figure 6The image shows a comparison of the effects of similar network models for deblurring out-of-focus images of wafer samples. This invention uses Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Max Gray Difference (MGD) to measure the image deblurring performance. The table below compares the PSNR and SSIM of this invention with other methods on a test set of out-of-focus images of wafers.

[0052] Defocusing and blur removal methods PSNR SSIM MGD (255) Parameters (M) Restormer 39.75 0.949 46 26.2 DPDNet 37.58 0.912 51 31.6 Ours 40.98 0.963 22 12.4

[0053] As can be seen from the table above, the network proposed in this invention outperforms Restormer and DPDNet in PSNR and SSIM, respectively, by 1.23dB and 3.4dB, respectively, and by 0.014 and 0.051 in SSIM, respectively. The maximum grayscale error of the restored image is reduced by 52.1% and 56.8%, respectively. At the same time, the number of network parameters is 1.1 times less than Restormer and 1.54 times less than DPDNet. Therefore, this invention maintains a higher quality restoration effect while being lightweight.

[0054] The above-disclosed embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of the invention. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments, and equivalent variations made in accordance with the claims of the present invention, still fall within the scope of the invention.

Claims

1. A method for removing out-of-focus blur from grayscale images of wafer samples based on an encoding / decoding network, characterized in that, Includes the following steps: (1) Obtain a single-channel grayscale image of the wafer to be restored from an industrial camera. (2) Construct an end-to-end network with a U-shaped encoding-decoding structure, and add upsampling and downsampling modules, encoding blocks and decoding blocks at different scales of each encoding layer and decoding layer. (3) Use depthwise separable convolution, spatial-channel collaborative attention, and gated feedforward network to build basic network blocks, namely encoding blocks and decoding blocks. (4) The captured data is made into a dataset and trained using a three-branch training method and a task-specific loss function to obtain a trained model and test and verify it.

2. As described in claim 1, an end-to-end network with a U-shaped encoder-decoder structure is constructed, and upsampling and downsampling modules, as well as encoder blocks and decoder blocks, are added at different scales in each encoder layer and decoder layer, characterized in that: The network consists of an input embedding layer, inter-layer upsampling and downsampling, a four-level coding layer, a four-level decoding layer, cross-scale skip connections, and an output layer; the coding and decoding blocks within each coding layer include a feature mixing module, a spatial-channel collaborative attention module, and a feedforward network (FFN) module, respectively. Scale transfer employs pixel rearrangement-based downsampling and upsampling to reduce interpolation artifacts and improve information fidelity; the coding layer of the U-shaped network consists of several coding blocks, the decoding layer consists of several decoding blocks, and skip connections are set between corresponding scales to fuse shallow and deep features; Spatial-channel collaborative attention is introduced into the basic blocks of different scale layers in the encoding and decoding layers to enhance directional spatial modeling and channel recalibration. To reduce the number of parameters and improve inference speed, depthwise separable convolutions and pointwise convolutions are used instead of standard convolutions for the main body of the entire network. Pixel rearrangement scale transfer is used for upsampling and downsampling between layers to transfer information with high fidelity across scales. Downsampling uses a concatenation of PixelUnshuffle and convolutions to map spatial displacements to the channel dimension to maintain energy consistency. Upsampling uses a concatenation of PixelShuffle and convolutions to achieve detail-friendly resolution restoration and reduce blurring and artifacts caused by interpolation, while progressively expanding the number of channels in the feature map to provide more information.

3. As described in claim 1, using depthwise separable convolution, spatial-channel cooperative attention, and gated feedforward networks to construct basic network blocks, namely encoding blocks and decoding blocks, characterized in that: The coding block in the coding layer consists of two layers of feature mixing and one layer of spatial-channel collaborative attention, followed by a set of feedforward networks (FFN), all of which form residual connections with the block input; Feature fusion is achieved by concatenating depthwise separable convolutions and pointwise convolutions, combined with nonlinear activation and normalization, to achieve efficient fusion of local features; spatial-channel collaborative attention uses directional spatial modeling and channel recalibration for collaborative computation; the decoding blocks in the decoding layer include upsampling, splicing and fusion with features of the corresponding coding scale using skip connections, and a network block structure consistent with the coding blocks; Upsampling employs a concatenation of PixelShuffle and convolution to restore spatial resolution and suppress ringing caused by deconvolution. After fusion, the reconstruction results are progressively refined through feature mixing, spatial-channel collaborative attention, and a feedforward network. The spatial-channel collaborative attention module is used for the collaborative computation of directional spatial modeling and channel recalibration. Firstly, the spatial attention unit applies one-dimensional depthwise separable convolutions in the horizontal and vertical directions, with kernel sizes taken from {3, 5, 7, 9} to cover different receptive fields, obtaining spatial weights along rows and columns. , ; Secondly, the channel attention unit generates query Q, key K, and value V by grouping windowed or downsampled features using 1×1 convolutions, calculates scaled dot product self-attention, and averages the channel weights in the spatial dimension. After being gated (sigmoid) and fused with spatial weights, the output is obtained. ,in Input features for the module, Indicates directional spatial fusion, Channel gating is indicated; feedforward networks (FFNs) are used for nonlinear transformation and channel reconfiguration. It includes 1×1 convolution to expand the channels to λ times the original number of channels, 3×3 depthwise convolution for local mixing, and multiplicative gating with channel splitting (e.g. The FFN is used to form residuals with the block input and stabilize the training; the output layer is used to fine-tune the restored image; preferably, several small depthwise separable convolutional blocks and spatial-channel co-attention modules are used to suppress color level / grayscale drift; and residual connections are used to maintain feature uniformity.

4. As described in claim 1, the captured data is compiled into a dataset and trained using a three-branch training method combined with a task-specific loss function to obtain a trained model, which is then tested and verified. The model is characterized by: The training data was created using grayscale BMP format images captured by an industrial camera. Images with varying degrees of out-of-focus and clear images of the same scene were acquired by moving the camera's focal plane. Paired supervised images were then created using a script to serve as training input and labels. During training, the training data underwent preprocessing such as flipping and random cropping before being fed into the network. The network used a combination of mixed block inputs, group consistency inputs, and paired inputs in a specific ratio, along with standard paired inputs. Monitoring losses, mixed blocks Maximum grayscale error loss 95th percentile error loss Identity loss, grouped within groups The group consistency loss is used for iterative training, specifically by using three-branch mixed training and a combined loss function to reduce the grayscale error of the image. First, a jumbled block input and a grouped input are constructed. The jumbled block input refers to two pairs of samples cropped from the same image. Sub-blocks are combined in a four-quadrant / four-row / four-column layout. The canvas is used to randomly permutate and record two pairs of clear / blurred positions. These pairs are used to calculate the maximum grayscale error (MGD) and p95 soft thresholding constraint for paired regions within the image. An identity preservation term is applied to the "clear → clear" path to suppress unnecessary changes. Group consistency input refers to labeling samples from the same background but with different blur levels with group IDs. This is used for group consistency branch calculation, i.e., sampling within the same group. For samples with different blur levels, compare the pixel-by-pixel range of the network output at the same pixel position. Calculate softening Consistency loss; additionally, a standard "fuzzy-clear" pairwise supervision is added for overall approximation and structure preservation. Initially, standard pairwise supervision is used to improve the ability to maintain the overall structure, and later, mixed block inputs and group-consistent inputs are used for a small amount of follow-up training to improve the ability to reduce grayscale errors; the total loss can be summarized as follows: Finally, the trained model is used for testing and validation.

Images