Sem image super-resolution reconstruction method, computer program product and computer device

Abstract

The application provides a SEM image super-resolution reconstruction method, a computer program product and a computer device. The method comprises the following steps: obtaining a SEM original image, determining the noise intensity level and the local edge gradient of the SEM original image; determining the filtering parameters of joint filtering according to the noise intensity level and the local edge gradient, and performing joint filtering on the SEM original image according to the filtering parameters; performing super-resolution reconstruction processing on the image filtered by joint filtering by using a preset super-resolution network; performing quality evaluation on the reconstructed image, and updating the filtering parameters of joint filtering according to the result of the quality evaluation; repeatedly performing the above process until the result of the quality evaluation indicates that the image quality meets the preset standard. The advantage of the application is that the filtering parameters can be adaptively determined according to the noise intensity level and the local edge gradient of the image, and the filtering parameters are updated in the reverse direction through the quality evaluation of the super-resolution reconstruction result to perform closed-loop iterative optimization, thereby improving the quality reliability of the reconstructed image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically provides a SEM image super-resolution reconstruction method, computer program product, and computer equipment. Background Technology

[0002] SEM (Scanning Electron Microscopy) images have wide applications in materials science, semiconductor manufacturing, biomedicine, and other fields, and their imaging quality directly affects the accuracy of subsequent analysis results. However, due to the physical constraints of hardware equipment, the imaging resolution of SEM images often falls short of the growing demand for observing fine structures.

[0003] In existing technologies, filtering algorithms are typically used to preprocess SEM images to suppress noise, followed by interpolation or super-resolution algorithms to improve image resolution. However, once the filtering parameters are set, they remain unchanged during processing, making it impossible to adaptively adjust them based on noise intensity and edge feature differences in different regions of the image. Furthermore, the lack of quality feedback and parameter optimization mechanisms based on the processing results makes it difficult to guarantee the quality of the final reconstructed image. Summary of the Invention

[0004] One object of the present invention is to overcome at least one technical defect in the prior art and provide a SEM image super-resolution reconstruction method, computer program product and computer equipment.

[0005] A further objective of this invention is to enable the filtering parameters to be adaptively determined based on the noise intensity level and local edge gradient of the image, and to update the filtering parameters in reverse through the quality evaluation of the super-resolution reconstruction results for closed-loop iterative optimization, thereby improving the quality reliability of the reconstructed image.

[0006] Another further objective of this invention is to fully leverage the complementary advantages of guided filtering and bilateral filtering in joint filtering to provide high-quality input images for super-resolution reconstruction processing.

[0007] In particular, according to a first aspect of the present invention, the present invention provides a method for super-resolution reconstruction of SEM images, comprising: Obtain the original SEM image and determine the noise intensity level and local edge gradient of the original SEM image; The filtering parameters for joint filtering are determined based on the noise intensity level and the local edge gradient, and the original SEM image is jointly filtered according to the filtering parameters. The image after joint filtering is reconstructed using a preset super-resolution network; The image after super-resolution reconstruction is subjected to quality assessment, and the filtering parameters of the joint filter are updated based on the results of the quality assessment. The process of joint filtering, super-resolution reconstruction, and quality assessment is repeated until the result of the quality assessment indicates that the image quality meets the preset image quality standard.

[0008] Optionally, the joint filtering includes guided filtering and bilateral filtering, and the step of determining the filtering parameters of the joint filtering based on the noise intensity level and the local edge gradient includes: The size of the filter window for the guided filter is determined based on the noise intensity level, and the higher the noise intensity level, the larger the filter window size. The gray-scale standard deviation of the bilateral filter is determined based on the local edge gradient, and the larger the local edge gradient, the smaller the gray-scale standard deviation.

[0009] Optionally, the step of performing a quality assessment on the super-resolution reconstructed image and updating the filtering parameters of the joint filter based on the quality assessment result includes: Calculate the edge gradient difference between the super-resolution reconstructed image and the reference high-resolution image; When the edge gradient difference is greater than the preset edge gradient difference threshold, the grayscale standard deviation of the bilateral filter is reduced. Calculate the global grayscale variance of the image after the super-resolution reconstruction process; When the global grayscale variance is greater than the preset grayscale variance threshold, the size of the filter window of the guided filter is increased.

[0010] Optionally, the preset image quality standard is: The edge gradient difference is less than or equal to the preset edge gradient difference threshold, and the global grayscale variance is less than or equal to the preset grayscale variance threshold.

[0011] Optionally, the preset super-resolution network is constructed based on the standard ESPCN network and includes a feature extraction part and an upsampling part; The feature extraction part includes a multi-scale feature extraction module, which is used to extract multiple scale features of the jointly filtered image and fuse the multiple scale features through a 1×1 convolution kernel to output fused features. The upsampling portion is used to upsample the fused features to generate a super-resolution reconstructed image.

[0012] Optionally, a residual connection is provided between the feature extraction part and the upsampling part, and the residual connection is used to superimpose the output features of the multi-scale feature extraction module onto the input features of the upsampling part; The feature extraction part uses the LeakyReLU activation function for non-linear activation; The upsampling portion includes subpixel convolutional layers, and the number of convolutional kernels in the subpixel convolutional layers is greater than the number of convolutional kernels in the standard ESPCN network.

[0013] Optionally, the preset super-resolution network is trained using a composite loss function, which includes a pixel-level loss term and an edge-weighted loss term. The edge-weighted loss term assigns a higher weight to the error in the image edge region than to the error in the non-edge region.

[0014] Optionally, the step of determining the noise intensity level of the original SEM image includes: The original SEM image is subjected to grayscale normalization processing, and the global grayscale variance of the original SEM image is calculated. Based on the magnitude of the global grayscale variance of the original SEM image, determine the noise intensity level of the original SEM image; and / or The steps for determining the local edge gradient of the original SEM image include: Edge detection is performed on the original SEM image; Based on the edge detection results, the local edge gradient of the original SEM image is determined.

[0015] According to a second aspect of the present invention, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the SEM image super-resolution reconstruction method described in any of the preceding claims.

[0016] According to a third aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program in accordance with the steps of the SEM image super-resolution reconstruction method described in any of the preceding claims.

[0017] The SEM image super-resolution reconstruction method of this invention adaptively determines the filtering parameters based on the noise intensity level and local edge gradient of the original SEM image. This ensures that the intensity of the joint filtering matches the noise level and edge features of different regions of the image, avoiding insufficient denoising or edge blurring caused by uniformly processing a single fixed parameter across the entire image. Furthermore, since the image quality assessment result after super-resolution reconstruction is used to update the filtering parameters, and the iterative process of joint filtering, super-resolution reconstruction, and quality assessment is repeated until the preset image quality standard is met, a quality feedback-driven closed-loop optimization mechanism is formed. This allows the filtering parameters to be continuously adjusted and gradually approach the optimal configuration during the iteration process. Therefore, this invention overcomes the technical shortcomings of traditional methods where the filtering parameters are fixed and lack a feedback optimization mechanism, improving the quality reliability of SEM image super-resolution reconstruction.

[0018] Furthermore, the SEM image super-resolution reconstruction method of the present invention includes guided filtering and bilateral filtering in its joint filtering. Since the size of the guided filtering window is adaptively determined based on the noise intensity level, and the higher the noise intensity level, the larger the window size, the more noise can be suppressed in high-noise regions with a larger filtering window, while in low-noise regions, a smaller filtering window can be used to avoid excessive smoothing of details. Simultaneously, since the gray-scale standard deviation of the bilateral filtering is adaptively determined based on the local edge gradient, and the larger the local edge gradient, the smaller the standard deviation, the clear edges can be maintained with a smaller gray-scale standard deviation in strong edge regions, while residual noise can be effectively removed with a larger gray-scale standard deviation in flat regions. Thus, guided filtering and bilateral filtering complement each other through adaptive adjustment of their respective parameters, effectively suppressing mixed noise in SEM images while preserving microscopic edge details to the maximum extent, providing a high-quality input image for subsequent super-resolution reconstruction processing.

[0019] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of a SEM image super-resolution reconstruction method according to an embodiment of the present invention; Figure 2 This is a flowchart of a filter parameter update method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a pre-defined super-resolution network according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 6 This is a schematic block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0021] Reference will now be made in detail to embodiments of the invention, one or more of which are illustrated in the accompanying drawings. The various embodiments provided are intended to explain the invention and not to limit it. In fact, various modifications and variations to the invention will be apparent to those skilled in the art without departing from the scope or spirit of the invention. For example, a feature illustrated or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, the invention is intended to cover such modifications and variations within the scope of the appended claims and their equivalents.

[0022] In the description of this embodiment, it should be understood that the term "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. When a feature "includes or contains" one or more of the features it covers, unless otherwise specifically described, this indicates that other features are not excluded and may be further included.

[0023] In the description of this embodiment, the terms "one embodiment," "some embodiments," "some examples," "one example," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0024] Figure 1 This is a flowchart of a SEM image super-resolution reconstruction method according to an embodiment of the present invention, as follows: Figure 1 As shown, the SEM image super-resolution reconstruction method may include the following steps S101 to S105.

[0025] Step S101: Acquire the original SEM image and determine its noise intensity level and local edge gradient. The original SEM image refers to a low-resolution grayscale image directly acquired and output by a scanning electron microscope (SEM) without any processing. The noise intensity level characterizes the degree of noise contamination in the original SEM image. In this embodiment, the noise intensity level can include high noise level, medium noise level, and low noise level. A higher noise intensity level indicates more severe noise contamination of the image. The local edge gradient characterizes the edge strength of each local region in the original SEM image. A larger local edge gradient indicates a higher probability that the region belongs to the image edge region, i.e., it contains richer microstructural details; conversely, a smaller local edge gradient indicates a higher probability that the region belongs to a flat region of the image, i.e., it contains less detailed information.

[0026] Step S102: Determine the filtering parameters for joint filtering based on the noise intensity level and local edge gradient, and perform joint filtering on the original SEM image according to the filtering parameters. In this embodiment, joint filtering refers to combining two or more filtering algorithms with different filtering characteristics. Through complementary advantages, it effectively suppresses image noise while preserving edge details to the maximum extent, thereby providing a high-quality input image for subsequent super-resolution reconstruction processing. The filtering parameters are the key parameters that need to be set for each filtering algorithm in joint filtering; their values ​​directly affect the filtering strength of the joint filtering and the degree of edge detail preservation.

[0027] Step S103: Perform super-resolution reconstruction processing on the jointly filtered image using a preset super-resolution network. Super-resolution reconstruction processing refers to the process of inputting the jointly filtered low-resolution image into a preset super-resolution network and generating a high-resolution output image through the network model.

[0028] Step S104 involves performing a quality assessment on the super-resolution reconstructed image and updating the filtering parameters of the joint filter based on the assessment results. Quality assessment refers to evaluating the quality of the image obtained after super-resolution reconstruction to quantitatively determine the quality level of the reconstructed image.

[0029] Step S105 involves repeating the joint filtering, super-resolution reconstruction, and quality assessment processes until the quality assessment result indicates that the image quality meets the preset image quality standard. The preset image quality standard refers to the pre-defined quantitative conditions used to determine whether the image quality meets the acceptable requirements.

[0030] The SEM image super-resolution reconstruction method in this embodiment adaptively determines the filtering parameters based on the noise intensity level and local edge gradient of the original SEM image. This allows the intensity of the joint filtering to match the noise level and edge features of different regions of the image, avoiding the insufficient denoising or edge blurring caused by uniform processing of a single fixed parameter across the entire image. Furthermore, since the image quality assessment results after super-resolution reconstruction are used to update the filtering parameters in reverse, and the iterative process of joint filtering, super-resolution reconstruction, and quality assessment is repeated until the preset image quality standard is met, a quality feedback-driven closed-loop optimization mechanism is formed. This allows the filtering parameters to be continuously adjusted and gradually approach the optimal configuration during the iteration process.

[0031] Therefore, this invention overcomes the technical defects of traditional methods where the filtering parameters are fixed and there is a lack of feedback optimization mechanism, thereby improving the quality and reliability of SEM image super-resolution reconstruction.

[0032] In an optional embodiment, determining the noise intensity level of the original SEM image in step S101 may include: performing grayscale normalization processing on the original SEM image, calculating the global grayscale variance of the original SEM image, and determining the noise intensity level of the original SEM image based on the magnitude of the global grayscale variance.

[0033] Specifically, grayscale normalization refers to mapping the pixel grayscale values ​​of the original SEM image to a predetermined numerical range (e.g., the [0, 1] interval) to eliminate the inconsistency in the dimensions of grayscale values ​​between different images due to differences in imaging conditions, so that subsequent global grayscale variance calculations are comparable.

[0034] The global grayscale variance is calculated as follows: The grayscale values ​​of all pixels in the original SEM image are statistically analyzed, and the average of the squares of the differences between each value and the image's grayscale mean is calculated. A larger global grayscale variance indicates a greater deviation of the grayscale values ​​of each pixel from the mean, meaning a more dispersed grayscale distribution and higher corresponding noise intensity. Conversely, a smaller global grayscale variance indicates a more concentrated grayscale distribution and lower corresponding noise intensity.

[0035] In this embodiment, noise intensity levels can be divided into multiple levels based on the magnitude of the global grayscale variance. For example, a global grayscale variance greater than a first predetermined threshold is determined to be a high noise level; a global grayscale variance less than or equal to the first predetermined threshold but greater than a second predetermined threshold is determined to be a medium noise level; and a global grayscale variance less than or equal to the second predetermined threshold is determined to be a low noise level. The first and second predetermined thresholds can be preset according to the actual application scenario and image quality requirements. For example, a global grayscale variance greater than 0.02 is determined to be a high noise level; a global grayscale variance between 0.01 and 0.02 is determined to be a medium noise level; and a global grayscale variance less than 0.01 is determined to be a low noise level.

[0036] In an optional embodiment, determining the local edge gradient of the original SEM image in step S101 may include: performing edge detection on the original SEM image; and determining the local edge gradient of the original SEM image based on the edge detection results.

[0037] Specifically, edge detection refers to processing the original SEM image using edge detection operators to identify the locations of pixels where grayscale values ​​change drastically. In this embodiment, the Sobel operator can be used for edge detection. The Sobel operator calculates approximate gradient values ​​in the horizontal and vertical directions of the image to obtain the gradient magnitude at each pixel, which is the local edge gradient of that pixel. For any pixel in the image, the larger the gradient magnitude, the more drastic the grayscale change at that point, and the higher the probability that the point belongs to the image edge region; the smaller the gradient magnitude, the more gradual the grayscale change at that point, and the higher the probability that the point belongs to a flat region of the image. Thus, a local edge gradient distribution map with the same size as the original SEM image can be obtained, where the value of each pixel represents the edge intensity at that location.

[0038] In an optional embodiment, the joint filtering in step S102 may include guided filtering and bilateral filtering. Guided filtering is a local linear filtering algorithm that uses a guide image to filter the input image. It establishes a linear relationship between the filtered output and the guide image within a local window centered on each pixel, effectively preserving the image's edge structure while filtering out noise. The size of the guided filtering window is a key parameter affecting its filtering performance: a larger window size results in a larger neighborhood involved in linear fitting, a smoother filtered output, and stronger noise suppression, but an excessively large window size may lead to over-smoothing of image edges; a smaller window size results in stronger edge preservation, but a correspondingly weaker noise reduction effect. Bilateral filtering is a nonlinear filtering algorithm that simultaneously considers pixel spatial proximity and gray-level similarity. It achieves filtering by jointly weighting Gaussian weights in the spatial and gray-level domains. The gray-scale standard deviation of a bilateral filter is a key parameter affecting its edge-preserving ability: the smaller the gray-scale standard deviation, the greater the weight difference between pixels with different gray values, and the pixels on both sides of the edge have less mutual influence due to the large gray-scale difference, thus the edge is preserved; the larger the gray-scale standard deviation, the weaker the influence of gray-scale similarity, the filter tends to be a regular Gaussian filter, and the edge-preserving ability decreases.

[0039] Based on the above principles, in this embodiment, the size of the guided filter window is determined according to the noise intensity level, and the higher the noise intensity level, the larger the filter window size. This is because a high noise level means that the image is severely contaminated by noise, requiring a larger filter window to provide sufficient noise suppression capability; a low noise level means that the image is less contaminated by noise, and a smaller filter window can meet the noise reduction requirements while better preserving image details.

[0040] In this embodiment, the gray-domain standard deviation of the bilateral filter is determined based on the local edge gradient, and the larger the local edge gradient, the smaller the gray-domain standard deviation. This is because in areas with large edge gradients, a smaller gray-domain standard deviation is needed to maintain edge sharpness during the filtering process, so that the pixels on both sides of the edge receive smaller weights in the gray-domain, thereby avoiding edge blurring; in areas with small edge gradients, a larger gray-domain standard deviation is used to give the filter a stronger smoothing effect, so as to fully suppress any noise that may remain in flat areas.

[0041] In one example, the window size of the guided filter can be set according to the noise intensity level: 7×7 for high noise levels, 5×5 for medium noise levels, and 3×3 for low noise levels. The spatial domain standard deviation of the bilateral filter can be fixed at σ_s=2, while the gray-scale domain standard deviation σ_r is dynamically adjusted based on the local edge gradient: when the local edge gradient is greater than 20 (strong edge region), σ_r takes 0.1 ~ 0.3 (weak filtering); when the local edge gradient is less than or equal to 20 (flat region), σ_r takes 0.5 ~ 0.8 (strong filtering).

[0042] In this embodiment, guided filtering and bilateral filtering are combined in series. First, guided filtering is performed on the original SEM image, and then the output of the guided filtering is used as the input image for bilateral filtering. Because guided filtering has excellent suppression capabilities for Gaussian-distributed electronic noise, and its filtering characteristics based on a local linear model can effectively preserve the overall edge structure of the image while reducing noise, it is used as the first-level filter to remove the main electronic noise in the original SEM image. Bilateral filtering further smooths flat areas based on guided filtering, and through dynamic adjustment of the gray-level standard deviation, it further refines the protection of edge details while maintaining the edge structure already achieved by guided filtering. Through the combined strategy of guided filtering for primary noise reduction and bilateral filtering for primary edge preservation, the two complement each other, effectively suppressing mixed noise while maximizing the preservation of microscopic edge details in the SEM image, providing a high-quality input image for subsequent super-resolution reconstruction processing.

[0043] It should be noted that the specific parameter values ​​for guided filtering and bilateral filtering described above are merely illustrative examples. Those skilled in the art can set different thresholds and parameter values ​​according to actual image quality and processing requirements. Furthermore, while maintaining the cascaded combination strategy of main noise reduction and main edge preservation, other traditional filtering algorithms can also be used to replace guided filtering or bilateral filtering. For example, non-local mean filtering can be used to replace guided filtering for noise reduction, or guided filtering can be used to replace bilateral filtering for edge preservation. These are all within the scope of this invention.

[0044] In an optional embodiment, the preset super-resolution network in step S103 can be constructed based on the standard ESPCN (Efficient Sub-Pixel Convolutional Neural Network) network. The standard ESPCN network typically includes a feature extraction part and an upsampling part. In this embodiment, the preset super-resolution network is an improvement upon the standard ESPCN network. Figure 3 This is a schematic diagram of a pre-defined super-resolution network according to an embodiment of the present invention, such as... Figure 3As shown, the feature extraction section adds a multi-scale feature extraction module to extract multiple scale features from the jointly filtered image, and fuses these features using a 1×1 convolution kernel to output the fused features. The upsampling section is used to upsample the fused features to generate the super-resolution reconstructed image.

[0045] The multi-scale feature extraction module can be configured with multiple parallel convolutional branches of different sizes (e.g., 3×3 and 5×5) to extract features from images under different receptive fields. This allows the network to simultaneously utilize local detail information and large-scale regional topography information, thereby more comprehensively reconstructing the microscopic texture and edge structure of SEM images. A 1×1 convolutional kernel is used to linearly combine and reduce the dimensionality of feature maps at multiple scales along the channel dimension, outputting a fused feature map with a uniform number of channels.

[0046] In an optional embodiment, a residual connection may be provided between the feature extraction part and the upsampling part. This connection is used to superimpose the output features of the multi-scale feature extraction module onto the input features of the upsampling part. This allows the upsampling part to directly acquire the original feature information in the low-resolution space when learning the high-resolution mapping, thereby alleviating the gradient vanishing problem in deep network training and accelerating network convergence. The feature extraction part may use the LeakyReLU activation function for non-linear activation. This function introduces a small non-zero slope (e.g., 0.01) when the input is negative, avoiding the neuron failure problem caused by the zero gradient in the negative region of the standard ReLU activation function. The upsampling part includes sub-pixel convolutional layers, and the number of convolutional kernels can be greater than that of the standard ESPCN network (e.g., increased from 64 to 128) to increase the number of trainable parameters, learn richer feature representations, and improve the quality of the reconstructed image.

[0047] It should be noted that, while maintaining the optimization logic of multi-scale feature extraction and residual connection, other lightweight super-resolution networks can also be used as the base network, such as replacing ESPCN with networks like FSRCNN and FPN. Only the parameter settings of the feature extraction module and the upsampling layer need to be adjusted adaptively, which is within the scope of this invention.

[0048] In one optional embodiment, the pre-defined super-resolution network is trained using a composite loss function, which includes a pixel-level loss term and an edge-weighted loss term. During training, this composite loss function uses a real high-resolution SEM image as a reference image. The pixel-level loss term constrains the difference in pixel-by-pixel grayscale values ​​between the reconstructed image output by the network and the reference image, ensuring overall pixel-level reconstruction accuracy. The edge-weighted loss term assigns a higher weight to the error in edge regions compared to the reference image compared to non-edge regions, allowing the prediction error of edge pixels to account for a larger proportion of the total loss, thereby guiding the network to focus more on the restoration of microscopic edge structures.

[0049] Specifically, during training, the Adam optimizer can be used to optimize the network parameters, with a learning rate of 0.001 and 200 iterations. Kaiming Normal Initialization can be used to initialize the convolutional kernels. The pixel-level loss term uses mean squared error loss, and the edge-weighted loss term uses edge-aware L1 loss. The weight ratio of these two terms can be set to 0.4:0.6, but this ratio is only an example. The expression for the composite loss function is: ; in, This is the mean square error loss term. This is the edge-aware L1 loss term.

[0050] The expression is: ; in, N Total number of pixels To reconstruct the image at the pixel level Pixel value at that location, For reference image at pixel points The pixel value at that location.

[0051] Edge maps between the reconstructed image and the reference image are extracted using the Canny operator. The L1 error of the edge region is calculated, and the edge pixel error is given a high weight. The expression is as follows: ; in, M The number of edge pixels, and The reconstructed image and the reference image are respectively located at the pixel level. Pixel value at that location, For weighting coefficients (e.g., edge pixels) =2, non-edge pixels =1).

[0052] After the network is trained, the image output after joint filtering is input into the trained network, and the network can then output a high-resolution image after super-resolution reconstruction. This network can support different resolution upscaling factors such as 2x, 3x, or 4x.

[0053] Figure 2 This is a flowchart of a filter parameter update method according to an embodiment of the present invention, as follows: Figure 2 As shown, the steps of performing quality assessment on the super-resolution reconstructed image and updating the filter parameters of the joint filter based on the quality assessment results may include steps S201 to S204.

[0054] Step S201: Calculate the edge gradient difference between the super-resolution reconstructed image and the reference high-resolution image. The reference high-resolution image can be a pre-acquired high-resolution SEM image within the same field of view as the original SEM image, or a high-quality reference image obtained by registering and fusing multiple low-resolution SEM images. The edge gradient difference is used to evaluate the closeness of the reconstructed image to the reference high-resolution image in terms of edge structure. It is calculated as follows: edge detection is performed on both the reconstructed image and the reference high-resolution image to obtain their respective edge gradient maps. Then, the pixel-by-pixel difference between the two edge gradient maps (e.g., the sum of absolute differences or mean square error) is calculated. This difference is the edge gradient difference. A larger edge gradient difference indicates that the edges of the reconstructed image are more blurred or have a larger deviation compared to the reference image.

[0055] Step S202: When the edge gradient difference is greater than the preset edge gradient difference threshold, reduce the grayscale standard deviation of the bilateral filter.

[0056] In this step, when the edge gradient difference is greater than the preset edge gradient difference threshold, it indicates that the current joint filtering is not protecting the edges enough, resulting in blurred edges in the reconstructed image. At this time, reducing the gray-scale standard deviation of the bilateral filter can enhance the edge preservation capability in the next round of filtering.

[0057] Step S203: Calculate the global gray-level variance of the super-resolution reconstructed image. The global gray-level variance is used to evaluate the overall uniformity of gray-level distribution and the degree of noise residue in the reconstructed image. It is calculated by statistically analyzing the gray-level values ​​of all pixels in the reconstructed image and averaging the squares of the differences between these values ​​and the image's gray-level mean. A larger global gray-level variance indicates more residual noise in the reconstructed image.

[0058] Step S204: When the global grayscale variance is greater than the preset grayscale variance threshold, increase the size of the filter window of the guided filter.

[0059] In this step, when the global gray-level variance is greater than the preset gray-level variance threshold, it indicates that the current joint filtering is insufficient to suppress noise, resulting in a lot of noise remaining in the reconstructed image. At this time, increasing the size of the guided filter window can enhance the noise suppression capability in the next round of filtering.

[0060] In this embodiment, the edge gradient difference threshold and the gray-level variance threshold can be preset according to the image quality requirements of the actual application scenario. For example, the edge gradient difference threshold can be set to 15 and the gray-level variance threshold can be set to 0.01. However, the above values ​​are only examples, and those skilled in the art can set different thresholds according to specific needs.

[0061] It is understandable that if the edge gradient difference is less than or equal to the preset edge gradient difference threshold, it indicates that the edge structure of the reconstructed image is well restored, and the current bilateral filtering parameters are already able to effectively protect the edges. Therefore, it is only necessary to keep the gray-level standard deviation of the bilateral filter unchanged. If the global gray-level variance is less than or equal to the preset gray-level variance threshold, it indicates that there is little residual noise in the reconstructed image, and the current guided filtering parameters are already able to effectively suppress noise. Therefore, it is only necessary to keep the guided filtering window size unchanged.

[0062] In one optional embodiment, the preset image quality standard is: the edge gradient difference is less than or equal to a preset edge gradient difference threshold, and the global grayscale variance is less than or equal to a preset grayscale variance threshold.

[0063] When both of the above conditions are met simultaneously, the reconstructed image quality is deemed satisfactory, the iterative process terminates, and the current reconstructed image is output as the final high-resolution, low-noise, high-fidelity SEM image. If either condition is not met, the filtering parameters are updated based on the corresponding quality assessment result, and the iterative process of joint filtering, super-resolution reconstruction, and quality assessment is re-executed. In practical applications, a maximum number of iterations (e.g., 3 to 5 rounds) can be preset to prevent the iteration from becoming infinite due to extreme image quality issues that make convergence difficult.

[0064] It should be noted that, while maintaining the core idea of ​​super-resolution effect feedback and filter parameter adjustment, other image quality assessment indicators can be used to replace edge gradient difference and global gray-level variance. For example, the structural similarity index (SSIM) can be used as the basis for parameter adjustment. All of these fall within the scope of this invention.

[0065] In one specific embodiment, a low-resolution raw SEM image is first acquired. This image is typically subject to interference from electronic noise (following a Gaussian distribution) and / or particle noise (following an impulse distribution), and the microscopic texture details are blurred, making it unsuitable for direct use in high-precision critical dimension measurement or micro-defect detection.

[0066] The pixel values ​​of the acquired SEM raw image are mapped to the range [0,1]. Noise is assessed in the normalized image by calculating the gray-level histogram and global gray-level variance. Based on the magnitude of the global gray-level variance, the noise intensity is divided into several preset levels; for example, a global gray-level variance greater than 0.02 indicates a high noise level, between 0.01 and 0.02 indicates a medium noise level, and less than 0.01 indicates a low noise level. Furthermore, the distribution pattern of the gray-level histogram can also help determine the dominant noise type of the image, but this determination result does not participate in the subsequent adjustment of filtering parameters.

[0067] Based on the noise intensity level in the noise assessment results, guided filtering is used to denoise the normalized image. The filter window size of the guided filter is adaptively adjusted according to the noise intensity level: 7×7 for high noise, 5×5 for medium noise, and 3×3 for low noise. Subsequently, bilateral filtering is used to preserve the edges of the image after guided filtering. The spatial domain standard deviation σ_s of the bilateral filter is fixed at 2. The local edge gradient of the image is calculated using the Sobel operator, and the gray domain standard deviation σ_r is adaptively adjusted according to the local edge gradient: when the local edge gradient is greater than 20, σ_r is 0.1 ~ 0.3; when the local edge gradient is less than or equal to 20, σ_r is 0.5 ~ 0.8. Through the cascade combination strategy of guided filtering and bilateral filtering, a preprocessed image with low noise and high edge preservation is output.

[0068] The preprocessed image is input into a pre-defined improved ESPCN network for super-resolution reconstruction. The improved ESPCN network is derived from the standard ESPCN network with three targeted improvements: a multi-scale feature extraction module is added to the feature extraction part, with two parallel convolutional branches of 3×3 and 5×5, and feature fusion is performed through 1×1 convolutional kernels; the ReLU activation function is replaced with LeakyReLU (with a negative slope of 0.01); a residual connection is added between the multi-scale feature extraction module and the upsampling layer; and the number of convolutional kernels in the sub-pixel convolutional layer is increased from 64 in the standard ESPCN network to 128.

[0069] The quality of the super-resolution reconstructed image is evaluated by calculating its edge gradient difference and global gray-level variance compared to the reference high-resolution image. When the edge gradient difference exceeds a preset edge gradient difference threshold, the gray-level standard deviation of the bilateral filter is reduced; when the global gray-level variance exceeds a preset gray-level variance threshold, the filter window size of the guided filter is increased. Based on the adjusted filter parameters, the joint filtering and super-resolution reconstruction process is re-executed, and the above quality evaluation and parameter adjustment process is repeated until the evaluation result meets the preset image quality standard, and the optimized high-resolution image is output.

[0070] The SEM image super-resolution reconstruction methods described in the above embodiments can be implemented by computer program instructions. Therefore, this embodiment also provides a computer program product 41, a computer-readable storage medium 42, and a computer device 43. Figure 4 This is a schematic diagram of a computer program product according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Figure 6 This is a schematic block diagram of a computer device according to an embodiment of the present invention.

[0071] Computer program product 41 includes computer program 411, which, when executed by processor 431, implements any of the aforementioned SEM image super-resolution reconstruction methods. Computer-readable storage medium 42 stores the aforementioned computer program 411, which, when executed by processor 431, implements any of the aforementioned SEM image super-resolution reconstruction methods. Computer device 43 may include memory 432, processor 431, and computer program 411 stored in memory 432 and running on processor 431.

[0072] The computer program 411 used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages.

[0073] Computer program 411 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of the invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0074] For the purposes of this embodiment, computer program product 41 refers to a related product containing computer program 411. Computer-readable storage medium 42 is a tangible device capable of holding and storing computer program 411, and can be any device capable of containing, storing, communicating, propagating, or transmitting computer program 411 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 42 include: portable computer disks, hard disks, random access memory 432 (RAM), read-only memory 432 (ROM), erasable programmable read-only memory 432 (EPROM or flash memory), static random access memory 432 (SRAM), portable optical disc read-only memory 432 (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0075] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for super-resolution reconstruction of SEM images, comprising: Obtain the original SEM image and determine the noise intensity level and local edge gradient of the original SEM image; The filtering parameters for joint filtering are determined based on the noise intensity level and the local edge gradient, and the original SEM image is jointly filtered according to the filtering parameters. The image after joint filtering is reconstructed using a preset super-resolution network; The image after super-resolution reconstruction is subjected to quality assessment, and the filtering parameters of the joint filter are updated based on the results of the quality assessment. The process of joint filtering, super-resolution reconstruction, and quality assessment is repeated until the result of the quality assessment indicates that the image quality meets the preset image quality standard.

2. The SEM image super-resolution reconstruction method according to claim 1, characterized in that, The joint filtering includes guided filtering and bilateral filtering. The step of determining the filtering parameters of the joint filtering based on the noise intensity level and the local edge gradient includes: The size of the filter window for the guided filter is determined based on the noise intensity level, and the higher the noise intensity level, the larger the filter window size. The gray-scale standard deviation of the bilateral filter is determined based on the local edge gradient, and the larger the local edge gradient, the smaller the gray-scale standard deviation.

3. The SEM image super-resolution reconstruction method according to claim 2, characterized in that, The steps of performing a quality assessment on the super-resolution reconstructed image and updating the filter parameters of the joint filter based on the quality assessment results include: Calculate the edge gradient difference between the super-resolution reconstructed image and the reference high-resolution image; When the edge gradient difference is greater than the preset edge gradient difference threshold, the grayscale standard deviation of the bilateral filter is reduced. Calculate the global grayscale variance of the image after the super-resolution reconstruction process; When the global grayscale variance is greater than the preset grayscale variance threshold, the size of the filter window of the guided filter is increased.

4. The SEM image super-resolution reconstruction method according to claim 3, characterized in that, The preset image quality standard is: The edge gradient difference is less than or equal to the preset edge gradient difference threshold, and the global grayscale variance is less than or equal to the preset grayscale variance threshold.

5. The SEM image super-resolution reconstruction method according to claim 1, characterized in that, The preset super-resolution network is constructed based on the standard ESPCN network and includes a feature extraction part and an upsampling part; The feature extraction part includes a multi-scale feature extraction module, which is used to extract multiple scale features of the jointly filtered image and fuse the multiple scale features through a 1×1 convolution kernel to output fused features. The upsampling portion is used to upsample the fused features to generate a super-resolution reconstructed image.

6. The SEM image super-resolution reconstruction method according to claim 5, characterized in that, A residual connection is provided between the feature extraction part and the upsampling part. The residual connection is used to superimpose the output features of the multi-scale feature extraction module onto the input features of the upsampling part. The feature extraction part uses the LeakyReLU activation function for non-linear activation; The upsampling portion includes subpixel convolutional layers, and the number of convolutional kernels in the subpixel convolutional layers is greater than the number of convolutional kernels in the standard ESPCN network.

7. The SEM image super-resolution reconstruction method according to claim 5, characterized in that, The preset super-resolution network is trained using a composite loss function, which includes a pixel-level loss term and an edge-weighted loss term. The edge-weighted loss term assigns a higher weight to the error in the image edge region than to the error in the non-edge region.

8. The SEM image super-resolution reconstruction method according to claim 1, characterized in that, The steps for determining the noise intensity level of the raw SEM image include: The original SEM image is subjected to grayscale normalization processing, and the global grayscale variance of the original SEM image is calculated. Based on the magnitude of the global grayscale variance of the original SEM image, determine the noise intensity level of the original SEM image; and / or The steps for determining the local edge gradient of the original SEM image include: Edge detection is performed on the original SEM image; Based on the edge detection results, the local edge gradient of the original SEM image is determined.

9. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the SEM image super-resolution reconstruction method according to any one of claims 1 to 8.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the SEM image super-resolution reconstruction method according to any one of claims 1 to 8.

Images