Scanning electron microscope image denoising method based on noise model
By constructing the SEM noise model and generating simulated noise images, and combining with the CNN denoising network for training, the problem of noise removal difficulties in SEM images in the semiconductor field is solved, efficient denoising processing and structure maintenance are achieved, and data costs are reduced.
Patent Information
- Application Number
- CN202510219767.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-26
- Publication Date
- 2025-06-13
AI Technical Summary
In the semiconductor field, especially in scanning electron microscopy (SEM) image processing, the prior art is difficult to effectively remove image noise without damaging the image structure, and due to the lack of clean data, the data cost of model training is relatively high.
By building a noise model based on SEM imaging principle, fit the noise parameters and generate simulated noise images, combine it with the CNN denoising network for training, and use dual-channel parallel format for denoising, avoiding the need for clean data.
It realizes effective removal of SEM image noise, maintaining image structure without requiring clean data, reducing the data cost of model training, and improving the performance of the denoiser.
Smart Images
Figure CN120147172A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image denoising, and in particular to a method for denoising scanning electron microscope images based on a noise model. Background Art
[0002] Image denoising is a basic task in the field of computer vision, and among them, the image denoising algorithm based on Convolutional Neural Network (CNN) has achieved good denoising performance. The image denoising algorithm based on CNN is mainly divided into two stages, namely the training stage and the testing stage. In the training stage, first, a noisy image is input into the CNN, and the output result of the CNN network is compared with the clean image. The difference between the predicted output and the target image is calculated through a loss function, and then the network weights are updated based on the backpropagation algorithm to minimize the loss function. This process requires a large number of labeled samples (noisy images and corresponding clean images) for training. In the testing stage, given a noisy image, the CNN will remove the noise as much as possible according to the features and patterns learned in the training stage. By forward-propagating the input image in the network, a denoised output image is finally obtained.
[0003] However, there are still some fields where it is difficult to obtain a large number of pairs of noisy images and clean images, especially in the semiconductor field. Process optimization is an important step in the semiconductor production and manufacturing process. Scanning Electron Microscope (SEM) images are an important basis for semiconductor process optimization. While containing process features, they also contain a large amount of image noise, which brings great difficulties to the process optimization based on SEM images. And due to the complex imaging characteristics of the SEM, it is also difficult to obtain clean images. The solution to this problem in the prior art is to adopt a self-supervised learning strategy, relying only on the internal structure and relationship of the noisy data to learn from the noisy data itself. However, these existing technologies all have certain requirements for the statistical characteristics of the image noise itself. Although the denoiser can be trained without clean data, the performance of the trained model is greatly limited. Therefore, how to reduce the data cost overhead in the model training stage while maintaining the performance of the neural network denoiser is a problem that the prior art needs to solve. Summary of the Invention
[0004] The purpose of the present invention is to provide a method for denoising scanning electron microscope images based on a noise model. Compared with the traditional non-blind denoising network, this method can effectively remove a large amount of noise in the SEM image while effectively maintaining the structure of the original SEM image, and can effectively remove the noise in the SEM image without the premise of clean data.
[0005] The object of the present invention is achieved by the following technical solutions:
[0006] A method for denoising scanning electron microscope images based on a noise model, the method comprising:
[0007] Step 1: By analyzing the imaging principle of a scanning electron microscope (SEM), construct an SEM noise model;
[0008] Step 2: Based on the constructed SEM noise model, perform noise parameter fitting on the SEM image Y to be denoised to obtain noise parameters k and σ 2 , and generate a simulated noise image N based on the noise parameters s1 and N s2 ;
[0009] Step 3: Train a CNN denoising network, and use the image Z = Y + N after re - adding noise s1 as one of the channels, and the other channel is the simulated noise image N s2 , and send them into the input end of the CNN denoising network in the form of dual - channel parallelism. The output end is the SEM image Y to be denoised, and repeat the training until the CNN denoising network converges;
[0010] Step 4: In the test stage, send the SEM image Y to be denoised and the simulated noise image N s2 into the converged CNN denoising network together to achieve denoising of the SEM image.
[0011] It can be seen from the above - mentioned technical solutions provided by the present invention that, compared with traditional non - blind denoising networks, the above - mentioned method can effectively remove a large amount of noise in the SEM image while effectively maintaining the structure of the original SEM image, remove the noise in the SEM image effectively without the need for clean data, thereby reducing the training cost of the CNN denoiser and maintaining the performance of the denoiser. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following - described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0013] Figure 1 It is a schematic flowchart of the method for denoising scanning electron microscope images based on a noise model provided by the embodiments of the present invention;
[0014] Figure 2 It is a schematic diagram of the process of fitting the SEM noise parameters described in the embodiments of the present invention;
[0015] Figure 3 Schematic diagram of the denoising results of SEM images of different methods in the examples of the present invention. Specific implementation manners
[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments, which does not constitute a limitation to the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.
[0017] As Figure 1 shown in the schematic flowchart of the method for denoising scanning electron microscope images based on a noise model provided by the embodiment of the present invention, the method includes:
[0018] Step 1: By analyzing the imaging principle of a scanning electron microscope (SEM), construct an SEM noise model;
[0019] In this step, in Step 1, by analyzing the imaging principle of a scanning electron microscope (SEM), a noise model of the SEM is introduced, and the mathematical expression of the model is shown in formula (1):
[0020] Var(x) = kE(x) + σ 2 (1)
[0021] where k and σ 2 represent noise parameters; x represents the pixel intensity value after being disturbed by noise; E(x) represents the pixel expectation after being disturbed by noise; Var(x) represents the pixel variance after being disturbed by noise.
[0022] Step 2: Based on the constructed SEM noise model, perform noise parameter fitting on the SEM image Y to be denoised to obtain the noise parameters k and σ 2 , and generate a simulated noise image N s1 and N s2 (two samplings under the same noise distribution);
[0023] In this step, since it is necessary to calculate the pixel expectation and variance after being disturbed by noise, it is necessary to statistically analyze the same pixel random event, such as Figure 2The figure shows a schematic diagram of the process of fitting SEM noise parameters according to an embodiment of the present invention. The SEM image Y to be denoised is segmented, and the gray intensity distribution of each segmented noise sample (image block) is balanced. The expectation and variance (data points) of each noise sample are calculated. Defining I(i,j) as the signal intensity at any pixel position in the image block I, the above calculation process can be expressed as:
[0024]
[0025] Among them, E(I) represents the expectation of the image block I; Var(I) represents the variance of the image block I; m and n respectively represent the pixel length and width of the image block I;
[0026] Taking the expectation as the X-axis and the variance as the Y-axis, a straight line is fitted by the least squares method. The slope of the straight line is the noise parameter k, and the intercept of the straight line on the Y-axis is the noise parameter σ 2 ;
[0027] Based on the noise parameters k and σ 2 , according to the constructed SEM noise model, a simulated noise image N s1 and N s2 are generated. Among them, the SEM image Y to be denoised and the simulated noise image N s1 are summed pixel by pixel to generate a re-noised image Z, that is, Z = Y + N s1 .
[0028] Step 3: Train the CNN (Convolutional Neural Network) denoising network. Take the re-noised image Z = Y + N s1 as one of the channels, and the other channel is the simulated noise image N s2 , and send them into the input end of the CNN denoising network in the form of dual-channel parallelism. The output end is the SEM image Y to be denoised, and repeat the training until the CNN denoising network converges;
[0029] In this step, during the training of the CNN denoising network in step 3, the loss function adopted is the L2 loss, and its mathematical expression is:
[0030] L(y i ,x i )=(y i -x i ) 2 (3)
[0031] Among them, x i represents the noisy image; y idenotes the corresponding clean image; assuming there are N pairs of noisy-clean images in the training set, then the subscript i represents any one of these pairs of images, where 0 < i < N + 1;
[0032] The above loss function is the negative logarithm of the likelihood function, and the optimization process of the loss function can be regarded as maximum likelihood estimation.
[0033] The embodiment of this application avoids the need for clean data in the training stage by the form of adding noise again.
[0034] Step 4: In the test stage, the SEM image Y to be denoised and the simulated noise image N s2 are fed into the converged CNN denoising network together to achieve SEM image denoising.
[0035] In this step, the specific mathematical description is expressed as:
[0036]
[0037] where denotes the denoised image; D() represents the converged CNN denoising network model.
[0038] It should be noted that the content not described in detail in the embodiments of the present invention belongs to the prior art well-known to those skilled in the art.
[0039] Next, specific examples are used to illustrate the process and effect of the method described in the present invention. This example applies the present invention to the SEM image denoising task of SRAM content, and two experiments are specifically designed to verify the innovation and practicability of the present invention:
[0040] The first experiment: First, an accurate noise-added dataset is generated based on the SEM noise model proposed in the present invention, and then 4 mutually independent fuzzy noise-added datasets are randomly generated. Based on these five datasets, the method described in the present invention (i.e., the Noisier2Noisy strategy) is respectively applied to train the CNN denoiser, where the network model used is Unet. Regarding the five noise-added datasets, based on the SEM noise model proposed in the present invention, noise parameter estimation is performed on the SEM images of SRAM content used in the experiment: Using this as the accurate noise parameter to add noise to the original noisy SEM image again; the noise parameters used for the other four noise-added datasets are respectively and The following Table 1 shows the comparison of the denoising performance of the networks trained with different datasets:
[0041] Table 1 Comparison of the denoising performance of the networks trained with different datasets
[0042] Noise level Accurate / Fuzzy Peak signal-to-noise ratio (dB) Structural similarity k＝0.39 Fuzzy 33.52 0.9453 <![CDATA[σ 2 = 28]]> Fuzzy 32.50 0.9402 <![CDATA[k = 0.20, σ 2 = 14]]> Fuzzy 33.22 0.9468 <![CDATA[k = 0.59, σ 2 = 42]]> Fuzzy 32.98 0.9396 <![CDATA[k = 0.39, σ 2 = 28]]> Accurate 34.13 0.9593
[0043] In the second experiment, the Noisier2Noisy strategy proposed by the present invention was applied to train the CNN denoiser of different network models, and the model training of the CNN denoiser was carried out only based on the accurately denoised dataset. In order to prove the applicability of the present invention, experiments were conducted on two network models, Unet and DnCNN respectively. In order to prove the innovation of the present invention, this example was also experimentally compared with other denoising techniques in the prior art, including the image denoising method BM3D and the self-supervised image denoising method N2V.
[0044] In the above two experiments, during the model training process, the initial learning rate η = 0.001, and it was reduced to half of the original every 50 rounds, batch_size = 32, and the maximum number of iterations was 2000 epochs.
[0045] The present invention (Noisier2Noisy) was compared with the traditional image denoising method BM3D and the self-supervised image denoising method N2V respectively. The performance after denoising the SEM images of SRAM content by different methods was recorded respectively, where the peak signal-to-noise ratio and the structural similarity were used to evaluate the denoising performance of different methods. The specific performance results are shown in Table 2:
[0046] Table 2 Comparison of SEM denoising performance of different methods
[0047] Method Blind / Non-blind Peak signal-to-noise ratio (dB) Structural similarity BM3D Non-blind 32.10 0.8994 N2V Blind 30.45 0.8969 Noisier2Noisy+Unet Non-blind 34.13 0.9593 Noisier2Noisy+DnCNN Non-blind 34.17 0.9434
[0048] As Figure 3 shown in the schematic diagram of the SEM image denoising results of different methods in the examples of the present invention, it can be seen from Table 2 and Figure 3 that the method described in the embodiments of the present invention, that is, the Noisier2Noisy strategy, can not only be widely applied to multiple CNN model architectures, but also has better denoising performance while maintaining the original structure of the image compared with several existing methods.
[0049] In summary, the method (Noisier2Noisy strategy) described in the embodiments of the present invention has the following advantages:
[0050] 1. The present invention estimates the noise parameters of the noisy SEM image based on the SEM noise model, generates a simulated noise image according to the noise parameters, and sums it with the original noisy SEM image pixel by pixel to generate a double-noise SEM image, realizing data augmentation based on model knowledge, and can effectively reduce the data cost of model training;
[0051] 2. The present invention realizes the training of the CNN denoiser by learning the denoising mapping between the SEM image after data augmentation (double noise) and the original noisy (single noise) SEM image. Compared with other training methods, the present invention does not require clean target data;
[0052] 3. The present invention adds the simulated noise image as prior knowledge to network training, enabling the trained network to achieve non-blind noise estimation and effectively improving the denoising performance and robustness of the denoiser.
[0053] 4. The Noisier2Noisy strategy proposed by the present invention is also applicable to training any CNN denoiser other than Unet and DnCNN, and has better denoising effects than traditional self-supervised image denoising methods.
[0054] In addition, those of ordinary skill in the art can understand that all or part of the steps in implementing the above-described embodiment methods can be completed by instructing relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, or the like.
[0055] As described above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims. The information disclosed in the background art part of this article is only intended to deepen the understanding of the overall background art of the present invention, and should not be regarded as an admission or any form of implication that this information constitutes the prior art already known to those skilled in the art.
Claims
1. A scanning electron microscope image denoising method based on a noise model, characterized in that: The method comprises: Step 1: Construct a SEM noise model by analyzing the imaging principle of a scanning electron microscope (SEM); Step 2: Based on the constructed SEM noise model, noise parameters are fitted on the denoised SEM image Y to obtain noise parameters k and σ 2 , and generate a simulated noise image N based on the noise parameters s1 and N s2 ; Step 3: Train the CNN denoising network and add the noise again to the image Z=Y+N s1 As one of the channels, the other channel is a simulated noise image N s2 , is sent to the input of the CNN denoising network in the form of dual channels in parallel, and the output is the SEM image Y to be denoised. The training is repeated until the CNN denoising network converges; Step 4: In the test phase, the SEM image to be denoised Y and the simulated noise image N s2 They are then sent to the converged CNN denoising network to achieve SEM image denoising.
2. The scanning electron microscope image denoising method based on the noise model according to claim 1, characterized in that: In step 1, by analyzing the imaging principle of the scanning electron microscope (SEM), the noise model of the SEM is introduced. The mathematical expression of the model is shown in formula (1): Var(x)=kE(x)+σ 2 (1) Among them, k and σ 2 represents the noise parameter; x represents the pixel intensity value after being interfered by noise; E(x) represents the pixel expectation after being interfered by noise; Var(x) represents the pixel variance after being interfered by noise.
3. The scanning electron microscope image denoising method based on noise model according to claim 2, characterized in that: In step 2, the denoised SEM image Y is segmented, the grayscale intensity distribution of each noise sample after segmentation is balanced, and the expectation and variance of each noise sample are calculated. I(i, j) is defined as the signal intensity of any pixel position in the image block I. The above calculation process is expressed as: Where E(I) represents the expectation of image block I; Var(I) represents the variance of image block I; m and n represent the pixel length and width of image block I respectively; With expectation as X-axis and variance as Y-axis, a straight line is fitted by the least squares method. The slope of the straight line is the noise parameter k, and the intercept of the straight line on the Y-axis is the noise parameter σ 2 ; Based on the noise parameters k and σ 2 , based on the constructed SEM noise model, generate a simulated noise image N s1 and N s2 , where the SEM image to be denoised Y and the simulated noise image N s1 The image Z after adding noise again is generated by summing the pixels, that is, Z = Y + N s1 .
4. The scanning electron microscope image denoising method based on noise model according to claim 1, characterized in that: In the process of training the CNN denoising network in step 3, the loss function used is L2 loss, and its mathematical expression is: L(and i ,x i )=(and i -x i ) 2 (3) where x i represents a noisy image; y i represents the corresponding clean image; assuming that there are N pairs of noise-clean images in the training set, then the subscript i represents any pair of images, 0 <i<N+1; The above loss function is the negative logarithm of the relief function, and the optimization process of the loss function can be regarded as the maximum likelihood estimation.
5. The scanning electron microscope image denoising method based on noise model according to claim 1, characterized in that: In step 4, the specific mathematical description is expressed as: in, represents the denoised image; D() represents the CNN denoising network model after convergence.