An IC equipment alignment mark image enhancement method and image enhancement device

Abstract

The application provides an IC equipment alignment mark image enhancement method and an image enhancement device, wherein the method comprises: acquiring a process degradation mark image to be enhanced; performing real-time quality evaluation on the process degradation mark image to be enhanced by using a preset image quality evaluation algorithm; if the evaluation result is lower than a preset quality threshold, inputting the process degradation mark image to a trained cycle generative adversarial network to output an enhanced alignment mark image; the cycle generative adversarial network is trained in the following manner: training the network by using a non-paired training data set, and in the training process, optimizing the network by using a loss function comprising a structural similarity loss and an edge phase consistency loss, and training and reasoning by using a dynamic slicing strategy. By the application, the signal-to-noise ratio, edge sharpness and geometric position fidelity of the alignment mark image are improved without paired data, thereby improving the alignment accuracy.

Description

Technical Field

[0001] This application relates to the field of semiconductor manufacturing technology, and more specifically, to an image enhancement method and an image enhancement device for IC equipment alignment marks. Background Technology

[0002] Currently, the overlay accuracy of integrated circuit (IC) equipment is a key factor determining chip performance and yield, and its realization highly depends on the precise capture of alignment mark positions on the wafer before exposure. However, in advanced semiconductor manufacturing processes, image quality is affected by a combination of complex factors: processes such as chemical mechanical polishing (CMP), high aspect ratio etching, and multilayer thin film deposition can severely damage the physical morphology of the marks and introduce non-uniform optical interference; optical imaging defects caused by defocusing or jitter result in blurring or ghosting; and electrical noise such as shot noise and readout noise introduced by camera sensors lead to low contrast, low brightness, and severe signal-to-noise ratio degradation in the image. Image quality directly affects the measurement accuracy of the alignment system.

[0003] To improve image quality, traditional image processing methods rely on idealized noise models and manual parameter tuning, which struggle to cope with complex nonlinear process interference in actual production and have limited generalization capabilities. In recent years, deep learning-based image enhancement technologies have made progress, but mainstream supervised learning methods require a large number of paired "sharp-degraded" images for training, which is difficult to achieve in actual production lines due to the inability to obtain a perfectly lossless state for the same marker. While unsupervised generative adversarial networks (GANs) that do not require paired data offer a solution, they are prone to producing texture "illusions" when enhancing images, leading to imperceptible pixel-level geometric positional drift at critical edges of markers. For the alignment of IC equipment (Integrated Circuit Equipment) with nanometer-level precision, this geometric distortion directly introduces measurement errors, representing a major drawback of existing unsupervised deep learning methods in practical applications. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image enhancement method and an image enhancement device for IC equipment alignment marks, so as to overcome at least one of the above-mentioned defects.

[0005] In a first aspect, embodiments of this application provide a method for enhancing alignment mark images of IC equipment. The method includes: acquiring a process degradation mark image to be enhanced; performing real-time quality assessment on the process degradation mark image to be enhanced using a preset image quality evaluation algorithm, and determining whether the assessment result is lower than a preset quality threshold; if the assessment result is lower than the quality threshold, inputting the process degradation mark image into a trained recurrent generative adversarial network (RGAN) and outputting the enhanced alignment mark image; if the assessment result is higher than or equal to the quality threshold, not performing enhancement and outputting the original image; wherein the RGAN is trained in the following manner: training the network using an unpaired training dataset, and during the training process, optimizing the network using a loss function that includes structural similarity loss and edge phase consistency loss, and using a dynamic slicing strategy for training and inference.

[0006] In one optional embodiment of this application, the training dataset includes process degradation labeled domain images and ideal standard labeled domain images. The training dataset is constructed as follows: multi-source degradation images covering physical morphological damage, optical interference texture, system imaging blur, and sensor electrical noise are collected from the actual production line; the degradation features of the original labeled images to be screened are quantified using the image quality evaluation algorithm; images whose quantization indicators conform to the degradation distribution characteristics are selected as process degradation labeled domain images; simulated labeled images with ideal edge characteristics are generated based on the imaging simulation model as ideal standard labeled domain images; and the images of the two domains are normalized and their sizes are unified to form a non-paired training dataset.

[0007] In one optional embodiment of this application, the recurrent generative adversarial network includes a forward generator, a backward generator, a first discriminator, and a second discriminator, wherein the forward generator is used to map a process degradation labeling domain image to an ideal standard labeling domain image; the backward generator is used to map the ideal standard labeling domain image to a process degradation labeling domain image; the first discriminator is used to distinguish between the image generated by the forward generator and the real ideal standard labeling domain image; and the second discriminator is used to distinguish between the image generated by the backward generator and the real process degradation labeling domain image.

[0008] In one optional embodiment of this application, both the forward generator and the reverse generator adopt an architecture based on a fully convolutional architecture where the encoder, converter, and decoder are connected in sequence, and the converter introduces a dilated convolution module; both the first discriminator and the second discriminator adopt a structure based on local patches for discrimination.

[0009] In one optional embodiment of this application, the structural similarity loss is obtained by: within a preset local window, calculating the pixel mean, pixel standard deviation, and covariance between the network input image and the network generated image; based on the pixel mean, pixel standard deviation, and covariance, calculating a structural similarity index that measures the similarity between the two images in three dimensions: brightness, contrast, and structure; and constructing the structural similarity loss based on the structural similarity index.

[0010] In one optional embodiment of this application, the edge phase consistency loss is obtained as follows: using a multi-scale, multi-directional logarithmic Gabor filter bank, frequency domain filtering is performed on the network input image and the network generated image respectively to obtain the filter response at each scale and direction; based on the filter response, the local energy of each pixel and the sum of the amplitudes at all scales are calculated respectively; based on the sum of the local energy and the amplitude, the phase consistency maps corresponding to the network input image and the network generated image are calculated respectively, and the high-value regions in the phase consistency maps represent the physical edge positions of the image; the difference norm between the two phase consistency maps is calculated, and the edge phase consistency loss is constructed based on the difference norm.

[0011] In one optional embodiment of this application, the dynamic slicing strategy is implemented through the following steps: During the training phase, the gray-level centroid of the full-resolution training image is calculated to determine the marker center, and a region of interest of a preset size is delineated based on the marker center. Local slices of a fixed size are randomly cropped from the region of interest, and random rotation and / or flipping enhancement operations are applied to the local slices as network input; During the inference phase, based on the fully convolutional architecture characteristics of the recurrent generative adversarial network, the recurrent generative adversarial network supports image input of arbitrary size; If the size of the process degradation marker image to be enhanced is greater than a preset threshold, it is divided into multiple sub-image blocks for distributed inference using an overlapping sliding window method, and the overlapping areas of each processed sub-image block are weighted and fused to eliminate stitching seams and output a complete enhanced image.

[0012] In one optional embodiment of this application, the loss function is a total loss function, which is constructed by weighting and summing the adversarial loss, cycle consistency loss, structural similarity loss, and edge phase consistency loss to obtain the total loss function.

[0013] In one optional embodiment of this application, the phase consistency map is obtained by: subtracting a preset noise threshold from the local energy and taking the positive value to obtain the effective local energy; dividing the effective local energy by the sum of the amplitudes at all scales and the sum of a constant to prevent division by zero to obtain the phase consistency value of each pixel; and forming the phase consistency map based on the phase consistency value of each pixel.

[0014] Secondly, embodiments of this application also provide an image enhancement device, the device comprising: a process degradation marker image acquisition module, used to acquire a process degradation marker image to be enhanced; an image output module, used to perform real-time quality evaluation of the process degradation marker image to be enhanced using a preset image quality evaluation algorithm, and determine whether the evaluation result is lower than a preset quality threshold; if the evaluation result is lower than the quality threshold, the process degradation marker image is input into a trained recurrent generative adversarial network, and the enhanced alignment marker image is output; if the evaluation result is higher than or equal to the quality threshold, no enhancement is performed, and the original image is output; wherein, the recurrent generative adversarial network is trained in the following manner: the network is trained using an unpaired training dataset, and during the training process, a loss function including structural similarity loss and edge phase consistency loss is used to optimize the network, and a dynamic slicing strategy is used for training and inference.

[0015] The IC equipment alignment mark image enhancement method and image enhancement device provided in this application acquire a process degradation mark image to be enhanced; input the process degradation mark image into a trained recurrent generative adversarial network (RGAN), and output the enhanced alignment mark image; wherein, the RGAN is trained in the following manner: the network is trained using an unpaired training dataset, and during the training process, a loss function including structural similarity loss and edge phase consistency loss is used to optimize the network, and a dynamic slicing strategy is used for training and inference. Through this application, the signal-to-noise ratio, edge sharpness, and geometric position fidelity of the alignment mark image are improved without the need for paired data, thereby improving alignment accuracy.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of the IC equipment alignment mark image enhancement method provided in the embodiments of this application; Figure 2 A schematic diagram of the overall architecture provided for embodiments of this application; Figure 3 A flowchart for constructing a training dataset is provided as an embodiment of this application; Figure 4 A flowchart for obtaining structural similarity loss is provided for embodiments of this application; Figure 5 A flowchart for obtaining edge phase consistency loss provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the phase consistency principle provided in the embodiments of this application; Figure 7 A flowchart for obtaining a phase consistency map provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of the image enhancement device provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of semiconductor manufacturing technology.

[0021] Research has revealed that the manufacturing precision of integrated circuits is highly dependent on the alignment system of core equipment such as photolithography, which requires precise capture of alignment marks on the wafer for overlay. However, in advanced processes, complex techniques such as chemical mechanical polishing, high aspect ratio etching, and multilayer thin film deposition can severely damage the physical morphology of the marks and introduce complex noise. Furthermore, defocusing blurring of the optical imaging system and electrical noise from the sensor further degrade image quality, resulting in decreased contrast and worsened signal-to-noise ratio in the acquired mark images, directly threatening alignment accuracy.

[0022] Traditional image enhancement methods are mostly based on ideal noise assumptions and rely on manual parameter adjustment, making them difficult to handle real-world complex process noise and lacking in generalization and automation capabilities. While deep learning-based image restoration methods have made progress in recent years, supervised learning requires a large amount of "sharp-degraded" paired data, which is unavailable in actual production lines. Unsupervised generative adversarial networks (GANs), while not requiring paired data, are prone to generating false textures during enhancement, leading to pixel-level geometric positional shifts in marked edges. For nanometer-precision integrated circuit manufacturing, such geometric distortion is an unacceptable fatal flaw, and current technologies lack enhancement methods that can effectively suppress such geometrical shifts within an unsupervised framework.

[0023] Based on this, embodiments of this application provide an image enhancement method and an image enhancement device for IC equipment alignment marks, which can effectively overcome the geometric position drift problem caused by existing unsupervised generative adversarial networks when enhancing the alignment mark images of integrated circuit equipment without the need for paired training data. By introducing an image quality evaluation and distribution mechanism, geometric consistency constraints and dynamic slicing strategy, the method can significantly improve the image signal-to-noise ratio and edge sharpness while strictly maintaining the physical geometric structure of the marks, thereby improving the measurement accuracy of the alignment system and the overlay yield of chip manufacturing.

[0024] To address the limitations of existing technologies in enhancing alignment mark images for IC equipment, such as the failure of traditional algorithms under complex processes and the tendency of unsupervised deep learning methods to cause geometric drift, this method first constructs a non-paired dual-domain dataset containing a "process degradation mark domain" collected from actual production lines and an "ideal standard mark domain" generated through simulation. Based on this, a recurrent generative adversarial network (GAN) with geometric consistency constraints is constructed, and the nonlinear mapping relationship between the two domains is learned through adversarial training. Structural similarity loss and edge phase consistency loss are introduced during training to maintain the macroscopic structural consistency of the image and lock the sub-pixel physical edge positions, respectively, thereby effectively suppressing geometric drift. Specifically, by calculating the structural similarity loss, the non-distortion of brightness, contrast, and structure in the images before and after enhancement is constrained; further, the edge phase consistency loss is introduced, utilizing the physical property that the phase spectrum is insensitive to contrast but highly sensitive to structural features, to construct an edge phase loss function, forcing the generative network to strictly lock the sub-pixel physical edge positions while enhancing texture. Finally, a dynamic slicing strategy is employed for efficient training and inference, ensuring the model's ability to process high-resolution images. During the inference phase, based on the input image size, a full-image input or overlapping sliding window input strategy is adopted to output an enhanced image with high signal-to-noise ratio, sharp edges, and faithful geometric position. This method achieves high signal-to-noise ratio, edge sharpening, and faithful geometric position enhancement of the alignment mark image without requiring pairing data, significantly improving the measurement accuracy of the IC equipment alignment system and the overlay yield of chip manufacturing.

[0025] Please see Figure 1 , Figure 1 A flowchart illustrating the IC equipment alignment mark image enhancement method provided in this application embodiment. Figure 1 As shown in the figure, the IC equipment alignment mark image enhancement method provided in this application embodiment includes: S101. Obtain the process degradation marker image to be enhanced.

[0026] This step inputs images of wafer alignment marks actually acquired from the semiconductor production line. These images, having undergone complex processes such as chemical mechanical polishing, etching, and thin film deposition, typically contain physical damage such as dish-shaped pits, rough edges, and low-frequency interference fringes. In addition, due to the limitations of the depth of focus of the lithography imaging system and vibrations, the images are often accompanied by defocus blur and sensor electrical noise, resulting in a significant reduction in the image signal-to-noise ratio and blurred edge features.

[0027] In practical systems, these images are directly captured by alignment sensors in integrated circuit manufacturing equipment (such as lithography machines) and input into subsequent processing modules in the form of a digital matrix.

[0028] S102. Use a preset image quality evaluation algorithm to perform real-time quality assessment on the process degradation marker image to be enhanced, and determine whether the evaluation result is lower than the preset quality threshold.

[0029] The direct benefit of this step is that it establishes a quality screening mechanism, which provides a realistic and repairable input source for subsequent enhancement models, while avoiding redundant processing of high-quality images, thus ensuring that the method is geared towards real-world industrial problems and is efficient.

[0030] S103. If the evaluation result is lower than the quality threshold, the process degradation label image is input into the trained recurrent generative adversarial network, and the enhanced alignment label image is output.

[0031] This step is the core reasoning process of the method. The trained network acts as a fixed function mapping, receiving degraded images and outputting enhanced images. The network employs a fully convolutional architecture and does not contain fully connected layers, thus supporting image inputs of arbitrary sizes. During the training phase, the network has already learned the mapping relationship from the "process degradation domain" to the "ideal standard domain".

[0032] S104. If the evaluation result is higher than or equal to the quality threshold, no enhancement is performed, and the original image is output.

[0033] Subsequently, a pre-defined image quality assessment algorithm is used to perform real-time quality evaluation of the image to be enhanced, which is a key step in improving production line yield. The system extracts indicators such as contrast-to-noise ratio (CNR), edge energy information, and local information entropy of the image. If the quality score is higher than or equal to the quality threshold (determined to be clear), a bypass jump is performed, and the original image is directly output to the alignment process without going through the neural network; if it is lower than the quality threshold, it is determined to be an image to be enhanced.

[0034] The recurrent generative adversarial network is trained in the following way: The network is trained using an unpaired training dataset. During training, a loss function that includes structural similarity loss and edge phase consistency loss is used to optimize the network. A dynamic slicing strategy is also employed for training and inference.

[0035] In this embodiment of the application, the training of the recurrent generative adversarial network is achieved in the following manner: First, an unpaired training dataset consisting of real process degradation images and simulated ideal images is constructed to provide the source and target domains for model learning.

[0036] Secondly, a recurrent generative adversarial network is designed and trained. This network achieves mutual mapping between two domains through two generators and ensures the authenticity of the generated images through two discriminators.

[0037] Then, loss function optimization is performed. During training, in addition to the standard adversarial loss and cycle consistency loss, the key innovation lies in the introduction of geometric consistency constraints. These constraints are specifically implemented through structural similarity loss and edge phase consistency loss. Structural similarity (SSIM) loss calculates the similarity index of brightness, contrast, and structure between the generated image and the input image within a local window, aiming to maximize this index to maintain the macroscopic topological structure of the label without distortion. Edge phase consistency loss, as the core geometric constraint, uses a set of multi-scale, multi-directional logarithmic Gabor filters to filter both the input and generated images, calculating their phase consistency maps (which eliminate brightness and contrast interference and accurately reflect physical edge positions), and then calculating the difference between the two maps as the loss. This loss constrains the phase alignment between the enhanced image and the input image in the frequency domain, ensuring that physical edges do not drift at the sub-pixel level. By minimizing this loss, the network is forced to lock the physical edge positions of the original image at the sub-pixel level, fundamentally eliminating the edge geometric drift problem common in unsupervised generative models.

[0038] Finally, a dynamic slicing strategy is adopted to adaptively segment high-resolution images during the training and inference phases, so as to balance model learning efficiency, memory constraints and the overall quality of output images.

[0039] The training process described in the above embodiments of this application can produce a highly specialized enhancement model. This model can be trained without paired data, strictly maintaining the key geometric dimensions and positions of alignment marks without nanometer-level drift when enhancing images (improving signal-to-noise ratio, sharpening edges), and can efficiently process industrial-grade high-resolution images. Finally, by inputting the image to be enhanced into this model, geometrically accurate enhancement results can be directly obtained, thus providing reliable input for subsequent high-precision alignment measurements.

[0040] Here, the training dataset includes process degradation labeled domain images and ideal standard labeled domain images.

[0041] For further details, please refer to Figure 2 , Figure 2 This is a schematic diagram of the overall architecture provided for an embodiment of this application. Figure 2 As shown, this application constructs an unpaired dataset containing a "process degradation marker domain" (domain A) and an "ideal standard marker domain" (domain B), and builds a recurrent generative adversarial network based on geometric consistency constraints. During training, degraded images from domain A are input to the forward generator, which maps them to generate enhanced images. These enhanced images are then fed into the discriminator B for authenticity judgment (true / false), and simultaneously used by the reverse generator to simulate degradation, generating reconstructed degraded images that form a recurrent consistency constraint with the original image from domain A. The geometric consistency constraint introduced in the network training consists of two parts: a structural similarity loss function to maintain the macroscopic structure, and a phase consistency loss function to lock sub-pixel level edges. This method can effectively solve the problem of alignment marker image degradation caused by chemical mechanical polishing (CMP), etching, thin film interference, optical defocusing, and sensor noise in advanced semiconductor processes, and overcomes the geometric position drift that is easily generated in traditional unsupervised algorithms, thereby improving the measurement accuracy of IC equipment alignment systems and the overlay yield of chip manufacturing.

[0042] For further details, please refer to Figure 3 , Figure 3 This is a flowchart illustrating the construction of a training dataset as provided in an embodiment of this application. Figure 3 As shown, the training dataset is constructed in the following manner: S201. Collect images of degradation markers affected by the process in the actual production line as the process degradation marker field.

[0043] This step aims to obtain degraded samples from the real world.

[0044] Specifically, the operation involves using a high-resolution image sensor built into integrated circuit manufacturing equipment (such as the alignment subsystem of a lithography machine) on the semiconductor front-end process production line (after chemical mechanical polishing, etching, and thin film deposition) to directly acquire images of preset alignment marks on the wafer surface. The collected data covers wafers from different levels and batches (such as 65nm, 28nm, and 7nm processes) to ensure data diversity and representativeness.

[0045] Subsequently, using the same image quality assessment algorithm as S102, these original images were quantitatively filtered. The system calculated the contrast-to-noise ratio (CNR), edge energy information, and local information entropy of the images, retaining only those images whose indicators were below a certain threshold and had typical damage characteristics. These characteristics included: dish-shaped pits caused by uneven chemical mechanical polishing (manifested as concentric nonlinear decay of gray values ​​in the marked grating area, with the gray value in the central area being significantly lower than that in the edge area); asymmetric edge roughness and contour distortion caused by the anisotropy of the etching process (manifested as rough edges and uneven chamfers of the grating lines, resulting in distortion of the waveform contour); low-frequency background fringes and overall contrast reduction caused by optical interference due to the stacking of multiple thin films (manifested as low-frequency interference fringes and low contrast noise in the background area due to the stacking of silicon oxide and silicon nitride films covering the wafer); high-frequency edge blurring caused by defocusing of the imaging system (manifested as edge gradient diffusion); and random electrical noise introduced by the sensor (manifested as graininess in uniform areas).

[0046] The collected images are saved in their original grayscale matrix format, forming an image library for the process degradation marker domain. This step establishes a data source that is completely consistent with the final application scenario, containing realistic and complex noise and physical deformation, laying the foundation for the model to learn how to cope with real industrial challenges.

[0047] S202. Generate a simulated marked image with ideal edge characteristics based on the imaging simulation model as an ideal standard marked domain.

[0048] Here, professional optical simulation software (such as PROLITH, SOLID-E, etc.) is used to establish an electromagnetic field simulation model of the alignment mark based on rigorous vector imaging theory. In the software, the material stacking structure of the wafer (including substrate material, refractive index and thickness of each thin film) needs to be defined first. Then, the three-dimensional geometric model of the alignment mark (usually a grating structure, and key dimensional parameters such as line width, spacing, and trench depth need to be set) is accurately constructed on the substrate according to the design drawings.

[0049] Next, the optical properties of the actual lithography alignment system are simulated. Key parameters include illumination wavelength (e.g., 193 nm), numerical aperture (e.g., 0.93), illumination mode (e.g., off-axis illumination), and some coherence factors. The spatial image is calculated using the software's built-in high numerical aperture vector imaging algorithm. The light intensity distribution matrix obtained from the simulation is the ideal marker image. This image possesses ideal step edge characteristics and a clean, noise-free background. By fine-tuning parameters such as marker size, film thickness, illumination wavelength, and numerical aperture within a small range, a total of 2000 simulated images of different shapes are generated, forming the "ideal standard marker domain" image library in domain B. This step provides a physically accurate, label-free "perfect" target domain, enabling deep neural networks to learn clear augmented targets and solving the problem of obtaining paired, clear images in practice.

[0050] S203. Normalize and unify the size of the images in the two domains to form an unpaired training data set.

[0051] This step involves standardizing and preprocessing the raw data to facilitate efficient training of the neural network. Specific operations include: Gray-level normalization: For each image in both the process degradation domain and the ideal standard domain, calculate the minimum and maximum values ​​among all pixels. Then, apply a linear transformation formula to map the original intensity value of each pixel to […]. Within the interval [1,1]. This operation eliminates the overall brightness and contrast deviations between different images caused by differences in lighting and sensor response, allowing the network to focus on learning structural information.

[0052] The calculation formula is as follows:

[0053] in, These are the original pixel values. and These represent the minimum and maximum pixel values ​​of the image, respectively. The samples in domain A and domain B do not need to correspond one-to-one; after random shuffling, they form an unpaired dual-domain training set.

[0054] Size standardization: All images (whether high-resolution measured or simulated images) are resized to the same pixel size (e.g., 512×512 pixels) by cropping the center region or scaling and interpolation. This ensures that the size of the neural network input layer is fixed, facilitating batch processing and convolution operations.

[0055] Constructing the unpaired set: After the preprocessing described above, the images from the two domains are stored in two separate sets. During training, the program independently and randomly selects images from the two sets to form training sample pairs. These samples do not need to correspond one-to-one in content (i.e., they are not two images labeled "good" and "bad"). Finally, the image sets from the two domains together constitute the unpaired training data set used to train the recurrent generative adversarial network. The direct benefits of this step are that it completes data preparation, accelerates the convergence process of model training through normalization, enables batch training through size uniformity, and the "unpaired" characteristic fundamentally eliminates the dependence on stringent paired data, greatly improving the practicality and deployability of the method.

[0056] Furthermore, the recurrent generative adversarial network includes a forward generator, a backward generator, a first discriminator, and a second discriminator. The forward generator maps the process degradation labeled domain image to the ideal standard labeled domain image; the backward generator maps the ideal standard labeled domain image to the process degradation labeled domain image; the first discriminator distinguishes the image generated by the forward generator from the real ideal standard labeled domain image; and the second discriminator distinguishes the image generated by the backward generator from the real process degradation labeled domain image.

[0057] Here, the forward generator is the enhancement engine in the network, whose function is to learn and perform the mapping transformation from the process degradation labeling domain to the ideal standard labeling domain. During training and inference, it receives a real labeling image from the production line with various process defects (such as CMP disc pits, uneven etching, etc.) as input and outputs an enhanced image with high signal-to-noise ratio, sharp edges, and a clean background. The goal of this generator is to make the output image visually indistinguishable from the ideal labeling image generated by simulation software.

[0058] The reverse generator, unlike the forward generator, learns the mapping from the ideal standard labeling domain to the process degradation labeling domain. It receives a simulated ideal labeling image and attempts to generate a "pseudo-degradation" image that can simulate the effects of real process damage. This reverse path, together with the forward path, forms a loop, which is crucial to ensuring the network can learn meaningful, content-fidelity mappings even without paired data—the principle of cycle consistency.

[0059] To guide the two generators in effective learning, the network is equipped with corresponding discriminators for adversarial training.

[0060] The first discriminator is specifically designed to determine whether an image belongs to the true ideal standard labeling domain. That is, it distinguishes between the enhanced image generated by the forward generator and the true ideal simulation image. It simultaneously sees the true ideal image generated by the simulation model and the generated enhanced image produced by the forward generator. Its task is to accurately distinguish them, thereby forcing the forward generator to continuously improve, making its generated images increasingly realistic, until they can "fool" the discriminator.

[0061] The second discriminator mirrors this function, specifically designed to determine whether an image belongs to the actual process degradation marker domain. That is, it distinguishes between the degradation image generated by the reverse engineer and the actual measured degradation image. It receives actual degradation images from the production line and generated degradation images from the reverse engineer, and performs a discrimination test to drive the reverse engineer to more accurately simulate real process damage patterns.

[0062] Through this recurrent adversarial architecture that includes bidirectional generation and bidirectional discrimination, the network can automatically learn complex and nonlinear mapping relationships from unpaired dual-domain data, providing a core model foundation for the subsequent introduction of geometric constraints and the realization of high-fidelity enhancement.

[0063] Furthermore, both the forward generator and the reverse generator adopt an architecture based on a fully convolutional architecture, in which the encoder, converter and decoder are connected in sequence to support image input of arbitrary size. A dilated convolution module is introduced in the converter to expand the receptive field. Both the first discriminator and the second discriminator adopt a structure based on local patches for discrimination.

[0064] Here, both the forward and backward generators employ a ResNet-based architecture where the encoder, converter, and decoder are sequentially connected. This design aims to remove microscopic noise (such as particle noise) while effectively eliminating large-scale low-frequency light spots caused by thin-film interference. A dilated convolution module is introduced into the converter, forming a complete image-to-image transformation pipeline.

[0065] The encoder consists of multiple convolutional layers. Each layer uses a specific kernel size (e.g., 3×3 or 5×5), a set stride (typically 2), and downsampling. An instance normalization layer and a ReLU activation function are then applied after the convolution operation. The encoder's input is the original image (a degraded image for the forward generator, and an ideal simulation image for the reverse generator). Its function is to extract abstract features from the image layer by layer while compressing its spatial size. As the number of layers increases, the feature map size decreases while the number of channels increases, thereby capturing features from edges and textures to higher-level semantics. The encoder's parameters (kernel weights, biases) are learned during training through backpropagation from random initialization. The direct effect of this part is to transform the input image into a compact, semantically rich intermediate feature representation, providing a foundation for subsequent depth transformations.

[0066] The converter section is the core of the architecture, consisting of a series of residual modules connected in series. To effectively address the global low-frequency light spots and uneven illumination caused by thin-film interference in integrated circuit manufacturing, and to solve the problem of limited receptive field and difficulty in capturing global illumination changes in traditional convolution, dilated convolution is introduced in some residual modules. Dilated convolution expands the receptive field by inserting spaces (controlled by the dilation rate parameter, for example, set to 2, 4, 8) between elements of the standard convolution kernel without increasing the number of parameters or computational complexity. The converter receives the feature map output from the encoder and performs a nonlinear transformation through these residual blocks with dilated convolutions. The direct effect of this process is that it enables the network to simultaneously perceive the widely distributed contextual information and local details in the image: the large receptive field helps the model identify and suppress low-frequency interference fringes and global intensity changes across the entire image, while the residual connections ensure that micro-texture features (such as grain noise and edge gradients) can also be effectively processed. The converter outputs a deeply processed feature map that has fused global and local information.

[0067] The decoder consists of multiple deconvolutional layers (or transposed convolutional layers), with a structure roughly symmetrical to the encoder. Each deconvolutional layer performs an upsampling operation, progressively restoring the spatial size of the feature map to the size of the original input image while reducing the number of channels. Each layer is followed by instance normalization and a ReLU activation function (the last layer may use a Tanh activation function to constrain pixel values ​​to a specific range). The decoder's parameters are also learned during training. The decoder receives the feature map output from the transformer and upsamples and reconstructs it into an output image of the same size as the input image. The direct effect of this part is to remap the high-level features, after global-local synthesis, back to pixel space, generating a visually coherent and detailed enhanced (or degraded) image.

[0068] Overall, the generator's encoder-converter-decoder architecture, combined with dilated convolution in the converter, achieves end-to-end transformation from input to output. Its direct effect is that the generator can not only remove localized grainy noise and edge roughness in the image, but also effectively suppress large-scale low-frequency artifacts caused by physical processes (such as thin-film interference fringes), thus outputting a high-quality image that possesses sharp edges and clean textures locally, while also exhibiting uniform illumination and a clean background globally.

[0069] On the other hand, both the first and second discriminators adopt a structure based on local map patches for discrimination, which is usually called the PatchGAN (Markov Discriminator) structure.

[0070] In this structure, the discriminator maps the input image to an N×N feature matrix. The network structure is designed so that each element in the output matrix corresponds to a local region in the original input image. The discriminator independently judges each element in the matrix as real or fake, and finally takes the average of the outputs. This design forces the discriminator to focus on the high-frequency texture details of the image rather than the global content consistency of the image, thus forcing the generator to generate images with sharp edges and realistic textures.

[0071] In this structure, the discriminator's input is a complete image (either a real image or an image generated by the generator). The discriminator itself is also a convolutional neural network, composed of several convolutional layers, but its final output is not a single true / false scalar, but a two-dimensional feature matrix (e.g., N×N). Each element in this output matrix corresponds to the discrimination result of a specific local region (i.e., a "patch") in the input image. The receptive field of each patch covers a corresponding local region of the input image, and the discriminator independently determines whether each patch comes from a real image distribution or a generated image.

[0072] In its implementation, the discriminator's convolutional layers use convolutions with small strides (such as 1 or 2) to progressively reduce spatial resolution and increase the number of channels. The final convolutional layer uses an appropriately sized kernel and stride to ensure that each position in the output matrix corresponds to a sufficiently large local region of the input image. Instance normalization and LeakyReLU activation functions are typically applied after each convolutional layer. The discriminator's parameters are also learned during training.

[0073] This local patch-based discrimination method shifts the discriminator's focus from evaluating the overall consistency of the image (e.g., the reasonableness of the overall content) to emphasizing high-frequency texture details. This forces the generator to produce sufficiently realistic textures in every local region of its output, particularly clear, sharp, and physically accurate edges, rather than merely achieving global outline accuracy or relying on blurring to evade discrimination. Therefore, under the pressure of adversarial training, the generator learns to produce images highly similar to the true ideal marker (or the true degraded marker) in local details, ensuring that the enhanced image not only has correct macroscopic structure but also meets the expected high-quality standards in microscopic texture.

[0074] For further details, please refer to Figure 4 , Figure 4 A flowchart illustrating the structural similarity loss provided in this application embodiment. For example... Figure 4 As shown in the figure, the structural similarity loss is obtained in the following way: S301. Within a preset local window, calculate the pixel mean, pixel standard deviation, and covariance between the network input image and the network generated image, respectively.

[0075] This step involves extracting local statistical features from the two images. Specifically, a fixed-size local window (e.g., 11x11 pixels) is defined, and the entire input image and the corresponding generated image are traversed using a sliding window approach. For each window location, the following calculations are performed: Calculate the pixel mean of all pixels in the input image within this window, denoted as μx. This value characterizes the average brightness level of this local region.

[0076] Calculate the pixel mean of all pixels in the generated image at the same window location, denoted as μy. Calculate the standard deviation of the input image pixels within that window, denoted as σx. The standard deviation reflects the dispersion of pixel values ​​relative to the mean, characterizing the contrast or texture richness of the region.

[0077] Calculate the standard deviation of the generated image pixels within this window, denoted as σy. Calculate the covariance of the input and generated images within this window, denoted as σxy. Covariance measures the joint trend of change of two variables (here, the pixel values ​​of the two images within the same window), and its value reflects the degree of consistency between them in spatial structure; a positive value indicates that the trends are the same and the structures are similar.

[0078] This application transforms the visual comparison of two images from the original pixel-level differences into a quantitative statistical analysis of three key perceptual attributes of the local region: brightness (represented by the mean), contrast (represented by the standard deviation), and structure (represented by the covariance), thus converting pixel differences into a comparison of quantifiable perceptual attributes.

[0079] S302. Based on the pixel mean, pixel standard deviation, and covariance, the structural similarity index, which measures the similarity between two images in three dimensions of brightness, contrast, and structure, is calculated.

[0080] This step uses the statistics obtained in step 301 to calculate a comprehensive similarity score. The calculation formula is the Structural Similarity Index (SSIM), whose basic form is as follows:

[0081] in, and Representing images respectively and The average pixel value within a local window represents brightness information; and These represent the standard deviation of the image and indicate contrast information. The covariance of two images represents structural information; and It is a very small constant used to prevent calculation instability when the denominator is zero.

[0082] By comparison and The local average gray level of the enhanced image is forced to maintain a linear relationship with the original image; by comparison and The difference in normalized signal strength; by calculating the covariance The SSIM index measures the spatial structural similarity between two images. By maximizing the SSIM index, the similarity of the images before and after enhancement can be constrained in three dimensions: brightness, contrast, and structure, thereby preventing macroscopic shape distortion of the alignment markers during the enhancement process.

[0083] This step outputs a value between [-1, 1] (usually closer to 1 indicates high similarity). This value is not a simple pixel-by-pixel comparison of differences, but rather a comprehensive evaluation from the perspective of the human visual system, assessing whether the overall brightness of the enhanced image and the original input image are consistent in each local region, whether the strength of texture contrast matches, and, most importantly, whether the spatial structure (such as the relative position and shape of edges and lines) has been distorted.

[0084] S303. Construct structural similarity loss based on structural similarity index.

[0085] This step transforms the similarity index into a loss term that can be used for backpropagation optimization in neural networks. A common implementation involves calculating the average structural similarity index (SSIM) and then constructing the loss function. Specifically, this involves averaging the SSIM values ​​calculated for all local windows in the image to obtain the mean structural similarity index (MeanSSIM) for the entire image. Then, the structural similarity loss is applied. Defined as:

[0086] Here, x is the input image, and y is the generated image. The network optimization is driven during training by minimizing this loss function (i.e., maximizing MeanSSIM).

[0087] This step provides a clear optimization objective for training the recurrent generative adversarial network (GAN)—maximizing the structural similarity between the enhanced image and the original input image. During backpropagation, the gradient generated by this loss guides the generator to adjust its parameters so that the generated image is strictly faithful to the original input in terms of macroscopic topology. Specifically, it forces the network to maintain the overall shape, critical dimensions (CD), and contours of the alignment markers without any visible distortion or deformation during noise removal and contrast enhancement, effectively preventing macroscopic geometric distortion introduced by the enhancement process.

[0088] For further details, please refer to Figure 5 , Figure 5 A flowchart illustrating the edge phase consistency loss provided in an embodiment of this application. Figure 5 As shown, the edge phase consistency loss is obtained in the following way: S401. Using a multi-scale, multi-directional logarithmic Gabor filter bank, frequency domain filtering is performed on the network input image and the network generated image respectively to obtain the filter response at each scale and direction.

[0089] A set of logarithmic Gabor filters is pre-designed and constructed. Each filter is defined by three key parameters: center frequency (determining the perceived texture coarseness), bandwidth (determining the frequency coverage), and orientation (e.g., 0°, 45°, 90°, 135°). By combining multiple different center frequencies (e.g., 4 scales) and multiple orientations (e.g., 6 orientations), a filter bank covering a wide spectrum is constructed. A significant feature of this filter bank is that its DC component is zero, which automatically filters out overall illumination unevenness in the image. The input image (process-degraded image) and the generated image (enhanced image) are then convolved with this filter bank (or multiplicative filtering is performed in the frequency domain). For each pixel, at each scale 's' and each orientation 'o', the filter outputs a pair of orthogonal responses: an even-symmetric real response es,o(x) (similar to a cosine wave) and an odd-symmetric imaginary response os,o(x) (similar to a sine wave). The direct effect of this step is to decompose the original image into frequency components of multiple scales and directions. At the physical edges of the image, the responses of odd-symmetric filters at all scales reach extreme values ​​and are phase-aligned, providing a separate frequency domain signal that is insensitive to changes in illumination for subsequent precise edge localization.

[0090] To fundamentally suppress edge geometric drift, a common problem in unsupervised image enhancement, this application introduces edge geometric constraints based on the principle of phase consistency. This principle states that the generation of physical structural features (especially edges) in an image originates from the high phase consistency of the peaks (i.e., phases) of the multi-scale, multi-directional frequency components of the signal at that spatial location. Phase consistency results in coherent enhancement when the component signals are superimposed, leading to an extremum of local energy at that location; this extremum is the precise location of the physical edge. Importantly, the phase consistency metric depends only on the phase alignment and is insensitive to changes in the overall brightness and contrast of the image, thus providing a stable and pure representation of the geometric structure.

[0091] For further details, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the phase consistency principle provided in an embodiment of this application. Figure 6 As shown in the diagram, this schematic simulates a one-dimensional ideal step edge signal, which is synthesized by superimposing four different frequency components (components 1 to 4, with frequencies increasing from low to high). At the center of the edge (near spatial position 0), the response amplitudes of all frequency components undergo synchronous and directional abrupt changes (transitioning from negative to positive values), indicating that their peaks are highly phase aligned at this location. This alignment results in the superimposed signal amplitude (i.e., local energy) reaching its maximum value, precisely corresponding to the location of the physical edge. Conversely, in the smooth region far from the edge, the amplitudes of each component are weak, change gradually, and have disordered phases, with the superimposed local energy approaching zero.

[0092] Therefore, the edge phase consistency loss constructed based on the phase consistency principle can force the neural network to strictly maintain the consistency of the generated image and the input image in the frequency domain phase structure during image enhancement. By forcing the network to lock the sub-pixel-level physical edge positions defined by phase alignment that are insensitive to changes in brightness / contrast, this method can effectively prevent geometric drift introduced by texture "illusion" or improper enhancement, thereby meeting the stringent requirements of geometric fidelity for nanometer-level alignment in IC equipment.

[0093] S402. Based on the filter response, calculate the local energy of each pixel and the sum of the amplitudes at all scales.

[0094] This step extracts key physical quantities that characterize "phase consistency" from the filter response. Specifically, for each pixel 'x': First, the image is convolved using a pair of orthogonal filters (with even-symmetric cosine waves in the real part and odd-symmetric sine waves in the imaginary part) contained in the Log-Gabor filter in the spatial domain, to obtain the image for each pixel. The response components at various scales and in all directions. The real part of the response is... The imaginary part response is Calculate the amplitude at each scale and direction. :

[0095] in, The filter response amplitude at position x, at the nth scale and in the direction represents the signal strength at that position at that frequency and in that direction. The square of the real part of the response. The square of the imaginary response and the square root of the sum of the two are the total amplitude of that frequency component.

[0096] The response components at each scale are treated as vectors and summed to calculate the local energy after superposition. First, the real part of the response at all scales in the same direction. and the virtual part Summing them separately:

[0097] in, For local energy, For the real part of the response in the same direction at all scales, This is the imaginary part of the response for all scales in the same direction.

[0098] The local energy reaches its maximum value at the point where the peaks (i.e., phases) of all frequency components are aligned, which is a sign of the physical edge.

[0099] Calculate the sum of amplitudes across all scales: , as a normalization factor.

[0100] The above steps yield two key indicators: local energy, which directly quantifies the degree of phase alignment of each frequency component at that pixel and is the core evidence of edge existence; and the sum of amplitudes, which represents the total signal strength at that point. By comparing these two quantities, the possibility of misjudging areas with simply high contrast (high amplitude) but disordered phase (such as noise) as edges can be ruled out.

[0101] S403. Based on the sum of local energy and amplitude, calculate the phase consistency maps corresponding to the network input image and the network generated image respectively. The high value region in the phase consistency map represents the physical edge position of the image.

[0102] This step synthesizes local energy and amplitude information into a clear edge map. The phase coherence value is then calculated by combining the sum of local energy and amplitude. The calculation formula is as follows:

[0103] in, The sum of amplitudes across all scales is represented by T, a preset noise threshold typically estimated based on noise statistics (such as Rayleigh distribution) of the image background region. Subtracting T effectively suppresses small, spurious local energy responses in smooth regions caused by random noise, improving the robustness of the image.

[0104] It is a very small positive number that prevents division by zero, ensuring the stability of the formula.

[0105] max( ,0) Ensure the result is non-negative. The calculated PCo(x) is a value between 0 and 1. The maximum value or weighted combination of the results for each direction is taken to obtain the phase consistency value PC(x) for each pixel. The PC(x) of all pixels constitute a phase consistency map. The direct effect of this step is to generate a binary trend map with "black background and white lines". The bright pixels (close to 1) in the map precisely correspond to the real, physical geometric edge positions in the input or generated image, completely eliminating differences in brightness and contrast, as well as noise in uniform regions, from the original image. This map is a purified representation of the "skeleton" of the image's geometry.

[0106] S404. Calculate the difference norm between the two phase consistency maps, and construct the edge phase consistency loss based on the difference norm.

[0107] This step quantifies the differences in geometric structure between the two images. Specifically, after obtaining the phase consistency map (Mapin) of the input image and the phase consistency map (Mapout) of the generated image, the L2 norm (Euclidean distance) between them is calculated as a measure of difference.

[0108] in, Phase consistency map of the input image Phase consistency map of the generated image The difference norm between them The phase consistency map of the input image. To generate a phase consistency map of the image.

[0109] This difference value serves as the edge phase consistency loss. During training, this loss value is minimized using the backpropagation algorithm to force the generator to maintain strict alignment between the input and generated images at their physical edge locations.

[0110] This step provides the generator with an extremely stringent, pixel-level geometric constraint objective. Minimizing this loss means forcing the enhanced image output by the generator to have all its physical edges precisely aligned with the physical edges of the original input image at the sub-pixel level. Since the phase consistency map is insensitive to changes in brightness and contrast, but only to structure, this loss does not penalize the network for denoising or contrast enhancement; it only penalizes even sub-pixel shifts in edge positions caused by texture "illusions" or improper processing. This fundamentally eliminates the possibility of geometric drift of alignment marker edges during enhancement, ensuring the absolute positional fidelity required for nanometer-level alignment metrology.

[0111] Furthermore, the dynamic slicing strategy is implemented through the following steps: During the training phase, the grayscale centroid of the full-resolution image is first calculated to roughly locate the marker center. Then, a region of interest (ROI) with a coverage area larger than the marker size (e.g., the marker occupies 900 pixels, and the ROI is defined as 1200 pixels) is delineated using the grayscale centroid as the origin. Local slices of fixed size (e.g., 256x256) are randomly cropped from this ROI, and random rotation and / or flipping enhancement operations are applied to the local slices, which are then used as network input.

[0112] This ensures that every slice entering the network contains effective labeled features, avoiding the network wasting training computation in featureless background areas (empty areas), significantly accelerating the model's convergence speed and improving the accuracy of feature representation.

[0113] A dynamic tiling (Patch) strategy is employed, randomly cropping fixed-size local slices (e.g., 256×256 pixels) from the original high-resolution training images and applying random rotation and / or flipping enhancement operations to these local slices as network input. This strategy forces the network to focus on local micro-texture and edge gradient restoration, while also expanding the sample size.

[0114] Here, during the training phase, the dynamic slicing strategy is implemented as follows: Local slices of a preset fixed size are cropped from the original high-resolution training image at random coordinates and used as network input. This fixed size is determined based on the computing device's memory capacity, the number of network model parameters, and the minimum region required for effective feature extraction. After cropping, random geometric transformations are applied to the obtained local slices, including rotations of 90, 180, or 270 degrees with specific probabilities, as well as horizontal or vertical flipping operations. The relevant transformation parameters are dynamically generated by a random number generator.

[0115] The direct effects of this strategy during the training phase are reflected in three aspects: First, it enables the network to be effectively trained on high-resolution images with limited hardware resources, solving the memory bottleneck problem. Second, since each input sample is only a small portion of the complete label, this strategy forces the generator and discriminator to focus on learning to repair microscopic granular noise and accurately reconstruct local features such as edge gradients. Finally, by sampling the same original image multiple times at different locations and orientations, the training dataset is physically expanded.

[0116] During the inference phase, a full-image input or overlapping sliding window strategy is adopted. Based on the fully convolutional architecture of the recurrent generative adversarial network (RGAN), the RGAN supports image input of arbitrary size. If the size of the process degradation marker image to be enhanced is larger than a preset threshold, an overlapping sliding window is used to divide it into multiple sub-image blocks for distributed inference. The overlapping areas of the processed sub-image blocks are then weighted and fused to eliminate stitching seams and output a complete enhanced image.

[0117] In the inference phase, the dynamic slicing strategy is implemented as follows: The system first acquires the process degradation marker image to be enhanced and detects its size, comparing it with a preset memory limit threshold. If the image size does not exceed the threshold, it is directly input into the network in full-image mode for enhancement. If the image size exceeds the threshold, an overlapping sliding window processing procedure is activated: a fixed size consistent with that in the training phase is used as the sliding window size, and the sliding window step size is set to be smaller than the slice size (e.g., generating a 50% overlap area). The large image is divided into multiple sub-image blocks according to spatial order; each sub-block is sequentially input into the trained network for processing; finally, all processed sub-image blocks are recombined, and the pixel values ​​of the overlapping areas are calculated using a weighted fusion method (such as linear gradient weighting or cosine weighting) to synthesize a complete output image. This strategy eliminates the block artifacts caused by direct stitching, ultimately outputting a complete enhanced image with high signal-to-noise ratio, sharp edges, and faithful geometric position.

[0118] The direct effects of this strategy during the inference phase are reflected in three aspects: First, it enables the trained model to overcome the limitations of input size during training and process arbitrarily high-resolution real-world production images, greatly expanding the model's applicability. Second, the overlapping sliding window design ensures that each pixel in the image is processed multiple times by the network along with sufficient contextual information, effectively avoiding the degradation of edge enhancement quality caused by boundary effects. Finally, the sophisticated weighted fusion algorithm smooths out the output differences between different sub-blocks at the seams, completely eliminating visible block effects or seam traces that may be introduced by direct stitching, ultimately outputting a visually coherent, high-resolution enhanced image with uniform overall quality, which can directly serve the subsequent alignment and measurement process.

[0119] In this embodiment of the application, the loss function is called the total loss function, which is constructed by weighting and summing the adversarial loss, the cycle consistency loss, the structural similarity loss, and the edge phase consistency loss.

[0120] The total loss function is constructed by combining multiple sub-loss terms and is the ultimate objective function that drives the optimization of the entire recurrent generative adversarial network training.

[0121] The overall loss function is constructed as follows: four key sub-loss terms—adversarial loss, cycle consistency loss, structural similarity loss, and edge phase consistency loss—are linearly weighted and summed according to preset weight coefficients. The weight coefficient of each sub-loss term is a hyperparameter greater than zero, manually set or determined through a small number of validation experiments before training begins, based on the magnitude of each loss term and its relative importance in the overall optimization objective. The adversarial loss is dynamically calculated by the generator and discriminator in a minimax game to ensure the realism of the generated image; the cycle consistency loss is calculated by comparing the difference between the original image and the reconstructed image after a complete cycle mapping to ensure the reversibility and consistency of image content during domain transformation; the structural similarity loss and edge phase consistency loss are calculated according to the aforementioned steps, together constituting a geometric consistency constraint.

[0122] By summing the weighted losses from each of the above items, we obtain the total loss function used for network parameter optimization. :

[0123] in, This is the least-squares adversarial loss function, used to ensure the realism of the generated images; This is the cycle consistency loss function, used to ensure the reversibility of content; The structural similarity loss function; This is the edge phase consistency loss function. These are the weighting coefficients for each loss function. These are the weighting coefficients of the least squares adversarial loss function. These are the weighting coefficients of the cycle consistency loss function. These are the weighting coefficients of the structural similarity loss function. The weighting coefficients of the edge phase consistency loss function are adjusted by... While removing complex process noise (dominated by adversarial loss), geometric fidelity (dominated by phase loss) was significantly enhanced, ensuring the measurement accuracy of the enhanced image in the IC equipment alignment system.

[0124] By adjusting the weighting coefficients of each loss term, the denoising intensity, visual realism, and geometric fidelity of the image can be balanced. This design aims to guide the network to effectively remove complex process noise while strictly ensuring the fidelity of geometric features such as the critical dimensions (CD) and center position of the alignment marks during the enhancement process, thereby directly improving the measurement accuracy of the subsequent alignment system.

[0125] During the backpropagation phase of training, this construction process optimizes the network's performance across all key dimensions simultaneously and in a coordinated manner by minimizing this single total loss value. Specifically, it forces the generator to output realistic images sufficient to fool the discriminator (dominated by adversarial loss) while maintaining content consistency (guaranteed by cycle consistency loss), while strictly locking the image's macroscopic structure (constrained by structural similarity loss) and sub-pixel-level edge locations (constrained by edge phase consistency loss). By appropriately configuring the weights, this total loss function balances image denoising intensity, visual realism, and geometric fidelity, thereby guiding the network to ultimately learn an optimal mapping that improves image quality while avoiding the introduction of harmful geometric drift.

[0126] For further details, please refer to Figure 7 , Figure 7 This is a flowchart illustrating the process of obtaining a phase consistency map, as provided in an embodiment of this application. Figure 7 As shown, the phase consistency map is obtained in the following way: S501. Subtract a preset noise threshold from the local energy and take the positive value to obtain the effective local energy.

[0127] This step denoises and thresholds the local energy. Specifically, it obtains the local energy of each pixel in each direction from previous calculations. Simultaneously, a preset noise threshold T is introduced. This threshold is not arbitrarily set but estimated based on the statistical characteristics of noise in the image background or smooth regions. A common method is to assume that the noise follows a Rayleigh distribution, analyze the pixel value fluctuations in flat regions without obvious structure in the image, statistically calculate the standard deviation of the noise level or the expected energy value, and use this as the threshold T. Subsequently, the local energy of each pixel is calculated as follows: Eo′(x)=max(Eo(x) T,0) Here, Eo′(x) represents the effective local energy at position x in direction o. It is the result of subtracting the noise floor from the original local energy while ensuring non-negativity, representing the signal energy at that location that significantly exceeds the noise level due to real physical edges; Eo(x) represents the original local energy at position x in direction o; T represents a preset noise threshold. This threshold is typically estimated based on the noise statistics of the image background or smooth regions (e.g., assuming the noise follows a Rayleigh distribution). Its function is to distinguish between the strong energy response generated by real edges and the weak energy fluctuations generated by random noise.

[0128] max(Eo(x) (T, 0) This is an operation that takes a positive value. Its purpose is to ensure that the result of the subtraction is non-negative. If Eo(x) If the result of T is negative, it means that the local energy at that point is lower than the noise level and is not an effective edge signal. In this case, it should be set to 0.

[0129] The estimated random noise floor is subtracted from the original local energy. This effectively suppresses spurious, weak phase consistency responses caused by random noise fluctuations in smooth regions, allowing subsequent calculations to focus only on strong energy signals that significantly exceed the noise level and are truly caused by physical edges. This greatly improves the signal-to-noise ratio of the phase consistency map and the robustness of edge detection.

[0130] S502. Divide the effective local energy by the sum of the amplitudes at all scales and a constant to prevent division by zero to obtain the phase consistency value of each pixel.

[0131] This step obtains the final phase consistency metric value through normalization calculation. Specifically, it involves taking the effective local energy obtained in step S501. As the numerator, the denominator consists of two parts: one is the sum of the amplitudes of the pixel across all scales and directions. This value represents the total signal strength at that location; secondly, it introduces a very small positive constant to prevent division by zero. (e.g., 1×10) 8 The calculation formula is:

[0132] in, Eo′(x) represents the phase coherence value at position x in direction o. It is a scalar between 0 and 1. A value close to 1 indicates that the phases of all frequency components are highly aligned at that position and in that direction, corresponding to a physical edge. A value close to 0 indicates that the phases of the frequency components are disordered, corresponding to smooth regions or noise. Eo′(x) represents the effective local energy at position x in direction o. The sum of amplitudes across all scales. It is a very small constant to prevent the denominator from being 0.

[0133] S503. A phase consistency map is formed based on the phase consistency values ​​of each pixel.

[0134] This step integrates the calculation results of all pixels to generate a complete feature image. Specifically, for each pixel, the maximum phase consistency value PC(x) = maxo(PCo(x)) across all directions is typically taken as the final value for that pixel, resulting in a two-dimensional matrix with the same spatial size as the original input image. Each element PC(x) in this matrix ranges from 0 to 1, where values ​​close to 1 indicate that the location is a physically aligned edge, while values ​​close to 0 correspond to smooth regions or noise. This two-dimensional matrix is ​​the phase consistency map. The direct effect of this step is to generate a clear "structural skeleton" map that removes brightness and contrast interference. This map, in grayscale or pseudo-color, accurately and intuitively marks the sub-pixel positions of all physical edges in the input image. It filters out uneven illumination, background variations, and random noise from the original image, retaining only the most essential geometric structural information, providing a clean and reliable benchmark for subsequent calculations of edge phase consistency loss.

[0135] Compared with traditional image processing methods and existing unsupervised deep learning methods, the IC equipment alignment mark image enhancement method and image enhancement device provided in this application overcome the problem of geometric consistency constraints composed of structural similarity loss and edge phase consistency loss in the unsupervised recurrent generative adversarial network, and combine them with a dynamic slicing strategy. This solves the technical problem of difficulty in simultaneously improving image quality and maintaining geometric fidelity without the need for paired training data. It achieves the effect of strictly maintaining the physical structure and sub-pixel positional accuracy of the alignment mark while enhancing the image signal-to-noise ratio and edge sharpness, thereby effectively improving the measurement accuracy and overlay yield of the integrated circuit equipment alignment system.

[0136] The advantages of this application compared to existing technologies are: 1) This application is based on an unpaired dual-domain training strategy, which automatically learns the mapping relationship between the "ideal standard labeling domain" generated by IC equipment simulation and the "process degradation labeling domain" collected from the production line through a recurrent generative adversarial network. This method does not require obtaining "clear-degraded" image pairs at the same time and location, effectively solving the data acquisition problem of deep learning models in industrial applications and significantly reducing the model deployment cost.

[0137] 2) To address the issue of edge position drift caused by spurious textures generated in general unsupervised generative networks, this application innovatively introduces geometric consistency constraints. On one hand, structural similarity (SSIM) loss constraints are used to enhance the non-distortion properties of the images before and after the image in terms of brightness, contrast, and structure. On the other hand, leveraging the physical property that the phase spectrum is insensitive to contrast changes but highly sensitive to structural features, an edge phase consistency loss is constructed to force the network to strictly lock the sub-pixel-level physical edge position of the alignment mark while removing complex process noise. Geometric consistency constraints ensure the geometric fidelity of the enhanced image, meeting the stringent requirements of nanometer-level precision for IC equipment alignment.

[0138] 3) To address the issues of memory overflow and insufficient local texture learning caused by the extremely high resolution of wafer images, this application employs dynamic random slicing during the training phase, forcing the network to focus on microscopic edge gradients and noise textures; during the inference phase, it combines full-image or overlapping sliding window inputs with the dilated convolution module introduced in the network architecture, effectively expanding the receptive field and thus effectively eliminating global low-frequency light spots caused by thin-film interference.

[0139] This application constructs a dual-domain dataset, solving the data acquisition challenge for deep learning models in industrial applications and significantly reducing model deployment costs. It innovatively introduces set consistency constraints into unsupervised learning, ensuring the geometric fidelity of enhanced images and meeting the stringent nanometer-level precision requirements of IC equipment alignment. Dynamic random slicing is employed during the training phase, addressing the issues of memory overflow and insufficient local texture learning under high-resolution input.

[0140] Based on the same inventive concept, this application also provides an image enhancement device corresponding to the IC equipment alignment mark image enhancement method. Since the principle of the device in this application is similar to the IC equipment alignment mark image enhancement method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0141] Please see Figure 8 Figure 8 This is a schematic diagram of the image enhancement device provided in an embodiment of this application. Figure 8 As shown, the image enhancement device 800 includes: The process degradation marker image acquisition module 801 is used to acquire the process degradation marker image to be enhanced; The image output module 802 is used to perform real-time quality assessment on the process degradation marker image to be enhanced using a preset image quality evaluation algorithm, and to determine whether the assessment result is lower than a preset quality threshold. If the assessment result is lower than the quality threshold, the process degradation marker image is input into a trained recurrent generative adversarial network (RGAN) and the enhanced alignment marker image is output. If the assessment result is higher than or equal to the quality threshold, no enhancement is performed and the original image is output. The RGAN is trained in the following way: the network is trained using an unpaired training dataset, and during the training process, a loss function including structural similarity loss and edge phase consistency loss is used to optimize the network, and a dynamic slicing strategy is used for training and inference.

[0142] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0145] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0146] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for enhancing alignment mark images in IC equipment, characterized in that, include: Obtain the image of the process degradation markers to be enhanced; A preset image quality evaluation algorithm is used to perform real-time quality assessment on the process degradation marker image to be enhanced, and to determine whether the evaluation result is lower than a preset quality threshold. If the evaluation result is lower than the quality threshold, the process degradation label image is input into the trained recurrent generative adversarial network, and the enhanced alignment label image is output. If the evaluation result is higher than or equal to the quality threshold, no enhancement is performed, and the original image is output. The recurrent generative adversarial network is trained in the following way: The network is trained using an unpaired training dataset. During training, a loss function that includes structural similarity loss and edge phase consistency loss is used to optimize the network. A dynamic slicing strategy is also employed for training and inference.

2. The method according to claim 1, characterized in that, The training dataset includes process degradation labeled domain images and ideal standard labeled domain images. The training dataset is constructed in the following manner: Multi-source degraded images from the actual production line are collected, including physical morphological damage, optical interference texture, system imaging blur, and sensor electrical noise. The image quality evaluation algorithm is used to quantify the degradation features of the original labeled images to be screened, and images whose quantification indicators meet the degradation distribution characteristics are selected as process degradation labeling domain images. The ideal standard labeling domain is generated based on the imaging simulation model, which produces a simulated labeled image with ideal edge characteristics. The images from the two domains are normalized and their sizes are standardized to form an unpaired training data set.

3. The method according to claim 1 or 2, characterized in that, The recurrent generative adversarial network includes a forward generator, a backward generator, a first discriminator, and a second discriminator. The forward generator is used to map the process degradation marker domain image to the ideal standard marker domain image; The reverse generator is used to map the ideal standard label domain image to the process degradation label domain image; The first discriminator is used to distinguish between the image generated by the forward generator and the real ideal standard labeled domain image; The second discriminator is used to distinguish the image generated by the reverse generator from the actual process degradation marker domain image.

4. The method according to claim 3, characterized in that, Both the forward generator and the reverse generator adopt an architecture based on a fully convolutional architecture, in which the encoder, converter and decoder are connected in sequence, and the converter introduces a dilated convolution module. Both the first discriminator and the second discriminator adopt a structure based on local map tiles for discrimination.

5. The method according to claim 1, characterized in that, The structural similarity loss is obtained in the following way: Within a preset local window, calculate the pixel mean, pixel standard deviation, and covariance between the network input image and the network generated image, respectively. Based on the pixel mean, pixel standard deviation, and covariance, a structural similarity index is calculated to measure the similarity between two images in three dimensions: brightness, contrast, and structure. The structural similarity loss is constructed based on the structural similarity index.

6. The method according to claim 1, characterized in that, The edge phase consistency loss is obtained in the following way: Using a multi-scale, multi-directional logarithmic Gabor filter bank, frequency domain filtering is performed on the network input image and the network generated image respectively to obtain the filter response at each scale and direction; Based on the filter response, the local energy of each pixel and the sum of the amplitudes at all scales are calculated respectively. Based on the sum of the local energy and amplitude, the phase consistency maps corresponding to the network input image and the network generated image are calculated respectively, and the high value regions in the phase consistency map represent the physical edge positions of the image; Calculate the difference norm between the two phase consistency maps, and construct the edge phase consistency loss based on the difference norm.

7. The method according to claim 1, characterized in that, The dynamic slicing strategy is implemented through the following steps: During the training phase, the gray-level centroid of the full-resolution training image is calculated to determine the marker center, and a region of interest of a preset size is delineated based on the marker center. Local slices of a fixed size are randomly cropped from the region of interest, and random rotation and / or flipping enhancement operations are applied to the local slices as network input. During the inference phase, based on the fully convolutional architecture of the recurrent generative adversarial network, the recurrent generative adversarial network supports image input of arbitrary size; If the size of the process degradation marker image to be enhanced is greater than a preset threshold, it is divided into multiple sub-image blocks using an overlapping sliding window method for distributed inference. The overlapping areas of each sub-image block are then weighted and fused to eliminate seams and output a complete enhanced image.

8. The method according to claim 1, characterized in that, The loss function is the total loss function, which is constructed in the following way: The total loss function is obtained by weighted summing of the adversarial loss, cycle consistency loss, structural similarity loss, and edge phase consistency loss.

9. The method according to claim 6, characterized in that, The phase consistency map was obtained using the following method: The effective local energy is obtained by subtracting a preset noise threshold from the local energy and then taking the positive value. The effective local energy is divided by the sum of the amplitudes at all scales and the sum of a constant to prevent division by zero to obtain the phase consistency value of each pixel. The phase consistency map is formed based on the phase consistency values ​​of each pixel.

10. An image enhancement device, characterized in that, include: The process degradation marker image acquisition module is used to acquire process degradation marker images to be enhanced; The image output module is used to perform real-time quality assessment on the process degradation marker image to be enhanced using a preset image quality evaluation algorithm, and to determine whether the assessment result is lower than the preset quality threshold. If the evaluation result is lower than the quality threshold, the degraded process-marked image is input into the trained recurrent generative adversarial network, and the enhanced alignment-marked image is output; if the evaluation result is higher than or equal to the quality threshold, no enhancement is performed, and the original image is output. The recurrent generative adversarial network is trained in the following way: The network is trained using an unpaired training dataset. During training, a loss function that includes structural similarity loss and edge phase consistency loss is used to optimize the network. A dynamic slicing strategy is also employed for training and inference.

Images