Image processing method and device, medium and computer program product

Abstract

The invention discloses an image processing method and device, a medium and a computer program product, and relates to the technical field of semiconductor manufacturing. The image processing method comprises the following steps: acquiring target image samples of a plurality of wafer samples through a scanning electron microscope, and obtaining a first contour image label of each target image sample; based on the target image sample, a first contour image result is generated through a generator of a preset image edge model, iteration training is carried out on the preset image edge model through pixel difference loss, adversarial loss and edge supervision loss until a convergence condition is met, and a target image edge model is obtained; the generator of the target image edge model can output an edge contour image based on the target image of the target wafer. Through multi-dimensional constraints of pixel difference loss, edge supervision loss and adversarial loss, the preset image edge model is trained in the direction of generating an actual edge contour, and the accuracy of the edge contour image is improved.

Description

Technical Field

[0001] This application belongs to the field of semiconductor manufacturing technology, and in particular relates to an image processing method, device, medium and computer program product. Background Technology

[0002] In semiconductor manufacturing, precise control of multilayer lithography processes is crucial for ensuring chip performance and yield. This requires acquiring SEM images containing microstructural information about the multilayer lithographic patterns on silicon wafers using a high-voltage scanning electron microscope (HVSEM). Based on the edge contours of the target structure in the SEM images, key processes such as aligning the design layout with the SEM image and measuring overlay (OVL) errors can be controlled. Therefore, extracting complete and accurate edge contours from SEM images is a critical step in ensuring the reliability of subsequent processes.

[0003] Currently, edge contour extraction from SEM images is generally achieved through unsupervised style transfer methods using Cycle-Consistent Adversarial Networks (CycleGANs) to transfer SEM and layout images. This method constructs two generators and two discriminators. The first generator maps the SEM image to a design layout style, while the second generator reconstructs the layout image back to the SEM style. The generators and discriminators are jointly trained to generate a layout style image from the SEM image using the first generator, thus extracting the edge contours. However, this method lacks direct geometric constraints between the input and generated images, leading to inconsistencies between the output layout image and the input SEM structure, resulting in poor accuracy in determining the edge contour image. Summary of the Invention

[0004] This application provides an image processing method, apparatus, medium, and computer program product that can improve the accuracy of edge contour images.

[0005] A first aspect of this application provides an image processing method, including:

[0006] Using a scanning electron microscope, target image samples of multiple wafer samples were acquired, and a first contour image label was obtained for each target image sample;

[0007] Based on the target image sample, a first contour image result is generated by a generator of a preset image edge model, and the pixel difference loss is determined based on the difference between the first contour image result and the first contour image label.

[0008] The discriminator of the preset image edge model is used to determine the authenticity of the first contour image result and the first contour image label, and the adversarial loss is determined based on the discrimination result.

[0009] The edge calculation module of the preset image edge model performs edge detection on the first contour image label and the first contour image result to obtain the second contour image label and the second contour image result. The edge supervision loss is determined based on the difference between the second contour image label and the second contour image result.

[0010] The preset image edge model is iteratively trained based on the pixel difference loss, the edge supervision loss, and the adversarial loss until the preset convergence condition is met, thereby obtaining the target image edge model.

[0011] A second aspect of this application provides an image processing apparatus, comprising:

[0012] The acquisition module is used to acquire target image samples of multiple wafer samples using a scanning electron microscope, and to acquire a first contour image label for each target image sample;

[0013] The pixel difference loss module is used to generate a first contour image result based on the target image sample through a generator of a preset image edge model, and to determine the pixel difference loss based on the difference between the first contour image result and the first contour image label.

[0014] The adversarial loss module is used to judge the authenticity of the first contour image result and the first contour image label by using the discriminator of the preset image edge model, and determine the adversarial loss based on the judgment result;

[0015] The edge supervision loss module is used to perform edge detection on the first contour image label and the first contour image result respectively through the edge calculation module of the preset image edge model to obtain the second contour image label and the second contour image result, and determine the edge supervision loss based on the difference between the second contour image label and the second contour image result.

[0016] The parameter update module is used to iteratively train the preset image edge model based on the pixel difference loss, the edge supervision loss, and the adversarial loss until the preset convergence condition is met, so as to obtain the target image edge model.

[0017] A third aspect of the embodiments of this application provides an electronic device, the device comprising: a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, they implement an image processing method as provided in any of the embodiments of this application described above.

[0018] A fourth aspect of the embodiments of this application provides a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement an image processing method as provided in any of the embodiments of this application described above.

[0019] A fifth aspect of the embodiments of this application provides a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform an image processing method as provided in any of the embodiments of this application described above.

[0020] The technical solution provided in this application has at least the following beneficial effects:

[0021] In an image processing method provided in this application embodiment, firstly, multiple target image samples of wafer samples are acquired using a scanning electron microscope, and a first contour image label is obtained for each target image sample. Secondly, based on the target image samples, a first contour image result is generated using a generator of a preset image edge model. Then, based on an edge detection module, edge detection is performed on the first contour image label and the first contour image result to generate a second contour image label and a second contour image result. Finally, based on the first contour image label, the first contour image result, the second contour image label, and the second contour image result, pixel difference loss, adversarial loss, and edge supervision loss are determined. The preset image edge model is then iteratively trained using these losses until convergence is met, resulting in a target image edge model. The generator of this target image edge model can output edge contour images based on the target image of the target wafer. In other words, through multi-dimensional constraints of pixel difference loss, edge supervision loss, and adversarial loss, the preset image edge model is trained to generate edge contours that more closely resemble actual edges, thereby improving the accuracy of the edge contour images. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of an image processing method provided in one embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the structure of a preset image edge model provided in one embodiment of this application;

[0025] Figure 3 This is a schematic flowchart of an image processing method provided in one embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this application;

[0027] Figure 5 This is a schematic diagram of an image processing device provided in one embodiment of this application. Detailed Implementation

[0028] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0029] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0030] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0031] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0032] Multilayer lithography refers to the process in semiconductor chip manufacturing where multiple layers of photoresist, exposure, and development are repeated on a silicon wafer to form multilayer stacked micro-circuit patterns.

[0033] Overlay error (OVL) refers to the spatial deviation (including translation, rotation, scaling, etc.) between the lithographic pattern of the next layer and the pattern formed in the previous layer in a multilayer lithography process.

[0034] High-voltage scanning electron microscopy (HVSEM) refers to the process of scanning the surface of a silicon wafer by emitting a high-voltage electron beam. The interaction between the electron beam and the silicon material generates signals such as secondary electrons and backscattered electrons, and the signals are received by a detector to obtain image data.

[0035] Cycle-Consistent Adversarial Network (CycleGAN) is an unsupervised deep learning model based on the idea of ​​adversarial learning. Its core structure includes two generators and two discriminators, which are used to achieve two image style transfers without paired data (such as converting SEM images to design layout styles and then converting the layout style back to SEM styles).

[0036] The Edge Calculation Module is a module used for edge detection of images.

[0037] In semiconductor manufacturing, precise control of multilayer lithography processes is crucial for ensuring chip performance and yield. This requires acquiring SEM images containing microstructural information about the multilayer lithographic patterns on silicon wafers using a high-voltage scanning electron microscope (HVSEM). Based on the edge contours of the target structure in the SEM images, key processes such as aligning the design layout with the SEM image and measuring overlay (OVL) errors can be controlled. Therefore, extracting complete and accurate edge contours from SEM images is a critical step in ensuring the reliability of subsequent processes.

[0038] Currently, edge contour extraction from SEM images is generally achieved through unsupervised style transfer methods using Cycle-Consistent Adversarial Networks (CycleGANs) to transfer SEM and layout images. This method constructs two generators and two discriminators. The first generator maps the SEM image to a design layout style, while the second generator reconstructs the layout image back to the SEM style. The generators and discriminators are jointly trained to generate a layout style image from the SEM image using the first generator, thus extracting the edge contours. However, this method lacks direct geometric constraints between the input and generated images, leading to inconsistencies between the output layout image and the input SEM structure, resulting in poor accuracy in determining the edge contour image.

[0039] To address the aforementioned technical problems, this application provides an image processing method and a computer program product. In the image processing method provided in this application embodiment, firstly, multiple target image samples of wafer samples are acquired using a scanning electron microscope, and a first contour image label is obtained for each target image sample. Secondly, based on the target image samples, a first contour image result is generated using a generator of a preset image edge model, and a pixel difference loss is determined based on the difference between the first contour image result and the first contour image label. The authenticity of the first contour image result and the first contour image label is judged by a discriminator of the preset image edge model, and an adversarial loss is determined based on the judgment result. Edge detection is performed on the first contour image label and the first contour image result using an edge calculation model of the preset image edge model, respectively, to obtain a second contour image label and a second contour image result, and an edge supervision loss is determined based on the difference between the second contour image label and the second contour image result. Finally, based on the pixel difference loss, the edge supervision loss, and the adversarial loss, the preset image edge model is iteratively trained until a preset convergence condition is met, resulting in a target image edge model. The generator of this target image edge model is used to process the target image of the target wafer to generate an edge contour image.

[0040] For example, the image processing method provided in this application can be applied to the production line of a semiconductor manufacturing company for quality inspection and precision control of key processes such as wafer multilayer lithography. In practical applications, a target image of the target wafer surface is acquired using a scanning electron microscope, and this target image is input into a trained target image edge model. The edge contour image is then generated by the generator of this target image edge model and stored in a data storage device. Operators can perform various mathematical analyses or quality assessments based on this edge contour image, such as measuring graphic overlay errors, locating and classifying defect positions, and comparing the consistency between the design layout and the actual contour.

[0041] Taking pattern overlay error measurement as an example, pattern overlay error refers to the spatial positional deviation between different lithographic layers calculated through edge contour images to ensure precise alignment of multilayer circuit patterns. First, operators can retrieve edge contour images of different lithographic layers from a data storage device (all generated by a target image edge model generator after acquiring the target image using a scanning electron microscope), and align these images using an image registration algorithm. Second, based on the aligned edge contour images, the physical coordinates of key alignment points such as rectangular corners are extracted, and the pattern overlay error is calculated. Finally, based on the pattern overlay error, it is determined whether the alignment accuracy between different lithographic layers meets the requirements. Therefore, the image processing method of this application can be used to measure pattern overlay error with high accuracy, no need for manual measurement, and high efficiency, ensuring the accuracy of multilayer lithography processes and chip yield.

[0042] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. The image processing method provided by the embodiments of this application can be applied to various application scenarios that require extracting the edge contours of structures from images and performing analysis or process control based on the edge contours.

[0043] The following describes an image processing method provided by an embodiment of this application. In practical applications, the executing entity of the image processing method of this application embodiment can be a terminal device, such as a desktop computer, a laptop computer, or a remote device similar to a server. Of course, the executing entity of this application embodiment can also be a software entity, such as a client or software program installed on a terminal device. The specific type of executing entity corresponding to the technical solution provided by this application embodiment is not strictly limited here, and can be flexibly selected according to the actual application scenario and actual needs.

[0044] The following describes specific embodiments of an image processing method, apparatus, electronic device, storage medium, and computer program product provided in this application. First, an image processing method is described.

[0045] Figure 1 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 1 As shown, the method includes steps S100 to S104.

[0046] S100: Using a scanning electron microscope, acquire target image samples of multiple wafer samples and obtain a first contour image label for each target image sample.

[0047] In one or more embodiments of this application, in order to provide training data for a preset image edge model in subsequent steps, and to ensure that the preset image edge model can have a mapping relationship from a wafer SEM image to an edge contour image through contrastive learning input to labels, this application needs to obtain training data for the preset image edge model in this step.

[0048] Specifically, this application obtains the first contour image label containing the edge contour corresponding to the target image sample.

[0049] It should be noted that this application does not limit the acquisition conditions of the target image samples, which can be set according to actual needs. For example, the target image samples can cover multiple process dimensions (different photolithography layers, different process nodes), the imaging parameters (accelerating voltage, scanning resolution, etc.) of the scanning electron microscope are consistent when acquiring each target image sample, and the target image samples can be a part of the wafer (edge ​​region, center region), etc. For example, multiple wafer samples may include conventional photolithography layers (photoresist layer, metal interconnect layer, dielectric etching layer, etc.) and conventional process nodes (such as 7nm, 14nm, etc.) to ensure sample diversity. At the same time, this application does not limit the specific method of obtaining the first contour image label. For example, the first contour image label can be obtained by extracting the edge contour of the structure by applying image filtering and thresholding algorithms to each target image sample; or for each target image sample, the design layout of the wafer sample to which the target image sample belongs can be called, and the design layout can be aligned with the target image sample, and then the first contour image label of the target image sample can be obtained based on the aligned design layout.

[0050] S101: Based on the target image sample, generate a first contour image result using a generator with a preset image edge model, and determine the pixel difference loss based on the difference between the first contour image result and the first contour image label.

[0051] In one or more embodiments of this application, in order to provide parameter optimization direction for the preset image edge model in subsequent steps, the model is transformed from "random output" to "contour generation close to the label" iteration. In this step, this application can generate a first contour image result through a generator and determine the pixel difference loss between the first contour image result and the first contour image label, providing an iterative direction at the pixel level for model training.

[0052] It should be noted that this application does not limit the generator structure of the preset image edge model, which can be set according to actual needs. For example, the preset image edge model can adopt a pix2pix architecture, and the generator can adopt an encoder and decoder architecture (U-Net 256 structure, ResNet structure). The encoder is used to extract features from the input target image sample, and the decoder is used to map the extracted features back to the contour image (first contour image result) corresponding to the target image sample. Figure 2 The figure shows a schematic diagram of the structure of a preset image edge model provided in an embodiment of this application. In this figure, the preset image edge model 200 includes a generator 201, a discriminator 202, and an edge calculation module 203.

[0053] Pixel difference loss refers to the degree of deviation between the first contour image result generated by the quantization generator and the first contour image label in the global pixel distribution. This application does not limit the method for determining pixel difference loss; it can be set according to actual needs, such as mean squared error, mean absolute error, structural similarity index, etc. In one or more embodiments of this application, this application can iterate through the pixel values ​​of each pixel position in the first contour image result and the first contour image label, and determine the absolute difference between the corresponding pixel values ​​of each pixel position in the first contour image result and the first contour image label. Then, based on each absolute difference, the pixel difference loss is determined by global averaging. The formula for calculating the pixel difference loss is as follows:

[0054] d i,j =|G(A) i,j -B i,j | (1)

[0055]

[0056] In equations (1) and (2), H*W represents the pixel dimensions of the first contour image result and the first contour image label; (i, j) represents the pixel position; G(A) i,j B represents the pixel value at the pixel location in the first contour image result generated by the generator; i,j This represents the pixel value at the pixel location in the first contour image label; d i,j L is the absolute difference between the pixel values ​​at the corresponding pixel locations. L1 Pixel difference loss is used to measure the average absolute error of the entire image at the pixel level, to ensure that the iterative direction of subsequent model training is that the first contour image result generated by the generator is close to the first contour image label in overall structure, that is, to ensure the closeness in overall structure.

[0057] S102: The discriminator of the preset image edge model is used to determine the authenticity of the first contour image result and the first contour image label, and the adversarial loss is determined based on the discrimination result.

[0058] In one or more embodiments of this application, in order to provide parameter optimization direction for the preset image edge model in subsequent steps, the model is transformed from "random output" to "contour generation close to the label" iteration. In this step, this application can use a discriminator to distinguish the authenticity of the first contour image result and the first contour image label, so as to ensure the structural rationality of the generated first contour image result through adversarial learning between the generator and the discriminator.

[0059] Specifically, this application inputs the first contour image result and the first contour image label into a discriminator of a preset image edge model. The discriminator then determines the authenticity of the first contour image result and the first contour image label, obtaining a discrimination result. Based on this discrimination result, an adversarial loss is determined. The discrimination result is the probability that the first contour image result and the first contour image label are both real labels (e.g., a value between 0 and 1, where 1 indicates real and 0 indicates generated).

[0060] It should be noted that this application does not limit the discriminator structure of the preset image edge model. It can be set according to actual needs, such as the PatchGAN architecture, which discriminates local regions of the input first contour image result and the first contour image label to output the probability of authenticity for each local region; or a fully convolutional network result, which discriminates the entire input first contour image result and the first contour image label. In other words, during training, the discriminator aims to maximize the distinguishing ability between the first contour image result and the first contour image label, while the generator aims to generate a first contour image result that better fits the first contour image label, i.e., minimizing the discriminator's distinguishing ability. Regardless of the discriminator architecture used, the design goal of the adversarial loss determined in this step is to allow the preset image edge model to learn the structural rules (such as edge topology and morphological features) of the first contour image label through an adversarial game between the generator and the discriminator, ensuring that the generated first contour image result not only closely resembles the first contour image label at the pixel level but also in its structural rules.

[0061] This application does not limit the method of determining adversarial loss, which can be set according to actual needs, such as binary cross-entropy loss. This quantifies the discriminator's deviation between the probability of the input image being true and the true label / result (1 for the first contour image label, 0 for the first contour image result), thereby improving the discriminator's distinguishing ability and allowing the generator to optimize the output to better fit the label. In one or more embodiments of this application, a mapping relationship discrimination between "input image - output contour" can be constructed, strengthening the generator's learning of the correlation between target image samples and contour images. Furthermore, by identifying whether the target image sample and the contour image are a true match, the generator can learn the mapping logic from a specific target image sample to the corresponding true contour, as follows:

[0062] First, this application uses the target image sample and the first contour image label as a label mapping sample pair (i.e., a matching pair of real input and real output), and determines the label discrimination error of the label mapping sample pair (i.e., the deviation between the probability of the discriminator judging the match as "real association" and the ideal value of 1) through a discriminator of a preset image edge model; and uses the target image sample and the first contour image result as a result mapping sample pair (i.e., a matching pair of real input and generated output), and determines the result discrimination error of the result mapping sample pair (i.e., the deviation between the probability of the discriminator judging the match as "real association" and the ideal value of 0) through a discriminator of a preset image edge model. Second, based on the label discrimination error and the result discrimination error, the discriminator loss is determined, and based on the result discrimination error, the generator loss is determined. Finally, based on the discriminator loss and the generator loss, the adversarial loss is determined.

[0063] The formulas for calculating the discriminator loss and generator loss are as follows:

[0064] L D =-[E A,B logD(A,B)+E A log(1-D(A,G(A)))] (3)

[0065] L GAN =-E A logD(A,G(A)) (4)

[0066] In equations (3) and (4), L D For discriminator loss; L GAN is the generator loss; D(*) is the discriminator function, with the input being sample pairs; G(*) is the generator function, with the input being the target image sample A; B is the first contour image label; G(A) is the first contour image result; (A,B) is the label-mapped sample pair; (A,G(A)) is the result-mapped sample pair.

[0067] In this embodiment, the adversarial loss can not only constrain the generator in the preset image edge model to generate edge contours that are close to the first contour image label, but also enable the generator to learn the correlation between the input target image sample and the first contour image label, thereby improving the accuracy and scene adaptability of the subsequent model in generating edge contours.

[0068] S103: The edge calculation module of the preset image edge model performs edge detection on the first contour image label and the first contour image result respectively to obtain the second contour image label and the second contour image result, and determines the edge supervision loss based on the difference between the second contour image label and the second contour image result.

[0069] In one or more embodiments of this application, in order to improve the learning of edge features by the preset image edge model in subsequent steps, the deviation at the edge level is focused to improve the generator's ability to accurately reproduce the details of the edge contours of the structure in the target image.

[0070] First, the first contour image label is input into the edge calculation module of a preset image edge model. This module performs edge detection on the first contour image label to obtain the second contour image label. Second, the first contour image result is input into the edge calculation module, which performs edge detection on the result to obtain the second contour image result. Finally, based on the difference between the second contour image label and the second contour image result, the edge supervision loss is determined.

[0071] It should be noted that this application does not limit the edge detection method of the edge computing module (compute_edge), and can be set according to actual needs, such as the first-order differential operator (Sobel operator), the second-order differential operator (Laplacian operator), etc. In one or more embodiments of this application, firstly, based on a preset first edge detection operator, edge detection is performed on the first contour image label and the first contour image result respectively, and by capturing regions with smooth grayscale, a third contour image label and a third contour image result are obtained. Secondly, according to a preset second edge detection operator, edge detection is performed on the first contour image label and the first contour image result respectively, and by capturing regions with abrupt grayscale changes, a fourth contour image label and a fourth contour image result are obtained. Finally, according to a preset edge fusion weight, the third contour image label and the fourth contour image label are fused to obtain a second contour image label; and, according to the edge fusion weight, the third contour image result and the fourth contour image result are fused to obtain a second contour image result. The first edge detection operator can be a Sobel operator to capture smooth boundaries; the second edge detection operator can be a Laplacian operator to capture sharp details. Edge fusion weights are used to fuse images obtained through different edge detection operators. The magnitude of these weights can be set according to actual needs. For example, if the focus is on smooth curves, the weight of the first edge detection operator is increased; if the focus is on sharp corners, the weight of the second edge detection operator is increased. Based on the fusion weights, the formula for calculating the fused image is as follows:

[0072]

[0073] LaplacianEdge = |Laplacian convolution stack| (6)

[0074] Edge=α*SobelEdge+(1-α)*LaplacianEdge (7)

[0075] In equations (5), (6), and (7), the Sobel operator is a first-order derivative edge detection operator. By calculating the gradient changes of pixel gray levels in the horizontal and vertical directions respectively, it locates regions with abrupt changes in gray levels in the image, focusing on the detection of smooth edges and fine-value features; grad x This is the horizontal gradient plot; grad yThe first image shows the vertical gradient map; SobelEdge is the result of edge detection based on the Sobel operator. The Laplacian operator is a second-derivative edge detection operator that does not require directional calculations and directly reflects the rate of change of pixel grayscale through the second derivative, focusing on the detection of sharp corners, subtle protrusions / depressions, and other features in the image; LaplacianEdge is the result of edge detection based on the Laplacian operator. α is an adjustable fusion weight, ranging from 0 to 1; Edge is the result of image fusion based on the fusion weight. By fusing two different edge detection operators, the limitations of edge detection by a single operator can be avoided, and the adjustable fusion weight allows for controllability of smooth edge detection or detailed edge enhancement.

[0076] Of course, this application does not limit the method of determining the edge supervision loss, and it can be set according to actual needs, such as based on the mean square error of edge pixels, the mean absolute error, etc. In one or more embodiments of this application, this application determines the binary cross-entropy loss and the Dessian loss based on the second contour image label and the second contour image result, and then determines the edge supervision loss by weighted summation based on the binary cross-entropy loss and the Dessian loss.

[0077] The formula for calculating the edge supervision loss is determined by weighted summation of the binary cross-entropy loss and the Dess loss, as follows:

[0078] EdgeLoss=β*BCE(fake B real B )+(1-β)DICE(fake B real B (8)

[0079] In equation (8), EdgeLoss is the edge supervision loss; fake B The result is the second contour image; real B The second contour image label; β is an adjustable weight; BCE (fake) B real B ) represents the binary cross-entropy loss; DICE (fake B real B ) represents the DICE loss. When β approaches 1, the edge supervision loss is dominated by BCE, which is suitable for scenarios with a low proportion of edge pixels and class imbalance to ensure that edge pixels are not missed. When β approaches 0, the edge supervision loss is dominated by DICE, which is suitable for scenarios with high requirements for edge continuity to ensure the structural integrity of the edges. When β is 0.5, the edge supervision loss takes into account both classification accuracy and structural integrity.

[0080] In this embodiment, the binary cross-entropy loss quantifies the deviation between the second contour image result and the second contour image label in terms of pixel-level category (foreground or background). It is used to prevent the model from missing edge pixels when there is a category imbalance between edge pixels and non-edge pixels in the second contour image result / edge, thus ensuring the basic classification accuracy of edge pixels. The Dess loss quantifies the deviation between the second contour image result and the second contour image label in terms of spatial overlap in the edge region. By calculating the intersection-union ratio (IUU) of the edge regions of the second contour image result and the second contour image label, it directly constrains the spatial consistency between the model-generated edges and the label edges.

[0081] S104: Based on the pixel difference loss, the edge supervision loss, and the adversarial loss, the preset image edge model is iteratively trained until the preset convergence condition is met, thereby obtaining the target image edge model.

[0082] Specifically, this application can iteratively train a preset image edge model based on the pixel difference loss, edge supervision loss, and adversarial loss determined in the aforementioned steps to construct multi-dimensional loss feedback, thereby guiding the preset image edge model to improve its ability to "generate edge contours from SEM images" until a preset convergence condition is met, thus obtaining the target image edge model. This application acquires a target image of a target wafer and performs image processing on the target image using the generator of the target image edge model to generate an edge contour image.

[0083] It should be noted that the iterative training of the preset image edge model needs to be carried out alternately in two stages: discriminator optimization and generator optimization. In one or more embodiments of this application, firstly, a portion of the target image samples is selected to calculate the adversarial loss, and based on the obtained adversarial loss, the discriminator is optimized through the backpropagation algorithm to improve its discrimination ability. Secondly, using the same portion of target image samples as in the previous step, pixel difference loss, edge supervision loss, and adversarial loss are calculated, and the total loss is determined by weighted summation (the weights can be adjusted according to actual needs). Then, based on this total loss, the generator is optimized through the backpropagation algorithm to improve its generation ability. Finally, the process of "discriminator optimization and generator optimization" is repeated. After each round of iterative training, some additional target image samples and first contour image labels can be used to verify the model performance until the preset convergence condition is met. Of course, the specific content of the preset convergence condition is not limited in this application, and can be set according to actual needs, such as loss stabilization, performance reaching expectations, and the number of iterations reaching a threshold. By using an alternating update strategy, we can avoid situations where the discriminator fails to converge due to frequent changes in the generator parameters, and where the generator is unable to generate a reasonable structure due to insufficient discriminator capabilities, thus ensuring that the generator achieves performance optimization under the constraints of the discriminator.

[0084] Furthermore, to further improve the training efficiency and noise resistance of the preset image edge model, and to ensure that the model can stably extract high-precision edges in SEM images with different noise levels, the training of the generator and discriminator is optimized. In one or more embodiments of this application, before generating the first contour image result in step S101, the target image samples can be divided based on the noise level to obtain low-noise image samples and high-noise image samples. This divides the model training process from steps S101 to S104 into two stages. This application uses low-noise image samples to perform the first stage of iterative training on the preset image edge model, and freezes the generator's general edge feature extraction layer after the first stage of iterative training. Then, high-noise image samples are used to perform the second stage of iterative training on the frozen preset image edge model. The method of measuring the noise level is not limited in this application; it can be set according to actual needs, such as grayscale standard deviation, signal-to-noise ratio, etc., or the distinction between low and high noise can be achieved through manual observation of the image.

[0085] In this embodiment, the target image samples are first divided according to noise level, and then the model is trained in two stages. Compared with the process of training with mixed noise samples, this avoids the situation where the model overfits to low-noise samples and has insufficient generalization ability to high-noise samples. The first stage of iterative training allows the generator's general edge feature extraction layer (such as the first N convolutional layers of the encoder) to learn the edge feature rules under noise-free interference, ensuring that the generator masters the basic edge extraction logic. After the first stage of iterative training is completed, the generator's general edge feature extraction layer is frozen, and only the decoder and part of the generator can be updated. Thus, in the second stage of iterative training, the preset image edge model learns the ability to separate edge contours under noise interference based on the existing general edge feature extraction ability, thereby achieving accurate edge contour extraction of low-noise image samples and stable adaptation of edge contour extraction of high-noise image samples.

[0086] In the aforementioned image processing method, firstly, target image samples from multiple wafer samples are acquired using a scanning electron microscope, and a precise first contour image label is matched to each target image sample. This provides direct geometric constraints from the input image to the edge contour for model training, ensuring structural consistency between the generator output and the input SEM image. Secondly, based on a pre-defined image edge model, the generator generates the first contour image result from the target image samples. The deviation is then quantified using three types of losses: pixel difference loss (measuring the global pixel deviation between the result and the label), adversarial loss (distinguishing the authenticity of the result from the label using a discriminator, constraining the structural rationality of the result), and edge supervision loss (further edge detection is performed on the result and label, focusing on edge detail deviations). Finally, the three types of losses are fused to iteratively train the model until convergence conditions are met, resulting in a directly applicable target image edge model. The generator can output edge contour images for the target wafer's target image. In other words, through multi-dimensional constraints of pixel difference loss, edge supervision loss, and adversarial loss, the pre-defined image edge model is trained to generate edge contours that more closely resemble actual edges, improving the accuracy of the edge contour images.

[0087] Furthermore, this application introduces an edge detection module based on the pix2pix architecture, integrating the Sobel operator, Laplacian operator, and fusion weight strategy. This dynamically enhances the emphasis on edge contours in SEM images; that is, adjustable fusion weights balance smooth edges and sharp corner details, improving the clarity and coherence of generated contours and reducing edge blurring and breakage. A combination of BCE and DICE edge supervision loss is used, with dual constraints ensuring the generation model maintains consistent edge direction, reducing breakage and artifacts, and ensuring the generation of structurally complete edge contours even in high-noise environments. Staged training is implemented for SEM data with different noise levels: the generation model is first trained on low-noise image samples, then the first N layers of the generator are frozen, with only subsequent layers and the discriminator fine-tuned to adapt to high-noise image samples. This retains the general edge feature extraction capability learned in the low-noise stage while enhancing the stable adaptability to high-noise image samples through later layer optimization, ensuring the integrity of extracted edge contours under high-noise conditions.

[0088] like Figure 3 The diagram shown is a flowchart illustrating the process of training a preset image edge model according to an embodiment of this application. Figure 3 Steps S300 to S305 are involved. For details of each step, please refer to the above content. The process of training the preset image edge model is as follows:

[0089] S300: Using a scanning electron microscope, acquire target image samples of multiple wafer samples, and for each target image sample, obtain the first contour image label containing the edge contour corresponding to that target image sample.

[0090] S301: Based on the noise level, the target image samples are divided to obtain low-noise image samples and high-noise image samples.

[0091] S302: By using a preset image edge model, determine the pixel difference loss, edge supervision loss, and adversarial loss corresponding to low-noise image samples.

[0092] S303: Based on the pixel difference loss, edge supervision loss and adversarial loss corresponding to low-noise image samples, the preset image edge model is trained in the first stage of iterative training, and the general edge feature extraction layer of the generator is frozen after the first stage of iterative training.

[0093] S304: By using a preset image edge model, determine the pixel difference loss, edge supervision loss, and adversarial loss corresponding to high-noise image samples;

[0094] S305: Based on the pixel difference loss, edge supervision loss and adversarial loss corresponding to the high-noise image samples, perform the second stage of iterative training on the frozen preset image edge model until the preset convergence condition is met, and obtain the target image edge model.

[0095] Based on the aforementioned image processing method, this application also provides a specific embodiment of an image processing apparatus.

[0096] like Figure 4 As shown, an image processing device 400 provided in this application embodiment includes an acquisition module 401, a pixel difference loss module 402, an adversarial loss module 403, an edge supervision loss module 404, and a parameter update module 405.

[0097] The acquisition module 401 is used to acquire target image samples of multiple wafer samples using a scanning electron microscope, and to acquire a first contour image label for each target image sample;

[0098] The pixel difference loss module 402 is used to generate a first contour image result based on the target image sample by a generator of a preset image edge model, and to determine the pixel difference loss based on the difference between the first contour image result and the first contour image label.

[0099] The adversarial loss module 403 is used to judge the authenticity of the first contour image result and the first contour image label through the discriminator of the preset image edge model, and determine the adversarial loss based on the judgment result;

[0100] The edge supervision loss module 404 is used to perform edge detection on the first contour image label and the first contour image result respectively through the edge calculation module of the preset image edge model to obtain the second contour image label and the second contour image result, and determine the edge supervision loss based on the difference between the second contour image label and the second contour image result.

[0101] The parameter update module 405 is used to iteratively train the preset image edge model according to the pixel difference loss, the edge supervision loss and the adversarial loss until the preset convergence condition is met, so as to obtain the target image edge model.

[0102] In some embodiments, the above-described apparatus further includes an acquisition module 401, specifically configured to: divide the target image samples according to noise levels to obtain low-noise image samples and high-noise image samples, wherein the low-noise image samples are used to perform a first-stage iterative training on the preset image edge model, and the high-noise image samples are used to freeze the general edge feature extraction layer of the generator after the first-stage iterative training, and to perform a second-stage iterative training on the frozen preset image edge model based on the high-noise image samples.

[0103] In some embodiments, the above-described apparatus further includes a parameter update module 405, specifically configured to: employ an alternating update strategy to iteratively train the generator and the discriminator of the preset image edge model, wherein each round of the iterative training includes: fixing the generator and updating the parameters of the discriminator according to the adversarial loss; fixing the discriminator and updating the generator according to the pixel difference loss, the edge supervision loss, and the adversarial loss.

[0104] In some embodiments, the above-described apparatus further includes an edge supervision loss module 404, specifically configured to: perform edge detection on the first contour image label and the first contour image result according to a preset first edge detection operator, and obtain a third contour image label and a third contour image result by capturing regions with smooth grayscale; perform edge detection on the first contour image label and the first contour image result according to a preset second edge detection operator, and obtain a fourth contour image label and a fourth contour image result by capturing regions with abrupt grayscale changes; fuse the third contour image label and the fourth contour image label according to a preset edge fusion weight to obtain a second contour image label; and fuse the third contour image result and the fourth contour image result according to a preset edge fusion weight to obtain the second contour image result.

[0105] In some embodiments, the above-described apparatus further includes a pixel difference loss module 402, specifically configured to: traverse the pixel values ​​of each pixel position in the first contour image result and the first contour image label, and determine the absolute difference between the pixel values ​​corresponding to each pixel position; and determine the pixel difference loss by global averaging based on each absolute difference.

[0106] In some embodiments, the above-described apparatus further includes an adversarial loss module 403, specifically configured to: take the target image sample and the first contour image label as a label mapping sample pair, and determine the label discrimination error of the label mapping sample pair using the discriminator of the preset image edge model; take the target image sample and the first contour image result as a result mapping sample pair, and determine the result discrimination error of the result mapping sample pair using the discriminator of the preset image edge model; determine the discriminator loss based on the label discrimination error and the result discrimination error, and determine the generator loss based on the result discrimination error; and determine the adversarial loss based on the discriminator loss and the generator loss.

[0107] In some embodiments, the above apparatus further includes an edge supervision loss module 404, specifically configured to: determine a binary cross-entropy loss and a Dessian loss based on the second contour image label and the second contour image result; and determine an edge supervision loss by weighted summation based on the binary cross-entropy loss and the Dessian loss.

[0108] According to an image processing method, this application also provides a specific embodiment of an image processing apparatus.

[0109] Figure 5 A schematic diagram of the hardware structure of an image processing device provided in an embodiment of this application is shown.

[0110] The image processing device may include a processor 501 and a memory 502 storing computer program instructions.

[0111] Specifically, the processor 501 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0112] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0113] The processor 301 implements any of the image processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0114] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 5 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0115] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0116] Bus 510 includes hardware, software, or both, that couples the components of the electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0117] Furthermore, in conjunction with the image processing methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image processing methods in the above embodiments.

[0118] In addition, in conjunction with the image processing method in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs an image processing method as provided in any aspect of the above embodiments of this application.

[0119] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0120] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0121] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0122] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0123] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Using a scanning electron microscope, target image samples of multiple wafer samples were acquired, and a first contour image label was obtained for each target image sample; Based on the target image sample, a first contour image result is generated by a generator of a preset image edge model, and the pixel difference loss is determined based on the difference between the first contour image result and the first contour image label. The discriminator of the preset image edge model is used to determine the authenticity of the first contour image result and the first contour image label, and the adversarial loss is determined based on the discrimination result. The edge calculation module of the preset image edge model performs edge detection on the first contour image label and the first contour image result to obtain the second contour image label and the second contour image result. The edge supervision loss is determined based on the difference between the second contour image label and the second contour image result. The preset image edge model is iteratively trained based on the pixel difference loss, the edge supervision loss, and the adversarial loss until the preset convergence condition is met, thereby obtaining the target image edge model.

2. The method as described in claim 1, characterized in that, Before generating the first contour image result based on the target image sample using a generator with a preset image edge model, the method further includes: The target image samples are divided according to the noise level to obtain low-noise image samples and high-noise image samples. The low-noise image samples are used to perform a first-stage iterative training on the preset image edge model. The high-noise image samples are used to freeze the general edge feature extraction layer of the generator after the first-stage iterative training, and to perform a second-stage iterative training on the frozen preset image edge model based on the high-noise image samples.

3. The method as described in claim 1, characterized in that, Iterative training of the preset image edge model includes: An alternating update strategy is used to iteratively train the generator and discriminator of the preset image edge model. Each round of iterative training includes: The generator is fixed, and the parameters of the discriminator are updated according to the adversarial loss; The discriminator is fixed, and the generator is updated based on the pixel difference loss, the edge supervision loss, and the adversarial loss.

4. The method as described in claim 1, characterized in that, Edge detection is performed on the first contour image label and the first contour image result respectively to obtain the second contour image label and the second contour image result, including: According to the preset first edge detection operator, edge detection is performed on the first contour image label and the first contour image result respectively. By capturing the gray-scale smooth area, the third contour image label and the third contour image result are obtained. According to the preset second edge detection operator, edge detection is performed on the first contour image label and the first contour image result respectively. By capturing the region of gray-scale abrupt change, the fourth contour image label and the fourth contour image result are obtained. According to the preset edge fusion weight, the third contour image label and the fourth contour image label are fused to obtain the second contour image label; The third contour image result and the fourth contour image result are fused according to the preset edge fusion weights to obtain the second contour image result.

5. The method as described in claim 1, characterized in that, Based on the difference between the first contour image result and the first contour image label, the pixel difference loss is determined, including: Traverse the first contour image result and the pixel value of each pixel position in the first contour image label, and determine the absolute difference between the pixel value corresponding to each pixel position; The pixel difference loss is determined by global averaging based on the absolute differences described above.

6. The method as described in claim 1, characterized in that, The discriminator of the preset image edge model is used to determine the authenticity of the first contour image result and the first contour image label, and based on the determination result, the adversarial loss is determined, including: The target image sample and the first contour image label are used as a label mapping sample pair, and the label discrimination error of the label mapping sample pair is determined by the discriminator of the preset image edge model. The target image sample and the first contour image result are used as a result mapping sample pair, and the result discrimination error of the result mapping sample pair is determined by the discriminator of the preset image edge model. Based on the label discrimination error and the result discrimination error, the discriminator loss is determined, and based on the result discrimination error, the generator loss is determined. The adversarial loss is determined based on the discriminator loss and the generator loss.

7. The method as described in claim 1, characterized in that, Based on the difference between the second contour image label and the second contour image result, the edge supervision loss is determined, including: Based on the second contour image label and the second contour image result, determine the binary cross-entropy loss and the Dess loss; The edge supervision loss is determined by weighted summation based on the binary cross-entropy loss and the Descein loss.

8. An electronic device, characterized in that, The electronic device includes a memory and a program or instructions stored in the memory and executable on a processor, wherein when the program or instructions are executed by the processor, they implement an image processing method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement an image processing method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs an image processing method as described in any one of claims 1-7.

Images