Target image segmentation method and device, and lithographic apparatus, model training method

Abstract

The application discloses a kind of alignment target image segmentation method and device and photoetching equipment, model training method, wherein, method includes: obtaining the original image of maskless lithography alignment target;Original image is input into ResUNet model including attention module, to obtain the target feature area in the original image of maskless lithography alignment target by the output of ResUNet model including attention module;Attention module includes spatial attention module and self-attention module, spatial attention module and self-attention module are arranged in parallel, and self-attention module is calculated based on spatial symmetry with the similarity between points as weight.The application can better carry out feature abstraction to target, realize the segmentation purpose of end-to-end pixel level of alignment target image in a variety of photoetching application scenarios.

Description

Technical Field

[0001] This invention relates to the field of visual image processing, and in particular to a method for segmenting aligned target images, as well as an apparatus for segmenting aligned target images and a maskless lithography device. This invention also relates to a model training method. Background Technology

[0002] Image segmentation is an important component of image recognition and computer vision, with broad applications across various fields. In the field of computer vision, image segmentation techniques have evolved over a long period and can be broadly categorized into two types: traditional image segmentation methods based on manual feature extraction, and image segmentation methods based on deep learning-based feature extraction.

[0003] Traditional image segmentation methods based on manually extracted features mainly employ template matching algorithms based on target shape or image region filtering algorithms based on target physical features. Therefore, different image processing methods and target recognition algorithms need to be developed based on the target image quality under different target manufacturing processes, lamination processes, imaging systems, and optical systems. Algorithm parameters are then configured according to the actual situation in practical applications. This presents two problems: (1) To meet customer needs, there are many types of target recognition algorithms, and their configuration is complex. Non-professional operators often make configuration errors, leading to exposure rejection or even misalignment and scrapping. (2) Due to differences in production processes and environments among equipment manufacturers, manual measurement of target parameters often results in mismatches between target imaging parameters and actual measurements. Furthermore, due to varying target manufacturing processes among different customer manufacturers, various problems such as reflection, blurring, damage, and burrs frequently occur in the aligned target imaging images, posing a significant challenge to the compatibility of traditional image processing algorithms.

[0004] With the increasing popularity of image segmentation methods based on deep learning frameworks, their applications are becoming more and more widespread, demonstrating remarkable performance in areas such as remote sensing image segmentation, landslide detection, road scene segmentation, and medical image detection. However, research on target segmentation tasks based on deep learning frameworks is extremely limited. The UNet series network architecture, initially used for medical segmentation, is based on an encoder-decoder model and, due to its fewer layers, can fully utilize shallow features, making it popular for other segmentation tasks. However, because other segmentation tasks differ from the industrial features of target segments, the algorithms for feature extraction and abstraction in these tasks do not incorporate attention mechanisms based on geometric meanings, such as rotational symmetry and geometric symmetry. This means that existing models lack the ability to express deep features of the target when guiding target segmentation tasks. Furthermore, as a target used for precise positioning, the positioning process involves identifying the geometric center of the target graphic distributed on the image. Typically, the geometric center is identified by calculating the centroid position of the target on the image. Therefore, the quality of the segmentation result is directly related to the centroid deviation between the segmented foreground and the ground truth image, but the current loss calculation function does not take this into consideration. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, one objective of this invention is to propose a maskless photolithography alignment target image segmentation method. This method can better extract features from small-sized segmentation targets with obvious symmetry characteristics, such as targets, solving the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing, and achieving end-to-end pixel-level segmentation of alignment target images in various photolithography application scenarios.

[0006] The second objective of this invention is to propose a ResUNet model training method.

[0007] The third objective of this invention is to provide a maskless photolithography alignment target image segmentation device.

[0008] The fourth objective of this invention is to provide a maskless photolithography device.

[0009] To achieve the above objectives, a maskless lithography alignment target image segmentation method according to a first aspect of the present invention includes: acquiring an original image of a maskless lithography alignment target; inputting the original image into a ResUNet model including an attention module to obtain target feature regions in the original image of the maskless lithography alignment target through the output of the ResUNet model including the attention module; wherein the attention module includes a spatial attention module and a self-attention module, the spatial attention module and the self-attention module are configured in parallel, and the self-attention module calculates based on spatial symmetry using the similarity between points as weights.

[0010] The maskless photolithography alignment target image segmentation method according to embodiments of the present invention uses the UNet network as the baseline model and introduces the ResNet network structure to form a ResUNet network model. In the ResUNet network model, spatial attention module and self-attention module are introduced and set in parallel, which improves the expressive power of the ResUNet network model and enables better feature abstraction of the target image. It solves the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing, further optimizes the segmentation effect, and achieves end-to-end pixel-level segmentation of alignment target images in various photolithography application scenarios.

[0011] In some embodiments, the encoder of the ResUNet model including the attention module includes a main branch network and a secondary branch network; the main branch network configures the ResUNet encoding network layer and the attention module, and the main branch attention module is cross-concatenated with the ResUNet encoding network layer; the secondary branch configures the attention module, and the attention module configured in the secondary branch is connected in parallel to the last layer of the main branch network through pooling; wherein, the convolution kernels of the attention module configured in the main branch and the convolution kernels of the attention module configured in the secondary branch are adaptively configured when building the ResUNet model including the attention module.

[0012] In some embodiments, the original image is input into a ResUNet model including an attention module to obtain the target feature region in the original image of the maskless lithography alignment target through the output of the ResUNet model including the attention module. This includes: inputting the original image into the ResUNet model including the attention module; extracting features through the ResUNet model including the attention module to obtain an original feature map; inputting the original feature map into the main branch network and the secondary branch network respectively; downsampling through the ResUNet encoding network layer of the main branch network and the attention module configured in the main branch to obtain a first feature image; obtaining a second feature image through the attention module configured in the secondary branch; and obtaining the target feature region in the original image of the maskless lithography alignment target based on the first feature image and the second feature image.

[0013] In some embodiments, the process of the spatial attention module processing data includes: obtaining a single-channel feature map by max pooling the original feature map in the channel dimension; calculating weights for the single-channel feature map using a two-dimensional convolution with a kernel size of n, where n is a positive integer; and performing pointwise multiplication of the calculated weights with the original feature map to obtain a first sub-feature image.

[0014] In some embodiments, the spatial attention module further includes the following data processing steps: performing average pooling on the original feature map in the spatial dimension to obtain a first computational feature map with a resolution half that of the original feature map; inverting the first computational feature map to obtain a second computational feature map; performing a ratio operation on the pixel values ​​of corresponding points based on the first and second computational feature maps, and activating them with a Gaussian function to obtain a first weight matrix; performing a point-by-point multiplication operation on the first weight matrix and the first computational feature map as weights to obtain a third computational feature map; upsampling the third computational feature map to obtain a second weight matrix whose single-channel resolution is restored to the size of the original feature map; performing a point-by-point multiplication operation on the second weight matrix and the original feature map in the channel dimension to obtain a second sub-feature image; and performing a point-by-point addition operation on the second sub-feature image and the first sub-feature image to obtain the first feature image.

[0015] In some embodiments, the process of the self-attention module processing data includes: obtaining a single-channel feature map by max pooling the original feature map in the channel dimension; calculating weights for the single-channel feature map by performing a two-dimensional convolution with a kernel size of n, where n is a positive integer; and performing a pointwise multiplication operation between the calculated weights and the original feature map to obtain a third sub-feature image.

[0016] In some embodiments, the self-attention module further includes the following steps in processing the data: performing average pooling on the original feature map in the spatial dimension to obtain a fourth computational feature map with a resolution half that of the original feature map; inverting the fourth computational feature map to obtain a fifth computational feature map; calculating the ratio between the fourth and fifth computational feature maps and activating it using a Gaussian function to obtain a third weight matrix; multiplying the third weight matrix as weights with the fourth computational feature map point-by-point to obtain a sixth computational feature map; upsampling the sixth computational feature map to obtain a fourth weight matrix whose single-channel resolution is restored to the size of the original feature map; multiplying the fourth weight matrix with the original feature map in the channel dimension to obtain a fourth sub-feature image; and adding the fourth sub-feature image and the third sub-feature image point-by-point to obtain a second feature image.

[0017] In some embodiments, the size of n is configured when building the ResUNet model including the attention module.

[0018] In some embodiments, the kernel size of the main branch attention module is n=7, and the kernel size of the secondary branch attention module is n=3.

[0019] To achieve the above objectives, a ResUNet model training method according to a second aspect of the present invention includes: providing a ResUNet model; obtaining a segmentation result of a maskless lithographic alignment target image output by the ResUNet model; calculating the centroid deviation between the segmentation result and the ground truth label of the alignment target; obtaining a loss function based on the centroid deviation; and adjusting the parameters of the ResUNet model based on the loss function.

[0020] According to the ResUNet model training method of the present invention, by introducing a spatial attention module and a self-attention module into the ResUNet network model, and proposing a loss function based on centroid deviation according to the geometric features of the target, the parameters of the ResUNet network model can be adjusted. This enables better feature abstraction of the target, enhances the acquisition of key information in the image, improves the segmentation capability of the ResUNet network model for alignment target images without mask lithography, and achieves the goal of end-to-end pixel-level segmentation of alignment target images in various lithography application scenarios.

[0021] In some embodiments, the ResUNet model includes an attention module, which includes a spatial attention module and a self-attention module. The spatial attention module and the self-attention module are connected in parallel. The self-attention module calculates the similarity between points based on spatial symmetry.

[0022] In some embodiments, the encoder of the ResUNet model includes a main branch network and a secondary branch network; the main branch network is configured with a ResUNet encoding network layer and the attention module, and the main branch attention module is cross-concatenated with the ResUNet encoding network layer; the secondary branch network is configured with the attention module, and the attention module configured in the secondary branch network is connected in parallel to the last layer of the main branch network through pooling; wherein, the kernel of the convolution of the attention module configured in the main branch network is larger than the kernel of the convolution of the attention module configured in the secondary branch network.

[0023] To achieve the above objectives, a maskless photolithography alignment target image segmentation apparatus according to a third aspect of the present invention includes: a processor and a memory communicatively connected to the processor; the memory stores a ResUNet network model and a computer program executed by the processor, wherein the computer program, when executed by the processor, implements the maskless photolithography alignment target image segmentation method described in the above embodiment.

[0024] The maskless photolithography alignment target image segmentation apparatus according to the present invention can perform feature abstraction of the target by executing the maskless photolithography alignment target image segmentation method of the above embodiment, which solves the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing, further optimizes the segmentation effect, and achieves the purpose of end-to-end pixel-level segmentation of alignment target images in various photolithography application scenarios.

[0025] To achieve the above objectives, the maskless lithography apparatus of the fourth aspect of the present invention includes an alignment system and the maskless lithography alignment target image segmentation device described in the above embodiment.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a maskless photolithography alignment target image segmentation method according to an embodiment of the present invention; Figure 2 This is a structural diagram of a ResUNet network model according to an embodiment of the present invention; Figure 3 This is a flowchart of obtaining target feature regions in an original image according to an embodiment of the present invention; Figure 4 This is a structural diagram of an attention module according to an embodiment of the present invention; Figure 5 (a)-(i) are comparison diagrams of the effect of target image feature extraction according to an embodiment of the present invention; Figure 6 This is a flowchart of a ResUNet model training method according to an embodiment of the present invention; Figure 7 This is a schematic diagram of centroid deviation according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a tor-ReLu activation function according to an embodiment of the present invention; Figure 9 This is a block diagram of a maskless photolithography alignment target image segmentation apparatus according to an embodiment of the present invention; Figure 10 This is a block diagram of a maskless photolithography apparatus according to an embodiment of the present invention.

[0028] Figure label: 100 maskless lithography equipment; Alignment system 1; Maskless photolithography alignment target image segmentation device 2; Memory 21, processor 22. Detailed Implementation

[0029] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0030] The structural pattern of alignment targets is typically designed by process engineers, digitally created by drafters, and then fabricated onto an exposure substrate using various mechanical processes. These designed target graphics are usually composed of one or more basic geometric patterns superimposed and combined. The spatial distribution of these patterns naturally exhibits certain relationships, providing abstractable features in terms of spatial attention and self-attention. Based on the geometric symmetry inherent in target images, this invention proposes a maskless photolithography alignment target image segmentation method and a ResUNet model training method. This method is primarily applied to pixel-level segmentation of alignment targets. Compared to the traditional method of manually filling in target parameters for localization, this deep learning-based image segmentation method achieves end-to-end pixel-level segmentation, eliminating the intermediate manual steps.

[0031] The following is for reference. Figures 1-5 This paper describes a maskless photolithography alignment target image segmentation method and a ResUNet model training method according to embodiments of the present invention.

[0032] in, Figure 1 This is a flowchart of a maskless photolithography alignment target image segmentation method according to an embodiment of the present invention, as follows: Figure 1As shown, the method includes steps S1-S2.

[0033] S1, acquire the original image of the maskless photolithography alignment target.

[0034] Specifically, the original image can be a single target image obtained by the CCD camera component in the alignment system capturing the alignment marks on the substrate and then segmenting it. The target image can be a three-channel RGB color image or a single-channel grayscale image.

[0035] S2, by inputting the original image into a ResUNet model including an attention module, the target feature region in the original image of the maskless lithographic alignment target is obtained through the output of the ResUNet model including the attention module.

[0036] The target feature region contains various features of the target image, such as color features, texture features, shape features, spatial relationship features, and other image features.

[0037] Specifically, in some embodiments of the present invention, the ResUNet model is a model based on the UNet network. To improve the expressive power of feature extraction, a ResNet structure is introduced into the UNet network model. Simultaneously, an attention module is also introduced into the ResUNet network model, which can further enhance the model's expressive power. Combining the attention module and the ResUNet model allows for better feature extraction from the original image to obtain a feature image, thereby further optimizing the segmentation effect.

[0038] The attention module comprises a spatial attention module and a self-attention module, which are connected in parallel. The self-attention module calculates weights based on the similarity between points, using spatial symmetry as the weight. Inputting the spatial attention module enhances the ResUNet network model's accuracy in locating target regions, improving its segmentation capabilities and resulting in feature images with strong semantic information. Inputting the self-attention module effectively learns the dependencies between each pixel in the image to be segmented and all pixels in the image, acquiring richer global contextual features and achieving higher image segmentation accuracy.

[0039] The maskless photolithography alignment target image segmentation method according to embodiments of the present invention uses the UNet network as the baseline model and introduces the ResNet network structure to form a ResUNet network model. In the ResUNet network model, spatial attention module and self-attention module are introduced and set in parallel, which improves the expressive power of the ResUNet network model and enables better feature abstraction of the target image. It solves the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing, further optimizes the segmentation effect, and achieves end-to-end pixel-level segmentation of alignment target images in various photolithography application scenarios.

[0040] Figure 2 This is a structural diagram of a ResUNet network model according to an embodiment of the present invention, as shown below. Figure 2 As shown, the ResUNet network model includes an encoder and a decoder. The encoder can perform high-level semantic feature extraction and semantic information condensation on the feature image. The encoder includes a main branch network and a secondary branch network. The main branch network is configured with ResUNet encoding network layers and attention modules. The attention modules of the main branch network are cross-concatenated with the ResUNet encoding network layers. The secondary branch network is configured with attention modules, which are then connected in parallel to the last layer of the main branch network through pooling.

[0041] The kernels of the convolutions in the main branch and the secondary branch attention modules are adaptively configured when building a ResUNet model that includes attention modules. The kernel of the convolutions in the main branch attention modules can be larger, smaller, or equal to the kernel of the convolutions in the secondary branch attention modules, and should be selected appropriately based on the purpose of model training.

[0042] It should be noted that adaptive configuration refers to configuring convolutional kernels of appropriate size according to the size of the input image when building the ResUNet network model. For example, if the model is used to detect high-resolution images, a larger convolutional kernel can be configured, thereby enabling the ResUNet network model to segment high-resolution target images.

[0043] The decoder can perform image semantic restoration and pixel recovery on the feature image. In this embodiment of the invention, the decoder is based on an upsampling structure in the VGG16 convolutional block style. The upsampling method adopts transposed convolution, and the feature map is decoded through a concat method similar to ResUNet short connections, so that the image is restored to its original size for further pixel classification. The segmentation head of the decoder uses 1*1 convolution (Conv1*1) to classify pixels. There are skip connections between the encoder and decoder of the ResUNet network model for information supplementation to improve segmentation accuracy.

[0044] Figure 3 This is a flowchart of obtaining target feature regions in an original image according to an embodiment of the present invention, such as... Figure 3 As shown, the original image is input into a ResUNet model including an attention module to obtain the target feature region in the original image of the maskless lithography alignment target through the output of the ResUNet model including the attention module, including steps S10-S40.

[0045] S10: Input the original image into the ResUNet model including the attention module, extract features through the ResUNet model including the attention module to obtain the original feature map, and input the original feature map into the main branch network and the secondary branch network respectively.

[0046] S20, downsampling is performed through the ResUNet encoding network layer of the main branch network and the attention module configured in the main branch to obtain the first feature image.

[0047] S30, the second feature image is obtained through the attention module configured in the secondary branch.

[0048] S40, obtain the target feature region in the original image of the maskless photolithography alignment target based on the first feature image and the second feature image.

[0049] Specifically, such as Figure 2 As shown, taking a B*3*512*512 original image, the ResUNet encoding network layer uses Res18 Blocks as an example. The overall ResUNet model is based on an encoder-decoder structure. The encoder has two branch networks. The main branch network consists of Res18 Blocks connected in series with attention modules, and the secondary branch network consists of attention modules connected in parallel to the last layer of the main branch network through pooling. In some embodiments of this invention, the ResUNet model uses two types of attention modules in the encoding stage: one with a kernel size of 7 and the other with a kernel size of 3, labeled A3 and A7 in the figure, respectively. The main branch network uses attention modules with a kernel size of 7 because, during the downsampling stage, the kernel size of 7 allows the main branch's attention module to have a larger receptive field, enabling it to focus on spatial features over a wider range during downsampling loss. The secondary branch network uses kernels with a kernel size of 3, which allows for feature extraction based on higher resolution images, enabling it to focus on smaller targets in a smaller field of view within the high-resolution image.

[0050] The original image is input into the ResUNet network model for feature extraction to obtain the original feature map. This original feature map is then input into the main branch network and the secondary branch network. The original feature map is downsampled through a Res18Block structure and a 7-kernel attention module in the main branch network to obtain the first feature image. The original feature map is then passed through a 3-kernel attention module in the secondary branch network, and further downsampled using max pooling to the size of the feature map output from the fourth layer of the main branch network to obtain the second feature image. The first and second feature images are then processed point-by-point, and the result is output from the last layer of the main branch network to obtain the target feature region in the maskless lithography alignment target image. The max pooling and downsampling of the original feature map in the secondary branch network preserves the original features while reducing the number of parameters and computational cost, thus mitigating overfitting to some extent.

[0051] Figure 4 This is a structural diagram of an attention module according to an embodiment of the present invention, such as... Figure 4 As shown, the attention module adopts a dual-attention parallel structure. Therefore, the data processing of both the spatial attention module and the self-attention module includes data processing through two channels.

[0052] The spatial attention module's data processing process in one channel includes: obtaining a single-channel feature map by max pooling the original feature map in the channel dimension; calculating weights on the single-channel feature map using a two-dimensional convolution with a kernel size of n, where n is a positive integer; and performing point-by-point multiplication of the calculated weights with the original feature map to obtain the first sub-feature image.

[0053] Specifically, the attention module in this embodiment of the invention is a scoring module with a variable kernel size. For an original feature map of size Bs*H*W*C as input, a single-channel feature map of size Bs*H*W*1 is first obtained by max pooling in the channel dimension. Then, the single-channel feature map of size Bs*H*W*1 is weighted using a two-dimensional convolution with a kernel size of n, where n is a positive integer. It should be noted that the size of n is configured when building the ResUNet model including the attention module. In some embodiments of the invention, the kernel size n=7 of the main branch attention module and the kernel size n=3 of the secondary branch attention module. Therefore, in the main branch network, a two-dimensional convolution with a kernel size of 7 is used for weight calculation, and in the secondary branch network, a two-dimensional convolution with a kernel size of 3 is used for weight calculation. The calculated weights are multiplied pointwise with the original feature map of size Bs*H*W*C, and the first sub-feature image is output.

[0054] Furthermore, the spatial attention module's data processing in another channel includes: performing average pooling on the original feature map in the spatial dimension to obtain a first computational feature map with half the resolution of the original feature map; inverting the first computational feature map to obtain a second computational feature map; performing a ratio operation on the corresponding pixel values ​​of the first and second computational feature maps and activating them with a Gaussian function to obtain a first weight matrix; multiplying the first weight matrix with the first computational feature map point by point to obtain a third computational feature map; upsampling the third computational feature map to obtain a second weight matrix whose single-channel resolution is restored to the size of the original feature map; multiplying the second weight matrix with the original feature map in the channel dimension to obtain a second sub-feature image; and adding the second sub-feature image and the first sub-feature image point by point to obtain a first feature image.

[0055] In fact, the attention module of this invention is a computational feature module based on central symmetry, using the similarity between points as weights. For an original feature map of size Bs*H*W*C, it first undergoes channel-wise average pooling and two-dimensional average pooling in width and height to merge channels and reduce the feature map, resulting in a first computational feature map of size Bs*H / 2*W / 2*1. The first computational feature map is then inverted in the X and Y directions to obtain a second computational feature map. For the two two-dimensional matrices of the first and second computational feature maps, the similarity between their corresponding points is calculated, and then activated by a Gaussian function to obtain the first weight matrix.

[0056] In this embodiment of the invention, the similarity calculation formula can use the ratio method. When the pixel values ​​of two points are equal, the ratio is 1; the larger the pixel difference, the closer it is to 0. The formula is as follows:

[0057]

[0058] The weight matrix f is normalized using a Gaussian kernel function with a mean of 1 and a variance equal to the variance of the pixel values ​​of the three channels in the set, as shown in the following formula:

[0059] Since both the spatial attention module and the self-attention module output feature maps of size Bs*H*W*1, and their dimensions are exactly the same (i.e., the number of channels, height, and width are equal), the data processing process of the self-attention module is consistent with that of the spatial attention module.

[0060] Similarly, the self-attention module's data processing process in one channel includes: for the original feature map of size Bs*H*W*C as input, firstly, it obtains a single-channel feature map of size Bs*H*W*1 through max pooling in the channel dimension; then, it calculates the weights of the single-channel feature map of size Bs*H*W*1 using a two-dimensional convolution with a kernel size of n, where n is a positive integer and its size is configured when building the ResUNet model including the attention module; then, it performs pointwise multiplication of the calculated weights with the original feature map of size Bs*H*W*C and outputs the third sub-feature image.

[0061] The self-attention module's data processing in another channel includes: performing average pooling on the original feature map of size Bs*H*W*C in the spatial dimension to obtain a fourth computational feature map with half the resolution of the original feature map, i.e., size Bs*H / 2*W / 2*1; inverting the fourth computational feature map in the X and Y directions to obtain a fifth computational feature map; calculating the ratio between the fourth and fifth computational feature maps and activating them with a Gaussian function to obtain a third weight matrix; multiplying the third weight matrix as weights with the fourth computational feature map point-by-point to obtain a sixth computational feature map; upsampling the sixth computational feature map to obtain a fourth weight matrix whose single-channel resolution is restored to the size of the original feature map; multiplying the fourth weight matrix with the original feature map in the channel dimension to obtain a fourth sub-feature image; and adding the fourth sub-feature image and the third sub-feature image point-by-point to obtain a second feature image.

[0062] In summary, based on the ResUNet network model, a multi-kernel attention module is introduced. This module consists of two parts. One part is introduced into the main branch of the ResUNet encoder, where the attention module is cross-connected with the ResUNet encoding layer. During the downsampling stage, a convolutional kernel with a kernel size of 7 is used, resulting in a larger receptive field for the spatial attention module, thus enabling the extraction of more global feature information. The other part is introduced into the secondary branch of the encoder, where the attention module is introduced and connected in parallel to the last layer of the ResNet network through pooling. During the downsampling stage... Using a convolutional kernel with a kernel size of 3, the segment can extract detailed features of small targets on high-resolution feature maps, enabling better segmentation of small targets. The first feature image output by the main branch network and the second feature image output by the secondary branch network are calculated at the pixel level, and the target feature region in the maskless lithography alignment target image is obtained through the output of the last layer of the main branch network. This improves the segmentation accuracy of the alignment target image and solves the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing. It achieves end-to-end pixel-level segmentation of alignment target images in various lithography application scenarios.

[0063] Figure 5 This is a comparison diagram of the effect of target image feature extraction according to an embodiment of the present invention. Figure 5 As shown, (a), (b), and (c) are the original images of the target; (d), (e), and (f) are the target image feature extraction results after introducing spatial attention and channel attention modules in the ResUNet model; and (g), (h), and (i) are the target image feature extraction results after introducing spatial attention and self-attention modules in the ResUNet model. It is clear that introducing both spatial attention and self-attention modules simultaneously improves the feature extraction performance of the ResUNet network model.

[0064] The following describes a ResUNet model training method according to a second aspect of the present invention.

[0065] Figure 6 This is a flowchart of a ResUNet model training method according to an embodiment of the present invention, as follows: Figure 6 As shown, the ResUNet model training method of this embodiment includes at least the following steps S100-S500.

[0066] S100 provides the ResUNet network model.

[0067] like Figure 2 As shown, the ResUNet network model is a symmetrical network structure. The left side is the downsampling encoding process, and the right side is the upsampling decoding process. The original image is input into the ResUNet network model, features are extracted through downsampling, and then the decoder restores it to the original size for pixel-by-pixel classification.

[0068] S200: Obtain the segmentation results of the maskless lithography alignment target image output by the ResUNet network model.

[0069] The target feature region in the maskless lithography alignment target image is obtained by using the ResUNet network model, and segmentation is performed based on the feature region to obtain the segmentation result of the maskless lithography alignment target image.

[0070] S300 calculates the centroid deviation between the segmentation result and the true value label of the target.

[0071] Figure 7 This illustrates the physical meaning of centroid deviation, which refers to the deviation between the centroid of the foreground segment in the model and the centroid in the ground truth labels. For example... Figure 7 As shown, Figure 7Hollow points correspond to the centroids in the ground truth labels, while solid points correspond to the centroids of the foreground segmented by the model. The distance between the two points is the centroid deviation. The specific method for obtaining the centroid deviation is as follows: convert both the segmentation result and the corresponding target ground truth labels into single-channel binary images containing the foreground. Convert the binary images into two-dimensional arrays, where the two-dimensional array includes the zeroth moment and the first moment. The zeroth moment represents the sum of the areas of the segmented foreground in the image, and the first moment represents the sum of the pixel indices of the segmented foreground in the two-dimensional direction. The centroid deviation is obtained based on the two-dimensional arrays of the segmentation result and the corresponding target ground truth labels. Taking a 512*512 input image as an example, the centroid deviation B is defined as:

[0072] Where GT represents the ground truth label and Pred represents the segmentation result of the model, both of which are two-dimensional arrays converted from single-channel binarized images containing only background and foreground. It is the zeroth moment of the image, representing the sum of the areas of the segmented foreground on the image. and It is the first moment of the image, representing the sum of the pixel indices of the segmented foreground in the X and Y directions.

[0073] S400, obtain the loss function based on the centroid deviation.

[0074] In this embodiment of the invention, centroid deviation can be used as an evaluation metric for target segmentation results. Furthermore, a centroid tolerance loss function (tor_B loss, tolerate-barycenterm loss) based on centroid deviation is used as the training loss calculation function for the model. The centroid deviation loss function is defined as follows: [Definition details omitted for brevity]

[0075] Where torReLu is the activation function of the centroid deviation B, and the centroid deviation loss is defined to have a differentiable gradient over all intervals of B, with the gradient increasing as B approaches zero. The formula is as follows:

[0076] The independent variable x is the centroid deviation distance, and tor is the deviation tolerance, representing the acceptable range of centroid deviation. Unlike the two-stage activation function in ReLU6 where the maximum activation value is 6, the maximum activation value in tor-ReLU can be set according to the segmentation accuracy requirements. Generally, the maximum acceptable deviation range is 5, which means that when the centroid of the segmented foreground and the centroid of the foreground in the ground truth label are within 5 pixels, the traditional localization algorithm can identify the segmented region.

[0077] S500 adjusts the parameters of the ResUNet network model according to the loss function.

[0078] Figure 8 It is the image of the tor-ReLu activation function according to an embodiment of the present invention. As Figure 8 shown, let tor be 2, 3, and 5 respectively. When the tor-ReLu is a three-segment function. When the deviation value x ≤ 1, the slope is taken as 1. When the deviation value 1 < x < tor, the slope is taken as tor. Within this range, the descending gradient of the centroid deviation loss will change with the value of tor. When the deviation value x > tor, the loss gradient of the centroid deviation is taken as 0.

[0079] According to the ResUNet model training method of the embodiment of the present invention, by providing a ResUNet network model, the ResUNet network model includes an encoder and a decoder. The encoder includes a main branch network and a secondary branch network. The main branch network configures a ResUNet coding network layer and an attention module. The main branch attention module is cross-connected in series with the ResUNet coding network layer. The secondary branch network configures an attention module. The attention module is connected in parallel to the last layer of the main branch network through pooling. By introducing a spatial attention module and a self-attention module, the feature map is weighted, which can enhance the feature extraction of key information in the image and improve the segmentation accuracy of the ResUNet network model for the target image. And, by providing a centroid deviation loss function, based on the geometric symmetry of the target image, therefore, using the deviation between the centroid of the segmented foreground and the center of the image can better express the geometric features of the target, solving the problems of complex configuration and unsatisfactory segmentation results based on traditional image processing, and achieving the end-to-end pixel-level segmentation purpose of the alignment target image in various lithography application scenarios.

[0080] Next, a maskless lithography alignment target image segmentation device according to the third aspect embodiment of the present invention will be described.

[0081] As Figure 9 shown, the maskless lithography alignment target image segmentation device 2 includes: a processor 22 and a memory 21 communicatively connected to the processor 22. The memory 21 stores a ResUNet network model and a computer program executed by the processor. When the computer program is executed by the processor, it implements the maskless lithography alignment target image segmentation method described in the above embodiment. The maskless lithography alignment target image segmentation method can refer to the description of the above embodiment and will not be elaborated here.

[0082] In some embodiments, the processor 22 may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or other general-purpose processors, etc., without specific limitations.

[0083] The following describes a maskless lithography apparatus according to an embodiment of a fourth aspect of the present invention.

[0084] like Figure 10 As shown, the maskless lithography equipment 100 includes an alignment system 1 and a maskless lithography alignment target image segmentation device 2 as described in the above embodiment. Specifically, when the substrate is located in the alignment system 1, the target on the substrate is illuminated by the light source of the alignment system 1. The CCD camera component in the alignment system 1 is responsible for taking pictures of the target, and the target image is acquired by the image acquisition unit. The target image is then transmitted to the maskless lithography alignment target image segmentation device 2 for image segmentation processing, thereby realizing intelligent recognition of the alignment target and obtaining the relative position of the target on the substrate, thus improving the accuracy of subsequent exposure operations.

[0085] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0086] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A maskless photolithography alignment target image segmentation method, characterized in that, include: Obtain the raw image of the alignment target without mask photolithography; The original image is input into a ResUNet model including an attention module, so that the target feature region in the original image of the maskless lithography alignment target is obtained through the output of the ResUNet model including the attention module. The attention module includes a spatial attention module and a self-attention module. The spatial attention module and the self-attention module are connected in parallel. Both the spatial attention module and the self-attention module include two parallel data processing channels. The self-attention module calculates the similarity between points based on spatial symmetry.

2. The maskless photolithography alignment target image segmentation method according to claim 1, characterized in that, The encoder of the ResUNet model, which includes an attention module, comprises a main branch network and a secondary branch network. The main branch network is configured with a ResUNet coding network layer and the attention module, and the attention module configured in the main branch network is cross-connected with the ResUNet coding network layer; The attention module is configured in the secondary branch network, and the attention module configured in the secondary branch network is connected in parallel to the last layer of the main branch network through pooling; The kernels of the convolutions of the attention modules configured in the main branch network and the kernels of the convolutions of the attention modules configured in the secondary branch network are adaptively configured when building the ResUNet model including the attention modules.

3. The maskless photolithography alignment target image segmentation method according to claim 2, characterized in that, The original image is input into a ResUNet model including an attention module to obtain the target feature region in the original image of the maskless lithography alignment target through the output of the ResUNet model including the attention module, including: The original image is input into the ResUNet model including the attention module, and features are extracted by the ResUNet model including the attention module to obtain the original feature map. The original feature map is then input into the main branch network and the secondary branch network respectively. The first feature image is obtained by downsampling through the ResUNet encoding network layer of the main branch network and the attention module configured in the main branch network; The second feature image is obtained through the attention module configured in the sub-branch network; The target feature region in the original image of the maskless photolithography alignment target is obtained based on the first feature image and the second feature image.

4. The maskless photolithography alignment target image segmentation method according to claim 3, characterized in that, The spatial attention module processes data as follows: The original feature map is then subjected to max pooling along the channel dimension to obtain a single-channel feature map. The single-channel feature map is weighted using a two-dimensional convolution with a kernel size of n, where n is a positive integer. The calculated weights are multiplied point by point with the original feature map to obtain the first sub-feature image.

5. The maskless photolithography alignment target image segmentation method according to claim 4, characterized in that, The spatial attention module's data processing procedure also includes: The original feature map is average pooled in the spatial dimension to obtain a first computational feature map with a resolution of half that of the original feature map; The first computational feature map is inverted to obtain the second computational feature map; The ratio of corresponding pixel values ​​is calculated based on the first and second computational feature maps, and then activated by a Gaussian function to obtain the first weight matrix. The first weight matrix is ​​used as the weight and the first computational feature map is multiplied point by point to obtain the third computational feature map. The third computed feature map is upsampled to obtain a second weight matrix that restores the single-channel resolution to the size of the original feature map. The second weight matrix is ​​multiplied with the original feature map along the channel dimension to obtain the second sub-feature image; The second sub-feature image and the first sub-feature image are added point by point to obtain the first feature image.

6. The maskless photolithography alignment target image segmentation method according to claim 3, characterized in that, The data processing process of the self-attention module includes: The original feature map is then subjected to max pooling along the channel dimension to obtain a single-channel feature map. The single-channel feature map is weighted by a two-dimensional convolution with a kernel size of n, where n is a positive integer; The calculated weights are multiplied point by point with the original feature map to obtain the third sub-feature image.

7. The maskless photolithography alignment target image segmentation method according to claim 6, characterized in that, The data processing process of the self-attention module also includes: The original feature map is average pooled in the spatial dimension to obtain a fourth computational feature map with a resolution of half that of the original feature map; The fourth computational feature map is inverted to obtain the fifth computational feature map; The ratio is calculated based on the fourth and fifth computational feature maps, and activated by a Gaussian function to obtain the third weight matrix. The third weight matrix is ​​used as the weight and the fourth computational feature map is multiplied point by point to obtain the sixth computational feature map. The sixth computed feature map is upsampled to obtain a fourth weight matrix that restores the single-channel resolution to the size of the original feature map; The fourth weight matrix is ​​multiplied with the original feature map in the channel dimension to obtain the fourth sub-feature image; The second feature image is obtained by adding the fourth sub-feature image and the third sub-feature image point by point.

8. The maskless photolithography alignment target image segmentation method according to claim 4 or 6, characterized in that, The size of n is configured when building the ResUNet model including the attention module.

9. The maskless photolithography alignment target image segmentation method according to claim 8, characterized in that, The attention module configured in the main branch network has a kernel size of n=7, and the attention module configured in the secondary branch network has a kernel size of n=3.

10. A ResUNet model training method, characterized in that, The ResUNet model includes an attention module, which comprises a spatial attention module and a self-attention module. The spatial attention module and the self-attention module are connected in parallel. Both the spatial attention module and the self-attention module include two parallel data processing channels. The self-attention module calculates the similarity between points based on spatial symmetry. The ResUNet model is used to obtain the target feature region in the original image of the maskless lithography alignment target. The ResUNet model training method includes: Provides the ResUNet model; Obtain the segmentation result of the maskless photolithography alignment target image output by the ResUNet model; Calculate the centroid deviation between the segmentation result and the true value label of the corresponding target; The loss function is obtained based on the centroid deviation. The parameters of the ResUNet model are adjusted according to the loss function.

11. The ResUNet model training method according to claim 10, characterized in that, The encoder of the ResUNet model includes a main branch network and a secondary branch network; The main branch network is configured with a ResUNet coding network layer and the attention module, and the attention module configured in the main branch network is cross-connected with the ResUNet coding network layer; The attention module is configured in the secondary branch network, and the attention module configured in the secondary branch network is connected in parallel to the last layer of the main branch network through pooling; Wherein, the kernel of the convolution of the attention module configured in the main branch network is larger than the kernel of the convolution of the attention module configured in the secondary branch network.

12. A maskless photolithography alignment target image segmentation device, characterized in that, include: A processor and a memory communicatively connected to the processor; The memory stores a ResUNet network model and a computer program executed by the processor. When the computer program is executed by the processor, it implements the maskless photolithography alignment target image segmentation method according to any one of claims 1-9.

13. A maskless photolithography apparatus, characterized in that, It includes an alignment system and the maskless photolithography alignment target image segmentation apparatus as described in claim 12.

Images