Deblurring method of target image, medium, apparatus thereof and maskless lithography apparatus

Abstract

The application discloses a target image deblurring method, a medium, a device and a maskless lithography equipment. The target image deblurring method comprises the following steps: obtaining an original image of a maskless lithography alignment target; inputting the original image into a Qiformer network model to obtain a residual image, wherein the Qiformer network model comprises an attention layer, and the attention layer extracts key features, value features and local context features of the original image; and obtaining a deblurred image of the maskless lithography alignment target according to the original image and the residual image. The method can improve the definition and details of the image, and reduce the calculation complexity.

Description

Technical Field

[0001] This invention relates to the field of image technology, and in particular to a method, medium and apparatus for deblurring target images, and a maskless lithography device. Background Technology

[0002] With the continuous development of semiconductor manufacturing technology, direct-write lithography has become a crucial step in semiconductor manufacturing. Direct-write lithography eliminates the need for mask fabrication, thus saving on mask production costs. Mask costs constitute a significant portion of the overall lithography cost, and currently, mask prices are rising sharply. Due to the soaring price of masks, developing maskless lithography has become a trend, offering a potential solution to the ever-increasing mask costs and representing a promising lithography technology. However, the quality of the target image directly affects the process capability of direct-write lithography; therefore, researching effective methods to remove target image blur is of great importance. Summary of the Invention

[0003] The present invention aims to at least solve one of the technical problems existing in the prior art. Therefore, one object of the present invention is to propose a method for deblurring target images, which can improve image clarity and detail while reducing computational complexity.

[0004] The second objective of this invention is to provide a computer storage medium.

[0005] The third objective of this invention is to provide a device for deblurring target images.

[0006] The fourth objective of this invention is to provide a maskless photolithography device.

[0007] To address the aforementioned problems, a first aspect of the present invention provides a method for deblurring a target image, comprising: acquiring an original image of a maskless lithographic alignment target; inputting the original image into a Qiformer network model to obtain a residual image, wherein the Qiformer network model includes an attention layer, and extracting key features, value features, and local context features of the original image in the attention layer; and obtaining a deblurred image of the maskless lithographic alignment target based on the original image and the residual image.

[0008] According to the target image deblurring method of the present invention, a Qiformer network model is used to process the original image of the maskless lithography alignment target. Since the attention layer of the Qiformer network model can aggregate local and non-local pixel interactions, it can effectively process high-resolution images, improve image clarity and detail, and improve image quality. In addition, the present application extracts key features, value features and local context features of the original image in the attention layer. That is, the extraction of query features in the Qiformer network model is cancelled, and local context features are introduced in the attention layer. This can not only make up for the feature loss of shallow networks, but also reduce time complexity and computational complexity.

[0009] In some embodiments, the Qiformer network model employs a global residual structure, comprising two convolutional layers and an N-level symmetric encoder-decoder, wherein each level of encoder-decoder comprises a Transformer block consisting of the attention layer and the feedforward layer, and N≥1.

[0010] In some embodiments, inputting the original image into a Qiformer network model to obtain a residual image includes: performing convolution processing on the original image to obtain a shallow feature image; performing deblurring processing on the shallow feature image through an N-level symmetric encoder-decoder to obtain a deep feature image; and performing convolution processing on the deep feature image to obtain the residual image.

[0011] In some embodiments, the attention layer includes a normalization layer, three deformable convolutional layers, and a 1×1 convolutional layer.

[0012] In some embodiments, the shallow feature image is deblurred using an N-level symmetric encoder-decoder to obtain a deep feature image, including: normalizing the shallow feature image in the attention layer to obtain a first normalized image; performing convolution on the first normalized image to extract the key features, the value features, and the local context features; performing pooling and normalization on the key features to obtain the weight values ​​of the value features; obtaining the global context features of the original image based on the value features and the weight values ​​of the value features; sequentially fusing and convolving the local context features and the global context features to obtain a residual feature image; obtaining an attention image based on the residual feature image and the original image; and inputting the attention image into the feedforward layer to obtain the deep feature image.

[0013] In some embodiments, the feedforward layer includes a normalization layer, two deformable convolutional layers, a Gelu activation function, and a 1×1 convolutional layer.

[0014] In some embodiments, inputting the attention image into the feedforward layer to obtain the deep feature image includes: normalizing the attention image to obtain a second normalized image; performing convolution on the second normalized image to obtain a first feature image; inputting the first feature image into the activation function to obtain a second feature image; performing a dot product interaction between the first feature image and the second feature image to obtain a third feature image; performing convolution on the third feature image to obtain a fourth feature image; and fusing the fourth feature image and the attention image to obtain the deep feature image.

[0015] A second aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the target image deblurring method described in the above embodiments.

[0016] A third aspect of the present invention provides a target image deblurring apparatus, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the at least one processor executes the computer program to implement the target image deblurring method described in the above embodiments.

[0017] The target image deblurring apparatus according to embodiments of the present invention, by having a processor execute the target image deblurring method of the above embodiments, can improve the clarity and detail of the image and reduce computational complexity.

[0018] A fourth aspect of the present invention provides a maskless lithography apparatus, including the target image deblurring device described in the above embodiments.

[0019] According to the maskless lithography apparatus of the present invention, by employing the target image deblurring device of the above embodiment, the image clarity and detail can be improved, and the computational complexity can be reduced.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a target image deblurring method according to an embodiment of the present invention; Figure 2This is an architecture diagram of a Qiformer network model according to an embodiment of the present invention; Figure 3 This is an architecture diagram of the attention layer according to an embodiment of the present invention; Figure 4 This is an architecture diagram of a feedforward layer according to an embodiment of the present invention; Figure 5 This is a structural block diagram of a target image deblurring device according to an embodiment of the present invention; Figure 6 This is a structural block diagram of a maskless lithography apparatus according to an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention are described in detail below. The embodiments described with reference to the accompanying drawings are exemplary. The embodiments of the present invention are described in detail below.

[0023] Image deblurring is a crucial task in computer vision and image processing, aiming to recover sharp images from blurred or distorted ones. While significant progress has been made in deblurring techniques, several challenges and problems remain. For example, blurring can originate from various causes, and different types of blur require different deblurring methods, making it difficult to design general-purpose deblurring algorithms. Many efficient deblurring algorithms, such as deep learning-based methods, require substantial computational resources and time, hindering their application in real-time applications and on resource-constrained devices. Furthermore, existing deblurring algorithms often perform well on specific types of blur but poorly on others.

[0024] Among the related technologies, image deblurring techniques include: (1) restoration methods based on image degradation models, which model the image degradation process as a mathematical model and then restore the image by solving the model parameters. However, this method requires understanding and modeling the image degradation process first; (2) restoration methods based on image edges, which use the edge information of the image for restoration. This method works well for images with obvious edge information; (3) restoration methods based on prior knowledge of the image, which use prior knowledge of the image for restoration, such as using the statistical features and texture information of the image; (4) restoration methods based on deep learning, which use deep learning networks for image restoration, such as using convolutional neural networks for image deblurring.

[0025] The above techniques have certain limitations in practical applications and still have the following drawbacks: (1) Some techniques do not achieve ideal restoration results when dealing with certain complex image blurring situations, and may cause distortion, artifacts, and other problems; (2) Some techniques require a large amount of computation and high computing resources, making them unsuitable for resource-constrained scenarios; (3) Some techniques require modeling of the image degradation process or specific models, making them unsuitable for image deblurring needs in some practical scenarios; (4) Some techniques are sensitive to image edge information, and may cause edge sharpening or edge blurring. In summary, image deblurring techniques still need continuous improvement and optimization to adapt to more complex image blurring situations and practical application needs.

[0026] To address the aforementioned problems, the first aspect of this invention proposes a method for deblurring target images. This method can improve image clarity and detail while reducing computational complexity.

[0027] The following is for reference. Figure 1 The target image deblurring method described in this embodiment of the invention is as follows: Figure 1 As shown, the deblurring method includes at least steps S1-S3.

[0028] Step S1: Obtain the original image of the maskless photolithography alignment target.

[0029] Specifically, the original image can be a single target image obtained by taking a picture of the alignment marks on the substrate by the CCD (charge coupled device) camera component in the alignment system of the maskless lithography equipment, and then segmenting it. The original image of the alignment target can be a three-channel RGB color image or a single-channel grayscale image, without limitation.

[0030] Step S2: Input the original image into the Qiformer (Query-irrelevant Transformer) network model to obtain the residual image. The Qiformer network model includes an attention layer, in which key features, value features, and local context features of the original image are extracted.

[0031] Specifically, the attention layer of the Qiformer network model has the advantage of aggregating local and non-local pixel interactions, which can meet the processing requirements of high-resolution images. Based on this, this application uses the Qiformer network model to process the original image of the maskless lithography alignment target to obtain a residual image. Then, when the residual image is used for subsequent processing, the image clarity and detail can be effectively improved, thus improving the image quality.

[0032] In related technologies, for the Transformer model, the time and memory complexity of the key-query dot product in its self-attention mechanism increases quadratically with the spatial resolution of the input image. That is, for a W×H pixel image, the time complexity is O(W×H). Therefore, directly applying it to target image deblurring tasks in industrial scenarios is not feasible. For the existing Restormer model, a multi-Dconv head attention block is introduced to replace the ordinary multi-head self-attention module with linear complexity. Because it is a cross-feature dimension attention mechanism, its time complexity for a W×H pixel image is O(n log n). However, this still places a significant burden on computational resources for target image deblurring tasks in industrial scenarios. To address this issue, this application considers the self-attention mechanism in the Transformer model, which exhibits query-independent behavior at deeper levels. The attention map displays almost consistent context globally, regardless of the query's location. Based on this, this application extracts key features, value features, and local contextual features of the original image at the attention layer. In other words, it eliminates the query feature extraction required in the Restormer network model, meaning the attention layer exhibits query-independent behavior. Introducing local contextual features at the attention layer compensates for feature deficiencies in shallow networks, reduces time and computational complexity, and makes this method more suitable for target image deblurring tasks in industrial scenarios.

[0033] Step S3: Obtain the deblurred image of the maskless photolithography alignment target based on the original image and the residual image.

[0034] Specifically, the original image and the residual image are fused to generate a clear image, which is to obtain the deblurred image of the maskless lithography alignment target.

[0035] According to the target image deblurring method of the present invention, a Qiformer network model is used to process the original image of the maskless lithography alignment target. Since the attention layer of the Qiformer network model can aggregate local and non-local pixel interactions, it can effectively process high-resolution images, improve image clarity and detail, and improve image quality. In addition, the present application extracts key features, value features and local context features of the original image in the attention layer. That is, the extraction of query features in the Qiformer network model is cancelled, and local context features are introduced in the attention layer. This can not only make up for the feature loss of shallow networks, but also reduce time complexity and computational complexity.

[0036] In some embodiments, the Qiformer network model employs a global residual structure. The Qiformer network model includes two convolutional layers and N levels of symmetric encoder-decoder, where each encoder-decoder level comprises a Transformer block consisting of an attention layer and a feedforward layer, and N ≥ 1. For example, refer to... Figure 2 As shown, the overall architecture of the Qiformer network model is a U-shaped hierarchical network, consisting of four levels of symmetric encoder-decoder, with each level composed of multiple Transformer blocks. The convolutional layers are 3×3 convolutional layers.

[0037] In some embodiments, reference Figure 2 As shown, inputting the original image into the Qiformer network model to obtain the residual image includes: performing convolution processing on the original image, denoted as I, to obtain a shallow feature image; performing deblurring processing on the shallow feature image through an N-level symmetric encoder-decoder to obtain a deep feature image; performing convolution processing on the deep feature image to obtain the residual image, denoted as net(I), and finally obtaining the deblurred image Y = net(I) + I of the maskless lithography alignment target.

[0038] For example, for the original image input to the Qiformer network model, a 3×3 convolutional layer is first used to extract shallow features to obtain a shallow feature image. The shallow feature image is then input into the encoder-decoder to deblur and obtain a deep feature image. Finally, the deep feature image is processed by a 3×3 convolutional layer to recover the features and obtain a residual image.

[0039] In some embodiments, reference Figure 3 As shown, the attention layer consists of a normalization layer, three deformable convolutional layers, and one 1×1 convolutional layer. The deformable convolutional layers can employ the DCNv3 operator.

[0040] In some embodiments, reference Figure 3 As shown, the process of deblurring a shallow feature image using an N-level symmetric encoder-decoder to obtain a deep feature image includes: in the attention layer, normalizing the shallow feature image to obtain a first normalized image; convolving the first normalized image to extract key features, value features, and local context features; pooling and normalizing the key features to obtain weight values ​​for the value features; obtaining the global context features of the original image based on the value features and their weight values; fusing and convolving the local and global context features sequentially to obtain a residual feature image; obtaining an attention image based on the residual feature image and the original image; and inputting the attention image into a feedforward layer to obtain the deep feature image.

[0041] For example, refer to Figure 3As shown, the first normalized image is obtained by passing the shallow feature image X through a normalization layer. Then, the DCNv3 convolution operator is used to generate key features (K), value features (V), and local context features. ,Right now Furthermore, pooling and normalization operations are performed on the key feature key(K) to generate weight values ​​for the value feature value(V), and the weight values ​​are then interacted with the value feature value(V) through a dot product to obtain the global context features. ,Right now Furthermore, local contextual features are then integrated. and global context features Then, the residual feature image is obtained by convolution through a 1×1 convolutional layer. The residual feature image is then added to the original image to finally output the attention image. ,Right now .

[0042] In some embodiments, reference Figure 4 As shown, the feedforward layer consists of a normalization layer, two deformable convolutional layers, a GeLU activation function, and a 1×1 convolutional layer. The deformable convolutional layers can employ the DCNv3 operator.

[0043] In some embodiments, reference Figure 4 As shown, the attention image is input into the feedforward layer to obtain a deep feature image, including: processing the attention image... Normalization is performed to obtain a second normalized image. The second normalized image is convolved to obtain the first feature image. The first feature image is input into the activation function to obtain the second feature image. The first feature image and the second feature image are dot-producted to obtain the third feature image. ,Right now The third feature image is convolved to obtain the fourth feature image. The fourth feature image and the attention image are fused to obtain the deep feature image. ,Right now Therefore, by using the feedforward neural network based on the gate mechanism and employing the DCNv3 operator in the feedforward layer, richer shallow contextual features can be obtained, which helps to improve image clarity and detail, and thus improve image quality.

[0044] A second aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the target image deblurring method of the above embodiments.

[0045] A third aspect of the present invention provides a deblurring apparatus for a target image, such as... Figure 5 As shown, the device 10 includes at least one processor 1 and a memory 2 communicatively connected to at least one processor 1.

[0046] The memory 2 stores a computer program that can be executed by at least one processor 1. When the at least one processor 1 executes the computer program, it implements the target image deblurring method of the above embodiment.

[0047] The target image deblurring apparatus 10 according to an embodiment of the present invention can improve the image clarity and detail and reduce computational complexity by executing the target image deblurring method of the above embodiment by the processor 1.

[0048] A fourth aspect of the present invention provides a maskless lithography apparatus, such as... Figure 6 As shown, the device 20 includes a target image deblurring device 10 according to the above embodiment.

[0049] According to the maskless lithography apparatus 20 of the present invention, by employing the target image deblurring device 10 of the above embodiment, the image clarity and detail can be improved, and the computational complexity can be reduced.

[0050] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0051] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for deblurring a target image, characterized in that, include: Obtain the raw image of the alignment target without mask photolithography; The original image is input into a Qiformer network model to obtain a residual image, wherein the Qiformer network model includes an attention layer, and the key features, value features and local context features of the original image are extracted in the attention layer; The deblurred image of the maskless photolithography alignment target is obtained based on the original image and the residual image; In the attention layer, the input image of the attention layer is normalized to obtain a first normalized image; The first normalized image is subjected to convolution processing to extract the key features, the value features, and the local context features, respectively. The key features are pooled and normalized to obtain the weight values ​​of the value features; The global context features of the original image are obtained based on the value features and the weight values ​​of the value features; The local context features and the global context features are sequentially fused and convolved to obtain the residual feature image; The attention image output by the attention layer is obtained based on the residual feature image and the image input to the attention layer; The attention image is input into the feedforward layer of the current level encoder-decoder to obtain the output image of the current level encoder-decoder.

2. The target image deblurring method according to claim 1, characterized in that, The Qiformer network model adopts a global residual structure. The Qiformer network model includes two convolutional layers and N levels of symmetric encoder-decoder, wherein each level of encoder-decoder includes a Transformer block composed of the attention layer and the feedforward layer, and N≥1.

3. The target image deblurring method according to claim 2, characterized in that, The original image is input into the Qiformer network model to obtain the residual image, including: The original image is convolved to obtain a shallow feature image; The shallow feature image is deblurred by an N-level symmetric encoder-decoder to obtain a deep feature image; The deep feature image is convolved to obtain the residual image.

4. The target image deblurring method according to claim 3, characterized in that, The attention layer consists of a normalization layer, three deformable convolutional layers, and a 1×1 convolutional layer.

5. The target image deblurring method according to claim 1, characterized in that, The feedforward layer includes a normalization layer, two deformable convolutional layers, a Gelu activation function, and a 1×1 convolutional layer.

6. The target image deblurring method according to claim 5, characterized in that, In the feedforward layer, the attention image is normalized to obtain a second normalized image; The second normalized image is convolved to obtain the first feature image; The first feature image is input into the activation function to obtain the second feature image; The first feature image and the second feature image are dot-producted to obtain the third feature image; The third feature image is convolved to obtain the fourth feature image; The fourth feature image and the attention image are fused together to obtain the output image of the current level encoder-decoder.

7. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target image deblurring method according to any one of claims 1-6.

8. A deblurring device for a target image, characterized in that, include: At least one processor; A memory that is communicatively connected to at least one of the processors; The memory stores a computer program that can be executed by at least one of the processors, and when the at least one processor executes the computer program, it implements the target image deblurring method according to any one of claims 1-6.

9. A maskless photolithography apparatus, characterized in that, The device includes the target image deblurring apparatus as described in claim 8.

Images