A method and device for identifying nanoscale and irregularity defects

Abstract

The application relates to the technical field of semiconductor detection, and discloses a semiconductor chip detection method, in particular to a nanoscale and irregular morphology defect recognition method and device. By taking a visual Transformer as a main image encoder, the encoder structure is hierarchized and locally enhanced in view of the characteristics of nanoscale defects, such as small size, weak local texture, discontinuous boundary and large calculation amount of high-resolution images. Firstly, the edge continuity between adjacent image blocks is reserved by embedding overlapping image blocks, then shallow texture, middle morphology and deep global semantic feature are respectively extracted through multi-level Transformer coding stages, and the output features of all levels are input into a feature pyramid module for fusion; subsequently, a deformation convolution branch is arranged to extract defect contour features, and a segmentation output branch is arranged to add and fuse the feature pyramid features and the semantic features of the last layer of the encoder, and then output the segmentation result, so that the nanoscale defects and irregular morphology defects can be accurately recognized.

Description

Technical Field

[0001] This application relates to the field of semiconductor testing technology, specifically a semiconductor chip testing method and apparatus for identifying nanoscale and irregular morphological defects. Background Technology

[0002] With the development of semiconductor manufacturing, micro-nano fabrication, display panels, precision thin films, optical components, and high-end equipment manufacturing technologies, the accuracy requirements for defect detection on or inside product surfaces are constantly increasing. In these application scenarios, defects may manifest as nanoscale point particles, micro-pinholes, slender cracks, uneven film layers, localized corrosion spots, edge chipping, or irregular scratches. These types of defects are typically characterized by small scale, low contrast, blurred boundaries, irregular shapes, and strong interference from background textures. Therefore, they place high demands on the detection model's ability to represent local details, understand global semantics, and adapt to boundaries.

[0003] Traditional defect identification methods typically employ algorithms such as threshold segmentation, edge detection, texture statistics, morphological processing, or template matching. These methods are effective in scenarios with stable lighting, simple backgrounds, and regular defect shapes. However, when the image acquisition environment changes, the sample surface has complex textures, the defect grayscale is close to the background grayscale, or the defect exhibits irregular shapes such as bending, diffusion, or breakage, traditional algorithms have low robustness and are prone to missed detections, false detections, and incomplete boundary segmentation. Summary of the Invention

[0004] To address the technical problems of loss of local details, insufficient global semantic understanding, and incomplete boundary segmentation in the prior art when dealing with nanoscale, low-contrast, blurred boundary, and irregular shape defects, this application provides a method and apparatus for identifying nanoscale and irregular shape defects.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, a method for identifying nanoscale and irregular morphological defects is provided. The method includes: dividing an image containing an object to be detected into multiple image blocks; converting the features of the multiple image blocks into feature tokens; performing gradient encoding on the feature tokens based on a multi-gradient encoder to obtain multi-scale features; the multi-gradient encoder includes an image block embedding layer, a position encoding layer, multiple encoding stages, and a multi-scale feature output layer; fusing the multi-scale features based on a feature pyramid and then fusing them with the output features of the last layer of the multi-gradient encoder to obtain fused features; and using the fused features to obtain a defect mask based on a segmentation prediction head.

[0006] In some specific implementations, the conversion of features of multiple image blocks into feature tokens includes converting the image blocks into feature tokens through linear mapping or convolutional mapping.

[0007] In some specific implementations, the multi-gradient encoder includes a first encoder, a second encoder, a third encoder, and a fourth encoder. The first encoder is used to retain high-resolution edge information and texture information, the second encoder is used to retain the local structure and short-range morphological information of the defect, the third encoder is used to retain high-level semantic information, and the fourth encoder is used to represent the global contextual relationship between the defect and the background.

[0008] In some specific implementations, the first encoder, the second encoder, the third encoder, and the fourth encoder each include a normalization layer, a multi-head self-attention layer, a residual connection layer, a feedforward network layer, and a local detail enhancement unit.

[0009] In some specific implementations, the multi-gradient encoder further includes an image patch embedding layer and a position encoding layer, wherein the position encoding layer is used to add spatial position information to the feature token.

[0010] In some specific implementations, the local detail enhancement unit is equipped with depthwise separable convolution or local receptive field convolution to compensate for the domain information of the feature map, and then the compensated features are mapped to the token sequence again.

[0011] In some specific implementations, the first encoder and the second encoder are equipped with local window self-attention or group self-attention, and the third encoder and the fourth encoder are equipped with cross-window interaction self-attention or global self-attention.

[0012] In some specific implementations, the multi-scale features are fused based on a feature pyramid, including: unifying the channels of the multi-scale features based on the feature pyramid, and upsampling the deep features step by step and merging them laterally with the shallow features to obtain pyramid features.

[0013] In some specific implementations, the pyramid features and the features output by the fourth encoder are scale-aligned and channel-aligned, and then added element by element to obtain the fused features.

[0014] Secondly, a nanoscale and irregular morphological defect recognition device is provided for use in a semiconductor inspection system, comprising: an image segmentation module for segmenting an image containing an object to be detected into multiple image blocks, and converting the features of the multiple image blocks into feature tokens; an encoding module for performing gradient encoding processing on the feature tokens based on a multi-gradient encoder to obtain multi-scale features; and a detection module for fusing the multi-scale features based on a feature pyramid with the output features of the last layer of the multi-gradient encoder to obtain fused features, and using the fused features to obtain a defect mask based on a segmentation prediction head.

[0015] The embodiments of the present invention bring the following beneficial effects: In the technical solution provided in this application embodiment, a visual Transformer is used as the backbone image encoder. Addressing the characteristics of nanoscale defects—small scale, weak local texture, discontinuous boundaries, and high computational cost of high-resolution images—the encoder structure is designed with hierarchical structure and local enhancements. First, overlapping image block embedding preserves the edge continuity between adjacent image blocks. Then, a multi-level Transformer encoding stage extracts shallow texture, mid-level morphology, and deep global semantic features, respectively, and the output features at each level are input into a feature pyramid module for fusion. Subsequently, a deformation convolution branch is set to extract defect contour features, and a segmentation output branch is set to add and fuse the feature pyramid features with the semantic features of the last layer of the encoder to output the segmentation result, thereby achieving accurate identification of nanoscale defects and irregularly shaped defects.

[0016] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0017] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] The methods, systems, and / or procedures shown in the accompanying drawings will be further described with reference to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example figures represent similar mechanisms in the various views of the drawings.

[0020] Figure 1 This is a schematic diagram of the process for identifying nanoscale and irregularly shaped defects.

[0021] Figure 2 This is a schematic diagram of the encoding structure.

[0022] Figure 3 This is a schematic diagram of the defect identification device.

[0023] Figure 4 This is a schematic diagram of a readable medium structure. Detailed Implementation

[0024] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0025] In the detailed description below, numerous specific details are illustrated with examples to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that this application can be practiced without these details. In other instances, well-known methods, procedures, systems, components, and / or circuits have been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of this application.

[0026] This application uses flowcharts to illustrate the execution process performed by a system according to embodiments of this application. It should be clearly understood that the execution processes in the flowcharts may not be executed sequentially. Instead, these execution processes may be executed in reverse order or simultaneously. Additionally, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0027] This application provides a semiconductor inspection system for detecting surface defects in semiconductor products, including abnormalities such as breakage, scratches, and contamination. Specifically, the system employs automated optical inspection, acquiring image information of the surface of the semiconductor product to be inspected and determining whether the product has corresponding defects based on pixel features in the image.

[0028] With the development of semiconductor manufacturing technology, the accuracy requirements for defect detection on or inside product surfaces are constantly increasing. Traditional image detection techniques, due to their low robustness, are prone to missed detections, false detections, and incomplete boundary segmentation, and are rarely used in high-precision semiconductor surface inspection scenarios. While existing deep learning methods can automatically learn features, ordinary convolutional neural networks rely on fixed sampling positions, which is insufficient for adapting to deformed targets, and continuous downsampling can easily lead to the loss of fine-grained edge information of nanoscale targets. On the other hand, directly applying visual Transformers to process high-resolution industrial images often faces the dilemma of excessive computation and insufficient ability to depict local texture details, making it difficult to simultaneously achieve global semantic understanding, local deformation adaptation, preservation of small target details, and pixel-level boundary integrity.

[0029] Therefore, to address the aforementioned crystal technology problems, embodiments of this application provide a method for identifying nanoscale and irregularly shaped defects, enabling defect identification in semiconductor products. For details regarding this method, please refer to [link to relevant documentation]. Figure 1 As shown, it includes the following steps: Step S11. Divide the image containing the object to be detected into multiple image blocks, and convert the features of the multiple image blocks into feature tokens.

[0030] In this embodiment, the image containing the object to be detected can be a high-resolution semiconductor product image acquired by imaging devices such as industrial cameras, microscopes, scanning electron microscopes, or line array cameras.

[0031] Furthermore, to ensure the accuracy of subsequent processing, the semiconductor product image needs to be preprocessed before it is segmented into blocks. This preprocessing includes flat-field correction, background estimation, noise filtering, local contrast enhancement, and scale normalization to eliminate interference from uneven illumination and sensor noise.

[0032] Among them, block processing divides the semiconductor product image into multiple sub-image blocks with overlapping or non-overlapping regions. This division method aims to reduce the problem of the boundary of small defects being truncated due to hard cutting. In particular, for nanoscale weak targets such as pinholes, particles or crack ends, overlapping cutting can preserve the edge continuity between adjacent image blocks.

[0033] Feature transformation projects the pixel data of each image patch into a high-dimensional feature space through linear or convolutional mapping, thereby generating a sequence of feature tokens. Specifically, linear mapping constructs a learnable embedding matrix, flattens each image patch into a vector, and performs matrix multiplication with the embedding matrix to obtain the corresponding feature token. Convolutional mapping uses a convolutional kernel to perform a sliding window operation on the image patch, extracting local texture and edge features, and reshaping the output feature map into a token sequence. Both mapping methods support an overlapping block strategy, where adjacent image patches share some pixel regions, thus preventing nanoscale defects from being truncated when they happen to be located at the block boundary, ensuring the integrity of defect edge information. For example, when the input image is divided into 16×16 pixel image patches, if convolutional mapping is used, a 3×3 convolutional kernel with a stride of 8 can be used for feature extraction, resulting in a 50% overlap between adjacent patches, thereby preserving the continuous structural information of tiny pinholes or crack endpoints. Through this learnable mapping mechanism, the model can adaptively capture subtle grayscale changes and texture patterns within image patches, significantly improving the feature tokens' responsiveness to low-contrast defects and providing high-quality input representations for subsequent gradient encoding processing.

[0034] Step S12. Perform gradient encoding on the feature token based on a multi-gradient encoder to obtain multi-scale features.

[0035] In this embodiment, the multi-gradient encoder is a hierarchical visual Transformer architecture used for deep feature extraction and abstraction of the input feature token sequence. For more information on this multi-gradient encoder structure, please refer to [link to relevant documentation]. Figure 2 As shown, the architecture includes an image patch embedding layer, a positional encoding layer, multiple encoding stages, and a multi-scale feature output layer. The image patch embedding layer is responsible for performing the block segmentation and mapping processing in step S11 and serves as the input interface for the encoder. The positional encoding layer adds spatial location information to the feature tokens. Since the Transformer architecture itself lacks the translation invariance and positional awareness of convolutional neural networks, the positional encoding layer adds learnable position vectors or relative positional biases, enabling the encoder to distinguish semantic differences between similar textures in different spatial coordinates during self-attention computation. This is crucial for locating the specific position of defects.

[0036] In this context, the image patch embedding layer refers to a processing unit that divides the preprocessed high-resolution image to be detected into multiple image patches with overlapping regions and maps each image patch to an initial feature vector. This layer specifically includes an overlap segmentation module and a linear / convolutional mapping module. The image patch embedding layer is formed by dividing the input image... Image patch sequence is obtained by sliding cropping according to preset step size and window size. Then through a learnable projection matrix Alternatively, the convolution kernel can convert each image patch into a feature token. The function of this is to perform the initial conversion from pixel space to Transformer token space, while using an overlapping segmentation strategy to preserve the edge continuity between adjacent image blocks and avoid nanoscale defect boundaries being truncated by hard segments.

[0037] The positional encoding layer is used to inject spatial location information into the feature token, including a learnable positional encoding sublayer or a relative positional bias sublayer. The positional encoding layer generates a position vector with the same dimensions as the feature token. This is obtained by adding it element-wise to the initial feature token sequence output by the image patch embedding layer, i.e. Its function is to enable the encoder to distinguish similar textures at different spatial locations during self-attention computation, recover the spatial topology of the image, and avoid positional confusion caused by the global self-attention mechanism. Specifically, spatial positional information is incorporated by fusing vectors representing the absolute coordinates of image patches into feature tokens. This process generates the corresponding positional encoding vector based on the row and column indices of the image patch in the original image, using a lookup table or function. For high-resolution semiconductor images, positional encoding can employ learnable parameters, i.e., optimizing a unique vector for each location during training, or it can use relative positional biases, directly introducing a bias term related to relative distance when calculating attention weights.

[0038] Specifically, after the image patch embedding layer outputs the initial token sequence, the position coding layer immediately intervenes to calculate the position vector. This is superimposed on each token. This operation makes subsequent self-attention calculations not only dependent on the similarity of image content, but also constrained by spatial distance.

[0039] In this embodiment, the image patch embedding layer and the positional encoding layer work together. The image patch embedding layer is responsible for extracting local content features and maintaining boundary integrity, while the positional encoding layer is responsible for assigning clear spatial coordinate attributes to these features. Through their synergy, the model is able to capture subtle gray-scale abrupt changes and accurately locate their specific coordinates within the entire image when processing nanoscale point particles or tiny pinholes, thus laying an accurate spatial foundation for subsequent multi-scale feature extraction. For example, when there are two tiny defects with the same gray value but different locations in the image to be detected, without the positional encoding layer, the self-attention mechanism may mistakenly treat them as the same feature and confuse them. However, with the addition of positional encoding, the model can accurately distinguish between the two based on spatial bias, effectively reducing the false detection rate. This step aims to construct an initial feature sequence with spatial awareness, providing input data containing complete geometric information for the multi-gradient encoder.

[0040] The encoding process comprises multiple parallel levels of encoding modules for each of the several encoding stages. Each level includes a normalization layer, a multi-head self-attention layer, a residual connection layer, a feedforward network layer, and a local detail enhancement unit. Specifically, the multi-head self-attention layer establishes long-range dependencies between defective and background regions, capturing global contextual information. The local detail enhancement unit employs depthwise separable convolutions or small convolutional kernels to compensate for texture, edge, and grayscale abrupt changes between adjacent image patches, enhancing the encoder's responsiveness to weak-contrast defects and fine structures. A multi-scale feature output layer extracts and outputs feature maps with different resolutions and semantic levels from different encoding stages.

[0041] Furthermore, the multi-gradient encoder is constructed as a four-stage, progressively functioning encoding system: the first encoder, the second encoder, the third encoder, and the fourth encoder. These four encoders are logically connected in series or in parallel, each responsible for extracting image features at different levels of abstraction. The first encoder, located at the beginning of the encoding process, directly receives the feature token sequence after positional encoding. The second encoder receives the output of the first encoder; the third encoder receives the output of the second encoder; and the fourth encoder, at the end of the encoding process, receives the output of the third encoder. This hierarchical setup allows the model to simulate the cognitive process of the human visual system, from perceiving local details to understanding global semantics.

[0042] The four encoders output four multi-scale features: C1, C2, C3, and C4. C1 corresponds to shallow features with high spatial resolution, suitable for describing point-like defects; C4 corresponds to deep features with low spatial resolution, suitable for determining the overall category of defects. Through this multi-gradient, hierarchical encoding process, the model can simultaneously acquire comprehensive feature representations from microscopic texture to macroscopic semantics, solving the problem that single-scale features cannot simultaneously capture details and semantics.

[0043] Specifically, the first encoder is configured to focus on extracting high-frequency detail features from the input image. Specifically, this encoder maintains high feature map spatial resolution during processing, typically avoiding large downsampling operations on the feature map, or performing only slight stride convolutions to ensure that pixel-level edges of nanoscale defects are not blurred. The first encoder's role is to capture regions in the image with abrupt changes in grayscale, such as the outlines of tiny particles, the edges of pinholes, and the subtle texture undulations of thin film surfaces. Its working principle is to use a shallow neural network to model the pixel relationships within a local neighborhood, generating feature representations containing rich geometric information. For example, when there are dust particles with a diameter of only a few pixels on the surface of the object to be detected, the first encoder can sensitively respond to gradient changes at the particle edges, generating a feature response map with high activation values, unlike deep networks which smooth out tiny targets due to an excessively large receptive field. Thus, the feature C1 output by the first encoder provides crucial detailed information for subsequent identification of nanoscale point defects.

[0044] The second encoder is configured to further integrate structural associations within the local neighborhood, building upon the first encoder, to extract short-range morphological features of defects. This encoder primarily describes the connectivity, orientation, and shape patterns within a small area of ​​the defect, such as the direction of crack extension, the continuity of scratches, or the local aggregation of irregular spots. The second encoder connects the discrete edge points extracted by the first encoder into semantically meaningful local structural fragments. Its processing typically involves a moderately sized receptive field, enabling the establishment of dependencies between adjacent image patches without losing too much spatial information. For example, for a curved micro-crack, the first encoder might only detect discontinuous edges on both sides of the crack, while the second encoder can analyze the spatial arrangement of these edge fragments to identify them as local segments belonging to the same continuous crack, thereby generating feature C2 representing the crack orientation and local curvature. This process effectively enhances the model's ability to structurally represent irregularly shaped defects.

[0045] The third encoder is configured to extract mid-to-high-level abstract semantic features from the image, primarily used to distinguish the essential differences between defective regions and complex background textures. As the encoding layers deepen, the receptive field of the third encoder significantly expands, covering a larger image area, thus ignoring local noise interference and focusing on regions with specific semantic patterns. This encoder utilizes prior knowledge learned through deep learning to determine whether an image patch belongs to the defect category, even if the defect has extremely low contrast or is similar to the background texture. Its working principle involves aggregating low-level texture and structural information into high-level class discriminative features through multi-layer nonlinear transformations and self-attention mechanisms. For example, when detecting display panels with complex lattice backgrounds, the background itself may exhibit regular periodic textures. The third encoder can learn this periodicity and mark abnormal regions deviating from this pattern as high-confidence defect features, outputting semantically discriminative feature C3. This allows the model to maintain a high detection rate even when facing strong background interference.

[0046] The fourth encoder, as the final layer of the multi-gradient encoder, is configured to model global contextual dependencies across the entire image. This encoder has the largest receptive field, potentially reaching the entire image, and its core function is to understand the macroscopic logical relationship between defects and their surrounding environment, as well as the distribution of defects across the entire workpiece surface. The fourth encoder eliminates local false positives and provides final confirmation of the overall shape and location of defects. Its working principle utilizes a global self-attention mechanism, allowing any location in the image to directly interact with all other locations, thus achieving the most complete scene understanding. For example, when a dark spot resembling a suspected defect exists in the image, the fourth encoder combines the overall image illumination distribution, the normal texture patterns of other areas, and the relative position of the dark spot globally to comprehensively determine whether it is a real defect. If the dark spot is in a normal shadow transition zone and conforms to the global illumination model, the fourth encoder may suppress its activation value; otherwise, it will enhance it. The resulting feature C4 contains the most comprehensive global contextual information between the defect and the background, providing strong semantic support for the final segmentation prediction.

[0047] This application constructs a complete feature extraction chain from microscopic details to macroscopic semantics through the collaborative operation of a first encoder, a second encoder, a third encoder, and a fourth encoder. The edge and texture information provided by the first encoder ensures that the details of nanoscale targets are not lost; the local structural information supplemented by the second encoder enhances the continuous expression of irregular morphological defects; the high-level semantic information extracted by the third encoder improves the classification accuracy in complex backgrounds; and the global context relationship established by the fourth encoder further optimizes the robustness of defect judgment. This multi-level division of labor mechanism enables the model to simultaneously consider the localization accuracy of small targets and the semantic integrity of large-scale targets, effectively solving the technical problem in existing technologies where single-scale features are difficult to adapt to diverse defect morphologies, and significantly improving the overall performance of nanoscale and irregular morphological defect recognition.

[0048] Furthermore, the first encoder and the second encoder in this embodiment are equipped with local self-attention or group self-attention mechanisms. Local self-attention divides the feature token sequence into multiple non-overlapping or partially overlapping local windows, and restricts self-attention calculation to only occur between tokens within the same window. Group self-attention divides the feature tokens into multiple groups according to preset rules, and self-attention calculation is performed only between tokens within the same group.

[0049] Specifically, the first and second encoders correspond to the shallow stages in the network architecture, primarily processing high-resolution feature maps. If global self-attention is directly applied at this stage, the computational complexity will increase quadratically with image resolution, making deployment in industrial settings difficult. By setting local window self-attention, for example, dividing the feature map into H×W windows of size M×M, with each window independently calculating its attention matrix, the computational complexity can be reduced from... Reduce to Where N represents token heatstroke. Grouped self-attention is employed, dividing the channel dimension or spatial dimension into G groups, each group independently interacting with attention, achieving the same goal of reducing memory usage and accelerating inference. This setup allows the first encoder to efficiently preserve high-frequency texture information of nanoscale point defects, enabling the second encoder to accurately capture the local orientation and short-range morphology of fine cracks without prematurely introducing global context, which could lead to the smoothing or obscuring of local details.

[0050] Furthermore, cross-window interactive self-attention or global self-attention are implemented in the third and fourth encoders. Cross-window interactive self-attention allows tokens from different local windows to exchange information during attention computation, achieved through a shifted window mechanism or coefficient connection mode, breaking down the isolation boundaries of local windows. Global self-attention performs fully connected attention computation between any two tokens in the sequence, establishing long-distance dependencies across the entire graph. These two attention mechanisms expand the receptive field of the deep encoding stage, enabling it to understand the overall semantic association between defective regions and complex backgrounds, as well as the spatial topological relationships between different parts of irregular defects. The third and fourth encoders correspond to the deep stages of the network architecture, where the spatial resolution of feature maps has decreased, but the semantic information is more abstract. At this stage, the importance of local details decreases, while understanding the global context becomes crucial. By implementing cross-window interactive self-attention, such as using a shifted window strategy, tokens from adjacent windows can pay attention to each other in the next layer computation, thus achieving the gradual fusion of global information without significantly increasing computational cost. Alternatively, global self-attention can be used, allowing each token in the fourth encoder to see all other regions of the entire image, thereby accurately distinguishing local noise from true global defect patterns. This design, from shallow local focus to deep global awareness, ensures that the model can both identify tiny pinholes and completely segment large-scale irregular scratches.

[0051] Furthermore, the encoder in this embodiment includes a normalization layer, a multi-head self-attention layer, a residual connection layer, a feedforward network layer, and a local detail enhancement unit. The normalization layer standardizes the input feature tokens to stabilize the neural network training process and accelerate convergence; this can be implemented using layer normalization or batch normalization. The multi-head self-attention layer establishes long-distance dependencies in the feature token sequence by mapping features to query vectors, key vectors, and value vectors, and calculating the association weights between different image patches, thereby capturing global contextual information about the defect region and its surrounding background. The residual connection layer directly superimposes the input of the current layer onto the output to alleviate the gradient vanishing problem in deep networks, ensuring that original detail information can be passed to subsequent layers. The feedforward network layer performs nonlinear transformations on each feature token, enhancing the model's feature representation capability.

[0052] Specifically, the Local Detail Enhancement Unit (LDU) is located within the encoding stage to compensate for the shortcomings of the standard Transformer architecture in depicting local texture details. This unit compensates for abrupt changes in texture, edges, and grayscale between adjacent image patches through convolutional operations, enhancing the encoder's response to point particles, tiny pinholes, and weak boundary cracks. For example, the LDU can employ depthwise separable convolution or 3×3 small convolutional kernels to first restore the feature tokens to a two-dimensional feature map for neighborhood information extraction, and then remap the compensated features back to the token sequence. Through this structural design, the encoder can not only acquire global semantics using a self-attention mechanism but also enhance local edge responses through convolutional operations, thereby significantly improving the recognition accuracy for low-contrast defects and nanoscale targets. This step, through the coordinated efforts of various layers, achieves an organic unity between global modeling and local enhancement, effectively solving the technical challenge of missing weak defects in complex backgrounds.

[0053] Furthermore, in this embodiment, depthwise separable convolutions or local receptive field convolutions are incorporated into the local detail units to compensate for neighborhood information in the feature maps, and then the compensated features are mapped a second time to the token sequence. The neighborhood information compensation for the feature maps involves generating local feature maps through convolution operations, and then fusing these local feature maps with the original feature maps input to the unit. Specifically, in the Transformer coding block, the feature tokens are first reshaped into two-dimensional feature maps with height, width, and number of channels. Convolution operations are performed in this two-dimensional space to capture local contextual dependencies. Subsequently, the compensated feature maps are flattened and converted back into token sequences, completing the secondary mapping to the token sequence. This process achieves a closed loop from global semantic modeling to local detail enhancement and back to serialization processing.

[0054] This application achieves organic synergy between depthwise separable convolutions or local receptive field convolutions set in the local detail enhancement unit and the multi-head self-attention mechanism in the Transformer encoding stage. Based on this, the local neighborhood information compensation provided by the convolutional operation effectively solves the problem of sluggish response of the pure attention mechanism when facing low-contrast, weak-texture defects, while the design of secondary mapping to the token sequence ensures that the local enhancement information can be seamlessly integrated into the global semantic modeling process. With this hybrid architecture, the model can not only establish the dependency between the defect region and the distant background using self-attention, but also accurately lock the edge and corner features of nanoscale defects through the convolutional kernel, thereby achieving accurate identification of irregularly shaped defects in complex backgrounds.

[0055] Step S13. After fusing the multi-scale features based on the feature pyramid, fuse them with the output features of the last layer of the multi-gradient encoder to obtain fused features, and use the fused features to obtain a defect mask based on the segmentation prediction head.

[0056] In this embodiment, the feature pyramid is a network module used to fuse multi-scale features. Its function is to unify the channels and align the semantics of multi-scale features at different levels output by the encoder. Specifically, the feature pyramid first unifies the number of channels of features at different levels through 1×1 convolution or linear mapping. Then, it upsamples the deep features level by level and connects them laterally with shallow features of the same resolution, thereby transferring the strong semantic information of the deep layers to the high-resolution features of the shallow layers, generating pyramid features containing rich details and semantic information. The output feature of the last layer of the multi-gradient encoder can refer to the feature with the strongest global contextual representation ability output by the deepest stage of the encoder. Fusion can refer to performing a scale-aligned and channel-aligned addition operation on the multi-scale features processed by the feature pyramid and the output feature of the last layer of the encoder. This fusion strategy aims to combine the multi-scale detailed information provided by the feature pyramid with the pure global semantic information provided by the encoder end to form a fused feature that has both high-resolution boundary details and strong semantic discriminative power. The segmentation prediction head is a classifier containing convolutional layers and activation functions, used to receive the fused features and output pixel-level defect masks. Specifically, the segmentation prediction head performs convolutional transformations on the fused features, mapping the feature dimensions to the number of categories, and generates a probability map of each pixel belonging to the defect region through an activation function. Finally, it binarizes the data to obtain the defect mask. For example, by fusing the P2, P3, and P4 features output from the feature pyramid with the C4 feature output from the encoder, and inputting this to a prediction head consisting of two 3×3 convolutional layers and a sigmoid layer, a defect segmentation result with the same size as the input image can be obtained. Through the above fusion and prediction process, this application can effectively improve the characterization accuracy of irregularly shaped defect boundaries, ensuring that nanoscale small targets are not missed, while also guaranteeing the semantic consistency within large-scale defects.

[0057] Specifically, the multi-scale features are fused based on a feature pyramid, including: channel unification of the multi-scale features based on the feature pyramid, and horizontal fusion of deep features with shallow features at each level to obtain pyramid features.

[0058] Scale alignment refers to adjusting the spatial resolution of the feature maps to ensure that the pyramid features and the features output by the fourth encoder are consistent in height and width. Channel alignment refers to adjusting the number of channels in the feature maps through linear mapping to match the channel dimensions. Specifically, the fourth encoder, as the last layer of the multi-gradient encoder, outputs a feature denoted as C4 that contains global contextual relationships and high-level semantic information between defects and the background, but has a low spatial resolution. In contrast, the pyramid features obtained based on the feature pyramid retain rich multi-scale detail information. To effectively combine the two, the feature C4 output by the fourth encoder is first upsampled to restore its spatial size to the same resolution as the pyramid features, achieving scale alignment. Simultaneously, a 1×1 convolutional layer is used to transform the number of channels in C4, unifying its channel dimensions with those of the pyramid features. For example, if the resolution of pyramid feature P2 is H / 4×W / 4 and the number of channels is 256, while the resolution of the fourth encoder output feature C4 is H / 32×W / 32 and the number of channels is 512, then by upsampling by 8 times, the spatial size of C4 is expanded to H / 4×W / 4, and the number of channels is compressed to 256 by a 1×1 convolution, thus completing the alignment operation. This alignment mechanism eliminates the differences in spatial and channel dimensions between deep semantic features and shallow detail features, laying the data foundation for subsequent feature fusion.

[0059] Furthermore, element-wise addition can refer to performing arithmetic addition on the pixel values ​​at corresponding positions in the two feature matrices after scale alignment and channel alignment. Specifically, after completing the alignment operation, the system superimposes the output features of the fourth encoder, which contain global semantic information, with the pyramid features, which contain multi-scale detail information, at the pixel level. This operation ensures that each pixel in the fused feature map contains both the fine texture, edge, and local structural information from the feature pyramid and the strong semantic discriminative information and global contextual relationships from the fourth encoder. For example, for a nanoscale crack region with extremely low contrast in an image, the pyramid features may only capture weak grayscale changes, while the features output by the fourth encoder can confirm that the region belongs to the defect category based on the global context. After the two are added together, the feature response value of the region is significantly enhanced, thus highlighting the defect signal. The resulting fused features possess both high-resolution spatial detail and deep semantic understanding capabilities, effectively solving the problem that single-scale features cannot simultaneously achieve both the accuracy of small target localization and the discriminability of complex backgrounds, providing the optimal feature representation for the subsequent segmentation prediction head to generate high-precision defect masks.

[0060] This application achieves deep complementarity between multi-scale detail information and global semantic information by performing scale alignment, channel alignment, and element-wise addition between the pyramid features and the features output by the fourth encoder. In this process, scale alignment ensures a one-to-one correspondence in spatial location, channel alignment guarantees compatibility of feature dimensions, and element-wise addition achieves lossless information superposition with low computational cost. Leveraging the strong global context constraints provided by the fourth encoder, the model can effectively suppress false detection noise generated by complex background textures; simultaneously, utilizing the high-resolution details preserved by the feature pyramid, the model can accurately reproduce the minute shapes and irregular boundaries of nanoscale defects. This fusion strategy not only enhances the richness of feature representation but also significantly improves the model's robustness in recognizing weak-contrast defects, texture-masked defects, and irregularly shaped defects, ultimately outputting a pixel-level defect mask with complete boundaries and accurate classification.

[0061] In summary, this application provides a method for identifying nanoscale and irregularly shaped defects. By using a visual Transformer as the backbone image encoder, and considering the characteristics of nanoscale defects such as small scale, weak local texture, discontinuous boundaries, and high computational cost of high-resolution images, the encoder structure is designed with hierarchical structure and local enhancement: First, overlapping image block embedding is used to preserve the edge continuity between adjacent image blocks. Then, a multi-level Transformer encoding stage is used to extract shallow texture, mid-level morphology, and deep global semantic features, and the output features at each level are input into a feature pyramid module for fusion. Subsequently, a deformation convolution branch is set to extract defect contour features, and a segmentation output branch is set to add and fuse the feature pyramid features with the semantic features of the last layer of the encoder to output the segmentation result, thereby achieving accurate identification of nanoscale and irregularly shaped defects.

[0062] This embodiment provides a nanoscale and irregularly shaped defect identification device, which is used to perform the defect identification method described in the above embodiment. The structure of the device is as follows: Figure 3 As shown, the device 30 includes an image segmentation module 31, an encoding module 32, and a detection module 33. The modules are interconnected via a data bus or internal communication interface to collaboratively perform precision testing of semiconductor products.

[0063] Specifically, the image segmentation module 31 is used to segment an image containing the object to be detected into multiple image blocks, and convert the features of the multiple image blocks into feature tokens; the encoding module 32 is used to perform gradient encoding on the feature tokens based on a multi-gradient encoder to obtain multi-scale features; and the detection module 33 is used to fuse the multi-scale features based on a feature pyramid and then fuse them with the output features of the last layer of the multi-gradient encoder to obtain fused features, and then use the fused features to obtain a defect mask based on a segmentation prediction head.

[0064] See Figure 4 The present invention also provides a readable medium 40, which stores computer-readable instructions 401, including instructions for performing the aforementioned semiconductor chip placement position deviation detection method.

[0065] The functions and technical effects of the readable medium 40 provided in the embodiments of the present invention can be referred to the technical effects in the foregoing embodiments, and will not be repeated here.

[0066] It should be noted that, in the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and / or methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units / modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0067] The units / modules described as separate components may or may not be physically separate. The components shown as units / modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units / modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] Furthermore, in the various embodiments of the present invention, the functional units / modules can be integrated into one processing unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated into one unit / module. The integrated unit / module described above can be implemented in hardware or in the form of hardware plus software functional units / modules.

[0069] The integrated unit / module implemented as a software functional unit / module described above can be stored in a computer-readable storage medium. The software functional unit, stored in a storage medium, includes several instructions to cause one or more processors of a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of the present invention.

[0070] The integrated unit / module implemented as a software functional unit / module described above can be stored in a computer-readable storage medium. The software functional unit, stored in a storage medium, includes several instructions to cause one or more processors of a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of the present invention.

[0071] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying nanoscale and irregularly shaped defects, characterized in that, The method includes: An image containing the object to be detected is divided into multiple image blocks, and the features of the multiple image blocks are converted into feature tokens. The feature tokens are gradient encoded using a multi-gradient encoder to obtain multi-scale features; the multi-gradient encoder includes an image patch embedding layer, a position encoding layer, multiple encoding stages, and a multi-scale feature output layer. The multi-scale features are fused based on the feature pyramid and then fused with the output features of the last layer of the multi-gradient encoder to obtain fused features. The fused features are then used to obtain a defect mask based on the segmentation prediction head.

2. The method for identifying nanoscale and irregular morphological defects according to claim 1, characterized in that, Converting multiple image blocks into feature tokens includes converting the image blocks into feature tokens via linear mapping or convolutional mapping.

3. The method for identifying nanoscale and irregular morphological defects according to claim 1, characterized in that, The multi-gradient encoder includes a first encoder, a second encoder, a third encoder, and a fourth encoder. The first encoder is used to retain high-resolution edge information and texture information, the second encoder is used to retain the local structure and short-range morphological information of defects, the third encoder is used to retain high-level semantic information, and the fourth encoder is used to represent the global contextual relationship between defects and the background.

4. The method for identifying nanoscale and irregular morphological defects according to claim 3, characterized in that, The first encoder, the second encoder, the third encoder, and the fourth encoder each include a normalization layer, a multi-head self-attention layer, a residual connection layer, a feedforward network layer, and a local detail enhancement unit.

5. The method for identifying nanoscale and irregular morphological defects according to claim 3, characterized in that, The multi-gradient encoder further includes an image patch embedding layer and a position encoding layer, wherein the position encoding layer is used to add spatial position information to the feature token.

6. The method for identifying nanoscale and irregular morphological defects according to claim 4, characterized in that, The local detail enhancement unit is equipped with depthwise separable convolution or local receptive field convolution to compensate for the neighborhood information of the feature map, and then the compensated features are mapped to the token sequence again.

7. The method for identifying nanoscale and irregular morphological defects according to claim 4, characterized in that, The first encoder and the second encoder are equipped with local window self-attention or group self-attention, and the third encoder and the fourth encoder are equipped with cross-window interactive self-attention or global self-attention.

8. The method for identifying nanoscale and irregular morphological defects according to claim 3, characterized in that, The multi-scale features are fused based on a feature pyramid, including: channel unification of the multi-scale features based on the feature pyramid, and horizontal fusion of deep features with shallow features at each level to obtain pyramid features.

9. The method for identifying nanoscale and irregular morphological defects according to claim 8, characterized in that, The pyramid features and the features output by the fourth encoder are scale-aligned and channel-aligned, and then added element by element to obtain the fused features.

10. A nanoscale and irregular morphological defect identification device, characterized in that, Applications in semiconductor testing systems include: The image segmentation module is used to segment an image containing the object to be detected into multiple image blocks, and to convert the features of the multiple image blocks into feature tokens. The encoding module is used to perform gradient encoding processing on the feature token based on a multi-gradient encoder to obtain multi-scale features; The detection module is used to fuse the multi-scale features based on the feature pyramid and then fuse them with the output features of the last layer of the multi-gradient encoder to obtain fused features, and then use the fused features to obtain a defect mask based on the segmentation prediction head.

Images