Chip surface defect detection method, system and device based on improved segmentation large model

Abstract

The application relates to a chip surface defect detection method, system and equipment based on an improved segmentation large model, which comprises the following steps: performing multi-scale feature extraction on random occlusion image blocks in a chip image, performing feature compensation on feature layers of each scale, and obtaining image coding features; performing position coding and category embedding mapping on user prompt points to generate prompt coding features matched with the dimensions of the image coding features; performing space-time cross-attention interaction between the image coding features and the prompt coding features after fusion and stored short-term and long-term memory features, reconstructing an occlusion area in the chip image according to an interaction result; decoding the reconstructed image features to generate a plurality of candidate segmentation masks, and outputting a chip defect segmentation mask matched with a user intention through confidence screening and sorting. The application solves the technical problems of high-quality training data limitation, poor generalization ability and high deployment cost in a chip defect detection scheme based on deep learning.

Description

Technical Field

[0001] This invention relates to the field of chip defect detection technology, and in particular to a chip surface defect detection method, system and device based on an improved segmentation large model. Background Technology

[0002] Chip manufacturing involves complex processes such as photolithography, etching, and deposition, which can easily lead to surface defects such as scratches, particle contamination, pits, and abnormal linewidth. These defects directly affect chip performance and yield, making chip surface defect detection a crucial safeguard for chip production. Currently, chip defect detection primarily employs traditional detection methods and novel deep learning-based detection methods.

[0003] Traditional inspection methods mainly rely on optical imaging, physical property analysis, or manual interpretation, using fixed rules and manually designed features for defect detection. Typical methods include: using optical lenses to magnify the chip surface and identifying defects by comparing it with standard templates manually or through semi-automatic systems, but this is highly dependent on the operator's experience; using laser beam scanning to detect morphological anomalies by changes in reflected light intensity or phase, but this is easily affected by surface reflectivity; and using image processing algorithms to achieve automated identification, but this has poor parameter robustness and is difficult to adapt to different processes.

[0004] Deep learning-based defect detection methods automatically extract defect features using neural networks. By training models with large amounts of labeled data, they can effectively capture complex features and hidden patterns that are difficult to define using traditional methods. However, existing neural network-based detection methods still have several limitations, including dependence on training data quality, insufficient generalization ability for defect types, and hardware resource constraints, making it difficult to achieve comprehensive application in certain practical industrial scenarios. For example, the detection performance of convolutional neural networks is highly dependent on the labeling quality of the training dataset; generative adversarial networks tend to generate a certain type of common defect, resulting in a lack of generalization ability in generating and detecting diverse defects; and in large-scale model applications, the extremely high memory consumption and computational resource requirements severely limit the feasibility of deployment on edge devices or low-configuration hardware environments.

[0005] Therefore, those skilled in the art urgently need a chip defect detection method that combines less reliance on training data, stronger generalization ability, and lower deployment cost. Summary of the Invention

[0006] (a) Technical problems to be solved In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a chip surface defect detection method, system and device based on an improved segmentation large model, which solves the technical problems of high-quality training dataset limitation, poor generalization ability and high deployment cost in the existing deep learning chip defect detection methods.

[0007] (II) Technical Solution To achieve the above objectives, the main technical solutions adopted by the present invention include: In a first aspect, embodiments of the present invention provide a chip surface defect detection method based on an improved segmentation large model, comprising: Using an image encoder based on a simplified hierarchical structure, multi-scale feature extraction is performed on randomly occluded image blocks in the input chip image. Image coding features are obtained by performing cross-scale and hierarchical feature compensation on feature layers at each scale. The input user cue points segmented by guidance are positionally encoded and category embedded to generate cue encoding features that match the dimensions of image encoding features; After fusing image coding features with cue coding features, spatiotemporal cross-attention interaction is performed with stored short-term and long-term memory features, and the occluded areas in the chip image are reconstructed based on the interaction results. A lightweight bidirectional decoder is used to decode the features of the reconstructed image to generate multiple candidate segmentation masks. After confidence screening and sorting, a chip defect segmentation mask matching the user's intent is output.

[0008] Optionally, before using an image encoder based on a simplified hierarchical structure to extract multi-scale features from randomly occluded image patches in the input chip image, and obtaining image coding features by performing cross-scale and hierarchical feature compensation on each scale feature layer, the method further includes: The obtained basic segmentation model is improved by removing the fusion layer from the image encoder and introducing a spatiotemporal cross-attention module to obtain the initial segmentation model. Using the image encoder and cue encoder in the initial segmentation large model, the preset chip defect images are traversed to encode defect features, and the generated defect features are stored in memory through the spatiotemporal cross attention module. The confidence analysis of defect features is performed using the decoder in the initial segmentation model. The mask loss is calculated based on the predicted segmentation mask determined by the confidence analysis results and the actual defect region annotation of the chip defect image to obtain the mask loss value. Using the mask loss value as a guide for decoder lightweighting, and combining it with preset model parameters based on Hessian matrix quantization, the decoder of the initial large segmentation model is lightweighted to obtain the optimal large segmentation model.

[0009] Optionally, using the image encoder and cue encoder in the initial segmentation large model, the preset chip defect images are traversed to encode defect features, and the generated defect features are stored in memory through a spatiotemporal cross-attention module, including: The obtained chip defect images are preprocessed by scaling, filling blank areas, and annotating defect areas to obtain a chip defect image training set. Random location points are extracted from each labeled defect region in the chip defect image training set to obtain a training cue point set; The image encoder is used to extract features from the randomly occluded image blocks in the chip defect image, and the initial defect image features are dynamically sparsed according to the preset sparse weight matrix to obtain the defect image features. The training cue points corresponding to the defect feature image are input into the cue encoder, so that the training cue point encoding and category embedding are mapped to the defect image feature dimension through the position encoding module to obtain the training cue features; The defect image features are fused with the training cue features, and the generated defect features are stored in memory through an empty cross-attention module.

[0010] Optionally, the decoder in the initial segmentation model is used to perform confidence analysis on the defect features, and the mask loss is calculated based on the predicted segmentation mask determined by the confidence analysis results and the actual defect region annotation of the chip defect image. The mask loss values ​​include: The defect features are split into dual-sequence features that are queries and key-value pairs in the spatial dimension. The attention enhancement of the dual-sequence features is performed through a spatiotemporal cross-attention module to enhance bidirectional information interaction and fusion, thereby generating multiple candidate prediction segmentation masks. The candidate predicted segmentation masks are sorted based on their confidence scores, and the candidate predicted segmentation mask with the highest confidence score is determined as the predicted segmentation mask for chip defects. The mask loss is calculated by comparing the predicted segmentation mask with the actual defect region annotations in the chip defect image, resulting in the mask loss value used for initial segmentation large model training.

[0011] Optionally, an image encoder based on a simplified hierarchical structure is used to extract multi-scale features from randomly occluded image patches in the input chip image, and the image coding features are obtained by performing cross-scale and hierarchical feature compensation on each scale feature layer, including: Based on the self-supervised pre-training strategy, occluded image patches are randomly sampled from the input chip image, and the occluded image patches are positionally encoded. Multi-scale feature extraction is performed on occluded image patches using multi-branch convolutional structures and deformable convolutional operations to obtain multi-scale initial feature layers. The initial feature layer is modeled in a global context, and redundant features are pruned using a dynamic sparse pruning mechanism to retain key features. Based on the feature compensation module, cross-scale feature interaction and hierarchical feature compensation are fused and encoded on the feature layers of each scale after pruning to obtain image coding features containing local detail information and global structural information. The feature compensation module generates compensation features based on the preset label image, and then fits the compensation features to the feature layer of the corresponding scale using a two-dimensional Gaussian fit.

[0012] Optionally, the input guided segmentation user cue points are subjected to location encoding and category embedding mapping to generate cue encoding features that match the image encoding feature dimensions, including: The coordinate information of the input user prompt point is positionally encoded to obtain a position embedding vector that matches the image encoding feature dimension; Map the type labels of user prompts to category embedding vectors that match the dimensions of image coding features; The location embedding vector and the category embedding vector are concatenated and interacted with the image coding features across modalities to generate cue coding features that are spatially aware and semantically consistent.

[0013] Optionally, after fusing image encoding features and cue encoding features, spatiotemporal cross-attention interaction is performed with stored short-term and long-term memory features, and the occluded region in the chip image is reconstructed based on the interaction result, including: Image coding features and cue coding features are aligned and fused along the channel dimension to obtain fused features for guiding the decoder's mask segmentation; The fused features are stored in a memory library, which includes a first queue for real-time memory storage of fused features and a second queue for storing historical prompt frames obtained from user interaction. The first and second queues are dynamically updated using a first-in-first-out strategy, and the features in the queues are stored in a weighted manner based on the configured feature importance level. Temporal and positional encoding is performed on the local features in the current fused features and the corresponding features in the first N frames of the first queue to obtain short-term motion trajectory prediction; Retrieve historical prompt frames from the second queue whose semantic similarity to the current fused feature scene reaches a preset threshold, and use the retrieved features as long-term memory references; Based on short-term motion trajectory prediction and long-term memory reference, a spatial transformation network is used to perform multi-scale feature compensation and fusion on the occluded area to obtain a chip image of the reconstructed occluded area.

[0014] Optionally, a lightweight bidirectional decoder is used to decode the reconstructed image features to generate multiple candidate segmentation masks. These masks are then filtered and sorted based on confidence levels to output a chip defect segmentation mask that matches the user's intent. Image features are split into dual-sequence features that are queries and key-value pairs in the spatial dimension, and attention enhancement is performed on the dual-sequence features for bidirectional information interaction and fusion to generate multiple candidate segmentation masks. A confidence analysis is performed on all candidate segmentation masks. The candidate segmentation masks are sorted according to the obtained confidence scores, and the candidate segmentation mask with the highest confidence score is selected as the optimal output to obtain a chip defect segmentation mask that matches the user's intent.

[0015] In a second aspect, embodiments of the present invention provide a chip surface defect detection system, comprising: The image feature encoding module is used to extract multi-scale features from randomly occluded image blocks in the input chip image using an image encoder based on a simplified hierarchical structure, and to obtain image encoded features by performing cross-scale and hierarchical feature compensation on feature layers at each scale. The prompt feature encoding module is used to perform position encoding and category embedding mapping on the input guided segmentation user prompt points to generate prompt encoding features that match the image encoding feature dimensions; The feature fusion and image occlusion reconstruction module is used to fuse image coding features and cue coding features, perform spatiotemporal cross-attention interaction with stored short-term and long-term memory features, and reconstruct the occluded area in the chip image based on the interaction result. The mask decoding module is used to decode the reconstructed image features using a lightweight bidirectional decoder to generate multiple candidate segmentation masks, and outputs a chip defect segmentation mask that matches the user's intent by filtering and sorting based on confidence.

[0016] Thirdly, embodiments of the present invention provide a chip surface defect detection device, comprising: The light source is configured to illuminate the chip surface at a preset wavelength and incident angle; Image sensors are used to capture light signals reflected from the surface of a chip and generate images of the chip. A chip surface defect detection component is used to perform the chip surface defect detection method based on the improved segmentation large model described above on the chip image.

[0017] (III) Beneficial Effects This invention effectively enhances the model's ability to learn and reconstruct features under limited training data by introducing a multi-scale feature extraction and cross-scale feature compensation mechanism for randomly occluded image patches, and by combining the interaction of short-term and long-term memory features, thereby reducing the model's dependence on large-scale, high-quality labeled data for training.

[0018] The multi-scale feature extraction and compensation, spatiotemporal cross-attention and other mechanisms employed in this invention enable the model to better understand defect features of different scales and forms, thereby exhibiting stronger adaptability and robustness in the identification of unseen defect types.

[0019] This invention employs a simplified hierarchical image encoder and a lightweight bidirectional decoder, which reduces the complexity and number of parameters of the model. At the same time, it avoids unnecessary calculations through a confidence screening and sorting mechanism, making the final model less demanding on hardware computing power and improving the deployment feasibility of edge devices or low-configuration hardware environments. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a chip surface defect detection method based on an improved segmentation large model, as provided in an embodiment of the present invention. Figure 2 This is a flowchart of feature fusion coding provided in an embodiment of the present invention; Figure 3 This is a flowchart of feature decoding provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of feature layer fusion performed by a simplified hierarchical image encoder according to an embodiment of the present invention. Detailed Implementation

[0021] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] refer to Figure 1 As shown in the embodiment of the present invention, a chip surface defect detection method based on an improved segmentation large model is proposed, which includes: First, using an image encoder based on a simplified hierarchical structure, multi-scale feature extraction is performed on randomly occluded image blocks in the input chip image, and image coding features are obtained by performing cross-scale and hierarchical feature compensation on each scale feature layer; Next, the input user prompt points for guided segmentation are positionally encoded and category embedded and mapped to generate prompt coding features that match the dimension of the image coding features; Then, the image coding features and prompt coding features are fused and subjected to spatiotemporal cross-attention interaction with stored short-term and long-term memory features, and the occluded region in the chip image is reconstructed based on the interaction result; Finally, a lightweight bidirectional decoder is used to decode the reconstructed image features to generate multiple candidate segmentation masks, and the chip defect segmentation mask that matches the user intent is output by filtering and sorting by confidence.

[0023] This embodiment effectively enhances the model's ability to learn and reconstruct features with limited training data by introducing multi-scale feature extraction and cross-scale feature compensation mechanisms for randomly occluded image patches, combined with the interaction of short-term and long-term memory features. This reduces the model's dependence on large-scale, high-quality labeled data for training. The multi-scale feature extraction and compensation, spatiotemporal cross-attention, and other mechanisms employed in this embodiment enable the model to better understand defect features of different scales and forms, thus exhibiting stronger adaptability and robustness in identifying unseen defect types. Furthermore, this embodiment uses a simplified hierarchical image encoder and a lightweight bidirectional decoder to reduce model complexity and parameter count, while avoiding unnecessary computation through confidence filtering and ranking mechanisms. This results in lower hardware computing power requirements for the final model, improving deployment feasibility on edge devices or low-configuration hardware environments.

[0024] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0025] Specifically, refer to Figure 1 As shown, this embodiment discloses a chip surface defect detection method based on an improved segmentation large model, the method steps of which may include the following steps S100 to S400: S100. Using an image encoder based on a simplified hierarchical structure, multi-scale feature extraction is performed on randomly occluded image blocks in the input chip image. Image coding features are obtained by performing cross-scale and hierarchical feature compensation on feature layers at each scale.

[0026] In this embodiment, the network structure of the segmentation model is optimized by removing the simplified hierarchical structure of the image encoder from the fusion layer, enabling deployment with limited hardware resources. Simultaneously, a hierarchical compensation structure is employed to enhance the robustness and completeness of feature representations at each scale, thereby compensating for the feature loss caused by the reduced fusion layer and significantly improving the accuracy and generalization ability of subsequent chip defect detection. Furthermore, this embodiment effectively alleviates the problem of local information loss caused by random occlusion in chip images by introducing a cross-scale feature fusion mechanism.

[0027] In this embodiment, before executing steps S100 to S400, it is necessary to fine-tune and improve the basic segmentation model retrieved from the database and train the model. Specific steps may include G100 to G400: G100: Improve the obtained basic segmentation model by removing the fusion layer from the image encoder and introducing a spatiotemporal cross-attention module to obtain the initial segmentation model.

[0028] Furthermore, this example employs a simplified hierarchical image encoder structure by removing the fusion layer from the image encoder, enabling the model to be deployed with limited hardware resources. The spatiotemporal cross-attention module utilizes memory and cross-attention mechanisms, allowing the current fused features to interact with short-term and long-term memory features. This enables the model to better understand defect features of different scales and forms, thus exhibiting stronger adaptability and robustness in identifying unseen defect types.

[0029] To further explain, the memory mechanism consists of three parts: a memory encoder, which downsamples the mask using a convolutional module and adds it element-wise to the feature map generated by the image encoder, then further fuses the information through a lightweight convolutional layer; a memory bank, which uses two first-in-first-out queues: a first queue for real-time storage of the fused features of the most recent N frames and a second queue for storing historical prompt frames obtained from user interactions. The first and second queues are dynamically updated using a first-in-first-out strategy, and the features in the queues are weighted based on the configured feature importance level to ensure long-term context consistency; and a memory attention module, which performs temporal and positional encoding on the local features in the current fused features and the corresponding features of the previous N frames in the first queue to generate short-term motion trajectory predictions, and retrieves historical prompt frames from the second queue whose scene semantic similarity to the current fused features reaches a preset threshold, so that the retrieved features can be used as long-term memory references.

[0030] Cross-attention mechanism is used to enhance attention by performing bidirectional information interaction and fusion on the reconstructed image features. It can simultaneously update prompt information and image features to generate a segmentation mask that matches the user's intent.

[0031] G200 utilizes the image encoder and cue encoder in the initial segmentation large model to traverse the preset chip defect images to encode defect features, and stores the generated defect features through the spatiotemporal cross-attention module.

[0032] Further, refer to Figure 2 As shown, step G200 may include sub-steps G210 to G250: G210. The obtained chip defect images are preprocessed by scaling, filling blank areas, and annotating defect areas to obtain a chip defect image training set.

[0033] G220: Extract random location points from each labeled defect region in the chip defect image training set to obtain a training cue point set.

[0034] G230: Use an image encoder to extract features from randomly occluded image blocks in a chip defect image, and dynamically sparse the initial defect image features according to a preset sparse weight matrix to obtain the defect image features.

[0035] Furthermore, randomly occluded image patches are input into the image feature encoder to extract image features. A sparse weight matrix is ​​dynamically generated based on the weight saliency measure (gradient magnitude) to dynamically sparse the feature map through a dynamic sparse pruning mechanism. In this process, non-significant weights are forced to zero, while significant weights continue to participate in the forward and backward propagation processes.

[0036] G240. Input the training cue points corresponding to the defect feature image into the cue encoder, so that the training cue point encoding and category embedding are mapped to the defect image feature dimension through the position encoding module to obtain the training cue features.

[0037] Furthermore, the training cue points corresponding to the defect feature image are input into the cue encoder. The coordinates of the points are converted into embedding vectors through position encoding, and then mapped to a unified 256-dimensional embedding space. The label type information is mapped to a unified 256-dimensional embedding feature, which facilitates fusion with the image features (64×64×256) output by the image encoder.

[0038] G250 performs feature fusion between defect image features and training prompt features, and stores the generated defect features through an empty cross-attention module.

[0039] G300 uses the decoder in the initial segmentation model to perform confidence analysis on the defect features, and calculates the mask loss based on the predicted segmentation mask determined by the confidence analysis results and the actual defect area annotation of the chip defect image to obtain the mask loss value.

[0040] Further, refer to Figure 3 As shown, step G300 may include sub-steps G310 to G330: G310 decomposes defect features into dual-sequence features that are queries and key-value pairs in the spatial dimension, and enhances the attention of the dual-sequence features through bidirectional information interaction and fusion using a spatiotemporal cross-attention module to generate multiple candidate prediction segmentation masks.

[0041] For example, defect features can be split into two sequences. Features from sequence A can be used as the query vector, and features from sequence B as the key and value vectors. Alternatively, features from sequence B can be used as the query vector, and features from sequence A as the key and value vectors. The attention outputs from both directions are then concatenated and integrated to form a bidirectionally enhanced joint representation. This design can simultaneously update prompts and image features, generating a segmentation mask that matches the user's intent. For instance, when the data source is a video stream, the decoder interacts with the features of the current frame and historical memories in the memory bank to enhance spatiotemporal consistency.

[0042] G320 sorts the candidate predicted segmentation masks based on their confidence levels and determines the candidate predicted segmentation mask with the highest confidence level as the predicted segmentation mask for chip defects.

[0043] G330: Perform mask loss calculation on the predicted segmentation mask and the actual defect region annotation of the chip defect image to obtain the mask loss value used for initial segmentation large model training.

[0044] To further explain, calculating the loss of the highest-scoring mask aims to quantify the difference between the model's predictions and the ground truth annotations, and to guide model parameter optimization through gradient backpropagation, thus achieving lightweight guidance for the decoder. Comparing the predicted mask and the ground truth mask, the cross-entropy loss is calculated using the ground truth mask (actual defect region annotations) and the predicted probability map: ; In the formula, N represents the number of sample points, M represents the number of categories, and y ic The sign function (0 or 1) is used, taking the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic This represents the probability that observed sample i belongs to category c.

[0045] G400 uses the mask loss value as a guide for decoder lightweighting. Combined with the preset model parameters based on Hessian matrix quantization, the decoder of the initial large segmentation model is lightweighted to obtain the optimal large segmentation model.

[0046] Furthermore, using the mask loss result as a guide for decoder lightweighting, 8-bit quantization technology is adopted to lightweight the decoder of the segmentation large model. It is based on the Hessian matrix quantization parameter to minimize the task loss and ensure the stable performance of the lightweight model.

[0047] In this embodiment, step S100 may include the following sub-steps S110 to S140: S110. Based on the self-supervised pre-training strategy, randomly sample occluded image blocks from the input chip image and encode the positions of the occluded image blocks.

[0048] Furthermore, after scaling and normalizing the input chip image, a self-supervised pre-training strategy is used to randomly extract occluded image patches from the processed chip image and encode the position of the occluded image patch. This enables the asymmetric image encoder and decoder to reconstruct the occluded region, enhancing the model's ability to learn defect features with limited training data and reducing the dependence of large segmentation models on high-quality training data.

[0049] S120. Multi-scale feature extraction is performed on occluded image patches using a multi-branch convolutional structure and deformable convolutional operations to obtain an initial feature layer at multiple scales.

[0050] S130. Perform global context modeling on the initial feature layer, and prune the identified redundant features through a dynamic sparse pruning mechanism to retain key features.

[0051] S140. Based on the feature compensation module, perform cross-scale feature interaction and hierarchical feature compensation fusion encoding on the pruned feature layers at each scale to obtain image coding features containing local detail information and global structural information.

[0052] Furthermore, the feature compensation module generates compensation features based on the preset label image, and then fits the compensation features to the feature layer of the corresponding scale using a two-dimensional Gaussian fit. (Reference) Figure 4 As shown, since the image encoder eliminates the fusion layer structure, when operating on the 1 / 4, 1 / 8, and 1 / 16 scale features in feature encoding, this embodiment proposes a compensatory feature fusion method to compensate for the feature loss caused by the fusion layer. First, based on the label image predefined by the user in conjunction with the characteristics of the chip to be detected, compensatory features associated with the chip to be detected are generated. Then, the compensatory features are fitted to the feature layer of the corresponding scale using a two-dimensional Gaussian, and cross-scale feature interaction results are performed based on the feature layers of each scale to obtain image encoding features containing local detail information and global structural information. This example uses an image encoder based on a simplified hierarchical structure to complete the encoding of the feature layer. Through cross-scale feature fusion and hierarchical feature fusion mechanisms, the network encoding capability is improved, and the encoding speed is accelerated while improving performance.

[0053] S200: Perform position encoding and category embedding mapping on the input user cue points segmented by the guidance to generate cue encoding features that match the image encoding feature dimensions.

[0054] In this embodiment, the prompt encoder uses points as input prompts. Since user prompt points are used to assist in localization, they need to be fused with feature information in the network. This requires mapping the user prompt point dimension to the image feature dimension. First, the coordinates of the points are converted into embedding vectors through position encoding. Then, the label type information of the user prompt points is mapped to a unified 256-dimensional embedding feature. Finally, the position encoding information of the user prompt points and the category encoding information of the labels are fused to form the prompt encoder features.

[0055] In this embodiment, step S200 may include the following sub-steps S210 to S230: S210. Perform position encoding on the coordinate information of the input user prompt point to obtain a position embedding vector that matches the dimension of the image encoding features.

[0056] S220. Map the type label of the user prompt point to a category embedding vector that matches the dimension of the image encoding features.

[0057] S230. Concatenate the location embedding vector and the category embedding vector, and perform cross-modal feature interaction with the image coding features to generate cue coding features with spatial awareness and semantic consistency.

[0058] S300: After fusing the image coding features and the prompt coding features, perform spatiotemporal cross-attention interaction with the stored short-term and long-term memory features, and reconstruct the occluded area in the chip image based on the interaction results.

[0059] In this embodiment, the encoded features of the occluded image patch are fused with the coded features of the cue indicating the occluded region. Subsequently, this example innovatively introduces short-term and long-term memories, and through a spatiotemporal cross-attention mechanism, enables deep interaction between the fused features and the memory features. This mechanism can associate and match the most relevant standard structural knowledge from long-term memory and reference temporal context information from short-term memory, thereby accurately inferring the content that the occluded region should contain. Finally, the repaired feature image is reconstructed based on the interaction results. This embodiment effectively utilizes prior knowledge and context, overcoming the ambiguity or structural errors that are easily caused by traditional methods, and significantly improving the realism, accuracy, and reliability of the reconstructed image results in complex industrial scenarios.

[0060] In this embodiment, step S300 may include the following sub-steps S310 to S350: S310. Align and fuse the image coding features and the cue coding features in the channel dimension to obtain the fused features used to guide the decoder mask segmentation.

[0061] S320. The fused features are stored in a memory library, which includes a first queue for real-time memory storage of fused features and a second queue for storing historical prompt frames obtained from user interaction. The first and second queues are dynamically updated using a first-in-first-out strategy, and the features in the queues are stored in a weighted manner based on the configured feature importance level.

[0062] S330. Perform time-position encoding on the local features in the current fused features and the corresponding features in the first N frames of the first queue to obtain short-term motion trajectory prediction.

[0063] S340. Retrieve historical prompt frames from the second queue whose semantic similarity to the current fused feature scene reaches a preset threshold, and use the retrieved features as long-term memory references.

[0064] S350, based on short-term motion trajectory prediction and long-term memory reference, performs multi-scale feature compensation and fusion on the occluded area through a spatial transformation network to obtain a chip image of the reconstructed occluded area.

[0065] The S400 uses a lightweight bidirectional decoder to decode the features of the reconstructed image to generate multiple candidate segmentation masks. After confidence screening and sorting, it outputs a chip defect segmentation mask that matches the user's intent.

[0066] In this embodiment, step S400 may include the following sub-steps S410 to S420: S410. The image features are split into dual-sequence features that are query and key-value pairs in the spatial dimension, and attention enhancement is performed on the dual-sequence features for bidirectional information interaction and fusion to generate multiple candidate segmentation masks.

[0067] Furthermore, this embodiment processes the fused features through a bidirectional attention mechanism. This bidirectional attention mechanism not only improves the completeness of feature representation but also significantly enhances the model's ability to perceive defect edges and fine structures, thereby generating high-quality and diverse candidate segmentation results even in complex backgrounds.

[0068] S420. Perform confidence analysis on all candidate segmentation masks, sort the candidate segmentation masks according to the obtained confidence scores, select the candidate segmentation mask with the highest confidence score as the optimal output, and obtain the chip defect segmentation mask that matches the user's intention.

[0069] Furthermore, this embodiment employs a confidence ranking mechanism to effectively avoid false detections and missed detections, thereby improving the reliability and robustness of the segmentation results and making them more in line with the needs of high-precision defect detection in practical applications.

[0070] Furthermore, this embodiment also proposes a chip surface defect detection system, which includes at least the following modules: The image feature encoding module is used to extract multi-scale features from randomly occluded image blocks in the input chip image using an image encoder based on a simplified hierarchical structure, and to obtain image encoded features by performing cross-scale and hierarchical feature compensation on feature layers at each scale.

[0071] The prompt feature encoding module is used to perform position encoding and category embedding mapping on the user prompt points of the input guidance segmentation to generate prompt encoding features that match the image encoding feature dimensions.

[0072] The feature fusion and image occlusion reconstruction module is used to fuse image-encoded features and cue-encoded features, perform spatiotemporal cross-attention interaction with stored short-term and long-term memory features, and reconstruct the occluded areas in the chip image based on the interaction results.

[0073] The mask decoding module is used to decode the reconstructed image features using a lightweight bidirectional decoder to generate multiple candidate segmentation masks, and outputs a chip defect segmentation mask that matches the user's intent by filtering and sorting based on confidence.

[0074] Finally, this embodiment also proposes a chip surface defect detection device, which includes: a light source, an image sensor, and a chip surface defect detection component. The light source is configured to illuminate the chip surface at a preset wavelength and incident angle. The image sensor is used to capture the reflected light signal from the chip surface and generate a chip image. The chip surface defect detection component is used to perform the chip surface defect detection method based on the improved segmentation large model described above on the chip image.

[0075] In summary, this embodiment proposes a chip surface defect detection method, system, and device based on an improved large-scale segmentation model. This embodiment uses a large-scale segmentation model as a foundation, fine-tuning a small sample dataset for chip defect detection to achieve good segmentation and detection results. This embodiment also utilizes prior knowledge of chip defects, selecting partial feature layer fusion during feature layer fusion and progressively compressing feature dimensions during feature decoding. This lightweights the model's network structure without sacrificing accuracy, enabling deployment on edge devices or low-configuration hardware environments. Finally, the improved large-scale segmentation model is a small neural network that projects features extracted from the backbone network to a lower dimension and then back to the original dimension, achieving fine-tuning of features. Furthermore, this embodiment improves fine-tuning speed by freezing the basic model parameters, allowing the model's adapter module to quickly adapt to new tasks and data distributions, eliminating the need for complex optimization of the entire model. This significantly shortens the model's inference time and improves the feasibility of deploying the model on mobile devices.

[0076] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0079] It should be noted that in the description of this invention, the word "a" or "an" preceding a component does not exclude the existence of multiple such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. The use of terms such as first, second, third, etc., is merely for convenience and does not indicate any order. These terms can be understood as part of the component names.

[0080] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0081] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning of the basic inventive concept, can make other changes and modifications to these embodiments.

[0082] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of the invention.

Claims

1. A chip surface defect detection method based on an improved segmentation large model, characterized in that, include: Using an image encoder based on a simplified hierarchical structure, multi-scale feature extraction is performed on randomly occluded image patches in the input chip image. Image coding features are obtained by performing cross-scale and hierarchical feature compensation on feature layers at each scale. The simplified hierarchical structure includes removing the fusion layer in the image encoder and introducing a spatiotemporal cross-attention module. The input user cue points segmented by guidance are positionally encoded and category embedded to generate cue encoding features that match the dimensions of image encoding features; After fusing image-encoded features and cue-encoded features, spatiotemporal cross-attention interaction is performed with stored short-term and long-term memory features, and the occluded region in the chip image is reconstructed based on the interaction result. This includes: aligning and fusing image-encoded features and cue-encoded features along the channel dimension to obtain fused features for guiding decoder mask segmentation; storing the fused features in a memory library, which includes a first queue for real-time storage of the fused features and a second queue for storing historical cue frames obtained from user interaction. The first and second queues are dynamically updated using a first-in-first-out strategy, and the features in the queues are weighted based on the configured feature importance level; performing temporal and positional encoding on the local features in the current fused features and the corresponding features in the first N frames to obtain short-term motion trajectory prediction; retrieving historical cue frames from the second queue whose scene semantic similarity to the current fused features reaches a preset threshold, and using the retrieved features as long-term memory references; and performing multi-scale feature compensation and fusion on the occluded region through a spatial transformation network based on the short-term motion trajectory prediction and long-term memory references to obtain a reconstructed chip image of the occluded region. A lightweight bidirectional decoder is used to decode the features of the reconstructed image to generate multiple candidate segmentation masks. After confidence screening and sorting, a chip defect segmentation mask matching the user's intent is output.

2. The method as described in claim 1, characterized in that, Before using an image encoder based on a simplified hierarchical structure to extract multi-scale features from randomly occluded image patches in the input chip image, and obtaining image coding features by performing cross-scale and hierarchical feature compensation on each scale feature layer, the following steps are also included: The obtained basic segmentation model is improved by removing the fusion layer from the image encoder and introducing a spatiotemporal cross-attention module to obtain the initial segmentation model. Using the image encoder and cue encoder in the initial segmentation large model, the preset chip defect images are traversed to encode defect features, and the generated defect features are stored in memory through the spatiotemporal cross attention module. The confidence analysis of defect features is performed using the decoder in the initial segmentation model. The mask loss is calculated based on the predicted segmentation mask determined by the confidence analysis results and the actual defect region annotation of the chip defect image to obtain the mask loss value. Using the mask loss value as a guide for decoder lightweighting, and combining it with preset model parameters based on Hessian matrix quantization, the decoder of the initial large segmentation model is lightweighted to obtain the optimal large segmentation model.

3. The method as described in claim 2, characterized in that, Using the image encoder and cue encoder in the initial segmentation model, the preset chip defect images are traversed to encode defect features, and the generated defect features are stored in memory through a spatiotemporal cross-attention module, including: The obtained chip defect images are preprocessed by scaling, filling blank areas, and annotating defect areas to obtain a chip defect image training set. Random location points are extracted from each labeled defect region in the chip defect image training set to obtain a training cue point set; The image encoder is used to extract features from randomly occluded image blocks in the chip defect image, and the initial defect image features are dynamically sparsed according to a preset sparse weight matrix to obtain the defect image features. The training cue points corresponding to the defect feature image are input into the cue encoder, so that the training cue point encoding and category embedding are mapped to the defect image feature dimension through the position encoding module to obtain the training cue features; The defect image features are fused with the training cue features, and the generated defect features are stored in memory through an empty cross-attention module.

4. The method as described in claim 2, characterized in that, The decoder in the initial segmentation model is used to perform confidence analysis on the defect features. Based on the predicted segmentation mask determined by the confidence analysis results and the actual defect region annotations in the chip defect image, the mask loss is calculated, yielding the following mask loss values: The defect features are split into dual-sequence features that are queries and key-value pairs in the spatial dimension. The attention enhancement of the dual-sequence features is performed through a spatiotemporal cross-attention module to enhance bidirectional information interaction and fusion, thereby generating multiple candidate prediction segmentation masks. The candidate predicted segmentation masks are sorted based on their confidence scores, and the candidate predicted segmentation mask with the highest confidence score is determined as the predicted segmentation mask for chip defects. The mask loss is calculated by comparing the predicted segmentation mask with the actual defect region annotations in the chip defect image, resulting in the mask loss value used for initial segmentation large model training.

5. The method as described in claim 1, characterized in that, Using an image encoder based on a simplified hierarchical structure, multi-scale feature extraction is performed on randomly occluded image patches in the input chip image. Through cross-scale and hierarchical feature compensation of each scale feature layer, the image encoded features are obtained, including: Based on the self-supervised pre-training strategy, occluded image patches are randomly sampled from the input chip image, and the occluded image patches are positionally encoded. Multi-scale feature extraction is performed on occluded image patches using multi-branch convolutional structures and deformable convolutional operations to obtain multi-scale initial feature layers. The initial feature layer is modeled in a global context, and redundant features are pruned using a dynamic sparse pruning mechanism to retain key features. Based on the feature compensation module, cross-scale feature interaction and hierarchical feature compensation are fused and encoded on the pruned feature layers at each scale to obtain image coding features containing local detail information and global structural information. The feature compensation module generates compensation features based on the preset label image, and then fits the compensation features to the feature layer of the corresponding scale using a two-dimensional Gaussian fit.

6. The method as described in claim 1, characterized in that, The input user cue points, segmented by guidance, are subjected to location encoding and category embedding mapping to generate cue encoding features that match the dimensions of the image encoding features, including: The coordinate information of the input user prompt point is positionally encoded to obtain a position embedding vector that matches the image encoding feature dimension; Map the type labels of user prompts to category embedding vectors that match the dimensions of image coding features; The location embedding vector and the category embedding vector are concatenated and interacted with the image coding features across modalities to generate cue coding features that are spatially aware and semantically consistent.

7. The method as described in claim 1, characterized in that, A lightweight bidirectional decoder is used to decode the reconstructed image features to generate multiple candidate segmentation masks. These masks are then filtered and sorted based on confidence levels to output chip defect segmentation masks that match the user's intent. Image features are split into dual-sequence features that are queries and key-value pairs in the spatial dimension, and attention enhancement is performed on the dual-sequence features for bidirectional information interaction and fusion to generate multiple candidate segmentation masks. A confidence analysis is performed on all candidate segmentation masks. The candidate segmentation masks are sorted according to the obtained confidence scores, and the candidate segmentation mask with the highest confidence score is selected as the optimal output to obtain a chip defect segmentation mask that matches the user's intent.

8. A chip surface defect detection system, characterized in that, include: The image feature encoding module is used to extract multi-scale features from randomly occluded image patches in the input chip image using an image encoder based on a simplified hierarchical structure, and to obtain image encoded features by performing cross-scale and hierarchical feature compensation on each scale feature layer; wherein, the simplified hierarchical structure includes removing the fusion layer in the image encoder and introducing a spatiotemporal cross-attention module; The prompt feature encoding module is used to perform position encoding and category embedding mapping on the input guided segmentation user prompt points to generate prompt encoding features that match the image encoding feature dimensions; The feature fusion and image occlusion reconstruction module is used to fuse image-encoded features and prompt-encoded features, perform spatiotemporal cross-attention interaction with stored short-term and long-term memory features, and reconstruct the occluded region in the chip image based on the interaction result. This includes: aligning and fusing image-encoded features and prompt-encoded features along the channel dimension to obtain fused features for guiding decoder mask segmentation; storing the fused features in a memory library, which includes a first queue for real-time storage of fused features and a second queue for storing historical prompt frames obtained from user interaction. The first and second queues are dynamically updated using a first-in-first-out strategy, and the features in the queues are weighted based on the configured feature importance level; performing temporal and positional encoding on the local features in the current fused features and the corresponding features in the first N frames to obtain short-term motion trajectory prediction; retrieving historical prompt frames from the second queue whose scene semantic similarity to the current fused features reaches a preset threshold, and using the retrieved features as long-term memory references; and performing multi-scale feature compensation and fusion on the occluded region through a spatial transformation network based on the short-term motion trajectory prediction and long-term memory references to obtain a reconstructed chip image of the occluded region. The mask decoding module is used to decode the reconstructed image features using a lightweight bidirectional decoder to generate multiple candidate segmentation masks, and outputs a chip defect segmentation mask that matches the user's intent by filtering and sorting based on confidence.

9. A chip surface defect detection device, characterized in that, include: The light source is configured to illuminate the chip surface at a preset wavelength and incident angle; Image sensors are used to capture light signals reflected from the surface of a chip and generate images of the chip. A chip surface defect detection component is used to perform chip surface defect detection based on an improved segmentation large model as described in any one of claims 1-7, based on a chip image.

Images