Chip image defect segmentation method based on improved SegFormer

Abstract

The invention discloses a chip image defect segmentation method based on improved SegFormer. The method comprises the following steps: constructing a chip surface defect image semantic segmentation data set; a SegFormer semantic segmentation network is improved, the self-attention of Transform Block in an encoder is replaced with cross covariance attention CrossCovAttn, a decoder is changed to be of a U-shaped structure, and an AFF dynamic feature fusion module is added; training is carried out to obtain an improved SegFormer semantic segmentation model; adjusting model training parameters to obtain an optimization model; and selecting an improved SegFormer semantic segmentation model with optimal performance, and carrying out semantic segmentation on the chip surface defect image to obtain a semantic segmentation result image. According to the method, the capability of processing the defects of different types and scales is enhanced, the segmentation effect of the SegFormer network is optimized, and the segmentation precision of the defects of different types and scales is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image or video recognition or understanding, and in particular, relates to a chip image defect segmentation method based on an improved SegFormer network. Background Art

[0002] As a core component of modern electronic devices, chip quality control in the manufacturing process is crucial. Tiny defects on the surface of a chip may seriously affect its performance. However, the variety of surface defects and the large differences in shapes and sizes make traditional manual detection methods inefficient and have high false alarm and missed alarm rates. Early image segmentation methods were mainly machine learning-based algorithms such as edge detection and region segmentation. These methods usually only consider simple features such as pixel intensity, shape, and similarity of the image, while ignoring all spatial, semantic and other information, and are greatly affected by noise. Therefore, they cannot meet the requirements of chip defect detection. The rise of deep learning has brought new solutions to image segmentation. End-to-end pixel-level prediction can be achieved through deep convolutional neural networks. Semantic segmentation models such as FCN, U-Net, and DeepLab have greatly improved the accuracy of segmentation.

[0003] SegFormer is a semantic segmentation model based on deep learning. It introduces a self-attention mechanism to improve the model's ability to capture long-distance dependencies and integrate multi-scale features. However, the SegFormer model is not detailed enough in feature extraction and the boundary segmentation is not accurate enough, resulting in low accuracy in chip surface defect segmentation scenarios, making it difficult to directly apply to production scenarios. Summary of the invention

[0004] In view of the problems of poor generalization ability and insufficient segmentation accuracy of semantic segmentation models in the prior art, the present invention provides a chip image defect segmentation method based on an improved SegFormer network. By constructing a multi-category and multi-scale data set, the model's ability to understand and process defects of different types and scales is enhanced; the segmentation effect of the SegFormer network in the field of chip surface defect segmentation is optimized, and the segmentation accuracy of the model for defects of different types and scales is improved; the semantic segmentation image of the defect is obtained by improving the SegFormer semantic segmentation model, which can provide a more accurate defect contour for subsequent defect measurement work.

[0005] The present invention is achieved through the following technical solutions.

[0006] One aspect of the present invention provides a chip image defect segmentation method based on an improved SegFormer, comprising:

[0007] Construct a multi-category and multi-scale semantic segmentation dataset of chip surface defect images, which includes a training set and a test set;

[0008] Build an improved SegFormer semantic segmentation network, including improved encoders and decoders:

[0009] Use the encoder part of the SegFormer network for feature extraction and replace the self-attention in the 4-layer Transformer Block of the encoder part with the cross covariance attention CrossCovAttn;

[0010] Decoder: The decoder part is changed to a U-shaped structure, and the multi-layer perceptron MLP is used to adjust the number of feature channels of each layer in the encoder. An adaptive feature fusion module AFF is added to fuse the features of the jump connection of this layer with the features of the upper layer upsampled;

[0011] Using the training set to train the improved SegFormer semantic segmentation network to obtain an improved SegFormer semantic segmentation model;

[0012] The improved SegFormer semantic segmentation model obtained through training is evaluated using the test set, and the training parameters are adjusted according to the training logs and evaluation indicators to obtain an optimized model;

[0013] The improved SegFormer semantic segmentation model with the best performance is selected to perform semantic segmentation on the chip surface defect image to obtain the semantic segmentation result image.

[0014] Preferably, the construction of a multi-category, multi-scale chip surface defect image semantic segmentation dataset comprises:

[0015] Use optical cameras and optical microscopes to collect images of chip surface defects of different sizes and categories;

[0016] Preprocess the collected chip surface defect images, including bilateral filtering and histogram equalization;

[0017] Amplifying the data set of the preprocessed image to obtain an amplified data set;

[0018] The images in the amplified dataset are annotated with pixel-level semantic segmentation, and divided into a training set and a test set to obtain a semantic segmentation dataset of chip surface defect images.

[0019] Preferably, the Transformer Block of the encoder part includes:

[0020] An Overlap Patch Merging, which is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation;

[0021] N cross-covariance attentions CrossCovAttn and the mixed feedforward network Mix-FFN constitute a combined module to extract and process image features and enhance the expressiveness and robustness of the improved SegFormer semantic segmentation model.

[0022] As a preferred choice, cross covariance attention CrossCovAttn is used to focus on the covariance relationship between features, capture more fine-grained relationships between features, enhance the ability of the improved SegFormer semantic segmentation model to capture global information, and introduce learning parameters to adjust the distribution of attention weights.

[0023] As a preferred embodiment, the decoder part is changed to a U-shaped structure, including a multi-layer perceptron MLP, an adaptive feature fusion module AFF and an upsample UpSample;

[0024] The decoder part obtains the extracted features in the 4-layer Transformer Block in the encoder through skip connections;

[0025] Each layer of features is adjusted through the multi-layer perceptron MLP to adjust the number of channels;

[0026] Use the adaptive feature fusion module AFF to fuse the low-level semantic features of the jump connection and the high-level semantic features obtained by upsampling;

[0027] The number of channels and size of the features are further adjusted through the multi-layer perceptron MLP and upsampling UpSample as feature output.

[0028] Preferably, the adaptive feature fusion module AFF fuses the feature map of the jump connection with the feature map of the upsampling module, calculates the feature weights at the channel level and the spatial level, adaptively adjusts the weight of each channel and each position, fuses features at different levels, and improves the segmentation accuracy of the model.

[0029] Preferably, the training set is used to train the improved SegFormer semantic segmentation network. The improved SegFormer network uses MiT-B0 to MiT-B5 backbone networks, loads the pre-trained weights corresponding to the backbone network, selects AdamW as the optimizer, and selects cross entropy loss and Focal loss as the loss function.

[0030] Preferably, the evaluation indicators of the semantic segmentation model include: mean intersection over union (mIoU), mean pixel accuracy (mPA), and mean precision (mPrecision).

[0031] As a preference, the improved SegFormer semantic segmentation model with the best performance is selected to perform semantic segmentation on chip surface defect images, including:

[0032] Input the original image into the improved SegFormer semantic segmentation model for segmentation;

[0033] Extract features of the image based on the encoder part to obtain feature maps output by four Transformer modules;

[0034] The feature maps output by the four Transformer modules are input into the decoder part for feature fusion and prediction, and finally the score of each pixel belonging to each category is obtained;

[0035] Based on the output of the decoder, a softmax classifier is used to convert the scores into probability values. The pixel with the highest probability is selected as the category of the pixel, and different colors are marked for pixels of different categories to finally generate a semantic segmentation result image.

[0036] Another aspect of the present invention provides a chip image defect segmentation device based on an improved SegFormer, comprising:

[0037] A construction module is used to construct a multi-category and multi-scale chip surface defect image semantic segmentation dataset including a training set and a test set;

[0038] Adjustment module, used to improve the SegFormer semantic segmentation network, replace the self-attention in the 4-layer TransformerBlock of the encoder part with the cross covariance attention CrossCovAttn; change the decoder part to a U-shaped structure for feature fusion;

[0039] A training module is used to train the improved SegFormer semantic segmentation network to obtain an improved SegFormer semantic segmentation model;

[0040] The testing module is used to evaluate the trained improved SegFormer semantic segmentation model, adjust the training parameters, and obtain the optimized model;

[0041] The semantic segmentation module is used to select the improved SegFormer semantic segmentation model with the best performance to perform semantic segmentation on the chip surface defect image and obtain the semantic segmentation result image.

[0042] Another aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the chip image defect segmentation method based on the improved SegFormer when executing the computer program.

[0043] The present invention adopts the above technical solution, which has the following beneficial effects:

[0044] 1) In view of the poor generalization ability of semantic segmentation models in the field of chip surface defect segmentation, the present invention solves the problem of lack of diversity in training data by constructing a multi-category, multi-scale dataset. The model can be exposed to richer feature information during the training process, thereby improving its adaptability and generalization performance to new samples. Ultimately, this enhanced dataset provides a more comprehensive learning foundation for the model, enabling it to more accurately identify and segment various chip surface defects in practical applications, significantly improving the practicality and reliability of the model.

[0045] 2) In response to the problem that the SegFormer network encoder is not detailed enough in feature extraction, the present invention uses cross-covariance attention CrossCovAttn to replace the original attention, so that the model can capture more fine-grained relationships between features and enhance the ability to capture global information.

[0046] 3) In order to address the problems of SegFormer network’s weak ability to capture contextual information and insufficient boundary segmentation accuracy, a U-shaped decoder is designed for the SegFormer network, so that the model can capture contextual information at different levels and improve the model’s ability to segment boundary information. An adaptive feature fusion module AFF is designed for feature fusion in the decoder, which adaptively adjusts the weights of each channel and each position in the feature, thereby more effectively fusing features at different levels and improving the segmentation accuracy of the model.

[0047] 4) In order to solve the problem that the existing defect measurement methods rely on manual experience and are highly subjective, the present invention obtains the semantic segmentation image of the defect by improving the SegFormer semantic segmentation model, which can provide a more accurate defect contour for subsequent defect measurement work. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 It is a flowchart of a chip image defect segmentation method based on SegFormer network in a preferred embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of the structure of an improved SegFormer semantic segmentation network in a preferred embodiment of the present invention;

[0051] Figure 3 It is a structural diagram of an adaptive feature fusion module AFF in a preferred embodiment of the present invention;

[0052] Figure 4 A schematic diagram of a chip image defect segmentation device based on an improved SegFormer according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the structure of a chip image defect segmentation electronic device based on an improved SegFormer shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0055] like Figure 1 As shown, the present invention provides a chip image defect segmentation method based on an improved SegFormer network, comprising the following steps:

[0056] S101. Construct a multi-category and multi-scale semantic segmentation dataset of chip surface defect images.

[0057] Since chip surface defects are diverse and defects of different categories and scales have great differences in features, this embodiment constructs a multi-category and multi-scale image dataset so that the model can learn more diverse features during the training process, thereby improving the generalization ability of defects of different types and scales.

[0058] The specific steps include:

[0059] S11. Use an optical camera and an optical microscope to collect images of chip surface defects of different sizes and categories;

[0060] S12. Preprocess the collected chip surface defect image, use the bilateralFilter() function in the OpenCV library to perform bilateral filtering to retain the image edge information; then use the equalizeHist() function in the OpenCV library to perform histogram equalization to enhance the image contrast;

[0061] S13. The preprocessed image data set is expanded, and the image is blurred, mirrored, rotated, and other operations are performed using functions such as blur(), flip(), and warpAffine() in the OpenCV library to obtain an expanded data set;

[0062] S14. Use automated annotation software to perform pixel-level semantic segmentation annotation on the images in the amplified dataset, annotate the segmentation mask and category label for each defect category, and then convert the annotated .json file into a .PNG image file that can be used for semantic segmentation training. Then divide the file into training and test sets to obtain a semantic segmentation dataset of chip surface defect images.

[0063] In step S101, since the surface defects of the chip are diverse, defects of different categories and scales have great differences in features. Therefore, by constructing a multi-category and multi-scale image dataset, the model can learn more diverse features during the training process, thereby improving the generalization ability of defects of different types and scales. The diversity of this dataset ensures that the model can effectively cope with complex and changeable actual defect scenarios, and improves the reliability and adaptability of the segmentation model.

[0064] S102. Build an improved SegFormer semantic segmentation network.

[0065] Constructing an improved SegFormer semantic segmentation network includes the following steps:

[0066] S21. Use the encoder part of the SegFormer network for feature extraction and replace the self-attention in the TransformerBlock in the encoder with the cross-covariance attention CrossCovAttn.

[0067] The Transformer Block of the encoder part includes:

[0068] An Overlap Patch Merging, which is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation;

[0069] N cross-covariance attentions CrossCovAttn and the mixed feedforward network Mix-FFN constitute a combined module to extract and process image features and enhance the expressiveness and robustness of the improved SegFormer semantic segmentation model.

[0070] Cross-covariance attention CrossCovAttn focuses on the covariance relationship between features, can capture more fine-grained relationships between features, enhance the ability of the improved SegFormer semantic segmentation model to capture global information, and introduces learnable parameters to adjust the distribution of attention weights. The calculation formula of CrossCovAttn is:

[0071]

[0072] The meanings of the parameters in the formula are as follows: where Q, K, and V are the query matrix, key matrix, and value matrix generated by the feature vector, respectively; represents the normalized query matrix; represents the transpose of the normalized bond matrix; τ is a learnable parameter used to adjust and Softmax is a normalization function used to output probability distribution.

[0073] S22. The decoder part is changed to a U-shaped structure, including a multi-layer perceptron MLP, an adaptive feature fusion module AFF and an upsampler UpSample.

[0074] A multi-layer perceptron (MLP) is used to adjust the number of feature channels of each layer in the encoder, and an adaptive feature fusion module (AFF) is added to fuse the features of the jump connection of this layer with the features of the upsampled layers.

[0075] The U-shaped structure decoder can retain more fine-grained features and enhance the model's ability to segment boundaries, enabling the model to capture contextual information at different levels, thereby better understanding the overall structure of the image and enhancing the robustness of the model.

[0076] The adaptive feature fusion module AFF fuses the feature map of the jump connection with the feature map of the upsampling module. By calculating the feature weights at the channel level and the spatial level, the module can adaptively adjust the weight of each channel and each position, thereby more effectively fusing features at different levels and improving the segmentation accuracy of the model.

[0077] The structure of the AFF feature fusion module is as follows Figure 3 As shown, the channel feature weights w that capture F1 and F2 respectively ch and spatial feature weight w sp For the channel feature weight w ch First, concatenate F1 and F2 along the channel dimension, extract features through average pooling, convolution and Sigmoid activation function in turn, and finally retain important feature channels through 1×1 convolution; for the spatial feature weight w sp , through 1×1 convolution and Sigmoid activation function to build the spatial dependency between features, emphasizing the important spatial regions; finally, the channel and spatial weights are combined as output. The calculation formula of AFF is:

[0078] w ch =Sigmoid(Conv1(AVGPool([F1;F2]))

[0079]

[0080] The meanings of the parameters in the formula are as follows: ch represents the spatial-level feature weight; w sp represents the channel-level feature weight; F1 and F2 represent the feature map of the input jump connection and the feature map of the upsampling module respectively; [F1; F2] represents the concatenation of the two feature maps in a specified dimension; Sigmoid() represents the Sigmoid activation function; AVGPool() represents average pooling; Conv1() represents a 1×1 convolution layer; F represents the output feature map of the DFF feature fusion module; It means element-by-element addition; Represents element-wise multiplication.

[0081] The specific structure of the improved SegFormer semantic segmentation network is as follows Figure 2 As shown:

[0082] The encoder part consists of 4 Transformer Blocks, which include the following parts: an Overlap Patch Merging, which is used to block the input image features and embed them into a new feature space, while completing the downsampling operation; N Cross Covariance Attention CrossCovAttn and Mixed Feedforward Network Mix-FFN combination modules, which are used to effectively extract and process features and enhance the expressiveness and robustness of the model. The above N is the number of modules in this layer, which depends on the scale of the backbone network MiT-B0 to MiT-B5.

[0083] The decoder part obtains the extracted features of the 4-layer Transformer Block in the encoder through skip connections. The number of channels of each layer of features is adjusted through the multi-layer perceptron MLP, and then the adaptive feature fusion module AFF is used to fuse the low-level semantic features of the skip connection and the high-level semantic features obtained by upsampling. Finally, the number of channels and size of the features are further adjusted through the multi-layer perceptron MLP and upsampling as the final feature output.

[0084] In step S102, an improved SegFormer semantic segmentation network is constructed, the encoder part of SegFormer is used for feature extraction, and the self-attention mechanism is replaced by the cross covariance attention CrossCovAttn, which focuses on the covariance relationship between features and enhances the model's ability to capture global information. The decoder part adopts a U-shaped structure, combines jump connections to retain fine-grained features, and adjusts the number of feature channels through a multi-layer perceptron MLP. At the same time, an adaptive feature fusion module AFF is added to adaptively fuse low-level and high-level semantic features by calculating channel-level and spatial-level feature weights, thereby improving the segmentation accuracy of the model. The overall design enables the improved SegFormer network to more accurately identify and segment targets when processing complex segmentation tasks, significantly improving performance and robustness.

[0085] S103. Use the training set obtained in step S1 to train the improved SegFormer semantic segmentation network, obtain the improved SegFormer semantic segmentation model, and record the training log.

[0086] Use the training set to train the improved SegFormer semantic segmentation network. The improved SegFormer network uses MiT-B0 to MiT-B5 backbone networks, loads the pre-trained weights corresponding to the backbone network, selects AdamW as the optimizer, and selects cross entropy loss and Focal loss as the loss function.

[0087] The specific steps include:

[0088] S31. Using the training set obtained in step S1, use the resize() function in the OpenCV library to resize the input image to 512×512 as the input for model training. The improved SegFormer network has 6 optional backbone networks from MiT-B0 to MiT-B5. Load the pre-trained weights corresponding to the backbone network. The optimizer can be AdamW. The loss function can be Cross Entropy Loss and Focal Loss. Set the initial learning rate to 1e-4, Batch Size to 36 and other training parameters, train the improved SegFormer semantic segmentation network, and obtain the improved SegFormer semantic segmentation model.

[0089] S32. Record the training log, which includes the loss value and mIoU indicator during training.

[0090] In step S103, the improved SegFormer semantic segmentation network can achieve good segmentation performance after careful configuration and training. Data preprocessing, backbone network selection, pre-trained weight loading, optimizer and loss function selection, and training parameter setting are all key factors to ensure effective model learning and improve generalization ability. At the same time, the recording of training logs provides an important basis for subsequent model evaluation and optimization.

[0091] S104. Use the test set obtained in step S101 to evaluate the trained improved SegFormer semantic segmentation model, adjust the training parameters according to the training log and evaluation indicators, optimize the performance of the model, and obtain the optimized model.

[0092] The specific steps include:

[0093] S41. Input the test set images into the improved SegFormer semantic segmentation model to obtain the segmentation results, and calculate the three evaluation indicators of the model: mean intersection over union (mIoU), mean pixel accuracy (mPA), and mean precision (mPrecision) according to the formula:

[0094]

[0095] The parameters in the formula have the following meanings: N is the number of categories; c is the specified category; TP c is a true positive, that is, the number of pixels correctly predicted as category c; FP c is a false positive, that is, the number of pixels that are incorrectly predicted to be of category c; FN c is a false negative, i.e., the number of pixels that are actually of class c but are not predicted to be of that class.

[0096] S42. Replace and improve the backbone network of the SegFormer semantic segmentation network, adjust parameters such as learning rate and Batch Size, continue to train the next generation model, and record the training parameters and evaluation indicators of each generation model.

[0097] In step S104, by evaluating and optimizing the trained model, the performance and stability of the segmentation model can be further improved to ensure its effectiveness in practical applications. The evaluation process uses a test set for verification, calculates indicators such as mIoU, mPA, and mPrecision, and comprehensively evaluates the segmentation performance of the model. According to the evaluation results, the training parameters are adjusted, and training and optimization are repeated to ensure the robustness and stability of the model in the actual application environment.

[0098] S105. Select the improved SegFormer semantic segmentation model with the best performance in step S104 to perform semantic segmentation on the chip surface defect image to obtain a semantic segmentation result image.

[0099] The specific steps include:

[0100] S51. Input the original image into the improved SegFormer semantic segmentation model for segmentation;

[0101] S52. Extract features of the image based on the encoder part to obtain feature maps output by four Transformer Blocks;

[0102] S53. The feature maps output by the four Transformer Blocks are input into the decoder part for feature fusion and prediction, and finally the score of each pixel belonging to each category is obtained;

[0103] S54. Based on the output of the decoder, use the softmax classifier to convert the score into a probability value, select the pixel with the highest probability as the category of the pixel, label different categories of pixels with different colors, and finally generate a semantic segmentation result image.

[0104] In step S105, by using the segmentation model with the best performance to segment the actual chip surface defect image, a high-precision segmentation result can be obtained, providing reliable data support for subsequent defect size measurement. The improved SegFormer model ensures accurate identification and segmentation of complex defects through multi-level feature extraction and fusion. By fusing feature outputs and using a softmax classifier to generate the final segmented image, accurate labeling of defect areas is achieved.

[0105] The present invention improves the accuracy of the model in segmenting defects of different types and scales by constructing a multi-category, multi-scale data set and optimizing the SegFormer network, solving the problem that the semantic segmentation model is difficult to distinguish when processing chip surface defect segmentation due to the high similarity between the background and the defect area. Compared with traditional segmentation methods, the present invention can significantly reduce the problem of insufficient generalization ability of the chip surface defect segmentation model when processing defects with diverse shapes and sizes.

[0106] According to an exemplary embodiment of the present invention, Figure 4 As shown, a chip image defect segmentation device 100 based on an improved SegFormer is used to implement the method, including:

[0107] A construction module 110 is used to construct a multi-category and multi-scale chip surface defect image semantic segmentation dataset including a training set and a test set;

[0108] The adjustment module 120 is used to improve the SegFormer semantic segmentation network by replacing the self-attention in the 4-layer Transformer Block of the encoder part with the cross covariance attention CrossCovAttn; and changing the decoder part to a U-shaped structure for feature fusion;

[0109] A training module 130 is used to train the improved SegFormer semantic segmentation network to obtain an improved SegFormer semantic segmentation model;

[0110] A testing module 140 is used to evaluate the trained improved SegFormer semantic segmentation model, adjust training parameters, and obtain an optimized model;

[0111] The semantic segmentation module 150 is used to select the improved SegFormer semantic segmentation model with the best performance to perform semantic segmentation on the chip surface defect image and obtain a semantic segmentation result image.

[0112] According to an exemplary embodiment of the present invention, Figure 5 As shown, the present invention provides an electronic device 200 for implementing the chip image defect segmentation method based on improved SegFormer, including: an acquisition card 210, a memory 220, a communication bus 230, and a processor 240.

[0113] The computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to implement an operation of a chip image defect segmentation method based on an improved SegFormer.

[0114] It should be noted that the above embodiment is a preferred implementation and should not be a limitation of the present invention. Professional and technical personnel can further combine the modules and algorithm steps of each example described in the embodiments disclosed herein to implement the present invention with electronic hardware, computer software or a combination of the two. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0115] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A chip image defect segmentation method based on improved SegFormer, characterized in that: include: Construct a multi-category and multi-scale semantic segmentation dataset of chip surface defect images, which includes a training set and a test set; Build an improved SegFormer semantic segmentation network, including improved encoders and decoders: Use the encoder part of the SegFormer network for feature extraction, and replace the self-attention in the 4-layer TransformerBlock of the encoder part with the cross covariance attention CrossCovAttn; The decoder part is changed to a U-shaped structure, and the multi-layer perceptron MLP is used to adjust the number of feature channels of each layer in the encoder. An adaptive feature fusion module AFF is added to fuse the features of the jump connection of the current layer with the features of the upsampled high-level layer. Using the training set to train the improved SegFormer semantic segmentation network to obtain an improved SegFormer semantic segmentation model; The improved SegFormer semantic segmentation model obtained through training is evaluated using the test set, and the training parameters are adjusted according to the training logs and evaluation indicators to obtain an optimized model; The improved SegFormer semantic segmentation model with the best performance is selected to perform semantic segmentation on the chip surface defect image to obtain the semantic segmentation result image.

2. The chip image defect segmentation method based on improved SegFormer according to claim 1, characterized in that: The method of constructing a multi-category and multi-scale chip surface defect image semantic segmentation dataset includes: Use optical cameras and optical microscopes to collect images of chip surface defects of different sizes and categories; Preprocess the collected chip surface defect images, including bilateral filtering and histogram equalization; Amplifying the data set of the preprocessed image to obtain an amplified data set; The images in the amplified dataset are annotated with pixel-level semantic segmentation, and divided into a training set and a test set to obtain a semantic segmentation dataset of chip surface defect images.

3. The chip image defect segmentation method based on improved SegFormer according to claim 1, characterized in that: The Transformer Block of the encoder part includes: An Overlap Patch Merging, which is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation; N combined modules consisting of cross-covariance attention CrossCovAttn and mixed feedforward network Mix-FFN are used to extract and process image features to enhance the expressiveness and robustness of the improved SegFormer semantic segmentation model.

4. The chip image defect segmentation method based on improved SegFormer according to claim 3, characterized in that: Cross-covariance attention CrossCovAttn is used to focus on the covariance relationship between features, capture finer-grained relationships between features, enhance the ability of the improved SegFormer semantic segmentation model to capture global information, and introduce learning parameters to adjust the distribution of attention weights.

5. The chip image defect segmentation method based on improved SegFormer according to claim 1, characterized in that: The decoder part is changed to a U-shaped structure, including a multi-layer perceptron MLP, an adaptive feature fusion module AFF, and an upsampler UpSample; The decoder part obtains the features extracted by the 4-layer Transformer Block in the encoder through skip connections; Each layer of features is adjusted through the multi-layer perceptron MLP to adjust the number of channels; Use the adaptive feature fusion module AFF to fuse the low-level semantic features of the jump connection and the upsampled high-level semantic features; The number of channels and size of the features are further adjusted through the multi-layer perceptron MLP and upsampling UpSample as feature output.

6. The chip image defect segmentation method based on improved SegFormer according to claim 5, characterized in that: The adaptive feature fusion module AFF fuses the feature map of the jump connection with the feature map of the upsampling module. By calculating the feature weights at the channel level and the spatial level, it adaptively adjusts the weight of each channel and each position, fuses features at different levels, and improves the segmentation accuracy of the model.

7. The chip image defect segmentation method based on improved SegFormer according to claim 1, characterized in that: Use the training set to train the improved SegFormer semantic segmentation network. The improved SegFormer network uses MiT-B0 to MiT-B5 backbone networks, loads the pre-trained weights corresponding to the backbone network, selects AdamW as the optimizer, and selects cross entropy loss and Focal loss as the loss function.

8. The chip image defect segmentation method based on improved SegFormer according to claim 1, characterized in that: The improved SegFormer semantic segmentation model with the best performance is selected to perform semantic segmentation on chip surface defect images, including: Input the original image into the improved SegFormer semantic segmentation model for segmentation; Based on the encoder part, feature extraction is performed on the picture to obtain feature maps output by four Transformer Blocks; The feature maps output by the four Transformer Blocks are input into the decoder part for feature fusion and prediction, and finally the score of each pixel belonging to each category is obtained; Based on the output of the decoder, a softmax classifier is used to convert the scores into probability values. The pixel with the highest probability is selected as the category of the pixel, and different colors are marked for pixels of different categories to finally generate a semantic segmentation result image.

9. A chip image defect segmentation device based on improved SegFormer according to any one of claims 1 to 8, characterized in that: include: A construction module is used to construct a multi-category and multi-scale chip surface defect image semantic segmentation dataset including a training set and a test set; Adjustment module, used to improve the SegFormer semantic segmentation network, replace the self-attention in the 4-layer Transformer Block of the encoder part with the cross covariance attention CrossCovAttn; change the decoder part to a U-shaped structure for feature fusion; A training module is used to train the improved SegFormer semantic segmentation network to obtain an improved SegFormer semantic segmentation model; The testing module is used to evaluate the trained improved SegFormer semantic segmentation model, adjust the training parameters, and obtain the optimized model; The semantic segmentation module is used to select the improved SegFormer semantic segmentation model with the best performance to perform semantic segmentation on the chip surface defect image and obtain the semantic segmentation result image.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is used to call the program stored in the memory to execute the steps of the chip image defect segmentation method based on the improved SegFormer as described in any one of claims 1 to 8.

Images