A chip image defect segmentation method based on an improved SegFormer

Abstract

This invention discloses a chip image defect segmentation method based on an improved SegFormer network. It constructs a semantic segmentation dataset of chip surface defect images; improves the SegFormer semantic segmentation network by replacing the self-attention of the Transformer Block in the encoder with Cross-Covariance Attention (CrossCovAttn), changing the decoder to a U-shaped structure, and adding an AFF dynamic feature fusion module; trains an improved SegFormer semantic segmentation model; adjusts the model training parameters to obtain an optimized model; selects the best-performing improved SegFormer semantic segmentation model to perform semantic segmentation on chip surface defect images, obtaining semantic segmentation result images. This invention enhances the ability to handle defects of different types and scales, optimizes the SegFormer network segmentation effect, and improves the segmentation accuracy of defects of different types and scales.

Description

Technical Field

[0001] This invention belongs to the field of image or video recognition or understanding technology, and specifically relates to a chip image defect segmentation method based on an improved SegFormer network. Background Technology

[0002] As a core component of modern electronic devices, chip quality control during manufacturing is crucial. Even minute defects on the chip surface can severely impact its performance. However, the wide variety of surface defects, with their diverse shapes and sizes, makes traditional manual inspection methods inefficient, resulting in high false positive and false negative rates. Early image segmentation methods primarily relied on machine learning algorithms such as edge detection and region segmentation. These methods typically considered only simple features like pixel intensity, shape, and similarity, neglecting spatial and semantic information, and were highly susceptible to noise, thus failing to meet the requirements for chip defect detection. The rise of deep learning has brought new solutions to image segmentation. Deep convolutional neural networks can achieve end-to-end pixel-level prediction, and semantic segmentation models such as FCN, U-Net, and DeepLab have significantly improved segmentation accuracy.

[0003] SegFormer is a semantic segmentation model based on deep learning. It introduces a self-attention mechanism to improve the model's ability to capture long-distance dependencies and fuse multi-scale features. However, SegFormer's feature extraction granularity is not fine enough, and its boundary segmentation is not precise enough, resulting in low accuracy in chip surface defect segmentation scenarios. Therefore, it is difficult to directly apply it to production scenarios. Summary of the Invention

[0004] This invention addresses the problems of poor generalization ability and insufficient segmentation accuracy of existing semantic segmentation models by providing a chip image defect segmentation method based on an improved SegFormer network. By constructing multi-class, multi-scale datasets, the method enhances the model's ability to understand and process defects of different types and scales. It optimizes the segmentation performance of the SegFormer network in chip surface defect segmentation, improving the model's segmentation accuracy for defects of different types and scales. Furthermore, by improving the SegFormer semantic segmentation model to obtain semantically segmented defect images, more accurate defect contours can be provided for subsequent defect measurement.

[0005] The present invention is achieved through the following technical solution.

[0006] One aspect of the present invention provides a chip image defect segmentation method based on an improved SegFormer, comprising:

[0007] Construct a semantic segmentation dataset for multi-class, multi-scale chip surface defect images, which includes a training set and a test set.

[0008] Construct an improved SegFormer semantic segmentation network, including an improved encoder and decoder:

[0009] Feature extraction is performed using the encoder part of the SegFormer network, and the self-attention in the 4-layer Transformer Block of the encoder part is replaced with CrossCovAttn (cross-covariance attention).

[0010] Decoder: The decoder part was changed to a U-shaped structure. A multilayer perceptron (MLP) was used to adjust the number of feature channels in each layer of the encoder. An adaptive feature fusion module (AFF) was added to fuse the features of the skip connections in the current layer with the features upsampled in the higher layers.

[0011] The improved SegFormer semantic segmentation network is trained using the training set to obtain the improved SegFormer semantic segmentation model.

[0012] The improved SegFormer semantic segmentation model was evaluated using the test set. The training parameters were adjusted based on the training logs and evaluation metrics to obtain the optimized model.

[0013] The improved SegFormer semantic segmentation model with the best performance was selected to perform semantic segmentation on the chip surface defect image, and the semantic segmentation result image was obtained.

[0014] Preferably, the construction of a multi-category, multi-scale chip surface defect image semantic segmentation dataset includes:

[0015] Images of surface defects of chips of different sizes and types were acquired using optical cameras and optical microscopes.

[0016] The acquired chip surface defect images are preprocessed, including bilateral filtering and histogram equalization;

[0017] The preprocessed image is augmented to obtain the augmented dataset;

[0018] The images in the amplified dataset are subjected to pixel-level semantic segmentation annotation, and the dataset is divided into training and testing sets to obtain a semantic segmentation dataset of chip surface defect images.

[0019] Preferably, the Transformer Block of the encoder section includes:

[0020] Overlap Patch Merging is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation.

[0021] N cross-covariance attention networks (CrossCovAttn) combined with a hybrid feedforward network (Mix-FFN) form a combined module for extracting and processing image features, thereby enhancing and improving the expressive power and robustness of the SegFormer semantic segmentation model.

[0022] As a preferred option, CrossCovAttn is used to focus on the covariance relationship between features, capture more granular relationships between features, enhance the ability of the improved SegFormer semantic segmentation model to capture global information, and introduce learning parameters to adjust the distribution of attention weights.

[0023] As a preferred option, the decoder section is modified to a U-shaped structure, including a multilayer perceptron (MLP), an adaptive feature fusion module (AFF), and an upsampling module (UpSample).

[0024] The decoder part obtains the extracted features from the four Transformer Blocks in the encoder through skip connections;

[0025] The number of channels for each layer of features is adjusted by a multilayer perceptron (MLP).

[0026] The Adaptive Feature Fusion (AFF) module is used to fuse low-level semantic features from skip connections and upsampled high-level semantic features.

[0027] The number of channels and size of the features are further adjusted by a multilayer perceptron (MLP) and upsampling (UpSample) to produce the feature output.

[0028] As a preferred embodiment, the Adaptive Feature Fusion (AFF) module fuses the feature maps of skip connections with those of the upsampling module. By calculating the feature weights at the channel and spatial levels, it adaptively adjusts the weights of each channel and each position, fusing features at different levels to improve the segmentation accuracy of the model.

[0029] As a preferred option, the improved SegFormer semantic segmentation network is trained using the training set. The improved SegFormer network uses the MiT-B0 to MiT-B5 backbone network, loads the pre-trained weights corresponding to the backbone network, selects AdamW as the optimizer, and selects cross-entropy loss and Focal focus loss as the loss function.

[0030] Preferred evaluation metrics for semantic segmentation models include: mean intersection-union ratio (mIoU), mean pixel accuracy (mPA), and mean precision (mPrecision).

[0031] As a preferred option, the improved SegFormer semantic segmentation model with the best performance is selected to perform semantic segmentation on chip surface defect images, including:

[0032] The original image is input into the improved SegFormer semantic segmentation model for segmentation.

[0033] Based on the encoder part, feature extraction is performed on the image to obtain feature maps output by the four Transformer modules;

[0034] The feature maps output by the four Transformer modules are input into the decoder for feature fusion and prediction, and finally the score of each pixel belonging to each category is obtained.

[0035] Based on the decoder output, a softmax classifier is used to convert the score into a probability value. The pixel with the highest probability is selected as its category, and different colors are used to label pixels of different categories, ultimately generating a semantic segmentation result image.

[0036] Another aspect of the present invention provides a chip image defect segmentation apparatus based on an improved SegFormer, comprising:

[0037] The building block is used to construct a multi-class, multi-scale semantic segmentation dataset of chip surface defect images, including training and testing sets.

[0038] The adjustment module is used to improve the SegFormer semantic segmentation network. The self-attention in the 4-layer TransformerBlock of the encoder part is replaced with CrossCovAttn; the decoder part is changed to a U-shaped structure for feature fusion.

[0039] The training module is used to train the improved SegFormer semantic segmentation network to obtain the improved SegFormer semantic segmentation model.

[0040] The testing module is used to evaluate the improved SegFormer semantic segmentation model obtained through training, adjust the training parameters, and obtain an optimized model.

[0041] The semantic segmentation module is used to select the best-performing improved SegFormer semantic segmentation model to perform semantic segmentation on chip surface defect images and obtain semantic segmentation result images.

[0042] In another aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the chip image defect segmentation method based on the improved SegFormer.

[0043] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0044] 1) To address the poor generalization ability of semantic segmentation models in chip surface defect segmentation, this invention solves the problem of insufficient diversity in training data by constructing a multi-class, multi-scale dataset. The model can access richer feature information during training, thereby improving its adaptability to new samples and its generalization performance. Ultimately, this enhanced dataset provides the model with a more comprehensive learning foundation, enabling it to more accurately identify and segment various chip surface defects in practical applications, significantly improving the model's practicality and reliability.

[0045] 2) To address the issue that the SegFormer network encoder is not detailed enough in feature extraction, this invention uses CrossCovAttn to replace the original attention, enabling the model to capture finer-grained relationships between features and enhancing its ability to capture global information.

[0046] 3) To address the issues of SegFormer network's weak ability to capture contextual information and insufficient boundary segmentation accuracy, a U-shaped decoder is designed for SegFormer network to enable the model to capture contextual information at different levels and improve the model's ability to segment boundary information. An adaptive feature fusion module (AFF) is designed for feature fusion in the decoder. By adaptively adjusting the weights of each channel and each position in the features, features at different levels are fused more effectively, thereby improving the model's segmentation accuracy.

[0047] 4) To address the problem that existing defect measurement methods rely on human experience and are highly subjective, this invention improves the SegFormer semantic segmentation model to obtain semantic segmentation images of defects, which can provide more accurate defect contours for subsequent defect measurement work. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, do not constitute an undue limitation of the invention. In the drawings:

[0049] Figure 1 This is a flowchart of a chip image defect segmentation method based on SegFormer network in a preferred embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the structure of the improved SegFormer semantic segmentation network in a preferred embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the adaptive feature fusion module (AFF) in a preferred embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of a chip image defect segmentation device based on an improved SegFormer, as shown in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of an electronic device structure for chip image defect segmentation based on an improved SegFormer, as shown in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention.

[0055] like Figure 1 As shown, the present invention provides a chip image defect segmentation method based on an improved SegFormer network, comprising the following steps:

[0056] S101. Construct a semantic segmentation dataset of multi-class, multi-scale chip surface defect images.

[0057] Because chip surface defects are diverse, and defects of different categories and scales have significant differences in features, this embodiment constructs an image dataset containing multiple categories and scales. This allows the model to learn more diverse features during training, thereby improving its generalization ability to different types and scales of defects.

[0058] Specifically, the following steps are included:

[0059] S11. Use optical cameras and optical microscopes to acquire images of chip surface defects of different sizes and types;

[0060] S12. Preprocess the acquired chip surface defect image by using the bilateralFilter() function in the OpenCV library to perform bilateral filtering and preserve image edge information; then use the equalizeHist() function in the OpenCV library to perform histogram equalization and enhance image contrast.

[0061] S13. Perform dataset augmentation on the preprocessed image. Use OpenCV library functions such as blur(), flip(), and warpAffine() to perform blurring, mirroring, and rotation operations on the image to obtain the augmented dataset.

[0062] S14. Use automated annotation software to perform pixel-level semantic segmentation annotation on the images in the augmented dataset, annotate the segmentation mask and category label for each defect category, then convert the annotated .json file into a .PNG image file that can be used for semantic segmentation training, and then divide the dataset into training and test sets to obtain the semantic segmentation dataset of chip surface defect images.

[0063] In step S101, due to the diversity of chip surface defects, different categories and scales of defects exhibit significant differences in features. Therefore, by constructing an image dataset containing multiple categories and scales, the model can learn more diverse features during training, thereby improving its generalization ability to different types and scales of defects. The diversity of this dataset ensures that the model can effectively cope with complex and varied real-world defect scenarios, improving the reliability and adaptability of the segmentation model.

[0064] S102. Construct an improved SegFormer semantic segmentation network.

[0065] The construction of the improved SegFormer semantic segmentation network includes the following steps:

[0066] S21. Use the encoder part of the SegFormer network for feature extraction, and replace the self-attention in the TransformerBlock of the encoder with cross-covariance attention (CrossCovAttn).

[0067] The Transformer Block in the encoder section includes:

[0068] Overlap Patch Merging is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation.

[0069] N cross-covariance attention networks (CrossCovAttn) combined with a hybrid feedforward network (Mix-FFN) form a combined module for extracting and processing image features, thereby enhancing and improving the expressive power and robustness of the SegFormer semantic segmentation model.

[0070] Cross-covariance attention (CrossCovAttn) primarily focuses on the covariance relationships between features, capturing finer-grained relationships and enhancing the SegFormer semantic segmentation model's ability to capture global information. It also introduces learnable parameters to adjust the distribution of attention weights. The formula for calculating CrossCovAttn is:

[0071]

[0072] The parameters in the formula have the following meanings: where Q, K, and V are the query matrix, key matrix, and value matrix generated from the feature vector, respectively; This represents the normalized query matrix; This represents the transpose of the normalized key matrix; τ is a learnable parameter used to... and The inner product is used for scaling; Softmax is a normalization function used to output the probability distribution.

[0073] S22. The decoder section is changed to a U-shaped structure, including a multilayer perceptron (MLP), an adaptive feature fusion module (AFF), and an upsampling module (UpSample).

[0074] The number of feature channels in each layer of the encoder is adjusted using a multilayer perceptron (MLP), and an adaptive feature fusion module (AFF) is added to fuse the features of the skip connections in the current layer with the features upsampled from higher layers.

[0075] The U-shaped structure decoder can retain finer-grained features, enhance the model's ability to segment boundaries, and enable the model to capture contextual information at different levels, thereby better understanding the overall structure of the image and enhancing the model's robustness.

[0076] The Adaptive Feature Fusion (AFF) module fuses the feature maps from skip connections with those from the upsampling module. By calculating channel-level and spatial-level feature weights, this module can adaptively adjust the weights of each channel and each location, thereby more effectively fusing features from different levels and improving the segmentation accuracy of the model.

[0077] The structure of the AFF feature fusion module is as follows: Figure 3 As shown, the channel feature weights w for capturing F1 and F2 are respectively. ch Spatial feature weights w sp For the channel feature weights w ch First, F1 and F2 are concatenated along the channel dimension. Then, features are extracted sequentially through average pooling, convolution, and the Sigmoid activation function. Finally, an important feature channel is preserved through a 1×1 convolution. For the spatial feature weights w... sp Spatial dependencies between features are constructed using 1×1 convolutions and a sigmoid activation function, emphasizing important spatial regions; finally, channel and spatial weights are combined as the output. The formula for calculating AFF is:

[0078] w ch =Sigmoid(Conv1(AVGPool([F1;F2]))

[0079]

[0080] The parameters in the formula have the following meanings: w ch Represents spatial-level feature weights; w sp The channel-level feature weights are represented; F1 and F2 represent the feature maps of the input skip connections and the upsampling module, respectively; [F1; F2] indicates concatenation of the two feature maps along a specified dimension; Sigmoid() represents the Sigmoid activation function; AVGPool() represents average pooling; Conv1() represents a 1×1 convolutional layer; F represents the output feature map of the DFF feature fusion module. This indicates element-wise addition; This indicates element-wise multiplication.

[0081] The specific structure of the improved SegFormer semantic segmentation network is as follows: Figure 2 As shown:

[0082] The encoder consists of four Transformer Blocks, each comprising: an Overlap Patch Merging module, used to segment the input image features and embed them into a new feature space, while simultaneously performing downsampling; and a combined module of N Cross-Covariance Attention (CrossCovAttn) and Mix-FFN feedforward networks, used to effectively extract and process features, enhancing the model's expressive power and robustness. N represents the number of modules in this layer, determined by the backbone network size from MiT-B0 to MiT-B5.

[0083] The decoder part obtains the extracted features from the four Transformer Blocks in the encoder through skip connections. The number of channels of each feature is adjusted by a multilayer perceptron (MLP). Then, an adaptive feature fusion module (AFF) is used to fuse the low-level semantic features from the skip connections and the upsampled high-level semantic features. Finally, the number of channels and size of the features are further adjusted by the MLP and upsampling to obtain the final feature output.

[0084] In step S102, an improved SegFormer semantic segmentation network is constructed. Feature extraction is performed using the encoder part of SegFormer, and the self-attention mechanism is replaced with Cross-Covariance Attention (CrossCovAttn). This mechanism focuses on the covariance relationship between features, enhancing the model's ability to capture global information. The decoder part adopts a U-shaped structure, combining skip connections to preserve fine-grained features, and adjusting the number of feature channels using a multilayer perceptron (MLP). Simultaneously, an adaptive feature fusion module (AFF) is added, which adaptively fuses low-level and high-level semantic features by calculating channel-level and spatial-level feature weights, thereby improving the model's segmentation accuracy. The overall design enables the improved SegFormer network to more accurately identify and segment targets when handling complex segmentation tasks, significantly improving performance and robustness.

[0085] S103. Use the training set obtained in step S1 to train the improved SegFormer semantic segmentation network, obtain the improved SegFormer semantic segmentation model, and record the training log.

[0086] The improved SegFormer semantic segmentation network was trained using the training set. The improved SegFormer network used the MiT-B0 to MiT-B5 backbone network, and the pre-trained weights corresponding to the backbone network were loaded. The optimizer was AdamW, and the loss function was cross-entropy loss and Focal focus loss.

[0087] Specifically, the following steps are included:

[0088] S31. Using the training set obtained in step S1, adjust the size of the input image to 512×512 using the resize() function in the OpenCV library, and use it as the input for model training. There are six backbone networks to choose from for the improved SegFormer network, ranging from MiT-B0 to MiT-B5. Load the pre-trained weights corresponding to the backbone network, select AdamW as the optimizer, and choose Cross Entropy Loss or Focal Loss as the loss function. Set the initial learning rate to 1e-4, the batch size to 36, and other training parameters to train the improved SegFormer semantic segmentation network, thus obtaining the improved SegFormer semantic segmentation model.

[0089] S32. Record the training log, which includes the loss value and mIoU metric during training.

[0090] In step S103, the improved SegFormer semantic segmentation network achieves good segmentation performance after careful configuration and training. Data preprocessing, backbone network selection, pre-training weight loading, optimizer and loss function selection, and training parameter settings are all key factors in ensuring effective model learning and improving generalization ability. Meanwhile, the recording of training logs provides important evidence for subsequent model evaluation and optimization.

[0091] S104. Use the test set obtained in step S101 to evaluate the improved SegFormer semantic segmentation model trained. Adjust the training parameters according to the training logs and evaluation metrics to optimize the model's performance and obtain the optimized model.

[0092] Specifically, the following steps are included:

[0093] S41. Input the test set images into the improved SegFormer semantic segmentation model to obtain the segmentation results, and calculate the model's average intersection-over-union ratio (mIoU), average pixel accuracy (mPA), and average precision (mPrecision) according to the formulas:

[0094]

[0095] The parameters in the formula have the following meanings: N is the number of categories; c is the specified category; TP c For true examples, this refers to the number of pixels correctly predicted as class c; FP c It is a false positive, that is, the number of pixels incorrectly predicted as class c; FN c These are false negatives, meaning the number of pixels that are actually of category c but were not predicted as belonging to that category.

[0096] S42. Replace the backbone network of the improved SegFormer semantic segmentation network, adjust parameters such as learning rate and batch size, continue to train the next generation model, and record the training parameters and evaluation metrics of each generation model.

[0097] In step S104, by evaluating and optimizing the trained model, the performance and stability of the segmentation model can be further improved, ensuring its effectiveness in practical applications. The evaluation process uses a test set for validation, calculating metrics such as mIoU, mPA, and mPrecision to comprehensively evaluate the model's segmentation performance. Based on the evaluation results, training parameters are adjusted, and repeated training and optimization are performed to ensure the model's robustness and stability in real-world application environments.

[0098] S105. Select the improved SegFormer semantic segmentation model with the best performance in step S104 to perform semantic segmentation on the chip surface defect image, and obtain the semantic segmentation result image.

[0099] Specifically, the following steps are included:

[0100] S51. Input the original image into the improved SegFormer semantic segmentation model for segmentation;

[0101] S52. Based on the encoder part, perform feature extraction on the image to obtain feature maps output by the four Transformer Blocks;

[0102] S53. Input the feature maps output by the four Transformer Blocks into the decoder part for feature fusion and prediction, and finally obtain the score of each pixel belonging to each category;

[0103] S54. Based on the decoder output, a softmax classifier is used to convert the score into a probability value. The pixel with the highest probability is selected as its category. Different colors are used to label pixels of different categories, and finally, a semantic segmentation result image is generated.

[0104] In step S105, by using the optimal segmentation model, the actual chip surface defect image is segmented, yielding high-precision segmentation results and providing reliable data support for subsequent defect size measurement. The improved SegFormer model ensures accurate identification and segmentation of complex defects through multi-level feature extraction and fusion. By fusing feature outputs and using a softmax classifier to generate the final segmented image, accurate annotation of the defect region is achieved.

[0105] This invention improves the accuracy of the model in segmenting defects of different types and scales by constructing multi-class, multi-scale datasets and optimizing the SegFormer network. It solves the problem that semantic segmentation models struggle to distinguish between background and defect regions when processing chip surface defects due to high similarity. Compared to traditional segmentation methods, this invention significantly reduces the generalization ability of chip surface defect segmentation models when handling defects with diverse shapes and sizes.

[0106] According to exemplary embodiments of the present invention, such as Figure 4 As shown, a chip image defect segmentation device 100 based on an improved SegFormer is used to implement the method, comprising:

[0107] Module 110 is used to construct a multi-class, multi-scale chip surface defect image semantic segmentation dataset that includes training and testing sets;

[0108] Adjustment module 120 is used to improve the SegFormer semantic segmentation network. The self-attention in the 4-layer Transformer Block of the encoder part is replaced with CrossCovAttn; the decoder part is changed to a U-shaped structure for feature fusion.

[0109] Training module 130 is used to train the improved SegFormer semantic segmentation network to obtain the improved SegFormer semantic segmentation model.

[0110] Test module 140 is used to evaluate the improved SegFormer semantic segmentation model obtained through training, adjust the training parameters, and obtain an optimized model.

[0111] The semantic segmentation module 150 is used to select the best-performing improved SegFormer semantic segmentation model to perform semantic segmentation on the chip surface defect image and obtain the semantic segmentation result image.

[0112] According to exemplary embodiments of the present invention, such as Figure 5 As shown, the present invention provides an electronic device 200 for implementing the chip image defect segmentation method based on the improved SegFormer, comprising: a data acquisition card 210, a memory 220, a communication bus 230, and a processor 240.

[0113] The computer device includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used to implement an operation of a chip image defect segmentation method based on an improved SegFormer.

[0114] It should be noted that the above embodiments are preferred implementations and should not be construed as limiting the present invention. Those skilled in the art can further implement the present invention using electronic hardware, computer software, or a combination of both, by combining the modules and algorithm steps of the various examples described in the embodiments disclosed herein. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0115] This invention is not limited to the above embodiments. Based on the technical solutions disclosed in this invention, those skilled in the art can make some substitutions and modifications to some of the technical features without creative effort, and all such substitutions and modifications are within the protection scope of this invention.

Claims

1. A chip image defect segmentation method based on an improved SegFormer, characterized in that, include: Construct a semantic segmentation dataset for multi-class, multi-scale chip surface defect images, which includes a training set and a test set. Construct an improved SegFormer semantic segmentation network, including an improved encoder and decoder: Feature extraction is performed using the encoder part of the SegFormer network, and the self-attention in the 4 layers of TransformerBlock in the encoder part is replaced with CrossCovAttn (cross-covariance attention). The decoder section was changed to a U-shaped structure. A multilayer perceptron (MLP) was used to adjust the number of feature channels in each layer of the encoder. An adaptive feature fusion module (AFF) was added to fuse the features of the skip connections in the current layer with the features upsampled in higher layers. The improved SegFormer semantic segmentation network is trained using the training set to obtain the improved SegFormer semantic segmentation model. The improved SegFormer semantic segmentation model was evaluated using the test set. The training parameters were adjusted based on the training logs and evaluation metrics to obtain the optimized model. The improved SegFormer semantic segmentation model with the best performance was selected to perform semantic segmentation on the chip surface defect image, and the semantic segmentation result image was obtained.

2. The chip image defect segmentation method based on the improved SegFormer according to claim 1, characterized in that, The construction of a multi-class, multi-scale chip surface defect image semantic segmentation dataset includes: Images of surface defects of chips of different sizes and types were acquired using optical cameras and optical microscopes. The acquired chip surface defect images are preprocessed, including bilateral filtering and histogram equalization; The preprocessed image is augmented to obtain the augmented dataset; The images in the amplified dataset are subjected to pixel-level semantic segmentation annotation, and the dataset is divided into training and testing sets to obtain a semantic segmentation dataset of chip surface defect images.

3. The chip image defect segmentation method based on the improved SegFormer according to claim 1, characterized in that, The Transformer Block in the encoder section includes: Overlap Patch Merging is used to divide the input image features into blocks and embed them into a new image feature space to complete the downsampling operation. N combined modules, consisting of CrossCovAttn and Mix-FFN, are used to extract and process image features, enhancing the expressive power and robustness of the SegFormer semantic segmentation model.

4. The chip image defect segmentation method based on the improved SegFormer according to claim 3, characterized in that, Cross-covariance attention (CrossCovAttn) is used to focus on the covariance relationship between features, capture more granular relationships between features, enhance the ability of the SegFormer semantic segmentation model to capture global information, and introduce learning parameters to adjust the distribution of attention weights.

5. The chip image defect segmentation method based on the improved SegFormer according to claim 1, characterized in that, The decoder section was changed to a U-shaped structure, including a multilayer perceptron (MLP), an adaptive feature fusion module (AFF), and an upsampling module (UpSample). The decoder part obtains the features extracted by the four Transformer Blocks in the encoder through skip connections; The number of channels for each layer of features is adjusted by a multilayer perceptron (MLP). The Adaptive Feature Fusion (AFF) module is used to fuse low-level semantic features from skip connections and upsampled high-level semantic features. The number of channels and size of the features are further adjusted by a multilayer perceptron (MLP) and upsampling (UpSample) to produce the feature output.

6. The chip image defect segmentation method based on the improved SegFormer according to claim 5, characterized in that, The Adaptive Feature Fusion (AFF) module fuses the feature maps from skip connections with those from the upsampling module. By calculating channel-level and spatial-level feature weights, it adaptively adjusts the weights of each channel and each location, fusing features from different levels to improve the model's segmentation accuracy.

7. The chip image defect segmentation method based on the improved SegFormer according to claim 1, characterized in that, The improved SegFormer semantic segmentation network was trained using the training set. The improved SegFormer network used the MiT-B0 to MiT-B5 backbone network, and the pre-trained weights corresponding to the backbone network were loaded. The optimizer was AdamW, and the loss function was cross-entropy loss and Focal focus loss.

8. The chip image defect segmentation method based on the improved SegFormer according to claim 1, characterized in that, The improved SegFormer semantic segmentation model, which has the best performance, is selected to perform semantic segmentation on chip surface defect images, including: The original image is input into the improved SegFormer semantic segmentation model for segmentation. Based on the encoder part, feature extraction is performed on the image to obtain feature maps output by the four Transformer Blocks; The feature maps output by the four Transformer Blocks are input into the decoder for feature fusion and prediction, ultimately yielding a score for each pixel belonging to each category. Based on the decoder output, a softmax classifier is used to convert the score into a probability value. The pixel with the highest probability is selected as its category, and different colors are used to label pixels of different categories, ultimately generating a semantic segmentation result image.

9. A chip image defect segmentation apparatus based on an improved SegFormer according to any one of claims 1-8, characterized in that, include: The building block is used to construct a multi-class, multi-scale semantic segmentation dataset of chip surface defect images, including training and testing sets. The adjustment module is used to improve the SegFormer semantic segmentation network. The self-attention in the 4-layer Transformer Block of the encoder part is replaced with CrossCovAttn; the decoder part is changed to a U-shaped structure for feature fusion. The training module is used to train the improved SegFormer semantic segmentation network to obtain the improved SegFormer semantic segmentation model. The testing module is used to evaluate the improved SegFormer semantic segmentation model obtained through training, adjust the training parameters, and obtain an optimized model. The semantic segmentation module is used to select the best-performing improved SegFormer semantic segmentation model to perform semantic segmentation on chip surface defect images and obtain semantic segmentation result images.

10. An electronic device, characterized in that, Including the processor and memory; The memory is used to store computer programs, the computer programs including program instructions; The processor is used to call a program stored in the memory to execute the steps of the chip image defect segmentation method based on the improved SegFormer as described in any one of claims 1-8.

Images