Method applied to defect detection of storage chips

Abstract

The invention relates to a method applied to storage type chip defect detection. The method is used for improving the detection speed and precision of storage type chip surface defects. According to the method, the Faster-RCNN network is adopted as a detection network, and a backbone network is replaced by a lightweight network MobileNetV3, so that the calculation amount and the parameter amount are greatly reduced. In addition, on this basis, a BiFPN network is connected, and the network balances information among different scale features through cross-scale connection and weighted feature fusion. Due to the fact that the sizes of the chips are different, the BiFPN network is good in performance in the aspect of processing the chips of different sizes, and therefore the quality of feature representation is improved while the calculation efficiency is guaranteed. And the robustness and the detection precision of the model are improved by balancing information among different scale features. The improved network model has superiority in the aspect of chip surface defect detection, and is suitable for chip surface detection in various scenes.

Description

Technical Field

[0001] The present invention relates to a method for defect detection of storage chips, and more specifically to a method for improving defect detection speed and accuracy based on an improved defect detection algorithm. Background Art

[0002] Chip surface defect detection technology is a key link in the semiconductor manufacturing process. It can ensure the quality and performance of the chip and improve the qualification rate of the product. In recent years, the successful application of deep learning models represented by convolutional neural networks in the field of computer vision has provided a new development direction for defect detection. The chip manufacturing process includes: chip design, chip manufacturing, chip surface defect detection, packaging and package defect detection, electrical performance detection and finished products. Among them, chip surface defect detection is an important link in the manufacturing process. The main research of this invention is the physical defect detection of the chip. In the process of chip production, there are many types of defects such as surface foreign matter, scratches, protrusions or missing surface materials, and residual contaminants. The relevant model algorithms of deep learning have achieved good detection results in many fields, which can also be applied to chip surface defect detection.

[0003] Due to different understandings of defects, deep learning can be divided into supervised learning, unsupervised learning, weakly supervised learning and semi-supervised learning in defect detection. The present invention uses supervised learning method for defect detection, using label-marked defect images to input into the network for training, extracting defect features and summarizing them. Summary of the invention

[0004] The present invention relates to the field of chip detection technology, and in particular to a method for detecting defects in storage chips. With the widespread application of electronic devices, storage chips are important components of electronic devices, and the quality and reliability of chips have an important impact on the performance of electronic devices. Therefore, it is of great significance to perform defect detection on storage chips.

[0005] The purpose of the present invention is to provide an efficient and accurate storage chip defect detection method to solve the problem that the manual detection method has relatively low efficiency and accuracy, is easily affected by subjective factors, and the detection efficiency cannot meet production needs. This method effectively improves the accuracy and speed of chip defect detection by combining image acquisition technology and image processing algorithms.

[0006] The present invention is implemented by adopting the following technical scheme: a method for surface defect detection of storage chips, including a Faster-RCNN model, which includes a feature extraction network (Conv layers), a region candidate network (Region Proposal Networks), a region of interest pooling layer (RoIPooling) and a classification and regression layer (Classification and Regression). The backbone network is replaced with a lightweight network MobileNetV3 to increase the speed of front-end processing. A BiFPN structure is connected to the model, and then the region candidate network and the interest pooling layer are connected respectively, and the interest pooling layer is connected to the classification and regression layer.

[0007] The BiFPN structure captures multi-scale information in the image by extracting and fusing features at different scales through bidirectional feature propagation. The general steps for processing feature maps are:

[0008] Top-down feature propagation: Starting from the highest resolution feature map, upsampling is performed to obtain a lower resolution feature map, which is then fused with the previous feature map. This operation allows the feature map to be fully expressed at different scales.

[0009] Bottom-up feature propagation: Starting from the lowest resolution feature map, a higher resolution feature map is obtained through downsampling, and then it is fused with the previous feature map. This operation can also make the feature map fully expressed at different scales.

[0010] Feature Fusion: BiFPN adjusts the scale of feature maps through upsampling and downsampling operations, and uses feature fusion techniques to fuse feature maps of different scales. This fusion method enables the network to simultaneously utilize feature information of different scales, thereby improving detection accuracy. At each scale, BiFPN learns a weight to fuse the features of that scale with other scales. This is achieved through a set of 1x1 convolutional layers to unify the number of channels of the feature map to the same value, and a set of 3x3 convolutional layers for feature fusion.

[0011] Horizontal connection: BiFPN introduces a horizontal connection operation to transfer information between feature maps at different levels. This can further enhance the expressiveness of feature maps and improve the accuracy of target detection.

[0012] A system for storage chip defect detection includes: a defect image input module, a feature extraction module, a feature classification module, and a detection result output module. Storage chip defect detection plays an important role in various production industries. Therefore, chip surface defect detection needs to achieve the following goals: ensure product quality while protecting consumer safety, improve production efficiency, reduce the time cost of workers' re-inspection, provide product follow-up tracking, and reduce problem maintenance time.

[0013] MobileNetV3 is selected as the backbone network of Faster R-CNN to reduce the computational complexity of the model and improve the processing speed. Feature extraction is performed on the input image to generate a feature map with rich semantic information. A feature pyramid is constructed based on MobileNetV3, and the model's detection ability for objects of different sizes is enhanced by fusing multi-scale features. The region proposal network (RPN) is improved to adapt to the feature map of the lightweight backbone network and quickly generate high-quality candidate regions.

[0014] Adapt the RoI pooling layer to ensure that candidate regions of different scales can be effectively mapped to a unified feature size for subsequent processing. Optimize the classification and bounding box regression heads to adapt to the feature representation of the lightweight network and improve the accuracy of detection. Design an efficient NMS strategy to remove overlapping prediction boxes and optimize the detection results. Propose an end-to-end training framework that jointly optimizes the parameters of all components through the back-propagation algorithm to achieve the best overall performance.

[0015] The storage chip defect detection method of the present invention can achieve efficient and accurate detection of storage chips, improve the quality and reliability of chips, and provide a strong guarantee for the stable operation of electronic equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. The present invention is further described in detail below in conjunction with the drawings.

[0017] Attached Figure 1 It is a flow chart of the present invention.

[0018] Attached Figure 2 This is the Faster-RCNN network structure diagram.

[0019] Attached Figure 3 This is the network structure diagram of Mobile net V3.

[0020] Attached Figure 4 This is the BiFPN network structure diagram. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is mainly aimed at identifying surface defects of storage chips. Through a deep learning model, it is improved by combining the classic lightweight model and the feature pyramid structure to achieve the purpose of accurately detecting chip surface defects, thereby improving product qualification rate and reducing the occurrence of subsequent failures.

[0022] The present invention aims to provide a target detection method based on the feature pyramid structure of Faster-RCNN and MobileNetV3, so as to realize rapid detection of chip surface defect areas under resource-constrained conditions. In order to improve the accuracy and efficiency of target detection, a target detection model based on Faster-RCNN as the backbone network is constructed. The model mainly includes three parts:

[0023] First, we build the Faster-RCNN network: We use MobileNetV3 as the backbone network of Faster-RCNN to replace the traditional convolutional neural network. MobileNetV3 achieves a significant reduction in parameters and computation through efficient network building modules such as deep separable convolution, inverted residual structure, and lightweight attention model.

[0024] Then construct a feature pyramid structure: In the Faster-RCNN network, a feature pyramid structure is introduced. This structure fuses features of different scales through a bottom-up pathway, a top-down pathway, and lateral connections. The bottom-up pathway is the backbone network of MobileNetV3, and the top-down pathway combines high-level semantic information with low-level detail information through upsampling and lateral connections.

[0025] RPN network generates candidate regions: On each layer of the feature pyramid, the RPN (Region Proposal Network) network is applied to generate candidate regions. The RPN network achieves efficient region generation by sharing the calculation of the backbone network.

[0026] ROI pooling and classification regression: Perform ROI pooling on the candidate regions generated by RPN and convert them into fixed-size feature maps. Finally, the feature maps are sent to the fully connected layer for classification and bounding box regression to obtain the final target detection results.

[0027] Next, the processing of the three important parts of the network model is described separately:

[0028] 1. Faster RCNN Network

[0029] First, the chip image to be tested is input, and the feature extraction network uses a pre-trained convolutional neural network to extract preliminary features; the convolutional neural network converts the original image into a feature map with rich semantic information through a series of convolutional layers, pooling layers, and activation functions. The feature map extracted in this step includes many scales and different semantic information.

[0030] Secondly, after the RPN region generates the candidate region, the network slides a small window on the feature map to predict candidate boxes of different scales and aspect ratios for each position and calculate the score of each candidate box, further preparing for subsequent classification regression.

[0031] Next, we proceed to the feature region pooling part. RoIPooling is used to pool the feature regions corresponding to the candidate boxes generated by RPN into feature maps of fixed size. This step is to map candidate boxes of different sizes to feature maps of the same size for subsequent classification and regression operations.

[0032] 2.MobileNet V3 Structure

[0033] The MobileNet V3 network combines the advantages of the previous two generations of lightweight networks and adopts the depthwise separable convolutions in MobileNet V1. This structure greatly reduces the parameters and computation of the model by decomposing the traditional convolution into two steps: depthwise convolution and pointwise convolution. The depth convolution applies a separate convolution kernel to each input channel, while the pointwise convolution uses a 1x1 convolution kernel to combine the outputs of the depth convolution.

[0034] Bottleneck structure and inverted residuals: The bottleneck structure and inverted residual design of MobileNetV2 are inherited. The bottleneck structure adopts the method of first increasing the dimension and then reducing the dimension. The number of channels is first increased through 1x1 convolution, then deep convolution is performed, and finally the number of channels is reduced back to the original through 1x1 convolution. The inverted residual structure first expands the feature channel, performs deep convolution and point-by-point convolution, and finally adds it to the input through a shortcut (residual connection).

[0035] The Squeeze-and-Excitation (SE) module is introduced to enhance the channel interaction and context information transmission in the network. The SE module specifically strengthens the correlation between channels by adaptively learning the processing weights of the channels, thereby improving the discriminative power and semantic expression ability of the feature map. MobileNetV3 uses the h-swish function as the activation function, replacing the ReLU6 in MobileNetV2. The h-swish function reduces the amount of calculation while maintaining the performance of the swish function, further improving the performance of the model.

[0036] The main improvement ideas of MobileNetV3 focus on combining hardware-aware network architecture search (NAS) and network optimization algorithms to optimize performance on mobile device CPUs. It uses a novel architecture design, including an inverted residual structure and a linear bottleneck layer, as well as a new efficient segmentation decoder (LR-ASPP) to improve performance in mobile classification, detection, and segmentation tasks. These improvements achieve higher accuracy, lower latency, and better performance in different resource usage scenarios through a carefully designed lightweight architecture.

[0037] 3. BiFPN structure

[0038] The weighted bidirectional feature pyramid network (BiFPN) can perform multi-scale feature fusion simply and quickly. Multi-scale feature fusion aims to fuse features of different resolutions. Given multiple features of different scales, its goal is to find a transformation that can effectively aggregate features of different scales and output multiple aggregated new features.

[0039] When fusing features of different resolutions, a common approach is to adjust their resolutions to equal size and then add them. Previous methods treat different input features equally, but the author observes that since different features have different resolutions, they usually contribute differently to the output.

[0040] According to the softmax-based fusion method, a fast normalized fusion is proposed. i Then add an activation function to ensure that it is greater than or equal to 0. The feature fusion of the sixth layer is as follows:

[0041]

[0042] in is the intermediate feature of the sixth layer in the top-down path, It is the output feature of the sixth layer in the bottom-up path. BN and activation functions are added after each depth-separable convolution. The BiFPN structure is based on FPN, and establishes two paths from top to bottom and bottom to top for feature fusion. If the input node and the output node are in the same layer, an additional edge is added to fuse more features without increasing the amount of calculation. This structure is based on the contribution rate of the output features. BiFPN is equivalent to giving different weights to each layer for fusion, allowing the network to pay more attention to important layers and also reducing node connections in some unnecessary layers.

[0043] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for detecting storage chip defects, characterized in that: The following steps are involved: (1) Obtain the storage chip to be tested; (2) Reading preset storage chip template data; (3) Capturing images of the chip at different directions and angles through an image acquisition device; (4) Processing the acquired image according to the defect detection method proposed by the present invention; (5) Determine whether the chip has defects and store the test results; (6) Analyze the test results and issue an instruction manual.

2. Includes the Faster-RCNN model, which includes a feature extraction network, a region candidate network, an interest region pooling layer, and a classification and regression layer; an Efficient Attention module is embedded in the Faster-RCNN model, that is, the feature extraction network adopts the Faster-RCNN model combined with EA, and an improved feature pyramid structure is connected after the Faster-RCNN model. The core improvement of Faster-RCNN is the introduction of RPN, which enables the network to automatically generate candidate regions without relying on external algorithms such as selective search. This greatly improves the detection speed, and due to the use of deep learning feature extraction, it also improves the accuracy of detection. Through end-to-end training, Faster-RCNN can simultaneously optimize feature extraction, candidate region generation, and target detection during training, thereby achieving better performance.

3. The method for detecting storage chip defects as claimed in claim 1, characterized in that: The Faster-RCNN network processes the provided chip surface image as follows: input the image into the network to obtain the corresponding feature map; use the RPN structure to generate candidate boxes, and project the candidate boxes generated by the RPN to the original feature map to obtain the corresponding feature matrix; The feature matrix is ​​scaled to a uniform size through the ROIpooling layer, and then the feature map is flattened to obtain the prediction result.

4. The method for detecting storage chip defects as claimed in claim 1, characterized in that: The EA (Efficient Aggregation) module is a component used to improve the performance of deep learning models, especially when dealing with image segmentation tasks. It enhances the representation ability of the network through more efficient feature aggregation. In the feature extraction stage, after the EA module is added to the residual network, richer contextual information is captured by aggregating feature maps from different layers. In addition, an EA module is applied to the fixed-size feature vector after RoI pooling before the fully connected layer to enhance the accuracy of classification and bounding box regression.

5. The method for detecting storage chip defects as claimed in claim 1, characterized in that: The BiFPN structure is mainly used to detect defects. It improves the performance of the target detection algorithm at different scales by improving the traditional feature pyramid network (FPN) and maintaining efficient computing performance. The BiFPN structure consists of four parts: feature extraction network, BiFPN network module, regression network and classification network. The feature extraction network is used to extract image features, the BiFPN network is used to fuse features at different levels, the regression network is used to predict the position of the target box, and the classification network is used to predict the category of the target.

6. The method for detecting storage chip defects as claimed in claim 2, characterized in that: Due to the small surface area of ​​the chip, poor defect detection, and slow network processing speed, the backbone network is replaced with Mobile-NetV3 as the main network processing to improve speed while maintaining accuracy. The Squeeze-and-excitation (SE) module and H-swish (HS) activation function are introduced in MobileNetV3 to improve model accuracy. The pre-trained MobileNetV3 model is used as the starting point of the feature extractor, which helps to utilize the feature representation learned on a large data set. Then the repeated weighted bidirectional feature pyramid network is re-adjusted according to the backbone network to extract features. When building the network, the channel of each feature layer is adjusted to 256 through 1x1 convolution; then the extracted feature information is input into the RPN region proposal network, and the feature map is input into the ROIPooling layer. The input of the ROIPooling layer is the original feature map and the candidate box output by the RPN, which is equivalent to corresponding each candidate box to different parts of the original feature map, and then these parts are cut off and passed to the ROIPooling layer separately.

7. The method for detecting storage chip defects as claimed in claim 1, characterized in that: Set up the MobileNetV3-Large network module to extract features. By specifying the extracted feature layer and the corresponding channel, combined with the height and width values ​​of each feature layer output, analyze whether the downsampling ratio of the extracted feature layer is correct. Next, build the backbone by instantiating Backbone With FPN; since the output of the submodule under the layer cannot be obtained by directly using Backbone With FPN, create-feature-extractor is used to reconstruct the skeleton to make the output more flexible.

8. The method for detecting storage chip defects as claimed in claim 1, characterized in that: The high-level feature fusion structure includes two processing paths (top-down and bottom-up). Traditional feature fusion is often just a simple superposition, while this network focuses on weighted feature fusion of the input feature map. After this processing, the contribution rate of the feature map can be effectively distinguished. There are three common weighted feature fusion methods. The present invention adopts Fastnormalized fusion with fast training speed. Its formula is as follows:

9. A method for storage chip defect detection as described in claim 3, wherein the extracted feature information is input into the RPN region proposal network to obtain a candidate box, and the feature maps of the region selected by the candidate box and different P2, P3, P4, and P5 are input into ROI (region of interest) Align to obtain a feature map of size 7×7, and finally the feature map is input into two consecutive activation function layers and a fully connected layer to obtain a classification result and a bounding box regression.

10. The method for detecting storage chip defects as claimed in claim 1, characterized in that: After the model is built, according to the characteristics of MobileNetV3, the model can be further optimized by adjusting the resolution factor and width factor to adapt to specific hardware and performance requirements.

Images