Flexible ic substrate surface defect recognition method based on improved FCOS network

Abstract

The application discloses a kind of based on improved FCOS network's flexible IC substrate surface defect identification method, including collection picture, expand picture dataset, mark picture, train and verify to improved FCOS network, utilize FCOS network to identify and the like steps. Among them, the improvement of FCOS network includes: in the up-sampling part in FCOS network, the nearest neighbor interpolation algorithm of original is replaced with PixelShuffle algorithm;Channel attention module is added before each detection head Head in FCOS network;Maxpool_BPA link is introduced after the FPN of FCOS network. Finally, by testing the improved FCOS network in the application, in the scene of detecting flexible IC substrate surface defect, compared with the recall rate of original network, mAP is increased by 4% and 4.9%.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection, in particular to a flexible IC substrate surface defect recognition method based on an improved FCOS network. BACKGROUND

[0002] In recent years, with the rapid development of technology, electronic devices have spread to every corner of our lives, and in this process, flexible IC substrate (FICS) is playing an irreplaceable role. At the same time, as electronic products continue to move towards miniaturization, the circuit is refined to the micron scale, making the management standards for electronic product quality and defects more stringent during the manufacturing process.

[0003] At present, the traditional recognition method has been unable to meet the high standard reliability requirements of flexible IC substrate. First, the traditional manual inspection method not only relies on a large number of human resources and low efficiency, but also is easily affected by the mental state and subjective judgment of the inspector. Secondly, the traditional image processing method (CN201510775840.X) such as template matching, morphological processing, wavelet transform and the like relies too much on stable detection environment and single category, and cannot well adapt to the complex environment and multiple categories in industrial production.

[0004] At the same time, with the rapid development of neural networks, people can extract image features and recognize them by using neural networks, which can recognize more types of defects and achieve higher recognition rates.

[0005] Under such circumstances, in order to solve these problems and achieve better automated defect detection, combined with the development trend of neural networks, an identification defect method based on a high-accuracy detection algorithm is invented, which has important practical application significance. SUMMARY

[0006] The present application provides a flexible IC substrate surface defect recognition method based on an improved FCOS network, which is applied to the appearance defect precision detection and analysis of flexible IC substrate.

[0007] The present application is realized at least by one of the following technical solutions.

[0008] The flexible IC substrate surface defect recognition method based on the improved FCOS network comprises the following steps:

[0009] Step one, collect the surface defect pictures of the measured flexible IC substrate;

[0010] Step two, expand the collected defect pictures;

[0011] Step three, labeling the obtained defective picture to obtain a surface defect dataset of the flexible IC substrate;

[0012] Step four, training and testing the improved FCOS network using the surface defect dataset of the flexible IC substrate;

[0013] Step five, defect detection on the picture to be detected using the trained improved FCOS network.

[0014] Further, the expansion method includes random rotation, random cropping, and random color transformation.

[0015] Further, the labeling method of using a rectangular bounding box to frame the defect and note the defect type is adopted, and the labeled defect types include open circuit, short circuit, and exposed copper defects.

[0016] Further, the image dataset used in training is randomly divided into a training set and a test set in proportion.

[0017] Further, the improved FCOS network is tested, and the recall (R), precision (P), and mean average precision (mAP) are selected to evaluate the detection ability of the network.

[0018] Further, the improved FCOS network is tested, and the recall (R), precision (P), and mean average precision (mAP) are selected to evaluate the detection ability of the network.

[0019] Further, the improved FCOS network includes:

[0020] (1) In the up-sampling part of the FCOS network, the PixelShuffle algorithm is used instead of the nearest neighbor interpolation algorithm;

[0021] (2) A channel attention module (CAM) is added before each detection head Head in the FCOS network;

[0022] (3) Maxpool_BPA is introduced after the FPN of the FCOS network.

[0023] Further, the PixelShuffle algorithm includes: performing convolution operation on the (C, H, W) feature vector, and then reorganizing the obtained (r 2 C, H, W) feature vector to obtain a (C, rH, rW) feature vector to complete the up-sampling of the feature vector (C, H, W), where r is the sampling multiple, C is the channel number of the input feature, H is the height of the input feature, and W is the width of the input feature.

[0024] Further, the channel attention module (CAM) comprises: descriptors are generated for each channel by performing maximum value pooling and average value pooling on the input features, then 1*1 convolution operations are performed on the two descriptors respectively, the two feature vectors obtained are superimposed, attention weights are generated by using a sigmoid activation function, and finally the calculated attention weights are multiplied with the original input features to obtain weighted features.

[0025] Further, the Maxpool_BPA link comprises: 4 times of Maxpooling operations are performed on the output of the P3 layer in the FPN, and then the P4, P5, P6 and P7 layers in the FPN are directly added respectively.

[0026] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0027] 1. In the improved FCOS network in the present application, the PixelShuffle algorithm is used to replace the original nearest neighbor interpolation algorithm in the up-sampling part, so that the deep information and the shallow information can be better fused.

[0028] 2. In the improved FCOS network in the present application, a channel attention module (CAM) is added in front of each detection head Head, which solves the defects that small targets are easily lost and disturbed by the background during sampling of the FCOS network, and enhances the detection ability of the FCOS network for the surface defects of the flexible IC substrate.

[0029] 3. In the improved FCOS network in the present application, the Maxpool_BPA link is introduced after the FPN, so that the bottom information flow of the FCOS network is faster, and the positioning ability of the entire feature level is further enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flow chart of the method for identifying surface defects of a flexible IC substrate based on the improved FCOS network of the embodiment;

[0031] Figure 2 is a structural schematic diagram of the original FCOS network;

[0032] Figure 3 is a structural diagram of the improved FCOS network of the embodiment of the present application;

[0033] Figure 4 is a structural schematic diagram of the PixelShuffle algorithm of the embodiment;

[0034] Figure 5 is a structural schematic diagram of the CAM of the embodiment;

[0035] Figure 6 is a schematic diagram of introducing Conv_BPA into the FCOS network according to an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of introducing Maxpool_BPA into the FCOS network according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in conjunction with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0038] As shown in the drawings, the flexible IC substrate surface defect recognition method based on the improved FCOS network of the present embodiment comprises the following steps: Figure 1 S1, using an existing flexible IC substrate image acquisition device, acquiring a surface defect picture of a measured flexible IC substrate;

[0039] S2, using a traditional image processing method, including but not limited to random rotation, random cropping, random color transformation, etc., to expand the collected defect picture;

[0040] S3, according to the existing flexible IC substrate production standard, using a rectangular bounding box to frame the defects and annotating the defect type annotation method to the obtained defect picture, the annotated defect type including but not limited to open circuit, short circuit, exposed copper, etc. Defects, to obtain a surface defect dataset of the flexible IC substrate;

[0041] S4, using the dataset to train and test the improved FCOS network;

[0042] S5, using the trained improved FCOS network to detect defects in the picture to be detected, and according to the defect result obtained by the network, judging whether the tested flexible IC substrate meets the industrial production standard.

[0043] As an embodiment, the expanded picture is resized to 800x1333 to adapt to the best size of the FCOS network for recognizing the flexible IC substrate picture.

[0044] Preferably, in step S3, the Labelimg tool is used to annotate the anchor box of the obtained picture, and the obtained label type is in Pascal VOC format, and the label type of the defect includes five types of short, open, gnawing hole, exposed copper and undercut, wherein the short label represents short circuit, the open label represents open circuit, the gnawing hole label represents hole biting, the exposed copper label represents exposed copper, and the undercut label represents undercut.

[0045] ​

[0046] As an embodiment, in the process of training and testing the improved FCOS network in step S4, the surface defect data set of the flexible IC substrate obtained in step S3 is randomly divided into a training set and a test set in a ratio of 7:3, training and testing are performed on a GeForce RTX 4090 GPU, the network is optimized in a Stochastic Gradient Descent (SGD) manner, and the parameters used during training are shown in Table 1.

[0047] Table 1 Training parameter settings

[0048] Parameter Value Epochs 150 Batch_size 4 Learning_rate 0.002 Momentum 0.9 Weight_decay 0.0001

[0049] Preferably, in step S4, the recall (Recall, R), precision (Precision, P) and mean average precision (mean Average Precision, mAP) are selected to evaluate the detection ability of the FCOS network, and the specific calculation formula is as follows:

[0050]

[0051]

[0052]

[0053]

[0054] wherein the prediction box is a bounding box generated by a target detection model or an object recognition model, used to represent the position and boundary of the target or object in the image that the model believes exists; the actual box is the annotation information from the data set, used to represent the position and boundary of the real target or object in the image; the Intersection Over Union (IOU) is a commonly used measurement standard for evaluating target detection performance, generally used to measure the degree of overlap between the model's predicted bounding box and the real target bounding box; Intersection refers to the overlapping area of the prediction box and the actual box, Union refers to the total area of the prediction box and the actual box combined together, TP refers to the number of prediction boxes whose IOU with the actual box is greater than the threshold (only one actual box is counted), FP refers to the number of prediction boxes whose IOU with the actual box is less than or equal to the threshold or the number of redundant detection boxes for the same actual box, FN refers to the number of prediction boxes that do not detect the actual box, and classes refer to the types of targets to be detected, is obtained by calculating the area under the two-dimensional curve with Precision and Recall as the vertical and horizontal axis coordinates.

[0055] Particularly, in the specific implementation of the present application, the threshold size of IOU is set to 0.5, and the class is 5.

[0056] The original FCOS network used in step S4 mainly includes two parts of feature extraction and detection head. In the feature extraction part, firstly, a ResNet50 backbone network is used to extract multi-scale information in the input image, and three scale feature maps P3, P4 and P5 are output. Then, through the FPN structure, convolution and up-sampling are used to fuse the three scale feature maps, and convolution and ReLU activation function are used to down-sample the P5 feature map to obtain the P6 and P7 scale feature maps. In the detection head part, three branches similar in structure are used to predict the class (Classification), anchor point (Center-ness) and the distance (Regression) of the four edges of the anchor point to the target bounding box. Therefore, the feature channel number output after the detection head is C, 1 and 4 respectively, wherein C represents the number of defect recognition categories; the structure of one branch includes four convolution layers + GN + ReLU activation function and one convolution layer, and the Regression branch and the Center-ness branch share the four convolution layers + GN + ReLU activation function.

[0057] The structure diagram of the original FCOS network is as shown in Figure 2 , wherein Conv is a convolution operation, the parameters k, s and p in Conv represent kernel size, stride and padding respectively, GN is a normalization method of deep learning, and ReLU is an activation function of deep learning.

[0058] The improvement of the FCOS network in step S4 is as follows:

[0059] (1) In the up-sampling part of the FCOS network, the original nearest neighbor interpolation algorithm is replaced by the PixelShuffle algorithm;

[0060] (2) A channel attention module (CAM) is added before each detection head Head in the FCOS network;

[0061] (3) The Maxpool_BPA link is introduced after the FPN of the FCOS network.

[0062] Further, the structure diagram of the improved FCOS network in the present application is as shown in Figure 3As shown in the figure, PS is a PixelShuffle algorithm, CAM is a channel attention module, Maxpool_BPA is a bottom-up path augmentation link based on maximum pooling, and other parameter meanings are consistent with Figure 2 .

[0063] The PixelShuffle algorithm mainly performs convolution operation on the (C, H, W) feature vector, and then reorganizes the obtained (r 2 C, H, W) feature vector into a (C, rH, rW) feature vector, so as to complete the up-sampling of the feature vector (C, H, W), wherein r is the sampling multiple, C is the channel number of the input feature, H is the height of the input feature, and W is the width of the input feature.

[0064] In particular, in the specific implementation of the present application, in order to match the feature map size obtained by the nearest neighbor interpolation algorithm in the original network, the sampling multiple of the PixelShuffle algorithm is set to 2.

[0065] Further, a structural schematic diagram of the PixelShuffle algorithm is shown in Figure 4 The schematic diagram is used to better show the specific operation during reorganization, taking a feature with C, H and W all being 1 as input, and in the (r*r, 1, 1) feature in the middle, the features from the 1st layer to the r 2 th layer are represented by serial numbers 1, 2, …, r 2 , respectively, wherein r is the sampling multiple, and Conv is the convolution operation.

[0066] The CAM mainly generates descriptors for each channel by performing maximum value pooling and average value pooling on the input feature, then performs 1*1 convolution operation on the two descriptors respectively, and stacks the two obtained feature vectors to generate attention weights using a sigmoid activation function, and finally multiplies the calculated attention weights with the original input feature to obtain the weighted channel feature, and the specific calculation formula is as shown below:

[0067] M(F) = σ(Conv(AvgPool(F) + Conv(MaxPool(F))

[0068] , wherein M() represents the CAM operation, F represents the input feature, sigma is the sigmoid activation function, Conv is the 1*1 convolution operation, AvgPool is the average value pooling operation, and MaxPool is the maximum value pooling operation.

[0069] Further, a structural schematic diagram of the CAM is shown in Figure 5 , wherein Conv is the 1*1 convolution operation, AvgPool is the average value pooling operation, and MaxPool is the maximum value pooling operation.

[0070] Further, in the surface defect data set of the flexible IC substrate, ablation experiments are conducted on the FCOS network, the PixelShuffle algorithm and the CAM, and the experimental results are shown in Table 2 below.

[0071] Table 2 PixelShuffle algorithm and CAM ablation experiment results table

[0072] Model PixelShuffle CAM Recall Precision mAP 1 90.6％ 93.6％ 89.1％ 2 √ 92.4％ 91.6％ 90.5％ 3 √ 93.2％ 92.8％ 91.9％ 4 √ √ 94.5％ 91.3％ 93.3％

[0073] From the above experimental results, it can be seen that the best detection effect is the No. 4 model, and it can be found that adding the PixelShuffle algorithm and the CAM in the FCOS network can make the detection performance of the FCOS network better.

[0074] The Maxpool_BPA link is similar to the bottom-up path augmentation (BPA) link in PANet. In PANet, in order to better preserve shallow feature information and make the bottom information flow faster, the BPA link is added after the FPN link, aiming to realize multi-layer feature fusion to improve the target detection performance.

[0075] The specific structure of the BPA link in PANet is as follows: first, the bottom layer feature map is convolved to adjust its size to match the size of the upper layer feature map, then the two feature maps are added to obtain a new feature map of the upper layer, and then the new feature map is convolved again and the above operation is performed with the feature map of the higher layer. Starting from the bottom layer, such operations are performed layer by layer to generate a new feature map of each layer, achieving the goal of feature fusion. This process helps to improve the network's perception of information of different scales and contexts, thereby improving the accuracy and performance of image detection and segmentation.

[0076] In the FCOS network, since its P6 and P7 layers only perform one bottom-up information transmission, and do not perform a second top-down information transmission, in order to pass more valuable information to each layer, the present application first attempts to introduce the Conv_BAP link into the FCOS network, and the specific structure is: directly performing 4 convolution operations on the output of the P3 layer in the FPN to obtain feature layers with sizes matching the sizes of the P4, P5, P6 and P7 layers, and then adding them to the P4, P5, P6 and P7 layers in the FPN to obtain new feature layers.

[0077] Further, the structure diagram of the Conv_BPA link and the way it is added to the FCOS network are as follows: Figure 6As shown, the link is based on the common BPA link, and the new feature map obtained from the bottom layer is no longer convolved and added to the upper layer feature, but multiple convolution operations are directly performed from the bottom layer, and the features obtained are added to the features of each layer, wherein Conv_BPA is a convolution-based bottom-up path enhancement link, and the meanings of other parameters are consistent with those of the BPA link. Figure 2

[0078] After that, in order to reduce the parameter amount and simplify the FCOS network, the Conv operation in the above Conv_BPA link is replaced by the Maxpooling operation, and the Maxpool_BPA link is obtained.

[0079] Further, the structural diagram of the Maxpool_BPA link and the way it is added to the FCOS network are as shown in Figure 7 , wherein Maxpool_BPA is a maximum pooling-based bottom-up path enhancement link, and the meanings of other parameters are consistent with those of the BPA link. Figure 2

[0080] Further, in the surface defect data set of the flexible IC substrate, the FCOS network and the PixelShuffle algorithm+CAM, the Conv_BPA link and the Maxpool_BPA link are subjected to ablation experiments, and the experimental results are shown in Table 3.

[0081] Table 3 PixelShuffle algorithm+CAM, Conv_BPA link and Maxpool_BPA link ablation experiment results

[0082] Model PixelShuffle+CAM Conv_BPA Maxpool_BPA Recall Precision mAP 1 90.6％ 93.6％ 89.1％ 2 √ 91.2％ 91.9％ 90.1％ 3 √ 91.2％ 89.7％ 88.7％ 4 √ 94.5％ 91.3％ 93.3％ 5 √ √ 92.2％ 92.4％ 90.6％ 6 √ √ 94.6％ 90.5％ 94.0％

[0083] From the above experimental results, it can be seen that in models 1-3, the best detection effect is model 2, and it can be found that when the PixelShuffle algorithm+CAM is not added to the FCOS network, the Conv_BPA link is added to the FCOS network, which is better; in models 4-6, the best detection effect is model 6, and it can be found that after the PixelShuffle algorithm+CAM is added to the FCOS network, the Maxpool_BPA link is added to the FCOS network, which is better, and the parameter amount of the link is less than that of the Conv_BPA link, so that the improved FCOS network is more lightweight, so the PixelShuffle algorithm+CAM+Maxpool_BPA link is adopted in the specific implementation of the present application.

[0084] ​​Further, in the surface defect data set of the flexible IC substrate, in order to further verify that the above-mentioned mode is the best scheme for implementation, the FCOS network and the PixelShuffle algorithm, CAM and Maxpool_BPA links are subjected to ablation experiments, and the experimental results are shown in Table 4.

[0085] Table 4 PixelShuffle algorithm, CAM and Maxpool_BPA link ablation experiment results table

[0086] Model PixelShuffle CAM Maxpool_BPA Recall Precision mAP 1 90.6％ 93.6％ 89.1％ 2 √ 92.4％ 91.6％ 90.5％ 3 √ 93.2％ 92.8％ 91.9％ 4 √ √ 93.2％ 87.0％ 92.6％ 5 √ √ 94.5％ 91.3％ 93.3％ 6 √ √ √ 94.6％ 90.5％ 94.0％

[0087] It can be known from the above experimental results that model 6 is the best model for detection effect, and compared with the original network, the recall rate of the improved FCOS network in the present application is increased by 4% and the mAP is increased by 4.9% in the scene of detecting the surface defects of the flexible IC substrate.

[0088] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement modes and shall be included in the protection scope of the present application.

Claims

1. A flexible IC substrate surface defect recognition method based on an improved FCOS network, characterized in that, The method comprises the following steps: Step one, collecting the surface defect pictures of the flexible IC substrate to be tested; Step two, expanding the collected defect pictures; Step three, labeling the obtained defect pictures to obtain the surface defect dataset of the flexible IC substrate; Step four, training and testing the improved FCOS network by using the surface defect dataset of the flexible IC substrate; Step five, detecting defects in the pictures to be tested by using the trained improved FCOS network; The improved FCOS network comprises: (1) replacing the nearest neighbor interpolation algorithm with the PixelShuffle algorithm in the up-sampling part of the FCOS network; The PixelShuffle algorithm includes: by... Perform convolution on the feature vectors, and then... The feature vectors are recombined to obtain a The feature vector, to complete the feature vector upsampling, where This is the sampling multiple. The number of channels for the input feature. For high input features, The width of the input feature; (2) adding a channel attention module before each detection head Head in the FCOS network, the channel attention module comprising: generating descriptors for each channel by performing maximum value pooling and average value pooling on the input features, then performing 1*1 convolution operation on the two descriptors respectively, and stacking the obtained two feature vectors to generate attention weights by using a sigmoid activation function, and finally multiplying the calculated attention weights with the original input features to obtain weighted features; (3) introducing a Maxpool_BPA link after the FPN of the FCOS network, the Maxpool_BPA being a bottom-up path enhancement link based on maximum pooling.

2. The flexible IC substrate surface defect recognition method based on the improved FCOS network according to claim 1, characterized in that, The expansion method comprises random rotation, random cropping and random color transformation.

3. The flexible IC substrate surface defect recognition method based on the improved FCOS network according to claim 1, characterized in that, The labeling method adopts a rectangular bounding box to frame the defects and notes the defect types, and the labeled defect types include open circuit, short circuit and copper exposure defects.

4. The flexible IC substrate surface defect recognition method based on the improved FCOS network according to claim 1, characterized in that, The image dataset used in the training is randomly divided into a training set and a test set in proportion.

5. The flexible IC substrate surface defect recognition method based on the improved FCOS network according to claim 1, characterized in that, The improved FCOS network is tested, and the recall rate, precision rate and average precision are selected to evaluate the detection ability of the network.

6. The flexible IC substrate surface defect recognition method based on the improved FCOS network according to claim 1, characterized in that, The trained improved FCOS network is used to detect defects in the pictures to be tested, and whether the tested flexible IC substrate meets the industrial production standard is judged according to the defect results obtained by the network.

Images