A fpc defect classification method based on cnn and transformer

Abstract

The application discloses a kind of FPC defect classification methods based on CNN and TRANSFORMER, the method with CTNet designed based on CNN and TRANSFORMER as classification model, steps are as follows: S1. collect target defect image, pre-process FPC original image data;S2. data enhancement processing is carried out to the image after pre-processing, expand image data set;S3. manual classification is carried out to the image data set after enhancement;S4. image data after artificial classification is proportionally divided into training set and verification set;S5. model training, the well divided image training set is sent into CTNet network model and is trained;S6. the performance evaluation of trained FPC classification detection model is carried out;S7. parameter optimization fine-tuning, in combination with the evaluation result of S6, the model is further optimized.The application realizes the automatic classification of FPC defect, has good universality, and classification speed is fast and low to computer performance requirement, can greatly reduce the production cost of enterprise.

Description

Technical Field

[0001] The present invention relates to the field of flexible circuit board defect classification, and in particular to an FPC defect classification method based on CNN and TRANSFORMER. Background Art

[0002] Flexible printed circuit boards (FPCs), also known as flexible printed circuit boards (FPCs), are thin, lightweight, and flexible. Compared to traditional circuit boards, FPCs occupy less space, significantly reducing package size and weight to meet the requirements of highly integrated and portable electronic products. They also enable three-dimensional circuit routing, enhancing product reliability and reducing assembly costs. However, due to the special materials, high integration density, and complex manufacturing process, FPCs are susceptible to defects caused by factors such as equipment, personnel, and the environment.

[0003] Convolutional neural networks (CNNs) are a common deep learning network architecture, inspired by biological visual cognition. They feature local perception and parameter sharing, resulting in high image recognition accuracy. The TRANSFORMER architecture first appeared in the field of natural language processing and became a mainstream model in this field within four years of its creation. Because each convolution in a CNN focuses only on information near the kernel and cannot incorporate distant information, CNNs are not very accurate in classifying FPC defects, which have limited feature information, a large number, and a wide variety. Traditional image processing methods also perform poorly on these types of FPC defect images. To address this issue, we propose an FPC defect classification method based on CNNs and TRANSFORMER. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology. The present invention provides an FPC defect classification method based on CNN and TRANSFORMER, which realizes the automatic classification of the main defects of FPC, saves the labor cost of the enterprise, has a fast model classification speed, and has low requirements on computer performance, which can greatly reduce the production cost of the enterprise and improve the efficiency of defect detection.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: a FPC defect classification method based on CNN and TRANSFORMER, comprising the following steps:

[0006] S1. Collect defect images of flexible circuit boards on site and preprocess the FPC raw image data;

[0007] S2. Perform data enhancement on the preprocessed image;

[0008] S3. Classify the enhanced image data;

[0009] S4. Divide the classified image dataset into a training set and a validation set in proportion;

[0010] S5. Model training: The divided image training set is fed into the constructed CTNet network model for training to produce an FPC defect classification model.

[0011] S6. Evaluate the performance of the trained FPC defect classification model.

[0012] S7. Parameter optimization, combined with the evaluation results of S6, further optimize and fine-tune the FPC defect classification model.

[0013] A further solution is that the step S1 of preprocessing the image data specifically includes:

[0014] Image data is cropped. Image cropping is used to crop the original high-resolution FPC image to obtain the part that needs to be classified, namely the ROI image, and crop the ROI image into K parts. The resolution of the K cropped images is between 512*512 and 1024*1024. K is a value selected from the natural number set {2, 4, 8, 16, 24} according to the original high-resolution FPC image.

[0015] A further solution is that the data enhancement processing method in step S2 adopts mirroring, brightness adjustment, translation and random deletion to increase the training data set.

[0016] A further solution is that the step S3 of classifying the image data specifically includes:

[0017] According to the types of FPC defects in actual production, images of the same type of defects are divided into the same folder, and the folder is named according to this defect type.

[0018] A further solution is that the method for dividing the data set specifically includes:

[0019] When the data set N after data augmentation is greater than or equal to 20,000, the ratio of the training set to the validation set is 3 to 1. When the data set 20,000>N≥10,000 after data augmentation is greater than or equal to 10,000, the ratio of the training set to the validation set is 4 to 1. When the data set 10,000>N after data augmentation is greater than or equal to 10,000, the ratio of the training set to the validation set is 5 to 1.

[0020] A further solution is to build a CTNet network model that specifically includes:

[0021] The input image is first scaled down to a dimension of 448*448*3, and then passed through a 3*3 convolutional layer and a maximum pooling layer to obtain a feature map with a dimension of 224*224*3.

[0022] A further solution is to first perform a 3*3 convolution layer on the feature map after dimensionality reduction to increase the channel dimension to 128; then input it into the convolution block, which consists of 4 branches, the first branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, the second branch is 2 3*3 convolution layers, the third branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, and the fourth branch is a maximum pooling layer; the feature map after dimension enhancement passes through the first 3 branches respectively, and then the feature maps generated by the first 3 branches are feature fused, and then concat fused with the feature map generated by the fourth branch. After fusion, it enters the maximum pooling layer to generate a feature map with a dimension of 112*112*128, and the obtained feature map is continuously passed through three CNN blocks with the same structure to finally generate a feature map with a dimension of 14*14*1024.

[0023] A further solution is that after the step of generating a feature map with a dimension of 14*14*1024, the following steps are further included:

[0024] First, the feature map generated after passing through four CNN blocks is Flattened to convert the 3D feature map into a 2D feature vector, which is then sent to the TRANSFORMER block. The TRANSFORMER block includes a self-attention layer, two fully connected layers, and an activation function layer. The 2D feature vector block first passes through the self-attention layer, then through a fully connected layer, an activation function layer, and a fully connected layer. The output result is then fused with the 2D feature vector that has not passed through the self-attention layer. The four TRANSFORMER block operations are repeated to finally generate a 2D feature vector with a dimension of 196*1024. The 2D feature vector with a dimension of 196*1024 then passes through two TRANSFORMER blocks with the same structure. The two TRANSFORMER blocks with the same structure are repeatedly superimposed n times. The number of superpositions n is calculated by formula (1). N is the number of defects that need to be detected in the FPC. The dimension of the final generated 2D feature vector is 196*1536.

[0025] (1).

[0026] A further solution is that the step S6 performs performance evaluation on the trained FPC defect classification model, specifically including:

[0027] The average accuracy ACC of the validation set is used as an indicator; if the average accuracy ACC of the model is ≥ 95%, it is determined that the model can accurately classify FPC defects; otherwise, the model needs to be optimized and fine-tuned.

[0028] A further solution is that the further optimization and fine-tuning of the FPC defect classification model in step S7 is achieved by increasing the training batch, increasing the number of training samples, modifying the learning rate, and the number of iterations.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The present invention solves the problem of low accuracy of FPC defect classification by traditional algorithms and deep learning models, saving labor costs;

[0031] (2) The present invention realizes the automatic classification of FPC defects;

[0032] (3) This invention solves the problem of slow inference speed of deep learning models and can meet the needs of online classification in industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of an FPC defect classification method based on CNN and TRANSFORMER according to an embodiment of the present invention;

[0034] Figure 2 This is a convolution block structure diagram of an embodiment of the present invention;

[0035] Figure 3 This is a diagram of the TRANSFORMER block structure of an embodiment of the present invention;

[0036] Figure 4 This is a network structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The purpose, technical solutions and advantages of the present invention are further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are not only used to explain the present invention, but also do not limit the scope of protection of the present invention.

[0038] like Figure 1 As shown in the figure, a FPC defect classification method based on CNN and TRANSFORMER is shown in the figure. The specific steps are as follows:

[0039] Step S1, collect the defect image of the flexible circuit board on site, and perform image cropping on the original FPC image data. The specific method of image cropping is to crop the original high-resolution FPC image to obtain the part that needs to be classified, that is, the ROI image, and then crop the ROI image into K parts. The maximum resolution of the K cropped images is 1024*1024, and the minimum resolution is 512*512. K is selected from the natural number set {2, 4, 8, 16, 24} according to the original high-resolution FPC image.

[0040] Step S2: perform data enhancement processing on the cropped image, wherein the data enhancement processing method uses mirroring, brightness adjustment, translation and random deletion to expand the data set by 5 times.

[0041] Step S3: manually classify the enhanced image data, wherein the classification method is to classify images with the same type of defects into the same folder based on the type of FPC defects in actual production, and the folder is named according to the defect type.

[0042] Step S4: Divide the classified image dataset into a training set and a validation set in proportion. When the dataset N after data enhancement processing is greater than or equal to 20,000, the ratio of the training set to the validation set is 3 to 1. When the dataset 20,000>N>10,000 after data enhancement processing is greater than or equal to 10,000, the ratio of the training set to the validation set is 4 to 1. When the dataset 10,000>N after data enhancement processing is greater than or equal to 10,000, the ratio of the training set to the validation set is 5 to 1.

[0043] Step S5: Model training: send the divided image training set to the CTNet network model for training to train the FPC defect classification model; please combine Figure 2 、 Figure 3 and Figure 4 Understand the CTNet network structure, Figure 2 and Figure 3 The plus sign (+) indicates element-by-element addition for feature fusion. The specific construction method of the CTNet model is as follows:

[0044] First, the image input to the CTNet model is reshaped to 448*448*3, and then passes through a 3*3 convolution layer and a maximum pooling layer to generate the feature map C0, the dimension of C0 is 224*224*3.

[0045] Before the feature map C0 enters the first CNN block, it needs to undergo another 3*3 convolution layer to increase the channel dimension to 128; the convolution block consists of 4 branches, the first branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, the second branch is two 3*3 convolution layers, the third branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, and the fourth branch is a maximum pooling layer; after the feature map C0 is dimensioned, it passes through the first 3 branches respectively, and then the feature maps generated by the first 3 branches are fused. , and then concat fused with the feature map generated by the fourth branch. After fusion, it enters the maximum pooling layer to generate feature map C1. The dimension of C1 is 112*112*128. Feature map C1 passes through the CNN block with the same structure to generate feature map C2. Feature map C2 passes through the CNN block with the same structure to generate feature map C3. Feature map C3 passes through the CNN block with the same structure to generate feature map C4. The dimension of C2 is 56*56*256, the dimension of C3 is 28*28*512, and the dimension of C4 is 14*14*1024.

[0046] Before entering the TRANSFORMER block, the feature map C4 is first flattened to convert the 3D feature map into a 2D feature vector, which is then sent to the TRANSFORMER block. The TRANSFORMER block is mainly composed of a self-attention layer, two fully connected layers, and an activation function layer. The 2D feature vector block first passes through the self-attention layer, then through a fully connected layer, an activation function layer, and a fully connected layer. The output result is then fused with the 2D feature vector that has not passed through the self-attention layer. The operation of 4 TRANSFORMER blocks is repeated to finally generate a 2D feature vector C5 with a dimension of 196*1024. The 2D feature vector C5 then passes through two TRANSFORMER blocks with the same structure. These two TRANSFORMER blocks with the same structure can be repeatedly superimposed n times. The number of superpositions n is calculated by formula (1). N is the number of defects that the FPC needs to detect. Finally, the dimension of the generated 2D feature vector C6 is 196*1536.

[0047] (1)

[0048] Step 6: Perform performance evaluation on the trained FPC defect classification model. The performance evaluation of the trained FPC defect classification model mainly uses the average accuracy ACC of the validation set as an indicator. If the average accuracy ACC of the model is ≥ 95%, it is determined that the model can accurately classify FPC defects. Otherwise, the model needs to be optimized and fine-tuned.

[0049] Step 7: Parameter optimization and fine-tuning. Combined with the evaluation results of step 6, the FPC defect classification model is further optimized and fine-tuned, mainly by increasing the training batch, increasing the number of training samples, modifying the learning rate, and the number of iterations. Example

[0050] In this embodiment, the FPC defect classification method based on CNN and TRANSFORMER first performs an image cropping operation on the collected original FPC image, then performs data enhancement on the pre-processed image to expand the FPC image data volume, then manually classifies the enhanced image data, divides the labeled data set into a training set and a validation set according to the proportion, then sends the divided data set to the model training, and finally evaluates the trained model to determine whether the model's classification accuracy meets the requirements. The specific steps are as follows:

[0051] Step 1: Collect the target defect image on site and crop the FPC original image data.

[0052] Step 2: Perform data augmentation on the preprocessed images. The data augmentation method uses mirroring, brightness adjustment, translation, and random deletion to expand the dataset by 5 times.

[0053] Step 3: Manual classification, classify the enhanced image data. In this embodiment, the types of FPC defects in actual production are manually classified, and images of the same type of defects are divided into the same folder, and the folder is named this defect type.

[0054] Step 4: Divide the manually classified image dataset into a training set and a validation set in proportion. The method of dividing the dataset is that when the dataset N after data augmentation is N≥20,000, the ratio of the training set to the validation set is 3 to 1; when the dataset after data augmentation is 20,000>N≥10,000, the ratio of the training set to the validation set is 4 to 1; when the dataset after data augmentation is 10,000>N, the ratio of the training set to the validation set is 5 to 1.

[0055] Step 5: Model training: send the divided image training set to the built CTNet network model for training to train the FPC defect classification model.

[0056] Step 6: Perform performance evaluation on the trained FPC defect classification model. The performance evaluation of the trained FPC defect classification model mainly uses the average accuracy ACC of the validation set as an indicator. If the average accuracy ACC of the model is ≥ 95%, it is determined that the model can accurately classify FPC defects. Otherwise, the model needs to be optimized and fine-tuned.

[0057] Step 7: Parameter optimization and fine-tuning. Combined with the evaluation results of step 6, the FPC defect classification model is further optimized and fine-tuned, mainly by increasing the training batch, increasing the number of training samples, modifying the learning rate, and the number of iterations.

[0058] This example uses the Pytorch 1.9 version of the deep learning framework. The experimental computer hardware configuration is: AMD Ryzen 9 3900X 12-core 24-thread processor, GeForce RTX 3090 graphics card, 32G running memory, 500G NVME solid-state drive; the experimental software environment is 64-bit Windows 10 Professional Edition, Pycharm integrated development environment, CUDA version 11.1, CUDNN version 8.0.4.30. In this environment, the model is trained using the GeForce RTX 3090 graphics card and predicted using the Ryzen 9 3900X 12-core 24-thread processor. The final results are shown in Table 1.

[0059] Table 1. FPC data information

[0060] Native resolution Crop resolution Defect image Data augmentation training set Validation set FPC1 5120*5120 1024*1024 1820 photos 9,100 sheets 7584 1516 FPC2 6576*4384 512*512 2214 11,070 photos 8856 2214

[0061] Table 2. Experimental results

[0062] Glue overflow Copper foreign body Rich crush Circuit breaker Short Circuit Average Accuracy ACC Average detection time FPC1 96.7% 98.6% 99.1% 97.5% 95.6% 99% 96.3% 97.5% 21ms FPC2 97.4% 98.1% 98.8% 97.9% 96.6% 98.7% 97.1% 97.8% 15ms

[0063] As can be seen from Table 2, after training, the classification method proposed in the present invention has an average accuracy ACC of more than 97% for classifying defects in two types of FPC images (the first type of FPC image (FPC1) and the second type of FPC image (FPC2)). The average detection time for predicting FPC1 defect images using the CPU is 21 milliseconds, and the average detection time for predicting FPC2 defect images using the CPU is 15 milliseconds. This shows that the present invention has high classification accuracy and fast classification speed for FPC appearance defects.

[0064] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and that the scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A FPC defect classification method based on CNN and TRANSFORMER, characterized in that: The steps include: S1. Collect defect images of flexible circuit boards on site and preprocess the FPC raw image data; S2. Perform data augmentation on the preprocessed image; S3. Classify the enhanced image data; S4. Divide the classified image dataset into a training set and a validation set in proportion; S5. Model training: The divided image training set is fed into the constructed CTNet network model for training to train an FPC defect classification model, which includes: The input image is first scaled down to a dimension of 448*448*3, and then passed through a 3*3 convolution layer and a maximum pooling layer to obtain a feature map with a dimension of 224*224*3. First, the feature map after dimensionality reduction is subjected to a 3*3 convolution layer to increase the channel dimension to 128; then it is input into the convolution block, which consists of four branches. The first branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, the second branch is two 3*3 convolution layers, the third branch is a 1*1 convolution layer, a 3*3 convolution layer and a 1*1 convolution layer, and the fourth branch is a maximum pooling layer; the feature map after dimension enhancement passes through the first three branches respectively, and then the feature maps generated by the first three branches are feature fused, and then concat fused with the feature map generated by the fourth branch. After fusion, it enters the maximum pooling layer to generate a feature map with a dimension of 112*112*128, and the obtained feature map is continuously passed through three CNN blocks with the same structure to finally generate a feature map with a dimension of 14*14*1024; The feature map generated after passing through the four CNN blocks is flattened to convert the 3D feature map into a 2D feature vector, which is then sent to the TRANSFORMER block. The TRANSFORMER block includes a self-attention layer, two fully connected layers, and an activation function layer. The 2D feature vector block first passes through the self-attention layer, then through a fully connected layer, an activation function layer, and a fully connected layer. The output result is then fused with the 2D feature vector that has not passed through the self-attention layer. The four TRANSFORMER block operations are repeated to finally generate a 2D feature vector with a dimension of 196*1024. The 2D feature vector with a dimension of 196*1024 then passes through two TRANSFORMER blocks with the same structure. The two TRANSFORMER blocks with the same structure are repeatedly superimposed n times. The number of superpositions n is calculated by formula (1). N is the number of defects that need to be detected in the FPC. The dimension of the final generated 2D feature vector is 196*1536. (1)； S6. Perform performance evaluation on the trained FPC defect classification model; S7. Parameter optimization. Combined with the evaluation results of S6, the FPC defect classification model is further optimized and fine-tuned. Further optimization and fine-tuning is achieved by increasing the training batch, increasing the number of training samples, modifying the learning rate, and the number of iterations.

2. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: The step S1 of pre-processing the image data specifically includes: Image data is cropped. Image cropping is used to crop the original high-resolution FPC image to obtain the part that needs to be classified, namely the ROI image, and crop the ROI image into K parts. The resolution of the K cropped images is between 512*512 and 1024*1024. K is a value selected from the natural number set {2, 4, 8, 16, 24} according to the original high-resolution FPC image.

3. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: The data enhancement processing method in step S2 adopts mirroring, brightness adjustment, translation and random deletion to increase the training data set.

4. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: The step S3 of classifying the image data specifically includes: According to the types of FPC defects in actual production, images of the same type of defects are divided into the same folder, and the folder is named according to this defect type.

5. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: The method for dividing the data set specifically includes: When the data set N after data augmentation is greater than or equal to 20,000, the ratio of the training set to the validation set is 3 to 1. When the data set 20,000>N≥10,000 after data augmentation is greater than or equal to 10,000, the ratio of the training set to the validation set is 4 to 1. When the data set 10,000>N after data augmentation is greater than or equal to 10,000, the ratio of the training set to the validation set is 5 to 1.

6. The FPC defect classification method based on CNN and TRANSFORMER according to claim 1, characterized in that: The step S6 of performing performance evaluation on the trained FPC defect classification model specifically includes: The average accuracy ACC of the validation set is used as an indicator; if the average accuracy ACC of the model is ≥ 95%, it is determined that the model can accurately classify FPC defects; otherwise, the model needs to be optimized and fine-tuned.

Images