A two-stage multi-classification industrial image defect detection method based on a twin residual network

Abstract

The application discloses a two-stage multi-classification defect detection method based on a twin residual network, wherein an image to be detected and template information thereof are jointly input into a network, difference features are extracted, and the difference features are combined into a residual network taking the template information as input according to a residual network design idea to extract multi-classification features; a two-classification detection network is trained by using the difference features; a proposal detection frame of the detection network is used for training of a multi-classification detection network to complete a multi-classification defect detection task. In the application, a twin network structure and a residual network structure decouple defect information and template information, the network more clearly learns a defect part and a classification part, has strong migration ability, and combines characteristics and advantages of a two-classification detection network and a multi-classification detection network to design a network structure which can accurately detect defects and accurately multi-classify.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection, and in particular relates to a two-stage multi-classification industrial image defect detection method based on a twin residual network. BACKGROUND

[0002] With the rapid development of artificial intelligence technology in the field of digital media processing and the rapid improvement of hardware performance in recent years, the performance of computer vision technology is constantly improving, and the deployment cost is constantly decreasing. Therefore, the landing conditions of vision-related artificial intelligence algorithms are mature, and in fact, they are increasingly applied to various industrial scenarios. In the industrial field of computer vision landing, automated defect detection is an important application scenario of related applications. Defect detection requires sensors to collect industrial product information, which is sent to the pre-set detection system, and the output is whether the product has defects and defect-related information.

[0003] In the defect detection process, different defects on the product will have different effects on the product, so manufacturers often have different detection standards for different types of defects. Therefore, in order to improve the detection accuracy, the detection system needs to have excellent defect multi-classification ability; at the same time, when the product background texture is complex and the variety is diverse, we need the detection system to have certain migration ability, and it can also correctly classify defects as much as possible in the application scenario that has not been learned. However, in the face of the above needs, the current detection scheme still has the following defects:

[0004] First, most of the existing defect detection schemes use the image to be detected as a separate input data, which may cause the network to learn some auxiliary information related to the specific background. In this case, the migration ability of the model cannot be guaranteed. The defect information and the background information are highly coupled during training, and the model is difficult to learn information only for the defect itself. Therefore, when the detection system encounters new products, it is difficult to focus on the defect, and a large number of false positives or false negatives may occur for such data;

[0005] Second, the current defect detection system does not fully utilize the prior knowledge of intermediate features. If a feature layer with prior information can be constructed, this information can be used to supervise the training of the model in the middle, making the network easier to train and perform better;

[0006] Third, the current defect detection system separates multi-classification and binary classification completely, does not combine the advantages of the two, and does not decouple the two in the network structure, resulting in a bottleneck in the multi-classification ability and migration ability of the network, which is difficult to improve. SUMMARY

[0007] The purpose of this invention is to provide a two-stage multi-class industrial image defect detection method based on twin residual networks, which has solved at least one of the many problems of the prior art.

[0008] In view of this, the specific solution of the present invention is as follows:

[0009] A two-stage multi-class industrial image defect detection method based on Siamese residual networks includes the following steps:

[0010] S1. Obtain the image to be detected and the corresponding template image, and preprocess the image data;

[0011] S2. Input the image to be detected and the template image into the Siamese backbone network to complete the image data processing, and input the resulting data representation features into the Siamese FPN network to obtain the Siamese FPN features;

[0012] S3. Input the template data into the residual network, combine it with the Siamese network feature extraction to obtain the residual network features, and then input it into the residual FPN network to obtain the residual FPN features;

[0013] S4. Feed the twin FPN features into the segmentation network for supervised training and optimize the features;

[0014] S5. Input the twin FPN features into the binary classification detection network for supervised binary classification training, and extract the proposed detection boxes of the detection network;

[0015] S6. Train a multi-class detection network using proposed detection boxes and residual FPN features;

[0016] S7. Use a multi-class detection network to detect image samples and output the detection results.

[0017] In this invention, step S5 is:

[0018] S51. Feed the twin FPN features into the RPN network to calculate the proposed detection boxes;

[0019] S52. Feed the proposed detection box and the twin FPN features into the binary classification detection network to obtain the detection box output and the corresponding binary classification score, and train the network using the detection box annotation as the label.

[0020] In this invention, step S6 is:

[0021] S61. Input the residual FPN features and the proposed detection boxes of the binary classification detection network into the multi-class classification detection network;

[0022] S62. Use the multi-class detection bounding boxes of defects as labels to train a multi-class detection network.

[0023] In this invention, the number of feature channels in the twin backbone network in step S2 is the same as the number of feature layers corresponding to the residual network in step S3.

[0024] Furthermore, in step S2, the shallow feature extraction layer of the twin backbone network is split into two input ends. The twin networks, which share common parameters, output two sets of features respectively. The result of the feature subtraction is input into the subsequent part of the network and merged into a single data stream. Subsequently, the normal feature extraction of the backbone network is restored.

[0025] Further, step S3 includes:

[0026] S31. Feed the template image into the residual network to extract the intermediate features of the input data;

[0027] S32. Find the last residual block before intermediate feature extraction, and merge the corresponding size of the Siamese network features extracted in step S2 into the residual block structure to obtain the residual network features.

[0028] S33. The residual network features are fed into the FPN feature extraction module to obtain residual FPN features with a fixed number of channels.

[0029] Preferably, both the twin FPN feature and the residual FPN feature are D4-D64 features.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. This invention uses a twin network structure to extract and learn features of defect areas, focusing on analyzing differential features and decoupling the analysis of defect information from background template information. This network structure can detect defect parts more sensitively, improve the detection accuracy of defects, and enhance the network's ability to transfer defect detection.

[0032] 2. This invention constructs a differential feature structure with a background expectation of 0, and incorporates the network structure before the segmentation network supervises the learning of this feature during the network training process, making full use of the above-mentioned prior information, making the network easier to train and more accurate;

[0033] 3. This invention designs a twin residual network structure, which decouples the defective part from the background information while fusing features, making the network have stronger migration detection capabilities.

[0034] 4. This invention integrates the roles of differential feature information and background information in the detection network. It uses differential features to find proposed detection boxes for defects and uses the fused features of background information and differential feature information to classify the detection boxes, enabling the detection system to have the ability to classify defects in multiple ways while maintaining binary classification performance. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the overall two-stage multi-classification industrial image defect detection method described in this invention.

[0037] Figure 2 This is a summary flowchart of the two-stage multi-classification industrial image defect detection method described in this invention.

[0038] Figure 3 This is a flowchart of the network feature extraction process for the two-stage multi-class industrial image defect detection method described in this invention.

[0039] Figure 4 This is a schematic diagram illustrating the network feature usage method of the two-stage multi-classification industrial image defect detection method described in this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described in this specification are merely for explaining the invention and are not intended to limit the invention.

[0041] This invention proposes a two-stage multi-class defect detection method based on Siamese residual networks. The core design concept is the information transmission mechanism of the residual network structure. It uses a Siamese network structure to extract residual feature information and incorporates it into the residual network structure, achieving a dual performance improvement in detection capability and transfer capability. A schematic diagram of the overall detection method is shown below. Figure 1 As shown.

[0042] Specifically, the flowchart of the two-stage multi-class defect detection method based on twin residual networks is as follows: Figure 2 As shown, the method includes the following steps:

[0043] S1. Obtain the image to be detected and the corresponding template image, preprocess the image data, scale the image to a uniform size, normalize the image contrast, and perform affine transformation correction.

[0044] S2. Extract the feature information of the Siamese network FPN, the process is as follows: Figure 3 As shown, the specific steps include:

[0045] a) Since this twin network needs to be used in conjunction with a residual network, the twin backbone network needs to be carefully designed so that the number of its D4-D32 feature channels is the same as the feature layers corresponding to the residual network.

[0046] b) Split the shallow feature extraction layer of the backbone network (before the D2 features) into two Siamese networks with common parameters. The network outputs two sets of D2 features. Subtract the two sets of output features and input them into the subsequent part of the network, merging them into a single data stream. Then, the normal feature extraction method of the backbone network is restored.

[0047] c) Extract the differential information features in the twin backbone network, and extract the features of D4, D8, D16, and D32 from the deep features of the backbone network;

[0048] d) The features of the Siamese network D4–D32 are fed into the FPN feature extraction module to obtain a feature with a fixed number of channels C. num The twins of FPN_D4 and FPN_D64.

[0049] S3. Extract the residual FPN features from the template image, the process is as follows: Figure 3 As shown, the specific steps include:

[0050] a) Feed the template image into the residual network and extract the D4–D32 intermediate features of the input data;

[0051] b) Find the last residual block before intermediate feature extraction, and merge the corresponding size of the Siamese network D4–D32 features extracted in step two into the residual block structure to obtain the residual network D4–D32 features.

[0052] c) The residual network D4–D32 features are fed into the FPN feature extraction module to obtain a feature with a fixed number of channels C. num The residual FPN_D4-FPN_D64 features.

[0053] S4. Feed the twin FPN_D4-FPN_D64 features into the segmentation network for supervised training, such as... Figure 4 As shown, the specific steps include:

[0054] a) Compress the twin FPN_D4-FPN_D64 features to 1 channel using two 1×1 convolution layers to obtain compressed D4-D64 single-channel features;

[0055] b) Activate the features using the sigmoid function, the specific formula is as follows:

[0056]

[0057] c) Transform the segmentation labels of the image to be detected to five sizes from D4 to D64 of the original image, generate five sets of boolean matrices as labels, and train the segmentation network. The training objective is to predict the defect as 1 and the background as 0.

[0058] d) The loss function uses binary cross-entropy:

[0059]

[0060] S5. Train a two-stage binary classification detection network using Siamese FPN features, such as Figure 4 As shown, the specific steps include:

[0061] a) Feed the twin FPN features into the RPN network to calculate the proposed detection boxes;

[0062] b) Feed the proposed detection box and the twin FPN features into the binary classification ROI detection network to obtain the detection box output and the corresponding binary classification score, and use the detection box annotation as the label for training.

[0063] S6. Train a multi-class detection network using residual FPN features and proposed detection boxes from a binary classification detection network, specifically including the following steps:

[0064] a) Input the residual FPN features and the proposed detection boxes of the binary classification detection network into the multi-class ROI detection network;

[0065] b) Use the multi-class detection bounding boxes of defects as labels to train a multi-class detection network.

[0066] S7. Use a multi-class detection network structure to detect the sample to be tested and obtain the detection results.

[0067] In one embodiment of the present invention, the two-stage multi-class defect detection method based on twin residual networks is as follows:

[0068] 1. Acquire the image to be tested of the industrial semiconductor device and the corresponding template image. During the process of cropping the template image, position calibration is required based on the difference gradient so that the deviation between the template and the image to be tested is within 0.5 pixels.

[0069] 2. Extracting Siamese FPN features requires splitting the backbone network and converting it into a Siamese network. Taking the ResNet50 backbone network as an example, the network layers before layer 1 (including the first convolutional layer, the first BN layer, the first ReLU layer, and the first maxpooling layer) are split into a Siamese structure. The Siamese network shares parameters and outputs two sets of features. The two sets of features are subtracted and used as the input to the subsequent ResNet50. D4, D8, D16, and D32 features are extracted from the ResNet50 feature layers, and these features are fed into the FPN module (FPN module has C channels). num Setting it to 256), we obtain twin FPN_D4, FPN_D8, FPN_D16, FPN_D32, and FPN_D64 features.

[0070] 3. Extract residual FPN features from the template image. To input the template image into the residual network (using ResNet50 as an example), extract the D4-D32 intermediate features of the input data. Determine the last residual block before feature extraction, and merge the corresponding size of the Siamese network D4-D32 features extracted in step 2 into the residual block structure to obtain the residual network D4-D32 features. Send the residual network D4-D32 features into the FPN feature extraction module to obtain residual FPN_D4-FPN_D64 features with a fixed number of 256 channels.

[0071] 4. Supervised training using a segmentation network. The twin FPN_D4-FPN_D64 features are fed into the segmentation network for supervised training. Two 1×1 convolutional layers (output channels 32 and 1) are used to compress the number of channels in the twin FPN_D4-FPN_D64 features to 1, resulting in compressed D4-D64 single-channel features. The sigmoid function is used to activate the features. The segmentation labels of the image to be detected are transformed to the five sizes of the original image (D4-D64), generating five sets of boolean matrices as labels. The segmentation network is then trained using binary cross-entropy as the loss function.

[0072] 5. Train a two-stage binary classification detection network using Siamese FPN features. Taking Faster R-CNN as an example, the two-stage detection network feeds Siamese FPN features into the RPN network to calculate proposed detection boxes; the proposed detection boxes and Siamese FPN features are then fed into a binary ROI detection network to obtain the detection box outputs and corresponding binary classification scores. The detection network is trained using the detection box labels as tags, with BCE loss and L1 loss functions used for both.

[0073] 6. Use residual FPN features and proposed detection boxes from a binary classification detection network to train a multi-class detection network. For example... Figure 4As shown, the residual FPN features and the proposed detection boxes of the binary classification detection network are input into the multi-class ROI detection network; the multi-class detection network is trained using the multi-class detection boxes of defects as labels.

[0074] 7. A multi-class detection network structure is used to detect the test samples, and the detection results are obtained. In this embodiment, we used existing data to divide the data into four groups of transfer data for testing. The results of this invention and the baseline experiment (mAP) are shown in Table 1.

[0075] Table 1:

[0076] Network architecture Migrated dataset 1 Migrated dataset 2 Migrated dataset 3 Migrated dataset 4 fasterrcnn (baseline) 83.75％ 60.34％ 85.19％ 72.23％ fasterrcnn (invention) 87.64％ 70.33％ 89.40％ 80.09％

[0077] As can be seen from the results in Table 1, the defect detection framework proposed in this invention outperforms the baseline algorithm in data transfer performance on various datasets, indicating that the method proposed in this invention has stronger data transfer capability in defect multi-classification tasks.

[0078] The present invention is not limited to the description in the specification and embodiments, and thus other advantages and modifications can be readily realized by those skilled in the art. Therefore, the present invention is not limited to the specific details, representative devices and illustrated examples shown and described herein without departing from the spirit and scope of the general concept as defined by the claims and their equivalents.

Claims

1. A two-stage multi-classification industrial image defect detection method based on Siamese residual networks, characterized in that, Includes the following steps: S1. Obtain the image to be detected and the corresponding template image, and preprocess the image data; S2. Input the image to be detected and the template image into the Siamese backbone network to complete the image data processing, and input the resulting data representation features into the Siamese FPN network to obtain the Siamese FPN features; S3. Input the template data into the residual network, combine it with Siamese network feature extraction to obtain residual network features, and then input it into the residual FPN network to obtain residual FPN features; including: S31. Feed the template image into the residual network to extract the intermediate features of the input data; S32. Find the last residual block before intermediate feature extraction, and merge the corresponding size of the Siamese network features extracted in step S2 into the residual block structure to obtain the residual network features. S33. The residual network features are fed into the FPN feature extraction module to obtain residual FPN features with a fixed number of channels; both the twin FPN features and the residual FPN features are D4-D64 features; S4. Feed the twin FPN features into the segmentation network for supervised training and optimize the features; S5. Input the twin FPN features into the binary classification detection network for supervised binary classification training, and extract the proposed detection boxes of the detection network; S6. Train a multi-class detection network using proposed detection boxes and residual FPN features; S7. Use a multi-class detection network to detect image samples and output the detection results.

2. The industrial image defect detection method according to claim 1, characterized in that, Step S5 is as follows: S51. Feed the twin FPN features into the RPN network to calculate the proposed detection boxes; S52. Feed the proposed detection box and the twin FPN features into the binary classification detection network to obtain the detection box output and the corresponding binary classification score, and train the network using the detection box annotation as the label.

3. The industrial image defect detection method according to claim 1, characterized in that, Step S6 is as follows: S61. Input the residual FPN features and the proposed detection boxes of the binary classification detection network into the multi-class classification detection network; S62. Use the multi-class detection bounding boxes of defects as labels to train a multi-class detection network.

4. The industrial image defect detection method according to claim 1, characterized in that, In step S2, the number of feature channels in the twin backbone network is the same as the number of feature layers corresponding to the residual network in step S3.

5. The industrial image defect detection method according to claim 4, characterized in that, In step S2, the shallow feature extraction layer of the twin backbone network is split into two input ends. The twin network, which shares the same parameters, outputs two sets of features respectively. The result of the feature subtraction is input into the subsequent part of the network and merged into a single data stream. Subsequently, the normal feature extraction of the backbone network is restored.

Images