Fast transfer method of production line surface defect detection model based on adversarial training

Abstract

The application provides a kind of fast migration method of production line surface defect detection model based on adversarial training, comprising obtaining the surface defect detection data set of new and old production line, and the surface defect detection data set of old production line is marked processing;Build the production line surface defect detection model based on yolov3, and use the surface defect detection data set of new and old production line to pretrain it;Global adversarial training network is used to the global and category level of the pre-trained production line surface defect detection model is trained;The trained production line surface defect detection model is migrated to new production line.The application only needs to mark the data of a production line, and the data of the remaining production line does not need to be marked, which can save a lot of marking time;The application carries out global and category level of the new and old production line data by global adversarial training module and category adversarial training module;And by category center alignment module, the new and old production line data is close to category center.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect detection, in particular to a rapid migration method of a production line surface defect detection model based on adversarial training. BACKGROUND

[0002] The surface defect detection technology based on machine vision is a key to ensure product quality and plays an important role in the development of industrial intelligence. At present, the surface defect equipment based on machine vision has been widely used to replace manual visual inspection in various industries, including 3C, automobile, home appliance, mechanical manufacturing, semiconductor and electronics, chemical industry, medicine, aerospace, light industry and other industries.

[0003] However, the existing surface defect detection method based on machine vision often needs to collect and label defect samples of each production line for training a neural network model of each production line. The entire training process needs to go through three stages of collection, labeling and training, which requires a large amount of time cost and labeling cost. At the same time, the model can only be trained after the data is labeled, and the time span from collecting data to training the model is relatively large. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a rapid migration method of a production line surface defect detection model based on adversarial training, which can save a large amount of data labeling time and model training time.

[0005] The technical scheme of the present application is as follows: the rapid migration method of a production line surface defect detection model based on adversarial training comprises the following steps:

[0006] S1), obtaining surface defect detection data sets of new and old production lines, and labeling the surface defect detection data set of the old production line;

[0007] S2), constructing a production line surface defect detection model based on yolov3, and pre-training the same using the surface defect detection data sets of the new and old production lines;

[0008] S3), constructing a global adversarial training network, and using the global adversarial training network to perform global and category level adversarial training on the pre-trained production line surface defect detection model;

[0009] S4), discarding the global discriminator and category discriminator of the production line surface defect detection model, and migrating the trained production line surface defect detection model to a new production line.

[0010] Preferably, in step S1), the bounding box and category of the defect of each defect image in the surface defect detection data set of the old production line are labeled.

[0011] Preferably, in step S2), the yolov3-based line surface defect detection model comprises a backbone network, a neck network and a prediction head network.

[0012] Preferably, in step S3), the global adversarial training network comprises a global adversarial training module, a category adversarial training module and a category center alignment module, the global and category adversarial training of the line surface defect detection model is performed through the global adversarial training module and the category adversarial training module, and the category center alignment module is used to narrow the distribution of the new and old line data at the category level.

[0013] Preferably, in step S3), the global and category adversarial training of the pre-trained line surface defect detection model is performed by using the global adversarial training network, and the method comprises the following steps:

[0014] S31), the features of the new and old line data output by the backbone network are input into the global adversarial training module and the category adversarial training module, respectively, to obtain corresponding global adversarial loss functions L The features output by the neck network and the features output by the prediction head network are input into the global adversarial training module and the category adversarial training module, respectively, to obtain corresponding global adversarial loss functions L g1 , L g2 , L g3 and category adversarial loss functions L c1 , L c2 , L c3 ;

[0015] S32), the features of the new and old line data output by the prediction head network and the image classification results p t , p s and the image confidence results o t , o s are input into the category center alignment module to obtain a category alignment loss function L c ;

[0016] S33), a total loss function L is calculated as follows:

[0017] L=L ori +L g1 +L g2 +L g3 +L c1 +L c2 +L c3 +L c ;

[0018] In the formula, L ori is a loss function of the yolov3-based line surface defect detection model.

[0019] Then the total loss function L is gradient backpropagation to perform global and class adversarial training and class closing on the line surface defect detection model based on yolov3;

[0020] Preferably, in step S31), the global adversarial training module comprises a gradient inversion layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

[0021] Preferably, in step S31), the global adversarial training module calculates the expression of the global adversarial loss function as follows:

[0022]

[0023] In the formula, indicates the features output by the line surface defect detection model for new and old line data; D indicates the global discriminator; and G indicates the gradient inversion layer.

[0024] Preferably, in step S31), the class adversarial training module comprises a gradient inversion layer G, k class discriminators E and a result selection layer P; k is the number of categories of the data set; and the output range of the class adversarial training module is [0, 1].

[0025] Preferably, in step S31), the class adversarial training module calculates the expression of the class adversarial loss function L ci as follows:

[0026]

[0027] In the formula, indicates the features output by the line surface defect detection model for new and old line data; E indicates the class discriminator; G indicates the gradient inversion layer; and P indicates the selection layer.

[0028] Preferably, in step S31), the gradient inversion layer G is an identity mapping layer, and the gradient passing through the gradient inversion layer G is multiplied by -1 during gradient backpropagation, thereby achieving the effect of adversarial training.

[0029] Preferably, in step S31), the global discriminator D and the class discriminator E each comprise three convolutional layers and one global pooling layer.

[0030] Preferably, in step S32), the class center alignment module is used to calculate the class alignment loss function L c to close the class centers of new and old line data, and specifically comprises the following steps:

[0031] S321), screening the image classification result p sand image confidence result o s elements greater than the preset threshold T; respectively obtain image classification masks and image confidence masks

[0032] S322), multiply the features image classification result p s , image classification masks and image confidence masks Pixel by pixel multiplication and global average pooling are performed to obtain k tensors with a length of c 1 , wherein c 1 is the dimension of the channel layer; and the feature center of the k tensors with a length of c 1 is represented as η s ;

[0033] S323), the image classification result p t and the image confidence result o t of the new production line are processed according to steps S321)-S322) to obtain the feature center η t ;

[0034] S324), the cosin similarity of the feature centers η t , η s of the new and old production lines and the category center of the new and old production lines is calculated to obtain the similarity matrix S ss , S st , S tt , S ts ;

[0035] S325), the weight of the diagonal element of the similarity matrix is set to-1, and the weight of the other elements is 1; then the elements of the similarity matrix S ss , S st , S tt , S ts are summed according to the weight to obtain the category alignment loss function L c .

[0036] The beneficial effects of the present application are:

[0037] 1. The present application only needs to label the data of one production line, and the data of the remaining production lines does not need to be labeled, which can save a lot of labeling time;

[0038] 2. The present application trains the detection model of the old production line, and after the training is completed, it is migrated to the new production line, without the need to train the detection model of each production line;

[0039] 3. This invention uses a global adversarial training module and a category adversarial training module to perform global and category-level adversarial training on new and old production line data; and uses a category center alignment module to bring the category centers of new and old production line data closer together. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the framework of the method of the present invention;

[0041] Figure 2 This is a flowchart illustrating the process of using the category center alignment module to pull in the category centers of new and old production line data in this invention. Detailed Implementation

[0042] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0043] like Figure 1 As shown, this embodiment provides a fast transfer method for a production line surface defect detection model based on adversarial training, including the following steps:

[0044] S1) Obtain the surface defect detection dataset for both new and old production lines;

[0045] This embodiment only requires annotation of the surface defect detection dataset of the old production line, and does not require annotation of the surface defect detection dataset of the new production line; thus saving a lot of time in data annotation.

[0046] In addition, the annotation mentioned in this embodiment refers to annotating the bounding box and category of the defect in each defect image in the surface defect detection dataset of the old production line.

[0047] S2) Construct a production line surface defect detection model based on YOLOv3 and pre-train it using surface defect detection datasets from both new and old production lines.

[0048] In this embodiment, the YOLOv3-based production line surface defect detection model includes a backbone network, a neck network, and a prediction head network. During training, the labeled surface defect detection datasets from the old production line and the new production line are input into the production line surface defect detection model to obtain the features of the old and new production line data output by the backbone network. Features of neck network output Prediction head network output features And image classification results p t p s Image confidence results t o s And the loss function L of the model ori Update.

[0049] S3), constructing a global adversarial training network; and using the global adversarial training network to perform global and category level adversarial training on the pre-trained production line surface defect detection model;

[0050] In this embodiment, the global adversarial training network includes a global adversarial training module, a category adversarial training module, and a category center alignment module. The global and category adversarial training of the production line surface defect detection model is performed through the global adversarial training module and the category adversarial training module, and the new and old production line data category centers are pulled closer through the category center alignment module.

[0051] In this embodiment, the global and category adversarial training of the pre-trained production line surface defect detection model is performed using the global adversarial training network, which includes the following steps:

[0052] S31), inputting the features of the new and old production line data output by the backbone network the features output by the neck network and the features output by the prediction head network to the global adversarial training module and the category adversarial training module respectively, to obtain corresponding global adversarial loss functions L g1 , L g2 , L g3 , and category adversarial loss functions L c1 , L c2 , L c3 respectively.

[0053] S32), inputting the features of the new and old production line data output by the prediction head network and the image classification results p t , p s , and the image confidence results o t , o s to the category center alignment module to obtain a category alignment loss function L c .

[0054] S33), calculating a total loss function L

[0055] L = L ori + L g1 + L g2 + L g3 + L c1 + L c2 + L c3 + L c .

[0056] In the formula, L ori is the loss function of the yolov3-based production line surface defect detection model.

[0057] Then the total loss function L is gradient backpropagation to the global and class-based adversarial training and class-based closing of the line surface defect detection model based on yolov3 training and model.

[0058] As preferred in the embodiment, in step S31), the global adversarial training module includes a gradient inversion layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

[0059] The expression of the global adversarial loss function calculated by the global adversarial training module is:

[0060]

[0061] In the formula, indicates the features output by the line surface defect detection model for new and old line data; D indicates the global discriminator; and G indicates the gradient inversion layer.

[0062] As preferred in the embodiment, in step S31), the class-based adversarial training module includes a gradient inversion layer G, k class discriminators E, and a result selection layer P; wherein k is the number of classes of the data set; and the output range of the class-based adversarial training module is [0, 1].

[0063] The expression of the class-based adversarial loss function L ci calculated by the class-based adversarial training module is:

[0064]

[0065] In the formula, indicates the features output by the line surface defect detection model for new and old line data; E indicates the class discriminator; G indicates the gradient inversion layer; and P indicates the selection layer.

[0066] As preferred in the embodiment, in step S31), the gradient inversion layer G is an identity mapping layer, and the gradient passing through the gradient inversion layer G is multiplied by -1 during gradient backpropagation; thereby achieving the effect of adversarial training.

[0067] The global discriminator D and the category discriminator E both include three convolutional layers and a global pooling layer. The convolutional kernel of the first convolutional layer of the global discriminator D is 1*1*c*c / 2, the convolutional kernel of the second convolutional layer is 1*1*c / 2*c / 4, and the convolutional kernel of the third convolutional layer is 1*1*c / 4*1; c is the dimension of the channel layer of the input feature; the input feature is a tensor of [n, c, h, w] (where n is the tensor size, c is the dimension of the channel layer, and h and w are the length and width of the tensor), the output after the first convolutional layer is a tensor of [n, c / 2, h, w], the output after the second convolutional layer is a tensor of [n, c / 4, h, w], the output after the third convolutional layer is a tensor of [n, 1, h, w], and the output after the global pooling layer is a tensor of [n, 1, 1, 1].

[0068] The convolutional kernel of the first convolutional layer of the category discriminator E is 1*1*c*c / 2, the convolutional kernel of the second convolutional layer is 1*1*c / 2*c / 4, and the convolutional kernel of the third convolutional layer is 1*1*c / 4*k, c is the dimension of the channel layer of the input feature, and k is the number of categories of the data set. The input feature is a tensor of [n, c, h, w] (where n is the tensor size, c is the dimension of the channel layer, and h and w are the length and width of the tensor), the output after the first convolutional layer is a tensor of [n, c / 2, h, w], the output after the second convolutional layer is a tensor of [n, c / 4, h, w], the output after the third convolutional layer is a tensor of [n, 1, h, w], and the output after the global pooling layer is a tensor of [n, k, 1, 1].

[0069] As preferred in the embodiment, in step S32), as shown in formula (1), the category alignment loss function L Figure 2 is calculated by using the category center alignment module. c In order to narrow the category centers of the new and old production line data, the following steps are included:

[0070] S321), screening the image classification result p s and the image confidence result o s of the old production line, which are greater than the preset threshold T; respectively obtaining the image classification mask and the image confidence mask

[0071] S322), multiplying the feature image classification result p s , the image classification mask and the image confidence mask pixel by pixel, and performing global average pooling to obtain k tensors with a length of c 1 , where c 1Let c be the dimension of the channel layer; and k elements of length c 1 The characteristic center of the tensor is represented as η s ;

[0072] S323), according to steps S321)-S322), classify the image results of the new production line p. t Image confidence results t After processing, the feature center η is obtained. t ;

[0073] S324) Calculate the characteristic center η of the old and new production lines. t η s Category center for new and old production lines The consine similarity is used to obtain the similarity matrix S. ss ,S st ,S tt ,S ts ;

[0074] (S325) Set the weight of the diagonal elements of the similarity matrix to -1, and the weight of the other elements to 1; then, set the similarity matrix S... ss ,S st ,S tt ,S ts The elements are summed according to their weights to obtain the class alignment loss function L. c .

[0075] S4) Discard the global discriminator and category discriminator of the production line surface defect detection model, and transfer the trained production line surface defect detection model to the new production line.

[0076] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A method for fast transfer of a line surface defect detection model based on adversarial training, characterized in that, The method comprises the following steps: S1), obtaining surface defect detection data sets of new and old production lines, and labeling the surface defect detection data set of the old production line; S2), constructing a production line surface defect detection model based on yolov3, and pre-training the same by using the surface defect detection data sets of the new and old production lines; S3), constructing a global adversarial training network, and using the global adversarial training network to perform global and category level adversarial training on the pre-trained production line surface defect detection model; The global adversarial training network comprises a global adversarial training module, a category adversarial training module, and a category center alignment module. The global and category adversarial training of the pre-trained production line surface defect detection model by using the global adversarial training network comprises the following steps: S31), input the features of the new and old production line data output by the backbone network , and the features output by the neck network , and the features output by the prediction head network , respectively into the global adversarial training module and the category adversarial training module, to obtain the corresponding global adversarial loss function and the category adversarial loss function ; The expression of the category adversarial training module calculating a category adversarial loss function is: The expression is: In the formula, represents the features output by the wafer surface defect detection model for new and old wafer line data; represents the category discriminator; represents the gradient reversal layer; represents the selection layer; S32)、the features of the new and old production line data output by the prediction head network 、 and the image classification result 、 and the image confidence result 、 are input into a category center alignment module to obtain a category alignment loss function ; S33), calculate total loss function : ； In the formula, is a loss function of the yolov3-based line surface defect detection model; Then the total loss function Gradient backpropagation is performed to conduct global and class adversarial training and class narrowing for the production line surface defect detection model based on yolov3. S4), discarding the global discriminator and the category discriminator of the production line surface defect detection model, and migrating the trained production line surface defect detection model to a new production line.

2. The method of claim 1, wherein the method further comprises: In step S2), the production line surface defect detection model based on yolov3 comprises a backbone network, a neck network, and a prediction head network.

3. The method of claim 1, wherein the method further comprises: In step S31), the global adversarial training module comprises a gradient inversion layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

4. The method of claim 3, wherein the method further comprises: In step S31), the expression of the global adversarial loss function calculated by the global adversarial training module is: In the formula, represents the features output by the wafer surface defect detection model for new and old wafer line data; represents a global discriminator; represents a gradient reversal layer.

5. The method of claim 4, wherein: In step S31), the category adversarial training module comprises a gradient inversion layer G, k category discriminators E, and a result selection layer P; wherein k is the number of categories of the data set; and the output range of the category adversarial training module is [0, 1].

6. The method of claim 5, wherein the method further comprises: In step S31), the gradient inversion layer G is an identity mapping layer, and the gradient passing through the gradient inversion layer G is multiplied by -1 during gradient backpropagation; the global discriminator D and the category discriminator E each comprise three convolutional layers and one global pooling layer.

7. The method of claim 1, wherein: In step S32, the category alignment loss function is calculated by using the category center alignment module to pull the category centers of the new and old production line data, specifically including the following steps: S321)、screening the image classification result of the old production line and the image confidence result greater than a preset threshold element respectively, an image classification mask and an image confidence mask ; S322), multiplying the features , the image classification result , the image classification mask , and the image confidence mask pixel by pixel, and performing global average pooling to obtain k tensors with a length of , where is the dimension of the channel layer; and the feature center of the k tensors with a length of is represented as ; S323), and classifying the image of the new line according to steps S321) - S322) and the image confidence result processing to obtain the feature center ; S324), calculate the feature center of the new and old production lines and the category center of the new and old production lines , the consin similarity, to obtain a similarity matrix ; S325)、set the weight of the diagonal elements of the similarity matrix to -1, and the weight of other elements to 1; then sum the elements of the similarity matrix according to the weights to obtain the category alignment loss function .

Images