Rapid migration method of production line surface defect detection model based on adversarial training

Abstract

The invention provides a production line surface defect detection model rapid migration method based on adversarial training, and the method comprises the steps: obtaining a surface defect detection data set of a new production line and a surface defect detection data set of an old production line, and carrying out the labeling processing of the surface defect detection data set of the old production line; constructing a production line surface defect detection model based on the yolov3, and pre-training the production line surface defect detection model by using the surface defect detection data sets of the new and old production lines; using a global adversarial training network to carry out global and category level adversarial training on the pre-trained production line surface defect detection model; and migrating the trained production line surface defect detection model to a new production line. According to the method, only the data of one production line needs to be labeled, and the data of the other production lines do not need to be labeled, so that a large amount of labeling time can be saved; according to the invention, through the global confrontation training module and the category confrontation training module, confrontation training at global and category levels is carried out on new and old production line data; and through a category center alignment module, the new and old production line data are subjected to category center zooming.

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a method for rapid migration of a production line surface defect detection model based on adversarial training. Background Art

[0002] Surface defect detection technology based on machine vision is the key to ensuring product quality and plays an important role in the development of industrial intelligence. At present, surface defect detection equipment based on machine vision has widely replaced manual eye detection in various industrial fields, including 3C, automobiles, home appliances, machinery manufacturing, semiconductors and electronics, chemicals, medicine, aerospace, light industry and other industries.

[0003] However, existing surface defect detection methods based on machine vision often require the collection and labeling of defect samples from each production line to train the neural network model of each production line. The entire training process needs to go through three stages: collection-labeling-training, which requires a lot of time and labeling costs. At the same time, the model can only be trained after the data is labeled, and the time span from data collection to model training is relatively long. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method for quickly migrating a production line surface defect detection model based on adversarial training. The present invention can save a lot of data labeling time and model training time.

[0005] The technical solution of the present invention is: a rapid migration method of a production line surface defect detection model based on adversarial training, comprising the following steps:

[0006] S1), obtaining the surface defect detection datasets of the new and old production lines, and annotating the surface defect detection datasets of the old production line;

[0007] S2) Build a production line surface defect detection model based on yolov3 and pre-train it using the surface defect detection datasets of new and old production lines;

[0008] S3) construct a global adversarial training network; and use the global adversarial training network to perform global and category-level adversarial training on the pre-trained production line surface defect detection model;

[0009] S4), discard the global discriminator and category discriminator of the production line surface defect detection model, and migrate the trained production line surface defect detection model to the new production line.

[0010] Preferably, in step S1), the bounding box and category of the defect of each defect image in the surface defect detection dataset of the old production line are labeled.

[0011] Preferably, in step S2), the production line surface defect detection model based on yolov3 includes a backbone network, a neck network and a prediction head network.

[0012] Preferably, in step S3), the global adversarial training network includes a global adversarial training module, a category adversarial training module and a category center alignment module. The global adversarial training module and the category adversarial training module are used to perform global and category adversarial training on the production line surface defect detection model, and the category center alignment module is used to bring the category level distribution of the new and old production line data closer.

[0013] Preferably, in step S3), a global adversarial training network is used to perform global and category adversarial training on the pre-trained production line surface defect detection model, which specifically includes the following steps:

[0014] S31), the characteristics of the new and old production line data output by the backbone network Characteristics of the neck network output And the prediction head network output features They are input into the global adversarial training module and the category adversarial training module respectively, and the corresponding global adversarial loss function L is obtained respectively. g1 , L g2 , L g3 , and the category adversarial loss function L c1 , L c2 , L c3 ;

[0015] S32) The characteristics of the new and old production line data output by the prediction head network And the image classification result p t 、p s ; and image confidence results o t , o s Input into the category center alignment module to obtain the category alignment loss function L c ;

[0016] S33), calculate the total loss function L:

[0017] L=L ori +L g1 +L g2 +L g3 +L c1 +L c2 +L c3 +L c ;

[0018] Where, L ori is the loss function of the production line surface defect detection model based on yolov3;

[0019] Then the total loss function L is back-propagated to perform global and category adversarial training and category narrowing on the production line surface defect detection model based on yolov3;

[0020] Preferably, in step S31), the global adversarial training module includes a gradient reversal layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

[0021] Preferably, in step S31), the global adversarial training module calculates the expression of the global adversarial loss function as follows:

[0022]

[0023] In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; D represents the global discriminator; G represents the gradient reversal layer.

[0024] Preferably, in step S31), the category adversarial training module includes a gradient reversal layer G, k category discriminators E and a result selection layer P; wherein k is the number of categories in the data set; and the output range of the category adversarial training module is [0,1].

[0025] Preferably, in step S31), the category adversarial training module calculates the category adversarial loss function L ci The expression is:

[0026]

[0027] In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; E represents the category discriminator; G represents the gradient reversal layer; P represents the selection layer.

[0028] Preferably, in step S31), the gradient reversal layer G is an identity mapping layer. During gradient back propagation, the gradient passing through the gradient reversal layer G will be multiplied by -1, thereby achieving the effect of adversarial training.

[0029] Preferably, in step S31), the global discriminator D and the category discriminator E both include three convolutional layers and one global pooling layer.

[0030] Preferably, in step S32), the category center alignment module is used to calculate the category alignment loss function L c , in order to bring the category center of the new and old production line data closer, the specific steps include:

[0031] S321), filter the image classification results of the old production line p sAnd the image confidence result o s Elements greater than the preset threshold T; image classification masks are obtained respectively and image confidence mask

[0032] S322), the feature Image classification results p s , Image Classification Mask and image confidence mask Multiply pixel by pixel and perform global average pooling to obtain k pixels of length c. 1 A tensor of 1 is the dimension of the channel layer; and k lengths of c 1 The characteristic center of the tensor is denoted as η s ;

[0033] S323), according to steps S321)-S322), the image classification result p of the new production line t And the image confidence result o t Processing is performed to obtain the feature center η t ;

[0034] S324), calculate the characteristic center η of the new and old production lines t , η s and category center for new and old production lines The consin similarity of the obtained similarity matrix S ss ,S st ,S tt ,S ts ;

[0035] S325), set the weight of the diagonal elements of the similarity matrix to -1, and the weight of other elements to 1; then convert the similarity matrix S ss ,S st ,S tt ,S ts The elements of are summed according to the weights to obtain the category alignment loss function L c .

[0036] The beneficial effects of the present invention are:

[0037] 1. The present invention only needs to mark the data of one production line, and the data of other production lines do not need to be marked, which can save a lot of marking time;

[0038] 2. The present invention trains the detection model of the old production line and migrates it to the new production line after the training is completed, without the need to train the detection model of each production line;

[0039] 3. The present invention performs adversarial training on the new and old production line data at the global and category levels through a global adversarial training module and a category adversarial training module; and brings the category centers of the new and old production line data closer through a category center alignment module. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A framework flow chart of the method of the present invention;

[0041] Figure 2 This is a process framework diagram of the present invention using the category center alignment module to pull in the category center of the new and old production line data. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0043] like Figure 1 As shown, this embodiment provides a method for quickly migrating a production line surface defect detection model based on adversarial training, comprising the following steps:

[0044] S1), obtain surface defect detection data sets of new and old production lines;

[0045] In this embodiment, only the surface defect detection dataset of the old production line needs to be labeled, and no labeling is required for the surface defect detection dataset of the new production line; thus, a large amount of data labeling time is saved;

[0046] In addition, the labeling described in this embodiment refers to labeling the bounding box and category of the defect of each defect image in the surface defect detection data set of the old production line.

[0047] S2) Build a production line surface defect detection model based on yolov3 and pre-train it using the surface defect detection datasets of new and old production lines;

[0048] In this embodiment, the production line surface defect detection model based on yolov3 includes a backbone network, a neck network and a prediction head network. During training, the surface defect detection data set of the marked old production line and the surface defect detection data set of the new production line are input into the production line surface defect detection model to obtain the characteristics of the new and old production line data output by the backbone network. Characteristics of the neck network output Prediction head network output features And the image classification result p t 、p s ; Image confidence result o t , o s , and the loss function L of the model ori to update.

[0049] S3) construct a global adversarial training network; and use the global adversarial training network to perform global and category-level adversarial training on the pre-trained production line surface defect detection model;

[0050] In this embodiment, the global adversarial training network includes a global adversarial training module, a category adversarial training module and a category center alignment module. The global adversarial training module and the category adversarial training module are used to perform global and category adversarial training on the production line surface defect detection model, and the category center alignment module is used to bring the category centers of the new and old production line data closer.

[0051] In this embodiment, a global adversarial training network is used to perform global and category adversarial training on a pre-trained production line surface defect detection model, specifically including the following steps:

[0052] S31), the characteristics of the new and old production line data output by the backbone network Characteristics of the neck network output And the prediction head network output features They are input into the global adversarial training module and the category adversarial training module respectively, and the corresponding global adversarial loss function L is obtained respectively. g1 , L g2 , L g3 , and the category adversarial loss function L c1 , L c2 , L c3 ;

[0053] S32) The characteristics of the new and old production line data output by the prediction head network And the image classification result p t 、p s ; and image confidence results o t , o s Input into the category center alignment module to obtain the category alignment loss function L c ;

[0054] S33), calculate the total loss function L

[0055] L=L ori +L g1 +L g2 +L g3 +L c1 +L c2 +L c3 +L c ;

[0056] Where, L ori is the loss function of the production line surface defect detection model based on yolov3;

[0057] Then the total loss function L is back-propagated to perform global and category adversarial training and category narrowing on the production line surface defect detection model based on yolov3;

[0058] As a preferred embodiment of the present invention, in step S31), the global adversarial training module includes a gradient reversal layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

[0059] The expression of the global adversarial training module for calculating the global adversarial loss function is:

[0060]

[0061] In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; D represents the global discriminator; G represents the gradient reversal layer.

[0062] As a preferred embodiment of this embodiment, in step S31), the category adversarial training module includes a gradient reversal layer G, k category discriminators E and a result selection layer P; wherein k is the number of categories in the data set; and the output range of the category adversarial training module is [0,1].

[0063] The category adversarial training module calculates the category adversarial loss function L ci The expression is:

[0064]

[0065] In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; E represents the category discriminator; G represents the gradient reversal layer; P represents the selection layer.

[0066] As a preferred embodiment of the present invention, in step S31), the gradient reversal layer G is an identity mapping layer. During gradient back propagation, the gradient passing through the gradient reversal layer G will be multiplied by -1, thereby achieving the effect of adversarial training.

[0067] The global discriminator D and the category discriminator E both include three convolutional layers and one global pooling layer. The convolution kernel of the first convolutional layer of the global discriminator D is 1*1*c*c / 2, the convolution kernel of the second convolutional layer is 1*1*c / 2*c / 4, and the convolution kernel of the third convolutional layer is 1*1*c / 4*1; c is the dimension of the channel layer of the input feature; the input feature is a tensor of [n,c,h,w] (where n is the size of the tensor, c is the dimension of the channel layer, and h and w are the length and width of the tensor), the output after the first convolutional layer is a tensor of [n,c / 2,h,w], the output after the second convolutional layer is a tensor of [n,c / 4,h,w], the output after the third convolutional layer is a tensor of [n,1,h,w], and the output after the global pooling layer is a tensor of [n,1,1,1].

[0068] The convolution kernel of the first convolution layer of the category discriminator E is 1*1*c*c / 2, the convolution kernel of the second convolution layer is 1*1*c / 2*c / 4, and the convolution kernel of the third convolution layer is 1*1*c / 4*k, where c is the dimension of the channel layer of the input feature, and k is the number of categories of the data set. The input feature is a tensor of [n,c,h,w] (where n is the size of the tensor, c is the dimension of the channel layer, and h and w are the length and width of the tensor), the output of the first convolution layer is a tensor of [n,c / 2,h,w], the output of the second convolution layer is a tensor of [n,c / 4,h,w], the output of the third convolution layer is a tensor of [n,1,h,w], and the output of the globalpooling layer is a tensor of [n,k,1,1].

[0069] As a preferred embodiment of this invention, in step S32), if Figure 2 As shown, the category center alignment module is used to calculate the category alignment loss function L c , in order to bring the category center of the new and old production line data closer, the specific steps include:

[0070] S321), filter the image classification results of the old production line p s And the image confidence result o s Elements greater than the preset threshold T; image classification masks are obtained respectively and image confidence mask

[0071] S322), the feature Image classification results p s , Image Classification Mask and image confidence mask Multiply pixel by pixel and perform global average pooling to obtain k pixels of length c. 1 A tensor of 1is the dimension of the channel layer; and k lengths of c 1 The characteristic center of the tensor is denoted as η s ;

[0072] S323), according to steps S321)-S322), the image classification result p of the new production line t And the image confidence result o t Processing is performed to obtain the feature center η t ;

[0073] S324), calculate the characteristic center η of the new and old production lines t , η s and category center for new and old production lines The consin similarity of the obtained similarity matrix S ss ,S st ,S tt ,S ts ;

[0074] S325), set the weight of the diagonal elements of the similarity matrix to -1, and the weight of other elements to 1; then convert the similarity matrix S ss ,S st ,S tt ,S ts The elements of are summed according to the weights to obtain the category alignment loss function L c .

[0075] S4), discard the global discriminator and category discriminator of the production line surface defect detection model, and migrate the trained production line surface defect detection model to the new production line.

[0076] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A rapid migration method for production line surface defect detection model based on adversarial training, characterized in that: The steps include: S1), obtaining the surface defect detection datasets of the new and old production lines, and annotating the surface defect detection datasets of the old production line; S2) Build a production line surface defect detection model based on yolov3 and pre-train it using the surface defect detection datasets of new and old production lines; S3) construct a global adversarial training network; and use the global adversarial training network to perform global and category-level adversarial training on the pre-trained production line surface defect detection model; S4), discard the global discriminator and category discriminator of the production line surface defect detection model, and migrate the trained production line surface defect detection model to the new production line.

2. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 1 is characterized by: In step S2), the production line surface defect detection model based on yolov3 includes a backbone network, a neck network and a prediction head network.

3. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 1 is characterized by: In step S3), the global adversarial training network includes a global adversarial training module, a category adversarial training module and a category center alignment module. The global adversarial training module and the category adversarial training module are used to perform global and category adversarial training on the production line surface defect detection model, and the category center alignment module is used to bring the category centers of the new and old production line data closer.

4. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 3 is characterized by: In step S3), a global adversarial training network is used to perform global and category adversarial training on the pre-trained production line surface defect detection model, which specifically includes the following steps: S31), the characteristics of the new and old production line data output by the backbone network Characteristics of the neck network output And the prediction head network output features They are input into the global adversarial training module and the category adversarial training module respectively, and the corresponding global adversarial loss function L is obtained respectively. g1 , L g2 , L g3 , and the category adversarial loss function L c1 , L c2 , L c3 ; S32) The characteristics of the new and old production line data output by the prediction head network And the image classification result p t 、p s ; and image confidence results o t , o s Input into the category center alignment module to obtain the category alignment loss function L c ; S33), calculate the total loss function L: L=L ori +L g1 +L g2 +L g3 +L c1 +L c2 +L c3 +L c ; Where, L ori is the loss function of the production line surface defect detection model based on yolov3; Then, the total loss function L is gradient back-propagated to perform global and category adversarial training and category narrowing on the production line surface defect detection model based on yolov3.

5. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 4 is characterized in that: In step S31), the global adversarial training module includes a gradient reversal layer G and a global discriminator D, and the output range of the global adversarial training module is [0, 1].

6. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 5 is characterized by: In step S31), the expression of the global adversarial loss function calculated by the global adversarial training module is: In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; D represents the global discriminator; G represents the gradient reversal layer.

7. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 6 is characterized by: In step S31), the category adversarial training module includes a gradient reversal layer G, k category discriminators E and a result selection layer P; wherein k is the number of categories in the data set; and the output range of the category adversarial training module is [0,1].

8. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 7 is characterized by: In step S31), the class adversarial training module calculates the class adversarial loss function L ci The expression is: In the formula, It represents the features of the new and old production line data output by the production line surface defect detection model; E represents the category discriminator; G represents the gradient reversal layer; P represents the selection layer.

9. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 8 is characterized by: In step S31), the gradient reversal layer G is an identity mapping layer. During gradient back propagation, the gradient passing through the gradient reversal layer G will be multiplied by -1; the global discriminator D and the category discriminator E both include three convolutional layers and one global pooling layer.

10. The rapid migration method of the production line surface defect detection model based on adversarial training according to claim 4 is characterized by: In step S32), the category center alignment module is used to calculate the category alignment loss function L c , in order to bring the category center of the new and old production line data closer, the specific steps include: S321), filter the image classification results of the old production line p s And the image confidence result o s Elements greater than a preset threshold T; Get the image classification mask respectively and image confidence mask S322), the feature Image classification results p s , Image Classification Mask and image confidence mask Multiply pixel by pixel and perform global average pooling to obtain k pixels of length c. 1 A tensor of 1 is the dimension of the channel layer; and k lengths of c 1 The characteristic center of the tensor is denoted as η s ; S323), according to steps S321)-S322), the image classification result p of the new production line t And the image confidence result o t Processing is performed to obtain the feature center η t ; S324), calculate the characteristic center η of the new and old production lines t , η s and category center for new and old production lines The consin similarity of the obtained similarity matrix S ss ,S st ,S tt ,S ts ; S325), set the weight of the diagonal elements of the similarity matrix to -1, and the weight of other elements to 1; then convert the similarity matrix S ss ,S st ,S tt ,S ts The elements of are summed according to the weights to obtain the category alignment loss function L c .

Images