Domain adaptive multi-class defect intelligent detection method based on class alignment

Abstract

The embodiment of the invention relates to the field of computer vision target detection, in particular to a domain self-adaptive multi-class defect intelligent detection method based on class alignment, and the method comprises the steps: obtaining an industrial defect image; processing the industrial defect image based on a domain shared feature extractor to obtain domain shared feature data, and processing the industrial defect image based on a domain private feature extractor to obtain domain private feature data; processing the source domain private feature data and the target domain shared feature data based on a classification neural network to obtain a first label type and a second label type; processing the domain shared feature data by a domain discriminator determined according to the first label type and the second label type to obtain a first probability value and a second probability value, and determining a defect classification detection result of the industrial defect image based on the first probability value and the second probability value. Model learning is guided, the accuracy and robustness of defect detection are improved, and defect detection in a complex industrial environment is more reliable.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision target detection, and in particular to a domain adaptive multi-class defect intelligent detection method based on class alignment. Background Art

[0002] In the field of defect detection, manual inspection and infrared technology were mainly relied on in the early days. These methods are not only inefficient, but also difficult to respond to defect problems in a timely manner. In contrast, detection technology based on machine vision has gradually become mainstream with its advantages of low cost and high efficiency. At present, most detection methods based on machine vision focus on specific types of defects and can distinguish between defect areas and non-defect areas. These methods perform well when detecting a single type of defect, but it is often difficult to achieve intelligent detection when faced with complex multi-class defect problems. In addition, due to the complexity of the external environment and the diversity of defect types, industrial target defect detection faces three major challenges: 1) The complexity of the external environment causes the image details to be blurred and contain a lot of noise, and the contrast difference between the defect and the background is small; 2) The scales of defects vary, and there are fewer information elements for small-sized defects; 3) There are many types of defects on the surface of industrial targets and their manifestations are diverse, which makes it difficult for existing methods to accurately detect multiple surface defects. Therefore, it is urgent to develop technologies that can adaptively identify and detect multiple types of surface defects to achieve intelligent detection of multiple types of defect targets. Summary of the invention

[0003] In view of this, an embodiment of the present application proposes an industrial defect detection method, which is aimed at industrial defect detection.

[0004] To achieve the above-mentioned purpose, an embodiment of the present application provides a domain adaptive multi-class defect intelligent detection method based on class alignment, comprising: acquiring an industrial defect image and inputting it into a pre-built domain adaptive multi-class defect detection model based on class alignment, wherein the domain adaptive multi-class defect detection model based on class alignment includes a domain shared feature extractor, a domain private feature extractor, a classification neural network and a domain discriminator; processing the industrial defect image based on the trained domain shared feature extractor to obtain domain shared feature data, and processing the industrial defect image based on the trained domain private feature extractor to obtain domain private feature data, wherein the domain shared feature data includes target domain shared feature data and domain private feature data. It includes source domain private feature data and target domain private feature data; based on the trained classification neural network, the source domain private feature data and the target domain shared feature data are processed to obtain a first label type and a second label type respectively; a domain discriminator determined according to the first label type and the second label type processes the domain shared feature data to obtain a first probability value and a second probability value, and a defect classification detection result of the industrial defect image is determined based on the first probability value and the second probability value, wherein the first probability value is the probability that the output result of the domain discriminator is judged to be the source domain private feature data, and the second probability value is the probability that the output result of the domain discriminator is judged to be the target domain private feature data.

[0005] Optionally, the trained domain-private feature extractor includes: a source domain-private feature extractor and a target domain-private feature extractor; wherein the source domain-private feature extractor includes a source domain image encoder, a first RPN module and a first convolution module connected in sequence, the source domain image encoder is used to encode the industrial defect image to obtain a first encoded image, the first RPN module is used to process the first encoded image to obtain a first encoded image carrying a candidate box, the first encoded image and the first encoded image carrying the candidate box are feature-concatenated to obtain a second encoded image, and the first convolution module is used to process the second encoded image to obtain the source domain-private feature data; The target domain private feature extractor includes a target domain image encoder, a second RPN module and a second convolution module connected in sequence, the target domain image encoder is used to encode the industrial defect image to obtain a third encoded image, the second RPN module is used to process the third encoded image to obtain a third encoded image carrying a candidate box, the third encoded image and the third encoded image carrying the candidate box are feature-concatenated to obtain a fourth encoded image, the second convolution module is used to process the fourth encoded image to obtain the target domain private feature data; the first convolution module, the second convolution module and the third convolution module are all VGG-19 convolutional neural networks.

[0006] Optionally, the trained domain shared feature extractor includes a shared domain image encoder, a third RPN module and a third convolution module connected in sequence; wherein, the shared domain image encoder is used to encode the industrial defect image to obtain a fifth encoded image, the third RPN module is used to process the fifth encoded image to obtain a fifth encoded image carrying a candidate box, the fifth encoded image and the fifth encoded image carrying the candidate box are feature-concatenated to obtain a sixth encoded image, and the third convolution module is used to process the sixth encoded image to obtain the domain shared feature data.

[0007] Optionally, before the industrial defect image is processed by the trained domain-shared feature extractor to obtain domain-shared feature data, the industrial defect detection further includes: constructing an image training set, the image training set including source domain data and target domain data, wherein the labeled training set is the source domain data and the unlabeled training set is the target domain data; processing the source domain data based on the domain-shared feature extractor to obtain source domain-shared feature data and a source domain-shared feature matrix; processing the target domain data based on the domain-shared feature extractor to obtain target domain-shared feature data and a target domain-shared feature matrix; processing the source domain data based on the source domain-private feature extractor to obtain a source domain-private feature matrix; processing the target domain data based on the target domain-private feature extractor to obtain a target domain-private feature matrix; A first product term is obtained based on the transpose of the source domain shared feature matrix and the square of the Frobenius norm of the source domain private feature matrix, a second product term is obtained based on the transpose of the target domain shared feature matrix and the square of the Frobenius norm of the target domain private feature matrix, and a first loss function is obtained based on the sum of the first product term and the second product term; the domain shared feature extractor and the domain private feature extractor are iteratively trained until the eigenvector of the source domain private feature matrix is ​​orthogonal to the eigenvector of the source domain shared feature matrix, the eigenvector of the target domain private feature matrix is ​​orthogonal to the eigenvector of the target domain shared feature matrix, and the value of the first loss function is minimized, thereby obtaining the trained domain shared feature extractor and the domain private feature extractor.

[0008] Optionally, the trained classification neural network includes a first fully connected neural network, the first fully connected neural network includes a plurality of first fully connected layers connected in sequence, and each of the first fully connected layers is connected to a first activation layer.

[0009] Optionally, before the trained classification neural network processes the source domain private feature data and the target domain shared feature data, the industrial defect detection also includes: inputting the source domain shared feature data into the pre-constructed classification neural network to obtain a first prediction result; obtaining the category label of the source domain data based on the label of the source domain data; determining a second loss function based on the cross entropy of the first prediction result and the category label of the source domain data; taking the second loss function as the objective function, training the pre-constructed classification neural network until the second loss function is minimized to obtain a trained pre-classification neural network; inputting the source domain private feature data into the first gradient reversal layer to obtain second source domain private feature data; inputting the second source domain private feature data into the pre-classification neural network to obtain a second prediction result; determining a third loss function based on the cross entropy of the category label of the source domain data and the second prediction result; taking the third loss function as the objective function, training the pre-classification neural network until the third loss function is minimized to obtain the trained classification neural network.

[0010] Optionally, after obtaining the trained classification neural network, the industrial defect detection further comprises: inputting target domain shared feature data into the classification neural network to obtain a pseudo label.

[0011] Optionally, the domain identifier includes a second gradient reversal layer and a second fully connected neural network connected in sequence, wherein the second fully connected neural network includes three second fully connected layers connected in sequence and a second activation layer arranged after each of the second fully connected layers.

[0012] Optionally, before the domain identifier determined according to the first label type and the second label type processes the domain shared feature data, the industrial defect detection also includes: inputting the domain shared feature data into the pre-built domain identifier to obtain source domain shared feature identification data and target domain shared feature identification data; determining a fourth loss function of the domain identifier based on the source domain shared feature identification data and the target domain shared feature identification data, the fourth loss function being a feature alignment square loss function; and training the domain identifier with the fourth loss function as the objective function to obtain a trained domain identifier.

[0013] Optionally, after obtaining the domain identifier, the industrial defect detection also includes: obtaining a pre-constructed third fully connected neural network; inputting source domain shared feature data, target domain shared feature data, source domain private feature data and target domain private feature data into the third fully connected neural network to obtain reconstructed feature data; and constructing a fifth loss function based on the reconstructed feature data and the original feature data, wherein the original feature data includes the source domain data and the target domain data.

[0014] The embodiment of the present application proposes a domain adaptive multi-class defect intelligent detection method based on class alignment, which obtains industrial defect images; processes the industrial defect images based on a trained domain shared feature extractor to obtain domain shared feature data, and processes the industrial defect images based on a trained domain private feature extractor to obtain domain private feature data, wherein the domain shared feature data includes target domain shared feature data, and the domain private feature data includes source domain private feature data and target domain private feature data; processes the source domain private feature data and the target domain shared feature data based on a trained classification neural network, and obtains a first label type and a second label type accordingly; determines a domain discriminator to process the domain shared feature data according to the first label type and the second label type to obtain a first probability value and a second probability value, and determines a defect classification detection result of the industrial defect image based on the first probability value and the second probability value, wherein the first probability value is the probability that the output result of the domain discriminator is judged to be the source domain private feature data, and the second probability value is the probability that the output result of the domain discriminator is judged to be the target domain private feature data, which can effectively utilize class alignment and domain adaptation technology to guide model learning, thereby improving the accuracy and robustness of defect detection, and making defect detection in complex industrial environments more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a process of a domain adaptive multi-class defect intelligent detection method based on class alignment provided in an embodiment of the present application Figure 1 ;

[0016] Figure 2 This is a process of a domain adaptive multi-class defect intelligent detection method based on class alignment provided in an embodiment of the present application Figure 2 ;

[0017] Figure 3 is a model schematic diagram of a domain adaptive multi-class defect intelligent detection method based on class alignment provided in an embodiment of the present application;

[0018] Figure 4 It is a discriminator network structure of a domain adaptive multi-class defect intelligent detection method based on class alignment provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0020] An embodiment of the present application proposes a domain adaptive multi-class defect intelligent detection method based on class alignment, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments are all described by taking the server as an example. The following is a specific description of the implementation details of a domain adaptive multi-class defect intelligent detection method based on class alignment proposed in this embodiment. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution.

[0021] The specific process of the domain adaptive multi-class defect intelligent detection method based on class alignment proposed in this embodiment can be as follows: Figure 1 As shown, including:

[0022] S10, acquiring an industrial defect image and inputting it into a pre-built domain adaptive multi-class defect detection model based on class alignment, wherein the domain adaptive multi-class defect detection model based on class alignment includes a domain shared feature extractor, a domain private feature extractor, a classification neural network and a domain discriminator;

[0023] Specifically, industrial defect images are the input data of the pre-built industrial defect detection model. Figure 3 The industrial defect detection model includes: a domain private feature extractor 11 and a domain shared feature extractor 12. The domain private feature extractor includes a source domain private feature extractor 111 and a target domain private feature extractor 112. The source domain private feature extractor 111 is connected to the first gradient reversal layer 13. The first gradient reversal layer 13 and the domain shared feature extractor 12 are both connected to the classification neural network 14. The domain shared feature extractor 12 is also connected to the domain discriminator 15. The domain discriminator 15 is connected to finally output the image defect detection result of the industrial device. The domain shared feature extractor 12 is connected to the source domain private feature extractor 111 and then connected to the source domain input layer 01. The domain shared feature extractor 12 is connected to the target domain private feature extractor 112 and then connected to the target domain input layer 02.

[0024] S20, processing the industrial defect image based on the trained domain shared feature extractor 12 to obtain domain shared feature data, and processing the industrial defect image based on the trained domain private feature extractor 11 to obtain domain private feature data, wherein the domain shared feature data includes target domain shared feature data, and the domain private feature data includes source domain private feature data and target domain private feature data;

[0025] In an embodiment of the present application, the domain-private feature extractor 11 includes: a source domain-private feature extractor 111 and a target domain-private feature extractor 112;

[0026] Among them, the source domain private feature extractor 111 includes a source domain image encoder 1111, a first RPN module 1112 and a first convolution module 1113 connected in sequence, the source domain image encoder 1111 is used to encode the industrial defect image to obtain a first encoded image, the first RPN module 1112 is used to process the first encoded image to obtain a first encoded image carrying a candidate frame, the first encoded image and the first encoded image carrying the candidate frame are feature-concatenated to obtain a second encoded image, and the first convolution module 1113 is used to process the second encoded image to obtain source domain private feature data;

[0027] The target domain private feature extractor 112 includes a target domain image encoder 1121, a second RPN module 1122 and a second convolution module 1123 connected in sequence. The target domain image encoder 1121 is used to encode the industrial defect image to obtain a third encoded image. The second RPN module 1122 is used to process the third encoded image to obtain a third encoded image carrying a candidate frame. The third encoded image and the third encoded image carrying the candidate frame are concatenated to obtain a fourth encoded image. The second convolution module 1123 is used to process the fourth encoded image to obtain target domain private feature data.

[0028] The trained domain shared feature extractor 12 includes a shared domain image encoder 121, a third RPN module 122, and a third convolution module 123 connected in sequence;

[0029] Among them, the shared domain image encoder 121 is used to encode the industrial defect image to obtain a fifth encoded image, the third RPN module 122 is used to process the fifth encoded image to obtain a fifth encoded image carrying a candidate frame, the fifth encoded image and the fifth encoded image carrying the candidate frame are concatenated to obtain a sixth encoded image, and the third convolution module 123 is used to process the sixth encoded image to obtain domain shared feature data.

[0030] The first convolution module 1113, the second convolution module 1123 and the third convolution module 123 are all VGG-19 convolutional neural networks.

[0031] Before step S20, the industrial defect detection further includes:

[0032] S51, constructing an image training set, where the image training set includes source domain data and target domain data, wherein the labeled training set is the source domain data, and the unlabeled training set is the target domain data;

[0033] Specifically, the industrial target defect image dataset is divided into a training set and a test set. For the training set, the high-resolution image is first blurred using a Gaussian blur kernel, and the initial training set is obtained by downsampling through bicubic interpolation. Then, the training set data is enhanced by flipping and rotating. The training set is divided into source domain data D according to whether there is a label or not. s and target domain data D t , where x s and x t are data samples from the source domain and the target domain respectively.

[0034] S52, processing the source domain data based on the domain shared feature extractor 12 to obtain source domain shared feature data and a source domain shared feature matrix;

[0035] S53, processing the target domain data based on the domain shared feature extractor 12 to obtain target domain shared feature data and a target domain shared feature matrix;

[0036] S54, processing the source domain data based on the source domain private feature extractor 111 to obtain a source domain private feature matrix;

[0037] S55, processing the target domain data based on the target domain private feature extractor 112 to obtain a target domain private feature matrix;

[0038] S56, obtaining a first product term based on the transpose of the source domain shared feature matrix and the square of the Frobenius norm of the source domain private feature matrix, obtaining a second product term based on the transpose of the target domain shared feature matrix and the square of the Frobenius norm of the target domain private feature matrix, and obtaining a first loss function based on the sum of the first product term and the second product term;

[0039] S57, iteratively train the domain shared feature extractor 12 and the domain private feature extractor 11 until the eigenvector of the source domain private feature matrix is ​​orthogonal to the eigenvector of the source domain shared feature matrix, the eigenvector of the target domain private feature matrix is ​​orthogonal to the eigenvector of the target domain shared feature matrix, and the value of the first loss function is minimized, thereby obtaining the trained domain shared feature extractor 12 and domain private feature extractor.

[0040] The source domain data and the target domain data share a common domain shared feature extractor 12 to extract domain shared features:

[0041]

[0042] in and They represent the shared feature data in the source domain and the shared feature data in the target domain, respectively, and E c represents the domain-shared feature extractor 12. The source domain data and the target domain data use their own private feature extractors to extract domain-private features:

[0043]

[0044] in, and Respectively represent the source domain private feature data and the target domain private feature data, and They represent the source domain private feature extractor 111 and the target domain private feature extractor 112 respectively. The domain shared feature extractor 12 and the domain private feature extractor 11 can be composed of an image encoder, a convolutional neural network and a region proposal network (RPN). The convolutional neural network is used to extract the features of the entire image, and the convolutional neural network can be a VGG-19 convolutional neural network. The RPN network is used to generate defect candidate boxes, and then the feature map of the candidate boxes is concatenated with the features of the entire image output by the image encoder to obtain the final feature data with the candidate boxes, and a feature matrix with batch is obtained. and in, and Represent the source domain shared feature matrix, the target domain shared feature matrix, the source domain private feature matrix, and the target domain private feature matrix respectively. The transposed matrix and the source domain private feature matrix Multiply to get the first product term, the target domain shares the feature matrix The transposed matrix of the target domain private feature matrix Multiply them to get the second product term, add the two product terms to get the extraction loss function, expressed as:

[0045]

[0046] in They represent the transposed matrix of the source domain shared feature matrix and the transposed matrix of the target domain shared feature matrix, respectively. Represents the square of the Frobenius norm. Minimize the first loss function to ensure that the eigenvectors of the source domain private feature matrix are orthogonal to the eigenvectors of the source domain shared feature matrix, and the eigenvectors of the target domain private feature matrix are orthogonal to the eigenvectors of the target domain shared feature matrix, so that the domain private features and domain shared features of each dimension will be independent of each other, completing the decoupling of domain private features and domain shared features.

[0047] S30, processing the source domain private feature data and the target domain shared feature data based on the trained classification neural network to obtain a first label type and a second label type accordingly;

[0048] Specifically, the trained classification neural network includes a first fully connected neural network 16, which includes a plurality of first fully connected layers connected in sequence, each of which is followed by an activation layer, wherein the first activation layer is a softmax function.

[0049] Before step S30, the industrial defect detection further includes:

[0050] S61, inputting source domain shared feature data into a pre-built classification neural network to obtain a first prediction result;

[0051] Among them, the first prediction result is the category label of the source domain shared feature data output by the classification neural network.

[0052] S62, obtaining a category label of the source domain data based on the label of the source domain data;

[0053] S63, determining a second loss function based on the cross entropy of the first prediction result and the category label of the source domain data;

[0054] Among them, the expression of the second loss function is:

[0055]

[0056] Wherein, N represents the number of candidate boxes selected from the first encoded image with candidate boxes, A single source domain data sample One-hot category labels, is the category prediction result of the source domain shared feature data after the classification neural network, that is, the source domain shared feature data, where the source domain shared feature data can be specifically expressed as:

[0057]

[0058] Where σ represents the softmax activation function and C represents the classification neural network.

[0059] The classification neural network in the present invention includes a plurality of fully connected layers of artificial neural networks, each fully connected layer is followed by a softmax activation function, and the softmax activation function is used to output the probability distribution of each category.

[0060] S64, taking the second loss function as the objective function, training the pre-constructed classification neural network until the second loss function reaches a minimum, thereby obtaining a trained pre-classification neural network;

[0061] The classification neural network is pre-trained using the source domain shared feature data, and the classification neural network obtains the multi-classification capability of the domain shared features by minimizing the second loss function. The classification neural network can be obtained by training the second loss function of the classification neural network with the source domain shared feature data.

[0062] S65, inputting the source domain private feature data into the first gradient inversion layer to obtain second source domain private feature data;

[0063] S66, inputting the second source domain private feature data into the pre-classification neural network to obtain a second prediction result;

[0064] S67, determining a third loss function based on the category label of the source domain data and the cross entropy of the second prediction result;

[0065] S68. Taking the third loss function as the objective function, train the pre-classification neural network until the third loss function is minimized to obtain a trained classification neural network.

[0066] The server may obtain a trained classification neural network by executing steps S61-S68. The training process of the classification neural network may include:

[0067] Secondly, the server sends the private features of the source domain The input is fed into a pre-trained classification neural network, and the server minimizes the classifier output and the category label y of the source domain data s The server uses the first gradient reversal layer GRL to automatically reverse the gradient before backpropagating to the source domain private feature extractor 11, prompting the source domain private feature extractor 11 to optimize in the direction of maximizing the cross entropy loss, so that the classification neural network cannot correctly classify the domain private features of the source domain, and the source domain private features are determined by the classification neural network. The third loss function is defined as:

[0068]

[0069] in, is the prediction result of the category label of the source domain private feature data by the classification neural network after the softmax activation function. The category label of the source domain private feature data is specifically expressed as:

[0070]

[0071] Among them, G represents the first gradient reversal layer, and its mathematical expression is:

[0072] G λ (x) = x

[0073]

[0074] Among them, I represents the unit matrix. In the gradient reversal layer, the parameter λ changes dynamically, and its change expression is shown in the following formula:

[0075]

[0076] Among them, t represents the relative value of the training process, that is, the ratio of the current number of iterations to the total number of iterations, and γ is a constant of 10.

[0077] After step S68, the industrial defect detection further includes:

[0078] S69, input the target domain shared feature data into the classification neural network to obtain a pseudo label. Finally, the server Input into the trained classification neural network to get the pseudo label.

[0079] S40. The domain identifier 15 determined according to the first label type and the second label type processes the domain shared feature data to obtain a first probability value and a second probability value, and determines the defect classification detection result of the industrial defect image based on the first probability value and the second probability value, wherein the first probability value is the probability that the output result of the domain identifier 15 is judged to be the private feature data of the source domain, and the second probability value is the probability that the output result of the domain identifier 15 is judged to be the private feature data of the target domain.

[0080] Specifically, in order to prevent the domain identifier 15 from correctly distinguishing which domain the shared features come from, the server can perform category-based feature alignment after obtaining the category labels of the source domain private feature data and the target domain private feature data. The server sets an independent domain identifier 15 for each category. The structure of the category-based domain identifier 15 used in the present invention is as follows: Figure 3 As shown, three fully connected layers and activation layers are used, and the second gradient reversal layer is used to cooperate with the category-based domain discriminator 15 training. In the forward propagation process, the category-based alignment acts as an identity transformation. When the gradient is updated to the domain discriminator 15 in the back propagation, the gradient is updated in the direction of improving the accuracy of the domain discriminator 15. When updated to the feature extraction layer, the gradient is reversed and updated in the direction of reducing the classification accuracy of the domain discriminator 15. If the domain shared feature data label or pseudo-label is c∈{1,2,...,n}, the domain shared feature data is input into the domain discriminator 15, and finally two values ​​are generated, respectively representing the probability p1 that the domain discriminator 15 believes that the candidate box comes from the source domain data and the probability p2 from the target domain data. Feature alignment uses a square loss function to achieve strong feature alignment. The source domain data and the target domain data use the same weights. The loss formula is expressed as:

[0081]

[0082] Where D ins is the domain discriminator based on class alignment15, M is the maximum number of categories of industrial defects, δ c Domain identifiers 15 representing different categories may correspond to different weights.

[0083] Specifically, the domain identifier 15 includes a second gradient reversal layer 151 and a third fully connected neural network 152 connected in sequence, wherein the third fully connected neural network 152 includes three second fully connected layers connected in sequence and a second activation layer arranged after each second fully connected layer.

[0084] Before step S40, the industrial defect detection further includes:

[0085] S71, inputting domain shared feature data into the pre-built domain identifier 15 to obtain source domain shared feature identification data and target domain shared feature identification data; determining a fourth loss function of the domain identifier 15 based on the source domain shared feature identification data and the target domain shared feature identification data, the fourth loss function being a feature alignment square loss function;

[0086] S72 . Train the domain identifier 15 using the fourth loss function as the objective function to obtain the trained domain identifier 15 .

[0087] After step S72, the industrial defect detection further includes:

[0088] S73, obtaining a pre-built third fully connected neural network 18;

[0089] S74, inputting the source domain shared feature data, the target domain shared feature data, the source domain private feature data, and the target domain private feature data into the third fully connected neural network 18 to obtain reconstructed feature data;

[0090] S75. Construct a fifth loss function based on the reconstructed feature data and the original feature data, wherein the original feature data includes source domain data and target domain data.

[0091] Among them, the server recombines the coupled features into the original data form. In order to ensure that the domain shared feature data and domain private feature data after decoupling do not lose the information of sample features, a decoder, that is, the third fully connected neural network 18, is built. The input is the combination of the domain shared feature data and domain private feature data of the source domain and the target domain respectively. The reconstructed feature data is obtained by decoding. The constraints are imposed by minimizing the regression loss function of the reconstructed feature data and the original feature data, and the model is optimized to reduce the loss of the original feature data in the feature decoupling process. The regression loss function of the decoder is the fifth loss function, and the fifth loss function can be defined as:

[0092]

[0093] Among them, f s and f t They represent the source domain feature data and the target domain feature data respectively, De is the decoder, and L MSE is the fifth loss function.

[0094] Through the above steps, the trained modules of the domain adaptive multi-class defect detection model based on class alignment are obtained. After obtaining the trained modules, all the aforementioned loss functions are multiplied by a weight coefficient, and the sum is used as the total loss function. The total loss function is optimized using the stochastic gradient descent method. The total loss function is expressed as:

[0095]

[0096] where λ i ,i∈{1,2,3,4,5} is the weight coefficient value of each loss function, and L is the total loss function of the domain adaptive multi-class defect detection model based on class alignment. This total loss function value reflects the difference between the model detection result and the actual detection result, and is used to measure the quality of the domain adaptive multi-class defect detection model based on class alignment. The smaller the total loss function value, the better the effect of the domain adaptive multi-class defect detection model based on class alignment.

[0097] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0098] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A domain adaptive multi-class defect intelligent detection method based on class alignment, characterized in that: include: Acquire an industrial defect image and input it into a pre-built domain adaptive multi-class defect detection model based on class alignment, wherein the domain adaptive multi-class defect detection model based on class alignment includes a domain shared feature extractor, a domain private feature extractor, a classification neural network and a domain discriminator; Processing the industrial defect image based on a domain-shared feature extractor to obtain domain-shared feature data, and processing the industrial defect image based on a trained domain-private feature extractor to obtain domain-private feature data, wherein the domain-shared feature data includes target domain-shared feature data, and the domain-private feature data includes source domain-private feature data and target domain-private feature data; Processing the source domain private feature data and the target domain shared feature data based on a classification neural network to obtain a first label type and a second label type accordingly; The domain identifier determined according to the first label type and the second label type processes the domain shared feature data to obtain a first probability value and a second probability value, and the defect classification detection result of the industrial defect image is determined based on the first probability value and the second probability value, wherein the first probability value is the probability that the output result of the domain identifier is judged to be the source domain private feature data, and the second probability value is the probability that the output result of the domain identifier is judged to be the target domain private feature data.

2. According to claim 1, a domain adaptive multi-class defect intelligent detection method based on class alignment is characterized in that: The trained domain-private feature extractor includes: a source domain-private feature extractor and a target domain-private feature extractor; The source domain private feature extractor includes a source domain image encoder, a first RPN module and a first convolution module connected in sequence, the source domain image encoder is used to encode the industrial defect image to obtain a first encoded image, the first RPN module is used to process the first encoded image to obtain a first encoded image carrying a candidate frame, the first encoded image and the first encoded image carrying the candidate frame are feature-concatenated to obtain a second encoded image, and the first convolution module is used to process the second encoded image to obtain the source domain private feature data; The target domain private feature extractor includes a target domain image encoder, a second RPN module and a second convolution module connected in sequence, the target domain image encoder is used to encode the industrial defect image to obtain a third encoded image, the second RPN module is used to process the third encoded image to obtain a third encoded image carrying a candidate frame, the third encoded image and the third encoded image carrying the candidate frame are feature-concatenated to obtain a fourth encoded image, and the second convolution module is used to process the fourth encoded image to obtain the target domain private feature data; The first convolution module and the second convolution module are both VGG-19 convolutional neural networks.

3. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 2 is characterized in that: The domain shared feature extractor includes a shared domain image encoder, a third RPN module and a third convolution module connected in sequence; The shared domain image encoder is used to encode the industrial defect image to obtain a fifth encoded image, the third RPN module is used to process the fifth encoded image to obtain a fifth encoded image carrying a candidate frame, the fifth encoded image and the fifth encoded image carrying the candidate frame are feature-concatenated to obtain a sixth encoded image, and the third convolution module is used to process the sixth encoded image to obtain the domain shared feature data; The third convolution module is a VGG-19 convolutional neural network.

4. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 3 is characterized in that: Before the trained domain shared feature extractor is used to process the industrial defect image to obtain domain shared feature data, the industrial defect detection further includes: Constructing an image training set, wherein the image training set includes a labeled training set and an unlabeled training set, wherein the data in the labeled training set is source domain data, and the data in the unlabeled training set is target domain data; Processing the source domain data based on the domain shared feature extractor to obtain source domain shared feature data and a source domain shared feature matrix; Processing the target domain data based on the domain shared feature extractor to obtain target domain shared feature data and a target domain shared feature matrix; Processing the source domain data based on the source domain private feature extractor to obtain a source domain private feature matrix; Processing the target domain data based on the target domain private feature extractor to obtain a target domain private feature matrix; Obtaining a first product term based on the transpose of the source domain shared feature matrix and the square of the Frobenius norm of the source domain private feature matrix, obtaining a second product term based on the transpose of the target domain shared feature matrix and the square of the Frobenius norm of the target domain private feature matrix, and obtaining a first loss function based on the sum of the first product term and the second product term; Iteratively train the domain shared feature extractor and the domain private feature extractor until the eigenvector of the source domain private feature matrix is ​​orthogonal to the eigenvector of the source domain shared feature matrix, the eigenvector of the target domain private feature matrix is ​​orthogonal to the eigenvector of the target domain shared feature matrix, and the value of the first loss function is minimized, thereby obtaining the trained domain shared feature extractor and the domain private feature extractor.

5. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 1 is characterized in that: The classification neural network includes a first fully connected neural network, which includes a plurality of first fully connected layers connected in sequence, and each of the first fully connected layers is connected to a first activation layer.

6. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 4 is characterized in that: Before processing the source domain private feature data and the target domain shared feature data based on the trained classification neural network, the industrial defect detection further includes: Inputting source domain shared feature data into the pre-built classification neural network to obtain a first prediction result; Obtaining a category label of the source domain data based on the label of the source domain data; Determine a second loss function based on the cross entropy of the first prediction result and the category label of the source domain data; Taking the second loss function as the objective function, training the pre-constructed classification neural network until the second loss function reaches a minimum, thereby obtaining a trained pre-classification neural network; Inputting the source domain private feature data into the first gradient reversal layer to obtain second source domain private feature data; Inputting second source domain private feature data into the pre-classification neural network to obtain a second prediction result; Determine a third loss function based on the category label of the source domain data and the cross entropy of the second prediction result; Taking the third loss function as the objective function, the pre-classification neural network is trained until the third loss function is minimized, thereby obtaining the trained classification neural network.

7. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 6 is characterized in that: After the trained classification neural network is obtained, the industrial defect detection further includes: The target domain shared feature data is input into the classification neural network to obtain a pseudo label, wherein the pseudo label is a type label of the target domain shared feature data.

8. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 1 is characterized in that: The domain identifier includes a second gradient reversal layer and a second fully connected neural network connected in sequence, wherein the second fully connected neural network includes three second fully connected layers connected in sequence and a second activation layer arranged after each of the second fully connected layers.

9. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 1 is characterized in that: Before the domain identifier determined according to the first label type and the second label type processes the domain shared feature data, the industrial defect detection further includes: Inputting domain shared feature data into the pre-built domain discriminator to obtain source domain shared feature identification data and target domain shared feature identification data; determining a fourth loss function of the domain discriminator based on the source domain shared feature identification data and the target domain shared feature identification data, wherein the fourth loss function is a feature alignment square loss function; The fourth loss function is used as the objective function to train the domain identifier to obtain a trained domain identifier.

10. The domain adaptive multi-class defect intelligent detection method based on class alignment according to claim 4, characterized in that: After obtaining the domain identifier, the industrial defect detection further includes: Get the pre-built third fully connected neural network; Inputting source domain shared feature data, target domain shared feature data, source domain private feature data, and target domain private feature data into the third fully connected neural network to obtain reconstructed feature data; A fifth loss function is constructed based on the reconstructed feature data and the original feature data, wherein the original feature data includes the source domain data and the target domain data.

Images