A defect detection method and system based on differentiable neural network architecture search

Abstract

The present invention discloses a defect detection method and system based on differentiable neural network architecture search. The method inputs the sample to be detected and the normal background sample into the defect detection model to obtain the defect detection result of the sample to be detected; wherein the defect detection model includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence; the backbone network is obtained by searching using a differentiable neural network architecture; the normal sample reference module outputs a multi-scale comparison reference map to provide a normal sample reference mechanism for the defect detection process; the multi-scale fusion module obtains the fusion feature map of the sample to be detected based on the multi-scale information fusion mechanism; the segmentation network segments the fusion feature map of the sample to be detected based on the multi-scale comparison reference map to obtain the defect detection result of the sample to be detected. The present invention can realize automatic detection of product defects and improve the accuracy and generalization of detection.

Description

Technical Field

[0001] The present invention relates to the field of defect detection, and in particular to a defect detection method and system based on differentiable neural network architecture search. Background Art

[0002] Product quality and production efficiency are important manifestations of industrial manufacturing capabilities. Detecting whether there are defects on the surface of products is an important part of ensuring product quality. With the improvement of mechanical level and the development of control technology, less and less manpower is required in the process of a product from the generation of parts to the assembly process. However, the quality inspection link after the product is assembled is often completed by human eyes. Due to the particularity, complexity and professionalism of product surface defects, the rate and accuracy of manual inspection are low, and long-term naked eye inspection work will also cause harm to the health of quality inspection workers. The above facts run counter to the general trend of industrial automation and the requirements of high-precision inspection. Defect detection methods based on machine vision and computer vision are gradually emerging and gradually play a role, which is very important for improving the level of industrial automation.

[0003] Traditional defect detection mainly relies on machine vision methods based on manual features to design defect detection systems. Scholars and experts summarize the color, texture, and geometric shape rules of specific industrial product surfaces, and then design industrial product surface defect detection methods suitable for specific categories and scenarios. The detection method based on traditional machine vision has improved the level of detection automation to a certain extent and has played a role in the product production line. However, for a certain industrial product, the manual feature extraction method requires experts and scholars to conduct a lot of subjective analysis of the data, and then decide the features to be extracted according to self-awareness and apply them to the algorithm. There are many kinds of products in the world, so the manual feature method requires a special analysis and design for a product, and a product also needs to be analyzed and designed once after it is updated. Although it replaces manual quality inspection to a certain extent, it requires scholars and experts to devote energy to tedious manual design, and the level of automation is not high to a large extent. In complex industrial imaging scenarios, affected by a series of factors such as lighting, the accuracy and generalization of this detection method are also difficult to achieve good results. Summary of the invention

[0004] Based on this, an embodiment of the present invention provides a defect detection method and system based on a differentiable neural network architecture search, which realizes automatic detection of product defects and improves the accuracy and generalization of detection.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A defect detection method based on differentiable neural network architecture search, comprising:

[0007] Obtain samples to be tested and normal background samples;

[0008] Inputting the sample to be detected and the normal background sample into a defect detection model to obtain a defect detection result of the sample to be detected;

[0009] Wherein, the defect detection model includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence;

[0010] The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output a multi-scale feature map of the sample to be detected and a multi-scale feature map of the normal background sample;

[0011] The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected;

[0012] The multi-scale fusion module is used to fuse the multi-scale feature maps of the sample to be detected to obtain the fused feature map of the sample to be detected;

[0013] The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected.

[0014] Optionally, the method for determining the trained backbone network is:

[0015] Determine a backbone network; the backbone network includes a multi-channel neural network structure search space and an inter-node search space;

[0016] A multi-scale fusion segmentation head is connected to the output of the backbone network; the multi-scale fusion segmentation head is determined by using a method of adding corresponding pixels of up-sampled serial feature maps;

[0017] The data set is input into the backbone network, the multi-scale fusion segmentation head fuses the multi-scale feature map of the data set output by the backbone network, and segments the fused feature map of the data set to obtain a segmentation prediction result; the data set includes a training set and a validation set;

[0018] Based on the segmentation prediction result, the parameters in the backbone network are trained using a gradient descent method to obtain a backbone training network;

[0019] The backbone training network is discretized by using a maximum retention method and an optimal path retention method with a fixed end node to obtain a discretized network, and the discretized network is determined as a trained backbone network.

[0020] Optionally, the multi-channel neural network structure search space includes M series channels; each of the series channels includes a plurality of channel units connected in series, each of the channel units is composed of a plurality of nodes connected in sequence, and the end node of each channel unit fuses the nodes of the channel unit except the end node in a jumping manner; from the first series channel to the Mth series channel, the number of nodes in each series channel is reduced by one in sequence; the first series channel is used to input images; adjacent series channels are connected in an upsampling and downsampling manner; the scale of the feature map output by each series channel is different.

[0021] Optionally, the inter-node search space includes a scale-invariant connection, a downsampling connection and an upsampling connection between two nodes; the scale-invariant connection is a connection between two nodes through a candidate change operation; the downsampling connection is a connection between two nodes through a candidate change operation, a resolution change operation and a scale transformation operation in sequence; the upsampling connection is the same as the downsampling connection; the candidate change operation includes a jump direct connection, a first convolution, a hole convolution, a deformable convolution and a separable convolution; the resolution change operation is a bilateral linear interpolation; and the scale transformation operation is a second convolution.

[0022] Optionally, the mechanism adopted by the normal sample reference module is a normal sample reference mechanism based on semantic embedding vectors.

[0023] Optionally, the multi-scale fusion module includes multiple scale fusion units and element addition units;

[0024] Each of the fusion units is used to fuse two feature maps of different scales in the multi-scale feature map of the sample to be detected to obtain a fused sub-map; the multiples of the scale difference of the two feature maps of different scales corresponding to each of the fusion units are the same;

[0025] The element adding unit is used to add the fusion sub-graphs output by all the fusion units to obtain the fusion feature graph of the sample to be detected.

[0026] Optionally, the multi-channel neural network structure search space includes four serially connected channels;

[0027] Among them, the node inputs in the second and third series channels are:

[0028] X i，j =β down ×f down (Xi-1，j-1 )+β keep ×f keep (X i，j-1 )+β up ×f up (X i+1，j-1 );

[0029] Where i represents the serial number of the serial channel, i=2 or 3; j represents the node number; X i，j represents the input of the jth node in the i-th series channel; X i，j-1 represents the input of the j-1th node in the i-th series channel; X i-1，h-1 represents the input of the j-1th node in the i-1th series channel; X i+1，j-1 represents the input of the j-1th node in the i+1th serial channel; f keep (X i，j-1 ) indicates that X i，j-1 The transformation of the scale-invariant connection; f up (X i+1，j-1 ) indicates that X i+1，j-1 The transformation of the upsampling connection; f dowm (X i-1，j-1 ) indicates that X i-1，j-1 The transformation of the downsampling connection; β keep Indicates the value obtained by calculating the Softmax of the parameters of the invariant scale connection; β down Indicates the value obtained by calculating the Softmax of the parameters of the downsampling connection; β up Indicates the value obtained by calculating the Softmax of the parameters of the upsampling connection.

[0030] Optionally, the output of the candidate change operation is:

[0031] y m =∑ o∈O β o ×f o (x m );

[0032] Among them, x m represents a node in the inter-node search space; y m Represents x m The output after the candidate change operation; O represents the set of candidate change operations, O = {skip, conv, diated, deform, sep}, f skip Indicates a jump direct connection; f conv represents the first convolution; f diated represents the dilated convolution; f deform represents deformable convolution; f sep represents separable convolution; β skipIndicates the value obtained by calculating the Softmax of the parameters of the jump direct connection; β conv Indicates the value obtained by calculating the Softmax of the first convolution parameter; β diated Indicates the value obtained by calculating the Softmax of the parameters of the dilated convolution; β deform Indicates the value obtained by calculating the Softmax of the deformable convolution parameters; β sep Indicates the value obtained by calculating Softmax on the parameters of the separable convolution.

[0033] Optionally, the distance is calculated as follows:

[0034]

[0035] Where p represents the pixel in the multi-scale feature map of the sample to be detected; q represents the pixel corresponding to p in the multi-scale feature map of the normal background sample; dis(p, q) represents the distance between p and q; e p represents the semantic embedding vector corresponding to p; e q Represents the semantic embedding vector corresponding to q.

[0036] The present invention also provides a defect detection system based on differentiable neural network architecture search, comprising:

[0037] A sample acquisition module is used to acquire samples to be detected and normal background samples;

[0038] A defect detection module, used for inputting the sample to be detected and the normal background sample into a defect detection model to obtain a defect detection result of the sample to be detected;

[0039] Wherein, the defect detection model in the defect detection module includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence;

[0040] The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output a multi-scale feature map of the sample to be detected and a multi-scale feature map of the normal background sample;

[0041] The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected;

[0042] The multi-scale fusion module is used to fuse the multi-scale feature maps of the sample to be detected to obtain the fused feature map of the sample to be detected;

[0043] The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The embodiment of the present invention proposes a defect detection method and system based on differentiable neural network architecture search, which uses a defect detection model to realize defect detection of samples to be detected. The defect detection model includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence. Among them, the backbone network is obtained by searching using a differentiable neural network architecture, and the network searched using a differentiable neural network architecture can avoid excessive human intervention, reduce the influence of people's subjective factors, and achieve good generalization; at the same time, the neural network automatically searched can better express the characteristics of the data, thereby mining more significant information, breaking through the bottleneck of the manually designed network, and still performing stably in a data set with large differences. The normal sample reference module outputs a multi-scale comparison reference image to provide a normal sample reference mechanism for the defect detection process. The mechanism is not sensitive to whether the product image is aligned, and has the ability to classify the sample defect category, which can improve the automation level; the multi-scale fusion module obtains the fusion feature map of the sample to be detected based on the multi-scale information fusion mechanism. Under the support of the multi-scale information fusion mechanism and the backbone network obtained by the neural network architecture search, high-precision detection can be achieved. Therefore, the present invention realizes automatic detection of product defects and improves the accuracy and generalization of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flowchart of a defect detection method based on differentiable neural network architecture search provided by an embodiment of the present invention;

[0048] Figure 2 A framework diagram of a defect detection model provided by an embodiment of the present invention;

[0049] Figure 3A structural diagram of a multi-channel neural network structure search space provided by an embodiment of the present invention;

[0050] Figure 4 A structural diagram of the inter-node search space provided by an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of the structure of a multi-scale fusion segmentation head provided in an embodiment of the present invention;

[0052] Figure 6 A flowchart of a method for determining a trained backbone network provided by an embodiment of the present invention;

[0053] Figure 7 A schematic diagram of a query mechanism provided by an embodiment of the present invention;

[0054] Figure 8 A structural diagram of a multi-scale fusion module provided in an embodiment of the present invention;

[0055] Fig. 9 A structural diagram of a segmentation network provided by an embodiment of the present invention;

[0056] Fig.10 A training flow chart provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] 1. Differentiable Neural Network Architecture Search

[0060] Defect detection algorithms based on deep convolutional neural networks have replaced manual automated quality inspection to a greater extent, but in the face of complex and diverse defect detection scenarios, scholars still need to make relevant adjustments, which cannot further improve the level of automation. The emergence of neural network architecture search, especially differentiable neural network architecture search technology, has greatly solved the above problems. The core of neural network architecture search technology is to search for network structures suitable for specific scenarios and data sets in discrete search spaces based on the distribution characteristics of data samples, so as to obtain candidate networks and achieve competitive performance without excessive human involvement in network structure design. The differentiable neural network architecture search method weakens the original discrete search space into a continuous space structure and forms a directed acyclic graph. After each optimization of the model weight, the performance of the intermediate model is directly evaluated, and the search weight is gradient decreased, which greatly reduces the resources used in the search process, making it possible to implement product surface defect detection algorithms based on neural network architecture search, and further improving the level of detection automation. Therefore, it is of practical significance to use differentiable neural network architecture search technology to design product surface defect detection methods.

[0061] 2. Normal sample reference mechanism and application of normal sample reference.

[0062] With the improvement of machining technology, the amount of defective sample data generated by the increasingly improved production line is very small, while the number of normal samples is countless. The data-driven deep convolutional neural network algorithm only uses defective samples for training, which results in a waste of normal sample data. From the definition of defects, the semantics of defects are different from the semantics in general scenarios such as "animals" and "buildings". Defects are more inclined to be a relative semantics, that is, non-normal samples are defective samples, which are manifested in the data as differences in shape, color, and area. A reality is that experienced quality inspection workers often compare defect-free products when judging whether there is a defect on a certain part of the product surface. This also brings inspiration to the defect detection algorithm on the product surface: after the convolutional neural network training is completed, the feature distribution is obtained by inputting normal samples as the control amount. When performing defect segmentation on the image to be inspected, the feature is used as a benchmark to analyze the difference between the two to improve the discrimination of defective parts. In theory, the convolutional neural network training process can be regarded as the algorithm's summary of the characteristics of sample defects, and the control can be regarded as a measurement. At the same time, an important fact is that the camera used to shoot sample images on the production line is usually fixed at a specific position at a specific angle, and the posture of the product causes different parts and different placements, which usually have some degree of difference, that is, the product imaging will not be completely aligned, but it will not change too much. Based on the analysis of the above facts, it is necessary and feasible to use normal samples as comparison in defect detection scenarios.

[0063] In past studies, there are two main ways to use normal samples. The first is to establish a reconstruction network through an autoencoder or GAN. The autoencoder reconstruction network compresses the input image to be detected into a low-dimensional space to extract representative information, and then reshapes it through a decoder. GAN uses the same principle, but the image reconstruction is generated by a generator. During the training process, only normal samples are used and defective samples are completely ignored. The reconstructor recognizes the characteristics of normal samples. In the actual detection process, the difference between the network output and the input is used to determine whether a defect exists. The second method is to use a twin network. During the training and testing process, by constructing pairs of defective samples and normal samples, the location of the defective sample is characterized by the difference between the heat map of the defective sample output by the network and the heat map of the normal sample.

[0064] The algorithm based on reconstruction network can only work in scenes with relatively uniform background texture, and it cannot judge the defect category because it has not learned the defect data. The method based on comparing heat of twin network directly subtracts the pixels of heat map, so it must require the product images to be completely aligned, which is not suitable for most defect detection scenarios.

[0065] Based on the above analysis, it is urgently needed by academia and industry to use normal samples as references in defect detection scenarios, design a more general comparison method, and use reasonable comparison significance results to correct the segmentation results of the network. This has important practical significance for improving the level of industrial automation.

[0066] 3. Multi-scale information fusion mechanism

[0067] In convolutional neural networks, the input image will obtain more recognizable semantic information after more convolution transformations, which improves the discrimination of whether a certain area of ​​the image is a defect. However, under the transformation of continuous convolution and downsampling, the texture detail information contained in the feature map becomes less and less, which affects the determination of the boundary of the target area. In the network structure obtained in the search space, large-scale features often undergo less convolution encoding and thus have more sufficient detail information, while small-scale features undergo pooling and more convolutions and have rich semantic information. Therefore, the fusion mechanism of multi-scale information has always been one of the important research points in the field of convolutional neural networks.

[0068] This is particularly important in defect detection scenarios. Defective parts tend to focus on low-level features such as color, texture, and shape, which is different from normal parts. This is different from general scenarios that focus on high-level features. Therefore, a suitable multi-scale fusion mechanism can help the algorithm improve its accuracy, and there is a need for it.

[0069] Based on the above three points, this embodiment provides a defect detection method based on differentiable neural network architecture search, such as Figure 1 As shown, the method combines the normal sample reference mechanism and the multi-scale information fusion mechanism. The method comprises:

[0070] Step 101: Obtain a sample to be detected and a normal background sample. The normal background sample is a sample without defects.

[0071] Step 102: Input the sample to be detected and the normal background sample into the defect detection model to obtain the defect detection result of the sample to be detected; the defect detection model includes a trained backbone network 1, a normal sample reference module 2, a multi-scale fusion module 3 and a segmentation network 4 connected in sequence. The framework of the defect detection model is as follows Figure 2 As shown in the figure, the defect detection result is the pixel-level segmentation result of the surface defect area.

[0072] 1) Trained backbone network

[0073] The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output a multi-scale feature map of the sample to be detected and a multi-scale feature map of the normal background sample.

[0074] The method for determining the trained backbone network is:

[0075] ① Determine the backbone network. Use the differentiable neural network architecture search technology to determine the appropriate network architecture as the backbone network of the overall method based on the characteristics of the data set. The search space of the backbone network includes the multi-channel neural network structure search space and the node search space. This step includes a series of sub-steps for differentiable neural network search, specifically:

[0076] Step 1.1 Setting up the search space for multi-channel neural network architecture

[0077] This embodiment adopts a multi-channel parallel and densely connected network search space. The multi-channel neural network structure search space includes M series channels; each of the series channels includes a plurality of channel units connected in series, each of the channel units is formed by connecting a plurality of nodes in sequence, and the end node of each channel unit fuses the nodes of the channel unit except the end node in a jumping manner; from the first series channel to the Mth series channel, the number of nodes in each series channel decreases by one; the first series channel inputs an image; adjacent series channels are connected in an upsampling and downsampling manner; the scale of the feature map output by each series channel is different.

[0078] The following is a detailed description using four series channels as an example. In the search space of the four-channel neural network structure, four scales are set for the image downsampling ratio. Assuming that the resolution of the input image after convolution downsampling is S, the resolutions of the four scales in the search space are S, S / 2, S / 4, and S / 8. Assume that the first series channel with a resolution of S contains L nodes, and each subsequent series channel reduces one node in turn. This is to ensure that the number of operations from the input to a column of nodes of each scale is consistent, and adjacent series channels interact densely with upsampling and downsampling connections, such as Figure 3 shown.

[0079] For the weakened continuous space analysis, assume that the input of the jth node in the i-th cascade channel in the search space is represented by X i，j (i.e., taking the non-merged nodes of the second serial channel with a resolution of S / 2 and the second serial channel with a resolution of S / 4 as an example, the number of input connections of the other serial channels is 2), the input has an invariant scale input X i，j-1 , downsample the input X i-1，j-1 And the upsampled input X i+1，j-1 , assuming that the transformation of the three connections is f keep , f up ,f dowm The architectural parameters of the three connections are set to α i，keep ,α i，up and α i，down , then the input of the jth node in the i-th series channel is expressed as:

[0080] X i，j =β down ×f down (X i-1，h-1 )+β keep ×f keep (X i，j-1 )+β up ×f up (X i+1，j-1 );

[0081] Where i represents the serial number of the serial channel, i=2 or 3; j represents the node number; X i，j-1 represents the input of the j-1th node in the ith series channel, that is, X i，j The invariant scale input; X i-1，j-1 represents the input of the j-1th node in the i-1th series channel, that is, X i，j Downsampled input; X i+1，j-1 represents the input of the j-1th node in the i+1th series channel, that is, X i，j The upsampled input of keep (X i，j-1 ) indicates that Xi，j-1 The transformation of the scale-invariant connection; f up (X i+1，j-1 ) indicates that X i+1，j-1 The transformation of the upsampling connection; f dowm (X i-1，j-1 ) indicates that X i-1，j-1 The transformation of the downsampling connection; β keep The parameter α represents the scale-invariant connection i，keep Find the value obtained by Softmax; β down Represents the parameter α for the downsampling connection i，down Find the value obtained by Softmax; β up Represents the parameter α for the upsampling connection i，up Find the value obtained by Softmax.

[0082] β keep For example, the calculation formula is as follows:

[0083]

[0084] α is used as an architecture parameter and is optimized alternately with the model weight parameter during the search process. There is a competitive relationship between the three inputs of the same node. After the network search is completed, relevant methods will be used to retain the connections with high significance.

[0085] Step 1.2 Setting up the search space between nodes

[0086] In the multi-channel search space of this embodiment, the edges between nodes are represented as candidate connection operations, and the edges are divided into scale-invariant connections, down-sampling connections, and up-sampling connections according to the different connected nodes. For the search space between nodes, the inter-node search space adopted in this embodiment is as follows Figure 4 shown.

[0087] The inter-node search space includes a scale-invariant connection, a downsampling connection, and an upsampling connection between two nodes. The scale-invariant connection is responsible for the processing and change of the feature map. The main candidate change operations are convolution, pooling, or jump connection. In addition to using the operations in the scale-invariant connection, the downsampling connection and the upsampling connection also require additional modules to achieve resolution changes. The main candidate operations considered are pooling, linear interpolation, and transposed convolution. The search space designed in this embodiment is ultimately to serve the defect detection of the product surface, and therefore combines the technical difficulties in the defect detection scenario and the requirements for deployment in the industrial scenario.

[0088] See also Figure 4, the invariant scale connection is a connection between two nodes through a candidate change operation. The downsampling connection is a connection between two nodes through a candidate change operation, a resolution change operation and a scale transformation operation in sequence. The upsampling connection is the same as the downsampling connection. The candidate change operations include jump direct connection, first convolution, hole convolution, deformable convolution and separable convolution, wherein the first convolution can be a 3x3 convolution, the hole convolution can be a hole convolution with a magnification of 2, the deformable convolution can be a 3x3 deformable convolution, and the separable convolution can be a superimposed 3x3 depth separable convolution. For up and down sampling connections, a resolution change operation is added after the output of the core design, such as bilateral linear interpolation to complete the resolution change and a second convolution, such as a 1x1 convolution, is used to complete the channel scale transformation.

[0089] The output of the candidate change operation may represent:

[0090] y m =∑ o∈O β o ×f o (x m );

[0091] Among them, x m represents a node in the inter-node search space; y m Represents x m The output after the candidate change operation; O represents the set of candidate change operations, O = {skip, conv, diated, deform, sep}, f skip Indicates a jump direct connection; f conv represents the first convolution; f diated represents the dilated convolution; f deform represents deformable convolution; f sep represents separable convolution; β skip Represents the parameter α for the jump direct connection skip Find the value obtained by Softmax; β conv Represents the parameter α for the first convolution conv Find the value obtained by Softmax; β diated Represents the parameter α for the dilated convolution diated Find the value obtained by Softmax; β deform Represents the parameter α of the deformable convolution deform Find the value obtained by Softmax; β sep Represents the parameter α for the separable convolution sep Find the value of Softmax. skip For example, the calculation formula is as follows:

[0092]

[0093] Figure 4 The β in represents the direct connection to avoid the phenomenon of “direct connection enrichment”. The value of β is a hyperparameter, which decays linearly to 0 as the network training progresses.

[0094] ② Design a search segmentation head, and connect a multi-scale fusion segmentation head to the output of the backbone network; the multi-scale fusion segmentation head is determined by adding the corresponding pixels of the up-sampled series feature map. Specifically, as shown in step 1.3:

[0095] Step 1.3 Design of multi-scale fusion segmentation head

[0096] After searching for a network structure that is highly adaptable to the data set as the backbone network, connect it to the normal sample reference module, the multi-scale fusion module, and the segmentation network to form the final network structure. If the backbone network structure is searched directly on the final network structure, overfitting will occur due to the complexity of the overall task. Therefore, when searching for the network structure, a simple multi-scale fusion segmentation head is connected to the multi-scale output to complete the network training, search, and decoding.

[0097] The design of the multi-scale fusion segmentation head adopts a simple upsampling and concatenating feature maps to add corresponding pixels, such as Figure 5 The multi-scale search space outputs feature results of four scales, represented as x s , x s / 2 , x s / 4 , x s / 8 The fusion segmentation head uses x s / 8 The feature map is upsampled twice by transposed convolution and then added to x s / 4 Sum to get y s / 4 In the same way, we can get y s / 2 and s After convolution, the prediction result is obtained s , pre s / 2 , pre s / 4 ,After the labeled true value is transformed to multiple scales, multi-scale supervised training is performed.

[0098] ③ Differentiable neural network architecture search. The data set is input into the backbone network, and the multi-scale fusion segmentation head fuses the multi-scale feature map of the data set output by the backbone network, and segments the fused feature map of the data set to obtain a segmentation prediction result; based on the segmentation prediction result, the parameters in the backbone network are trained using the gradient descent method to obtain a backbone training network. The data set includes a training set and a validation set. Specifically, as shown in step 1.4:

[0099] Step 1.4 Differentiable Neural Network Architecture Search

[0100] This step uses the search method to perform gradient descent. Assuming that the architecture parameter in the network search space is α and the weight in the model is w, the optimization goal of the search process is:

[0101] min α L val (w * (α), α)

[0102] swt * (α) = argmin w L train (w, α)

[0103] Among them, L val is the loss of the network on the validation set, L train is the loss of the network on the training set. The above formula can be expressed as a classic two-layer optimization problem, where α is the upper layer variable and w is the lower layer variable. The optimization of the architecture parameter α can be performed using gradient descent, but the accurate calculation of the gradient requires the optimization of the weight parameter to the best, and this process consumes a lot of computation. Therefore, the differentiable neural network architecture search technology makes the following approximation:

[0104]

[0105] That is, one-step optimization replaces global optimization. Therefore, the search process can be simply summarized as follows:

[0106] a. Divide the dataset into training set and validation set.

[0107] b. Use the validation set loss to perform gradient descent on the architecture parameters and update the architecture parameters.

[0108] c. Use the training set loss to perform gradient descent on the model weight parameters and update the model parameters.

[0109] ④ Network space decoding method. The backbone training network is discretized using the maximum retention method to obtain a discretized network, and the discretized network is determined as the trained backbone network. Specifically, as shown in step 1.5:

[0110] Step 1.5 Cyberspace decoding method

[0111] After the model search is completed, the network structure needs to be discretized according to the architecture weight parameters, which is also called continuous space decoding. In the local search space between nodes, candidate operations need to be selected according to the architecture parameter values. This embodiment adopts the simplest method - the maximum retention method.

[0112] In the product surface defect detection method proposed in this embodiment, a premise is that the network structure search is to provide an adaptive skeleton network for the segmentation head of defect detection, and the segmentation head requires the features of multiple scales output by the skeleton network. Therefore, it is necessary to retain all the terminal nodes, search for the optimal path from the starting node to each terminal node as the final discretized network, and four paths will be retained in the network structure. In order to further retain more wheel diameters, this embodiment sets the number of paths to each terminal node to be retained to k, which can be selected.

[0113] As for the path retention algorithm, this embodiment adopts a path retention rule based on dynamic programming, which is called the optimal path retention method with a fixed end node.

[0114] Assuming that the number of nodes in the series channel with a search space resolution of S is L, the number of nodes in the series channel with resolutions of S / 2, S / 4, and S / 8 are L-1, L-2, and L-3 respectively. The resolution from the starting node to the ending node is S x The optimal distance to the i-th node of , where i is calculated relative to the starting node, represents the connection architecture weight from the i-th node with resolution S1 to the 1-th node with resolution S2. The general transfer equation obtained in this embodiment is as follows:

[0115] dp i，s =max(dp i-1，s +β i-1，s，s , dp i-1，s / 2 +β i-1，s / 2，s )

[0116]

[0117]

[0118]

[0119] The overall process for step 1) is as follows Figure 6 shown.

[0120] 2) Normal sample reference module

[0121] The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected.

[0122] The mechanism adopted by the normal sample reference module described in this embodiment is a normal sample reference mechanism based on semantic embedding vectors. For each pixel on the multi-scale feature map of the sample to be detected, the optimal matching point is found in the multi-scale feature map of the normal background sample, and the distance between the optimal matching point is used as the comparison result of the pixel. The comparison result of the entire image is used as one of the inputs of the image segmentation network. Since it is a global match, there is no need to consider the problem of product image alignment. At the same time, the network uses defective sample images as training sets, and the defect category of the image can still be judged.

[0123] Semantic embedding vectors and embedding space are important concepts in the field of metric learning. In this field, it is believed that the distance between the semantic embedding vectors of the same object is close, while the distance between the semantic embedding vectors of different objects is far away. Metric learning often uses different object image pairs or positive and negative sample images to perform distance loss on the semantic embedding vectors to train related neural networks.

[0124] The feature map F output by the trained backbone network in step 1) of this embodiment H，W，C It is considered as an embedding space, where H, W and C are the height, width and number of channels of the feature map respectively. The feature map is sliced ​​in the first and second dimensions to form H*W feature vectors with dimension C, which can be represented as semantic embedding vectors.

[0125] Assume that in the feature map F G，W，C There are two pixel points p and q in the multi-scale feature map, p represents the pixel point in the multi-scale feature map of the sample to be detected; q represents the pixel point corresponding to p in the multi-scale feature map of the normal background sample; the calculation formula for the distance between p and q is:

[0126]

[0127] Where dis(p,q) represents the distance between p and q; e p represents the semantic embedding vector corresponding to p; e q Represents the semantic embedding vector corresponding to q.

[0128] dis ranges from 0 to 1. When two pixels are exactly the same:

[0129]

[0130] And when two pixels are completely dissimilar:

[0131]

[0132] Assume that the multi-scale feature map of the sample to be detected is P, the multi-scale feature map of the input normal background sample is Q, and the positive sample reference result is D W*H , the query mechanism is as follows Figure 7 As shown, the calculation formula is as follows:

[0133]

[0134] Where min * It represents the average value of the minimum top5%-top10% of all distances. This is done to avoid the influence of noise.

[0135] 3) Multi-scale fusion module

[0136] Different from the detection in general scenarios, the visual difference between the defective area and the normal area on the surface of the product is mainly manifested in low-order features such as texture and color, while the multi-scale fusion module designed in this embodiment mainly guides the low-order feature information with high-order feature information, and gradually fuses it in multiple scale features. The multi-scale fusion module is used to fuse the multi-scale feature map of the sample to be detected to obtain a fused feature map of the sample to be detected. The multi-scale fusion module includes a plurality of scale fusion units and an element addition unit. Each of the fusion units is used to fuse two feature maps of different scales in the multi-scale feature map of the sample to be detected to obtain a fused sub-map; the scale difference multiples of the two feature maps of different scales corresponding to each fusion unit are the same. The element addition unit is used to add the fused sub-maps output by all fusion units to obtain a fused feature map of the sample to be detected. Specifically:

[0137] Taking the fusion of four feature maps of different scales output by four serial channels as an example, the specific structure of the multi-scale fusion module is as follows: Figure 8 shown.

[0138] On the whole, this embodiment divides the feature maps of four scales into two groups: the feature maps with resolutions of s and s / 4 are divided into one group, and the feature maps with resolutions of s / 2 and s / 8 are divided into another group. As the scale decreases, the detail information contained in the feature is lower and the semantic information is higher. The greater the scale change, the greater the difference, so this embodiment selects the feature map with a scale difference of 4 times to perform the first step of fusion. Take the group of scales s and s / 4 as an example: the feature map of scale s is first subjected to deep convolution to process the information of a single feature map, and then batch normalization (Batch Normalization, BN) is used to standardize and remap the data, and finally, through 1x1 convolution, each element of the same pixel on different feature maps is transformed to output the feature map F detailFor the feature map of scale s / 4, we first use transposed convolution to upsample it by one time, that is, to scale s / 2, and then use the depth-to-space module D2S to reorganize the information to become a feature map F of scale s. semantic . Plan to use F semantic F detail The semantics is filled in through the following formula:

[0139] F full，s =F detial ×sigmoid(F semantic );

[0140] Similarly, the same strategy is used to generate F for resolutions s / 2 and s / 8. full，s / 2 . Then to F full，s The channel size is reduced to HxWxC / 2. For F whose size is originally H / 2xW / 2x2C, full，s / 2 , and once again use the depth-to-space module D2S to convert it into a HxWxC / 2 feature map. The corresponding pixels of the two feature maps are added to obtain the final feature fusion result, that is, the fused feature map.

[0141] 4) Segmentation network

[0142] The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected. Specifically:

[0143] In this embodiment, no direct distance supervision information is added to the normal sample comparison results during the network training process. Instead, the comparison results with the normal samples are input into the segmentation head, and the segmentation loss is used for indirect supervision. Therefore, in this embodiment, the positive sample comparison results are not used as direct segmentation results, but as an existence similar to the attention mechanism. Fig. 9 As shown, Fig. 9 In the above method, the fused feature map (the fused feature map) and the scale comparison reference map (the normal sample reference result) are input into three convolutional layers connected in sequence. After the serial convolution, the softmax function is used for threshold processing to obtain the segmentation result.

[0144] After step 1) is completed, a backbone network of the first stage will be searched. Steps 2)-4) build the functional components of the second stage. Next, the overall training process of the whole method will be shown.

[0145] A supernet is a weakened continuous search space. The weight parameters of a convolutional neural network model are usually assigned using a random initialization method. In the early stages of training, the network's performance is poor, and the effects of each candidate connection cannot directly reflect the role of the connection. The differentiable neural network architecture search method replaces the optimal model weight with the model weight value after one-step tuning. However, when the model weight is very poor, the overall result is still poor after one-step tuning. Therefore, in the early stages of network search, the architecture parameters are frozen, and the model weight parameters are trained for about 20 generations. This stage is called supernet training.

[0146] The second stage of training is the search for the neural network architecture. The training data set will be divided into two parts, one for optimizing the model weights and the other for optimizing the architecture weights. The gradient descent of the two will be performed alternately until the network converges. The third stage is the decoding stage of the network. The decoding method proposed in step 1.5 is used to discretize the continuous space that has been searched to obtain the result network.

[0147] Pre-training models can speed up the convergence of neural networks and improve generalization, which is very helpful for network training. In the fourth stage of training, we replaced the number of categories in the simple segmentation head, used the PASCAL VOC dataset to pre-train the model, and saved the weights of the converged backbone neural network model to complete the pre-training of the model.

[0148] In the fifth stage of network training, the searched neural network structure, normal sample reference mechanism, multi-scale fusion module and segmentation network are combined to form the final model. The pre-trained model saved in the fourth stage is used to initialize the backbone network model weights, and gradient optimization is performed on the defect dataset.

[0149] The training process is as follows Fig.10 shown.

[0150] The defect detection method based on differentiable neural network architecture search in the above embodiment has the following advantages:

[0151] 1. Good stability. The differentiable neural network architecture is used to search for a suitable network structure based on the characteristics of the data set, thus avoiding excessive human intervention, reducing the influence of subjective factors, and achieving good generalization. At the same time, the automatically searched neural network can better represent the characteristics of the data, thereby mining more significant information, breaking through the bottleneck of manually designed networks, and still performing stably in data sets with large differences.

[0152] 2. Strong versatility. The normal sample comparison mechanism used takes into account the global semantic embedding vector, so it is not sensitive to whether the product image is aligned, and has the ability to classify the sample defect category.

[0153] 3. High precision. With the support of the multi-scale information fusion mechanism and the backbone network obtained by the neural network architecture search, the algorithm can achieve high precision through a complete training process.

[0154] The present invention also provides a defect detection system based on differentiable neural network architecture search, comprising:

[0155] The sample acquisition module is used to obtain samples to be detected and normal background samples.

[0156] The defect detection module is used to input the sample to be detected and the normal background sample into a defect detection model to obtain a defect detection result of the sample to be detected.

[0157] Among them, the defect detection model in the defect detection module includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network that are connected in sequence.

[0158] The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output the multi-scale feature map of the sample to be detected and the multi-scale feature map of the normal background sample.

[0159] The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected.

[0160] The multi-scale fusion module is used to fuse the multi-scale feature maps of the sample to be detected to obtain a fused feature map of the sample to be detected.

[0161] The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected.

[0162] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0163] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A defect detection method based on differentiable neural network architecture search, characterized in that: include: Obtain samples to be tested and normal background samples; Inputting the sample to be detected and the normal background sample into a defect detection model to obtain a defect detection result of the sample to be detected; Wherein, the defect detection model includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence; The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output a multi-scale feature map of the sample to be detected and a multi-scale feature map of the normal background sample; The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected; The multi-scale fusion module is used to fuse the multi-scale feature maps of the sample to be detected to obtain the fused feature map of the sample to be detected; The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected.

2. A defect detection method based on differentiable neural network architecture search according to claim 1, characterized in that: The method for determining the trained backbone network is: Determine a backbone network; the backbone network includes a multi-channel neural network structure search space and an inter-node search space; A multi-scale fusion segmentation head is connected to the output of the backbone network; the multi-scale fusion segmentation head is determined by using a method of adding corresponding pixels of up-sampled serial feature maps; The data set is input into the backbone network, the multi-scale fusion segmentation head fuses the multi-scale feature map of the data set output by the backbone network, and segments the fused feature map of the data set to obtain a segmentation prediction result; the data set includes a training set and a validation set; Based on the segmentation prediction result, the parameters in the backbone network are trained using a gradient descent method to obtain a backbone training network; The backbone training network is discretized by using a maximum retention method and an optimal path retention method with a fixed end node to obtain a discretized network, and the discretized network is determined as a trained backbone network.

3. A defect detection method based on differentiable neural network architecture search according to claim 2, characterized in that: The multi-channel neural network structure search space includes M series channels; each of the series channels includes a plurality of series-connected channel units, each of the channel units is formed by sequentially connecting a plurality of nodes, and the end node of each channel unit fuses the nodes of the channel unit except the end node in a jumping manner; From the first serial channel to the Mth serial channel, the number of nodes in each serial channel decreases by one; The first series channel is used to input an image; adjacent series channels are connected in an upsampling and downsampling manner; and the scale of the feature map output by each series channel is different.

4. A defect detection method based on differentiable neural network architecture search according to claim 2, characterized in that: The inter-node search space includes a scale-invariant connection, a downsampling connection and an upsampling connection between two nodes; the scale-invariant connection is a connection between two nodes through a candidate change operation; the downsampling connection is a connection between two nodes through a candidate change operation, a resolution change operation and a scale transformation operation in sequence; the upsampling connection is the same as the downsampling connection; the candidate change operation includes a jump direct connection, a first convolution, a hole convolution, a deformable convolution and a separable convolution; the resolution change operation is a bilateral linear interpolation; the scale transformation operation is a second convolution.

5. The defect detection method based on differentiable neural network architecture search according to claim 1 is characterized in that: The mechanism adopted by the normal sample reference module is a normal sample reference mechanism based on semantic embedding vectors.

6. According to the defect detection method based on differentiable neural network architecture search in claim 1, the multi-scale fusion module comprises a plurality of scale fusion units and element addition units; Each of the fusion units is used to fuse two feature maps of different scales in the multi-scale feature map of the sample to be detected to obtain a fused sub-map; the multiples of the scale difference of the two feature maps of different scales corresponding to each of the fusion units are the same; The element adding unit is used to add the fusion sub-graphs output by all the fusion units to obtain the fusion feature graph of the sample to be detected.

7. A defect detection method based on differentiable neural network architecture search according to claim 3, wherein the multi-channel neural network structure search space includes four serially connected channels; in, The node inputs in the second and third cascade channels are: X i，j =b down ×f down (X i-1，j-1 )+b keep ×f keep (X i，j-1 )+b up ×f up (X i+1，j-1 ); Where i represents the serial number of the serial channel, i=2 or 3; j represents the node number; X i，j represents the input of the jth node in the i-th series channel; X i，j-1 represents the input of the j-1th node in the i-th series channel; X i-1，j-1 represents the input of the j-1th node in the i-1th series channel; X i+1，j-1 represents the input of the j-1th node in the i+1th serial channel; f keep (X i，j-1 ) indicates that X i，j-1 The transformation of the scale-invariant connection; f up (X i+1，j-1 ) indicates that X i+1，j-1 The transformation of the upsampling connection; f dowm (X i-1，j-1 ) indicates that X i-1，j-1 The transformation of the downsampling connection; β keep Indicates the value obtained by calculating the Softmax of the parameters of the invariant scale connection; β down Indicates the value obtained by calculating the Softmax of the parameters of the downsampling connection; β up Indicates the value obtained by calculating the Softmax of the parameters of the upsampling connection.

8. According to the defect detection method based on differentiable neural network architecture search in claim 4, the output of the candidate change operation is: y m =∑ o∈O b o ×f o (x m ); in, x m represents a node in the inter-node search space; y m Represents x m The output after the candidate change operation; O represents the set of candidate change operations, O = {skip, conv, diated, deform, sep}, f skip Indicates a jump direct connection; f conv represents the first convolution; f diated represents the dilated convolution; f deform represents deformable convolution; f sep represents separable convolution; β skip Indicates the value obtained by calculating the Softmax of the parameters of the jump direct connection; β conv Indicates the value obtained by calculating the Softmax of the first convolution parameter; β diated Indicates the value obtained by calculating the Softmax of the parameters of the dilated convolution; β deform Indicates the value obtained by calculating the Softmax of the deformable convolution parameters; β sep Indicates the value obtained by calculating Softmax on the parameters of the separable convolution.

9. According to the defect detection method based on differentiable neural network architecture search according to claim 5, the calculation formula of the distance is: in, p represents the pixel point in the multi-scale feature map of the sample to be detected; q represents the pixel point corresponding to p in the multi-scale feature map of the normal background sample; dis(p, q) represents the distance between p and q; e p represents the semantic embedding vector corresponding to p; e q Represents the semantic embedding vector corresponding to q.

10. A defect detection system based on differentiable neural network architecture search, characterized in that: include: A sample acquisition module is used to acquire samples to be detected and normal background samples; A defect detection module, used for inputting the sample to be detected and the normal background sample into a defect detection model to obtain a defect detection result of the sample to be detected; Wherein, the defect detection model in the defect detection module includes a trained backbone network, a normal sample reference module, a multi-scale fusion module and a segmentation network connected in sequence; The trained backbone network is obtained by training the backbone network using a data set; the backbone network is obtained by searching using a differentiable neural network architecture; the trained backbone network is used to output a multi-scale feature map of the sample to be detected and a multi-scale feature map of the normal background sample; The normal sample reference module is used to calculate the distance between each pixel point in the multi-scale feature map of the sample to be detected and the optimal matching point, and output a multi-scale comparison reference map composed of the distances corresponding to all pixel points; the optimal matching point is the pixel point in the multi-scale feature map of the normal background sample corresponding to the pixel point in the multi-scale feature map of the sample to be detected; The multi-scale fusion module is used to fuse the multi-scale feature maps of the sample to be detected to obtain the fused feature map of the sample to be detected; The segmentation network is used to segment the fused feature map of the sample to be detected based on the multi-scale contrast reference map, and determine the segmentation result as the defect detection result of the sample to be detected.

Images