A defect detection method, device, apparatus and storage medium

Abstract

The application provides a defect detection method, device and equipment and a storage medium. The method comprises: using an encoder network to extract features of a to-be-detected image to obtain a plurality of encoding feature vectors; splicing the plurality of encoding feature vectors and decoding the spliced vectors using a decoder network to obtain decoding feature vectors corresponding to the plurality of encoding feature vectors respectively; comparing each encoding feature vector with the decoding feature vector corresponding to the encoding feature vector to obtain a plurality of similarity matrices; determining a target similarity matrix based on the plurality of similarity matrices; and determining that the to-be-detected image has a defect if there is an element less than a preset threshold in the target similarity matrix. Through the above method, the problem that a deep learning defect detection algorithm is affected due to the difficulty in obtaining a defect sample is avoided.

Description

Technical Field

[0001] This application relates to the fields of image processing and computer vision technology, and in particular to a defect detection method, apparatus, device and storage medium. Background Technology

[0002] Industrial defect detection covers a wide range of applications, including steel, wood, paper, fabrics, and LCD screens. Traditional image processing methods are no longer sufficient for increasingly complex objects requiring detection. Against this backdrop, deep learning technology is considered a breakthrough for overcoming current technical challenges. Extensive research and practical applications in industry and academia have demonstrated the crucial role of deep learning methods in defect feature extraction; when sufficient defect samples are available, defect detection is no longer a major industrial problem. However, with advancements in production processes, the number of defective products decreases, and the potential types and forms of defects become difficult to predict and enumerate, leading to a shortage of defect samples. Therefore, traditional deep learning defect detection algorithms, which rely on a large number of defect samples, are also encountering technical bottlenecks. Summary of the Invention

[0003] This application provides a defect detection method, apparatus, device, and storage medium, which avoids the problem of the deep learning defect detection algorithm being affected by the difficulty in obtaining defect samples.

[0004] In a first aspect, embodiments of this application provide a defect detection method, the method comprising:

[0005] An encoder network is used to extract features from the image to be detected, resulting in multiple encoded feature vectors;

[0006] The multiple encoded feature vectors are concatenated, and the concatenated vector is decoded using a decoder network to obtain the decoded feature vectors corresponding to the multiple encoded feature vectors respectively.

[0007] By comparing each encoded feature vector with the corresponding decoded feature vector, multiple similarity matrices are obtained.

[0008] The target similarity matrix is ​​determined based on the multiple similarity matrices;

[0009] If there are elements in the target similarity matrix that are less than a preset threshold, then the image to be detected is determined to have a defect.

[0010] The decoder network is obtained by concatenating multiple sample encoded feature vectors and training them multiple times on a defect-free sample set based on the similarity between each sample encoded feature vector and the sample decoded feature vector corresponding to each sample encoded feature vector. The sample decoded vector is obtained by decoding the vector concatenated with the multiple sample encoded feature vectors. The multiple sample encoded feature vectors are obtained by feature extraction of the sample image by the encoder network.

[0011] In the above embodiment, an encoder network-decoder network structure is adopted. Defective samples are detected by the similarity between the output of the encoder network and the output of the decoder network. Since the network is trained only with samples without defects, when a defective image is input into the network, the similarity of its output will be low. Samples without defects are easy to obtain, and the network training is simple and convenient.

[0012] In one possible implementation, the decoder network is trained as follows:

[0013] The similarity loss value corresponding to each element coordinate is determined based on each sample encoding feature vector and the sample decoding feature vector corresponding to each sample encoding feature vector. The element coordinates of the elements in the sample encoding feature vector correspond one-to-one with the element coordinates of the elements in the sample decoding feature vector corresponding to the sample encoding feature vector.

[0014] The target loss function is determined by using the values ​​of various similarity loss functions. The target loss function is used to perform multiple rounds of parameter adjustment for each pair of decoder networks. In each round of parameter adjustment, the parameters after the previous round of parameter adjustment are used as the initial values. The error between the similarity predicted by the decoder network and the actual similarity is determined according to the target loss function, and the error is backpropagated through gradient to update the parameters in the decoder network. After the current round of parameter adjustment is completed, the adjusted parameters are used as the initial values ​​for the next round of parameter adjustment.

[0015] In the above embodiments, a target loss function is constructed using the sample encoding feature vector and the corresponding sample decoding feature vector. The parameters of the decoder network are then adjusted using the loss function through backpropagation, making the training efficiency of the network more efficient.

[0016] In one possible implementation, determining the similarity loss value corresponding to each element coordinate based on each sample encoded feature vector and the sample decoded feature vector corresponding to each sample encoded feature vector includes:

[0017] For each first element in each sample encoded feature vector and the second element in the corresponding sample decoded feature vector that has the same coordinates as the first element, perform the following operation:

[0018] Determine the ratio of the product of the first and second elements to the product of the L2 norm of the first and second elements;

[0019] The difference between the preset value and the ratio is used as the similarity loss value of the sample encoded feature vector and the sample decoded feature vector at the element coordinates.

[0020] In one possible implementation, determining the target loss function using the values ​​of each similarity loss function includes:

[0021] Formula As the target loss function, where L KD Let H be the target loss function, k represent the encoded feature vector or the decoded feature vector of the k-th sample, N be the number of encoded feature vectors or the number of decoded feature vectors, and H be the target loss function. k Let h be the row number in the encoded feature vector or decoded feature vector of the k-th sample, and let h represent the h-th row in the encoded feature vector or decoded feature vector of the k-th sample. k Let M be the column number in the encoded feature vector or decoded feature vector of the k-th sample, where w represents the w-th column in either the encoded or decoded feature vector of the k-th sample. k (h,w) represents the similarity loss value between the encoded feature vector of the k-th sample and the decoded feature vector of the k-th sample at the element coordinates (h,w), where W is the similarity loss value. k (h,w) is M k The weight values ​​corresponding to (h,w) are such that the dimension of the k-th sample variable code feature vector and the k-th sample decoding feature vector are the same.

[0022] In the above embodiments, the target loss function constructed by using the similarity loss value of each sample's encoded feature vector or decoded feature vector at each element coordinate and the weight value corresponding to each similarity loss value can reflect the error between the predicted similarity and the actual similarity, so that the parameters of the decoder network can be adjusted according to the error.

[0023] In one possible implementation, before concatenating the plurality of encoded feature vectors, the method further includes:

[0024] The dimensions of each encoded feature vector are transformed according to a preset dimension.

[0025] In the above embodiment, since the multiple encoded feature vectors extracted by the encoder network represent features at different levels, their dimensions are also different. Therefore, before splicing, it is necessary to unify the dimensions of the multiple encoded feature vectors.

[0026] In one possible implementation, determining the target similarity matrix based on the plurality of similarity matrices includes:

[0027] Each similarity matrix is ​​weighted, and each weighted similarity matrix is ​​upsampled according to the dimension of the pixel matrix of the image to be detected, so that the dimension of each upsampled similarity matrix matches the dimension of the pixel matrix of the image to be detected.

[0028] The elements of each similarity matrix after upsampling are summed together, and the average value of each element's coordinates is taken to obtain the target similarity matrix.

[0029] In the above embodiments, if the image to be detected has defects, in order to facilitate the determination of the shape and location information of the defects, the dimension of the similarity matrix is ​​restored to the dimension of the pixel matrix of the image to be detected.

[0030] In one possible implementation, the plurality of similarity matrices are weighted in the following manner:

[0031] For each similarity matrix, each element in the similarity matrix is ​​multiplied by the weight value of the same element coordinate in the weight matrix to obtain a weighted similarity matrix. The weight matrix is ​​determined based on the similarity matrix obtained by comparing the sample encoding feature vector with the sample decoding feature vector corresponding to the sample encoding feature vector.

[0032] In the above embodiments, to facilitate subsequent determination of the shape and location information of defects, the similarity is in the form of a two-dimensional matrix, and therefore the weights corresponding to the similarity are also in the form of a two-dimensional matrix. Different weight values ​​correspond to the coordinates of each element, which can make the features of foreground objects in the image more prominent.

[0033] Secondly, embodiments of this application provide a defect detection device, the device comprising:

[0034] The feature extraction module is used to extract features from the image to be detected using the encoder network, and obtain multiple encoded feature vectors.

[0035] The decoding module is used to concatenate the multiple encoded feature vectors and use a decoder network to decode the concatenated vectors to obtain decoded feature vectors corresponding to the multiple encoded feature vectors respectively.

[0036] The comparison module is used to compare each encoded feature vector with the corresponding decoded feature vector to obtain multiple similarity matrices.

[0037] The determination module is used to determine a target similarity matrix based on the plurality of similarity matrices; if there are elements in the target similarity matrix that are less than a preset threshold, then it is determined that the image to be detected has a defect;

[0038] The decoder network is obtained by concatenating multiple sample encoded feature vectors and training them multiple times on a defect-free sample set based on the similarity between each sample encoded feature vector and the sample decoded feature vector corresponding to each sample encoded feature vector. The sample decoded vector is obtained by decoding the vector concatenated with the multiple sample encoded feature vectors. The multiple sample encoded feature vectors are obtained by feature extraction of the sample image by the encoder network.

[0039] Thirdly, embodiments of this application provide a defect detection device, the device comprising:

[0040] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect above.

[0041] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for causing a computer to perform the method described in the first aspect above. Attached Figure Description

[0042] Figure 1 A schematic diagram of cosine similarity calculation provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram illustrating an application scenario of a defect detection method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic flowchart of a defect detection method provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of a deep learning model structure provided in an embodiment of this application;

[0046] Figure 5 This application provides a schematic diagram of a model training process.

[0047] Figure 6This is a schematic diagram illustrating a specific process of a defect detection method provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of a defect detection device provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of a defect detection device provided in an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0051] First, some concepts involved in the embodiments of this application will be introduced:

[0052] 1. Attention Mechanism: This mimics the visual signal processing pattern in the human brain. Through a series of weight calculations, the trained learning model can automatically distinguish between key and non-key information in the input image. The traditional attention mechanism calculation process consists of three steps: first, calculating the similarity between the target object (Query, Q) and each object (Key, K) in the original image according to Formula 1, denoted as the score. i .

[0053] score i =F(Q,k) i Formula 1, where i represents the feature vector of the i-th thing.

[0054] Then use the softmax function to adjust the score. i After normalization calculation, the probability distribution of the Attention weights is obtained:

[0055] Where N represents the number of transaction keys, and score j This represents the influence weight value of the feature vector of the j-th item on the target item Query.

[0056] Calculate the vectorized representation containing important image information:

[0057] v i The key-value vector represents the object. Attention(K,V,Q) represents the expressiveness of the target representation (Query) under the influence of the object (Key) and its key-value (Value). By applying different weights to different values, the selection tendency of the value is realized.

[0058] 2. Cosine Similarity: Cosine similarity uses the cosine of the angle between two vectors in a vector space to measure the difference between the two individuals. The closer the cosine value is to 1, the closer the angle is to 0 degrees, meaning the two vectors are more similar. This is called "cosine similarity".

[0059] like Figure 1 As shown, taking a two-dimensional space as an example, vector A is (x1, y1) and vector B is (x2, y2). We need to calculate the angle θ between them. The cosine similarity can be obtained using formula 4:

[0060]

[0061] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0062] like Figure 2 The diagram illustrates an application scenario of a defect detection method provided in this application. The application scenario includes: a server 201, a database 202, and at least one image to be detected (images 203_1, 203_2, and 203_N are shown in the example). The server 201 supports the operation of the encoder network and decoder network, and the database 202 stores the data and programs required to execute the defect detection method.

[0063] In existing defect detection methods, the training process of deep learning models to obtain defect detection results for images requires a large number of defect samples, which are often difficult to obtain in actual production processes. Furthermore, using the difference features between two images can only compare the overall similarity between them; however, regional similarity and pixel-level similarity are difficult to determine. To address this problem, embodiments of this application provide a defect detection method, such as... Figure 3 As shown, the method includes:

[0064] S301: Use the encoder network to extract features from the image to be detected, and obtain multiple encoded feature vectors.

[0065] In this embodiment, the encoder network is a pre-trained neural network used to extract multiple features from the image to be detected. For example, the encoder network can be a residual network-50, or other networks can be used. No specific limitation is made here. The weights of the encoder network are not updated during training and remain fixed.

[0066] Among them, multiple encoded feature vectors represent features from shallow to deep (images to be detected at different resolutions). Shallow features contain more pixel information, such as the color, texture, edges, corners, position and details of the image; deep features contain abstract information, such as semantic information, and have poor ability to perceive details.

[0067] The encoded feature vector is in the form of a two-dimensional matrix, and therefore can also be called a feature map. In this embodiment, the number of shallow encoded feature vectors and the number of deep encoded feature vectors extracted by the encoder network are not specifically limited.

[0068] S302: Concatenate the multiple encoded feature vectors and use a decoder network to decode the concatenated vectors to obtain decoded feature vectors corresponding to the multiple encoded feature vectors respectively.

[0069] In one possible implementation, before concatenating the plurality of encoded feature vectors, the method further includes:

[0070] The dimensions of each encoded feature vector are transformed according to a preset dimension.

[0071] The preset dimension can be any dimension of one of the multiple encoded feature vectors, or it can be any other dimension; no specific limitation is made here.

[0072] Concatenating multiple encoded feature vectors allows for the complementary advantages of shallow and deep features, thus enabling better utilization of features with different characteristics. The concatenation of multiple encoded feature vectors can be achieved using a multi-scale feature fusion (MMF) module; no specific limitations are specified here.

[0073] After concatenating multiple encoded feature vectors, a convolutional kernel of a preset size (e.g., 1x1) can be used to reduce the number of channels, simplify the dimension of the encoded feature vectors, and remove redundant information. Before decoding the concatenated vectors using the decoder network, a bottleneck embedding module can be used to increase the nonlinear mapping capability of the encoder network, while reducing the complexity of the encoder network and the number of weights. The decoded feature vectors obtained by the decoder network have the same dimension as the encoded feature vectors. For example, if encoded feature vector E1 corresponds to decoded feature vector D1, then E1 and D1 have the same dimension; if encoded feature vector E2 corresponds to decoded feature vector D2, then E2 and D2 have the same dimension, while the dimensions of E1(D1) and E2(D2) can be different.

[0074] The structure of the entire network model can be as follows: Figure 4 As shown, the encoder network is a pre-trained neural network, which can use the Residual Neural Network-50 pre-trained model. E1, E2, and E3 are three encoded feature vectors extracted by the residual neural network. E1 to E3 correspond to the progression from shallow features to deep features, and these features serve as the fitting targets for the subsequent decoder network.

[0075] Bottleneck Network: The bottleneck network consists of two parts: a multi-scale feature fusion module (MFF) and a bottleneck embedding module (BE). The MMF module's main process is as follows: It applies one or more 3x3 convolutional operations with a stride of 2 to E1 and E2, making the feature dimensions of these three convolutional layers the same as the dimension of E3. These convolutions are then concatenated, and then passed through a 1x1 convolutional kernel to reduce the number of channels, simplify the convolutional feature dimensions, and remove redundant information. The bottleneck embedding module is similar to the bottleneck structure in residual neural networks. Its main function is to increase the network's ability to perform non-linear mappings while reducing the network's complexity and the number of network weights. Figure 4 The network structure shown is only one implementation method. This application does not specifically limit the type of each module, the parameters in each module (such as the size of the convolution kernel and the stride), or the number of encoded feature vectors.

[0076] In the embodiments of this application, the encoder network and decoder network have symmetrical structures and are opposite to the encoder network in the data flow direction. Compared with traditional knowledge distillation frameworks that use the same or similar structures, this effectively solves the problem of insufficient output variability on abnormal samples and has strong robustness to changes in illumination.

[0077] The decoder network is a vector obtained by concatenating multiple sample encoded feature vectors, and the similarity between each sample encoded feature vector and the corresponding sample decoded feature vector is obtained by training on a defect-free sample set for multiple rounds. The sample decoded vector is obtained by decoding the vector obtained by concatenating the multiple sample encoded feature vectors. The multiple sample encoded feature vectors are obtained by feature extraction of the sample image by the encoder network.

[0078] In one possible implementation, the decoder network is trained in the following manner, with the specific process as follows: Figure 5 As shown:

[0079] S501: Determine the similarity loss value corresponding to the coordinates of each element based on the encoded feature vector of each sample and the decoded feature vector of the sample corresponding to each encoded feature vector.

[0080] The element coordinates of the elements in the sample encoding feature vector correspond one-to-one with the element coordinates of the elements in the sample decoding feature vector corresponding to the sample encoding feature vector. That is, if the sample encoding feature vector is a 2*3 matrix, then the sample decoding feature vector corresponding to the sample encoding feature vector is also a 2*3 matrix.

[0081] Taking the encoded feature vectors E1, E2, and E3 as an example, the sample image is input into the encoder-decoder network structure. After steps S301-S302, the decoded feature vectors D1, D2, and D3 are obtained. The similarity loss value between the encoded feature vector of the k-th sample and the decoded feature vector of the k-th sample at the element coordinates (h, w) (i.e., the element in the h-th row and w-th column) is:

[0082] Let be the first eigenvalue of the k-th E eigenvector at element coordinates (h, w). Let be the second eigenvalue of the k-th D feature vector at element coordinates (h, w), where h represents the h-th row in the k-th sample encoded feature vector or the k-th sample decoded feature vector, w represents the w-th column in the k-th sample encoded feature vector or the k-th sample decoded feature vector, and a is a preset value, such as 1. Let L2 be the arithmetic square root of the square of the first eigenvalue, which is the L2 norm of the first eigenvalue. The arithmetic square root of the square of the second eigenvalue is represented, i.e., the L2 norm of the second eigenvalue. This application does not specifically limit the location of the coordinate origin in its embodiments.

[0083] S502: Determine the target loss function using the values ​​of each similarity loss function, and use the target loss function to perform multiple rounds of parameter adjustment for each pair of decoder networks.

[0084] In this round of parameter adjustment, the parameters after the previous round of parameter adjustment are used as the initial values. The error between the similarity predicted by the decoder network and the actual similarity is determined according to the target loss function, and the error is backpropagated through gradient to update the parameters in the decoder network. After this round of parameter adjustment is completed, the adjusted parameters are used as the initial values ​​for the next round of parameter adjustment.

[0085] Formula As the target loss function, gradient backpropagation is performed according to this formula to update the network weights, but the weights of the encoder network are not updated and remain fixed. Where L... KD Let H be the target loss function, k represent the encoded feature vector or the decoded feature vector of the k-th sample, N be the number of encoded feature vectors or the number of decoded feature vectors, and H be the target loss function. k Let h be the row number in the encoded feature vector or decoded feature vector of the k-th sample, and let h represent the h-th row in the encoded feature vector or decoded feature vector of the k-th sample. k Let M be the column number in the encoded feature vector or decoded feature vector of the k-th sample, where w represents the w-th column in either the encoded or decoded feature vector of the k-th sample. k (h,w) represents the similarity loss value between the encoded feature vector of the k-th sample and the decoded feature vector of the k-th sample at the element coordinates (h,w), where W is the similarity loss value. k (h,w) is M k The weight values ​​corresponding to (h,w) are such that the dimension of the k-th sample variable code feature vector and the k-th sample decoding feature vector are the same.

[0086] S303: Compare each encoded feature vector with the corresponding decoded feature vector to obtain multiple similarity matrices.

[0087] Taking the encoded feature vectors E1, E2, and E3, and the decoded feature vectors D1, D2, and D3 as an example, the similarity between E1 and D1, E2 and D2, and E2 and D2 are calculated respectively. The similarity can be calculated using the cosine similarity formula 4 mentioned above, or other methods can be used; no specific limitation is made here. Based on the above, three similarity matrices can be obtained.

[0088] S304: Determine the target similarity matrix based on the plurality of similarity matrices; if there are elements in the target similarity matrix that are less than a preset threshold, then it is determined that the image to be detected has a defect.

[0089] In one possible implementation, determining the target similarity matrix based on the plurality of similarity matrices includes:

[0090] Each similarity matrix is ​​weighted, and each weighted similarity matrix is ​​upsampled according to the dimension of the pixel matrix of the image to be detected, so that the dimension of each upsampled similarity matrix matches the dimension of the pixel matrix of the image to be detected.

[0091] For each similarity matrix, each element in the similarity matrix is ​​multiplied by the weight value of the same element coordinate in the weight matrix to obtain a weighted similarity matrix. The weight matrix is ​​determined based on the similarity matrix obtained by comparing the sample encoding feature vector with the sample decoding feature vector corresponding to the sample encoding feature vector.

[0092] The weight matrix of the attention mechanism is set to have the same dimension as the corresponding encoded or decoded feature vector. This attention mechanism primarily aims to make the network focus more on the features of foreground objects in the input image while ignoring background feature interference. Figure 4 As shown, the attention mechanism can be set between the encoder network and the decoder network. The weight matrix of the attention mechanism can be calculated according to Equations 1-3 above. That is, Equation 1 is used to obtain the similarity matrix between the sample encoded feature vector and the sample decoded feature vector corresponding to the sample encoded feature vector, then Equation 2 is used to normalize the similarity matrix, and finally Equation 3 is used to obtain the weight matrix.

[0093] For each similarity matrix, each element in the similarity matrix is ​​multiplied by the weight value of the same element coordinate in the weight matrix to obtain a weighted similarity matrix. The weight matrix is ​​determined based on the similarity matrix obtained by comparing the sample encoding feature vector with the sample decoding feature vector corresponding to the sample encoding feature vector.

[0094] The elements of each similarity matrix after upsampling are summed together, and the average value of each element's coordinates is taken to obtain the target similarity matrix.

[0095] For example, the similarity matrix A1 is... Corresponding weight matrix Similarity matrix A2 is Corresponding weight matrix Then, weighting each similarity matrix yields... and Then add the corresponding element coordinates of A11 and A21 together to get Then, the average value is taken to obtain the target similarity matrix.

[0096] The specific process of the defective method provided in the embodiments of this application is described below, such as... Figure 6 As shown:

[0097] S601: Input the image to be detected;

[0098] S602: Feature extraction is performed through the encoder network to obtain multiple encoded feature vectors;

[0099] S603: Perform feature fusion on multiple encoded feature vectors, compress the fused feature vectors, and extract redundant information;

[0100] S604: After passing through the decoder network, the feature vector after feature fusion and compression is symmetrically restored with the features extracted by the encoder network to obtain multiple decoded feature vectors, wherein the multiple decoded feature vectors correspond to the multiple encoded feature vectors respectively;

[0101] S605: Determine whether to train the decoder network. If yes, execute S606; otherwise, execute S607.

[0102] S606: Calculate the similarity matrix between multiple encoded feature vectors and their corresponding decoded feature vectors, and update the parameters using backpropagation of the gradient of the target loss function. Here, the decoded feature vector is the decoded feature vector corresponding to the encoded feature vector.

[0103] S607: Calculate the similarity matrix between multiple encoded feature vectors and their corresponding decoded feature vectors, and obtain the target similarity matrix (pixel map) based on the similarity matrix, wherein the decoded feature vector is the decoded feature vector corresponding to the encoded feature vector;

[0104] S608: Determine whether there are elements in the target similarity matrix that are less than a preset threshold. If they exist, execute S609; otherwise, execute S610.

[0105] S609: The image to be inspected has a defect;

[0106] S610: The image to be inspected does not have defects.

[0107] In actual production, defect samples are difficult to obtain, and the possible categories and forms of defects are difficult to predict and enumerate. Defect detection methods that rely on pre-obtained defect samples have many limitations in practical applications. However, the defect detection method provided in this application only requires a single category of defect-free samples to train the neural network. Defect-free samples are easy to obtain, and network training is simple and convenient. Secondly, an end-to-end single-framework neural network is used, which directly obtains results from a single input. The network structure is simple and the running speed is fast. Moreover, the use of a structurally symmetrical encoder network and decoder network effectively solves the problem of insufficient output variability on abnormal samples compared to traditional knowledge distillation frameworks that use the same or similar structures. It also has strong robustness to changes in illumination. In addition, by using a bottleneck network and adopting a multi-scale feature fusion approach, multi-level features (rich in shallow spatial details) and deep features (poor in spatial details) are fused together, which can better supply the decoder network for feature restoration. At the same time, the use of a bottleneck embedding module increases the nonlinearity of the encoder network while reducing the model parameters. The spatial attention mechanism added to the decoder network can make the network pay more attention to foreground objects in the input image and suppress false detections that may be caused by background differences.

[0108] Based on the same inventive concept, this application also provides a defect detection device 700, such as... Figure 7 As shown, the apparatus includes:

[0109] The feature extraction module 701 is used to extract features from the image to be detected using the encoder network to obtain multiple encoded feature vectors.

[0110] The decoding module 702 is used to concatenate the multiple encoded feature vectors and use a decoder network to decode the concatenated vectors to obtain decoded feature vectors corresponding to the multiple encoded feature vectors respectively.

[0111] The comparison module 703 is used to compare each encoded feature vector with the corresponding decoded feature vector to obtain multiple similarity matrices.

[0112] The determination module 704 is used to determine a target similarity matrix based on the plurality of similarity matrices; if there are elements in the target similarity matrix that are less than a preset threshold, then it is determined that the image to be detected has a defect;

[0113] The decoder network is obtained by concatenating multiple sample encoded feature vectors and training them multiple times on a defect-free sample set based on the similarity between each sample encoded feature vector and the sample decoded feature vector corresponding to each sample encoded feature vector. The sample decoded vector is obtained by decoding the vector concatenated with the multiple sample encoded feature vectors. The multiple sample encoded feature vectors are obtained by feature extraction of the sample image by the encoder network.

[0114] In one possible implementation, the device 700 further includes a training module for training the decoder network in the following manner:

[0115] The similarity loss value corresponding to each element coordinate is determined based on each sample encoding feature vector and the sample decoding feature vector corresponding to each sample encoding feature vector. The element coordinates of the elements in the sample encoding feature vector correspond one-to-one with the element coordinates of the elements in the sample decoding feature vector corresponding to the sample encoding feature vector.

[0116] The target loss function is determined by using the values ​​of various similarity loss functions. The target loss function is used to perform multiple rounds of parameter adjustment for each pair of decoder networks. In each round of parameter adjustment, the parameters after the previous round of parameter adjustment are used as the initial values. The error between the similarity predicted by the decoder network and the actual similarity is determined according to the target loss function, and the error is backpropagated through gradient to update the parameters in the decoder network. After the current round of parameter adjustment is completed, the adjusted parameters are used as the initial values ​​for the next round of parameter adjustment.

[0117] In one possible implementation, the training module is used to determine the similarity loss value corresponding to each element coordinate based on each sample encoded feature vector and the sample decoded feature vector corresponding to each sample encoded feature vector, including:

[0118] For each first element in each sample encoded feature vector and the second element in the corresponding sample decoded feature vector that has the same coordinates as the first element, perform the following operation:

[0119] Determine the ratio of the product of the first and second elements to the product of the L2 norm of the first and second elements;

[0120] The difference between the preset value and the ratio is used as the similarity loss value of the sample encoded feature vector and the sample decoded feature vector at the element coordinates.

[0121] In one possible implementation, the training module is used to determine the target loss function using the values ​​of each similarity loss function, including:

[0122] Formula As the target loss function, where L KD Let H be the target loss function, k represent the encoded feature vector or the decoded feature vector of the k-th sample, N be the number of encoded feature vectors or the number of decoded feature vectors, and H be the target loss function. k Let h be the row number in the encoded feature vector or decoded feature vector of the k-th sample, and let h represent the h-th row in the encoded feature vector or decoded feature vector of the k-th sample. k Let M be the column number in the encoded feature vector or decoded feature vector of the k-th sample, where w represents the w-th column in either the encoded or decoded feature vector of the k-th sample. k (h,w) represents the similarity loss value between the encoded feature vector of the k-th sample and the decoded feature vector of the k-th sample at the element coordinates (h,w), where W is the similarity loss value. k (h,w) is M k The weight values ​​corresponding to (h,w) are such that the dimension of the k-th sample variable code feature vector and the k-th sample decoding feature vector are the same.

[0123] In one possible implementation, the device 700 further includes a dimension transformation module, which, before concatenating the plurality of encoded feature vectors, also includes:

[0124] The dimensions of each encoded feature vector are transformed according to a preset dimension.

[0125] In one possible implementation, the determining module 704 is used to determine a target similarity matrix based on the plurality of similarity matrices, including:

[0126] Each similarity matrix is ​​weighted, and each weighted similarity matrix is ​​upsampled according to the dimension of the pixel matrix of the image to be detected, so that the dimension of each upsampled similarity matrix matches the dimension of the pixel matrix of the image to be detected.

[0127] The elements of each similarity matrix after upsampling are summed together, and the average value of each element's coordinates is taken to obtain the target similarity matrix.

[0128] In one possible implementation, the determining module 704 is used to weight the plurality of similarity matrices in the following manner:

[0129] For each similarity matrix, each element in the similarity matrix is ​​multiplied by the weight value of the same element coordinate in the weight matrix to obtain a weighted similarity matrix. The weight matrix is ​​determined based on the similarity matrix obtained by comparing the sample encoding feature vector with the sample decoding feature vector corresponding to the sample encoding feature vector.

[0130] Based on the same inventive concept, this application also provides a defect detection device, the device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the defect detection methods in the above embodiments.

[0131] The following reference Figure 8 To describe an electronic device 130 according to this embodiment of the present application. Figure 8 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0132] like Figure 8 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0133] The processor 131 is used to read and execute instructions from the memory 132, so that the at least one processor can execute the defect detection method provided in the above embodiments.

[0134] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0135] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0136] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0137] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] In some possible implementations, various aspects of the defect detection method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps of a defect detection method according to various exemplary embodiments of this application as described above.

[0139] In addition, this application also provides a computer-readable storage medium storing a computer program for causing a computer to perform the method described in any of the above embodiments.

[0140] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0142] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0143] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A defect detection method characterized by, The method comprises: feature extraction on the to-be-detected image by using an encoder network to obtain a plurality of encoded feature vectors; splicing the plurality of encoded feature vectors and decoding the spliced vectors by using a decoder network to obtain decoded feature vectors corresponding to the plurality of encoded feature vectors respectively; comparing each encoded feature vector with the decoded feature vector corresponding to the encoded feature vector to obtain a plurality of similarity matrices; determining a target similarity matrix based on the plurality of similarity matrices; if there is an element less than a preset threshold in the target similarity matrix, it is determined that the to-be-detected image has a defect; wherein the decoder network is subjected to multiple rounds of parameter adjustment by using a target loss function, the target loss function is determined based on a plurality of similarity loss values, and for each first element in each sample encoded feature vector and a second element in a sample decoded feature vector corresponding to the sample encoded feature vector and having the same element coordinates as the first element, the following operations are performed: determining the ratio of the product of the first element and the second element to the product of the Euclidean L2 norm of the first element and the L2 norm of the second element; and taking the difference between a preset value and the ratio as the similarity loss value of the sample encoded feature vector and the sample decoded feature vector at the element coordinates, the sample decoded vector being obtained by decoding a vector spliced based on the plurality of sample encoded feature vectors, and the plurality of sample encoded feature vectors being obtained by feature extraction on sample images by using the encoder network.

2. The method of claim 1, wherein, The multiple rounds of parameter adjustment by using the target loss function comprise: in the current round of parameter adjustment, taking the parameters after the last round of parameter adjustment as initial values, determining the error between the predicted similarity by the decoder network and the actual similarity according to the target loss function, and performing gradient backpropagation on the error to update the parameters in the decoder network, and after the current round of parameter adjustment, taking the adjusted parameters as the initial values in the next round of parameter adjustment.

3. The method according to claim 1 or 2, characterized in that, Determining the target loss function by using the plurality of similarity loss values comprises: The formula is taken as the target loss function, where L KD is the target loss function, k represents the kth sample encoding feature vector or the kth sample decoding feature vector, N is the number of sample encoding feature vectors or the number of sample decoding feature vectors, H k is the number of rows in the kth sample encoding feature vector or the kth sample decoding feature vector, h represents the hth row in the kth sample encoding feature vector or the kth sample decoding feature vector, W k is the number of columns in the kth sample encoding feature vector or the kth sample decoding feature vector, w represents the wth column in the kth sample encoding feature vector or the kth sample decoding feature vector, is the similarity loss value of the kth sample encoding feature vector and the kth sample decoding feature vector at the element coordinates (h, w), is the M k (h, w) corresponding weight value, and the kth sample encoding feature vector and the kth sample decoding feature vector have the same dimension.

4. The method of claim 1, wherein, Before the splicing of the plurality of encoded feature vectors, the method further comprises: dimensionally converting the dimension of each encoded feature vector to a preset dimension.

5. The method of claim 1, wherein, Determining the target similarity matrix based on the plurality of similarity matrices comprises: weighting each similarity matrix, upsampling the weighted similarity matrix according to the dimension of the to-be-detected image pixel matrix to make the dimension of the upsampled similarity matrix match the dimension of the to-be-detected image pixel matrix; and adding the elements corresponding to each element coordinate of the upsampled similarity matrices and taking the average of the elements of each element coordinate to obtain the target similarity matrix.

6. The method of claim 5, wherein, The plurality of similarity matrices are weighted in the following manner: For each similarity matrix, each element in the similarity matrix is multiplied by a weight value at the same element coordinate of a weight matrix to obtain a weighted similarity matrix, the weight matrix being determined based on a sample decoding feature vector corresponding to a sample encoding feature vector and the similarity matrix.

7. A defect detection apparatus characterized by comprising: The apparatus comprises: a feature extraction module configured to extract features of the to-be-detected image using an encoder network to obtain a plurality of encoding feature vectors; a decoding module configured to concatenate the plurality of encoding feature vectors and decode the concatenated vectors using a decoder network to obtain a plurality of decoding feature vectors corresponding to the plurality of encoding feature vectors respectively; a comparison module configured to compare each encoding feature vector with a decoding feature vector corresponding to the encoding feature vector to obtain a plurality of similarity matrices; a determination module configured to determine a target similarity matrix based on the plurality of similarity matrices, and determine that the to-be-detected image has a defect if there is an element less than a preset threshold in the target similarity matrix. The decoder network performs multiple rounds of parameter adjustment using a target loss function, the target loss function being determined based on a plurality of similarity loss values, for each first element in each sample encoding feature vector and a second element in a sample decoding feature vector corresponding to the sample encoding feature vector at the same element coordinate as the first element, the following operations are performed: determining a ratio of a product of the first element and the second element to a product of a Euclidean L2 norm of the first element and an L2 norm of the second element, and taking a difference between a preset value and the ratio as a similarity loss value of the sample encoding feature vector and the sample decoding feature vector at the element coordinate, the sample decoding vector being obtained by decoding a vector obtained by concatenating the plurality of sample encoding feature vectors, the plurality of sample encoding feature vectors being obtained by extracting features of sample images using an encoder network.

8. A defect detection apparatus characterized by comprising: The apparatus comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program for causing a computer to perform the method of any one of claims 1-6.

Images