Industrial defect detection method and equipment based on deep learning chip, and medium

Abstract

The invention provides an industrial defect detection method and device based on a deep learning chip, and a medium. The method comprises the following steps: splicing features of different to-be-detected industrial images to obtain a target splicing feature; obtaining a first pixel group and a second pixel group from the target splicing feature through a sliding window and pooling; obtaining a first attention score and a second attention score according to the first pixel group and the second pixel group, and setting a first parameter matrix and a second parameter matrix; determining a spatial relative coordinate and a logarithmic interval continuous position deviation between the second parameter matrix and the query, and further determining an attention weight of the first pixel group, an attention weight of the second pixel group and an attention value; carrying out industrial defect detection according to the attention value; in combination with different types of image features extracted in different modes and corresponding attention features, high-precision detection of small or hidden defects can be realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of industrial inspection, and in particular, to industrial defect detection methods, equipment, and media based on deep learning chips. Background Art

[0002] Industrial defect detection is an important part of industrial production. There are currently three commonly used methods for industrial defect detection: one is to rely on manual inspections and inspections, where workers identify industrial defects with their naked eyes, but this requires a lot of manpower and is inefficient. The second is to use X-rays and radiographic imaging, which can provide detailed three-dimensional structural information inside objects through computer tomography, but tomography machines are expensive. A large number of 3D scans will be generated during the tomography process, which consumes a lot of storage space and time, and in the end, workers still need to identify them with their naked eyes. The third is to achieve this through deep learning algorithms, but the deep learning algorithms currently used for industrial defect detection still have a high detection error rate for smaller or hidden defects. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide an industrial defect detection method, equipment and medium based on deep learning chips, which can achieve high-precision detection of smaller or hidden defects.

[0005] An embodiment of the first aspect of the present application is an industrial defect detection method based on a deep learning chip, comprising:

[0006] Acquire the industrial image to be inspected;

[0007] The features of different industrial images to be detected are spliced ​​to obtain target splicing features;

[0008] Acquire a first pixel group from the target stitching feature by using a sliding window, and acquire a second pixel group from the target stitching feature by using pooling;

[0009] Obtain a first attention score according to the first pixel group, and obtain a second attention score according to the second pixel group;

[0010] Setting a first parameter matrix according to the first pixel group, and setting a second parameter matrix according to the second pixel group;

[0011] determining spatial relative coordinates between the second parameter matrix and the attention mechanism query, and determining logarithmically spaced continuous position deviations of the spatial relative coordinates;

[0012] Obtaining an attention weight of the first pixel group and an attention weight of the second pixel group according to a concatenation result of the first parameter matrix and the logarithmically spaced continuous position deviations of the spatial relative coordinates and a concatenation result of the first attention score and the second attention score;

[0013] Obtaining an attention value according to an attention weight of the first pixel group and an attention weight of the second pixel group;

[0014] Masking is performed according to the attention value to obtain a mask feature;

[0015] Detection is performed according to the mask features to obtain industrial defect detection results.

[0016] According to certain embodiments of the first aspect of the present application, the step of stitching together features of different to-be-detected industrial images to obtain target stitching features includes:

[0017] Extracting first-type features and second-type features of the to-be-detected industrial image, where the first-type features and the second-type features are of different types;

[0018] Splicing first-type features of different to-be-detected industrial images to obtain first spliced ​​sub-features;

[0019] splicing the second type features of different to-be-detected industrial images to obtain second spliced ​​sub-features;

[0020] Performing adaptive instance normalization on the first splicing sub-feature and the second splicing sub-feature to obtain a third splicing sub-feature and a fourth splicing sub-feature;

[0021] Linearly combining the industrial image to be detected, the third splicing sub-feature and the fourth splicing sub-feature to obtain a plurality of linear combination features;

[0022] Multiple linear combination features are linearly interpolated according to the bounding box to obtain the target splicing feature.

[0023] According to certain embodiments of the first aspect of the present application, the target splicing feature is expressed by the following formula: m =T⊙g 11 +(MR c -R s +T)⊙g 22 +(R c -T)⊙g 12 +(R s -T)⊙g 21 ; In the formula, x m is the target splicing feature; g 11 , g 22 , g 12 and g 21is a different linear combination feature; ⊙ is principal component multiplication; R c is the preset parameter, R c ∈R W×H×C ; M is the initialization matrix with the same size as the linear combination feature; R s is the preset parameter, R s ∈R W×H×C And R s =r s M, r s is a scalar value that controls the strength of the second type of feature; T is a preset parameter, T∈R W×H×C And max(0,R c +R s -M)≤T≤min(R c ,R s ).

[0024] According to some embodiments of the first aspect of the present application, obtaining a first attention score according to the first pixel group and obtaining a second attention score according to the second pixel group includes:

[0025] Obtaining a first key according to the first pixel group;

[0026] Obtaining a second key according to the second pixel group;

[0027] A fixed query is obtained according to the color image corresponding to the target stitching feature, a learning query is obtained according to the grayscale image corresponding to the target stitching feature, and a total query is obtained according to the fixed query and the learning query;

[0028] Obtaining the first attention score according to the first key and the total query;

[0029] The second attention score is obtained according to the second key and the total query.

[0030] According to certain embodiments of the first aspect of the present application, the attention weight of the first pixel group and the attention weight of the second pixel group are both related to the number of valid keys interacting with the query of the attention mechanism.

[0031] According to certain embodiments of the first aspect of the present application, the number of valid keys for query interaction with the attention mechanism is expressed by the following formula: N (i，j) =||ρ (i，j) ||+||σ (X) ||-||μ (i，j) ||; where N (i，j) is the number of valid keys interacting with the query of the attention mechanism, ρ (i，j) is the first pixel group, σ (X) is the second pixel group, μ (i，j)It is the mask segmentation corresponding to the pixel in the i-th row and j-th column of the industrial image to be detected.

[0032] According to certain embodiments of the first aspect of the present application, the attention value is expressed by the following formula: In the formula, X is the industrial image to be detected, X (i,j) is the pixel in the i-th row and j-th column of the industrial image to be detected, A(X (i,j) ) is the attention value corresponding to the pixel in the i-th row and j-th column of the industrial image to be detected, ρ(i,j) is the first pixel group, A (i,j)～ρ(i,j) is the attention weight of the first pixel group, is the attention score obtained by multiplying the learning key and the total query, V ρ(i,j) is the value obtained from the target splicing feature through the sliding window, σ(X) is the second pixel group, A (i,j)～ρ(X) is the attention weight of the second pixel group, V σ(X) is the value obtained from the target concatenated feature by pooling, and T is a matrix used to adjust the dynamic position deviation of the total query.

[0033] According to certain embodiments of the first aspect of the present application, the step of performing masking according to the attention value to obtain a mask feature includes:

[0034] The attention value is processed by linear projection to be divided into a first sub-attention value and a second sub-attention value;

[0035] The first sub-attention value is sequentially subjected to convolution processing and activation processing to generate gating information;

[0036] Multiplying the gating information and the second sub-attention value element by element to retain a portion of the elements in the second sub-attention value and suppress another portion of the elements in the sub-attention value to obtain a multiplication result;

[0037] The multiplication result is added to the attention value to obtain a mask feature.

[0038] An embodiment of the second aspect of the present application is an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the industrial defect detection method based on deep learning chip described in the embodiment of the first aspect of the present application is implemented.

[0039] An embodiment of the third aspect of the present application is a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the industrial defect detection method based on deep learning chip described in the embodiment of the first aspect of the present application is implemented.

[0040] The above scheme has at least the following beneficial effects: obtaining a target stitching feature by stitching the features of different industrial images to be detected; obtaining a first pixel group from the target stitching feature through a sliding window, and obtaining a second pixel group from the target stitching feature through pooling; obtaining a first attention score according to the first pixel group, and obtaining a second attention score according to the second pixel group; setting a first parameter matrix according to the first pixel group, and setting a second parameter matrix according to the second pixel group; determining the spatial relative coordinates between the second parameter matrix and the query of the attention mechanism, and determining the logarithmically spaced continuous position deviation of the spatial relative coordinates; obtaining the attention weight of the first pixel group and the attention weight of the second pixel group according to the stitching result of the first parameter matrix and the logarithmically spaced continuous position deviation of the spatial relative coordinates and the stitching result of the first attention score and the second attention score; obtaining an attention value according to the attention weight of the first pixel group and the attention weight of the second pixel group; masking according to the attention value to obtain a mask feature; performing detection according to the mask feature to obtain an industrial defect detection result; combining different types of image features extracted by different methods and the corresponding attention features, high-precision detection of smaller or hidden defects can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0042] Figure 1 It is a step diagram of an industrial defect detection method based on a deep learning chip provided in an embodiment of the present application;

[0043] Figure 2 It is a sub-step diagram of step S200 provided in an embodiment of the present application;

[0044] Figure 3 It is a sub-step diagram of step S400 provided in an embodiment of the present application;

[0045] Figure 4 is a schematic diagram of determining a first splicing sub-feature, a second splicing sub-feature, a third splicing sub-feature, and a fourth splicing sub-feature provided in an embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of a feature map for determining attention values ​​from a color image and a grayscale image provided by an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of step S700 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0050] The embodiments of the present application provide an industrial defect detection method, device and medium based on a deep learning chip, which can achieve high-precision detection of smaller or hidden defects.

[0051] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0052] Reference Figure 1 , an industrial defect detection method based on deep learning chip, including but not limited to the following steps:

[0053] Step S100, obtaining an industrial image to be detected;

[0054] Step S200, stitching the features of different industrial images to be detected to obtain target stitching features;

[0055] Step S300, obtaining a first pixel group from a target stitching feature by using a sliding window, and obtaining a second pixel group from the target stitching feature by using pooling;

[0056] Step S400, obtaining a first attention score according to the first pixel group, and obtaining a second attention score according to the second pixel group;

[0057] Step S500, setting a first parameter matrix according to the first pixel group, and setting a second parameter matrix according to the second pixel group;

[0058] Step S600, determining the spatial relative coordinates between the second parameter matrix and the query of the attention mechanism, and determining the logarithmically spaced continuous position deviations of the spatial relative coordinates;

[0059] Step S700, obtaining an attention weight of the first pixel group and an attention weight of the second pixel group according to a concatenation result of the first parameter matrix and the logarithmically spaced continuous position deviations of the spatial relative coordinates and a concatenation result of the first attention score and the second attention score;

[0060] Step S800, obtaining an attention value according to the attention weight of the first pixel group and the attention weight of the second pixel group;

[0061] Step S900, masking is performed according to the attention value to obtain a mask feature;

[0062] Step S1000, performing detection according to the mask features to obtain industrial defect detection results.

[0063] This industrial defect detection method is suitable for the following systems: using the Hongmeng operating system and the open source deep learning framework cambricon Pytorch and using domestic Haiguang x86 and Cambricon MLU370-S4 / S8 hardware to achieve deep learning training and reasoning.

[0064] The system is built as follows: install the Cambricon MLU370-S4 / S8 graphics card and Hygon x86CPU, use the docker image with pre-installed software stack provided by Cambricon to create a docker container, install cambricon Pytorch, CNToolkit, MagicMind and MagicMind Python in this environment to provide a deep learning environment for subsequent algorithm training, and finally transfer the operating environment to the Hongmeng system for global operations.

[0065] For step S100, multiple industrial images to be detected are obtained by shooting with an industrial camera, and the multiple industrial images to be detected are formed into a data set. During the training phase, labeling software such as labelme is used to label the data set, and the labeled data set is divided into a training set and a test set. For example, the data volume ratio of the training set and the test set can be 3:1.

[0066] Reference Figure 2 For step S200, the features of different industrial images to be detected are spliced ​​to obtain target splicing features, including the following steps:

[0067] Step S210, extracting first type features and second type features of the industrial image to be detected;

[0068] Step S220, splicing the first type features of different to-be-detected industrial images to obtain a first spliced ​​sub-feature;

[0069] Step S230, splicing the second type features of different industrial images to be detected to obtain a second spliced ​​sub-feature;

[0070] Step S240, performing adaptive instance normalization on the first splicing sub-feature and the second splicing sub-feature to obtain a third splicing sub-feature and a fourth splicing sub-feature;

[0071] Step S250, linearly combining the industrial image to be detected, the third splicing sub-feature and the fourth splicing sub-feature to obtain a plurality of linear combination features;

[0072] Step S260, linearly interpolating multiple linear combination features according to the bounding box to obtain a target splicing feature.

[0073] Reference Figure 4 For step S210, the industrial image to be detected is passed through an encoder f to extract the content and style of the industrial image to be detected to form a first type of feature and a second type of feature. The first type of feature and the second type of feature are different types of features. For example, the first type of feature is a content feature, and the second type of feature is a style feature.

[0074] The encoder f is the starting layer to the relu4_1 layer of the VGG-19 model. Its main function is to extract representative content features and style features from the input feature map.

[0075] For step S220, the first type features of different industrial images to be detected are spliced ​​to obtain a first spliced ​​sub-feature. For example, the first type features x1 and x2 of two industrial images to be detected are spliced ​​to obtain a first spliced ​​sub-feature f 11 .

[0076] For step S230, the second type features of different industrial images to be detected are spliced ​​to obtain a second spliced ​​sub-feature. For example, the second type features x1 and x2 of two industrial images to be detected are spliced ​​to obtain a second spliced ​​sub-feature f 22 .

[0077] For step S240, the first splicing sub-feature and the second splicing sub-feature are adaptively normalized to obtain a third splicing sub-feature and a fourth splicing sub-feature. 11 and the second concatenated sub-feature f 22 Perform adaptive instance normalization to obtain the third concatenated sub-feature f 12 and the fourth splicing sub-feature f 21 The role of adaptive instance normalization is to align the mean and variance of the combined content features with the mean and variance of the style features.

[0078] For step S250, the industrial image to be detected, the third splicing sub-feature and the fourth splicing sub-feature are linearly combined to obtain a plurality of linear combination features. 12 and the fourth splicing sub-feature f 21Decode the text through the decoder g to generate new text, and then make adaptive adjustments according to the selected style. The adjusted result is linearly combined with the industrial image to be detected to obtain the linear combination feature g 12 and g 21 , can be expressed by the formula: g 12 =γx1+(1-γ)g(f 12 ), g 21 =γx2+(1-γ)g(f 21 ).

[0079] Linear interpolation can solve the problem that the feature image is much smaller than the image frame size used to piece together the images x1 and x2. 12 The style has the content of x1 and has a mixed style of images x1 and x2, with a ratio of (γ,1-γ). Therefore, the new parameter γ adjusts the degree of mixing of the style image and the content image in the linear combination feature image; in addition, since the two input images are two random training images, there is a possibility that the style difference is too large. By setting a higher γ value to suppress the degree of mixing between images, the poor performance of the mixed image can be prevented.

[0080] For step S260, linear interpolation is performed on multiple linear combination features according to the bounding box to obtain a target splicing feature.

[0081] Define the bounding box, the coordinates of the bounding box are B = (r x ,r y ,r w ,r h ), sample the bounding box, crop the region from image x1 and paste it into image x2. 11 , g 22 , g 12 , g 21 Create a mixed image x of the target stitching features m , can be expressed by the formula: m =T⊙g 11 +(MR c -R s +T)⊙g 22 +(R c -T)⊙g 12 +(R s -T)⊙g 21 In the formula, x m is the target splicing feature; g 11 , g 22 , g 12 and g 21 is a different linear combination feature; ⊙ is principal component multiplication; R c is the preset parameter, R c ∈RW×H×C ; M is the initialization matrix with the same size as the linear combination feature; R s is the preset parameter, R s ∈R W×H×C And R s =r s M, r s is a scalar value that controls the strength of the second type of feature; T is a preset parameter, T∈R W×H×C And max(0,R c +R s -M)≤T≤min(R c ,R s ).

[0082] From x m As can be seen from the formula, the content of the bounding box is composed of the content of x1 inside the box and the content of x2 outside the box. The style of the box is 1-(1-r s )(1-γ) and (1-r s )(1-γ), and outside the box is r s (1-γ) and 1-r s (1-γ).

[0083] Assuming the area of ​​the bounding box is λ, the content ratio of x1 and x2 in the mixed image is λ and 1-λ, and the style ratio is γλ+(1-γ)r s and 1-(γλ+(1-γ)r s ). Based on this, the label y m By content tag y c and style tag y s Definition, y c and s It is a linear combination of label y1 and label y2. The loss function is the mean square error loss L s for: Where n is the total number of data sets, and y is the label mask map.

[0084] For step S300, a first pixel group is obtained from the target stitching feature through a sliding window, and a second pixel group is obtained from the target stitching feature through pooling.

[0085] Exemplarily, a set of pixels centered at the pixel at (i, j) in the sliding window is defined as ρ(i, j). The set of pixels obtained by pooling the feature map is defined as σ(X).

[0086] Reference Figure 3 For step S400, obtaining a first attention score according to the first pixel group and obtaining a second attention score according to the second pixel group includes the following steps:

[0087] Step S410, obtaining a first key according to a first pixel group;

[0088] Step S420, obtaining a second key according to the second pixel group;

[0089] Step S430, obtaining a fixed query according to the color image corresponding to the target splicing feature, obtaining a learning query according to the grayscale image corresponding to the target splicing feature, and obtaining a total query according to the fixed query and the learning query;

[0090] Step S440, obtaining a first attention score according to the first key and the total query;

[0091] Step S450, obtaining a second attention score according to the second key and the total query.

[0092] Reference Figure 5 For example, a fixed query is obtained based on a color image corresponding to the target splicing feature, such as an RGB image, and a learning query is generated with the pixel of the grayscale image corresponding to the target splicing feature as the center. The fixed query and the learning query are added to obtain the total query Q (i，j) Total queries Q (i，j) Features and location information of color images and grayscale images corresponding to target splicing features.

[0093] The key of the target splicing feature, i.e., the first key, is obtained based on the first pixel group through a sliding window of size k×k. The smallest form of a 3x3 sliding window is usually used to capture features near the visual focus and expand the receptive field of the model, then ||ρ(i, j)||=k 2 It can be understood that the value of the target splicing feature can also be obtained according to the first pixel group through the sliding window.

[0094] Learn the first key and multiply the learned first key by the total query Q (i，j) , get the first attention score S (i,j)～ρ(i,j) Then, the padding mask is used to fill the pixels at the edge of the obtained feature map to prevent the pixels at the edge of the feature map from affecting the Softmax operation due to the similarity between the zero-filled calculations outside the boundary.

[0095] Parameter-free adaptive average pooling is used to downsample in the spatial dimension, where the downsampling operator is: LayerNorm(AvgPool(GELU(Linear(X)))). A single-layer neural network is used for projection and activation before feature map pooling to prevent the average pooling operator from losing information, compress and extract useful information in advance, and thus improve the information compression rate after downsampling. Given a pooling size of H p ×W p , then ||σ(X)||=H p W pAfter pooling, the output is normalized using layer normalization to ensure the variance consistency of X and σ(X). In this way, the key of the target splicing feature, i.e., the second key, is obtained. It can be understood that the value of the target splicing feature can also be obtained according to the second pixel group through pooling.

[0096] The second key is learned, and the learned second key is multiplied by the total query Q (i，j) Get the second attention score S (i,j)～σ(X) .

[0097] For step S500, a first parameter matrix is ​​set according to the first pixel group, and a second parameter matrix is ​​set according to the second pixel group.

[0098] A learnable first parameter matrix B is set according to the characteristics of the first pixel group (i,j)～ρ(i,j) , the relative position deviation of the sliding window can be generalized.

[0099] The second pixel group is input into a multilayer perceptron with two hidden layers to obtain the second parameter matrix B (i，j)～σ(X) , where the activation function of the hidden layer is ReLU, and the multi-layer perceptron helps the network learn complex data patterns and features.

[0100] For step S600, the spatial relative coordinates between the second parameter matrix and the query of the attention mechanism are determined, and the logarithmically spaced continuous position deviations of the spatial relative coordinates are determined.

[0101] Exemplarily, the second parameter matrix B is calculated (i，j)～σ(X) With the total query Q (i，j) The relative spatial coordinates Δ (i，j)～σ(X) , and then use the logarithmic interval continuous position deviation log-CPB = log(x i )-log(x i-1 ) for the space relative coordinate Δ (i，j)～σ(X) Calculate log-CPB(Δ (i，j)～σ(X) ), where x i Indicates the current position, x i-1 Indicates the previous position of the current position. The introduction of logarithmically spaced continuous position deviations plays a role in generalizing the relative position deviations of features.

[0102] For step S700, the attention weight of the first pixel group and the attention weight of the second pixel group are obtained according to the splicing result of the logarithmically spaced continuous position deviations of the first parameter matrix and the spatial relative coordinates and the splicing result of the first attention score and the second attention score.

[0103] Reference Figure 6 , the first parameter matrix B (i,j)～ρ(i,j)The logarithmically spaced continuous position deviations from the spatial relative coordinates log-CPB (Δ (i，j)～σ(x) ) to splice and score the first attention (i,j)～ρ(i,j) With the second attention score S (i，j)～σ(X) The splicing result of the first parameter matrix and the logarithmically spaced continuous position deviation of the spatial relative coordinates and the splicing result of the first attention score and the second attention score are added, and the addition result is adjusted according to the adjustable parameter, and then the Softmax normalization and division are performed to obtain the attention weight A of the first pixel group. (i,j)～ρ(i,j) and the attention weight A of the second pixel group (i,j)～σ(X) .

[0104] It should be noted that the attention weight of the first pixel group and the attention weight of the second pixel group are both related to the number of valid keys interacting with the query of the attention mechanism.

[0105] In order to better combine the learnable keys and total queries, scaled cosine attention is introduced, which introduces an additional learnable coefficient to multiply the cosine similarity results of the total query and the learnable key, so that the attention mechanism can effectively ignore unimportant tokens. As the length of the input sequence increases, the confidence of the attention output decreases. Therefore, the attention is designed to be entropy invariant to better generalize to unknown lengths. When the query and key are approximately of size The entropy of the scaled dot product of a vector of sequence length n is estimated as:

[0106] τlogN is an adjustable parameter, where N is the number of valid keys that interact with the query of the attention mechanism. N does not include the number of masked tokens. Specifically, when applied to the attention decoder, tokens masked by the mask should not be counted in N. Under pixel attention, N is calculated as N (i，j) =||ρ (i，j) ||+||σ (X) ||-||μ (i，j) ||, where μ (i，j) represents the set of padding mask tokens at position (i, j). τ is a learnable variable.

[0107] For step S800, a feature map of attention values ​​is obtained according to the attention weights of the first pixel group and the attention weights of the second pixel group.

[0108] The attention value is expressed as follows: In the formula, X is the industrial image to be detected, X (i,j) is the pixel in the i-th row and j-th column of the industrial image to be detected, A(X (i,j)) is the attention value corresponding to the pixel in the i-th row and j-th column of the industrial image to be detected, ρ(i,j) is the first pixel group, A (i,j)～ρ(i,j) is the attention weight of the first pixel group, is the attention score obtained by multiplying the learning key and the total query, V ρ(i,j) is the value obtained from the target splicing feature through the sliding window, σ(X) is the second pixel group, A (i,j)～σ(X) is the attention weight of the second pixel group, V σ(X) is the value obtained from the target concatenated feature by pooling, and T is the matrix used to adjust the dynamic position deviation of the total query.

[0109] Step S700 and step S800 may be implemented by an encoder and a decoder.

[0110] For step S900, masking is performed according to the attention value to obtain a mask feature.

[0111] In gated channel attention, the gate branch has more decision-making power than the value branch, and ultimately decides whether the corresponding output element is reset to zero. The gated channel attention includes the gated linear unit (GLU), which is a channel mixer. The minimum form of adding a 3×3 depth convolution before the activation function of the gated linear unit can help the model learn richer and more abstract representations and transform it into a gated channel attention mechanism based on the nearest neighbor feature. First, the input feature map X is divided into two parts X1 and X2 through a linear projection unit. X1 is convolved with a 3×3 depth and then sent to the sigmoid activation function to generate gating information, which determines whether each element in X2 is retained or suppressed. Then the result of element-by-element multiplication of the gated information and X2 is added to the input feature map X through a linear projection unit to obtain a mask map containing mask features. Each token in the convolutional gated linear unit has a unique gating signal based on its closest fine-grained feature, which solves the shortcoming of the global average pooling in other mechanisms that is too coarse-grained.

[0112] In step S1000, detection is performed according to the mask features to obtain industrial defect detection results.

[0113] In order to better distinguish the types of industrial defect detection, the segmentation head outputs the category and probability. Specifically, a layer of deep convolution is added before the segmentation head to extract and integrate feature information through deep convolution. The feature extraction map X containing the mask feature is integrated into k×m×m, where k is the number of industrial defect categories and m is the length and width of the feature map. The size of m×m enables each layer to maintain an explicit m×m object space layout without collapsing it into a vector representation lacking spatial dimensions, truly retaining the explicit spatial correspondence of each pixel. The extracted feature map is then sent to the segmentation head, which has two branches, one for predicting the category of industrial defects and the other for predicting the probability of the corresponding category. In order to connect the two paths and realize the end-to-end detection of the industrial defect detection model, the cross entropy loss is used. The cross entropy loss allows the network to generate a mask for each class without the need for competition between classes. A dedicated classification head is relied on to predict the class label used to select the output mask. Finally, the network parameters are optimized by minimizing the cross entropy loss. The loss function L e Defined as: Among them, N represents the total number of samples, C represents the total number of categories, and y i,j Indicates whether the true label of the i-th sample is category j, Indicates the model's prediction result on whether the i-th sample belongs to category j.

[0114] The images after inspection are displayed visually, so that the staff can further judge whether the inspection results are wrong based on the inspection results. The industrial defect inspection results obtained by the industrial defect inspection method using the images collected by the automatic optical inspection camera will be automatically updated through the UI interface of the Hongmeng system. Real-time display is achieved so that the staff can more accurately judge whether the inspection results of the defect are correct.

[0115] An embodiment of the present application provides an electronic device. The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the industrial defect detection method based on deep learning chip as described above when executing the computer program.

[0116] The electronic device may be any intelligent terminal including a computer.

[0117] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute related programs to implement the technical solutions provided in the embodiments of the present application.

[0118] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory, and the processor is called to execute the methods of the embodiments of this application.

[0119] The input / output interface is used to realize information input and output.

[0120] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0121] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0122] An embodiment of the present application provides a computer storage medium storing computer executable instructions for executing the industrial defect detection method based on deep learning chip as described above.

[0123] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. In the above description of this specification, the description of reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0124] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0125] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0129] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. An industrial defect detection method based on deep learning chip, characterized in that: include: Acquire the industrial image to be inspected; The features of different industrial images to be detected are spliced ​​to obtain target splicing features; Acquire a first pixel group from the target stitching feature by using a sliding window, and acquire a second pixel group from the target stitching feature by using pooling; Obtain a first attention score according to the first pixel group, and obtain a second attention score according to the second pixel group; Setting a first parameter matrix according to the first pixel group, and setting a second parameter matrix according to the second pixel group; determining spatial relative coordinates between the second parameter matrix and the attention mechanism query, and determining logarithmically spaced continuous position deviations of the spatial relative coordinates; Obtaining an attention weight of the first pixel group and an attention weight of the second pixel group according to a concatenation result of the first parameter matrix and the logarithmically spaced continuous position deviations of the spatial relative coordinates and a concatenation result of the first attention score and the second attention score; Obtaining an attention value according to an attention weight of the first pixel group and an attention weight of the second pixel group; Masking is performed according to the attention value to obtain a mask feature; Detection is performed according to the mask features to obtain industrial defect detection results.

2. The industrial defect detection method based on deep learning chip according to claim 1 is characterized in that: The step of splicing the features of different industrial images to be detected to obtain target splicing features includes: Extracting first-type features and second-type features of the to-be-detected industrial image, where the first-type features and the second-type features are of different types; Splicing first-type features of different to-be-detected industrial images to obtain first spliced ​​sub-features; Splicing the second type features of different to-be-detected industrial images to obtain second spliced ​​sub-features; Performing adaptive instance normalization on the first splicing sub-feature and the second splicing sub-feature to obtain a third splicing sub-feature and a fourth splicing sub-feature; Linearly combining the industrial image to be detected, the third splicing sub-feature and the fourth splicing sub-feature to obtain a plurality of linear combination features; Multiple linear combination features are linearly interpolated according to the bounding box to obtain the target splicing feature.

3. The industrial defect detection method based on deep learning chip according to claim 1 is characterized in that: The target splicing feature is expressed by the following formula: m =T⊙g 11 +(MR c -R s +T)⊙g 22 +(R c -T)⊙g 12 +(R s -T)⊙g 21 ; In the formula, x m is the target splicing feature; g 11 , g 22 , g 12 and g 21 are different linear combination features; ⊙ is principal element multiplication; R c is the preset parameter, R c ∈R W×H×C ; M is the initialization matrix with the same size as the linear combination feature; R s is the preset parameter, R s ∈R W×H×C And R s =r s M, r s is a scalar value that controls the strength of the second type of feature; T is a preset parameter, T∈R W×H×C And max(0,R c +R s -M)≤T≤min(R c ,R s ).

4. The industrial defect detection method based on deep learning chip according to claim 1 is characterized in that: The obtaining a first attention score according to the first pixel group and obtaining a second attention score according to the second pixel group includes: Obtaining a first key according to the first pixel group; Obtaining a second key according to the second pixel group; A fixed query is obtained according to the color image corresponding to the target stitching feature, a learning query is obtained according to the grayscale image corresponding to the target stitching feature, and a total query is obtained according to the fixed query and the learning query; Obtaining the first attention score according to the first key and the total query; The second attention score is obtained according to the second key and the total query.

5. The industrial defect detection method based on deep learning chip according to claim 1 is characterized in that: The attention weight of the first pixel group and the attention weight of the second pixel group are both related to the number of valid keys interacting with the query of the attention mechanism.

6. The industrial defect detection method based on deep learning chip according to claim 5 is characterized in that: The number of valid keys interacting with the query of the attention mechanism is expressed as follows: N (i，j) =||ρ (i，j) ||+||σ (X) ||-||μ (i，j) ||; where N (i，j) is the number of valid keys interacting with the query of the attention mechanism, ρ (i，j) is the first pixel group, σ (X) is the second pixel group, μ (i，j) It is the mask segmentation corresponding to the pixel in the i-th row and j-th column of the industrial image to be detected.

7. The industrial defect detection method based on deep learning chip according to claim 4 is characterized in that: The attention value is expressed by the following formula: In the formula, X is the industrial image to be detected, X (i,j) is the pixel in the i-th row and j-th column of the industrial image to be detected, A(X (i,j) ) is the attention value corresponding to the pixel in the i-th row and j-th column of the industrial image to be detected, ρ(i,j) is the first pixel group, A (i,j)～ρ(i,j) is the attention weight of the first pixel group, is the attention score obtained by multiplying the learning key and the total query, V ρ(i,j) is the value obtained from the target splicing feature through the sliding window, σ(X) is the second pixel group, A (i,j)～σ(X) is the attention weight of the second pixel group, V σ(X) is the value obtained from the target concatenated feature by pooling, and T is a matrix used to adjust the dynamic position deviation of the total query.

8. The industrial defect detection method based on deep learning chip according to claim 1 is characterized in that: The step of performing masking according to the attention value to obtain a mask feature includes: The attention value is processed by linear projection to be divided into a first sub-attention value and a second sub-attention value; The first sub-attention value is sequentially subjected to convolution processing and activation processing to generate gating information; Multiplying the gating information and the second sub-attention value element by element to retain a portion of the elements in the second sub-attention value and suppress another portion of the elements in the sub-attention value to obtain a multiplication result; The multiplication result is added to the attention value to obtain a mask feature.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the industrial defect detection method based on deep learning chip as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the industrial defect detection method based on a deep learning chip according to any one of claims 1 to 8 is implemented.

Images