Transformer-based defect detection method and apparatus

Abstract

This invention provides a defect detection method and apparatus based on Transformer. The method includes: dividing a sample set into a training set and a test set, and further dividing the training set into a query set and a support set; determining the encoding vector corresponding to each sample image in the query set and the support set; constructing a defect detection network, including a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence; obtaining the query attention features of the query set and the support attention features of the support set; obtaining high-dimensional features; obtaining a predicted segmentation mask image; training based on the segmentation mask image and mask labels; and performing defect detection through the trained defect detection network. Therefore, it eliminates the need for large amounts of training data and extensive training sample annotation, enabling defect detection with a small number of samples. This effectively reduces the cost and time of sample data annotation, improves the efficiency of small-sample defect detection, and enhances the defect detection effect.

Description

Technical Field

[0001] This invention relates to the field of industrial quality inspection technology, specifically to a defect detection method and a defect detection device based on Transformer. Background Technology

[0002] In industrial manufacturing, any abnormality caused by multiple factors can lead to product defects. Therefore, defect detection is an important means to improve production quality and efficiency, and thus it is of great significance. Early product defect detection mainly relied on X-ray imaging followed by manual comparison of product defects with the naked eye. However, this method was time-consuming, labor-intensive, and had a high false detection rate. As machine vision methods have gradually matured, manual inspection has been gradually replaced by automated equipment integrating vision algorithms. However, early detection algorithms had low recognition accuracy and required manual feature extraction. In recent years, the emergence of deep convolutional neural networks has accelerated the development of computer vision. Due to their powerful feature extraction capabilities, they have been widely used in defect detection tasks.

[0003] Compared with traditional methods, convolutional neural networks have a higher recognition rate in defect detection. However, this method requires a large amount of data for training. Nowadays, industrial production processes are very sophisticated, and the yield rate of products is very high. Therefore, it is very difficult to obtain training data containing defects. Moreover, most of them are based on supervised models, which require a large number of training samples to be labeled. This process is very time-consuming and labor-intensive. This situation can also be called the small sample problem, which leads to low efficiency and poor effect of defect detection with a small number of samples. Summary of the Invention

[0004] To address the problem of low efficiency and poor performance in defect detection due to the need for large amounts of training data and high data annotation costs, this invention proposes the following technical solution.

[0005] A first aspect of this invention proposes a Transformer-based defect detection method, comprising the following steps: acquiring a sample set containing sample images and their corresponding mask labels, and dividing the sample set into a training set and a test set, and further dividing the training set into a query set and a support set; determining the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set; constructing a defect detection network, the defect detection network comprising a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence, wherein the cross-Transformer network is constructed based on a cross-Transformer module; inputting the query encoding vector and the support encoding vector into the cross-Transformer network to obtain query attention features of the query set and support attention features of the support set; inputting the query attention features and the support attention features into the feature enhancement network to obtain high-dimensional features; inputting the high-dimensional features into the decoder to obtain a predicted segmentation mask image, and training the defect detection network based on the segmentation mask image and the mask labels; acquiring a product image to be detected, and performing defect detection on the product image to be detected using the trained defect detection network.

[0006] In addition, the defect detection method based on Transformer according to the above embodiments of the present invention may also have the following additional technical features.

[0007] According to one embodiment of the present invention, determining the query encoding vector corresponding to each sample image of the query set and the support encoding vector corresponding to each sample image of the support set includes: determining the feature vector of each sample image of the query set and the support set; determining the position encoding information and branch encoding information corresponding to each sample image, and embedding the position encoding information and the branch encoding information into the feature vector of the corresponding sample image to obtain the query encoding vector corresponding to each sample image of the query set and the support encoding vector corresponding to each sample image of the support set.

[0008] According to an embodiment of the present invention, determining the feature vector of each sample image in the query set and the support set includes: setting the resolution of each sample image in the query set and the support set to a first preset value and then cropping it into multiple image blocks, wherein all image blocks of each sample image constitute the image block sequence of the sample image; and projecting the image block sequence of each sample image through a linear projection layer to obtain the feature vector of each sample image.

[0009] According to an embodiment of the present invention, the cross-Transformer network includes a cross-Transformer module and a feedforward network. The cross-Transformer module employs a multi-head cross-attention structure, inputting the query encoding vector and the support encoding vector into the cross-Transformer network to obtain query attention features of the query set and support attention features of the support set. This includes: inputting the query encoding vector and the support encoding vector into the cross-Transformer network; the cross-Transformer module mapping the query encoding vector and the support encoding vector to a query vector, a first key vector, and a first value vector, respectively; the cross-Transformer module reducing the size of the first key vector and the first value vector of the query set to obtain a second key vector and a second value vector of the query set, and reducing the size of the first key vector and the first value vector of the support set to obtain a third key vector and a third value vector of the support set; and the cross-Transformer module aggregating the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set.

[0010] According to one embodiment of the present invention, the cross-transformer module uses the SR module to reduce the size of the first key vector and the first value vector of the query set, and uses the SR module to reduce the size of the first key vector and the first value vector of the support set.

[0011] According to one embodiment of the present invention, the cross-Transformer module aggregates the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set, including: the cross-Transformer module performs average pooling on the third key vector to match the batch size of the second key vector to obtain the fourth key vector of the support set, and performs average pooling on the third value vector to match the batch size of the second value vector to obtain the fourth value vector of the support set; concatenates the second key vector and the fourth key vector to obtain the first concatenated key vector, and concatenates the second value vector and the fourth value vector to obtain the first concatenated value vector; and aggregates the query attention features of the query set and the support attention features of the support set. The query vector, the first concatenation key vector, and the first concatenation value vector are input into the feedforward network to obtain the query attention features of the query set. The cross-transformer module performs average pooling on the second key vector to match the batch size of the third key vector to obtain the fifth key vector of the query set, and performs average pooling on the second value vector to match the batch size of the third value vector to obtain the fifth value vector of the query set. The third key vector and the fifth key vector are concatenated to obtain the second concatenation key vector, and the third value vector and the fifth value vector are concatenated to obtain the second concatenation value vector. The query vector, the second concatenation key vector, and the second concatenation value vector of the support set are input into the feedforward network to obtain the support attention features of the support set.

[0012] According to one embodiment of the present invention, the query attention features and the support attention features are input into the feature enhancement network for aggregation processing to obtain high-dimensional features, including: the feature enhancement network using a cosine similarity function to calculate the relationship score between the query attention features and the support attention features; and the feature enhancement network obtaining high-dimensional features based on the relationship score, the query attention features, and the support attention features.

[0013] According to an embodiment of the present invention, the feature enhancement network obtains high-dimensional features based on the relation score, the query attention feature, and the support attention feature, including: the feature enhancement network multiplies the support attention feature with the relation score as a weight to obtain a weighted support attention feature, and fuses the weighted support attention feature with the query attention feature to obtain the high-dimensional feature.

[0014] According to one embodiment of the present invention, the feedforward network employs the LayerNorm function and the GELU activation function.

[0015] A second aspect of this invention provides a Transformer-based defect detection device, comprising: an acquisition module, configured to acquire a sample set containing sample images and their corresponding mask labels, and divide the sample set into a training set and a test set, further dividing the training set into a query set and a support set; a first determination module, configured to determine the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set; and a construction module, configured to construct a defect detection network, the defect detection network comprising a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence, wherein the cross-Transformer network is based on a cross-Transformer network. The network is constructed using the rmer module; the second determining module is used to input the query encoding vector and the support encoding vector into the cross-Transformer network to obtain the query attention features of the query set and the support attention features of the support set; the third determining module is used to input the query attention features and the support attention features into the feature enhancement network to obtain high-dimensional features; the fourth determining module is used to input the high-dimensional features into the decoder to obtain the predicted segmentation mask map, and to train the defect detection network based on the segmentation mask map and the mask label; the detection module is used to acquire the product image to be detected, and to perform defect detection on the product image to be detected through the trained defect detection network.

[0016] The technical solution of this invention does not require a large amount of training data or extensive training sample annotation, and can achieve defect detection with a small number of samples, effectively reducing the cost and time of sample data annotation, improving the efficiency of small sample defect detection, and enhancing the defect detection effect. Attached Figure Description

[0017] Figure 1 A schematic diagram of internal defects in a chip, taken by an X-ray inspection device.

[0018] Figure 2 This is a flowchart of a Transformer-based defect detection method according to an embodiment of the present invention.

[0019] Figure 3 for Figure 1 The corresponding segmentation mask image.

[0020] Figure 4 This is a schematic diagram illustrating the principle of obtaining a segmentation mask map for sample image prediction, as an example of the present invention.

[0021] Figure 5 This is a schematic diagram illustrating the principle of obtaining attention features as an example of the present invention.

[0022] Figure 6This is a schematic diagram illustrating the working principle of a cross-Transformer module as an example of the present invention.

[0023] Figure 7 This is a block diagram of a Transformer-based defect detection device according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Current defect detection research mainly focuses on strongly supervised tasks with a large number of samples, while there is relatively little research on supervised learning with a small number of samples. Since a small number of samples can effectively reduce the data processing cost in visual tasks, it has high research significance in industrial inspection, which can effectively characterize industrial productivity and reduce production costs.

[0026] In industrial settings, products such as chips are typically imaged using X-ray inspection equipment for defect detection. The choice of defect detection method is crucial. This invention employs a semantic segmentation paradigm to accomplish this task. Figure 1 As can be seen, the background changes within the chip are relatively small, and the feature differences between targets of the same category are large. For this type of problem, experiments have shown that the semantic segmentation paradigm is more suitable for this task.

[0027] Therefore, to ensure that defect detection can be achieved using a small amount of data, this invention proposes a defect detection method based on Transformer, which does not require a large amount of training data or extensive training sample annotation, and can also achieve defect detection with a small number of samples. This effectively reduces the cost and time of sample data annotation, improves the efficiency of small sample defect detection, and enhances the defect detection effect.

[0028] Figure 2 This is a flowchart of a Transformer-based defect detection method according to an embodiment of the present invention.

[0029] like Figure 2 As shown, the defect detection method based on Transformer includes the following steps S1 to S7.

[0030] S1. Obtain a sample set containing sample images and their corresponding mask labels, and divide the sample set into a training set and a test set. The training set is further divided into a query set and a support set.

[0031] Specifically, images of defective products can be acquired using X-ray inspection equipment to obtain sample images. These sample images are then labeled with defects to obtain corresponding mask labels. All sample images and their corresponding mask labels constitute a sample set. The sample set can be divided into a training set (for training the network) and a test set (for testing the network) at an 8:2 ratio. Furthermore, the training and test sets can be further divided into a support set and a query set at a 5:1 ratio.

[0032] It should be noted that when acquiring sample images, they can be acquired according to the defect type, such as acquiring sample images corresponding to six types of defects (missing solder joints, broken wires, twisted gold wires, gold wires too high, gold wires too low, and multiple lines).

[0033] S2, determine the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set.

[0034] It should be noted that in this embodiment of the invention, the encoding vector corresponding to each image in the query set is called the query encoding vector, and the encoding vector corresponding to each image in the support set is called the support encoding vector, in order to distinguish them.

[0035] Specifically, for each sample image (query image) in the query set, it can be encoded to obtain the encoded vector of that sample image, thus obtaining the query encoded vector. For each sample image (support image) in the support set, it can be encoded in the same way to obtain the encoded vector of that sample image, thus obtaining the support encoded vector.

[0036] This yields the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set, which can then be used for training the subsequent network model.

[0037] S3. Construct a defect detection network, which includes a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence. The cross-Transformer network is built based on the cross-Transformer module.

[0038] The cross-Transformer network serves several purposes. First, it can extract features, improving few-sample similarity learning between the two branches (query and support branches) and enhancing information interaction between samples. It can also be called a feature extraction network. Second, the feature enhancement network performs computation and fusion processing. Third, the decoder performs dimensionality reduction and upsampling. The cross-Transformer network consists of a cross-Transformer module (serving as the backbone of the defect detection network) and a feedforward network. The feedforward network includes two feedforward layers. The cross-Transformer module is residually connected to the feedforward network. The cross-Transformer module employs a cross-attention structure, meaning it contains cross-attention layers.

[0039] Specifically, a cross-Transformer network is first constructed based on the cross-Transformer module, and then a defect detection network is constructed. The cross-Transformer module adopts a cross-attention structure.

[0040] S4. Input the query encoding vector and support encoding vector into the cross-transformer network to obtain the query attention features of the query set and the support attention features of the support set.

[0041] It should be noted that in this embodiment of the invention, the attention features of the query set are referred to as query attention features and the attention features of the support set are referred to as support attention features for easy distinction.

[0042] Specifically, after obtaining the encoding vector of the sample image and constructing the defect detection network, the query encoding vector and the support encoding vector can be input into the cross-Transformer network one after another. Then, after a series of processing, the cross-Transformer network obtains the query attention features of the query set and the support attention features of the support set.

[0043] Each time an encoding vector is input, the query encoding vector and the support encoding vector can be input in a ratio of 1:k (k is greater than 1), where k is a hyperparameter that can be set according to actual needs. For example, it can be 5, which means inputting one query encoding vector and five support encoding vectors.

[0044] S5 strengthens the network by combining query attention features and support attention features with input features to obtain high-dimensional features.

[0045] Specifically, after obtaining the query attention features and support attention features, they are input into the feature enhancement network and processed through certain calculations and aggregations to obtain high-dimensional features.

[0046] S6. Input the high-dimensional features into the decoder to obtain the predicted segmentation mask map, and train the defect detection network based on the segmentation mask map and mask labels.

[0047] Specifically, after obtaining the high-dimensional features, they are input into the decoder for dimensionality reduction and upsampling to obtain the segmentation mask map (predicted) corresponding to each sample image predicted by the defect detection network. Since each sample image has a corresponding mask label (real), the segmentation loss is calculated, and the parameters of the defect detection network are optimized accordingly. The above steps are repeated until the iteration termination condition is reached, thus completing the training of the defect detection network.

[0048] for Figure 1 The sample image shown is obtained after the above steps as follows. Figure 3 The segmentation mask shown.

[0049] S7: Acquire the image of the product to be inspected, and perform defect detection on the image of the product to be inspected through the trained defect detection network.

[0050] Specifically, when it is necessary to perform defect detection on the product to be inspected, an image of the product to be inspected can be obtained by any feasible means, such as an X-ray source inspection device, and the image of the product to be inspected can be input into a trained defect detection network so that the trained defect detection network can output the defect detection result of the product to be inspected.

[0051] Therefore, the defect detection method based on Transformer in this invention does not require a large amount of training data or extensive training sample annotation, and can also achieve defect detection with a small number of samples, effectively reducing the cost and time of sample data annotation, improving the efficiency of small sample defect detection, and enhancing the defect detection effect.

[0052] In one embodiment of the present invention, step S2 may include: determining the feature vector of each sample image in the query set and the support set; determining the position encoding information and branch encoding information corresponding to each sample image, and embedding the position encoding information and branch encoding information into the feature vector of the corresponding sample image to obtain the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set.

[0053] Among them, the location encoding information represents the location information of the sample image, and the branch encoding information represents whether the branch in which the sample image is located is the query set or the support set.

[0054] Further, determining the feature vector of each sample image in the query set and support set may include: setting the resolution of each sample image in the query set and support set to a first preset value and then cropping it into multiple image blocks, wherein all image blocks of each sample image constitute the image block sequence of the sample image; and projecting the image block sequence of each sample image through a linear projection layer to obtain the feature vector of each sample image.

[0055] Specifically, such as Figure 4 As shown, for each sample image (query image) in the query set, a corresponding feature vector can be obtained after processing. This feature vector is then encoded using the positional encoding information and branch encoding information of the sample image to obtain the encoded vector of that sample image, thus yielding the query encoded vector. Similarly, for each sample image (support image) in the support set, a corresponding feature vector can be obtained after the same processing. This feature vector is then encoded using the positional encoding information and branch encoding information of the sample image to obtain the encoded vector of that sample image, thus yielding the support encoded vector.

[0056] Specifically, for each sample image in the query set and support set, its resolution and image size are first set to a preset value, such as 256×128, to ensure an approximate aspect ratio with the real image and reduce distortion. Then, each sample image is cropped into 4×4×1 image patches. This results in 256×128 / 4×4 = 2048 image patches per sample image. These 2048 patches constitute the image patch sequence of the sample image, with a length of 2048. Unfolding these image patches yields a 16-dimensional vector. Projecting this vector through a linear projection layer maintains its size. The linear projection layer has a dimension of 2048×N, so the dimension remains unchanged after projection. The image patch sequence, after being projected through the linear projection layer, yields the feature vector.

[0057] After converting an image into image patches, location information is missing. Therefore, additional location encoding information (location embedding) and branch encoding information (branch embedding) need to be added to preserve the location and branch information.

[0058] This involves determining the encoded position information and branch position information for each sample image. This encoded information can be viewed as a table with N rows, where N is the same as the length of the image patch sequence. Each row represents a vector with the same dimension as the image patch sequence, which is 2048. The encoded position information and encoded branch information of the sample image are then added to the feature vector of that sample image to obtain the corresponding encoded vector, which retains the position and branch information.

[0059] The positional encoding information and branch encoding information are added together, so the dimension of the input sequence is not changed. This can be expressed by the formula:

[0060]

[0061]

[0062] In the above formula, X q X s X represents the query encoding vector and the support encoding vector, respectively. q0 X and X represent the feature vectors of the sample images in the query set and support set, respectively. E represents the location encoding information of the sample images in the query set and support set, respectively. bra [0]、E bra [1] represents the branch encoding information of the query set and the support set respectively, and the encoding information is incorporated into the feature vector by addition.

[0063] Then, proceed with the following steps, i.e., refer to... Figure 4 The query encoding vector and support encoding vector are input into a cross-transformer network to obtain query attention features of the query set and support attention features of the support set. The query attention features and support attention features are then input into a feature enhancement network for aggregation processing to obtain high-dimensional features. The high-dimensional features are then input into a decoder to obtain the predicted segmentation mask.

[0064] In one embodiment of the present invention, the cross-Transformer network includes a cross-Transformer module and a feedforward network (two feedforward layers). The cross-Transformer module employs a multi-head cross-attention structure (such as a three-head cross-attention structure), and the feedforward network uses the LayerNorm function (to normalize the entire image) and the GELU (Gaussian Error Linear Unit) activation function. Using this cross-Transformer network can better facilitate information interaction between the two branches (the query set branch and the support set branch), enabling efficient acquisition of semantic features of the image.

[0065] The formula for the GELU activation function is:

[0066]

[0067]

[0068] The formula for the LayerNorm function is:

[0069]

[0070] Where x represents the input value of the GELU activation function, erf represents the Gaussian error function, γ and β are trainable parameters, E(x) represents the expectation, and Var represents the sample variance.

[0071] In this embodiment, step S4, which involves inputting the query encoding vector and support encoding vector into a cross-Transformer network to obtain the query attention features of the query set and the support attention features of the support set, may include: inputting the query encoding vector and support encoding vector into the cross-Transformer network; the cross-Transformer module mapping the query encoding vector and support encoding vector to a query vector, a first key vector, and a first value vector, respectively; the cross-Transformer module reducing the size of the first key vector and the first value vector of the query set to obtain the second key vector and the second value vector of the query set, and reducing the size of the first key vector and the first value vector of the support set to obtain the third key vector and the third value vector of the support set; and the cross-Transformer module aggregating the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set.

[0072] Specifically, the query set encoding vector and support set encoding vectors can be simultaneously input into the cross-Transformer network at a ratio of 1:k, that is, one query set encoding vector and k support set encoding vectors are input. In the cross-Transformer network, such as... Figure 5 As shown, the cross-Transformer module is a three-head cross-attention structure. The query set encoding vector and the support set encoding vector are processed by three stages of the three-head cross-Transformer module (each stage includes a three-head cross-attention structure) and the feedforward network to obtain query attention features and support attention features. Since feature interaction is performed at each stage, the few-shot learning similarity between the two branches is improved.

[0073] Specifically, the cross-transformer module first encodes the query set vector X separately. q Support set encoding vector X s The mapping is represented by a query vector Q, a key vector K, and a value vector V, i.e., in a multi-head cross-attention structure, such as... Figure 6 As shown, the query encoding vector X q It is mapped to Q via the cross Transformer module. iq ,K iq V iq (Query vector, first key vector, and first value vector of the query set), supporting encoded vector X s Mapped to Q is,K is V is (The query vector, first key vector, and first value vector of the support set).

[0074] Subsequently, to reduce the computational cost and complexity of attention, the cross-Transformer module modifies the first key vector K of the query set. iq and the first value vector V iq The reduction process is performed to obtain the second key vector and the second value vector of the query set, and the first key vector K of the support set is also reduced. is and the first value vector V is The reduction process is performed to obtain the third key vector and the third value vector of the support set. Figure 6 (Not shown in the image).

[0075] After reducing the size, refer to Figure 6 To effectively aggregate feature information between query set branches and support set branches, the cross-transformer module aggregates the second key vector, second value vector, third key vector, and third value vector to obtain query attention features of the query set and support attention features of the support set.

[0076] In one example, the SR (spatial-reduction) module can be used to reduce the size of the first key vector and the first value vector of the query set, and the SR module can also be used to reduce the size of the first key vector and the first value vector of the support set.

[0077] Specifically, the traditional Transformer's MHA (Multi-Head Attention) can be replaced with an SR module (including a Reshape layer, a Conv layer, and a liner layer). The SR module also takes Q, K, and V as inputs, but the difference is that the SR module scales down the key vector K and the value vector V. The calculation process of the SR module is as follows:

[0078] SR(x) = Norm(Reshape(x,R) i W s )

[0079] In the above formula, x represents the input vector of the SR module, and R... i Indicates the scaling ratio, Reshape indicates the operation of changing the vector shape, W s This represents a linear projection that reduces the dimensionality of the input vector, and Norm() represents a regularization layer.

[0080] Furthermore, the cross-Transformer module aggregates the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set, including:

[0081] The cross-transformer module performs average pooling on the third key vector to match the batch size of the second key vector, thus obtaining the fourth key vector of the support set. It also performs average pooling on the third value vector to match the batch size of the second value vector, again obtaining the fourth value vector of the support set. The second and fourth key vectors are concatenated to obtain the first concatenated key vector, and the second and fourth value vectors are concatenated to obtain the first concatenated value vector. The query vector of the query set, the first concatenated key vector, and the first concatenated value vector are then input into the feedforward network to obtain the query attention features of the query set.

[0082] The cross-transformer module performs average pooling on the second key vector to match the batch size of the third key vector, resulting in the fifth key vector of the query set. It also performs average pooling on the second value vector to match the batch size of the third value vector, again resulting in the fifth value vector of the query set. The third and fifth key vectors are concatenated to obtain the second concatenated key vector, and the third and fifth value vectors are concatenated to obtain the second concatenated value vector. The query vector, the second concatenated key vector, and the second concatenated value vector of the support set are then input into the feedforward network to obtain the support attention features of the support set.

[0083] Specifically, to aggregate key information (reduced KV pairs) from the support branch to the query branch, the cross-Transformer module performs average pooling on the third key vector in the KV pairs of the support set to match the batch size of the second key vector in the KV pairs of the query set, thus obtaining the fourth key vector in the support set. It also performs average pooling on the third value vector in the KV pairs of the support set to match the batch size of the second value vector in the KV pairs of the query set, thus obtaining the fourth value vector in the support set. Then, the second and fourth key vectors are concatenated to obtain the first concatenated key vector, and the second and fourth value vectors are concatenated to obtain the first concatenated value vector. Finally, the query vector, the first concatenated key vector, and the first concatenated value vector of the query set are input into the feedforward network to obtain the query attention features of the query set. Figure 6 Only the mapping and splicing processes are shown; the reduction and pooling processes are not shown.

[0084] Similarly, the cross-Transformer module performs average pooling on the second key vector of the query set to match the batch size of the third key vector of the support set, resulting in the fifth key vector of the query set. It also performs average pooling on the second value vector of the query set to match the batch size of the third key vector, resulting in the fifth value vector of the query set. Then, the third and fifth key vectors are concatenated to obtain the second concatenated key vector, and the third and fifth value vectors are concatenated to obtain the second concatenated value vector. Finally, the query vector, the second concatenated key vector, and the second concatenated value vector of the support set are input into the feedforward network to obtain the support attention features of the support set. Figure 6 Only the mapping and splicing processes are shown; the reduction and pooling processes are not shown.

[0085] In other words, the cross-Transformer module first performs average pooling on one branch to match the batch size of the other branch, and then concatenates the key-value pairs of the two branches to form the aggregated attention features (the concatenated key-value pairs), which can be expressed by the formula:

[0086]

[0087]

[0088]

[0089] head j =Atten(QW j Q ,SR(K cat W j K ,SR(V cat W j V )

[0090] Where Cat represents splicing, W j Q W j K W j V N represents the linear projection parameter. i This represents the number of heads in the cross-attention layer of stage i.

[0091] Finally, a feedforward network is used to enhance the representation of the concatenated attention features:

[0092]

[0093]

[0094] In the above formula, Representing the query attention features and support attention features after passing through the feedforward network, respectively, MLP() represents the multilayer perceptron operator, and LN represents LayerNorm. These represent the attention features after the query set and support set are aggregated (attention features without the feedforward network), respectively.

[0095] After obtaining the query attention features and support attention features, step S5 is executed, which involves aggregating the query attention features and support attention features into the feature enhancement network to obtain high-dimensional features.

[0096] In one embodiment of the present invention, step S5 may include: the feature enhancement network using a cosine similarity function to calculate the relationship score between the query attention feature and the support attention feature; and the feature enhancement network obtaining high-dimensional features based on the relationship score, the query attention feature, and the support attention feature.

[0097] Furthermore, the feature enhancement network obtains high-dimensional features based on relation scores, query attention features, and support attention features. This can include: the feature enhancement network multiplies the relation scores as weights with the support attention features to obtain weighted support attention features, and then fuses the weighted support attention features with the query attention features to obtain high-dimensional features.

[0098] Specifically, the query attention features and support attention features processed by the feedforward network are used to calculate a relation score using the cosine similarity function to determine the difference between the query set and the support set. The relation score is then multiplied by the support attention features as weights to obtain weighted support attention features. Finally, the weighted support attention features are fused with the query attention features to obtain high-dimensional features. The expression for the cosine similarity function is:

[0099]

[0100] Finally, a lightweight decoder is used to extract the high-dimensional feature R. i,j Dimensionality reduction and upsampling restore the original image size, yielding a segmentation mask. The resolution of this mask is set to a first preset value. Finally, the segmentation results are used to determine if any defects exist. Essentially, segmentation is achieved by extracting more effective object information from the support set and then transferring this information to the query image.

[0101] As described above, this embodiment of the invention uses meta-learning. In the feature encoding stage, a cross-transformer is constructed to enhance the information interaction between the query set and the support set. Furthermore, an SR module is added to the multi-head attention structure to reduce the computational cost of the network. The similarity between the query set features and the support set features is compared using a cosine similarity function. Finally, a decoder is used for dimensionality reduction and upsampling to obtain the final result. This embodiment of the invention does not use any convolutional neural network structure, but uses a complete Transformer structure to achieve defect detection for small samples.

[0102] During the model testing phase, since the test set and training set do not have the same data, it is necessary to extract and construct a new support set and query set from the test set. After the same feature extraction and cosine similarity calculation as in training, the relationship score is obtained, and finally the segmentation mask image with the highest relationship score is output.

[0103] In summary, this invention provides a solution to the problem of small-sample defect detection in industrial environments. Instead of using traditional convolutional neural networks, this invention employs a pure Transformer architecture for defect detection. It uses cross-Transformers to construct feature extractors for the query and support sets and introduces an SR module to reduce the network's computational load. Unlike traditional deep learning networks that require large amounts of training data, this invention uses meta-learning methods based on the Transformer framework to achieve training and detection with a small number of samples. This effectively reduces the cost and time of data annotation in industrial production processes.

[0104] Corresponding to the Transformer-based defect detection method in the above embodiments, the present invention also proposes a Transformer-based defect detection device.

[0105] Figure 7 This is a block diagram of a Transformer-based defect detection device according to an embodiment of the present invention.

[0106] like Figure 7 As shown, the Transformer-based defect detection device includes: an acquisition module 10, a first determination module 20, a construction module 30, a second determination module 40, a third determination module 50, a fourth determination module 60, and a detection module 70.

[0107] The acquisition module 10 is used to acquire a sample set containing sample images and their corresponding mask labels, and divide the sample set into a training set and a test set, and further divide the training set into a query set and a support set; the first determination module 20 is used to determine the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set; the construction module 30 is used to construct a defect detection network, which includes a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence, wherein the cross-Transformer network is constructed based on the cross-Transformer module; the second determination module... The third determining module 50 is used to input the query encoding vector and the support encoding vector into the cross-transformer network to obtain the query attention features of the query set and the support attention features of the support set; the fourth determining module 60 is used to input the query attention features and the support attention features into the feature enhancement network to obtain high-dimensional features; the fifth determining module 60 is used to input the high-dimensional features into the decoder to obtain the predicted segmentation mask map, and train the defect detection network based on the segmentation mask map and the mask label; the detection module 70 is used to acquire the product image to be detected, and perform defect detection on the product image to be detected through the trained defect detection network.

[0108] It should be noted that the specific implementation method and implementation principle of the Transformer-based defect detection device can be found in the specific implementation method of the Transformer-based defect detection method described above. To avoid redundancy, they will not be described in detail here.

[0109] The defect detection device based on Transformer in this invention requires neither a large amount of training data nor a large amount of training sample annotation, and can also achieve defect detection with a small number of samples, effectively reducing the cost and time of sample data annotation, improving the efficiency of small sample defect detection, and improving the defect detection effect.

[0110] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0113] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0115] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A defect detection method based on Transformer, characterized in that, Includes the following steps: Obtain a sample set containing sample images and their corresponding mask labels, and divide the sample set into a training set and a test set, and divide the training set into a query set and a support set; Determine the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set; A defect detection network is constructed, comprising a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence, wherein the cross-Transformer network is constructed based on the cross-Transformer module; The query encoding vector and the support encoding vector are input into the cross-Transformer network to obtain the query attention features of the query set and the support attention features of the support set. The query attention features and the support attention features are input into the feature enhancement network to obtain high-dimensional features; The high-dimensional features are input into the decoder to obtain the predicted segmentation mask image, and the defect detection network is trained based on the segmentation mask image and the mask label. An image of the product to be inspected is acquired, and a trained defect detection network is used to perform defect detection on the image. The cross-Transformer network includes a cross-Transformer module and a feedforward network, wherein the cross-Transformer module adopts a multi-head cross-attention structure. The process of inputting the query encoding vector and the support encoding vector into the cross-Transformer network to obtain the query attention features of the query set and the support attention features of the support set includes: inputting the query encoding vector and the support encoding vector into the cross-Transformer network; the cross-Transformer module mapping the query encoding vector and the support encoding vector into a query vector, a first key vector, and a first value vector, respectively; the cross-Transformer module reducing the size of the first key vector and the first value vector of the query set to obtain the second key vector and the second value vector of the query set, and reducing the size of the first key vector and the first value vector of the support set to obtain the third key vector and the third value vector of the support set; and the cross-Transformer module aggregating the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set. The cross-transformer module aggregates the second key vector, the second value vector, the third key vector, and the third value vector to obtain the query attention features of the query set and the support attention features of the support set. This includes: the cross-transformer module performs average pooling on the third key vector to match the batch size of the second key vector to obtain the fourth key vector of the support set; and performs average pooling on the third value vector to match the batch size of the second value vector to obtain the fourth value vector of the support set; concatenates the second key vector and the fourth key vector to obtain the first concatenated key vector; and concatenates the second value vector and the fourth value vector to obtain the first concatenated value vector; and aggregates the query attention features of the query set and the support attention features of the support set. The first concatenation key vector and the first concatenation value vector are input into the feedforward network to obtain the query attention features of the query set; the cross-transformer module performs average pooling on the second key vector to match the batch size of the third key vector to obtain the fifth key vector of the query set, and performs average pooling on the second value vector to match the batch size of the third value vector to obtain the fifth value vector of the query set; the third key vector and the fifth key vector are concatenated to obtain the second concatenation key vector, and the third value vector and the fifth value vector are concatenated to obtain the second concatenation value vector; the query vector of the support set, the second concatenation key vector, and the second concatenation value vector are input into the feedforward network to obtain the support attention features of the support set.

2. The defect detection method based on Transformer according to claim 1, characterized in that, Determining the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set includes: Determine the feature vector of each sample image in the query set and the support set; Determine the positional encoding information and branch encoding information corresponding to each sample image, and embed the positional encoding information and branch encoding information into the feature vector of the corresponding sample image to obtain the query encoding vector corresponding to each sample image in the query set and the support encoding vector corresponding to each sample image in the support set.

3. The defect detection method based on Transformer according to claim 2, characterized in that, Determining the feature vector of each sample image in the query set and the support set includes: After setting the resolution of each sample image in the query set and the support set to a first preset value, the images are cropped into multiple image blocks, wherein all image blocks of each sample image constitute the image block sequence of the sample image; The feature vector of each sample image is obtained by projecting the sequence of image blocks of each sample image through a linear projection layer.

4. The defect detection method based on Transformer according to claim 1, characterized in that, The cross-transformer module uses the SR module to reduce the size of the first key vector and the first value vector of the query set, and also uses the SR module to reduce the size of the first key vector and the first value vector of the support set.

5. The defect detection method based on Transformer according to claim 1, characterized in that, The query attention features and the support attention features are input into the feature enhancement network for aggregation processing to obtain high-dimensional features, including: The feature enhancement network uses a cosine similarity function to calculate a relationship score between the query attention feature and the support attention feature; The feature enhancement network obtains high-dimensional features based on the relation score, the query attention feature, and the support attention feature.

6. The defect detection method based on Transformer according to claim 5, characterized in that, The feature enhancement network obtains high-dimensional features based on the relation score, the query attention feature, and the support attention feature, including: The feature enhancement network multiplies the relation score as a weight with the support attention feature to obtain a weighted support attention feature, and then fuses the weighted support attention feature with the query attention feature to obtain a high-dimensional feature.

7. The defect detection method based on Transformer according to claim 1, characterized in that, The feedforward network uses the LayerNorm function and the GELU activation function.

8. A Transformer-based defect detection device for implementing the defect detection method of claim 1, characterized in that, include: The acquisition module is used to acquire a sample set containing sample images and their corresponding mask labels, and divide the sample set into a training set and a test set, and divide the training set into a query set and a support set; The first determining module is used to determine the query encoding vector corresponding to each sample image of the query set and the support encoding vector corresponding to each sample image of the support set. A construction module is used to construct a defect detection network, which includes a cross-Transformer network, a feature enhancement network, and a decoder connected in sequence, wherein the cross-Transformer network is constructed based on the cross-Transformer module; The second determining module is used to input the query encoding vector and the support encoding vector into the cross-Transformer network to obtain the query attention features of the query set and the support attention features of the support set. The third determining module is used to input the query attention features and the support attention features into the feature enhancement network to obtain high-dimensional features; The fourth determination module is used to input high-dimensional features into the decoder to obtain a predicted segmentation mask image, and to train the defect detection network based on the segmentation mask image and the mask label. The detection module is used to acquire an image of the product to be detected and to perform defect detection on the image of the product to be detected through a trained defect detection network.

Images