A solder joint detection method based on machine vision

Abstract

The present invention proposes a solder joint detection method based on machine vision, which relates to the technical field of image recognition. The present invention proposes a solder joint detection process, including the preparation of a solder joint image data set, the construction of an efficient attention module ESA, the construction of a front convolution module FM, the construction of a back convolution module BM, the construction of an efficient feature block EFB, the construction of an efficient convolution encoder ECE, the construction of an efficient feature unit EFU, the construction of a machine vision solder joint detection model, and the training and detection of the machine vision solder joint detection model; at the same time, an efficient attention module ESA is proposed, linear element-by-element multiplication is used instead of quadratic matrix multiplication operations, and a machine vision solder joint detection model is formed based on the efficient attention module ESA, which is applied to solder joint detection scenarios, achieving an excellent balance between accuracy and efficiency.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition, and in particular relates to a welding spot detection method based on machine vision. Background Art

[0002] In the automobile manufacturing process, weld spot detection is of vital importance. Weld spots are the main connection process for automobile bodies and other parts, and their connection quality directly determines the overall structural rigidity and integrity of the automobile. The quality of weld spots is crucial to the safety and performance of the automobile. Therefore, strict and accurate detection of weld spots is an important part of ensuring automobile quality.

[0003] The use of image recognition technology to replace manual judgment of whether a solder joint is qualified or not is significant in achieving efficient and accurate automated testing, avoiding interference from human factors, and improving testing efficiency and product quality. At the same time, this technology can monitor the quality of solder joints in real time online, discover and solve potential problems in a timely manner, reduce production costs and risks, and provide strong support for the sustainable development of the automotive manufacturing industry.

[0004] Self-attention has become the de facto choice for capturing global context in various vision applications, however the computational complexity quadratic in image resolution limits its use in real-time applications, especially deployment on resource-constrained mobile devices. Although hybrid methods have been proposed to combine the advantages of convolution and self-attention to achieve better speed-accuracy trade-off, the expensive matrix multiplication operations in self-attention remain a bottleneck. Summary of the invention

[0005] The present invention provides a solder joint detection method based on machine vision, aiming to propose an efficient attention module ESA, use linear element-by-element multiplication instead of quadratic matrix multiplication operation, and eliminate explicit key-value interaction, so as to reduce the amount of self-attention operation without compromising the detection accuracy. At the same time, a front convolution module FM and a back convolution module BM are proposed, and the efficient feature block EFB is composed of the efficient feature block EFB and the efficient convolution encoder ECE to form an efficient feature unit EFU, thereby forming a machine vision solder joint detection model, and applying it in the solder joint detection scenario, achieving an excellent balance between accuracy and efficiency.

[0006] The present invention aims to propose an efficient attention module ESA and provide a solder joint detection method based on machine vision, comprising the following steps:

[0007] S1. Preparation of welding spot image dataset: collecting automobile manufacturing process images containing welding spots, marking welding spot areas in the automobile manufacturing process images, and obtaining welding spot image dataset;

[0008] S2, build an efficient attention module ESA, which includes linear layer, batch normalization, query operation and context operation;

[0009] S3, build the front convolution module FM, including 1×1 convolution, 3×3 depth convolution and batch normalization;

[0010] S4, construct the post-convolution module BM, which includes 1×1 convolution, batch normalization and GeLU activation function;

[0011] S5, construct an efficient feature block EFB, which is composed of the front convolution module FM, the efficient attention module ESA and the back convolution module BM.

[0012] S6. Build an efficient convolutional encoder ECE, which includes 3×3 depth convolution, batch normalization, 1×1 convolution and GeLU activation function;

[0013] S7, construct an efficient feature unit EFU, which is composed of an efficient convolutional encoder ECE and an efficient feature block EFB in sequence;

[0014] S8. Build a machine vision solder joint detection model, including input, tile embedding, multiple efficient feature units EFU, multiple downsampling, classifiers and output;

[0015] S9. Train a machine vision solder joint detection model and conduct detection. Use a solder joint image dataset to train the machine vision solder joint detection model. After the training is completed, input the images to be detected taken at the automobile manufacturing site into the machine vision solder joint detection model to obtain classification results. The classification results are divided into qualified solder joints and unqualified solder joints.

[0016] Preferably, in step S2, for the efficient attention module ESA, input feature X ESAIn , X ESAIn ∈R H×W×C , H, W and C are the input features X ESAIn The height, width and channels of q and W k Generate query Q and key K, Q = W ｑ X ESAIn , K=W k X ESAIn , W q ∈R d×d W k ∈R d×d , Q∈R n×d , K∈R n×d , where n is the sequence length of Q and K, d is the sequence dimension of Q and K, and then the global attention query vector α is obtained, α∈R n , w α ∈R d, · is matrix multiplication, w α is a learnable parameter vector, and then a query operation is performed, which inputs α and Q and outputs a single global query vector q. * is the broadcast element-wise multiplication, α i is the i-th element in the vector α, α i ∈R 1 , Q i is the i-th sequence in Q, 1≤i≤n, Q i ∈R 1×d , then calculate BN stands for batch normalization, Then perform context operation, context operation input q and K, the output is X ESAOut , X ESAOut This is the output of the efficient attention module ESA. X ESAOut ∈R n×d , n×d=H×W×C, + is element-by-element addition, * is broadcast element-by-element multiplication, where L1 represents linear layer L1, and L2 represents linear layer L2.

[0017] Preferably, in step S3, for the front convolution module FM, the input feature X FMIn , X FMIn ∈R H×W×C , H, W and C are the input features X FMIn The height, width and channels of the front convolution module FM are FMOut , X FMOut =Conv(DWConv(BN(X FMIn ))), X FMOut ∈R H×W×C , where BN stands for batch normalization, DWConv stands for 3×3 depthwise convolution, and Conv stands for 1×1 convolution.

[0018] Preferably, in step S4, for the post-convolution module BM, the input feature X BMIn , X BMIn ∈R H×W×C , H, W and C are the input features X BMIn The height, width and channels of the post-convolution module BM are BMOut , X BMOut =Conv2(BN(Conv1(GeLU(X BMIn ))))+X BMIn , X BMOut ∈R H×W×C, where BN stands for batch normalization, Conv1 is the first 1×1 convolution, GeLU stands for the GeLU activation function, Conv2 is the second 1×1 convolution, and + stands for element-wise addition.

[0019] Preferably, in step S5, for the efficient feature block EFB, input feature X EFBIn , X EFBIn ∈R H×W×C , H, W and C are the input features X EFBIn The height, width and channel of X EFBIn First, it passes through the front convolution module FM, then the efficient attention module ESA, and finally the back convolution module BM, and the output X of the efficient feature block EFB is obtained. EFBOut , X EFBOut ∈R H×W×C .

[0020] Preferably, in step S6, for the efficient convolutional encoder ECE, the input feature X ECEIn , X ECEIn ∈R H×W×C , H, W and C are the input features X ECEIn The height, width and channels of the efficient convolutional encoder ECE are output as X ECEOut , X ECEOut =Conv2(GeLU(Conv1(BN(DWConv(X ECEIn ))))))+X ECEIn , X ECEOut ∈R H×W×C , where DWConv stands for 3×3 depthwise convolution, BN stands for batch normalization, Conv1 is the first 1×1 convolution, GeLU stands for GeLU activation function, Conv2 is the second 1×1 convolution, and + stands for element-wise addition.

[0021] Preferably, in step S7, for the efficient feature unit EFU, input feature X EFUIn , X EFUIn ∈R H×W×C , H, W and C are the input features X EFUIn The height, width and channel of X EFUIn First, it passes through the efficient convolution encoder ECE, then passes through the efficient feature block EFB, and obtains the output X of the efficient feature unit EFU EFUOut , X EFUOut ∈R H×W×C .

[0022] Preferably, in step S8, for the machine vision solder joint detection model, the tile embedding is implemented by two 3×3 convolutions with a stride of 2, and the classifier is implemented by multiple fully connected layers. The input image to be detected I, I∈R H×W×3, H, W and 3 are the height, width and channel of the input image I, I is embedded into the image patch to get the feature X, and C1 are the height, width and channel of feature X. Input X into the first efficient feature unit EFU to obtain feature X E1 , and C1 is feature X E1 The height, width and channel of X E1 Input to the first downsampling to get feature X de1 , and C2 is feature X de1 The height, width and channel of X de1 Input to the second efficient feature unit EFU to obtain feature X E2 , and C2 is feature X E2 The height, width and channel of X E2 Input to the second downsampling to get feature X de2 , and C3 is feature X de2 The height, width and channel of X de2 Input to the third efficient feature unit EFU to obtain feature X E3 , and C3 is feature X E3 The height, width and channel of X E3 Input to the third downsampling to get feature X de3 , and C4 is feature X de3 The height, width and channel of X de3 Input to the fourth efficient feature unit EFU to obtain feature X E4 , and C4 is feature X E4 The height, width and channel of feature X E4 Input into the classifier and get the classification result.

[0023] Compared with the prior art, the present invention has the following technical effects:

[0024] The technical solution provided by the present invention proposes an efficient attention module ESA, uses linear element-by-element multiplication instead of quadratic matrix multiplication operations, and eliminates explicit key-value interactions, thereby reducing the amount of self-attention operations without compromising detection accuracy. At the same time, a front convolution module FM and a post convolution module BM are proposed, which together with the efficient attention module ESA form an efficient feature block EFB, and the efficient feature block EFB and the efficient convolution encoder ECE form an efficient feature unit EFU, thereby forming a machine vision solder joint detection model, and applying it to solder joint detection scenarios, achieving an excellent balance between accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a solder joint detection flow chart provided by the present invention.

[0026] Figure 2 This is a structural diagram of the efficient attention module ESA provided by the present invention.

[0027] Figure 3 This is a structural diagram of the pre-convolution module FM provided by the present invention.

[0028] Figure 4 This is a structural diagram of the post-convolution module BM provided by the present invention.

[0029] Figure 5 This is a structural diagram of the efficient feature block EFB provided by the present invention.

[0030] Figure 6 This is a structural diagram of the efficient convolutional encoder ECE provided by the present invention.

[0031] Figure 7 This is a structural diagram of the high-efficiency characteristic unit EFU provided by the present invention.

[0032] Figure 8 It is a structural diagram of the machine vision solder joint detection model provided by the present invention. DETAILED DESCRIPTION

[0033] The present invention aims to propose a solder joint detection method based on machine vision, proposes an efficient attention module ESA, uses linear element-by-element multiplication instead of quadratic matrix multiplication operation, and eliminates explicit key-value interaction, while reducing the amount of self-attention operation without compromising detection accuracy. At the same time, a front convolution module FM and a back convolution module BM are proposed, and the efficient feature block EFB is composed of the efficient feature block EFB and the efficient convolution encoder ECE to form an efficient feature unit EFU, thereby forming a machine vision solder joint detection model, and applying it in the solder joint detection scenario, achieving an excellent balance between accuracy and efficiency.

[0034] See also Figure 1 As shown, a method for detecting solder joints based on machine vision in an embodiment of the present application:

[0035] S1. Preparation of welding spot image dataset: collect automobile manufacturing process images containing welding spots, a total of 3000 images, annotate the welding spot areas in the automobile manufacturing process images, and obtain welding spot image dataset;

[0036] S2, build an efficient attention module ESA, which includes 2 linear layers, 1 batch normalization, 1 query operation and 1 context operation;

[0037] S3, build the front convolution module FM, including 1 1×1 convolution, 1 3×3 depth convolution and 1 batch normalization;

[0038] S4, build the post-convolution module BM, which includes 2 1×1 convolutions, 1 batch normalization and 1 GeLU activation function;

[0039] S5, construct an efficient feature block EFB, which consists of a front convolution module FM, an efficient attention module ESA and a back convolution module BM.

[0040] S6. Build an efficient convolution encoder ECE, which includes 1 3×3 depth convolution, 1 batch normalization, 2 1×1 convolutions and 1 GeLU activation function;

[0041] S7, construct an efficient feature unit EFU, which is composed of an efficient convolutional encoder ECE and an efficient feature block EFB in sequence;

[0042] S8. Build a machine vision solder joint detection model, including 1 tile embedding, 4 efficient feature units (EFUs), 3 downsampling and 1 classifier;

[0043] S9. Train a machine vision solder joint detection model and conduct detection. Use a solder joint image dataset to train the machine vision solder joint detection model. Divide the solder joint image dataset into a training set and a validation set at a ratio of 7 to 3. After the training is completed, input the images to be detected taken at the automobile manufacturing site into the machine vision solder joint detection model to obtain classification results. The classification results are divided into qualified solder joints and unqualified solder joints.

[0044] Furthermore, in step S2, for the efficient attention module ESA, its structure is as follows Figure 2 As shown, input feature X ESAIn , X ESAIn ∈R H×W×C , H, W and C are the input features X ESAIn The height, width and channels of q and W k Generate query Q and key K, Q = W ｑ X ESAIn , K=W k X ESAIn , W q ∈R d×d , W k ∈R d×d , Q∈R n×d , K∈R n×d , where n is the sequence length of Q and K, d is the sequence dimension of Q and K, and then the global attention query vector α is obtained, α∈R n , w α ∈Rd , · is matrix multiplication, w α is a learnable parameter vector, and then a query operation is performed, which inputs α and Q and outputs a single global query vector q. q∈R d , * is broadcast element-by-element multiplication, α i is the i-th element in the vector α, α i ∈R 1 , Q i is the i-th sequence in Q, 1≤i≤n, Q i ∈R 1×d , then calculate BN stands for batch normalization, Then perform context operation, context operation input q and K, the output is X ESAOut , X ESAOut This is the output of the efficient attention module ESA. X ESAOut ∈R n×d , n×d=H×W×C, + is element-by-element addition, * is broadcast element-by-element multiplication, where L1 represents linear layer L1, and L2 represents linear layer L2.

[0045] Furthermore, in step S3, for the front convolution module FM, its structure is as follows Figure 3 As shown, input feature X FMIn , X FMIn ∈R H×W×C , H, W and C are the input features X FMIn The height, width and channels of the front convolution module FM are FMOut , X FMOut =Conv(DWConv(BN(X FMIn )))

[0046] X FMOut ∈R H×W×C , where BN stands for batch normalization, DWConv stands for 3×3 depthwise convolution, and Conv stands for 1×1 convolution.

[0047] Further, in step S4, for the post-convolution module BM, its structure is as follows Figure 4 As shown, input feature X BMIn , X BMIn ∈R H×W×C , H, W and C are the input features X BMIn The height, width and channels of the post-convolution module BM are BMOut , X BMOut =Conv2(BN(Conv1(GeLU(X BMIn ))))+X BMIn, X BMOut ∈R H×W×C , where BN stands for batch normalization, Conv1 is the first 1×1 convolution, GeLU stands for the GeLU activation function, Conv2 is the second 1×1 convolution, + stands for element-wise addition, and + and Figure 4 In correspond.

[0048] Further, in step S5, for the efficient feature block EFB, its structure is as follows Figure 5 As shown, input feature X EFBIn , X EFBIn ∈R H×W×C , H, W and C are the input features X EFBIn The height, width and channel of X EFBIn First, it passes through the front convolution module FM, then the efficient attention module ESA, and finally the back convolution module BM, and the output X of the efficient feature block EFB is obtained. EFBOut , X EFBOut ∈R H×W×C .

[0049] Further, in step S6, for the efficient convolutional encoder ECE, its structure is as follows Figure 6 As shown, input feature X ECEIn , X ECEIn ∈R H×W×C , H, W and C are the input features X ECEIn The height, width and channels of the efficient convolutional encoder ECE are output as X ECEOut , X ECEOut =Conv2(GeLU(Conv1(BN(DWConv(X ECEIn ))))))+X ECEIn X ECEOut ∈R H×W×C , where DWConv stands for 3×3 depthwise convolution, BN stands for batch normalization, Conv1 is the first 1×1 convolution, GeLU stands for GeLU activation function, Conv2 is the second 1×1 convolution, + stands for element-wise addition, and + and Figure 6 In correspond.

[0050] Further, in step S7, for the high efficiency characteristic unit EFU, its structure is as follows Figure 7 As shown, input feature X EFUIn , X EFUIn ∈R H×W×C , H, W and C are the input features X EFUIn The height, width and channel of X EFUIn First, it passes through the efficient convolution encoder ECE, then passes through the efficient feature block EFB, and obtains the output X of the efficient feature unit EFUEFUOut , X EFUOut ∈R H×W×C .

[0051] Further, in step S8, for the machine vision solder joint detection model, its structure is as follows Figure 8 As shown in the figure, the patch embedding is implemented by two 3×3 convolutions with a stride of 2, and the classifier is implemented by multiple fully connected layers. The input image to be detected is I, I∈R H×W×3 , H, W and 3 are the height, width and channel of the input image I, I is embedded into the image patch to get the feature X, and C1 are the height, width and channel of feature X. Input X into the first efficient feature unit EFU to obtain feature X E1 , and C1 is feature X E1 The height, width and channel of X E1 Input to the first downsampling to get feature X de1 , and C2 is feature X de1 The height, width and channel of X de1 Input to the second efficient feature unit EFU to obtain feature X E2 , and C2 is feature X E2 The height, width and channel of X E2 Input to the second downsampling to get feature X de2 , and C3 is feature X de2 The height, width and channel of X de2 Input to the third efficient feature unit EFU to obtain feature X E3 , and C3 is feature X E3 The height, width and channel of X E3 Input to the third downsampling to get feature X de3 , and C4 is feature X de3 The height, width and channel of X de3 Input to the fourth efficient feature unit EFU to obtain feature X E4 , and C4 is feature X E4 The height, width and channel of feature X E4 Input into the classifier and get the classification result.

[0052] Further, in step S8, for the machine vision solder joint detection model, its structure is as follows Figure 8 As shown in the figure, the patch embedding is implemented by two 3×3 convolutions with a stride of 2, and the classifier is implemented by multiple fully connected layers. The input image to be detected is I, I∈R640 ×640×3 , 640, 640 and 3 are the height, width and channel of the input image I, I is embedded into the image patch to obtain the feature X, X∈R 160×160×48 , 160, 160 and 48 are the height, width and channel of feature X, X is input into the first efficient feature unit EFU, and feature X is obtained E1 , X E1 ∈R 160×160×48 , 160, 160 and 48 are feature X E1 The height, width and channel of X E1 Input to the first downsampling to get feature X de1 , X de1 ∈R 80×80×96 , 80, 80 and 96 are features X de1 The height, width and channel of X de1 Input to the second efficient feature unit EFU to obtain feature X E2 , X E2 ∈R 80×80×96 , 80, 80 and 96 are features X E2 The height, width and channel of X E2 Input to the second downsampling to get feature X de2 , X de2 ∈R 40×40×192 , 40, 40 and 192 are features X de2 The height, width and channel of X de2 Input to the third efficient feature unit EFU to obtain feature X E3 , X E3 ∈∈R 40×40×192 , 40, 40 and 192 are features X E3 The height, width and channel of X E3 Input to the third downsampling to get feature X de3 , X de3 ∈R 20×20×384 , 20, 20 and 384 are features of X de3 The height, width and channel of X de3 Input to the fourth efficient feature unit EFU to obtain feature X E4 , X E4 ∈R 20×20×384 , 20, 20 and 384 are features of X E4 The height, width and channel of feature X E4 Input into the classifier and get the classification result.

[0053] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A solder joint detection method based on machine vision, characterized in that: The following steps are involved: S1. Preparation of welding spot image dataset: collecting automobile manufacturing process images containing welding spots, marking welding spot areas in the automobile manufacturing process images, and obtaining welding spot image dataset; S2, build an efficient attention module ESA, including linear layer, batch normalization, query operation and context operation; S3, build the front convolution module FM, including 1×1 convolution, 3×3 depth convolution and batch normalization; S4, construct the post-convolution module BM, which includes 1×1 convolution, batch normalization and GeLU activation function; S5, construct an efficient feature block EFB, which is composed of the front convolution module FM, the efficient attention module ESA and the back convolution module BM. S6. Build an efficient convolutional encoder ECE, which includes 3×3 depth convolution, batch normalization, 1×1 convolution and GeLU activation function; S7, construct an efficient feature unit EFU, which is composed of an efficient convolutional encoder ECE and an efficient feature block EFB in sequence; S8. Build a machine vision solder joint detection model, including input, tile embedding, multiple efficient feature units EFU, multiple downsampling, classifiers and output; S9, training a machine vision solder joint detection model and conducting detection, using a solder joint image data set to train the machine vision solder joint detection model, after the training is completed, inputting the images to be detected taken at the automobile manufacturing site into the machine vision solder joint detection model to obtain classification results, the classification results being qualified solder joints and unqualified solder joints; In step S2, for the efficient attention module ESA, the input feature , , , and is the input feature The height, width and channels of and Generate a query and key , , , , , , ,in yes and The sequence length, yes Q and K The sequence dimension of , and then get the global attention query vector , , , , is matrix multiplication, is a learnable parameter vector, and then performs query operations ， Query operation input and , outputs a single global query vector , , , is the element-wise multiplication for broadcasting, is a vector The i-th element in , yes The i-th sequence in , , then calculate , , stands for batch normalization, , and then perform context operation, context operation input , and , the output is , This is the output of the efficient attention module ESA. , , , is element-wise addition, is a broadcast element-wise multiplication, where represents the linear layer L1, represents the linear layer L2; In step S3, for the front convolution module FM, the input feature , , , and is the input feature The height, width and channels of the front convolution module FM are , , ,in stands for batch normalization, represents a 3×3 depthwise convolution, Represents 1×1 convolution; In step S4, for the post-convolution module BM, the input feature , , , and is the input feature The height, width and channels of the post-convolution module BM are , , ,in stands for batch normalization, is the first 1×1 convolution, represents the GeLU activation function, is the second 1×1 convolution, stands for element-by-element addition; In step S5, for the efficient feature block EFB, the input feature , , , and is the input feature height, width and passages, First, it passes through the front convolution module FM, then the efficient attention module ESA, and finally the back convolution module BM to get the output of the efficient feature block EFB , ; In step S6, for the efficient convolutional encoder ECE, the input feature , , , and is the input feature The height, width and channels of the efficient convolutional encoder ECE are output as follows: , , ,in represents a 3×3 depthwise convolution, stands for batch normalization, is the first 1×1 convolution, represents the GeLU activation function, is the second 1×1 convolution, stands for element-by-element addition; In step S7, for the efficient feature unit EFU, enter the feature , , , and is the input feature height, width and passages, First, it passes through the efficient convolution encoder ECE, then passes through the efficient feature block EFB, and obtains the output of the efficient feature unit EFU , ; In step S8, for the machine vision solder joint detection model, the tile embedding is implemented by two 3×3 convolutions with a stride of 2, and the classifier is implemented by multiple fully connected layers. , , , and 3 is the input image height, width and passages, Obtaining features through tile embedding , , , and It is a feature The height, width and channels of Input to the first efficient feature unit EFU to get the feature , , , and It is a feature The height, width and channels of Input to the first downsampling to get the features , , , and It is a feature The height, width and channels of Input to the second efficient feature unit EFU to obtain the feature , , , and It is a feature The height, width and channels of Input to the second downsampling to get the features , , , and It is a feature The height, width and channels of Input to the third efficient feature unit EFU to obtain the feature , , , and It is a feature The height, width and channels of Input to the third downsampling to get the features , , , and It is a feature The height, width and channels of Input to the fourth efficient feature unit EFU to obtain the feature , , , and It is a feature The height, width and channel of Input into the classifier and get the classification result.

Images