A method for extracting the position of weld points based on the Transformer neural network

Abstract

The present invention provides a method for extracting the position of weld points based on a Transformer neural network. The method includes: collecting weld pictures as training samples, and extracting feature points from the training samples; obtaining the coordinate information of the feature points and performing normalization processing; inputting the processed data into a training network, outputting a training result, comparing the training result with the true value, and outputting and saving the optimal training parameters; inputting the optimal training parameters into the training network to obtain a final weld network model; collecting the weld pictures to be detected, performing feature point extraction and data processing, and then inputting them into the final weld network model to obtain the coordinate information of the weld points of the weld pictures to be detected; outputting the obtained coordinate information of the weld points to a welding robot, and performing an automated welding operation through the welding robot. The present invention improves the accuracy and efficiency of extracting the position of weld points, so as to meet the real-time requirements of robot welding and relevant industrial standards.

Description

Technical Field

[0001] The present invention relates to the technical field of welding robots, and in particular to a method for extracting the position of weld points based on a Transformer neural network. Background Art

[0002] A welding robot is an industrial robot engaged in welding (including cutting and spraying). During the actual welding process of the welding robot, due to the influence of environmental or human factors, such as high temperature, dust, strong light radiation, hand tremors, human error in precision judgment, etc., problems such as partial welding, porosity, and insufficient penetration in the weld seam are caused. These problems not only pose safety hazards to the welding process and subsequent use process, but also cause losses to the interests of enterprises or factories. Therefore, it is crucial to improve the accuracy of the welding robot in extracting the weld position, which can not only make up for safety hazards, but also improve the efficiency of enterprises or factories.

[0003] Traditional weld detection methods require manual heuristic repeated search for in-weld points, which consumes a large amount of time for each detection, and when there is too much data, the accuracy is poor. Although some existing methods based on convolutional neural networks have significantly improved the accuracy compared with traditional methods, they cannot accurately locate the position of defects in the weld image, can only have a high recognition rate for certain types of defects, the overall recognition accuracy is not high, and some detection methods have improved the accuracy, but they are still trained for weld images. The images are not robust enough to the size during training and detection, and the preprocessing is also more troublesome. In addition, the number of network layers is too large, which reduces the calculation efficiency and increases the time cost. It cannot well transmit the coordinates of the weld points to the robot in real time to achieve automated welding of the weld seam. At the same time, there are also problems such as difficult model design and failure to reach the expected performance. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for extracting the position of weld points based on a Transformer neural network to achieve accurate and rapid extraction of the position of weld points.

[0005] The present invention is implemented as follows: A method for extracting the position of weld points based on a Transformer neural network includes the following steps:

[0006] Step 1: Collect weld pictures as training samples, and extract feature points from the training samples. The feature points include weld points and non-weld points;

[0007] Step 2: Obtain the coordinate information of the feature points and perform normalization processing;

[0008] Step 3: Input the processed data into the training network, output the training results, compare the training results with the true values, and output and save the optimal training parameters;

[0009] Step 4: Input the optimal training parameters into the training network to obtain the final weld network model;

[0010] Step 5: Collect the weld images to be detected, extract the feature points and process the data, then input them into the final weld network model to obtain the weld point coordinate information of the weld images to be detected;

[0011] Step 6: Output the obtained weld point coordinate information to the welding robot, and perform automated welding operations through the welding robot.

[0012] Further, the specific steps of Step 1 include:

[0013] Connect the industrial camera to the terminal for communication through the IP address of the industrial camera, and collect a series of weld images as training samples through the industrial camera;

[0014] For each training sample, obtain the feature points of the weld through the Canny operator of the feature extraction algorithm. The feature points include weld points and non-weld points. Mark the weld points as positive samples and the non-weld points as negative samples.

[0015] Further, the specific steps of Step 2 include: Obtain the coordinate information of the feature points. The data dimension of the feature points is N×2, where N represents N points, the feature dimension of each point is 2, and the value is the coordinate value X(s, t) of the extracted feature point. s represents the abscissa value in the same coordinate system of each image, and t represents the ordinate value in the same coordinate system of each image. Normalize the obtained position data of the weld points.

[0016] Further, the specific steps of Step 3 include:

[0017] Step 3-1: Pre-define the weld point label as 1 and the non-weld point label as 0. After corresponding the coordinates of the weld feature points in each image with their corresponding labels one by one, the label is in the format of N×1, representing the true labels of N points;

[0018] Step 3-2: Input the matrix of the processed feature point data N×2 into the training network, and the training network is a Transformer neural network;

[0019] Step 3-3: Use the encoding part of the Transformer neural network to learn the feature information of the global points and encode each feature information;

[0020] Step 34: Input the encoding result, the obtained weld point labels, and non-weld point labels into the Transformer decoding part for joint training to obtain matrix decoding, weld point categories, and non-weld point categories. Then, through operations, the purified weld information is obtained. Finally, the weld information is restored to the original data size, and the predicted category information is output;

[0021] Step 35: Compare the output predicted category information with the true value, calculate the error backpropagation, and update the network parameters until the network reaches the convergence state, and then save the optimal training parameters.

[0022] Further, the specific steps of Step 33 include:

[0023] Step 331: Split the coordinate values X of N feature points of each weld image into N small block vectors, and embed the X(s, t) in the horizontal and vertical 2 dimensions of the N small block vectors into 128 dimensions, which is represented by the high-dimensional vector a, and the dimension of each a is 1×128. [a1, a2,..., a n represents the set of N input information converted into an N×128 matrix. Multiply a by three different vectors , , respectively to obtain the initial representations of the Q, K, and V vectors:

[0024] ;

[0025] ;

[0026] ;

[0027] Among them, represents the parameters learned by the three networks, and the dimension is equal to the transpose of a;

[0028] Step 332: Input Q, K, and V into the multi-head attention module. This module divides the network into multiple heads to form multiple subspaces, allowing the network to focus on information at different levels and obtain feature information at multiple levels;

[0029] Step 333: Input the feature information at multiple levels into the residual and layer normalization layers. The residual connection adds the input of the previous layer and the output of the previous layer, that is, x + Multi-Head(Q, K, V). The layer normalization normalizes each layer; where x refers to the weld point coordinate matrix after embedding.

[0030] Step 334: Finally, it is input into the feed-forward network + residual and layer normalization. The feed-forward network is a simple two-layer linear mapping followed by an operation of an activation function, and then processed by the residual and layer normalization layer to complete the network encoding and encode the information of each feature point.

[0031] Further, the specific content of step 332 is as follows:

[0032] The obtained Q, K, and V are input into the linear layer, and the operations are as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] i = 1, 2,..., 8;

[0037] Among them, , , represent the respective weight parameters of Q, K, and V in the i-th head, , , represent the output of the linear layer of the i-th head;

[0038] Enter the scaled dot-product attention layer, and the operations are as follows:

[0039] ) ;

[0040] ;

[0041] Softmax is a function for calculating scores here, is its calculation result; represents the output of the i-th head in the scaled dot-product attention layer;

[0042] Perform the calculations of the concatenation layer and the linear layer:

[0043] Multi-Head(Q, K, V) = Concat( ,..., ) ;

[0044] Concat means to merge and concatenate the results of h heads, represents the weight of the last linear layer of the multi-head attention, and Multi-Head(Q, K, V) is the output of the entire multiple attentions, that is, the multi-level feature information.

[0045] Further, the specific network modules in the Transformer decoding part are the same as those in the network encoding part. The label information of both weld points and non-weld points is added simultaneously to obtain the encoded information of the categories. The encoded information of the categories is dot-multiplied with the decoding result of the Transformer decoding part to obtain N category information with a dimension of N×1, and the coordinate information of the category of weld points is obtained, thereby obtaining the weld track of the weld picture to be detected.

[0046] Further, cross entropy is used as the error function for network training. The probabilities predicted for each category under binary classification are defined as p and 1-p, and the expression of the error function is:

[0047] ;

[0048] where represents the category label value of sample i, 1 for weld points and 0 for non-weld points;

[0049] represents the probability that sample i is predicted as an in-weld point, M represents the number of samples, and the base of Log is e.

[0050] Further, step 5 specifically includes:

[0051] Collect the weld pictures to be detected through an industrial camera, obtain the feature points of the welds through the Canny operator of the feature extraction algorithm for each weld picture. The feature points include weld points and non-weld points, obtain the coordinate information of the feature points, and normalize the position data of the obtained weld points.

[0052] The present invention has the following advantages: By using the Transformer as the basic network, the Attention mechanism therein can learn the global information of weld points. The welding points are trained through an algorithm and the weld point track is detected, improving the recognition accuracy of the robot for welds, increasing the weight of the region of interest, assisting the network to more quickly locate the region of interest, further improving the operation speed, realizing highly parallelized training, and greatly improving the efficiency of weld extraction. In addition, the neural network extracts the position information of the weld and transmits its coordinate information to the robot to achieve automatic welding of the weld. The present invention improves the overall operation efficiency and meets the real-time requirements of robot welding and relevant industrial standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described below with reference to the accompanying drawings in conjunction with embodiments.

[0054] Figure 1 is the flowchart of the method of the present invention.

[0055] Figure 2Schematic diagram of the industrial camera capturing pictures in the specific embodiment of the present invention.

[0056] Figure 3 Schematic diagram of the overall distribution of weld points in the picture feature extraction diagram in the embodiment of the present invention.

[0057] Figure 4 Schematic diagram of the network framework in the embodiment of the present invention.

[0058] Figure 5 Schematic diagram of the structure of the encoding part in the network framework in the embodiment of the present invention.

[0059] Figure 6 Schematic diagram of the principle of the multi-head attention module in the embodiment of the present invention. Detailed implementation manners

[0060] The overall design idea of the technical solution in the embodiment of this application is as follows: Through the attention mechanism and the network, comprehensively learn the key points of the weld. When learning the global information, introduce the category encoding information of the weld points and non-weld points, integrate it into the network for joint learning, enable the network to better learn the weld point features, improve the accuracy of weld key point detection, achieve accurate and fast weld trajectory detection, and realize high-efficiency automated welding of the weld.

[0061] Please refer to Figures 1 to 6 , the present invention provides a method for extracting the position of weld points based on a Transformer neural network, including the following steps:

[0062] Step 1: Collect weld pictures as training samples, and extract feature points from the training samples. The feature points include weld points and non-weld points;

[0063] Step 2: Obtain the coordinate information of the feature points and perform normalization processing;

[0064] Step 3: Input the processed data into the training network, output the training results, compare the training results with the true values, and output and save the optimal training parameters;

[0065] Step 4: Input the optimal training parameters into the training network to obtain the final weld network model;

[0066] Step 5: Collect the weld pictures to be detected. After performing feature point extraction and data processing, input them into the final weld network model to obtain the weld point coordinate information of the weld pictures to be detected;

[0067] Step 6: Output the obtained weld point coordinate information to the welding robot, and perform automated welding operations through the welding robot.

[0068] Preferably, the specific content of step 1 includes:

[0069] Connect the industrial camera to the terminal for communication by setting the IP address of the industrial camera, and collect a series of weld images through the industrial camera as training samples;

[0070] For each training sample, obtain the feature points of the weld through the Canny operator of the feature extraction algorithm. The feature points include weld points and non-weld points. Mark the weld points as positive samples and the non-weld points as negative samples.

[0071] Preferably, step 2 specifically includes: obtaining the coordinate information of the feature points. The data dimension of the feature points is N×2, where N represents N points, and the feature dimension of each point is 2, with the value being the coordinate value X(s, t) of the extracted feature point. s represents the abscissa value in the same coordinate system of each image, and t represents the ordinate value in the same coordinate system of each image. Normalize the position data of the obtained weld points.

[0072] Preferably, step 3 specifically includes:

[0073] Step 31: Pre-define the weld point label as 1 and the non-weld point label as 0. After corresponding the coordinates of the weld feature points in each image with their corresponding labels one by one, the label is in the format of N×1, representing the true labels of N points;

[0074] Step 32: Input the processed matrix of feature point data N×2 into the training network, and the training network is a Transformer neural network;

[0075] Step 33: Use the encoding part of the Transformer neural network to learn the feature information of the global points and encode each feature information;

[0076] Step 34: Input the encoding result, the obtained weld point labels and non-weld point labels into the decoding part of the Transformer for training together to obtain matrix decoding, weld point categories, and non-weld point categories. Then, through operations, obtain the purified weld information, and finally restore the weld information to the original data size and output the predicted category information; as Figure 2 shown, the weld points and non-weld points on the left refer to the labels constructed before training, that is, the true values, and the weld points and non-weld points on the right are the representations after being encoded by the decoding part of the Transformer;

[0077] Step 35: Compare the output predicted category information with the true values, calculate the error backpropagation, and update the network parameters until the network reaches the convergence state and save the optimal training parameters.

[0078] Preferably, step 33 specifically includes:

[0079] Step 331: Split the coordinate values X of the N feature points of each weld seam diagram into N small block vectors, and embed the X (s, t) in the horizontal and vertical dimensions of the N small block vectors into 128 dimensions, which is represented by a high-dimensional vector a, and the dimension of each a is 1×128. [a1, a2, …, a n represents the set of N input information converted into an N×128 matrix. Multiply a with three different vectors , , respectively to obtain the initial representations of the three vectors Q, K, and V:

[0080] ;

[0081] ;

[0082] ;

[0083] where represents the parameters learned by the three networks, and the dimension is equal to the transpose of a;

[0084] Step 332: Input Q, K, and V into the multi-head attention module. This module divides the network into multiple heads to form multiple subspaces, allowing the network to focus on information at different levels and obtain feature information at multiple levels.

[0085] Step 333: Input the feature information at multiple levels into the residual and layer normalization layers. The residual connection adds the input of the previous layer and the output of the previous layer, that is, x + Multi-Head(Q, K, V). Layer normalization normalizes each layer; where x refers to the weld point coordinate matrix after embedding.

[0086] Step 334: Finally, input it into the feed-forward network + residual and layer normalization. The feed-forward network is a simple two-layer linear mapping followed by an operation of an activation function, such as =Relu( ), refers to the input of the feed-forward network, , respectively refer to the weights of the two-layer linear mapping, refers to the output of the feed-forward network, and then it is processed by the residual and layer normalization layers to complete the network encoding and encode the information of each feature point.

[0087] Preferably, the specific operation of step 332 is as follows:

[0088] Input the obtained Q, K, and V into the linear layer, and the operation is as follows:

[0089] ;

[0090] ;

[0091] ;

[0092] i = 1, 2, …, 8;

[0093] Among them, , , represent the respective weight parameters of Q, K, and V in the i-th head, , , represent the output of the i-th head linear layer;

[0094] Enter the scaled dot-product attention layer and perform the following operations:

[0095] ) ;

[0096] ;

[0097] Softmax is a function for calculating scores here, is its calculation result; represents the output of the i-th head in the scaled dot-product attention layer;

[0098] Perform the calculations of the concatenation layer and the linear layer:

[0099] Multi-Head(Q, K, V) = Concat( ,..., ) ;

[0100] Concat means to merge and concatenate the results of h heads, represents the weight of the last linear layer of the multi-head attention, and Multi-Head(Q, K, V) is the output of the entire multiple attentions, that is, the multi-level feature information.

[0101] Preferably, the specific network modules in the Transformer decoding part are the same as those in the network encoding part. At the same time, the label information of the weld points and non-weld points is added to obtain the encoded information of the categories. The encoded information of the categories is multiplied by the decoding result of the Transformer decoding part to obtain N category information with a dimension of N×1, and the coordinate information of the weld points is obtained, and then the weld track of the weld image to be detected is obtained.

[0102] Preferably, the cross-entropy is used as the error function for network training. The probabilities predicted for each category in binary classification are defined as p and 1-p, and the expression of the error function is:

[0103] ;

[0104] where, represents the class label value of sample i, with the weld point being 1 and the non-weld point being 0;

[0105] represents the probability that sample i is predicted as an in-weld point, M represents the number of samples, and the base of Log is e.

[0106] Preferably, step 5 specifically includes:

[0107] Collect the weld images to be detected through an industrial camera, obtain the feature points of the weld through the Canny operator of the feature extraction algorithm for each weld image. The feature points include weld points and non-weld points, obtain the coordinate information of the feature points, and perform normalization processing on the position data of the obtained weld points.

[0108] Normalization means scaling the data to the range [0,1], and its formula is:

[0109]

[0110] is the original feature point data, with a dimension of N×2, is the data after normalization.

[0111] The above technical solutions of the present invention have at least the following advantages: By using the Transformer as the basic network and combining the Attention mechanism to learn the global information of the weld points, training the welding points through an algorithm and detecting the weld point trajectory, the recognition accuracy of the robot for the weld is improved. And by increasing the weight of the region of interest, assisting the network to more quickly locate the region of interest, further improving the operation speed, realizing highly parallelized training, so that the efficiency of weld extraction is greatly improved. In addition, the neural network extracts the position information of the weld and transmits its coordinate information to the robot to realize automatic welding of the weld. The present invention improves the overall efficiency of weld point extraction, meeting the real-time requirements of robot welding and relevant industrial standards.

[0112] Although the specific embodiments of the present invention have been described above, those skilled in the art of this technology should understand that the specific embodiments we described are illustrative rather than used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope protected by the claims of the present invention.

Claims

1. A method for extracting the position of weld points based on a Transformer neural network, characterized in that: It includes the following steps: Step 1: Collect weld images as training samples, and obtain the feature points of the weld through the Canny operator of the feature extraction algorithm. The feature points include weld points and non-weld points. Mark the weld points as positive samples and the non-weld points as negative samples; Step 2: Obtain the coordinate information of the feature points and perform normalization processing; Step 31: Pre-define the weld point label as 1 and the non-weld point label as 0. After corresponding the coordinates of the weld feature points in each image with their corresponding labels one by one, the label is in the format of N×1, representing the true labels of N points; Step 32: Input the processed feature point data matrix of N×2 into the training network, and the training network is a Transformer neural network; Step 33: Use the encoding part of the Transformer neural network to learn the feature information of the global points and encode each feature information; Specifically, it includes: Step 331: Split the coordinate values X of the N feature points in each weld seam diagram into N small block vectors, and embed the X(s, t) in the horizontal and vertical dimensions of the N small block vectors into 128 dimensions, which is represented by the high-dimensional vector a, and the dimension of each a is 1×128. [a1, a2, …, a n represents the set of N input information converted into an N×128 matrix, and multiply a by three different vectors , , respectively to obtain three vectors Q, K, and V. Among them, represents the parameters of three network learning, and the dimension is equal to the transpose of a; Step 332: Input Q, K, and V into the multi-head attention module to obtain feature information at multiple levels; Step 333: Input the feature information at multiple levels into the residual and layer normalization layer. The residual connection adds the input of the previous layer and the output of the previous layer, and layer normalization normalizes each layer; Step 334: Finally, input it into the feed-forward network. The feed-forward network is a simple two-layer linear mapping followed by the operation of an activation function, and then processed by the residual and layer normalization layer to complete the network encoding and encode each feature point information; Step 34: Input the encoding result, the obtained weld point labels, and non-weld point labels into the decoding part of the Transformer for joint training to obtain matrix decoding, weld point categories, and non-weld point categories. Then, through operations, the purified weld information is obtained. Finally, the weld information is restored to the original data size and the predicted category information is output; Step 35: Compare the output predicted category information with the true value, calculate the error backpropagation, and update the network parameters until the network reaches the convergence state, and then save the optimal training parameters; Step 4: Input the optimal training parameters into the training network to obtain the final weld network model; Step 5: Collect the weld images to be detected, perform feature point extraction and data processing, and then input them into the final weld network model to obtain the weld point coordinate information of the weld images to be detected; Step 6: Output the obtained weld point coordinate information to the welding robot, and the welding robot performs automated welding operations.

2. The method according to claim 1, wherein: The specific content of Step 1 includes: Connect the industrial camera to the terminal for communication through the IP address of the industrial camera, and collect a series of weld images as training samples through the industrial camera.

3. The method according to claim 1, wherein: The specific content of Step 2 includes: Obtain the coordinate information of the feature points. The dimension of the feature point data is N×2, where N represents N points, and the feature dimension of each point is 2, and the value is the coordinate value X(s, t) of the extracted feature point. s represents the abscissa value in the same coordinate system of each image, and t represents the ordinate value in the same coordinate system of each image. Perform normalization processing on the obtained position data of the weld points.

4. The method according to claim 1, characterized in that: The specific content of Step 332 is: The obtained Q, K, and V are input into the linear layer, and the operations are as follows: ； ； ； i=1,2，…，8； Among them, , , represent the respective weight parameters of Q, K, and V in the i-th head, , , represent the output of the i-th head linear layer; Enter the scaled dot-product attention layer, and the operations are as follows: ) ； ； Softmax is a function that calculates scores here, which is the result of its calculation; represents the output of the i-th head in the scaled dot-product attention layer; Perform the calculations of the concatenation layer and the linear layer: Multi-Head(Q,K,V)=Concat( ,..., ) ; Concat means to combine and splice the results of h heads. It represents the weight of the last linear layer of multi-head attention. Multi-Head(Q, K, V) is the output of the entire multiple attentions, that is, multi-level feature information.

5. The method according to claim 1, wherein: The specific network modules in the Transformer decoding part are the same as those in the network encoding part. At the same time, the label information of the weld points and non-weld points is added to obtain the encoded information of the categories. The encoded information of the categories is dot-multiplied with the decoding result of the Transformer decoding part to obtain N category information with a dimension of N×1, and the coordinate information of the weld points is obtained, and then the weld track of the weld image to be detected is obtained.

6. The method according to claim 1, characterized in that: Cross-entropy is used as the error function for network training. Define the probabilities predicted for each category under binary classification as p and 1-p. The expression of the error function is: ； Among them, represents the class label value of sample i, where the weld point is 1 and the non-weld point is 0; It represents the probability that sample i is predicted as an in-weld point, M represents the number of samples, and the base of Log is e.

7. The method according to claim 1, characterized in that: The specific content of step 5 includes: Collect the weld images to be detected through an industrial camera. For each weld image, obtain the feature points of the weld through the Canny operator of the feature extraction algorithm. The feature points include weld points and non-weld points. Obtain the coordinate information of the feature points, and normalize the position data of the obtained weld points.

Images