Material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network

Abstract

The invention provides a material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network, and the method comprises the steps: S1, capturing a point cloud of a single structure light segment of laser light in a camera view field through a binocular structure light depth vision camera, and calculating the coordinates of the point cloud through a spatial point cloud collection method; s2, preprocessing coordinate data of the spatial point cloud; s3, creating a deep learning neural network model; s4, training the deep learning neural network model to obtain a deep neural network used for extracting the three-dimensional welding seam point cloud; and S5, inputting point cloud data scanned and recorded by the binocular structured light depth vision camera by using the deep neural network for extracting the three-dimensional welding seam point cloud, and judging whether the data is a welding seam area or not. The efficient and intelligent welding seam feature recognition method for recognizing and selecting the welding seam based on the three-dimensional point cloud convolutional deep neural network provides a new thought and practice basis for intelligent and automatic technology development of welding.

Description

Technical Field

[0001] The present invention belongs to the technical field of material processing, and in particular relates to a material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network. Background Art

[0002] The recognition of the surface features of materials is an important basis for material processing, sorting, and post-processing. For example, in the field of visual welding, the welding process is a process of bonding two or more adjacent metal parts into a whole by means of laser or arc heating. Welding technology has been widely used in various fields of vehicles, ships, and industrial manufacturing. The welding process relies on experienced welders to perform welding operations, and the dust and ultraviolet radiation during the welding process can easily damage the welder. In recent years, researchers have developed a robot automatic welding system based on two-dimensional vision, which uses industrial cameras to collect the morphology of the workpiece, analyze the welding trajectory, and guide the industrial robot to perform welding operations. Although robots using two-dimensional vision can solve the welding process of visual tracking and trajectory planning of single-pass welds, two-dimensional vision is difficult to handle for grooves or multi-layer multi-pass welds that require base welding. Therefore, it is urgent to develop a machine vision welding process that can realize multi-layer multi-pass welding processes.

[0003] In recent years, with the development of 3D vision technologies such as structured light and binocular vision, more and more 3D vision technologies have been applied to welding processes, which has led to the rapid development of visual welding technology. 3D vision uses structured light or binocular cameras to scan the welding area to obtain the spatial point cloud of the welding area. Through point cloud screening algorithms or template matching and other technologies, combined with manual screening methods, the position and size of the weld are obtained to guide the welding operation. Although the current 3D vision technology can solve the problem of multi-layer and multi-pass weld recognition to a certain extent, it still strongly relies on manual intervention during the welding implementation process, so the degree of automation of welding equipment still has a lot of room for improvement.

[0004] In summary, in order to address the technical difficulties in identifying three-dimensional point clouds in welding areas, it is very meaningful to propose a material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network to achieve rapid recognition of welds.

[0006] The technical solution of the present invention is: a material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network, comprising:

[0007] S1: Use a line laser to project a laser line segment onto the surface of the welding workpiece, capture the point cloud of a single structured light segment of the laser light in the camera field of view through the left and right cameras of the binocular structured light depth vision camera, and use a spatial point cloud acquisition method to calculate the coordinates of the point cloud;

[0008] S2: preprocessing the coordinate data of the spatial point cloud, and convolving the spatial point cloud using a three-dimensional convolution kernel;

[0009] S3: Create a deep learning neural network model, and use the deep learning neural network model to build a regression analysis model of the weld point cloud;

[0010] S4: creating a deep learning training set and a label set, training a deep learning neural network model, and obtaining a deep neural network for extracting a three-dimensional weld point cloud;

[0011] S5: Using the deep neural network for extracting three-dimensional weld point cloud, input the point cloud data scanned and recorded by the binocular structured light depth vision camera to determine whether the data is a weld area.

[0012] Preferably, the coordinates of the spatial point cloud in S1 are calculated according to the following formula:

[0013] X=((xC x )×Z) / F x

[0014] Y=((yC y )×Z) / F x

[0015] Z=(F x ×Cd) / d

[0016] Where: X, Y, Z are the spatial coordinates of the point cloud; F x is the focal length of the camera; Cd is the distance between the center lines of the two camera lenses; d is the pixel distance between the left and right cameras; C x , C y is the camera constant.

[0017] Preferably, the convolution of the spatial point cloud using a three-dimensional convolution kernel in S2 includes: compressing the spatial point cloud data along the Y direction, that is, setting the Y coordinates of all point clouds to 0, and obtaining point cloud data consisting of only X and Z coordinates; scaling the coordinates of the compressed point cloud data to 1000mm×500mm; creating a voxelized matrix to load the point cloud data, using a 5×5 mean convolution kernel and a moving value of a step size of 5mm to perform a convolution operation on the point cloud of the voxelized matrix to obtain a 200×100 point cloud.

[0018] Preferably, the creating of a deep learning neural network in S3 comprises: creating a deep neural network with an input layer of 20,000 nodes, a first hidden layer of 1,000 nodes, a second hidden layer of 100 nodes, a third hidden layer of 50 nodes, and an output layer of 3 nodes;

[0019] The cross entropy loss function is used to calculate the expected value error of the neural network, and the weights of the neural network are corrected by feedback. The cross entropy loss function is defined as follows:

[0020]

[0021]

[0022] Where: y i The true label of the i-th category; p i is the probability of each label; L is the loss value.

[0023] Preferably, a deep learning training set and a label set are created in S4, including: a deep neural network training data set constructed by creating a V-shaped point cloud and arc-shaped and rectangular space point clouds, and the value range of each point cloud x is: 0-1000, and the value range of z is 0-500.

[0024] Preferably, the V-shaped point cloud:

[0025]

[0026] The arc point cloud:

[0027]

[0028] The rectangular space point cloud:

[0029]

[0030] Where: k is the coefficient; R is the radius of the circle; x and y are the coordinates of the point cloud.

[0031] Preferably, the training of the deep learning neural network model in S4 to obtain a deep neural network for extracting a three-dimensional weld point cloud comprises:

[0032] Neural network forward propagation: propagation from the input layer to the first hidden layer, assuming that the input x of the input layer is an input vector with 20,000 features. The weight matrix W of the first layer 1 is a 1000×20000 matrix, the bias term b1 is a vector of length 1000, and the activation function of the first layer is represented by f;

[0033] Weighted sum calculation:

[0034]

[0035] Activation function calculation:

[0036] a 1 =f(z 1 )

[0037] Where f is the activation function;

[0038] The propagation from the first hidden layer to the second hidden layer, the weight matrix W of the second layer 2 is a 100×1000 matrix, the bias term b 2 is a vector of length 100; the activation function of the second layer is also f;

[0039] Weighted sum calculation:

[0040] z 2 =w 2 a 1 +b 2

[0041] Activation function calculation:

[0042] a 2 =f(z 2 )

[0043] Propagation from the second hidden layer to the third hidden layer, the weight matrix W of the third layer 3 is a 50×100 matrix, the bias term b 3 is a vector of length 50; the activation function of the third layer is still f;

[0044] Weighted sum calculation:

[0045] z 3 =w 3 a 2 +b 3

[0046] Activation function calculation:

[0047] a 3 =f(z 3 )

[0048] The propagation from the third hidden layer to the output layer, the weight matrix W of the output layer 4 is a 3×50 matrix, the bias term b 4 is a vector of length 3;

[0049] Weighted sum calculation:

[0050] z 4 =w 4 a 3 +b 4

[0051] Activation function calculation:

[0052] y=Softmax(z 4 )

[0053] Among them, the Softmax function is defined as:

[0054]

[0055] Backpropagation of Neural Network:

[0056] Calculate the error of the output layer, the cross entropy loss function L, which is defined as follows:

[0057]

[0058] Among them, yi is the true label, y^i is the predicted value output by the model;

[0059] The error of the output layer is the gradient of the loss function with respect to the activation value z4 of the output layer:

[0060]

[0061] Among them, y^ is the predicted value of the output layer, and y is the actual label:

[0062] Update the weights and biases of the output layer and calculate the gradients of the weights and biases according to the gradient descent method:

[0063]

[0064]

[0065] Update weights and biases:

[0066]

[0067]

[0068] Where η is the learning rate:

[0069] Calculate the error of the third layer and backpropagate the error from the output layer to the third layer:

[0070]

[0071] f'(z 3 ) is the derivative of the activation function f in the third layer:

[0072] Update the weights and biases of the third layer, and calculate the gradients of the weights and biases of the third layer:

[0073]

[0074]

[0075] Update weights and biases:

[0076]

[0077]

[0078] Calculate the error of the second layer and backpropagate the error from the third layer to the second layer:

[0079]

[0080] Compute the gradients of the weights and biases of the second layer:

[0081]

[0082]

[0083] Update weights and biases:

[0084]

[0085]

[0086] Calculate the error of the first layer and backpropagate the error from the second layer to the first layer:

[0087]

[0088] Compute the gradients of the weights and biases of the first layer:

[0089]

[0090]

[0091] Update weights and biases

[0092]

[0093]

[0094] Through forward propagation, the activation value of each layer is obtained; through back propagation, the error of each layer is calculated, and the weights and biases are updated according to the error.

[0095] Preferably, S5: using the deep neural network for extracting three-dimensional weld point cloud, inputting the point cloud data scanned and recorded by the binocular structured light depth vision camera, and determining whether the data is a weld area:

[0096] Select one of the weld point clouds collected by the binocular structured light depth vision camera in turn, take the actual weld length of the welded workpiece as the weld search width (w), and the moving step length is (s);

[0097] Start searching from the zero point of the point cloud curve and take the data with serial numbers from 0 to w;

[0098] Compress the spatial point cloud data along the Y direction, that is, set the y value of all points on the point cloud curve to 0, and obtain point cloud data consisting of only X and Z coordinates;

[0099] The coordinates of the compressed point cloud data are scaled to 1000x 500mm, and the point cloud data is voxelized and passed into the trained point cloud convolutional neural network to determine whether the predicted value of the area is greater than the threshold. If the area is greater than the threshold, the area is the weld area, and the point cloud of the area is saved in a point cloud array; the threshold is set to a value between 0.6-1;

[0100] By traversing the point clouds of all structured light segments in sequence, the three-dimensional point cloud of the weld can be obtained.

[0101] The present invention provides a material surface feature recognition method based on a three-dimensional point cloud convolution deep neural network. The method uses a point cloud convolution kernel to convolve the spatial point cloud of the welding area, and uses a deep neural network to construct a regression analysis model of the weld point cloud. By training the deep neural network model, a deep neural network for extracting three-dimensional weld point clouds is finally obtained. The point cloud convolution deep neural network is used to extract V-shaped, circular arc, and rectangular structural features on the material surface. For example, in the welding field, it can be applied to the rapid recognition and modeling of V-shaped groove welds. It has the advantages of high recognition accuracy and fast recognition speed, and has broad market application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0103] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0104] Figure 1 A schematic diagram of obtaining a spatial point cloud provided by the present invention;

[0105] Figure 2 The collected 3D point cloud of the welding area. DETAILED DESCRIPTION

[0106] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0107] In order to solve the technical problem that the three-dimensional point cloud of the welding area is difficult to identify, the present invention constructs a deep learning neural network algorithm based on the three-dimensional point cloud convolution, uses the three-dimensional convolution kernel to convolve the spatial point cloud, and uses the deep neural network to construct a regression analysis model of the weld point cloud. By training the deep neural network model, the deep neural network is finally used to extract the three-dimensional weld point cloud, thereby realizing the rapid identification and modeling of the weld.

[0108] A material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network is characterized by comprising:

[0109] S1: Use a line laser to project a laser line segment onto the surface of the welding workpiece, capture the point cloud of a single structured light segment of the laser light in the camera field of view through the left and right cameras of the binocular structured light depth vision camera, and use a spatial point cloud acquisition method to calculate the coordinates of the point cloud;

[0110] The structured light depth vision camera and spatial point cloud acquisition method are used to project the laser line segment onto the surface of the welding workpiece through the line laser;

[0111] The pixel coordinates of the laser light in the camera field of view are captured by the left and right cameras. The coordinates of the spatial point cloud are calculated according to the following formula.

[0112] X=((xC x )×Z) / F x

[0113] Y=((yC y )×Z) / F x

[0114] Z=(F x ×Cd) / d

[0115] Where: X, Y, Z are the spatial coordinates of the point cloud; Fx is the focal length of the camera; Cd is the distance between the center lines of the two camera lenses; d is the pixel distance between the left and right cameras; Cx and Cy are camera constants.

[0116] S2: preprocessing the coordinate data of the spatial point cloud, and convolving the spatial point cloud using a three-dimensional convolution kernel;

[0117] The spatial point cloud data is compressed along the Y direction, that is, the Y coordinates of all point clouds are set to 0, and point cloud data consisting of only X and Z coordinates are obtained; the coordinates of the compressed point cloud data are scaled to 1000mmx500mm; a voxelized matrix is ​​created to load the point cloud data, and a 5x5 mean convolution kernel is used with a moving value of 5mm in step size to perform convolution operation on the point cloud of the voxelized matrix to obtain a 200X100 point cloud.

[0118] Specifically, the point cloud is voxelized, and the area of ​​1000mm×500mm is evenly divided into small areas of 1mm×1mm. A voxelization matrix Point_Cloud

[1000]

[500] is created, and each element of the array is assigned a value of 0. The value in the brackets represents the coordinate of the voxel. For example: the voxelization matrix Point_Cloud

[100]

[50] , 100 represents the x coordinate in the range of 95-100, 50 represents the z coordinate in the range of 45-50. If there is a point cloud in this range, the array element is assigned a value of 1, otherwise it is assigned a value of 0. If the point cloud coordinates fall in the corresponding range after traversing the weld point cloud, the corresponding element of the voxelization matrix is ​​assigned a value of 1; (3) A 5×5 mean convolution kernel is used to perform a convolution operation on the point cloud with a moving value of a step size of 5mm to obtain a 200×100mm point cloud;

[0119] The method of scaling the weld area point cloud to 1000mmx500mm is as follows: traverse all point cloud data, obtain the minimum x coordinate (Min_x), maximum x coordinate (Max_x), minimum z coordinate (Min_z), and maximum z coordinate (Max_z) in all point clouds. Subtract Min_x from the x of all point clouds and multiply by 1000 / (Max_x-Min_x), and subtract Min_z from the z of all point clouds and multiply by 500 / (Max_z-Min_z) to obtain the new point cloud coordinates.

[0120] Use a 5x5 mean convolution kernel to perform convolution operations on the point cloud with a moving value of 5mm in step size. The 5x5 mean convolution kernel is expressed in the form of a matrix: K[5][5] = {{1,1,1,1,1}, {1,1,1,1,1}, {1,1,1,1,1}, {1,1,1,1,1}, {1,1,1,1,1}} In the above formula, each curly bracket represents a row of elements, and there are 5 rows in total. The convolution kernel slides during convolution: The convolution kernel K slides on the x and z of the input matrix. In order to perform the convolution operation, the convolution kernel will slide one position in the input matrix each time, and the step size of the convolution kernel sliding is 5 (the step size range is 1 to 5, preferably 5). Each time the convolution kernel is moved, a 5x5 area is covered. For example, if the current coordinates are x and z, the x coordinate range selected by the convolution kernel is: x to x+5; the selected z coordinate range is: z to z+5. Count the number Num and the average value Mean of all points falling into the convolution kernel. The convolution process is calculated according to the following formula:

[0121]

[0122] Create a convolution point cloud array Conv_Point_Cloud

[200]

[100] , and put the mean obtained by this convolution into the convolution point cloud array Conv_Point_Cloud[Num_x][Num_z]. Num_x is equal to x divided by the step size and rounded, and Num_z is equal to z divided by the step size and rounded.

[0123] S3: Create a deep learning neural network model, and use the deep learning neural network model to build a regression analysis model of the weld point cloud;

[0124] The deep learning neural network architecture used creates a deep neural network with an input layer of 20,000 nodes, a first hidden layer of 1,000 nodes, a second hidden layer of 100 nodes, a third hidden layer of 50 nodes, and an output layer of 3 nodes. The number of nodes in the input layer, hidden layer, and output layer can be adjusted according to the data sample, but the number of nodes from the input layer to the output layer should show a decreasing trend.

[0125] The cross entropy loss function is used to calculate the expected value error of the neural network and feedback is used to correct the weights of the neural network. The cross entropy loss function is defined as follows:

[0126]

[0127]

[0128] Where: y i The true label of the i-th category; p i is the probability of each label; L is the loss value;

[0129] S4: creating a deep learning training set and a label set, training a deep learning neural network model, and obtaining a deep neural network for extracting a three-dimensional weld point cloud;

[0130] The deep neural network training data set is constructed by creating V-shaped point clouds and arc-shaped and rectangular space point clouds. The value range of x of each point cloud is: 0-1000, and the value range of z is 0-500;

[0131] Specifically, in order to train the deep accelerated point cloud convolutional deep neural network, we designed a series of spatial point cloud data by writing a program, including V-shaped point clouds similar to the shape of welds, as well as arc-shaped and rectangular spatial point clouds as comparison data, and trained the neural network together. We used the comparison data as interference items for V-shaped point cloud recognition to improve the accuracy of V-shaped point cloud recognition.

[0132] The functions of the three point clouds are:

[0133] (1) V-shaped point cloud

[0134]

[0135] (2) Arc point cloud

[0136]

[0137] (3) Rectangular space point cloud

[0138]

[0139] Where k is the coefficient; R is the radius of the circle; x and y are the coordinates of the point cloud.

[0140] The neural network constructed by the present invention is a deep neural network consisting of an input layer with 20,000 nodes, a first hidden layer with 1,000 nodes, a second hidden layer with 100 nodes, a third hidden layer with 50 nodes, and an output layer with 3 nodes. The training of the neural network consists of forward propagation and back propagation of the neural network.

[0141] The created spatial point cloud construction deep neural network training data set is input into the point cloud convolutional deep neural network, and the weights and bias values ​​of the model are corrected by using the forward propagation and back propagation of the neural network to obtain the point cloud convolutional deep neural network model.

[0142] The forward propagation method of the neural network is as follows: assuming that the input x of the input layer is an input vector with 20,000 features, the neural network is forward propagated from the input layer to the first hidden layer. The weight matrix W of the first layer 1 is a 1000×20000 matrix, and the bias term b1 is a vector of length 1000. The activation function of the first layer is denoted by f.

[0143] The specific method is as follows:

[0144] (1) Neural Network Forward Propagation

[0145] Propagation from the input layer to the first hidden layer, assuming that the input x of the input layer is an input vector with 20,000 features. The weight matrix W of the first layer 1 is a 1000×20000 matrix, and the bias term b1 is a vector of length 1000. The activation function of the first layer is denoted by f.

[0146] Weighted sum calculation:

[0147] z 1 =w 1 x+b 1

[0148] Activation function calculation:

[0149] a 1 =f(z 1 )

[0150] Among them, f is the activation function.

[0151] The propagation from the first hidden layer to the second hidden layer, the weight matrix W of the second layer 2 is a 100×1000 matrix, the bias term b 2 is a vector of length 100. The activation function of the second layer is also f.

[0152] Weighted sum calculation:

[0153] z 2 =w 2 a 1 +b 2

[0154] Activation function calculation:

[0155] a 2 =f(z 2 )

[0156] Propagation from the second hidden layer to the third hidden layer, the weight matrix W of the third layer 3 is a 50×100 matrix, the bias term b 3 is a vector of length 50. The activation function of the third layer is still f.

[0157] Weighted sum calculation:

[0158] z 3 =w 3 a 2 +b 3

[0159] Activation function calculation:

[0160] a 3 =f(z 3 )

[0161] The propagation from the third hidden layer to the output layer, the weight matrix W of the output layer 4 is a 3×50 matrix, the bias term b 4 is a vector of length 3.

[0162] Weighted sum calculation:

[0163] z 4 =w 4 a 3 +b 4

[0164] Activation function calculation:

[0165] y=Sofmax(z 4 )

[0166] Among them, the Softmax function is defined as:

[0167]

[0168] (2) Backpropagation of Neural Networks

[0169] Calculate the error of the output layer, the cross entropy loss function L, which is defined as follows:

[0170]

[0171] Among them, y i is the true label, y^ i is the predicted value output by the model.

[0172] The error of the output layer is the loss function on the activation value z of the output layer 4 The gradient is:

[0173]

[0174] Among them, y^ is the predicted value of the output layer, and y is the actual label.

[0175] Update the weights and biases of the output layer and calculate the gradients of the weights and biases according to the gradient descent method:

[0176]

[0177]

[0178] Update weights and biases:

[0179]

[0180]

[0181] Where η is the learning rate.

[0182] Calculate the error of the third layer and backpropagate the error from the output layer to the third layer:

[0183]

[0184] Here, f'(z 3 ) is the derivative of the activation function f at the third layer.

[0185] Update the weights and biases of the third layer, and calculate the gradients of the weights and biases of the third layer:

[0186]

[0187]

[0188] Update weights and biases:

[0189]

[0190]

[0191] Calculate the error of the second layer and backpropagate the error from the third layer to the second layer:

[0192]

[0193] Compute the gradients of the weights and biases of the second layer:

[0194]

[0195]

[0196] Update weights and biases:

[0197]

[0198]

[0199] Calculate the error of the first layer and backpropagate the error from the second layer to the first layer:

[0200]

[0201] Compute the gradients of the weights and biases of the first layer:

[0202]

[0203]

[0204] Update weights and biases

[0205]

[0206]

[0207] Through forward propagation, we get the activation values ​​of each layer; through backpropagation, we calculate the errors of each layer and update the weights and biases based on these errors.

[0208] S5: Using the deep neural network for extracting three-dimensional weld point cloud, input the point cloud data scanned and recorded by the binocular structured light depth vision camera to determine whether the data is a weld area.

[0209] The binocular structured light depth camera is placed at the end of a six-degree-of-freedom robotic arm. The robotic arm drives the camera and line laser to scan along the length of the groove weld on the workpiece surface and record the point cloud to obtain a three-dimensional point cloud of the welding area. For the scanning of the groove weld, a V-shaped continuous area can be obtained, which is the weld area. The weld area is composed of point clouds of multiple structured light segments.

[0210] In the weld area point cloud, select the point cloud of a single structured light segment from bottom to top, and perform point cloud convolution deep neural network analysis. Take the actual weld length of the welded workpiece as the weld search width (w), and the moving step length is (s). Start the search from the zero point of the point cloud curve, and take the data with sequence numbers 0 to w. First, compress the spatial point cloud data along the Y direction, that is, the y value of all points on the point cloud curve is equal to 0, and obtain the point cloud data consisting of only X and Z coordinates. Scale the coordinates of the compressed point cloud data to 1000x 500mm, and voxelize the point cloud data, and pass it into the point cloud convolution neural network created by the present invention to determine whether the predicted value of the area is greater than the threshold (the threshold is set to a value between 0.6-1). If the area is greater than the threshold, the area is the weld area, and the point cloud of the area is saved in a point cloud array. After traversing all the point clouds of the structured light segments in turn, the three-dimensional point cloud of the weld can be obtained. Figure 1 and 2 As shown, Figure 1 , 2 Center binocular camera-1; welding area-2; weld-3; left camera field of view-4; right camera field of view-5; weld area point cloud-6; single weld point cloud-7;

[0211] In summary, the present invention constructs a deep learning neural network algorithm based on three-dimensional point cloud convolution, uses three-dimensional convolution kernel to convolve the spatial point cloud, and uses a deep neural network to construct a regression analysis model of the weld point cloud. By training the deep neural network model, the deep neural network is finally used to extract the three-dimensional weld point cloud, thereby realizing the rapid identification and modeling of welds.

[0212] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A material surface feature recognition method based on a three-dimensional point cloud convolutional deep neural network, characterized in that: include: S1: Use a line laser to project a laser line segment onto the surface of the welding workpiece, capture the point cloud of a single structured light segment of the laser light in the camera field of view through the left and right cameras of the binocular structured light depth vision camera, and use a spatial point cloud acquisition method to calculate the coordinates of the point cloud; S2: preprocessing the coordinate data of the spatial point cloud, and convolving the spatial point cloud using a three-dimensional convolution kernel; S3: Create a deep learning neural network model, and use the deep learning neural network model to build a regression analysis model of the weld point cloud; S4: creating a deep learning training set and a label set, training a deep learning neural network model, and obtaining a deep neural network for extracting a three-dimensional weld point cloud; S5: Using the deep neural network for extracting three-dimensional weld point cloud, input the point cloud data scanned and recorded by the binocular structured light depth vision camera to determine whether the data is a weld area.

2. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: The coordinates of the spatial point cloud in S1 are calculated according to the following formula: X=((x-C x )×Z) / F x Y=((y-C y )×Z) / F x Z=(F x ×Cd) / d Where: X, Y, Z are the spatial coordinates of the point cloud; F x is the focal length of the camera; Cd is the distance between the center lines of the two camera lenses; d is the pixel distance between the left and right cameras; C x , C y is the camera constant.

3. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: The method described in S2 uses a three-dimensional convolution kernel to convolve the spatial point cloud, including: compressing the spatial point cloud data along the Y direction, that is, setting the Y coordinates of all point clouds to 0, and obtaining point cloud data consisting of only X and Z coordinates; scaling the coordinates of the compressed point cloud data to 1000mm×500mm; creating a voxelized matrix to load the point cloud data, using a 5×5 mean convolution kernel and a moving value of a step size of 5mm to convolve the point cloud of the voxelized matrix to obtain a 200×100 point cloud.

4. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: Creating a deep learning neural network as described in S3 includes: creating a deep neural network with an input layer of 20,000 nodes, a first hidden layer of 1,000 nodes, a second hidden layer of 100 nodes, a third hidden layer of 50 nodes, and an output layer of 3 nodes; The cross entropy loss function is used to calculate the expected value error of the neural network, and the weights of the neural network are corrected by feedback. The cross entropy loss function is defined as follows: Where: y i The true label of the i-th category; p i is the probability of each label; L is the loss value.

5. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: A deep learning training set and label set are created in S4, including the V-shaped point cloud and arc-shaped and rectangular space point clouds created to construct a deep neural network training data set. The value range of x for each point cloud is 0-1000, and the value range of z is 0-500.

6. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 5 is characterized in that: The V-shaped point cloud: The arc point cloud: The rectangular space point cloud: Where: k is the coefficient; R is the radius of the circle; x and y are the coordinates of the point cloud.

7. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: The deep learning neural network model is trained as described in S4 to obtain a deep neural network for extracting a three-dimensional weld point cloud, including: Neural network forward propagation: propagation from the input layer to the first hidden layer, assuming that the input x of the input layer is an input vector of 20,000 features. The weight matrix W1 of the first layer is a 1000×20,000 matrix, the bias term b1 is a vector of length 1000, and the activation function of the first layer is represented by f; Weighted sum calculation: z1=w1x+b1 Activation function calculation: a1=f(z1) Where f is the activation function; The propagation from the first hidden layer to the second hidden layer, the weight matrix W2 of the second layer is a 100×1000 matrix, the bias term b2 is a vector of length 100; the activation function of the second layer is also f; Weighted sum calculation: z2=w2a1+b2 Activation function calculation: a2=f(z2) The propagation from the second hidden layer to the third hidden layer, the weight matrix W3 of the third layer is a 50×100 matrix, and the bias term b3 is a vector of length 50; the activation function of the third layer is still f; Weighted sum calculation: z3=w3a3+b3 Activation function calculation: a3=f(z3) The propagation from the third hidden layer to the output layer, the weight matrix W4 of the output layer is a 3×50 matrix, and the bias term b4 is a vector of length 3; Weighted sum calculation: z4=w4a3+b4 Activation function calculation: y=Softmax(z4) Among them, the Softmax function is defined as: Backpropagation of Neural Network: Calculate the error of the output layer, the cross entropy loss function L, which is defined as follows: Among them, yi is the true label, y^i is the predicted value output by the model; The error of the output layer is the gradient of the loss function with respect to the activation value z4 of the output layer: Among them, y^ is the predicted value of the output layer, and y is the actual label: Update the weights and biases of the output layer and calculate the gradients of the weights and biases according to the gradient descent method: Update weights and biases: Where η is the learning rate: Calculate the error of the third layer and backpropagate the error from the output layer to the third layer: f'(z3) is the derivative of the activation function f at the third layer: Update the weights and biases of the third layer, and calculate the gradients of the weights and biases of the third layer: Update weights and biases: Calculate the error of the second layer and backpropagate the error from the third layer to the second layer: Compute the gradients of the weights and biases of the second layer: Update weights and biases: Calculate the error of the first layer and backpropagate the error from the second layer to the first layer: Compute the gradients of the weights and biases of the first layer: Update weights and biases Through forward propagation, the activation value of each layer is obtained; through back propagation, the error of each layer is calculated, and the weights and biases are updated according to the error.

8. The material surface feature recognition method based on three-dimensional point cloud convolutional deep neural network according to claim 1 is characterized in that: S5: Using the deep neural network for extracting three-dimensional weld point cloud, inputting the point cloud data scanned and recorded by the binocular structured light depth vision camera, and determining whether the data is a weld area: Select one of the weld point clouds collected by the binocular structured light depth vision camera in turn, take the actual weld length of the welded workpiece as the weld search width (w), and the moving step length is (s); Start searching from the zero point of the point cloud curve and take the data with serial numbers from 0 to w; Compress the spatial point cloud data along the Y direction, that is, set the y value of all points on the point cloud curve to 0, and obtain point cloud data consisting of only X and Z coordinates; The coordinates of the compressed point cloud data are scaled to 1000x 500mm, and the point cloud data is voxelized and passed into the trained point cloud convolutional neural network to determine whether the predicted value of the area is greater than the threshold. If the area is greater than the threshold, the area is the weld area, and the point cloud of the area is saved in a point cloud array; the threshold is set to a value between 0.6-1; By traversing the point clouds of all structured light segments in sequence, the three-dimensional point cloud of the weld can be obtained.

Images