A binocular narrow butt weld detection method based on deep learning

Abstract

The present invention relates to a binocular narrow butt weld detection method based on deep learning, comprising the following steps: projecting stripe structured light and passive light by a projector, collecting weld images, and obtaining the point cloud of the welded part; performing two-dimensional data annotation based on image binarization processing and binocular consistency correction; constructing a two-dimensional weld extraction model based on spatial information mining to extract two-dimensional welds; mapping two-dimensional pixels into three-dimensional space coordinates based on a binocular vision model. Pose estimation is performed based on the point cloud of the welded part and the local neighborhood feature information of the weld points. The binocular narrow butt weld detection system of the present invention realizes accurate and efficient weld extraction; specifically, it is an accurate and reliable automatic data annotation method; specifically, it is a weld extraction network that realizes the extraction of two-dimensional coordinates of the weld to assist three-dimensional weld positioning. The present invention provides a necessary strategy for realizing more accurate and efficient narrow butt weld detection.

Description

Technical Field

[0001] The present invention relates to a binocular narrow butt weld detection method based on deep learning, belonging to the technical field of image processing. Background Art

[0002] Laser welding is a non-contact welding method, which has the characteristics of deep penetration, high precision, and small heat affected zone of the welded joint, enabling the loss and deformation of the welded parts to be minimized. In recent years, with the development of industrial lasers and the in-depth research of scientific researchers on welding processes, laser welding has been applied in fields such as the automotive industry, shipbuilding industry, and aerospace industry. However, with the wide application of laser welding technology, the detection of narrow butt welds in high-precision welded parts has become an urgent problem to be solved. High-precision welded parts are generally closely butted together, and the weld features are not obvious, making it impossible to detect them through common weld detection means, which seriously affects the efficiency of laser welding. Summary of the Invention

[0003] The purpose of the present invention is to provide a binocular narrow butt weld detection method based on deep learning.

[0004] To solve the above technical problems, the present invention provides a binocular narrow butt weld detection method based on deep learning, and its specific technical solutions are as follows:

[0005] A method for detecting narrow butt welds, comprising the following steps:

[0006] Step 1: Based on the projector projecting stripe structured light and passive light, collect weld images and obtain the point cloud of the welded part;

[0007] Step 2: Perform two-dimensional data annotation based on image binarization processing and binocular consistency correction;

[0008] Step 3: Construct a two-dimensional weld extraction model based on spatial information mining to extract two-dimensional welds;

[0009] Step 4: Map the weld two-dimensional pixels to three-dimensional space coordinates based on the binocular vision model;

[0010] Step 5: Perform pose estimation based on the point cloud of the welded part and the local neighborhood feature information of the weld points.

[0011] Further, in Step 1, a projector in the stripe coding sensing system module projects a group of passive light and stripe structured light onto the welded part, triggering the camera to collect the corresponding passive light image and stripe structured light image. The light band projected by the projector is 450 nm, the incident angle between the projection center and the welded part is 60°, and at the same time, the optical axes of the binocular cameras are placed on both sides of the weld at 60°.

[0012] Phase shifting profilometry (PSP) and phase unwrapping algorithm are used to analyze the modulated phase to obtain an unambiguous phase, and finally the point cloud information of the measured target is calculated through the system calibration data.

[0013] Further, in step two, adaptive gray threshold is used for binary image processing. The relationship between the gray threshold of a pixel and the gray values of its neighborhood is shown in formula (1).

[0014]

[0015] where i and j are the row and column of the image respectively; f(i, j) is the gray value of the pixel at the i-th row and j-th column; K is the size of the calculation neighborhood; C is an empirical constant; m and n are the row and column positions of the image; T is the calculated threshold.

[0016] Further, in step two, a binocular consistency correction algorithm based on phase transformation is adopted. The stripe structured light is used to align the pixels of the binocular cameras, and the redundant weld edge pixel points are removed to achieve precise correction of the weld position. There will be informationless pixel points after the binary image is spatially mapped, and the mapping result needs to be filled by closing operation. The weld correction algorithm is shown in formula (2).

[0017]

[0018] where P cam , P warp are the camera image and the mapped image respectively, and B is the closing operation convolution kernel.

[0019] Further, in step three, the different scale features of the left and right camera images are analyzed, and spatial information constraints and pixel position constraints are combined with the annotation data to construct a two-dimensional weld extraction model.

[0020] Further, in step four, the two-dimensional weld extraction network (SWENet) based on spatial information mining reduces the amount of calculation and ensures that the network has sufficient global and detail perception capabilities.

[0021] The Encoder-Decoder structure is adopted, including a downsampling module, a transposed convolution module, and a feature extraction module. The structure of the feature extraction module uses two groups of 1D convolutions of 3×1 and 1×3 to reduce the amount of calculation. The ReLU between the two convolutions increases the learning ability of the network. In addition, dilated convolutions are used alternately to allow more context information to enter the next layer.

[0022] To achieve more accurate prediction and enable the network to learn image feature information and binocular spatial structure information, this paper uses two labels with different spatial angles to constrain the prediction results. The Loss function for model constraint is shown in Equation (3).

[0023] Loss =

[0024] 1 / 2 Cross(Pr′, Pr″) + 1 / 2 Cross(Warp(Pl)·B, Pr′) + 1 / 2 Cross(Pl, Pl″) +

[0025] 1 / 2 Cross(Warp(Pr′)·B, Pl″) (3)

[0027] In the formula, Cross is the cross-entropy loss function for evaluating the prediction error; Warp is the left-right pixel mapping function; Pr′, Pr″, and Pl″ are the prediction result, the right camera label, and the left camera label respectively, and B is the closing operation convolution kernel.

[0028] To further reduce the model prediction error, binocular consistency is used to correct the prediction results respectively to obtain the accurate two-dimensional weld position. The inference formula for the weld position of the right camera is shown in Equation (4), where Pr′ and Pl are the prediction results of the left and right cameras respectively.

[0029]

[0030] Furthermore, in step five, the left and right cameras are calibrated to obtain the correspondence between the image coordinate system and the world coordinate system of the cameras, realizing the mapping from two-dimensional pixels to three-dimensional spatial positions, as shown in Equation (5).

[0031]

[0032] In the formula, (u1, v1) and (u2, v2) are the corresponding pixel points in the left and right cameras respectively, M1 and M2 are the projection matrices of the left and right cameras, (X, Y, Z) is the three-dimensional spatial point, and Z1 and Z2 are the scaling constants. Eliminating M1 and M2 from the above formula gives

[0033]

[0034]

[0035]

[0036]

[0037] where Let \(M_{1}\) and \(M_{2}\) be the elements in the \(i\)-th row and \(j\)-th column. The spatial point coordinates \((X, Y, Z)\) can be solved by the least squares method.

[0038] The multi-width three-dimensional weld seam obtained is refined into a single-width path by using the nearest neighbor iterative method: Take any point \(F\) on the multi-width point cloud. The points near \(F\) can be approximated as vectors with direction \(L\). * Calculate the set \(\beta\) of points whose distance from \(F\) is less than \(d\). By calculating the vector angles between each point in \(\beta\) and point \(F\), the set can be divided into two parts \(\beta_{1}\) and \(\beta_{2}\).

[0039]

[0040]

[0041] Then take the two points \(F_{1}\) and \(F_{2}\) that are farthest from \(F\) in \(\beta_{1}\) and \(\beta_{2}\), and repeat the above steps in two directions with \(F_{1}\) and \(F_{2}\) as the center points respectively until it can no longer continue, then the single-width weld seam path can be obtained.

[0042] The pose is adjusted in real time by using the local neighborhood feature information of the weld seam points to improve the welding quality. Calculate the local neighborhood of each weld seam point, and calculate its covariance matrix \(C\) for each neighborhood.

[0043]

[0044]

[0045] In the formula, \(p\) i is the neighborhood point, is the neighborhood center, and \(n\) is the number of points in the neighborhood. Calculate the eigenvalues \(\lambda_{1}\), \(\lambda_{2}\), \(\lambda_{3}\) (\(\lambda_{1}>\lambda_{2}>\lambda_{3}\)) of the covariance matrix \(C\), and \(\lambda_{3}\) is the neighborhood normal vector value.

[0046] The present invention has remarkable effects compared with the prior art:

[0047] 1. Accurate and efficient weld seam extraction is realized; 2. A precise and reliable automatic data annotation method is designed; 3. A weld seam extraction network is designed to realize the extraction of two-dimensional coordinates of the weld seam to assist three-dimensional weld seam positioning; 4. The present invention provides a necessary strategy for realizing more accurate and efficient narrow butt weld seam detection. Brief Description of the Drawings

[0048] Figure 1 is a schematic diagram of the overall experimental system device of the present invention.

[0049] [[ID=~45]] Figure 2 is a schematic diagram of the binocular fringe sensor system of the present invention.

[0050] Figure 3 is a flowchart of data acquisition of the present invention.

[0051] Figure 4 It is a flowchart for the present invention to obtain point cloud from an image through fringe phase analysis.

[0052] Figure 5 It is a flowchart for two-dimensional data annotation of the present invention.

[0053] Figure 6 It is a comparison chart of binarization of adaptive gray threshold and fixed gray threshold of the present invention.

[0054] Figure 7 It is a flowchart for binocular consistency correction of the present invention.

[0055] Figure 8 It is a model structure diagram of a two-dimensional weld extraction network of the present invention.

[0056] Figure 9 It is a calibration plate for the experiment of the present invention.

[0057] Figure 10 It is a standard part and a point cloud picture of the present invention.

[0058] Figure 11 It is a weld extraction error chart of the present invention without using binocular self-constraint.

[0059] Figure 12 It is a weld extraction error chart of the present invention using binocular self-constraint. Detailed implementation manners

[0060] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0061] A binocular narrow butt weld detection method based on deep learning according to the present invention includes the following steps:

[0062] First, a projector in the fringe coding sensing system module projects a group of passive light and fringe structured light onto the welded part, triggering the camera to collect the corresponding passive light image and fringe structured light image. The light band projected by the projector is 450 nm, the incident angle between the projection center and the welded part is 60°, and at the same time, the optical axes of the binocular cameras are respectively placed on both sides of the weld with an angle of 60°.

[0063] The phase shifting profilometry (PSP) and phase unwrapping algorithm are used to analyze the modulated phase to obtain an unambiguous phase, and finally the point cloud information of the measured target is calculated through the system calibration data.

[0064] The weld is binarized using an adaptive gray threshold, and the relationship between the gray threshold of a pixel and the gray values of its neighborhood is shown in formula (1).

[0065]

[0066] Wherein, i and j are the row and column of the image respectively; f(i, j) is the gray value of the pixel at the j-th column of the i-th row; K is the size of the calculation neighborhood; C is an empirical constant; m and n are the row and column positions of the image respectively; T is the calculated threshold value.

[0067] Adopt a binocular consistency correction algorithm based on phase transformation, use stripe structured light to align the pixels of the binocular camera, remove redundant weld edge pixel points, and achieve precise correction of the weld position. After the binarized image is spatially mapped, there will be informationless pixel points, and it is necessary to perform closing operation processing on the mapping result for filling. The weld correction algorithm is shown in formula (2).

[0068]

[0069] Wherein, P cam , P warp are the camera image and the mapped image respectively, and B is the closing operation convolution kernel.

[0070] Then, analyze the different scale features of the left and right camera images, combine the annotation data for spatial information constraint and pixel position constraint, and construct a two-dimensional weld extraction model.

[0071] The two-dimensional weld extraction network (SWENet) based on spatial information mining reduces the amount of calculation and ensures that the network has sufficient global and detail perception capabilities.

[0072] Adopt an Encoder-Decoder structure, including a downsampling module, a transposed convolution module, and a feature extraction module. The structure of the feature extraction module uses two groups of 1D convolutions of 3×1 and 1×3 to reduce the amount of calculation, and the ReLU between the two convolutions increases the learning ability of the network. In addition, dilated convolutions are alternately used to allow more context information to enter the next layer.

[0073] In order to achieve more accurate prediction and let the network learn the image feature information and the binocular spatial structure information, this paper uses two labels with different spatial angles to constrain the prediction result. The Loss function for model constraint is shown in formula (3).

[0074] Loss =

[0075] 1 / 2Cross(Pr, Pr″) + 1 / 2Cross(Warp(Pl)·B, Pr′) + 1 / 2Cross(Pl′, Pl″) +

[0076] 1 / 2Cross(Warp(Pr′)·B, Pl″) (3)

[0078] Where Cross is the cross-entropy loss function, which evaluates the magnitude of the prediction error; Warp is the left-right pixel mapping function; Pr′, Pr″, and Pl′ are the prediction result, the right camera label, and the left camera label respectively, and B is the closing operation convolution kernel.

[0079] Finally, to further reduce the model prediction error, binocular consistency is used to correct the prediction results respectively to obtain the accurate two-dimensional weld position. The inference formula for the weld position of the right camera is shown in formula (4), where Pr′ and Pl′ are the prediction results of the left and right cameras respectively.

[0080]

[0081] Calibrate the left and right cameras to obtain the correspondence between the image coordinate system and the world coordinate system of the cameras, and realize the mapping from two-dimensional pixels to three-dimensional space positions, as shown in formula (5).

[0082]

[0083] Where (u1, v1) and (u2, v2) are the corresponding pixel points in the left and right cameras respectively, M1 and M2 are the projection matrices of the left and right cameras, (X, Y, Z) is the three-dimensional space point, and Z1 and Z2 are the scaling constants. Eliminating M1 and M2 from the above formula gives

[0084]

[0085]

[0086]

[0087]

[0088] Where is the i-th row and j-th column of M1 and M2, and the spatial point coordinates (X, Y, Z) can be solved by the least squares method.

[0089] The multi-line-width three-dimensional weld obtained is refined into a single-line-width path using the nearest neighbor iteration method: Take any point F on the multi-line-width point cloud. The points near F can be approximated as vectors in the direction of L * Calculate the set β of points whose distance from F is less than d. By calculating the vector angles between each point in β and point F, the set can be divided into two parts, β1 and β2.

[0090]

[0091]

[0092] Then, take the two points F1 and F2 that are farthest from F among β_1 and β_2, and then repeat the above steps in two directions with F1 and F2 as the center points respectively until it can no longer continue, and the weld path with a single line width can be obtained.

[0093] The pose is adjusted in real time by using the local neighborhood feature information of the weld points to improve the welding quality. Calculate the local neighborhood of each weld point, and calculate its covariance matrix C for each neighborhood.

[0094]

[0095]

[0096] In the formula, p i is the neighborhood point, is the neighborhood center, and n is the number of points in the neighborhood. Calculate the eigenvalues λ1, λ2, λ3 (λ1 > λ2 > λ3) of the covariance matrix C, and λ3 is the neighborhood normal vector value.

[0097] This system uses the classic Zhang Dingyou planar calibration method to calibrate the binocular camera, and finally obtains the internal and external parameters of the camera. In order to obtain accurate calibration results, a circular calibration plate with higher accuracy than the checkerboard is used for calibration in the experiment, and the center distance is 4 mm. After the system calibration is completed, two standard parts with fixed thickness are used for accuracy testing. The point cloud data of the standard parts can be obtained by scanning the standard parts with the system, and then plane fitting is performed on the obtained point clouds respectively. The system error is obtained by comparing the actual thickness of the two standard parts with the thickness calculated by the fitting plane. This system obtains the point clouds of the standard parts at 5 different positions in the field of view respectively, and the calculated errors are as Figure 11 shown. In addition to the measurement accuracy of the system, the system resolution also affects the extraction accuracy of narrow butt welds. The point distance of the point cloud obtained by this experimental system is about 0.065 mm, which is suitable for the extraction of 0.3 mm welds. Use a feeler gauge to limit the width of the narrow butt weld to 0.3 mm (due to the machining error of the workpiece, the actual width will fluctuate within the range of 0.3 mm). Compare the extracted weld with the position of the weld marked manually to obtain the extraction error of the weld. As Figure 12 shown, for the weld extraction error using binocular self-constraint, the average error is 0.0155, among which 63.63% of the points have an error of zero, and the errors of the remaining points are all within one point distance. It shows that the binocular self-constraint greatly improves the weld extraction accuracy, and the error fluctuation range is relatively concentrated.

[0098] Inspired by the above ideal embodiments according to the present invention, through the above description, relevant staff can completely make various changes and modifications without departing from the technical idea of this invention. The technical scope of this invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A binocular narrow butt weld detection method based on deep learning, characterized in that, It includes the following steps: Step 1: Based on the projector projecting stripe structured light and passive light, collect the weld image and obtain the point cloud of the welded part; Step 2: Conduct two-dimensional data annotation based on image binarization processing and binocular consistency correction; Step 3: Construct a two-dimensional weld extraction model based on spatial information mining to extract the two-dimensional weld; Step 4: Map the two-dimensional pixels of the weld into three-dimensional space coordinates based on the binocular vision model; Step 5: Conduct pose estimation based on the point cloud of the welded part and the local neighborhood feature information of the weld points.

2. The binocular narrow butt weld detection method based on deep learning according to claim 1, wherein: The projector in the stripe coding sensing system module projects a group of passive light and stripe structured light onto the welded part, triggering the camera to collect the corresponding passive light image and stripe structured light image; the light band projected by the projector is 450nm, the incident angle between the projection center and the welded part is 60°, and at the same time, the optical axes of the binocular cameras are kept at 60° and placed on both sides of the weld respectively; Adopt phase shifting profilometry and phase unwrapping algorithm to analyze the modulated phase to obtain an unambiguous phase, and finally calculate the point cloud information of the measured target through the system calibration data.

3. The binocular narrow butt weld detection method based on deep learning according to claim 1, characterized in that: Adopt adaptive gray threshold to conduct binarization processing on the image. The relationship between the gray threshold of the pixel and the gray value of its neighborhood is shown in formula (1). Where, i and j are the rows and columns of the image respectively; f(i, j) is the gray value of the pixel in the i-th row and j-th column; K is the size of the calculated neighborhood; C is an empirical constant; m and n are the row and column positions of the image respectively; T is the calculated threshold.

4. The binocular narrow butt weld detection method based on deep learning according to claim 1, characterized in that: Adopt a binocular consistency correction algorithm based on phase transformation. Use the stripe structured light to align the pixels of the binocular cameras, remove redundant weld edge pixel points, and achieve precise correction of the weld position; there will be informationless pixel points after the binarized image is spatially mapped, and it is necessary to perform closing operation processing on the mapping result for filling. The weld correction algorithm is shown in formula (2). Among them, P cam , P warp are the camera image and the mapped image respectively, and B is the closing operation convolution kernel.

5. The binocular narrow butt weld detection method based on deep learning according to claim 1, wherein: Analyze the different scale features of the left and right camera images, combine the annotation data for spatial information constraint and pixel position constraint, and construct a two-dimensional weld extraction model.

6. The binocular narrow butt weld detection method based on deep learning according to claim 1, characterized in that: The two-dimensional weld extraction network based on spatial information mining reduces the calculation amount and ensures that the network has sufficient global and detail perception capabilities; Adopt the Encoder-Decoder structure, including a downsampling module, a transposed convolution module, and a feature extraction module; The structure of the feature extraction module uses two groups of 1D convolutions of 3×1 and 1×3 to reduce the calculation amount. The ReLU between the two convolutions increases the learning ability of the network. In addition, dilated convolutions are used alternately to enable more context information to enter the next layer; In order to achieve more accurate prediction, let the network learn the image feature information and the binocular spatial structure information, and use two labels with different spatial angles to constrain the prediction result; the Loss function for model constraint is shown in formula (3). Loss = 1 / 2Cross(Pr′, Pr″)+1 / 2Cross(Warp(Pl′)·B, Pr′′)+1 / 2Cross(Pl′, Pl″)+1 / 2Cross(Warp(Pr′)·B, Pl″) (3) Where Cross is the cross-entropy loss function, which evaluates the magnitude of the prediction error; Warp is the left-right pixel mapping function; Pr′ and Pl′ are the prediction results of the right camera and the left camera respectively, Pr" and Pl" are the labels of the right camera and the left camera respectively, and B is the closing operation convolution kernel; To further reduce the model prediction error, binocular consistency is used to correct the prediction results respectively to obtain the accurate two-dimensional weld position; the inference formula for the weld position of the right camera is shown in formula (4); 7. The binocular narrow butt weld detection method based on deep learning according to claim 1, characterized in that: The left and right cameras are calibrated to obtain the correspondence between the image coordinate system and the world coordinate system of the cameras, realizing the mapping from two-dimensional pixels to three-dimensional spatial positions, as shown in formula (5), Where (u1, v1) and (u2, v2) are the corresponding pixel points in the left and right cameras respectively, M1 and M2 are the projection matrices of the left and right cameras, (X, Y, Z) is the three-dimensional spatial point, and Z1, Z2 are the scaling constants; eliminating M1 and M2 from the above formula gives, Among them is the i-th row and j-th column of M1 and M2, and the spatial point coordinates (X, Y, Z) can be solved by the least squares method; The obtained multi-line-width three-dimensional weld seam is refined into a single-line-width path by using the nearest neighbor iteration method: Take any point F on the multi-line-width point cloud. The points near F can be approximated as vectors with the direction of L * Calculate the set β of points whose distance from F is less than d. By calculating the vector angles between each point in β and point F, the set can be divided into two parts, β1 and β2 Then, take the two points F1 and F2 that are farthest from F among β1 and β2, and then repeat the above steps in two directions with F1 and F2 as the center points respectively until it cannot continue, and the weld path with a single line width can be obtained; The pose is adjusted in real time using the local neighborhood feature information of the weld points to improve the welding quality. Calculate the local neighborhood of each weld point, and calculate its covariance matrix C for each neighborhood, where p i is a neighborhood point, is the neighborhood center, and n is the number of points in the neighborhood; Calculate the eigenvalues λ1, λ2, λ3 of the covariance matrix C, where λ1 > λ2 > λ3, and λ3 is the neighborhood normal vector value.

Images