A method and device for initial welding elongation measurement based on binocular vision

Abstract

The application relates to the technical field of robots, and discloses a method and device for measuring initial welding dry elongation based on binocular vision, mainly including camera calibration, image acquisition and processing, stereo matching to obtain a parallax map and a depth map, positioning of a welding gun axis and identification of welding gun key points, identification and positioning of a point to be welded, after a welding area ROI (region of interest) is intercepted, a point to be welded is determined through an intersection of an extension line of the welding gun axis and a weld; and the three-dimensional coordinates of the welding gun key points and the point to be welded are used to indirectly calculate the welding dry elongation. The method can quickly and stably detect the welding dry elongation before welding, and is helpful for timely adjusting the welding dry elongation according to welding parameters or timely adjusting the welding parameters according to the welding dry elongation. The application can not only improve the welding quality and efficiency of the robot, but also improve the automation degree of robot welding and reduce the labor intensity of welders.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more specifically to a method and apparatus for measuring the initial weld extension based on binocular vision. Background Technology

[0002] Welding plays a vital role in industrial production, and the wire extension length has a crucial impact on weld quality. When the wire extension is too long, the resistance voltage of the welding wire increases, while the voltage across the welding arc decreases, resulting in less arc heat and affecting weld quality. Conversely, if the extension is too short, the arc can easily backtrack and burn out the contact tip, leading to cracks in the weld due to contamination from the contact tip material. Generally, when the welding wire diameter is less than 3 mm, the wire extension length should be between 20 and 60 mm. Therefore, it is necessary to monitor and control the wire extension length according to welding requirements to ensure weld quality.

[0003] In actual production environments, the distance from the bottom of the contact tip to the surface of the workpiece is usually used as a reference for convenient measurement of the welding wire extension length. However, manual measurement using tools such as rulers is inefficient and unsafe. Therefore, an effective, non-contact method is needed for the preliminary measurement of welding wire extension. Summary of the Invention

[0004] To address the aforementioned problems, this invention proposes a method and apparatus for measuring initial weld extension based on binocular vision, aiming to achieve automatic, rapid, and non-contact measurement of initial weld extension. This invention ensures rapid and accurate measurement of initial weld extension, reducing the labor intensity of welders and improving welding quality and efficiency. To solve the problems existing in current initial weld extension measurement technologies, the technical solution adopted by this invention is as follows:

[0005] A method for measuring initial weld extension based on binocular vision, the method comprising the following steps:

[0006] Step 1, camera calibration; obtain the internal and external parameters of each camera module and the structural parameters between the two camera modules in the binocular camera.

[0007] Step 2, Image Acquisition and Image Processing: The acquired raw RGB image is subjected to distortion correction and stereo correction to obtain distortion-free and coplanar aligned RGB and grayscale images from the left and right perspectives.

[0008] Step 3: Stereo matching and obtaining disparity and depth maps; using the semi-global block matching algorithm SGBM to calculate binocular disparity, thereby obtaining point cloud maps and depth maps, and aligning the left view of the binocular camera with the depth map.

[0009] Step 4: Adaptive positioning of the welding torch and its axis; by locating the point on the image at the end of the welding torch nozzle connector located on the welding torch axis, the welding torch axis is positioned, providing a basis for positioning the welding area. Because the surface of the welding torch nozzle is not only smooth and textureless but also has specular reflection, feature point matching is difficult, making direct identification of the welding torch nozzle and calculation of the three-dimensional coordinates of the nozzle apex difficult and unstable. Note that there is a standard brass nozzle connector coaxial with the welding torch above the nozzle, which allows for more stable identification and positioning. Therefore, the weld extension is indirectly measured by calculating the distance from the point at the end of the welding torch nozzle connector located on the welding torch axis to the point to be welded.

[0010] Step 5: Estimate the pixel position of the point to be welded in the image; In order to obtain the pixel coordinates of the point to be welded in the left-view image, the welding area ROI (Region of Interest) is cropped, and the point to be welded is determined by the intersection of the extension line of the welding gun axis and the weld seam. Finally, the coordinates of the point to be welded in the cropped welding area ROI image are converted into pixel coordinates in the left-view image.

[0011] Step 6: Key point coordinate transformation and distance calculation; the two-dimensional coordinates of the point to be welded and the key points of the welding gun are transformed into three-dimensional coordinates. By calculating the Euclidean distance between the two points in three-dimensional space, after excluding outliers, the average measurement value of every 40 frames is taken as the final result of the initial welding extension of the binocular vision measurement.

[0012] Furthermore, the binocular cameras are calibrated to obtain the intrinsic and extrinsic parameters of each camera and the structural parameters between the two cameras. The steps include:

[0013] First, the two cameras on the binocular camera module simultaneously capture images of the checkerboard calibration board at different locations.

[0014] Next, OpenCV is used for camera calibration and stereo calibration. The intrinsic and extrinsic parameters, distortion parameters, and structural parameters between the two cameras are obtained for each camera module, including the rotation matrix and translation vector from the right camera coordinate system to the left camera coordinate system.

[0015] Further, the steps of image acquisition, and performing distortion correction and stereo correction on the acquired images to obtain two images with distortion-free and coplanar alignment from the left and right perspectives include:

[0016] First, the two three-channel RGB images from the left and right perspectives of the binocular camera are converted into two single-channel grayscale images.

[0017] Next, the distortion parameters obtained from camera calibration were used to correct the distortion of the two grayscale images.

[0018] Then, the camera intrinsic parameters obtained from camera calibration and the structural parameters between the binocular cameras are used to perform stereo correction on the two grayscale images to obtain coplanar aligned binocular images from the left and right perspectives.

[0019] Further, stereo matching is performed to obtain disparity and depth maps. The steps are as follows:

[0020] First, the disparity in binocular vision is calculated using the semi-global block matching algorithm SGBM.

[0021] Next, using the disparity map and the reprojection matrix obtained during stereo correction, a mapping map is generated. This mapping map is a three-channel image of the same size as the disparity map. Each channel stores the values ​​of the pixel position on the X, Y, and Z axes in the camera coordinate system, that is, the three-dimensional coordinates (x, y, z) of each pixel in the camera coordinate system, where the value of z represents the distance from the object to the binocular camera plane, in millimeters (mm).

[0022] Furthermore, adaptive positioning of the welding torch and its axis. The steps include:

[0023] First, the corrected left-view image is converted into an HSV color gamut image. By adjusting the H, S, and V thresholds, the nozzle connector part is segmented according to color and displayed on the binary image.

[0024] Next, small connected regions are filtered to remove noise, and then a closing operation is used to make the segmented image of the nozzle connector more complete. A contour extraction algorithm is used to obtain the edge contour of the nozzle connector in the segmented image, and the minimum bounding rectangle is generated based on the contour.

[0025] Next, calculate the midpoints of the two sides of the smallest bounding rectangle perpendicular to the welding torch axis. The line connecting these two points is the straight line containing the welding torch axis. Solve the equation of this line in the pixel coordinate system of the left view to provide a basis for the next step of truncating the welding area. In addition, record the pixel coordinates of the midpoint closer to the welding point in the corrected left view.

[0026] Further, the welding area is extracted and cropped from the left-view image, and the pixel coordinates of the points to be welded are estimated on the cropped welding area image. Finally, these pixel coordinates are converted into pixel coordinates on the left-view image. The steps include:

[0027] First, draw the rectangular region of interest (ROI) for welding on the corrected left view. Since the relative positions of the camera and the welding torch are fixed, the pixel coordinates of the torch vertex are also fixed. On the left-view image, draw a line segment of appropriate length perpendicular to the torch axis, passing through the torch vertex, as the width of the ROI. Specifically, the torch vertex is the midpoint of this line segment. Based on the width of the ROI, the longer side of the ROI is parallel to the torch axis.

[0028] Next, trim the welding area ROI rectangle. Since this rectangle is a rotated rectangle, rotate the left view so that the rotation angle of the welding area ROI is 0. Trim the welding area ROI; the vertical center line of the trimmed welding area ROI is now the axis of the welding torch.

[0029] Next, extract the points to be welded. Convert the ROI image to grayscale, use the Canny algorithm to extract the weld seam, and record the intersection of the weld seam and the vertical centerline (welding torch axis) of the ROI. This intersection is the point to be welded.

[0030] Finally, the pixel coordinates of the points to be soldered are calculated. Using the reverse operation of the step of cropping the ROI (Region of Interest) for soldering, the pixel coordinates of the points to be soldered on the cropped ROI image are transformed into pixel coordinates on the corrected left view.

[0031] Furthermore, the two-dimensional coordinates of the points to be soldered and the key points of the welding torch are transformed into three-dimensional coordinates. By calculating the Euclidean distance between the two points in three-dimensional space, and after eliminating outliers, the average measurement value for every 40 frames is taken as the final result of the binocular vision measurement. The specific steps are as follows:

[0032] First, calculate the three-dimensional coordinates of the welding torch nozzle joint vertex and the point to be welded in the camera coordinate system, obtained from the previous steps.

[0033] Next, the Euclidean distance between the two points in three-dimensional space is calculated using the following formula.

[0034]

[0035] Among them, (x a ,y a ,z a (x) represents the three-dimensional coordinates of a point located at the end of the welding torch nozzle connector on the welding torch axis in the camera coordinate system. b ,y b ,z b ) represents the three-dimensional coordinates of the point to be welded in the camera coordinate system. length represents the point (x, y) located at the end of the welding torch nozzle connector on the welding torch axis, as measured visually. a ,y a ,z a ) to the point to be welded (x b ,y b ,z b The Euclidean distance of ).

[0036] Next, the distance from the end of the nozzle connector to the welding point (length) minus the length (len) of the welding torch nozzle. weld , which is the distance len from the nozzle tip to the welding point, is also the wire extension, and satisfies the following formula.

[0037] len = length - len weld

[0038] Where len is the final calculated wire extension. weld This refers to the nozzle length.

[0039] Then, determine whether the measured distance len from the previous step is within a reasonable range (0 cm to 7.5 cm); otherwise, skip directly to the next frame for measurement. After measuring 40 frames, take the average of these 40 measurements as the final result of the binocular vision measurement.

[0040] The present invention also provides an initial weld extension measurement device based on binocular vision, comprising a frame-synchronized binocular camera module, a camera fixture, a computer, an illumination module, a welding torch, and a device for moving the welding torch, wherein:

[0041] The frame-synchronized binocular camera module has a focal length of 3.0mm, a baseline of 63mm, and an image resolution of 640×480, and is used for raw image acquisition.

[0042] The camera clamp is used to fix the relative position between the binocular camera module and the welding torch, so that the baseline of the camera is approximately perpendicular to the axis of the welding torch on a certain projection plane, and the camera is fixed to the side of the welding torch's forward direction. This design reduces the number of parameters that need to be estimated separately, simplifies the measurement algorithm, improves measurement accuracy, and speeds up the algorithm's calculation speed.

[0043] The computer is used for real-time image acquisition, image processing, feature matching, stereo matching, and key point estimation and recognition to complete the measurement of the initial weld extension.

[0044] The lighting module is fixed to the device used to move the welding torch, and is used to improve ambient lighting and increase measurement accuracy.

[0045] The welding torch is a Panasonic MIG / MAG welding torch. The nozzle outer diameter is 24mm, and the overall length is 73mm. The nozzle connector is a standard brass-colored connector. The welding torch and camera clamp are fixed together on a device for moving the welding torch. The welding torch, camera, and lighting module maintain a fixed relative position.

[0046] The beneficial effects of this invention are:

[0047] This invention can quickly and stably detect weld extension, thereby allowing for timely adjustments to the weld extension or vice versa based on the extension. This not only improves the quality and efficiency of machine welding but also enhances the automation level of robotic welding and reduces the labor intensity of welders. Attached Figure Description

[0048] Figure 1 This is a flowchart of the initial weld extension measurement method based on binocular vision according to the present invention;

[0049] Figure 2 This is a flowchart of the adaptive positioning welding torch and the welding torch axis of the present invention;

[0050] Figure 3 This is a flowchart illustrating the estimation of the pixel position of the solder joint in an image according to the present invention.

[0051] Figure 4 The flowchart shows the average of every 40 measurements in this invention as the final measurement result;

[0052] Figure 5 This is an overall structural diagram of the binocular vision-based initial weld extension measurement device of the present invention;

[0053] Figure 6 This is a side view of the binocular vision-based initial weld extension measurement device of the present invention.

[0054] Figure 7 Experimental results of utilizing the adaptive positioning welding torch, welding torch axis, and welding area of ​​the present invention;

[0055] Figure 8 Experimental results of weld identification using the present invention; (a) Display of weld area positioning and welding point positioning results; (b) Weld identification of the weld area after rotating and cropping the weld area;

[0056] Figure 9 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 2.0 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0057] Figure 10 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 2.5 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0058] Figure 11 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 3.0 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0059] Figure 12 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 3.5 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0060] Figure 13 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 4.0 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0061] Figure 14 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 4.5 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0062] Figure 15 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 5.0 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0063] Figure 16 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 5.5 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0064] Figure 17 Experiments were conducted to verify the stability and accuracy of visual measurements using this invention when the wire extension was 6.0 cm; (a) the results of 100 consecutive measurements with each frame as a measurement result; (b) the results of 50 measurements with the average of 40 consecutive frames as the measurement result.

[0065] Figure 18 This relates the measurement error, standard deviation, and the number of frames used to calculate the average value when the actual initial weld extension is 2.5 cm in a certain measurement experiment.

[0066] Figure 19This relates to the calculation error and time consumption for every 10 additional frames used in calculating the average value when the actual initial weld extension is 2.5 cm in a certain measurement experiment.

[0067] The attached diagram is labeled as follows: 1. Nozzle connector; 2. Nozzle; 3. Binocular camera module; 4. Illumination module; 5. Material to be welded; 6. Camera fixture; 7. Base; 8. Welding extension. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] See Figure 1 This invention provides a method for measuring initial weld extension based on binocular vision, the steps of which include:

[0070] Step 1: Camera calibration. For binocular cameras, it is necessary to obtain the intrinsic and extrinsic parameters of each camera module and the structural parameters between the two camera modules.

[0071] First, the two imaging systems in the binocular camera module used in this invention are on a single plane. The calibration board is placed within the overlapping field of view of the binocular cameras, and the image area of ​​the calibration board captured occupies approximately 1 / 3 of the entire image area, ensuring that both cameras can simultaneously identify all corner points. With a fixed aperture and focal length, the two cameras on the binocular camera module simultaneously capture 15 images of the checkerboard calibration board (9×6, 19.5mm) at different positions (the checkerboard is imaged in the upper left, upper right, lower left, lower right, and center of the camera's field of view).

[0072] Next, OpenCV is used for camera calibration and stereo calibration. The intrinsic and extrinsic parameters, distortion parameters, and structural parameters between the two cameras are obtained, including the rotation matrix and translation vector between the two coordinate systems.

[0073] Obtain the intrinsic parameters K and distortion parameters d of the two cameras respectively.

[0074]

[0075] d = [k1, k2, p1, p2, k3]

[0076] Among them, f x f y c x c yIt is an intrinsic parameter of the camera's linear model. f x f y These are the scale factors along the u-axis and v-axis of the pixel coordinate system, respectively, c x c y It is an optical center. i (i = 1, 2, 3) are the radial distortion coefficients, p i (i = 1, 2) are the tangential distortion coefficients.

[0077] Using the camera parameters described above, stereo calibration yields a rotation matrix R (3×3, an orthogonal matrix with a determinant of 1) and a translation vector t (3×1). Given the 3D coordinates (x1, y1, z1) of a point in the first camera coordinate system, the 3D coordinates (x2, y2, z2) in the second camera coordinate system can be calculated.

[0078]

[0079] Step two, image acquisition and image processing. Distortion correction and stereo correction are performed on the acquired raw RGB images to obtain distortion-free and coplanar aligned RGB and grayscale images from both left and right perspectives.

[0080] First, the two three-channel RGB images from the left and right perspectives of the binocular camera are converted into two single-channel grayscale images.

[0081] GRAY←0.299·R+0.587·G+0.114·B

[0082] Where R, G, and B are the pixel brightness values ​​of the three channels of the color image, and GRAY is the corresponding pixel gray value of the grayscale image.

[0083] Next, using the distortion parameters obtained from camera calibration, distortion correction is performed on the two grayscale images from the left and right viewpoints of the binocular camera, respectively, resulting in two distortion-free grayscale images. First, the coordinates of point P on the distortion-free image are calculated as [u, v]. T The corresponding coordinates on the distorted image [u distorted ,v distorted ] T First, determine the coordinates [u,v] of point P in the distortion-free image. T Transforming to the normalized plane, we obtain the coordinates [x, y]. T ,

[0084]

[0085] For a point [x,y] on the normalized plane T Calculate radial and tangential distortion.

[0086]

[0087] Where, [x,y] T The coordinates of point P on the normalized plane are [x distorted ,y distorted ] T These are the normalized coordinates of the points after distortion. Let k be the distance between point P and the origin of the coordinate system. i (i = 1, 2, 3) are the radial distortion coefficients, p i (i = 1, 2) are the tangential distortion coefficients.

[0088] The normalized coordinates of the distorted points are projected onto the pixel plane using an intrinsic parameter matrix to obtain the point [u,v] on the distortion-free image. T The coordinates [u] on the distorted image distorted ,v distorted ] T ,

[0089]

[0090] Finally, the distorted image is located in [u distorted ,v distorted ] T The value on the graph is assigned to the coordinates [u, v] of the distortion-free image. T Location,

[0091] image undistorted (u,v)←image distorted (u distorted ,v distorted )

[0092] Among them, [u,v] T This represents the correct location of point P on the image. distorted ,v distorted ] T This represents the coordinates of point P on the undistorted image to the coordinates on the distorted image. undistorted This represents a distortion-free image after distortion correction. distorted This represents the original image.

[0093] Then, using the camera intrinsic parameters obtained from the camera calibration in step one and the structural parameters between the binocular cameras, stereo correction is performed on the two grayscale images to obtain coplanar row-aligned binocular images of the left and right views. In this invention, the left and right camera views of the binocular camera are mainly horizontally moved along the X-axis (possibly with a small vertical offset). After coplanar row alignment correction, the corresponding epipolar lines in the left and right views are horizontal, the optical axes are parallel, the imaging planes of the left and right views are coplanar, and the two pixel coordinate systems projected onto the image planes of the two cameras have the same y-coordinate. Simultaneously, a parallax-to-depth mapping matrix is ​​obtained during stereo correction.

[0094]

[0095] Among them, T x is the length of the binocular camera baseline, [c_x1,c_y] is the origin of the left image, c_x2 is the origin of the X-axis in the right view, and f is the focal length.

[0096] Step 3: Stereo matching and acquisition of disparity and depth maps. The SGBM algorithm is used to calculate binocular disparity, thereby obtaining point cloud and depth maps. The left view of the binocular camera is aligned with the depth map.

[0097] First, the disparity in binocular vision is calculated using the semi-global block matching algorithm SGBM, which utilizes the functions cv::StereoSGBM::create() and cv::StereoMatcher::compute().

[0098] Next, using the disparity image and the reprojection matrix Q obtained in step two, a mapping map is obtained. This mapping map is a three-channel image of the same size as the disparity image. Each channel stores the values ​​of the pixel position on the X, Y, and Z axes in the camera coordinate system, that is, the three-dimensional coordinates (x, y, z) of each pixel in the camera coordinate system, where the value of z represents the distance from the object to the binocular camera plane, in millimeters (mm).

[0099] Step 4: Adaptive positioning of the welding torch and its axis. Because the surface of the welding torch nozzle is not only smooth and textureless but also has specular reflection, feature point matching is difficult, making direct identification of the nozzle and calculation of the three-dimensional coordinates of the nozzle apex challenging and unstable. Note that there is a standard brass nozzle connector coaxial with the welding torch above the nozzle, which allows for more stable identification and positioning. Therefore, the weld extension is indirectly measured by calculating the distance from the point at the end of the nozzle connector located on the welding torch axis to the weld point. The specific steps are as follows:

[0100] First, the corrected left-view RGB image is converted to an HSV color gamut image. By adjusting the H, S, and V thresholds, the nozzle connector portion is segmented according to color and displayed on a binary image. The formula for converting an RGB image to an HSV color gamut image is as follows.

[0101] V = max(R, G, B)

[0102]

[0103]

[0104] When H < 0, H = H + 360. Output V ∈ [0, 1], S ∈ [0, 1], H ∈ [0, 360]. When displaying an 8-bit image,

[0105] V = 255V

[0106] S = 255S

[0107] H = H / 2

[0108] Where R, G, and B represent the pixel brightness of the R, G, and B channels in the color image, respectively, and H, S, and V represent the pixel values ​​of the corresponding H, S, and V channels in the HSV color gamut. By adjusting the HSV threshold, a binary image is generated where only the target area has a pixel value of 255, and the remaining non-target areas have a pixel value of 0.

[0109] Next, the binary image generated in the previous step still has many noise points in the non-target area. Small connected regions are filtered to remove the noise. Then, a dilation operation is performed to eliminate small holes and cracks in the target area of ​​the binary image with pixel values ​​of 0, followed by erosion to restore the target area to its original size. The closing operation makes the segmented image of the nozzle connector more stable and complete. Then, the minimum bounding rectangle of the target area contour is obtained. This minimum bounding rectangle is a rotated rectangle, defined by ((x...)... center ,y center ), (width, height), angle) represent. Where (x, height), angle) center ,y center Let () be the center coordinates of the rotating rectangle, (width, height) be the width and height of the rotating rectangle, and angle be the angle of rotation of the horizontal rectangle around the center point. Additionally, θ = angle * π / 180, in rad. Therefore, the rotation matrix is:

[0110]

[0111] In a plane coordinate system, for any point (xa, ya), after rotating by an angle θ around a coordinate point (xb, yb), the new coordinates are set as (x, y). The formula for calculating this is:

[0112]

[0113] Then, the midpoints of the two sides of the minimum bounding rectangle perpendicular to the welding torch axis are calculated. Since the welding torch nozzle of the experimental equipment used in this invention is a standard cylindrical shape, the line connecting the midpoints of these two sides is the straight line containing the welding torch axis. By sorting the four vertices of the minimum bounding rectangle in ascending order according to the X-axis and Y-axis, the two sides of the minimum bounding rectangle perpendicular to the welding torch axis can be obtained, and then the midpoints of these two sides can be obtained. Using these two midpoints, the two-variable linear equation of the welding torch axis in the left-view image pixel coordinate system can be obtained:

[0114] Let the four vertices of the bounding rectangle of the nozzle connector region be (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Sort them in ascending order along the X-axis as vector_x = [(x4, y4), (x1, y1), (x3, y3), (x2, y2)], and in ascending order along the Y-axis as vector_y = [(x1, y1), (x2, y2), (x4, y4), (x3, y3)]. The intersection points of the welding torch axis and this minimum bounding rectangle are (x5, y5) and (x6, y6).

[0115]

[0116] In this context, vector_x[0] and vector_y[0] represent the first elements of vectors vector_x and vector_y, respectively, namely (x4, y4) and (x1, y1). vector_x[-1] and vector_y[-1] represent the last elements of vectors vector_x and vector_y, respectively, namely (x2, y2) and (x3, y3). " / 2" means dividing each element by 2.

[0117] The slope of the welding torch axis in the pixel coordinate system of the left view can be obtained from (x5, y5) and (x6, y6) using the linear equation in two variables. The intercept is b = y⁵ - k·x⁵, and the equation is expressed as y = k·x + b. The flowchart is as follows: Figure 2 As shown.

[0118] Step 5: Estimate the pixel position of the solder joint in the image. Extract the ROI (Region of Interest) for soldering and estimate the pixel position of the solder joint within that region. Finally, convert the coordinates of the solder joint on the ROI image to pixel coordinates on the left-view image.

[0119] When the welding torch axis is aligned with the weld seam in space for welding, the welding torch axis intersects with the weld seam. The location of the weld point is represented on the image as the intersection of the welding torch axis and the weld seam. The steps include:

[0120] First, draw the rectangular region of interest (ROI) for welding in the corrected left view. Since the relative positions of the camera and the welding torch are fixed, the pixel coordinates of the torch vertex are also fixed. In the left-view image, draw a line segment of appropriate length perpendicular to the torch axis, passing through the torch vertex, as the width of the ROI. Specifically, the torch vertex is the midpoint of this line segment. Based on the width of the ROI, the longer side of the ROI is parallel to the torch axis.

[0121] From step four, we know the equation of the welding torch axis in the left-view image pixel coordinate system, let it be y = k·x + b. Then, through the known pixel coordinates (_x,_y) of the welding torch vertex, draw a line of appropriate length perpendicular to the welding torch axis with the equation y = (-1 / k)·x + (_y + (1 / k)·_x). Take the welding torch vertex as the midpoint and take a length of 20 pixels as the width of the welding region ROI. Take the two endpoints of the line segment with the width as the above and draw a line segment parallel to the welding torch axis with a length of 60 pixels, extending from the welding torch vertex to the point to be welded, as the long side of the welding region ROI.

[0122] Next, the welding area ROI is cropped. The welding area ROI obtained in the above steps is a rotated rectangle, which is not convenient for cropping the image and identifying the points to be welded. Therefore, the original left view is rotated so that the rotation angle of the welding area ROI is 0. Then, the welding area ROI is cropped from the left view. At this time, the vertical center line of the rectangular area (i.e., the cropped welding area) is the axis of the welding gun.

[0123] Then, the points to be welded are extracted. The ROI image of the welding area is converted to grayscale, and the weld seam is extracted using the Canny algorithm. The weld seam extraction result is as follows. Figure 8 As shown in (b), the white line in the figure represents the weld location. Next, record the coordinates of the intersection point of the weld and the vertical center line of the ROI image; this intersection point is the point to be welded. Specifically, starting from the center point of the rectangular area, explore one pixel upwards and downwards each time. When the pixel value is 255, stop exploring and record the current pixel coordinates.

[0124] Finally, calculate the pixel coordinates of the point to be soldered in the left-view image. This is done by reversing the steps described above: first, restore the pixel coordinates of the point to be soldered in the cropped ROI image to the pixel coordinate system of the rotated left-view image; then, rotate this coordinate point in the reverse direction of the previous steps. This process transforms the pixel coordinates of the point to be soldered in the cropped ROI image into pixel coordinates in the left-view image. The flowchart is as follows. Figure 3 As shown.

[0125] Step six: Convert the two-dimensional coordinates of the solder joint and the key points of the welding gun into three-dimensional coordinates. Calculate the Euclidean distance between the two points in three-dimensional space. After excluding outliers, take the average measurement value of every 40 frames as the final result of the binocular vision measurement.

[0126] First, the two-dimensional coordinates of the points to be soldered and the key points of the welding gun are transformed into three-dimensional coordinates.

[0127]

[0128] Among them, (x c ,y c ,zc ) represents the three-dimensional coordinates of the key point, (u c ,v c f represents the two-dimensional coordinates of the key point. x f y c x c y It is an intrinsic parameter of the camera's linear model. f x f y These are the scale factors along the u-axis and v-axis of the pixel coordinate system, respectively, c x c y It is the optical center, and depth is the depth.

[0129] Calculate the Euclidean distance between two points in three-dimensional space.

[0130]

[0131] Among them, (x a ,y a ,z a (x) represents the three-dimensional coordinates of a point located at the end of the welding torch nozzle connector on the welding torch axis in the camera coordinate system. b ,y b ,z b ) represents the three-dimensional coordinates of the point to be welded in the camera coordinate system. length represents the point (x, y) located at the end of the welding torch nozzle connector on the welding torch axis, as measured visually. a ,y a ,z a ) to the point to be welded (x b ,y b ,z b The Euclidean distance of ).

[0132] Next, the distance from the end of the welding torch nozzle connector to the point to be welded (length) minus the length of the welding torch nozzle (len) is calculated. weld , which is the distance len from the tip of the welding torch nozzle to the point to be welded. This distance is also the wire extension, and satisfies the following formula.

[0133] len = length - len weld

[0134] Where len is the final calculated wire extension. weld This refers to the length of the welding torch nozzle.

[0135] according to Figure 9 (a) Figure 10 (a) Figure 11 (a) Figure 12 (a) Figure 13 (a) Figure 14 (a) Figure 15(a) Figure 16 (a) Figure 17 (a) It is known that taking the measurement result of a single frame as the final result of binocular vision measurement results in significant inaccuracies and instabilities. The experimental results from 100 measurement experiments are shown in the table below:

[0136] Table 1 shows the experimental results when the measurement results of a single frame were used as the final result of the binocular vision measurement.

[0137]

[0138]

[0139] according to Figure 18 It is known that averaging multiple measurements as the final result can effectively reduce measurement error, decrease the standard deviation, and suppress abrupt changes in the measurement results. Furthermore, according to... Figure 19 It is known that although the measurement error decreases as the number of frames used to calculate the average increases, the time required to obtain a single measurement result also increases significantly. Therefore, to balance measurement error and measurement time, the average measurement value per 40 frames is selected as the final result of the binocular vision measurement. The results of 50 consecutive measurements are as follows: Figure 9 (b) Figure 10 (b) Figure 11 (b) Figure 12 (b) Figure 13 (b) Figure 14 (b) Figure 15 (b) Figure 16 (b) Figure 17 As shown in (b), the experimental results are as follows:

[0140] Table 2 shows the experimental results when the average measurement value per 40 frames was used as the final result of the binocular vision measurement.

[0141] Average value (cm) 1.88 2.55 3.04 3.56 4.21 4.47 4.85 5.49 5.75 Maximum value - Minimum value (cm) 0.60 0.71 0.49 1.00 1.26 1.41 1.05 0.54 1.90 Standard deviation 0.13 0.15 0.11 0.21 0.26 0.31 0.19 0.14 0.35

[0142] Compared to the above experiments, selecting the average measurement value from every 40 frames as the final result of binocular vision measurement is better than using the measurement value from a single frame, both in terms of fluctuation range and dispersion. Furthermore, the measurement time is within 3 to 4 seconds, meeting the requirements of speed and stability. The specific process is as follows: Figure 4 As shown.

[0143] See Figure 5 and Figure 6 The present invention also provides an initial weld extension measurement device based on binocular vision, comprising a frame-synchronized binocular camera module, a camera fixture, a computer, an illumination module, a welding torch, and a device for moving the welding torch, wherein:

[0144] The frame-synchronized binocular camera module has a focal length of 3.0mm, a baseline of 63mm, and an image resolution of 640×480, and is used for raw image acquisition.

[0145] The camera clamp is used to fix the relative position between the binocular camera module and the welding torch, so that the baseline of the camera is approximately perpendicular to the axis of the welding torch on a certain projection plane, and the camera is fixed to the side of the welding torch's forward direction. This design reduces the number of parameters that need to be estimated separately, simplifies the measurement algorithm, improves measurement accuracy, and speeds up the algorithm's calculation speed.

[0146] The computer is used for real-time image acquisition, image processing, feature matching, stereo matching, and key point estimation and recognition to complete the measurement of the initial weld extension.

[0147] The lighting module is fixed to the device used to move the welding torch, thereby improving ambient lighting and increasing measurement accuracy.

[0148] The welding torch is a Panasonic MIG / MAG welding torch. The nozzle outer diameter is 24mm, and the overall length is 73mm. The nozzle connector is a standard brass-colored nozzle connector. The welding torch and camera clamp are fixed together on a device for moving the welding torch. The welding torch, camera, and lighting module maintain a fixed relative position.

[0149] In summary, by using the method of this invention during the welding process, welders or robots can quickly and stably obtain the welding extension amount before welding, which helps to autonomously adjust working parameters based on the extension amount, improve the welding quality and efficiency of the robot, and increase the automation level of machine welding, thereby reducing the labor intensity of welders.

[0150] Based on the above analysis, the method provided by this invention can help welders or robots quickly and stably obtain the initial welding extension during the welding process. The extension measured by the method of this invention can be used to autonomously adjust working parameters or to autonomously adjust the extension according to welding working parameters, which can improve the welding quality and efficiency of robots, significantly enhance the automation level of machine welding, and reduce the labor intensity of welders.

[0151] The above examples are merely one specific embodiment of the present invention. Obviously, the present invention is not limited to the above embodiments, and many similar modifications are possible. All modifications directly derived or conceived by those skilled in the art from the disclosure of this invention should be considered within the scope of protection of this invention.

Claims

1. A method for measuring initial weld extension based on binocular vision, characterized in that, The method includes the following steps: Step 1, Camera Calibration; Obtain the internal and external parameters of each camera module and the structural parameters between the two camera modules in the binocular camera system; Step 2, Image Acquisition and Image Processing: The acquired raw RGB image is subjected to distortion correction and stereo correction to obtain distortion-free and coplanar aligned RGB and grayscale images from the left and right perspectives. Step 3: Stereo matching and obtaining disparity map and depth map; use the semi-global block matching algorithm SGBM to calculate binocular disparity, and then obtain point cloud map and depth map, aligning the left view of the binocular camera with the depth map; Step 4, adaptive positioning of the welding torch and welding torch axis; by positioning the point at the end of the welding torch nozzle joint located on the welding torch axis on the image, the welding torch axis is positioned, and the dry extension is indirectly measured by calculating the distance from the point at the end of the welding torch nozzle joint located on the welding torch axis to the point to be welded. Step 5: Estimate the pixel position of the point to be welded in the image; extract the ROI of the welding area, determine the point to be welded by the intersection of the extension line of the welding gun axis and the weld seam, and finally convert the coordinates of the point to be welded on the ROI image into pixel coordinates on the left-view image. Step 6: Key point coordinate transformation and distance calculation; the two-dimensional coordinates of the point to be welded and the key points of the welding gun are transformed into three-dimensional coordinates. By calculating the Euclidean distance between the two points in three-dimensional space, after excluding outliers, the average measurement value of every 40 frames is taken as the final result of the initial welding extension of the binocular vision measurement.

2. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 1 includes: Step 1.1: The two cameras on the binocular camera module simultaneously capture images of the checkerboard calibration board at different positions; Step 1.2: Perform camera calibration and stereo calibration using OpenCV; obtain the intrinsic and extrinsic parameters, distortion parameters, and structural parameters between the two cameras for each camera module, including the rotation matrix and translation vector between the two camera coordinate systems.

3. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 2 includes: Step 2.1: Convert the two three-channel RGB images from the left and right perspectives of the binocular camera into two single-channel grayscale images; Step 2.2: Use the distortion parameters obtained from camera calibration to correct the distortion of the two grayscale images respectively; Step 2.3: Use the camera intrinsic parameters obtained from camera calibration and the structural parameters between the binocular cameras to perform stereo correction on the two grayscale images, and obtain binocular images with coplanar alignment from the left and right perspectives.

4. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 3 includes: Step 3.1: Calculate the disparity in binocular vision using the SGBM algorithm; Step 3.2: Using the disparity image and the reprojection matrix obtained during image stereo correction, a mapping map is obtained. This mapping map is a three-channel image of the same size as the disparity image. Each channel stores the value of the pixel position on the X-axis, Y-axis and Z-axis in the camera coordinate system, that is, the three-dimensional coordinates (x, y, z) of each pixel in the camera coordinate system. The value of z represents the distance from the object to the binocular camera plane, in millimeters.

5. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 4 includes: Step 4.1: Convert the corrected left-view image into an HSV color gamut image. By adjusting the H, S, and V thresholds, segment the nozzle connector part according to color and display it on the binary image. Step 4.2: Filter small connected regions to remove noise, and then use a closing operation to make the segmented image of the nozzle connector more complete; obtain the minimum bounding rectangle of the contour by the contour of the segmented image. Step 4.3: Calculate the midpoints of the two sides of the smallest bounding rectangle that are perpendicular to the welding torch axis. The line connecting these two midpoints is the straight line containing the welding torch axis. Solve the equation of the straight line containing the welding torch axis in the left view pixel coordinate system. Record the pixel coordinates of the midpoint that is closer to the point to be welded in the corrected left view.

6. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 5 includes: Step 5.1: Draw the rectangle of the welding area ROI on the corrected left view; draw a line segment perpendicular to the welding gun axis through the vertex of the welding gun in the left view image, which is the width of the welding area ROI. The vertex of the welding gun is the midpoint of this line segment. Based on the width of the welding area ROI, the long side of the welding area ROI is parallel to the welding gun axis. Step 5.2, trim the welding area ROI; rotate the left view so that the rotation angle of the welding area ROI is 0; trim the welding area ROI, and the center line of the trimmed rectangular area is the axis of the welding gun. Step 5.3, extract the point to be welded; convert the cropped ROI image to grayscale, use the Canny algorithm to extract the weld, and record the intersection of the vertical center line of the weld and the ROI. This intersection is the point to be welded. Step 5.4: Calculate the pixel coordinates of the point to be soldered; using the reverse operation of the step of trimming the ROI, convert the pixel coordinates of the point to be soldered on the trimmed ROI image into pixel coordinates on the corrected left view.

7. The method for measuring initial weld extension based on binocular vision according to claim 1, characterized in that, Step 6 includes: Step 6.1: Calculate the three-dimensional coordinates of the point at the end of the welding torch nozzle connector located on the welding torch axis and the point to be welded in the camera coordinate system; the formula for calculating the Euclidean distance between the two points in three-dimensional space is as follows: Among them, (x a ,y a ,z a (x) represents the three-dimensional coordinates of a point located at the end of the welding torch nozzle connector on the welding torch axis in the camera coordinate system. b ,y b ,z b ) represents the three-dimensional coordinates of the point to be welded in the camera coordinate system, and length represents the point (x, y) located at the end of the welding torch nozzle connector on the welding torch axis as measured visually. a ,y a ,z a ) to the point to be welded (x b ,y b ,z b The Euclidean distance of ) Step 6.2: The distance from the end of the welding torch nozzle connector on the welding torch axis to the point to be welded is length minus the length of the welding torch nozzle. weld , which is the distance len from the nozzle tip to the welding point, is also the wire extension, and satisfies the following formula. len=length-len weld Where len is the final calculated wire extension. weld This refers to the nozzle length; Step 6.3: Determine if the measured distance len in the previous step is within 0 cm to 7.5 cm; otherwise, skip directly to the next frame for measurement. After measuring 40 frames, take the average value of these 40 frames as the final result of the binocular vision measurement.

8. A binocular vision-based initial weld extension measurement device, used to implement the binocular vision-based initial weld extension measurement method according to any one of claims 1-7, characterized in that, It includes a frame-synchronized binocular camera module, a camera fixture, a computer, a lighting module, a welding torch, and a device for moving the welding torch, wherein: The frame-synchronized binocular camera module has a focal length of 3.0mm, a baseline of 63mm, and an image resolution of 640×480, and is used for raw image acquisition. The camera clamp is used to fix the relative position between the binocular camera module and the welding torch, so that the baseline of the camera is approximately perpendicular to the axis of the welding torch on the imaging plane of the binocular camera, and to fix the camera to one side of the direction of the welding torch's movement. The computer is used for real-time image acquisition, image processing, feature matching, stereo matching, and key point estimation and recognition to complete the measurement of the initial weld extension. The lighting module is fixed to the device for moving the welding torch and is used to improve ambient lighting; The welding torch is a Panasonic MIG / MAG welding torch; the nozzle has an outer diameter of 24mm and a total length of 73mm; the nozzle connector is a standard brass nozzle connector; the welding torch and camera clamp are fixed together on the device for moving the welding torch; the welding torch, camera, and lighting module are kept in a fixed relative position.

Images