Grouped straight line welding start and end point recognition method using machine vision

Abstract

The application discloses a small-group linear welding start and end point identification method using machine vision in the field of robot intelligent welding, and the welding assembly is provided with a welding gun, a motion mechanism and an ArUco characteristic code plate. Three ArUco characteristic codes with ids of 0, 1 and 2 are arranged side by side on the ArUco characteristic code plate, and two ArUco characteristic codes with ids of 3 and 4 are arranged on a small-group workpiece to be welded. A 3D vision camera takes a picture, if the ArUco characteristic code with id of 0 is found, the welding assembly is identified, if the ArUco characteristic code with id of 2 is found, the end of the welding gun is positioned, the vector composed of the ArUco characteristic codes with ids of 0, 1 and 2 is the motion direction of the welding assembly, and the vectors composed of the ArUco characteristic codes with ids of 3 and 4 are respectively the welding start point and end point positions of the workpiece to be welded. The 3D vision camera is far away from the welding gun, the positions and states of the workpieces to be welded do not need to be changed, the start and end point setting mode is flexible, the identification speed is fast, and the welding assembly and the workpiece can be identified and positioned simultaneously.

Description

Technical Field

[0001] This invention belongs to the field of robotic intelligent welding technology, specifically a small-group vertical linear welding method using machine vision. Background Technology

[0002] Small group vertical welding is widely used in the manufacturing and maintenance of steel structures, ships and bridges. For example, in the small group vertical straight welding work in shipyards, most welding work is still in the stage of manual and semi-automatic operation. The welding work has high technical repetition, high labor intensity, and high difficulty in ensuring personnel safety. In addition, the welding quality is directly affected by the welder's skills, making it difficult to stabilize the welding quality of small group vertical welding.

[0003] Current automated welding technologies typically include weld seam identification and tracking methods, weld seam start and end point identification methods, and welding torch control methods. Among these, weld seam start and end point identification methods are an important component in improving the level of welding automation. When identifying the start and end points of welds, components such as sensors, controllers, and welding equipment are typically used. Sensors can be categorized by type, with typical sensors including ranging sensors, 2D vision image sensors, and 3D scanning sensors. They can also be categorized by the type of welding object and weld, such as arc-shaped welds on pipes and straight-line welds on right-angle steel plates. For example, the starting point identification and positioning method disclosed in Chinese patent application number 2015100770966, entitled "A CCD-based Method for Identifying and Locating the Starting Point of Large Marine Pipelines," uses a 2D vision image sensor to perform arc-shaped welds on pipes. In its workflow, the pipe is required to rotate while the welding equipment and CCD sensor remain stationary. This method has the following problems: 1. It is only suitable for welding rotatable pipes. When applied to shipbuilding, the weldment cannot be rotated, making it difficult to achieve group stand-up welding; 2. During the welding process, the CCD sensor is affected by welding fumes, causing identification failures and reducing system reliability. The method disclosed in Chinese Patent Application No. 2015104084617, entitled "Method for Identifying and Positioning the Starting Point of a Fillet Weld Based on an Ultrasonic Sensor," uses an ultrasonic sensor, specifically a rangefinder, to achieve straight-line welding of right-angled steel plates. Its working principle involves aligning the ultrasonic sensor with the steel plate and measuring the distance between the sensor and the plate. However, this method has the following problems: 1. If the sensor moves outside the steel plate, the distance reading will change significantly; 2. Due to the small measurement range of the ultrasonic sensor, it needs to be close to the welding object, i.e., close to the steel plate. During the welding process, the sensor is exposed to high welding temperatures, weld slag, and welding fumes, which can cause sensor failure and reduce system reliability. The method disclosed in Chinese Patent Application No. 2023115163922, entitled "Positioning Method, Device, and Electronic Equipment Based on Laser Sensing," uses a laser sensor, specifically a rangefinder. It requires prior knowledge of the approximate position of the welding target point and focuses on using the laser sensor to guide the robot for high-precision positioning, avoiding collisions between the robot and the workpiece. This method requires manual input of the approximate directions of the welding start and end points, while the robot uses a laser sensor for precise positioning, which still requires human intervention. At the same time, the laser sensor is mounted on the robot and is close to the welding torch, making it susceptible to the effects of high welding temperatures and fumes during welding, which is detrimental to ensuring the reliability of the system. Summary of the Invention

[0004] The purpose of this invention is to solve the many problems existing in the prior art and to propose a method for identifying the start and end points of vertical straight welding using a 3D vision camera and machine vision technology, so as to improve the reliability of the system.

[0005] To achieve the above objectives, the present invention employs the following technical solution for a machine vision-based method for identifying the start and end points of small-group vertical straight welding:

[0006] Its characteristics include the following steps:

[0007] Step 1): The welding assembly has a welding torch, a motion mechanism and an ArUco feature code plate. The motion mechanism drives the ArUco feature code plate and two welding torches arranged symmetrically relative to the ArUco feature code plate to move synchronously. The ArUco feature code plate is set with three ArUco feature codes with IDs of 0, 1 and 2 in parallel. The workpiece to be welded is set with two ArUco feature codes with IDs of 3 and 4.

[0008] Step 2): The 3D vision camera takes a picture and transmits the 2D photo and 3D point cloud signal to the industrial control computer. The industrial control computer performs ArUco feature code recognition on the 2D photo to obtain the id set ids of ArUco feature codes. If an ArUco feature code with a value of 0 is found in the id set ids, the welding component is identified.

[0009] Step 3): Find the ArUco feature code a3 with id 2 from the id set ids, obtain the pixel number corresponding to ArUco feature code a3 in the 2D image and the three-dimensional coordinates of the point corresponding to it in the 3D point cloud, and then locate the end of the welding gun using the three-dimensional coordinates.

[0010] Step 4): Find the ArUco feature codes a1, a2, and a3 with IDs of 0, 1, and 2 from the ID set ids. Obtain the pixel numbers corresponding to the ArUco feature codes a1, a2, and a3 in the 2D image and the three-dimensional coordinates of the three points corresponding to them in the 3D point cloud. Transform a straight line passing through the three-dimensional coordinates of the three points to obtain two candidate motion direction vectors. Calculate the angle between the two candidate motion direction vectors and the roughly estimated direction vector. Take the vector corresponding to the smaller angle as the motion direction of the welding component.

[0011] Step 5): Find the ArUco feature code a4 with id=3 from the id set ids, obtain the pixel number corresponding to ArUco feature code a4 in the 2D image and its corresponding three-dimensional coordinates in the 3D point cloud, which is the welding start point of the workpiece to be welded; find the ArUco feature a5 with id=3 from the id set ids, obtain the pixel number corresponding to ArUco feature a5 in the 2D image and its corresponding three-dimensional coordinates in the 3D point cloud, which is the welding end point of the workpiece to be welded.

[0012] The beneficial effects of adopting the above technical solution in this invention are as follows:

[0013] 1. High system reliability. The 3D vision camera is located far from the welding torch. During the welding process, the 3D vision camera will not heat up due to the high temperature generated by welding, nor will the camera lens be contaminated by welding fumes, thus preventing sensor damage or failure.

[0014] 2. Flexible start and end point setting. The start and end points of welding are marked using the ArUco feature code on the aluminum substrate, without changing the position and state of the workpiece being welded. This method is flexible, simple, and highly adaptable.

[0015] 3. Fast recognition speed. It adopts a combination of 2D vision and 3D vision, which has low computational complexity and short calculation cycle, enabling high-frequency monitoring.

[0016] 4. Strong system security. This invention uses 3D vision cameras and machine vision technology to identify, locate, and estimate the motion state of welding components, as well as to identify and locate the workpiece to be welded. It can simultaneously identify and locate both welding components and workpieces. Attached Figure Description

[0017] Figure 1 This is a simplified structural diagram of the welding robot used to implement the method for identifying the start and end points of vertical straight welding in this invention.

[0018] Figure 2 yes Figure 1 Enlarged structural schematic diagram of the welding assembly;

[0019] Figure 3 yes Figure 2 Arrangement diagram of the three ArUco signature plates on the welding assembly;

[0020] Figure 4 This is a flowchart of the method for identifying the start and end points of linear welding in this invention. Detailed Implementation

[0021] like Figure 1 and Figure 2 As shown, the welding robot includes a robot support 3, welding components 2, and a 3D vision camera 1. The robot support 3 supports all components of the welding robot, and both the welding components 2 and the 3D vision camera 1 are connected to the robot support 3. To ensure the reliability of the robot support 3, a gantry structure is adopted. To ensure the portability of the robot support 3, heavy-duty casters are installed at the bottom of the gantry, and handling handles are installed on both sides of the gantry.

[0022] like Figure 3 andFigure 4 As shown, welding assembly 2 is used to perform welding actions and includes welding torches 22, a motion mechanism 21, and an ArUco feature code plate 23. Two welding torches 22 and one ArUco feature code plate 23 are connected to the motion mechanism 21, with the two welding torches 22 arranged symmetrically relative to the ArUco feature code plate 23. The motion mechanism 21 drives the welding torches 22 and the ArUco feature code plate 23 to move synchronously. The welding torches 22 and the motion mechanism 21 are respectively connected to an industrial control computer via an electrical control system. The industrial control computer sends commands, which are then used by the electrical control system to control the operation of the welding torches 22 and the movement of the motion mechanism 21. The ArUco feature code plate 23 is made using an aluminum substrate anodizing process. Three ArUco QR codes are arranged side by side on the ArUco feature code plate 23, and all three ArUco QR codes belong to the dictionary "DICT_5×5_50". The IDs of the three ArUco QR codes are id=0, id=1, and id=2, respectively. Of these three ArUco QR codes, the ArUco feature code with id 0 contains two functions: a welding component identification mark and a motion direction vector mark. The ArUco feature code with id 1 functions as a motion direction vector mark, and the ArUco feature code with id 2 functions as a positioning mark and a motion direction vector mark. The welding component identification mark is used for identifying the welding component, the positioning mark is used for estimating the position of the welding component, and the motion direction vector mark is used for estimating the motion direction vector of the welding component.

[0023] 3D vision camera 1 is a weld start and end point sensing system used to determine whether the welding torch has reached the welding start and end points. It uses a 3D vision camera as a scene perception device. 3D vision camera 1 is connected to an industrial control computer via a signal line. The 3D vision camera 1 transmits the sensed information to the industrial control computer, which then uses mathematical methods such as recognition and positioning to identify the welding start and end points.

[0024] The industrial control computer includes a host computer and a PLC (Programmable Logic Controller). The host computer is used to execute mathematical algorithms related to the 3D vision camera 1 to identify and locate the welding component 2, as well as to identify and locate the welding start and end points. The PLC is used to connect the welding component 2 and the 3D vision camera 1, and to control the start and stop of the welding component 2 and the 3D vision camera 1.

[0025] like Figure 1 The workpiece 4 to be welded is assembled into a group, including the workpiece body and a welding start and end point marking fixture. The welding start and end point marking fixture uses ArUco feature codes and is manufactured using an aluminum substrate anodizing process. Two ArUco feature codes are used, both belonging to the dictionary "DICT_5×5_50", to represent the start and end points of the welding, respectively. Among them, the ArUco feature code representing the welding start point has an ID of 3; the ArUco feature code representing the welding end point has an ID of 4.

[0026] In the industrial control computer, the three ArUco feature codes on welding assembly 2 and the two ArUco feature codes on the workpiece 4 to be welded, totaling five ArUco feature codes, are combined into a set A and stored. Tool The set A of two ArUco signatures on the workpiece 4 to be welded Target And save it, defined as follows:

[0027] A = {a1, ..., a5}

[0028] A Tool ={a1, a2, a3}

[0029] A Target ={a4, a5},

[0030] Where a1 represents the first ArUco feature code on welding assembly 2, a2 ​​represents the second ArUco feature code on welding assembly 2, a3 represents the third ArUco feature code on welding assembly 2, a4 represents the first ArUco feature code on workpiece 4 to be welded, and a5 represents the second ArUco feature code on workpiece 4 to be welded. Sets A and T Tool A Target It exists in the industrial control computer.

[0031] The five ArUco signatures are numbered 1, 2, 3, 4, and 5 respectively. These five ArUco signature numbers form a set N, which is stored in the industrial control computer. The three ArUco signature numbers on welding assembly 2 form another set N. Tool The two ArUco feature codes on the workpiece 4 to be welded form a set N, which is stored in the industrial control computer. Target And it is stored in the industrial computer, defined as follows:

[0032] N = {1, ..., 5}

[0033] N Tool ={1, 2, 3}

[0034] N Target ={4, 5}.

[0035] The IDs of all ArUco signatures, namely the IDs of the ArUco signatures on welding assembly 2 and the IDs of the two ArUco signatures on workpiece 4 to be welded, are each formed into a set and stored in the industrial control computer, as defined below:

[0036] ID ArUco ={id n; n∈N, id1=0, id2=1, id3=2, id4=3, id5=4}

[0037] ID ArUco，Tool ={id n ; n∈N Tool}

[0038] ID ArUco，Target ={id n ;n∈NT arget},

[0039] In a 3D vision camera, based on the camera's inherent functions, each shot captures a 2D image and a 3D point cloud. The 2D pixel numbers are the same as the point numbers in the 3D point cloud, and these are set together as I... set The definition is as follows:

[0040] I set ={1, ..., i, ..., I},

[0041] In the above formula, I set Let I represent the set of pixel numbers; i represents any pixel number in the set of pixel numbers; 1 represents the last pixel number in the set of pixel numbers. set It is stored in the industrial control computer.

[0042] 2D photos are RGB photos, where pixels form a set called P. set，2D The definition is as follows:

[0043] P set，2D ={p 2D，i ;p 2D，i ≡(R i G i B i )},i∈I set ,

[0044] Among them, P set，2D p represents the set of all pixels in a 2D photograph. 2D，i This represents the pixel numbered i in a 2D photograph. 2D photographs are RGB images, using the RGB color mode. i G i B i These represent the values ​​of the red, green, and blue channels for pixel number i, respectively. Set P... set，2D It is stored in the industrial control computer.

[0045] A 3D point cloud is a collection of points, called P. set，3D The definition is as follows:

[0046]

[0047] Among them, P set，3D p represents the set of all points in a 3D point cloud. 3D，i This represents point i in the 3D point cloud. The 3D point cloud is obtained by scanning with a 3D vision camera, and each point contains a 3D coordinate, which is in the coordinate system of the 3D vision camera. Therefore, Cam x i The first coordinate component of the point numbered i in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i The second coordinate component represents the point numbered i in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i This represents the third coordinate component of point i in the 3D point cloud, with its top-left subscript Cam indicating the coordinate system of the 3D vision camera. The above set P... set，3D Stored in the industrial computer. It represents a three-dimensional real space, where the superscript Cam indicates the coordinate system of the 3D vision camera.

[0048] ArUco signatures are used to identify the start and end points of welding. The workflow is as follows: Figure 4 As shown, its working steps are as follows:

[0049] Step 1: Start the welding robot. The motion mechanism 22 of welding component 2 begins operation, the welding torch 22 is powered on, the 3D vision camera 1 is activated, and the start signal is transmitted to the industrial control computer. The industrial control computer stores the following set: A, A Tool A Target , N, N Tool N Target ID ArUco ID ArUco，Tool ID ArUco，Target .

[0050] Step 2: During the operation of welding component 2, 3D vision camera 1 identifies welding component 2, represented by the ArUco feature code with id=0. If the ArUco feature code with id=0 is identified, it indicates that welding component 2 has been identified. The set used in this step is as follows: I set P set，2D A Tool ID ArUco，Tool .

[0051] To ensure continuous monitoring of welding component 2, the industrial control computer (ICC) is required to perform periodic calculations at a certain frequency, typically 10Hz. The 3D vision camera 1 initiates image capture and scanning, transmitting the 2D image signal to the ICC. The ICC obtains the 2D image from the 3D vision camera 1 and executes the ArUco signature recognition algorithm on it. This algorithm is defined in the `cv::aruco::ArucoDetector::detectMarkers(InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArrayOfArraysrejectedImgPoints = noArray())` function from the `aruco_detector.hpp` file in the OpenCV library. It then obtains a set of ArUco signature IDs named "ids". The ICC searches for elements with a value of 0 in the set "ids". If an element with a value of 0 exists in the set "ids", welding component 2 is identified, and the identification process is complete. If no element with a value of 0 exists in the set "ids", the welding component is not identified, and the program issues an alarm.

[0052] Step 3: After identifying welding component 2, estimate its position. The ArUco feature code with id 2 represents the end of the welding torch for welding component 2. This ArUco feature code with id 2 remains rigidly fixed to the end of the welding torch. The industrial control computer stores the spatial transformation relationship between the ArUco feature code with id 2 and the end of the welding torch. If the ArUco feature code with id 2 is identified and located, it indicates that the end of the welding torch has been located.

[0053] The set used in this step is as follows: I set P set，2D P set，3D A, A Tool ID ArUco ID ArUco，Tool .

[0054] To ensure continuous monitoring of the position of welding component 2, the industrial control computer performs periodic calculations at a certain frequency, typically 10Hz. The 3D vision camera 1 initiates image capture and scanning, transmitting the data to the industrial control computer in the form of 2D images and 3D point clouds. The industrial control computer obtains the 2D images and 3D point clouds from the 3D vision camera 1 and establishes the correspondence between pixel numbers in the 2D images and point numbers in the 3D point clouds. It then executes the ArUco feature code recognition algorithm on the 2D images. This algorithm is defined by the function `cv::aruco::ArucoDetector::detectMarkers(InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArrayOfArraysrejectedImgPoints = noArray())` in the `aruco_detector.hpp` file of the OpenCV library, obtaining a set of ArUco feature code IDs named "ids". The element with a value of 2 is then selected from the ID set "ids" and defined as ArUco feature code a3.

[0055] a3∈A Tool id3=2∈ID ArUco，Tool

[0056] Obtain the pixel number corresponding to ArUco feature code a3 in the 2D image, defined as i. This obtained pixel number is denoted by i. Since the function can recognize the four corners of the ArUco feature code, this invention uniformly uses the top-left corner of the ArUco feature code to represent the ArUco feature code body. Obtain the point corresponding to ArUco feature code a3 in the 3D point cloud and find the corresponding three-dimensional coordinates, defined as p. 3D，i ; with p 3D，i express;

[0057] p 3D，i =( Cam x i , Cam y i Cam z i ),

[0058] During the movement of welding component 2, its three-dimensional coordinates p 3D，i The position of the end of the welding torch 22 in welding assembly 2 remains unchanged, that is, the position of the end of the welding torch 22 is p. 3D，i A spatial transformation of is represented as follows:

[0059]

[0060] Where, p ToolThis indicates the coordinates of the end of the welding torch 22 in the coordinate system of the welding assembly 2. This represents the spatial transformation matrix between the ArUco feature code plate 23 and the end of the welding torch 22. The upper left subscript "Tool" indicates the coordinate system of the welding assembly 2, and "Marker" indicates the coordinate system of the ArUco feature code plate 23. This matrix remains unchanged and is known during the welding process. 3D，i This represents the coordinates of the point corresponding to the ArUco feature code a3 in the 3D point cloud, where, Tool Marker T represents the spatial transformation matrix between the ArUco feature code plate 23 and the end of the welding torch 22. This matrix remains unchanged during the welding process and is known.

[0061] Step 4: Estimate the motion direction vector of welding assembly 2. The motion direction vector of welding assembly 2 is estimated using ArUco feature code board 23. ArUco feature code board 23 is consistent with the movement direction of welding assembly 2. The vector composed of ArUco feature codes with id values ​​of 0, 1, and 2 represents the motion direction of welding assembly 2.

[0062] The set used in this step is as follows: I set P set，2D P set，3D A, A Tool ID ArUco ID ArUco，Tool .

[0063] To ensure continuous monitoring of the motion direction vector of welding assembly 2, 3D vision camera 1 is required to perform periodic calculations at a certain frequency, typically 10Hz. The work content of each cycle is as follows:

[0064] The 3D vision camera 1 starts taking pictures and scanning, transmitting the data to the industrial control computer. The data includes 2D photos, 3D point clouds, and the correspondence between pixel numbers in the 2D photos and point numbers in the 3D point clouds. The industrial control computer executes the ArUco feature code recognition algorithm on the 2D photos. This algorithm is the `cv::aruco::ArucoDetector::detectMarkers(InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArrayOfArrays rejectedImgPoints = noArray())` function in the `aruco_detector.hpp` file of the OpenCV library, and obtains the set of ArUco feature codes `ids`, named "ids". Elements with values ​​of 0, 1, and 2 are found from the `ids` set and defined as ArUco feature codes a1, a2, and a3 as follows;

[0065] a1, a2, a3∈A, id1=0, id2=1, id3=2∈ID ArUco

[0066] Obtain the pixel numbers corresponding to ArUco feature codes a1, a2, and a3 in the 2D image, and define them as i1, i2, and i3 respectively. Since the function can identify the four corners of the ArUco feature code, this invention uniformly uses the upper left corner of the ArUco feature code to represent the ArUco feature code body.

[0067] Obtain the points corresponding to a1, a2, and a3 in the 3D point cloud, and find their corresponding 3D coordinates, defined as p. 3D，i as follows;

[0068] p 3D，iI =( Cam x i1 , Cam y i1 , Cam z i1 )

[0069] p 3D，i2 =( Cam x i2 , Cam y i2 , Cam z i2 )

[0070] p 3D，i3 =( Cam x i3 , Cam y i3 , Cam z i3 )

[0071] Where, p 3D，i1 p represents the coordinates of the point corresponding to the ArUco feature code a1 in the 3D point cloud. 3D，i2 p represents the coordinates of the point corresponding to the ArUco feature code a2 in the 3D point cloud. 3D,i3 This represents the coordinates of the point corresponding to the ArUco feature code a3 in the 3D point cloud. Cam x i1 The first coordinate component of the point numbered i1 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i1 The second coordinate component represents the point numbered i1 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i1 The third coordinate component represents the point numbered i1 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Camx i2 The first coordinate component of the point numbered i2 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i2 The second coordinate component represents the point numbered i2 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i2 The third coordinate component represents the point numbered i2 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam x i3 The first coordinate component of the point numbered i3 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i3 The second coordinate component represents the point numbered i3 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i3 The third coordinate component represents the point numbered i3 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera.

[0072] Suppose a straight line theoretically passes through p 3D,i1 p 3D,i2 p 3D,i3 However, due to errors in actual observation, namely:

[0073]

[0074] Where, p 3D,i This represents the coordinates of point i in the 3D point cloud. The theoretical value of the first coordinate component of point i in the 3D point cloud is the measurement result under the condition of no observation error of the 3D vision camera. Its upper left subscript Cam indicates the coordinate system of the 3D vision camera. The second coordinate component representing the point numbered i in the 3D point cloud is the measurement result under the condition of no observation error of the 3D vision camera. Its upper left subscript Cam indicates the coordinate system of the 3D vision camera. The third coordinate component represents the point i in the 3D point cloud, and is the measurement result under the condition of no observation error from the 3D vision camera. Its upper-left subscript Cam indicates the coordinate system of the 3D vision camera. ε represents the random error, and this error follows a Gaussian distribution with an expected value of 0. Here, ~ indicates that it follows a certain distribution. Let μ represent a Gaussian distribution, and let μ be the expected value (which is 0) and σ be the standard deviation.

[0075] Here, ε represents the random error, and this error follows a Gaussian distribution with an expected value of 0. Let the equation of the line be as follows;

[0076]

[0077] Where x, y, and z represent the independent variables in the equation of the line, x0, y0, and z0 represent the coordinates of a point on the line, and m, n, and p represent the three parameters of the equation of the line.

[0078] The equation of the transformed line is as follows;

[0079] (y-y0)m+(x0-x)n=0

[0080] (z-z0)m+(x0-x)p=0

[0081] Where x, y, and z represent the independent variables in the equation of the line, x0, y0, and z0 represent the coordinates of a point on the line, and m, n, and p represent the three parameters of the equation of the line.

[0082] p 3D，i1 Given x0, y0, and z0, we obtain:

[0083] x0 = Cam x i1 y0 = Cam y i1 z0 = Cam z i1 ,

[0084] p 3D，i2 P 3D，i3 Substituting these equations into the system of linear equations, we obtain the system of linear equations.

[0085]

[0086] The matrix K is represented as follows;

[0087]

[0088] Cam x i1 The first coordinate component of the point numbered i1 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i1 The second coordinate component represents the point numbered i1 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i1 The third coordinate component represents the point numbered i1 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam x i2The first coordinate component of the point numbered i2 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i2 The second coordinate component represents the point numbered i2 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i2 The third coordinate component represents the point numbered i2 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam x i3 The first coordinate component of the point numbered i3 in the 3D point cloud is represented by its upper left subscript Cam, which indicates the coordinate system of the 3D vision camera. Cam y i3 The second coordinate component represents the point numbered i3 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera. Cam z i3 The third coordinate component represents the point numbered i3 in the 3D point cloud, and its upper left subscript Cam indicates the coordinate system of the 3D vision camera.

[0089] Using the least squares method for estimation, then [m, n, p] T For K T The eigenvector corresponding to the smallest eigenvalue of K;

[0090] Obtain two candidate motion direction vectors;

[0091] r Tool，1 = (m, n, p)

[0092] r Tool，2 = (-m, -n, -p)

[0093] The roughly estimated direction vector is set as follows;

[0094]

[0095] The angles between the two candidate motion direction vectors and the roughly estimated direction vectors are calculated separately.

[0096]

[0097]

[0098] Where, θ Tool，1 represents the angle between the first candidate motion direction vector and the coarse direction vector, and arccos represents the inverse cosine function. Tool，1 Represents the first candidate motion direction vector, |r Tool，1 | represents the magnitude of the first candidate motion direction vector. rTool，2 Represents the first candidate motion direction vector, |r Tool，2 | represents the magnitude of the second candidate motion direction vector. This represents a rough direction vector. This represents the magnitude of a rough direction vector.

[0099] The vector corresponding to the smaller included angle is taken as the motion direction vector of welding component 2 as follows:

[0100]

[0101] Step 5: Estimate the welding start point position of the workpiece 4 to be welded. The ArUco feature code plate 23 is used to estimate the motion direction vector of the welding assembly 2. The ArUco feature code plate 23 is consistent with the movement direction of the welding assembly 2. The vector composed of ArUco feature codes with id of 3 represents the welding start point position of the workpiece 4 to be welded.

[0102] The set used in this step is as follows: I set P set，2D P set，3D A, A Tool A Target ID ArUco ID ArUco，Tool ID ArUco，Target .

[0103] To ensure continuous monitoring of the motion direction vector of welding assembly 2, 3D vision camera 1 is required to perform periodic calculations at a certain frequency, typically 10Hz. The work content of each cycle is as follows:

[0104] 3D vision camera 1 initiates image capture and scanning, which is completed by the 3D vision camera itself. The data is transmitted to the industrial control computer, consisting of a 2D image, a 3D point cloud, and the correspondence between pixel numbers in the 2D image and point numbers in the 3D point cloud. The industrial control computer obtains the 2D image and 3D point cloud from 3D vision camera 1, and also obtains the correspondence between pixel numbers in the 2D image and point numbers in the 3D point cloud. The ArUco signature recognition algorithm is executed on the 2D image. This algorithm is the `cv::aruco::ArucoDetector::detectMarkers(InputArray image, OutputArrayOfArrays comers, OutputArray ids, OutputArrayOfArrays rejectedImgPoints = noArray())` function in the `aruco_detector.hpp` file of the OpenCV library, obtaining a set of ArUco signature IDs named "ids". The element with a value of 3 is found from the set "ids" and defined as ArUco signature a4 as follows;

[0105] a4∈A Target id4=3∈ID ArUco，Target ,

[0106] Obtain the pixel number corresponding to a4 in the 2D image, and define it as i4. Since the function can identify the four corners of the ArUco feature code, the upper left corner of the ArUco feature code is used to represent the ArUco feature code body.

[0107] Obtain the point corresponding to a4 in the 3D point cloud and find its corresponding 3D coordinates, defined as p. 3D，i4 ,as follows;

[0108] p 3D，i1 =( Cam x i4 , Cam y i4 , Cam z i4 ).

[0109] p 3D，i4 This is the starting point for welding workpiece 4.

[0110] Step 6: Estimate the welding termination point position of the workpiece 4 to be welded. The ArUco feature code plate 23 is used to estimate the motion direction vector of the welding component 2. The principle is that the ArUco feature code plate 23 and the moving direction of the welding component 2 are consistent. The vector composed of ArUco feature codes with id of 4 represents the welding termination point position of the workpiece 4 to be welded.

[0111] The set used in this step is as follows: Iset P set，2D P set，3D A, A Tool A Target ID ArUco ID ArUco，Tool ID ArUco，Target .

[0112] To ensure continuous monitoring of the motion direction vector of welding assembly 2, 3D vision camera 1 performs periodic calculations at a certain frequency, typically 10Hz. The work content of each cycle is as follows:

[0113] 3D vision camera 1 initiates image capture and scanning, which is completed by the 3D vision camera itself. The data is transmitted to the industrial control computer, consisting of a 2D image, a 3D point cloud, and the correspondence between pixel numbers in the 2D image and point numbers in the 3D point cloud. The industrial control computer obtains the 2D image and 3D point cloud from 3D vision camera 1, and also obtains the correspondence between pixel numbers in the 2D image and point numbers in the 3D point cloud. The ArUco feature code recognition algorithm is executed on the 2D image. This algorithm is the `cv::aruco::ArucoDetector::detectMarkers(InputArray image, OutputArrayOfArrays corners, OutputArray ids, OutputArrayOfArrays rejectedImgPoints = noArray())` function in the `aruco_detector.hpp` file of the OpenCV library, obtaining a set of ArUco feature code IDs named "ids". The element with a value of 4 is found from the "ids" set and defined as ArUco feature code a5.

[0114] a5∈A Target id5 = 4 ∈ ID ArUco，Target ,

[0115] Obtain the pixel number corresponding to a5 in the 2D image, defined as i5. Since the function can recognize the four corners of the ArUco feature code, the top left corner of the ArUco feature code is used to represent the ArUco feature code body. Obtain the point corresponding to a5 in the 3D point cloud and find the corresponding 3D coordinates, defined as p. 3D，i5 ;

[0116] p 3D，i5 =( Cam x i5 , Cam y i5 , Cam z i5 ),

[0117] p 3D，i5This is the welding termination point of workpiece 4 to be welded.

[0118] Step 7: Determine if the tip of the welding torch 22 has reached the welding start point. The set used in this step is as follows: I set P set，2D P set，3D A, A Tool A Target ID ArUco ID ArUco，Tool ID ArUco，Target .

[0119] The method for determining whether the end of the welding torch 22 has reached the welding start point includes two judgment conditions:

[0120] The first criterion is that the distance between the welding start point and the ArUco feature code with id=2 on the welding assembly 22 is minimized. This requires that the plane formed by the ArUco feature code with id=2 and the ends of the two welding torches 22 is perpendicular to the direction of travel of the welding assembly 2. Under this requirement, if the end of the welding torch 22 reaches the welding start point, the welding start point will form a right angle with the ArUco feature code with id=2 and the ArUco feature code with id=1, referred to as θ. Start , means as follows:

[0121]

[0122] Where, θ Start This represents the angle between two vectors, where the first vector (p) 3D，i3 -p 3D，i4 Starting with the ArUco feature code with id=2, which is the third ArUco feature code on welding assembly 2, its 3D coordinates are p. 3D，i3 The ArUco feature code with id 3, which is the first ArUco feature code on the workpiece 4 to be welded, has 3D coordinates p. 3D，i4 , is the vector formed by the endpoints. |p 3D，i3 -p 3D，i4 This represents the modulus of the first vector.

[0123] The second vector (p) 3D，i3 -p 3D，i1 Starting with the ArUco feature code with id=2, which is the third ArUco feature code on welding assembly 2, its 3D coordinates are p. 3D，i3 The ArUco signature with id=0, which is the first ArUco signature on welding assembly 2, has 3D coordinates p. 3D，i1 , is the vector formed by the endpoints.

[0124] |p 3D，i3 -p 3D，i1| represents the modulus of the second vector.

[0125] The determination method is as follows: if θ Start The following conditions must be met:

[0126] θ Start ∈90°-δ,90°+δ]

[0127] Where [90°-δ, 90°+δ] represents a closed interval, with the lower limit being 90°-δ and the upper limit being 90°+δ, where δ represents a constant determined by industry experience. If the first condition is met, the industrial control computer controls the welding assembly 2 to move towards the welding start point.

[0128] The second criterion is that the moving direction of welding component 2 is in the same direction and parallel to the vector formed by the welding start point and welding end point. However, due to the installation error of the ArUco feature code, the two vectors cannot be guaranteed to be strictly parallel. Therefore, the parallelism condition is relaxed to an acute angle, as described below:

[0129]

[0130] Where, α Start Let represent the angle between two vectors, and arccos denote the inverse cosine function. The first vector... This represents a rough direction vector. This represents the magnitude of a coarse direction vector. The second vector (p) 3D，i5 -p 3D，i4 Starting with the ArUco feature code with id 4, which is the second ArUco feature code on the workpiece 4 to be welded, its 3D coordinates are p. 3D，i5 The ArUco feature code with id 3, which is the first ArUco feature code on the workpiece 4 to be welded, has 3D coordinates p. 3D，i4 , is the vector formed by the endpoints. |p 3D，i5 -p 3D，i4 | represents the magnitude of the second vector. (-90°, 90°) represents an open interval with a lower limit of -90° and an upper limit of 90°.

[0131] If both conditions are met simultaneously, the end of the welding torch 22 reaches the welding start point. The industrial control computer transmits the welding start signal of the welding torch 22 and the running signal of the motion mechanism 21 to the welding torch 22 and the motion mechanism 21 of the welding assembly 2. Then the motion mechanism 21 runs and the welding torch 22 starts welding.

[0132] Step 8: Determine whether the end of the welding torch 22 has reached the welding termination point. The set used in this step is as follows: I set P set，2D P set，3D A, A ToolA Target ID ArUco ID ArUco，Tool ID ArUco，Target .

[0133] It includes two judgment conditions:

[0134] The first criterion is that the industrial control computer determines the minimum distance between the welding termination point and the ArUco feature code with id=2 on the welding assembly 2. This requires that the plane formed by the ArUco feature code with id=2 and the ends of the two welding torches 22 is perpendicular to the direction of travel of the welding assembly 2. Under this requirement, if the end of the welding torch 22 reaches the welding termination point, the welding termination point will form a right angle with the ArUco feature code with id=2 and the ArUco feature code with id=1, referred to as θ. End :

[0135]

[0136] Where, θ End Let represent the angle between two vectors, and arccos denote the inverse cosine function. The first vector (p...) 3 / D，i3 -p 3D，i5 Starting with the ArUco feature code with id=2, which is the third ArUco feature code on welding assembly 2, its 3D coordinates are p. 3D，i3 The ArUco feature code with id 4, which is the second ArUco feature code on the workpiece 4 to be welded, has 3D coordinates p. 3D，i5 , is the vector formed by the endpoints. |p 3D，i3 -p 3D，i5 | represents the modulus of the first vector.

[0137] The second vector (p) 3D，i3 -p 3D，i1 Starting with the ArUco feature code with id=2, which is the third ArUco feature code on welding assembly 2, its 3D coordinates are p. 3D，i3 The ArUco signature with id=0, which is the first ArUco signature on welding assembly 2, has 3D coordinates p. 3D，i1 , is the vector formed by the endpoints. |p 3D，i3 -p 3D，i1 | represents the modulus of the second vector.

[0138] The determination method is as follows: if θ End The following conditions must be met.

[0139] θ End ∈[90°-δ, 90°+δ],

[0140] Where [90°-δ, 90°+δ] represents a closed interval, with the lower limit being 90°-δ and the upper limit being 90°+δ, where δ represents a constant determined empirically. When the first condition is met, the industrial control computer controls the welding assembly 2 to move towards the welding termination point.

[0141] The second criterion is that the moving direction of welding component 2 is in the same direction and parallel to the two vectors formed by the welding start point and welding end point. However, due to the installation error of the ArUco feature code, the two vectors cannot be guaranteed to be strictly parallel. Therefore, the parallel condition is relaxed to an acute angle, as described below.

[0142]

[0143] If both conditions are met simultaneously, the industrial control computer determines that the end of the welding torch 22 has reached the welding termination point and transmits the welding torch 22 stop welding signal and the motion mechanism 21 stop signal to the welding torch 22 and the motion mechanism 21 of the welding assembly 2. Then the welding torch 22 stops welding and the motion mechanism 21 stops.

[0144] By following the steps above, the identification of the start and end points of welding for one cycle can be completed.

Claims

1. A method for identifying the start and end points of linear welding in small groups using machine vision, characterized by: Includes the following steps: Step 1): The welding assembly has a welding torch, a motion mechanism and an ArUco feature code plate. The motion mechanism drives the ArUco feature code plate and two welding torches arranged symmetrically relative to the ArUco feature code plate to move synchronously. The ArUco feature code plate is set with three ArUco feature codes with IDs of 0, 1 and 2 in parallel. The workpiece to be welded is set with two ArUco feature codes with IDs of 3 and 4. Step 2): The 3D vision camera takes a picture and transmits the 2D photo and 3D point cloud signal to the industrial control computer. The industrial control computer performs ArUco feature code recognition on the 2D photo to obtain the id set ids of ArUco feature codes. If an ArUco feature code with a value of 0 is found in the id set ids, the welding component is identified. Step 3): Find the ArUco feature code a3 with id 2 from the id set ids, obtain the pixel number corresponding to ArUco feature code a3 in the 2D image and the three-dimensional coordinates of the point corresponding to it in the 3D point cloud, and then locate the end of the welding gun using the three-dimensional coordinates. Step 4): Find the ArUco feature codes a1, a2, and a3 with IDs of 0, 1, and 2 from the ID set ids. Obtain the pixel numbers corresponding to the ArUco feature codes a1, a2, and a3 in the 2D image and the three-dimensional coordinates of the three points corresponding to them in the 3D point cloud. Transform a straight line passing through the three-dimensional coordinates of the three points to obtain two candidate motion direction vectors. Calculate the angle between the two candidate motion direction vectors and the roughly estimated direction vector. Take the vector corresponding to the smaller angle as the motion direction of the welding component. Step 5): Find the ArUco feature code a4 with id=3 from the id set ids, obtain the pixel number corresponding to ArUco feature code a4 in the 2D image and its corresponding three-dimensional coordinates in the 3D point cloud, which is the welding start point of the workpiece to be welded; find the ArUco feature code a5 with id=4 from the id set ids, obtain the pixel number corresponding to ArUco feature code a5 in the 2D image and its corresponding three-dimensional coordinates in the 3D point cloud, which is the welding end point of the workpiece to be welded.

2. The method for identifying the start and end points of linear welding in a group as described in claim 1, characterized in that: If the distance between the welding start point and the ArUco feature code with id=2 on the welding component is minimized, and the moving direction of the welding component is in the same direction and parallel to the vector formed by the welding start point and the welding end point, then the end of the welding torch reaches the welding start point and the welding torch begins welding.

3. The method for identifying the start and end points of linear welding in a group as described in claim 1, characterized in that: If the distance between the welding termination point and the ArUco feature code with id=2 on the welding component is minimized, and the moving direction of the welding component is in the same direction and parallel to the vector formed by the welding start point and the welding end point, then the end of the welding torch reaches the welding termination point and the welding torch stops welding.

4. The method for identifying the start and end points of linear welding in a small group according to claim 2 or 3, characterized in that: The parallel condition is relaxed to an acute angle.

5. The method for identifying the start and end points of linear welding in a small group according to claim 2, characterized in that: If the end of the welding torch reaches the welding start point, the welding start point forms a right angle with the ArUco feature code with id=2 and the ArUco feature code with id=1.

6. The method for identifying the start and end points of linear welding in a group as described in claim 1, characterized in that: The three ArUco signatures on the welding assembly and the two ArUco signatures on the workpiece to be welded form a set A. Tool The set A of two ArUco signatures on the workpiece to be welded Target , The numbers of the five ArUco signatures form a set N, and the numbers of the three ArUco signatures on the welded assembly form a set N. Tool The numbers of the two ArUco feature codes on the workpiece to be welded form a set N. Target The IDs of all ArUco signatures, the IDs of the ArUco signatures on the welding assembly, and the IDs of the two ArUco signatures on the workpiece to be welded each form a separate set.

7. The method for identifying the start and end points of linear welding in a group as described in claim 6, characterized in that: The pixel numbers in a 2D photograph are the same as the point numbers in a 3D point cloud. The pixels in a 2D photograph form a set, and the pixels in a 3D point cloud form a set.

8. The method for identifying the start and end points of linear welding in a group as described in claim 1, characterized in that: In step 4), a straight line passing through the three-dimensional coordinates of the three points is: ， p 3D,i This represents the coordinates of point i in the 3D point cloud. The theoretical value of the first coordinate component of point i in the 3D point cloud is the measurement result under the condition of no observation error of the 3D vision camera. Its upper left superscript Cam indicates the coordinate system of the 3D vision camera. The second coordinate component representing point i in the 3D point cloud is the measurement result under the condition of no observation error of the 3D vision camera. The upper left corner label Cam indicates the coordinate system of the 3D vision camera. The third coordinate component of point i in the 3D point cloud represents the measurement result under the condition of no observation error of the 3D vision camera. Its upper left superscript Cam indicates the coordinate system of the 3D vision camera; ε represents random error, and the error follows a Gaussian distribution with an expected value of 0; ~ indicates that it follows a certain distribution. Let μ represent a Gaussian distribution, wherein μ is the expected value and is 0, σ is the standard deviation, and ε is the random error, which follows a Gaussian distribution with an expected value of 0.

9. The method for identifying the start and end points of linear welding in a group as described in claim 8, characterized in that: The equation of the line is: The equation of the transformed line is: (y-y0)m+(x0-x)n=0, (z- z0)m + (x0- x)p=0, p 3D,i1 As x0, y0, z 0， get; ， A system of linear equations is obtained; ， K is a matrix [m,n,p] T For K T The eigenvector corresponding to the smallest eigenvalue of K is then the two candidate motion direction vectors are: ， The set, roughly estimated direction vector; , Then the angle between the two candidate motion direction vectors and the roughly estimated direction vector; ， θ Tool,1 The first candidate motion direction vector is represented by the angle between the first vector and the approximate direction vector; arccos represents the inverse cosine function. This represents the first candidate motion direction vector. This represents the magnitude of the first candidate motion direction vector. This represents the second candidate motion direction vector. This represents the magnitude of the second candidate motion direction vector. This represents a rough direction vector. This represents the magnitude of a rough direction vector.

Images