A machine vision-based pre-assembly subassembly weld positioning system and method

Abstract

The application discloses a kind of based on machine vision's preceding group subassembly weld joint positioning system and method, comprising the following steps: camera is hoisted above welding platform by tooling, relative position is fixed and camera center line is perpendicular to the plane of the workpiece to be measured;Camera intrinsic calibration is carried out, and hand-eye calibration between camera-welding robot or hand-eye calibration between line laser sensor-welding robot;Workpiece feeding, scene picture is shot;Based on the shooting picture, workpiece feature point set calculation is carried out using gray scale segmentation and Harris corner detection, for the scene that needs to be shot by multiple cameras, image stitching based on calibration board guide map is carried out before workpiece feature point set calculation;Seeking location feature point set and calculating weld joint end point;Weld joint tracking is carried out, and weld joint data is corrected in real time, to obtain actual weld joint data.The application can effectively improve the welding efficiency and automation degree of shipyard small group subassembly unit.

Description

Technical Field

[0001] This invention belongs to the field of machine vision technology, and in particular to a machine vision-based system and method for locating weld seams in prefabricated components. Background Technology

[0002] Pre-assembly is a stage in shipbuilding, generally referring to the fabrication of the simplest hull structural components in a fixed location, such as assembling T-sections and installing reinforcements on ribs. Currently, in China's shipbuilding enterprises, pre-assembly welding is still mainly done manually, with a low degree of automation. Most have not yet achieved intelligent assembly line operations, resulting in significant waste of human resources and energy. Under these circumstances, accelerating the promotion and application of pre-assembly welding robots to achieve intelligent manufacturing of pre-assembly is urgently needed.

[0003] Welding is a crucial step in ship assembly and a key factor determining hull quality. Multiple processes and workstations in shipbuilding rely on welding. According to relevant data, welding workload accounts for 30-40% of the total shipbuilding workload, welding cost accounts for 30-50% of the total shipbuilding cost, and welding time accounts for 40% of the total shipbuilding time. Currently, ship welding in my country is still primarily manual, with an automation level of only 20-30%.

[0004] Chinese patent CN113369761A, entitled "A Method and System for Weld Seam Positioning Based on Vision Guidance Robot," discloses the following features: acquiring image information of the workpiece to be welded; determining the starting position of the weld seam on the workpiece based on the image information; and guiding the welding robot to perform welding based on the determined starting position of the weld seam. This patent is a hand-eye integrated detection solution, which requires pre-input of workpiece position information and has a long working time. Summary of the Invention

[0005] The purpose of this invention is to provide a machine vision-based pre-assembly unit weld seam positioning system and method, which can effectively improve the welding efficiency and automation level of shipyard pre-assembly units, and has high robustness and resistance to interference from ambient light.

[0006] The technical solution to achieve the objective of this invention is: a method for locating weld seams in preliminary sub-assemblies based on machine vision, comprising the following steps:

[0007] The camera is hoisted above the welding platform using a fixture, with its relative position fixed and its center line perpendicular to the plane of the workpiece to be measured.

[0008] Perform camera intrinsic parameter calibration, as well as hand-eye calibration between the camera and the welding robot, or hand-eye calibration between the line laser sensor and the welding robot;

[0009] Load the workpiece and take pictures of the scene;

[0010] The workpiece feature point set is calculated based on grayscale segmentation and Harris corner detection of the captured images. For scenarios that require multiple cameras to capture images, image stitching based on the calibration board guide map is performed before calculating the workpiece feature point set.

[0011] Locate the feature point set and calculate the weld endpoints;

[0012] Weld seam tracking is performed, and weld seam data is corrected in real time to obtain actual weld seam data.

[0013] Furthermore, for the hand-eye calibration between the camera and the welding robot, a nine-point calibration method is used; for the hand-eye calibration between the line laser sensor and the welding robot, a TCP calibration method is used.

[0014] Furthermore, the workpiece has a rectangular base, and the selected feature points are two points on each of the two short sides of the base plate and one point on the elbow plate.

[0015] Furthermore, the image stitching based on the calibration board guidance map specifically includes:

[0016] Step 1: Construct a double calibration plate. Fix two identical calibration plates onto the same long strip, ensuring that the centerlines of the two calibration plates lie on the same straight line and the relative distance between their centers is [missing information]. And measure the height difference between the calibration surface of the dual calibration plate and the test platform. Pixel precision in world coordinate system ;

[0017] Step 2: Place the dual calibration boards under the two cameras, with the two calibration boards located within the field of view of each camera, designated as C1 for the left camera and C2 for the right camera. Acquire guide images ImgGuide1 and ImgGuide2, and images to be stitched Img1 and Img2 (the guide image is the original scene image containing the calibration boards, and the images to be stitched are the images to be processed). Measure the overlap ratio of the two guide images along their column directions. The staggered ratio of row directions Based on the calibrated camera intrinsic parameters, and taking the center of the calibration board in the two guidance images ImgGuide1 and ImgGuide2 as the origin, the corresponding extrinsic parameters of the left camera C1 and the right camera C2 are obtained respectively. and ,

[0018]

[0019] in, Indicates translation. Indicates rotation, Indicates the direction of rotation, and can be either 1 or 0;

[0020] Step 3: Perform coordinate transformation based on... and Determine new external parameters and ,according to and Perform row-direction stitching on the images to be stitched, Img1 and Img2 respectively, to obtain the stitched result image ImgRes.

[0021] Furthermore, step 3 specifically includes:

[0022] Calculate the pixel coordinate system to world coordinate system projection transformation matrix for each of the two cameras. ,in:

[0023]

[0024] in, For the camera intrinsic parameter matrix, These are the camera's own external parameters;

[0025] Conversion from pixel coordinates to world coordinates:

[0026]

[0027] in, For camera depth, For pixel coordinates, For world coordinates, u and v are the column and row coordinates of the pixel;

[0028] According to ImgGuide1 Trim the extra rows to define the top left vertex, and use 0.5* Define the bottom right vertex, and determine the world coordinates of the new top left and bottom right vertices as follows: and Define the new extrinsic parameter of C1 as:

[0029]

[0030] Where RotNew1 is the new rotation of C1, TransNew1 is the new translation of C1, and DirectionNew1 is the new direction of C1;

[0031] For ImgGuide2, define the matrix:

[0032]

[0033]

[0034] right Find the inverse, and you will get ,Will Convert to 4x4 array :

[0035]

[0036] Will and Multiply to get Then and Multiply to get Finally Transform into a 1*3 array .

[0037] Furthermore, the specific steps for calculating the workpiece feature point set using grayscale segmentation and Harris corner detection for the stitched image are as follows:

[0038] Step 1: Accurately extract the base plate area of ​​the elbow plate. First, perform median filtering and graphics opening and closing operations on the image to be processed to filter out noise and small areas that meet the set requirements. Then, use threshold segmentation to extract the base plate area of ​​the elbow plate.

[0039] Step 2: Perform Harris corner detection on the outer contour of the elbow plate base area and calculate the corresponding workpiece feature point set VectorF. In the corner point set VectorC obtained by Harris corner detection on the outer contour, calculate the slope of two adjacent points to obtain the slope set VectorK. Determine the screening threshold and remove the grip points in VectorK that are less than the threshold in VectorC to obtain the set VectorF.

[0040] Step 3: Based on the set VectorF, determine the location of feature points according to the shape of the workpiece base plate, and calculate the feature point set.

[0041] A system based on the aforementioned preliminary group stand weld seam positioning method includes a camera unit, a welding robot unit, an information processing unit, and a welding platform. The camera unit includes a camera and a lighting source for capturing images of the workpiece within the scene. The welding robot unit includes a welding robot and a line laser sensor for precise weld seam positioning. The information processing unit is connected to the camera, a light source controller, and the welding robot. It controls the light source through the light source controller, controls the camera to capture images of the workpiece within the scene, simultaneously calculates the workpiece feature point set, locates the feature point set, calculates the weld seam endpoints, and transmits the feature point set data to the welding robot unit. The welding platform is used to place the elbow plate to be welded.

[0042] Furthermore, the camera is a CCD / CMOS camera, which is hoisted above the welding platform by a fixture, and the center line of the camera is perpendicular to the upper surface of the welding platform; the workpiece to be welded is placed on the welding platform with the elbow plate facing upward.

[0043] Furthermore, there are multiple cameras, and the relative positions of the multiple cameras are fixed.

[0044] Furthermore, the information processing unit is an industrial control computer.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention adopts a hand-eye separation detection scheme, which can automatically acquire workpiece position information; The present invention can realize one-click operation, eliminating manual teaching or manual welding, greatly reducing labor costs, and greatly improving safety and the automation level of shipyards; The weld positioning system of the present invention supports open operation, which is not only highly accurate but also has high robustness and anti-interference ability against external ambient light. Attached Figure Description

[0046] Figure 1 This is a system hardware architecture diagram of a specific embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the elbow plate to be welded according to a specific embodiment of the present invention.

[0048] Figure 3 for Figure 2 A diagram illustrating the processing results. Detailed Implementation

[0049] Exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The description of the exemplary embodiments is for illustrative purposes only and is not intended to limit the invention, its applications, or uses.

[0050] like Figure 1 As shown, a machine vision-based pre-assembly component weld seam positioning system includes a camera unit, a welding robot unit, an information processing unit, a light source controller, and a welding platform.

[0051] The information processing unit is connected to the camera, the light source controller, and the welding robot, respectively.

[0052] The information processing unit controls the light source through the light source controller, controls the camera to take pictures of the workpiece in the scene, and transmits the feature point set data to the welding robot unit.

[0053] The welding platform is used to place the elbow plate parts to be welded.

[0054] The welding robot unit includes a welding robot and a line laser sensor, and uses the TCP method for hand-eye calibration between the line laser sensor and the welding robot.

[0055] The imaging unit includes two CCD / CMOS cameras and an illumination source. The cameras are mounted above the welding platform using fixtures, with their relative positions fixed and their center lines perpendicular to the plane of the workpiece being measured. The illumination source provides suitable imaging brightness. After mounting, camera intrinsic parameter calibration, camera-welding robot hand-eye calibration, and image stitching based on the calibration board guidance diagram are performed. Specifically, for camera-welding robot hand-eye calibration, this system uses a nine-point calibration method.

[0056] The information processing unit is an industrial control computer. First, it calculates the weld feature points based on the image data returned by the camera. Then, it uses a line laser sensor to locate the feature points and calculate the weld endpoints. Finally, it performs real-time tracking along the rough path of the weld to obtain weld data.

[0057] A machine vision-based method for locating weld seams in preliminary assembly components includes the following steps:

[0058] Step 1: Loading the workpiece, such as... Figure 2 As shown, this method takes a rectangular bottom workpiece as an example. When feeding the elbow plate, the workpiece to be welded is placed on the welding platform with the elbow plate facing upward.

[0059] Step 2: Take photos of the scene. The photos should be grayscale images.

[0060] Step 3: Calculate the feature point set of the workpiece. The information processing unit uses grayscale segmentation and Harris corner detection to locate the four corner points of the rectangular base, and further calculates the coordinates of the feature point set. Specifically, the feature points selected in this method are two points on each of the two short sides of the base plate and one point on the elbow plate;

[0061] Step 4: Locate the feature point set and calculate the weld endpoint. The information processing unit sends the feature point set data obtained in the previous step to the welding robot unit. The welding robot unit, while maintaining a safe distance, locates the feature point set and determines the weld endpoint by finding the intersection points.

[0062] Step 5: Weld seam tracking and acquisition of actual weld seam data. The welding robot tracks the weld seam using a method suitable for straight weld seams and corrects the weld seam data in real time to obtain the actual weld seam data.

[0063] Specifically, the image stitching based on the calibration board guide map is as follows:

[0064] Step 1: Construct a double calibration plate. Fix two identical calibration plates onto the same long strip, ensuring that the centerlines of the two calibration plates lie on the same straight line and the distance between their centers is [missing information]. And measure the height difference between the calibration surface of the dual calibration plate and the test platform. Pixel precision in world coordinate system .

[0065] Step 2: Place the dual calibration boards under the two cameras, with each board positioned within the field of view of one camera (C1 for the left camera and C2 for the right camera). Acquire the guide images ImgGuide1 and ImgGuide2, and the images to be stitched (Img1 and Img2) respectively. Measure the overlap ratio of the dual guide images along their respective directions. The staggered ratio of row directions Based on the calculated camera intrinsic parameters, and taking the center of the calibration plate in the guidance images ImgGuide1 and ImgGuide2 as the origin, the corresponding extrinsic parameters were obtained respectively. , Calculate the pixel coordinate system-to-world coordinate system projection transformation matrix for each of the two cameras. ,in

[0066]

[0067] in, For the camera intrinsic parameter matrix, For camera external parameters:

[0068]

[0069] in, Describe translation, Describe rotation, Describes the direction of rotation; value is 1 or 0.

[0070] The transformation formula from pixel coordinates to world coordinates is as follows:

[0071]

[0072] in, For camera depth, For pixel coordinates, Used as world coordinates.

[0073] According to ImgGuide1 Trim the extra rows to define the top left vertex, and use 0.5* Define the bottom right vertex, and determine the world coordinates of the new top left and bottom right vertices as follows: and ,in

[0074] Define a new extrinsic parameter for C1

[0075]

[0076] For ImgGuide2, define a matrix.

[0077]

[0078]

[0079] right Find the inverse, and you will get ,Will Convert to 4x4 array

[0080]

[0081] will and Multiply to get Then and Multiply to get Finally Transform into a 1*3 array That is, the new extrinsic parameters of the right camera C2:

[0082]

[0083] according to and Perform row-wise stitching on Img1 and Img2 respectively to obtain the stitched image ImgRes, as shown below. Figure 3 .

[0084] The specific process for calculating the workpiece feature point set based on grayscale segmentation and Harris corner detection is as follows:

[0085] Step 1: Accurately extract the base plate area of ​​the elbow plate. First, perform median filtering and graphics opening / closing operations on the image to be processed to remove noise and small areas. Then, use threshold segmentation to extract the base plate area of ​​the elbow plate.

[0086] Step 2: Perform Harris corner detection on the outer contour of the elbow plate base area to calculate the corresponding workpiece feature point set VectorF. Further, in the corner point set VectorC obtained from Harris corner detection on the outer contour, calculate the slope of any two adjacent points to obtain the slope set VectorK. Determine the filtering threshold, and remove the grips in VectorK that are less than the threshold from VectorC, thus obtaining VectorF.

[0087] Step 3: Determine the location of feature points based on the shape of the workpiece base plate, and use the corresponding method to obtain the feature point set. The corresponding method for obtaining the feature point set is well known in the field and will not be elaborated here.

[0088] The present invention proposes a machine vision-based pilot group stand weld seam positioning system that supports open operation, which is not only highly accurate but also highly robust and resistant to interference from ambient light.

[0089] The above are preferred embodiments of this method. It should be noted that improvements and modifications made by those skilled in the art without departing from the principles described in this invention should also be considered within the scope of protection of this invention.

Claims

1. A method for locating weld seams in preliminary assembly components based on machine vision, characterized in that, Including the following steps: The camera is hoisted above the welding platform using a fixture, its position is fixed and the center line of the camera is perpendicular to the plane of the workpiece to be measured. Perform camera intrinsic parameter calibration, as well as hand-eye calibration between the camera and the welding robot, or hand-eye calibration between the line laser sensor and the welding robot; Load the workpiece and take scene photos; The workpiece feature point set is calculated based on grayscale segmentation and Harris corner detection of the captured images. For scenarios that require multiple cameras to capture images, image stitching based on the calibration board guide map is performed before calculating the workpiece feature point set. Locate the feature point set and calculate the weld endpoints; Weld seam tracking is performed, and weld seam data is corrected in real time to obtain actual weld seam data; The workpiece has a rectangular base, and the feature points are selected as two points on each of the two short sides of the base plate and one point on the elbow plate. The image stitching based on the calibration board guidance map specifically includes: Step 1: Construct a double calibration plate. Fix two identical calibration plates onto the same long strip, ensuring that the centerlines of the two calibration plates lie on the same straight line and the relative distance between their centers is [missing information]. And measure the height difference between the calibration surface of the dual calibration plate and the test platform. Pixel precision in world coordinate system ; Step 2: Place the dual calibration plates under the two cameras, with the two calibration plates located within the field of view of the two cameras respectively, designated as C1 for the left camera and C2 for the right camera. Acquire the guide images ImgGuide1 and ImgGuide2 and the images to be stitched Img1 and Img2 respectively; measure the overlap ratio of the dual guide images in the column direction. The staggered ratio of row directions Based on the calibrated camera intrinsic parameters, and taking the center of the calibration board in the two guidance images ImgGuide1 and ImgGuide2 as the origin, the corresponding extrinsic parameters of the left camera C1 and the right camera C2 are obtained respectively. and , in, Indicates translation. Indicates rotation, Indicates the direction of rotation, and can be either 1 or 0; Step 3: Perform coordinate transformation based on... and Determine new external parameters and ,according to and Perform row-direction stitching on the images to be stitched, Img1 and Img2 respectively, to obtain the stitched result image ImgRes.

2. The method for locating weld seams in preliminary sub-assemblies based on machine vision as described in claim 1, characterized in that: For hand-eye calibration between the camera and the welding robot, the nine-point calibration method is used; for hand-eye calibration between the line laser sensor and the welding robot, the TCP calibration method is used.

3. The method for locating weld seams in preliminary assembly components based on machine vision as described in claim 1, characterized in that: Step 3 specifically includes: Calculate the pixel coordinate system to world coordinate system projection transformation matrix for each of the two cameras. ,in: in, For the camera intrinsic parameter matrix, This refers to the camera's intrinsic parameters. Conversion from pixel coordinates to world coordinates: in, For camera depth, For pixel coordinates, For world coordinates, u and v are the column and row coordinates of the pixel; According to ImgGuide1 Trim the extra rows to define the top left vertex, and use 0.5* Define the bottom right vertex, and determine the world coordinates of the new top left and bottom right vertices as follows: and Define the new extrinsic parameter of C1 as: Where RotNew1 is the new rotation of C1, TransNew1 is the new translation of C1, and DirectionNew1 is the new direction of C1; For ImgGuide2, define the matrix: right Find the inverse, and you will get ,Will Convert to 4x4 array : Will and Multiply to get Then and Multiply to get Finally Transform into a 1*3 array .

4. The method for locating weld seams in preliminary sub-assemblies based on machine vision as described in claim 3, characterized in that: The specific steps for calculating the workpiece feature point set based on captured images using grayscale segmentation and Harris corner detection are as follows: Step 1: Accurately extract the base plate area of ​​the elbow plate. First, perform median filtering and graphics opening and closing operations on the image to be processed to filter out noise and small areas that meet the set requirements. Then, use threshold segmentation to extract the base plate area of ​​the elbow plate. Step 2: Perform Harris corner detection on the outer contour of the elbow plate base area and calculate the corresponding workpiece feature point set VectorF. In the corner point set VectorC obtained by Harris corner detection on the outer contour, calculate the slope of two adjacent points to obtain the slope set VectorK. Determine the screening threshold and remove the grip points in VectorK that are less than the threshold in VectorC to obtain the set VectorF. Step 3: Based on the set VectorF, determine the location of feature points according to the shape of the workpiece base plate, and calculate the feature point set.

5. A machine vision-based pre-assembly component weld seam positioning system, employing the pre-assembly component weld seam positioning method as described in any one of claims 1-4, characterized in that: The system includes a camera unit, a welding robot unit, an information processing unit, and a welding platform. The camera unit includes a camera and a lighting source for capturing images of the workpiece within the scene. The welding robot unit includes a welding robot and a line laser sensor for precise weld seam positioning. The information processing unit is connected to the camera, the light source controller, and the welding robot. It controls the light source through the light source controller, controls the camera to capture images of the workpiece within the scene, calculates the workpiece feature point set, locates the feature point set, calculates the weld seam endpoints, and transmits the feature point set data to the welding robot unit. The welding platform is used to place the elbow plate parts to be welded.

6. The machine vision-based pre-assembly component weld positioning system as described in claim 5, characterized in that: The camera is a CCD / CMOS camera, which is hoisted above the welding platform by a fixture, and the center line of the camera is perpendicular to the upper surface of the welding platform; the workpiece to be welded is placed on the welding platform with the elbow plate facing upward.

7. The machine vision-based pre-assembly component weld positioning system as described in claim 5, characterized in that: The camera is a multi-camera unit, and the relative positions of the cameras are fixed.

8. The machine vision-based pre-assembly component weld positioning system as described in claim 5, characterized in that: The information processing unit is an industrial control computer.

Images