A method for feature extraction and 3D reconstruction of weld shape based on laser scanning
Through the weld shape feature extraction and three-dimensional reconstruction method based on laser scanning, weld images are collected using optical structure system and CCD camera, combined with Harris corner detection and greedy projection triangulation algorithm, the problem of missing information in two-dimensional image analysis is solved. It achieves the accuracy and efficiency of the welding process, and provides detailed three-dimensional weld information and visualization effects.
Patent Information
- Application Number
- CN202110732138.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-06-30
AI Technical Summary
The existing welding appearance detection methods analyze the lost information through two-dimensional image, making it difficult to guarantee the accuracy and efficiency of welding.
Weld shape feature extraction and three-dimensional reconstruction method based on laser scanning are adopted, and weld images are collected through optical structure system and CCD cameras, digital morphological filtering, denoising, and feature point recognition are carried out, and Harris corner point detection algorithm and greedy projection triangle are used. The transformation algorithm performs three-dimensional reconstruction to restore the overall three-dimensional image after welding.
It achieves the accuracy and efficiency of the welding process, provides detailed three-dimensional information of the weld, enhances the reliability of the welding product and the visualization of detection, reduces the amount of calculated data, and ensures the accuracy of reconstruction.
Smart Images

Figure CN113379902B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a feature extraction and three-dimensional reconstruction method of a weld shape based on laser scanning. Background Art
[0002] At present, most companies involve welding technology in the process of mounting vehicles. In actual processing, various defects often appear in the appearance of welds due to the influence of processing accuracy, welding thermal deformation, etc. The appearance quality of welds directly affects the reliability of welded products. The appearance inspection of welds after welding is an effective means to improve and ensure the quality of welding. In addition, the traditional method uses two-dimensional images for analysis, which will lose a lot of valuable information, such as width, height, distance, etc., making it difficult to ensure the accuracy and efficiency of welding.
[0003] The 3D reconstruction technology can solve this problem, and combined with the advantages of monocular vision, it has the advantages of non-contact, fast, and real-time, providing technical support for weld appearance detection. The present invention designs weld recognition and reconstruction technology based on the above situation and 3D reconstruction technology. Summary of the invention
[0004] The purpose of the present invention is to solve the problem of information loss in the image analysis process of the existing appearance inspection method, which makes it difficult to ensure the accuracy and efficiency of welding, and to propose a feature extraction and three-dimensional reconstruction method of the weld shape based on laser scanning.
[0005] A method for feature extraction and 3D reconstruction of weld shape based on laser scanning, wherein the method first designs an optical structure system according to the principle of laser scanning and optical triangulation, and calculates the change information of the horizontal and vertical positions of the weld through geometric relationships; wherein the whole system is mainly composed of a camera and an industrial control machine, a CCD camera is used to collect weld images, and a J1900 industrial control machine is used as a processing unit, and the effects of image collection and display are given through debugging; then, the original image obtained is preprocessed by digital filtering, denoising, etc. through a weld recognition algorithm, and the feature points of the preprocessed weld image are identified; finally, the deviation of the actual weld center point is derived according to the position change of the weld image center point, and the overall 3D image after welding is restored by 3D reconstruction of the identified image point cloud data;
[0006] Preferably, the feature extraction and three-dimensional reconstruction method comprises the following steps:
[0007] Step 1: Obtain a weld image, establish a model through visual system calibration, convert the image coordinates of the weld in the weld image in the image coordinate system of the structured light stripes into camera coordinates, and then convert them into coordinates based on the robot base coordinate system;
[0008] Step 2: Image preprocessing;
[0009] Step 3: Use Harris corner detection algorithm to extract feature points;
[0010] Step 4: Use the greedy projection triangulation algorithm to perform triangulation processing to realize the three-dimensional reconstruction process of the weld image.
[0011] Preferably, in the feature extraction and three-dimensional reconstruction method, the specific process of step 1 is:
[0012] The visual system calibration includes three parts: camera calibration, line structured light plane calibration and hand-eye calibration;
[0013] Camera calibration includes two parts: calibration of internal parameters and calibration of external parameters. The calibration of internal parameters refers to the calibration of the focal length, distortion coefficient, and center point of the camera. The calibration of external parameters refers to the conversion relationship between the rotation transformation matrix and the movement matrix between the camera coordinate system and the world coordinate system.
[0014] There are two methods for camera calibration: traditional camera calibration and self-calibration. Self-calibration does not require calibration and is widely used in hand-eye calibration. It can be achieved by simply taking multiple images from various angles and directions.
[0015] The calibration of the structured light plane is performed after the camera is calibrated, and the line structured light plane is obtained using multiple feature points.
[0016] Preferably, in the feature extraction and three-dimensional reconstruction method, the specific process of image preprocessing in step 1 is:
[0017] First, image denoising effectively removes various noises and interferences in the weld image; second, image binarization separates the weld information from the background information in the image, improves the contrast, and highlights the weld information; third, weld edge detection aims to perform image segmentation and extract important information about the weld cross-sectional profile;
[0018] Preferably, the process of extracting the characteristic points in step 3 is to take the lowest point of the V-groove weld as the characteristic point and extract the characteristic point;
[0019] Specifically:
[0020] Step 31: Confirm the center line.
[0021] After edge extraction, the center line between the two edges is confirmed as the midline. The midline retains the original characteristics of the image, including shape and continuity. By analyzing the characteristics of the center line, various features of the original image are obtained;
[0022] Step 32: Scan the image to obtain the row coordinates of the middle point;
[0023] The binary image obtained after edge extraction is scanned using an up-and-down pixel scanning method, namely:
[0024] Starting from the first column of pixels, scan pixels from top to bottom and from bottom to top respectively, and record the row coordinates x1 and x2 of the pixel with a gray value of 255 encountered for the first time, and assign the gray value of the pixel to Oo, calculate the difference between the two row coordinates, and when the difference is less than 5, consider it a single edge and discard it, and when the difference is greater than or equal to 5, calculate the average of the two row coordinates, which is the row coordinate of the midpoint of the two edges;
[0025] Step 33: Assign the grayscale value of the pixel at the middle point to 255;
[0026] The center line represents the cross section of the weld. In the neighborhood of the lowest point of the weld, the gray value at the point changes in two directions; one is along the left edge of the weld cross section, and the other is along the right edge of the weld cross section.
[0027] Preferably, in the feature extraction and three-dimensional reconstruction method, the three-dimensional reconstruction process of the weld image in step 4 is specifically as follows:
[0028] Surface reconstruction is applied to the display of weld image data after acquisition. Visual Studio is used in combination with PCL to realize surface reconstruction, and the greedy projection triangulation algorithm is used for triangulation. First, an initial triangle is created, and new triangles are added until all points in the point cloud are traversed.
[0029] The beneficial effects of the present invention are:
[0030] The present invention is applied in the field of welding. The application of visual inspection and recognition technology can be said to be very extensive. Nearly 40% of domestic welding companies have used machine vision as the main guiding technology for welding. Compared with the previous weld inspection methods, machine vision inspection is widely used in product inspection due to its sensitivity, anti-interference, and no contact with the workpiece, which will not cause damage to the workpiece.
[0031] The present invention applies surface reconstruction to the display after the weld image data is acquired to enhance the visualization effect of the weld image. Surface reconstruction is implemented using Visual Studio in combination with PCL. PCL (Point Cloud Library) currently implements a variety of point cloud-based surface reconstruction algorithms, such as Poisson surface reconstruction, greedy projection triangulation, moving cubes, EarClipping and other algorithms. In addition, the present invention uses a greedy projection triangulation algorithm for triangulation processing for the rapid triangulation of unordered point clouds. By controlling a series of point lists (edge points), the grid can be grown and expanded outward until all possible points are connected.
[0032] (1) The use of three-dimensional reconstruction for presentation after inspection provides great convenience for some welding environments where it is inconvenient to inspect the weld status.
[0033] (2) The mature Harris corner detection algorithm is used to extract feature points, which not only reduces the amount of calculated data but also ensures the efficiency of feature point extraction.
[0034] (3) Using greedy projection triangulation for 3D reconstruction can effectively solve the problem of irregular weld image data and ensure the accuracy of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flow chart of the method according to the present invention;
[0036] Figure 2 This is a demonstration of the principle of the feature point extraction method involved in step three of the present invention;
[0037] Figure 3 This is a flow chart of triangulation processing using a greedy projection triangulation algorithm involved in step 4 of the present invention. DETAILED DESCRIPTION Specific implementation method one:
[0039] The present embodiment provides a method for extracting and reconstructing weld shapes based on laser scanning, and a method for extracting and reconstructing weld shapes based on laser scanning, which is characterized in that: the method first designs an optical structure system according to the principles of laser scanning and optical triangulation, and calculates the change information of the horizontal and vertical positions of the welds through geometric relationships; wherein the entire system is mainly composed of a camera and an industrial control machine, a CCD camera is used to collect weld images, and a J1900 industrial control machine is used as a processing unit, and the effects of image collection and display are given through debugging; then, through the weld recognition algorithm, the obtained original image is preprocessed by digital filtering, denoising and other preprocessing work, and the feature points of the preprocessed weld image are identified; finally, the deviation of the actual weld center point is derived according to the position change of the weld image center point, and the identified image point cloud data is reconstructed in three dimensions to restore the overall three-dimensional image after welding. Specific implementation method 2:
[0041] Different from the first embodiment, the present embodiment is a method for extracting and reconstructing the weld shape based on laser scanning, such as Figure 1 As shown, the feature extraction and 3D reconstruction method comprises the following steps:
[0042] Step 1: Obtain a weld image, establish a model through visual system calibration, convert the image coordinates of the weld in the weld image in the image coordinate system of the structured light stripes into camera coordinates, and then convert them into coordinates based on the robot base coordinate system;
[0043] Step 2: Image preprocessing;
[0044] Step 3: Use Harris corner detection algorithm to extract feature points;
[0045] Step 4: Use the greedy projection triangulation algorithm to perform triangulation processing to realize the three-dimensional reconstruction process of the weld image. Specific implementation method three:
[0047] Different from the second specific implementation mode, the present implementation mode is a method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning. In the feature extraction and three-dimensional reconstruction method, the specific process of step one is:
[0048] The visual system calibration includes three parts: camera calibration, line structured light plane calibration and hand-eye calibration;
[0049] Camera calibration includes two parts: calibration of internal parameters and calibration of external parameters. The calibration of internal parameters refers to the calibration of the focal length, distortion coefficient, and center point of the camera. The calibration of external parameters refers to the conversion relationship between the rotation transformation matrix and the movement matrix between the camera coordinate system and the world coordinate system.
[0050] There are two methods for camera calibration: traditional camera calibration and self-calibration. Self-calibration does not require calibration and is widely used in hand-eye calibration. It can be achieved by simply taking multiple images from various angles and directions.
[0051] The calibration of the structured light plane is performed after the camera is calibrated, and the line structured light plane is obtained using multiple feature points. Specific implementation method four:
[0053] Different from the second or third specific implementation, the present implementation is a method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning. In the feature extraction and three-dimensional reconstruction method, the specific process of image preprocessing in step 1 is as follows:
[0054] The chip in the CCD camera is the first place to come into contact with the image signal. At this time, the image signal is an analog signal. The camera will convert the analog signal into a digital signal and then transmit it to the host computer. Noise will also be generated during the conversion process. These noises will blur the image and make the weld information less obvious, so these noises must be eliminated to highlight the information we want to extract. The process of completing this work is called preprocessing. The preprocessing stage of this invention mainly completes three tasks: first, image denoising, which effectively removes various noises and interferences in the weld image; second, image binarization, which separates the weld information from the background information in the image, improves the contrast, and highlights the weld information; third, weld edge detection, its purpose is to perform image segmentation and extract important information about the weld cross-sectional contour; Specific implementation method four:
[0056] Different from the second or third specific implementation, the present implementation is a method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning. In the feature extraction and three-dimensional reconstruction method, the feature point extraction process of step three is to take the lowest point of the V-shaped groove weld as the feature point and extract the feature point. Therefore, for the V-shaped groove weld, the specific point to be welded is the lowest point of the groove, that is, welding is performed along the line connecting the lowest points. Based on this, the feature point extraction process is specifically as follows:
[0057] Step 31: Confirm the center line.
[0058] After edge extraction, the center line between the two edges is confirmed as the midline. The midline retains the original characteristics of the image, including shape, continuity, etc. By analyzing the characteristics of the center line, the various features of the original image are obtained; this step is a key step before feature point extraction.
[0059] Step 32: Scan the image to obtain the row coordinates of the middle point;
[0060] The image after edge extraction is a binary image. There are only two gray values in the image, one is the background with a gray value of 0, and the other is the edge with a gray value of 255. Therefore, the binary image obtained after edge extraction is scanned using an up-and-down pixel scanning method, that is:
[0061] Starting from the first column of pixels, scan pixels from top to bottom and from bottom to top respectively, and record the row coordinates x1 and x2 of the pixel with a gray value of 255 encountered for the first time, and assign the gray value of the pixel to Oo, calculate the difference between the two row coordinates, and when the difference is less than 5, consider it a single edge and discard it, and when the difference is greater than or equal to 5, calculate the average of the two row coordinates, which is the row coordinate of the midpoint of the two edges;
[0062] Step 33: Assign the grayscale value of the pixel at the middle point to 255;
[0063] The center line represents the cross section of the weld. In the neighborhood of the lowest point of the weld, the gray value at the point changes in two directions; one is along the left edge of the weld cross section, and the other is along the right edge of the weld cross section. Figure 2 As shown, it can be seen that the lowest point is a relatively regular corner point, which retains the important features of the image, reduces the amount of information data, and improves efficiency. Therefore, the present invention selects the widely used Harris corner point detection algorithm for feature point extraction. Specific implementation method five:
[0065] Different from the fourth embodiment, the present embodiment is a method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning. In the method for feature extraction and three-dimensional reconstruction, the three-dimensional reconstruction process of the weld image in step four is as follows:
[0066] Surface reconstruction is applied to the display of weld image data after acquisition to enhance the visualization of weld images. Surface reconstruction is implemented using Visual Studio combined with PCL. PCL (Point Cloud Library) currently implements a variety of point cloud-based surface reconstruction algorithms, such as Poisson surface reconstruction, greedy projection triangulation, marching cubes, EarClipping and other algorithms.
[0067] When measuring the detailed data of an object, deviations will occur in the collection, processing and other processes. These deviations will cause data irregularities. If these data are directly used for surface reconstruction, the three-dimensional reconstructed surface will be uneven and uneven, and will be accompanied by the generation of reconstruction holes. For the rapid triangulation of unordered point clouds, the present invention uses a greedy projection triangulation algorithm for triangulation processing; the principle of the greedy projection algorithm is: by controlling a series of point lists (edge points), the mesh can be grown and expanded outward until all possible points are connected. Local triangulation is to map the local area points of the point along the normal of the point and connect the unconnected points. The algorithm is based on the principle of incremental surface growth and follows a greedy type method. The algorithm first creates an initial triangle and continues to add new triangles until all points in the point cloud are traversed, or there are no more valid triangles that can be connected to the mesh. The specific algorithm is as follows Figure 3 shown.
[0068] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for feature extraction and three-dimensional reconstruction of weld shape based on laser scanning, characterized in that: The method first designs an optical structure system according to the principles of laser scanning and optical triangulation, and calculates the change information of the horizontal and vertical positions of the weld through geometric relationships; wherein the entire system is composed of a camera and an industrial control machine, a CCD camera is used to collect weld images, and a J1900 industrial control machine is used as a processing unit, and the effects of image collection and display are given through debugging; then, the obtained original image is preprocessed by digital filtering and denoising through a weld recognition algorithm, and the feature points of the preprocessed weld image are recognized; finally, the deviation of the actual weld center point is derived according to the position change of the weld image center point, and the overall three-dimensional image after welding is restored by performing three-dimensional reconstruction on the recognized image point cloud data; The feature extraction and three-dimensional reconstruction method specifically comprises the following steps: Step 1: Obtain a weld image, establish a model through visual system calibration, convert the image coordinates of the weld in the weld image in the image coordinate system of the structured light stripes into camera coordinates, and then convert them into coordinates based on the robot base coordinate system; Step 2: Image preprocessing; Step 3: Use Harris corner detection algorithm to extract feature points; Step 4: Use the greedy projection triangulation algorithm to perform triangulation processing to realize the three-dimensional reconstruction process of the weld image; The specific process of step one is: The visual system calibration includes three parts: camera calibration, line structured light plane calibration and hand-eye calibration; Camera calibration includes two parts: calibration of internal parameters and calibration of external parameters. The calibration of internal parameters refers to the calibration of the focal length, distortion coefficient and center point of the camera. The calibration of external parameters refers to the conversion relationship between the rotation transformation matrix and the movement matrix between the camera coordinate system and the world coordinate system. Self-calibration is used in hand-eye calibration, which can be achieved simply by taking multiple images from various angles and directions; Calibration of the line structured light plane: after the camera is calibrated, the line structured light plane is obtained by using multiple feature points; The specific process of image preprocessing in step 2 is as follows: First, image denoising effectively removes various noises and interferences in the weld image; second, image binarization separates the weld information from the background information in the image, improves the contrast, and highlights the weld information; third, weld edge detection aims to perform image segmentation and extract important information about the weld cross-sectional profile; The feature point extraction in step 3 is specifically as follows: Take the lowest point of the V-groove weld as the feature point and extract the feature point; Specifically: Step 31: Confirm the center line. After edge extraction, the center line between the two edges is confirmed as the midline. The midline retains the original characteristics of the image, including shape and continuity. By analyzing the characteristics of the center line, various features of the original image are obtained; Step 32: Scan the image to obtain the row coordinates of the middle point; The binary image obtained after edge extraction is scanned using an up-and-down pixel scanning method, namely: Starting from the first column of pixels, scan pixels from top to bottom and from bottom to top respectively, and record the row coordinates x1 and x2 of the pixel with a gray value of 255 encountered for the first time, and assign the gray value of the pixel to Oo, calculate the difference between the two row coordinates, and when the difference is less than 5, consider it a single edge and discard it, and when the difference is greater than or equal to 5, calculate the average of the two row coordinates, which is the row coordinate of the midpoint of the two edges; Step 33: Assign the grayscale value of the pixel at the middle point to 255; The center line represents the cross section of the weld. In the neighborhood of the lowest point of the weld, the gray value at the point changes in two directions; one is along the left edge of the weld cross section, and the other is along the right edge of the weld cross section. The three-dimensional reconstruction process of the weld image in step 4 is specifically as follows: Surface reconstruction is applied to the display of weld image data after acquisition. VisualStudio is used in combination with PCL to realize surface reconstruction, and the greedy projection triangulation algorithm is used for triangulation. First, an initial triangle is created, and new triangles are added until all points in the point cloud are traversed.
