Laser-electric arc hybrid welding monitoring method and device based on molten pool image

Abstract

The invention provides a laser-arc hybrid welding monitoring method and device based on a molten pool image, and the device comprises a laser device which transmits laser to a laser head through an optical fiber; the laser head is positioned above the molten pool of the weldment and is used for emitting laser to the molten pool; the electric arc welding gun is located above the weld pool of the weldment, and the electric arc welding gun is connected with the welding power source; the multi-view-angle high-speed cameras are installed above and on the side portion of the molten pool so as to capture molten pool images containing the molten pool and a key hole in the welding process and transmit the molten pool images to the industrial personal computer; the industrial personal computer preprocesses the molten pool image, the preprocessed molten pool image is aligned, registered and fused, and three-dimensional point cloud data of the molten pool and the key hole are generated; converting the three-dimensional point cloud data into a continuous three-dimensional surface model by using a surface reconstruction algorithm; and the three-dimensional surface model is input into the pre-trained convolutional neural network, whether defects are generated in the welding process or not is judged in real time, the welding defect detection precision and real-time performance can be improved, and the welding quality and the production efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of welding technology, and in particular, to a method and device for monitoring laser-arc hybrid welding based on molten pool images. Background Art

[0002] Welding technology, as an important means of material processing, plays a key role in modern industrial production. From early manual arc welding to later gas shielded welding, laser welding, etc., various welding methods have emerged continuously to meet the welding requirements in different scenarios. For example, arc welding has been widely used in the field of medium and thick plate welding due to its simple equipment and flexible operation; laser welding, with its advantages of high energy density, fast welding speed, and small heat affected zone, has shown unique charm in precision manufacturing and thick plate welding. However, with the continuous improvement of industrial manufacturing levels, the requirements for welding quality are getting higher and higher. Especially in the field of medium and thick plate welding, such as the manufacturing of key structures like rail transit vehicle bodies, bolster beams, and large ship hulls, traditional single welding methods often struggle to meet the high-quality and high-efficiency welding requirements.

[0003] To address the deficiencies of traditional single welding methods, laser-arc hybrid welding technology has emerged. This technology combines laser welding and arc welding, leveraging the high energy density of the laser and the stability of the arc to achieve efficient and high-quality welding of thick plates. In practical applications, laser-arc hybrid welding technology is particularly suitable for high-quality welding of lightweight materials such as honeycomb aluminum. However, even with this advanced welding technology, various defects may still occur during the welding process, such as pores, cracks, lack of fusion, etc. These defects seriously affect the quality and safety of welded parts.

[0004] To identify the above-mentioned defects during the welding process, existing vision-based defect identification methods mainly rely on the surface features of molten pool images for defect judgment during welding. These methods typically include gray-scale changes of images, texture analysis, edge detection, etc. These methods often ignore the internal structure and size information of the molten pool, resulting in low detection accuracy. Summary of the Invention

[0005] The present invention provides a method and device for monitoring laser-arc hybrid welding based on molten pool images to solve the technical defect that existing vision-based defect identification methods mainly rely on the surface features of molten pool images for defect judgment during welding, resulting in low detection accuracy.

[0006] The present invention provides a device for monitoring laser-arc hybrid welding based on molten pool images, comprising:

[0007] A laser, which transmits laser light to the laser head through an optical fiber;

[0008] A laser head, which is located above the molten pool of the welded part and is used to shoot laser towards the molten pool;

[0009] An arc welding torch, which is located above the molten pool of the welded part, and the arc welding torch is connected to a welding power source;

[0010] A high-speed camera with multiple perspectives, which is installed above and on the side of the molten pool to capture the molten pool image containing the molten pool and the keyhole during the welding process and transmit it to an industrial control computer;

[0011] An industrial control computer, which is used to preprocess the molten pool image, align, register and fuse the preprocessed molten pool image to generate three-dimensional point cloud data of the molten pool and the keyhole; use a surface reconstruction algorithm to convert the three-dimensional point cloud data into a continuous three-dimensional surface model; input the three-dimensional surface model into a pre-trained convolutional neural network to perform real-time discrimination on whether defects occur during the welding process.

[0012] According to the laser-arc hybrid welding monitoring device based on molten pool images provided by the present invention, the high-speed camera includes:

[0013] A top-view high-speed camera, which is installed above the laser head to capture the top image of the molten pool and the keyhole during the welding process and transmit it to the industrial control computer;

[0014] Two side structured light high-speed cameras, which are installed on both sides of the arc welding torch to capture the side images of the molten pool and the keyhole during the welding process and transmit them to the industrial control computer.

[0015] According to the laser-arc hybrid welding monitoring device based on molten pool images provided by the present invention, the top-view high-speed camera captures the top image of the molten pool and the keyhole at a frequency of 200 Hz and transmits it to the industrial control computer in real time;

[0016] The two side structured light high-speed cameras are installed on both sides of the arc welding torch through mechanical jigs, project grating or stripe patterns at a frequency of 200 Hz, capture the side images of the reflected molten pool and the keyhole, and transmit them to the industrial control computer in real time.

[0017] The present invention provides a laser-arc hybrid welding monitoring method based on molten pool images for the laser-arc hybrid welding monitoring device based on molten pool images as described above. The method includes:

[0018] Receiving the molten pool image captured by the high-speed camera through the industrial control computer and preprocessing the molten pool image;

[0019] Aligning, registering and fusing the preprocessed molten pool image through the industrial control computer to generate three-dimensional point cloud data of the molten pool and the keyhole;

[0020] The industrial control computer uses a surface reconstruction algorithm to convert the three-dimensional point cloud data into a continuous three-dimensional surface model;

[0021] The industrial control computer inputs the three-dimensional surface model into a pre-trained convolutional neural network to perform real-time discrimination on whether defects occur during the welding process.

[0022] According to the laser-arc hybrid welding monitoring method based on the molten pool image provided by the present invention, the molten pool image includes a top image and two side structured light images;

[0023] Preprocess the molten pool image, including: denoising and enhancing the top image and the two side structured light images to obtain a processed top image and processed two side structured light images; perform edge detection on the processed top image to extract the contour information of the top image; decode the structured light pattern of the processed two side structured light images and extract the depth information of the structured light images.

[0024] According to the laser-arc hybrid welding monitoring method based on the molten pool image provided by the present invention, the industrial control computer aligns, registers, and fuses the preprocessed molten pool image to generate three-dimensional point cloud data of the molten pool and the keyhole, including: registering the preprocessed top image and the preprocessed two side structured light images using a feature matching algorithm; according to the registered top image and structured light images, fusing the contour information of the top image and the depth information of the structured light images to generate the three-dimensional point cloud data of the molten pool and the keyhole.

[0025] According to the laser-arc hybrid welding monitoring method based on the molten pool image provided by the present invention, use a surface reconstruction algorithm to convert the three-dimensional point cloud data into a continuous three-dimensional surface model, including: performing denoising processing on the three-dimensional point cloud data; using the Poisson surface reconstruction algorithm to perform surface reconstruction according to the denoised three-dimensional point cloud data; performing smoothing processing on the reconstructed surface to obtain a continuous three-dimensional surface model.

[0026] According to the laser-arc hybrid welding monitoring method based on the molten pool image provided by the present invention, input the three-dimensional surface model into a pre-trained convolutional neural network to perform real-time discrimination on whether defects occur during the welding process, including: converting the three-dimensional surface model into a three-dimensional vector and inputting it into the pre-trained convolutional neural network to obtain an output result; judging whether defects occur during the welding process according to the output result. If the output is 0, it means that no welding defect occurs. If the output is 1, it means that a welding defect occurs.

[0027] According to the laser-arc hybrid welding monitoring method based on the molten pool image provided by the present invention, the training method of the convolutional neural network includes:

[0028] Record the image data and actual defect conditions of each welding experiment as training data; among them, the experimental parameters of the welding experiment include: the current is between 80 - 300A, the laser power is between 2000 - 5000W, and the welding speed is between 0.5 - 3m / min.

[0029] Divide the training data into a training set, a validation set, and a test set. Input the training set into the established initial convolutional neural network, set the initial learning rate to 0.05, halve the learning rate every 10 training rounds, verify the training effect through the validation set every 5 training rounds, the number of training rounds is 150, and use the PyCharm framework for model training to finally obtain a trained convolutional neural network.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for monitoring laser - arc hybrid welding based on molten pool images as described in any one of the above.

[0031] The method and device for monitoring laser - arc hybrid welding based on molten pool images provided by the present invention collect molten pool images during the welding process through high - speed cameras from multiple perspectives, use an industrial control computer to pre - process the images, align, register, fuse, and perform three - dimensional reconstruction to generate a three - dimensional surface model of the molten pool and keyhole, and input it into a pre - trained convolutional neural network for real - time defect discrimination. This device can significantly improve the accuracy and real - time performance of welding defect detection, effectively solve the problem of insufficient detection accuracy caused by ignoring the internal size information of the molten pool in traditional visual information defect recognition methods, and at the same time reduce the cost and workload of post - welding manual detection, improving the welding quality and production efficiency. Description of the Drawings

[0032] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0033] Figure 1 It is a schematic structural diagram of the laser - arc hybrid welding monitoring device provided by the present invention.

[0034] Figure 2 It is a schematic diagram of a molten pool image including a molten pool and a keyhole provided by the present invention.

[0035] Figure 3 It is a schematic diagram after three - dimensional reconstruction of the molten pool and keyhole provided by the present invention.

[0036] Figure 4 It is a schematic flow diagram of the laser-arc hybrid welding monitoring method provided by the present invention.

[0037] Figure 5 It is a comparison diagram of the effect of defect recognition on the three-dimensional reconstructed surface model and the effect of defect recognition on a single image provided by the present invention.

[0038] Figure 6 It is a schematic structural diagram of the electronic device provided by the present invention.

[0039] Reference numerals

[0040] 101—Welded part; 102—Laser head; 103—Arc welding torch;

[0041] 104—Top-view high-speed camera; 105—Welding power source;

[0042] 106—Structured light high-speed camera; 107—Industrial control computer;

[0043] 108—Weld pool; 109—Keyhole. Detailed implementation manners

[0044] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] First, a schematic explanation is given to the noun terms involved in the embodiments of the present invention.

[0046] High-speed camera: A high-speed camera is a camera device that can capture dynamic scenes at a high frame rate (usually higher than 30 frames per second of an ordinary camera). Its frame rate can reach hundreds or even thousands of frames per second. During the welding process, the high-speed camera is used to capture the rapid dynamic changes of the weld pool and the keyhole, ensuring that the subtle changes and defects during the welding process can be recorded. This high-frame-rate shooting can provide more detailed and accurate image data, which is helpful for subsequent image processing and defect recognition.

[0047] Structured Light High-Speed Camera: A structured light high-speed camera is a device that combines structured light technology and high-speed imaging capabilities. Structured light technology projects specific grating or stripe patterns and captures the deformation of these patterns on the object surface to obtain depth information of the object. During the welding process, the structured light high-speed camera projects grating or stripe patterns and captures the side images of the molten pool and keyhole to obtain depth information. Combining these depth information with the top-view images can generate three-dimensional point cloud data of the molten pool and keyhole, providing a basis for subsequent three-dimensional model reconstruction.

[0048] Molten Pool: The molten pool refers to the liquid metal region formed by the local melting of the welding material and the base material due to the high temperature during the welding process. Characteristics such as the shape, size, and temperature distribution of the molten pool are crucial for controlling welding defects. The molten pool is the core area where metal melting and solidification occur during welding, and its stability directly affects the welding quality and the formation of defects. By monitoring the dynamic changes of the molten pool, it is possible to judge in real time whether defects occur during the welding process, thereby improving the welding quality.

[0049] Keyhole: The keyhole refers to a deep and narrow hole generated inside the molten pool due to the high energy density of the laser during laser welding or laser-arc hybrid welding. The keyhole usually forms during the welding process and closes as the molten pool solidifies after welding. The formation and stability of the keyhole have an important impact on the welding quality. The depth and shape of the keyhole can affect the penetration degree of the welded joint and the formation of defects. By monitoring the three-dimensional shape of the keyhole, it is possible to more accurately judge whether defects occur during the welding process, thereby improving the welding quality.

[0050] Laser-Arc Hybrid Welding: A welding technology that combines laser welding and arc welding. Utilizing the high energy density of the laser and the stability of the arc, it realizes efficient and high-quality welding of thick plates. Laser welding has the advantages of fast welding speed, small heat-affected zone, and beautiful weld formation, while arc welding has the characteristics of simple equipment and flexible operation. By combining the two, the respective advantages can be fully utilized to improve the welding efficiency and welding quality. It is suitable for welding medium and thick plates, especially in the manufacturing of key structures such as rail transit vehicle bodies, bolster beams, and large ship hulls, and can effectively solve the problems of difficult fusion of the side walls of medium and thick plate welds and unsightly welds.

[0051] Welding Power Source: A device that provides electrical energy for the welding process. Welding power sources usually include two types: DC power sources and AC power sources. In the present invention, the welding power source adopts the method of DC reverse connection, connecting the positive pole of the power source to the welding torch and connecting the negative pole of the power source to the welded part through an electronic controller and wires. The current of the welding power source is set to 150 - 250A, and welding is carried out using a double-pulse mode.

[0052] Arc welding torch: A welding tool used to generate and maintain an electric arc. An arc welding torch usually consists of parts such as an electrode, a nozzle, and an insulating housing. In the present invention, the angle between the arc welding torch and the vertical direction can be set to 30°.

[0053] Laser: A device that generates a laser beam. The laser generates a laser beam through stimulated emission and has characteristics such as high energy density, high directivity, and high monochromaticity. In the present invention, the laser transmits the laser through an optical fiber to the laser head, and the laser power is set to 2000 - 5000W.

[0054] Laser head: An optical device used to focus the laser beam onto the welding area. The laser head usually consists of optical elements such as lenses, mirrors, and collimators. In the present invention, the laser head welds perpendicular to the welding surface, the laser power is set to 2000 - 5000W, and the welding speed is set to 1 - 2m / min. The laser head is widely used in welding processes such as laser welding and laser - arc hybrid welding.

[0055] To solve the technical defect that the defect recognition method based on visual information in the prior art mainly relies on the surface features of the molten pool image to judge defects during the welding process, resulting in low detection accuracy, and to achieve the technical effects of improving the accuracy and real - time performance of welding defect detection, and enhancing the welding quality and production efficiency, an embodiment of the present invention discloses a single - wire laser - arc hybrid welding device, see Figure 1 , including: a laser, a laser head 102, an arc welding torch 103, a high - speed camera with multiple perspectives, and an industrial control computer 107.

[0056] Among them, the laser transmits the laser to the laser head 102 through an optical fiber. The output power of the laser can be adjusted according to welding requirements and is usually set between 2000 - 5000W. The laser head 102 is located above the molten pool 108 of the welded part 101 and is responsible for accurately shooting the laser at the molten pool 108 to promote the welding process. The position and angle of the laser head 102 can be accurately adjusted through a motion mechanism to ensure that the laser can accurately act on the weld seam. This high - precision laser positioning not only improves the welding efficiency but also ensures the welding quality.

[0057] Secondly, the arc welding torch 103 is also located above the molten pool 108 of the welded part 101 and is connected to the welding power supply 105. The arc welding torch 103 generates an arc during the welding process to provide additional heat for welding. The position and angle of the arc welding torch 103 can also be accurately adjusted through a motion mechanism to ensure that the arc can accurately act on the weld seam. The stability of the arc is crucial for the continuity and stability of the welding process. It can not only improve the welding efficiency but also reduce the generation of welding defects.

[0058] During the welding process, high-speed cameras with multiple perspectives are installed above and on the side of the molten pool 108, responsible for capturing molten pool images containing the molten pool 108 and the keyhole 109, as Figure 2 shown. These images are transmitted to the industrial control computer 107 in real time through the high-speed cameras, providing the basic data for subsequent image processing and defect discrimination. The high frame rate and high resolution of the high-speed cameras ensure the clarity and accuracy of the images, enabling the precise capture of the dynamic changes of the molten pool 108 and the keyhole 109.

[0059] In this embodiment, the high frame rate and high resolution of the high-speed cameras ensure the clarity and accuracy of the images, enabling the precise capture of the dynamic changes of the molten pool 108 and the keyhole 109.

[0060] The high-speed cameras include: a top-view high-speed camera 104 and two side structured-light high-speed cameras 106. Among them, the top-view high-speed camera 104 is installed above the laser head 102 and captures the top images of the molten pool 108 and the keyhole 109 at a frequency of 200 Hz. This high frame rate shooting can record the rapid dynamic changes of the molten pool 108 and the keyhole 109 during the welding process, ensuring the integrity and accuracy of the image data. The top-view image provides a top-down view of the molten pool 108 and the keyhole 109, which helps to analyze the shape, size and dynamic changes of the molten pool 108, as well as the formation and closing process of the keyhole 109.

[0061] The specific functions and advantages of the top-view high-speed camera 104 include:

[0062] High frame rate: The high frame rate of 200 Hz can capture the rapid dynamic changes of the molten pool 108 and the keyhole 109 during the welding process, ensuring the continuity and integrity of the image data.

[0063] High resolution: The high resolution of the top-view high-speed camera 104 ensures the clarity of the image, enabling the precise capture of the details of the molten pool 108 and the keyhole 109.

[0064] Real-time transmission: The top-view high-speed camera 104 transmits the image data to the industrial control computer 107 in real time through a high-speed transmission interface (such as USB 3.0 or GigE Vision), ensuring the timeliness and accuracy of subsequent image processing.

[0065] Precise alignment: The position and angle of the top-view high-speed camera 104 can be precisely aligned through a precision adjustment mechanism, ensuring the coordinated operation of the laser head 102 and the arc welding torch 103.

[0066] The two-sided structured light high-speed cameras 106 are installed on both sides of the arc welding torch 103, projecting grating or stripe patterns at a frequency of 200 Hz to capture the side images of the molten pool 108 and the keyhole 109 that are reflected back. Structured light technology obtains the depth information of an object by projecting specific grating or stripe patterns and capturing the deformation of these patterns on the object surface. The two-sided structured light images provide a side view of the molten pool 108 and the keyhole 109. Combining with the top view image, complete three-dimensional point cloud data can be generated, providing rich geometric information for subsequent three-dimensional reconstruction and defect discrimination.

[0067] The specific functions and advantages of the two-sided structured light high-speed cameras 106 include:

[0068] Structured light technology: The structured light high-speed cameras 106 project grating or stripe patterns and capture the side images of the molten pool 108 and the keyhole 109 to obtain depth information. This technology can accurately measure the three-dimensional shapes of the molten pool 108 and the keyhole 109, providing basic data for subsequent three-dimensional reconstruction, as Figure 3 shown.

[0069] High frame rate: The high frame rate of 200 Hz of the structured light high-speed cameras 106 ensures the continuity and integrity of the image data, and can capture the rapid dynamic changes of the molten pool 108 and the keyhole 109 during the welding process.

[0070] High resolution: The high resolution of the structured light high-speed cameras 106 ensures the clarity of the images, enabling the details of the molten pool 108 and the keyhole 109 to be accurately captured.

[0071] The two-sided structured light high-speed cameras 106 transmit the image data to the industrial control computer 107 in real time through a high-speed transmission interface (such as USB 3.0 or GigE Vision), ensuring the timeliness and accuracy of subsequent image processing.

[0072] The top-view high-speed camera 104 and the two-sided structured light high-speed cameras 106 work together to ensure the comprehensive monitoring of the molten pool 108 and the keyhole 109. Specifically, the top-view image provides a top-down view of the molten pool 108 and the keyhole 109, while the two-sided structured light images provide a side view. This multi-view monitoring can comprehensively capture the three-dimensional shapes of the molten pool 108 and the keyhole 109, providing rich geometric information for subsequent three-dimensional reconstruction and defect discrimination.

[0073] All cameras transmit the image data to the industrial control computer 107 in real time through a high-speed transmission interface, ensuring the timeliness and accuracy of subsequent image processing. Moreover, the positions and angles of all cameras can be accurately aligned through a precision adjustment mechanism, ensuring the coordinated work of the laser head 102 and the arc welding torch 103, and improving the accuracy and stability of the welding process.

[0074] For the industrial control computer 107, after receiving the molten pool images transmitted by the high-speed camera, it first performs preprocessing on them. The preprocessing includes operations such as denoising, enhancement, edge detection, and structured light decoding. These operations can effectively improve the quality and usability of the images. The preprocessed images are aligned, registered, and fused through a feature matching algorithm to generate the three-dimensional point cloud data of the molten pool 108 and the keyhole 109. This three-dimensional point cloud data contains the detailed geometric information of the molten pool 108 and the keyhole 109, providing a basis for subsequent surface reconstruction.

[0075] In this embodiment, the purpose of denoising is to remove the random noise and interference signals in the images and improve the clarity and contrast of the images. Gaussian Filter is used to smooth the images and remove the high-frequency noise. Gaussian Filter replaces each pixel value in the image with the weighted average of the pixel values in its neighborhood through a convolution operation, and the weights are determined by the Gaussian function. This method can effectively remove the random noise while retaining the main features of the images.

[0076] The purpose of enhancement is to enhance the contrast and details of the images, making the features of the molten pool 108 and the keyhole 109 more obvious. Histogram Equalization is used to enhance the contrast of the images. Histogram Equalization adjusts the pixel value distribution of the images to make the contrast of the images more uniform, thereby enhancing the details of the images. In addition, Adaptive Histogram Equalization (AHE) can also be used to process the contrast enhancement of local regions to avoid the over-enhancement problem caused by global enhancement.

[0077] Optionally, Contrast Enhancement is used to improve the clarity of the images, and the OpenCV library is used for image processing.

[0078] The purpose of edge detection is to extract the contour information of the molten pool 108 and the keyhole 109, providing a basis for subsequent feature matching and three-dimensional reconstruction. Canny Edge Detection algorithm is used to perform edge detection on the top-view images, and the threshold parameters are adjusted (the low threshold is 50 and the high threshold is 150), and the OpenCV library is used for edge detection.

[0079] The purpose of structured light decoding is to extract depth information from the structured light images on both sides, providing a basis for subsequent 3D reconstruction. The structured light images are decoded using the Gray Code method or the Phase Shifting method. The Gray Code method encodes the projected grating pattern as gray values and then recovers the depth information through a decoding algorithm. The Phase Shifting method projects multiple grating patterns with different phases and calculates the phase difference to obtain the depth information. These methods can accurately measure the depth changes of the molten pool 108 and the keyhole 109.

[0080] Optionally, in this embodiment, the Scale-Invariant Feature Transform (SIFT) feature matching algorithm can be used to perform feature matching on the top image and the side image. The threshold for feature point detection is set to 0.04, and the OpenCV library is used for feature matching.

[0081] The preprocessed images are aligned, registered, and fused through a feature matching algorithm to generate the 3D point cloud data of the molten pool 108 and the keyhole 109. The specific steps are as follows:

[0082] 1) Feature matching: Find the corresponding feature points in the top-view image and the side-view image, providing a basis for subsequent image registration.

[0083] The Scale-Invariant Feature Transform (SIFT) algorithm is used for feature point detection and matching. The SIFT algorithm calculates the extreme points in the scale space of the image, extracts the descriptors of the feature points, and then finds the corresponding feature points through the nearest neighbor matching method. This method can effectively handle the scale changes and rotation changes of the image, ensuring the accuracy of feature point matching.

[0084] 2) Image registration: Align the top-view image and the side-view image so that they can be fused in the same coordinate system.

[0085] Using the feature matching results, calculate the geometric relationship between the images through the Homography Matrix or the Essential Matrix, and then perform image registration. The Homography Matrix is applicable to planar scenes, while the Essential Matrix is applicable to 3D scenes. Through image registration, the top-view image and the side-view image can be aligned, providing a basis for subsequent 3D reconstruction.

[0086] 3) 3D point cloud generation: Generate the 3D point cloud data of the molten pool 108 and the keyhole 109, providing a basis for subsequent surface reconstruction.

[0087] Combining the contour information of the top-view image and the depth information of the side-view image, a point cloud generation algorithm based on feature matching is used to generate the three-dimensional point cloud data of the molten pool 108 and the keyhole 109. Specifically, through the feature matching results, each pixel point in the top-view image is associated with its corresponding pixel point in the side-view image. Combining the depth information, the three-dimensional coordinates of each pixel point are calculated, thereby generating the three-dimensional point cloud data.

[0088] Among them, the generated three-dimensional point cloud data contains the detailed geometric information of the molten pool 108 and the keyhole 109, which provides a basis for subsequent surface reconstruction. Specifically, the three-dimensional point cloud data includes:

[0089] The shape and size of the molten pool 108: Through the three-dimensional point cloud data, the shape and size of the molten pool 108 can be accurately measured, including the length, width, and depth of the molten pool 108. These information are of great significance for analyzing the stability and development trend of the molten pool 108.

[0090] The morphology of the keyhole 109: The three-dimensional point cloud data can accurately describe the morphology of the keyhole 109, including the depth, width, and shape changes of the keyhole 109. These information play a key role in judging whether defects occur during the welding process.

[0091] The dynamic changes of the molten pool 108 and the keyhole 109: By continuously capturing and processing the molten pool images, a series of three-dimensional point cloud data can be generated, reflecting the dynamic changes of the molten pool 108 and the keyhole 109 during the welding process. These dynamic change information are of great significance for real-time monitoring of the welding process and defect discrimination.

[0092] Next, the industrial control computer 107 uses a surface reconstruction algorithm to convert the three-dimensional point cloud data into a continuous three-dimensional surface model. The surface reconstruction algorithm can generate a smooth and continuous surface model according to the point cloud data, so that the three-dimensional morphology of the molten pool 108 and the keyhole 109 can be visually displayed. The generated three-dimensional surface model can not only help the operator better understand the morphology of the molten pool 108 and the keyhole 109 during the welding process, but also provide rich geometric information for defect discrimination.

[0093] The surface reconstruction algorithm is the key technology to convert discrete point cloud data into a continuous surface model. In this embodiment, the Poisson surface reconstruction algorithm is used for surface reconstruction. The Poisson surface reconstruction algorithm generates a smooth and continuous surface model by solving the Poisson equation, which can accurately describe the three-dimensional morphology of the molten pool 108 and the keyhole 109.

[0094] The Poisson surface reconstruction algorithm treats the point cloud data as sparse sampling points. By using the normal information of these points, an implicit function is constructed. This function has a high gradient value near the point cloud data and a low gradient value in the region far from the point cloud data. By solving the Poisson equation, a smooth and continuous implicit function can be obtained, thus generating a surface model.

[0095] The specific solution process can include the following steps:

[0096] 1) Normal estimation: Estimate the normal direction for each point cloud data point. This can be achieved by calculating the principal component analysis (PCA) within the local neighborhood of the point cloud data point.

[0097] 2) Gradient calculation: Calculate the gradient field according to the normal direction of the point cloud data point. The gradient field is represented as a sparse matrix, where the gradient value of each point is determined by its normal direction.

[0098] 3) Solving the Poisson equation: Generate an implicit function by solving the Poisson equation. The form of the Poisson equation is:

[0099] ▽ 2 f = ▽·N

[0100] where ▽ 2 is the Laplace operator, f is the implicit function, and ▽·N is the divergence of the gradient field.

[0101] 4) Surface extraction: Extract the isosurface from the implicit function to generate a continuous surface model. This can be achieved by the Marching Cubes algorithm.

[0102] The advantages of the Poisson surface reconstruction algorithm are as follows:

[0103] Smoothness: The Poisson surface reconstruction algorithm can generate a smooth and continuous surface model, avoiding the discreteness and noise influence of the point cloud data.

[0104] Detail preservation: The algorithm can preserve the detail information in the point cloud data, enabling the fine structures of the molten pool 108 and the keyhole 109 to be accurately displayed.

[0105] Robustness: The algorithm has strong robustness to the noise and irregularity of the point cloud data and can generate high-quality surface models in complex data environments.

[0106] The three-dimensional surface model generated by the Poisson surface reconstruction algorithm can intuitively display the shape, size, and depth of the molten pool 108 and the keyhole 109, enabling the operator to clearly see the dynamic changes during the welding process. Moreover, by continuously capturing and processing the molten pool images to generate a series of three-dimensional surface models, the dynamic changes of the molten pool 108 and the keyhole 109 can be monitored in real time, and potential defects can be detected in a timely manner.

[0107] In addition, the three-dimensional surface model provides rich geometric information, including the shape, size, depth, and variation trend of the molten pool 108 and the keyhole 109. This information is of great significance for judging whether defects occur during the welding process. By analyzing the geometric features of the three-dimensional surface model, features related to defects can be extracted, such as the irregular shape of the molten pool 108 and the abnormal depth change of the keyhole 109. These features can be used as the basis for defect discrimination.

[0108] Finally, the industrial control computer 107 inputs the generated three-dimensional surface model into a pre-trained convolutional neural network to perform real-time discrimination on whether defects occur during the welding process. The convolutional neural network has learned the defect features under different welding conditions through a large amount of training data and can accurately identify defects during the welding process. Through real-time monitoring and discrimination, operators can promptly discover and handle welding defects, thereby improving the welding quality and production efficiency.

[0109] In practical applications, the device can further optimize the welding process through the multi-layer multi-pass welding method. For example, when using the three-layer three-pass welding method, the welding parameters and the control strategy of the arc shape for each weld pass can be adjusted according to specific requirements. Through multi-layer multi-pass welding, complete fusion of the weld seam can be ensured, lack of fusion defects can be reduced, and the welding quality can be improved. In addition, the electronic controller generates 200 control signals within each cycle, and the control time for each control signal is 5 ms, achieving high-frequency control of 200 times per second. This high-frequency control enables the arc shape to be precisely adjusted according to different requirements during the welding process, ensuring complete fusion of the weld seam sidewalls and reducing lack of fusion defects.

[0110] Through the above embodiments, the present invention provides an efficient and accurate single-wire laser-arc hybrid welding device. The molten pool images during the welding process are collected by high-speed cameras from multiple perspectives, and the industrial control computer 107 preprocesses, aligns, registers, fuses, and performs three-dimensional reconstruction on the images to generate a three-dimensional surface model of the molten pool 108 and the keyhole 109, and inputs it into a pre-trained convolutional neural network for real-time defect discrimination. The device can significantly improve the accuracy and real-time performance of welding defect detection, effectively solve the problem of insufficient detection accuracy caused by ignoring the internal dimension information of the molten pool 108 in traditional visual information defect recognition methods, and at the same time reduce the cost and workload of post-weld manual inspection, improve the welding quality and production efficiency, and has important practical application value.

[0111] An embodiment of the present invention also discloses a laser-arc hybrid welding monitoring method based on molten pool images, which is used for the laser-arc hybrid welding monitoring device as described above. Refer to Figure 4 , and the method includes:

[0112] 401. Receive the molten pool images captured by the industrial control computer and preprocess the molten pool images.

[0113] In this embodiment, the industrial control computer 107 first receives the molten pool images captured by the high-speed camera through a high-speed transmission interface (such as USB 3.0 or GigE Vision). These images include top-view images and side structured light images, which are transmitted to the industrial control computer 107 in real time at a frequency of 200 Hz. After receiving the images, the industrial control computer 107 first preprocesses them to improve the quality and usability of the images.

[0114] Specifically, the molten pool images include top images and side structured light images; step 401 specifically includes: performing denoising and enhancement processing on the top images and side structured light images to obtain the processed top images and processed side structured light images; performing edge detection on the processed top images to extract the contour information of the top images; decoding the structured light patterns of the processed side structured light images and extracting the depth information of the structured light images.

[0115] Specifically, for denoising: Use Gaussian Filter to perform smoothing processing on the molten pool images to remove high-frequency noise. Gaussian Filter replaces each pixel value in the molten pool image with the weighted average of the pixel values in its neighborhood through a convolution operation, and the weights are determined by the Gaussian function. The denoising process can remove random noise and interference signals in the molten pool images and improve the clarity and contrast of the images.

[0116] For enhancement: Use Histogram Equalization to enhance the contrast of the molten pool images. Histogram Equalization adjusts the pixel value distribution of the molten pool images to make the contrast of the molten pool images more uniform, thereby enhancing the details of the molten pool images. The enhancement process can enhance the contrast and details of the molten pool images, making the features of the molten pool 108 and the keyhole 109 more obvious.

[0117] For edge detection: Use the Canny edge detection algorithm to perform edge detection on the processed top images. The Canny algorithm detects the edges in the images by calculating the gradient magnitude and direction of the images and combining the double-threshold method. Edge detection can extract the contour information of the molten pool 108 and the keyhole 109, providing a basis for subsequent feature matching and three-dimensional reconstruction.

[0118] For structured light decoding: Use the Gray Code method or the Phase Shifting method to decode the structured light images. The Gray Code method encodes the projected grating pattern as a gray value and then restores the depth information through a decoding algorithm. Structured light decoding can extract the depth information from the side structured light images, providing a basis for subsequent three-dimensional reconstruction.

[0119] 402. Align, register, and fuse the preprocessed molten pool images through the industrial control computer to generate three-dimensional point cloud data of the molten pool and the keyhole.

[0120] In this step, use the feature matching algorithm to register the preprocessed top image and the preprocessed side structured light images; according to the registered top image and structured light images, fuse the contour information of the top image and the depth information of the structured light image to generate three-dimensional point cloud data of the molten pool 108 and the keyhole 109.

[0121] The specific steps are as follows:

[0122] Feature matching: Use the Scale-Invariant Feature Transform (SIFT) algorithm for feature point detection and matching. The SIFT algorithm calculates the extreme points in the scale space of the image, extracts the descriptors of the feature points, and then finds the corresponding feature points through the nearest neighbor matching method. Feature matching can find the corresponding feature points in the top view image and the side view image, providing a basis for subsequent image registration.

[0123] Image registration: Use the feature matching results to calculate the geometric relationship between the images through the Homography Matrix or the Essential Matrix, and then perform image registration. Image registration can align the top view image and the side view image so that they can be fused in the same coordinate system.

[0124] Three-dimensional point cloud generation: Combine the contour information of the top view image and the depth information of the side view image, and use the point cloud generation algorithm based on feature matching to generate three-dimensional point cloud data of the molten pool 108 and the keyhole 109. Generating the three-dimensional point cloud data of the molten pool 108 and the keyhole 109 can provide a basis for subsequent surface reconstruction.

[0125] 403. Use the surface reconstruction algorithm through the industrial control computer to convert the three-dimensional point cloud data into a continuous three-dimensional surface model.

[0126] Step 403 specifically includes: denoising the three-dimensional point cloud data; using the Poisson surface reconstruction algorithm to perform surface reconstruction according to the denoised three-dimensional point cloud data; smoothing the reconstructed surface to obtain a continuous three-dimensional surface model.

[0127] Use the Poisson Surface Reconstruction algorithm to convert the three-dimensional point cloud data into a continuous three-dimensional surface model. The Poisson surface reconstruction algorithm generates a smooth and continuous surface model by solving the Poisson equation. The specific steps include:

[0128] Normal estimation: Estimate the normal direction for each point cloud data point. This can be achieved by calculating the principal component analysis (PCA) within the local neighborhood of the point cloud data point.

[0129] Gradient calculation: Calculate the gradient field based on the normal direction of the point cloud data point. The gradient field is represented as a sparse matrix, where the gradient value of each point is determined by its normal direction.

[0130] Poisson equation solving: Generate an implicit function by solving the Poisson equation. The form of the Poisson equation is: ▽ 2 f = ▽·N. Where, ▽ 2 is the Laplace operator, f is the implicit function, and ▽·N is the divergence of the gradient field.

[0131] Surface extraction: Extract the isosurface from the implicit function to generate a continuous surface model. This can be achieved by the Marching Cubes algorithm.

[0132] 404. Input the three-dimensional surface model into a pre-trained convolutional neural network through an industrial control computer to perform real-time discrimination on whether defects occur during the welding process.

[0133] Step 404 specifically includes: Convert the three-dimensional surface model into a three-dimensional vector, input it into a pre-trained convolutional neural network, and obtain the output result; Determine whether defects occur during the welding process according to the output result. If the output is 0, it indicates that no welding defects occur. If the output is 1, it indicates that welding defects occur.

[0134] Among them, the training method of the convolutional neural network includes:

[0135] Record the image data and actual defect conditions of each welding experiment as training data; Among them, the experimental parameters of the welding experiment include: the current is between 80 - 300 A, the laser power is between 2000 - 5000 W, and the welding speed is between 0.5 - 3 m / min.

[0136] Divide the training data into a training set, a validation set, and a test set. Input the training set into the established initial convolutional neural network, set the initial learning rate to 0.05, halve the learning rate every 10 training rounds, verify the training effect through the validation set every 5 training rounds, the number of training rounds is 150, and use the PyCharm framework for model training to finally obtain a trained convolutional neural network.

[0137] Embodiments of the present invention can accurately identify defects during the welding process through a convolutional neural network, improving the accuracy of defect detection. See Figure 5 , Figure 5Shows a comparison chart of the effect of defect recognition on the three-dimensional reconstructed surface model and the effect of defect recognition on a single image in this embodiment.

[0138] As can be seen from Figure 5 in it, when the surface model of the three-dimensional reconstruction is input into the convolutional neural network (CNN) for defect discrimination, its accuracy can reach 98.6%; when a single image is input into the convolutional neural network (CNN) for defect discrimination, its accuracy can reach 82.6%. By inputting the surface model of the three-dimensional reconstruction into the CNN model for defect recognition, its accuracy is significantly higher than the defect recognition result of inputting a single image into the CNN (16% higher). This indicates that using the surface model of the three-dimensional reconstruction for defect discrimination is more accurate than using a single image method, probably because the three-dimensional model provides richer geometric information, which helps to improve the accuracy of defect detection.

[0139] The laser-arc hybrid welding monitoring method based on the molten pool image provided by the embodiment of the present invention collects the molten pool images during the welding process through high-speed cameras with multiple perspectives, uses an industrial computer to preprocess, align, register, fuse, and three-dimensionally reconstruct the images, generates a three-dimensional surface model of the molten pool and the keyhole, and inputs it into a pre-trained convolutional neural network for real-time defect discrimination. This device can significantly improve the accuracy and real-time performance of welding defect detection, effectively solve the problem of insufficient detection accuracy caused by ignoring the internal dimension information of the molten pool in the traditional visual information defect recognition method, and at the same time reduce the cost and workload of manual inspection after welding, improving the welding quality and production efficiency.

[0140] In addition, the technical effects of the embodiment of the present invention also include:

[0141] (1) Multi-perspective molten pool image acquisition: The embodiment of the present invention uses a top-view high-speed camera and two side structured-light high-speed cameras to capture the top view and side depth information of the molten pool and the keyhole at a frequency of 200 Hz respectively. This multi-perspective image acquisition method can comprehensively and accurately reflect the three-dimensional morphology of the molten pool and the keyhole, providing a high-quality data basis for subsequent model reconstruction.

[0142] (2) Image preprocessing and registration fusion: By performing preprocessing steps such as denoising, enhancement, edge detection, and structured-light decoding on the top image and the side image, the quality and usability of the images are effectively improved. In particular, the top-view image and the structured-light image are registered through a feature matching algorithm, and combined with the contour information of the top-view image and the depth information of the structured-light image, three-dimensional point cloud data of the molten pool and the keyhole are generated. This process ensures the consistency and accuracy of the data, laying a solid foundation for subsequent three-dimensional surface model reconstruction.

[0143] (3) 3D surface model reconstruction and surface reconstruction: The present invention uses a surface reconstruction algorithm to convert 3D point cloud data into a continuous 3D surface model. This process can not only intuitively display the 3D morphology of the molten pool and keyhole, but also capture subtle structural changes, providing rich information for defect identification. By matching the 3D surface model with actual welding defects, the corresponding relationship between defect data and the 3D surface model is established, providing a reliable basis for subsequent defect discrimination.

[0144] (4) Application of 3D convolutional neural network: In the embodiment of the present invention, the 3D surface model is converted into a 3D vector of 80×80×80 and input into a convolutional neural network for training. Through 5 convolutional layers and 2 fully connected layers, the convolutional neural network can effectively extract the features of the molten pool and keyhole models and match them with welding defects, enabling real-time and accurate discrimination of whether defects occur during the welding process, greatly improving the accuracy of defect monitoring.

[0145] Figure 6 Illustrates a schematic diagram of the physical structure of an electronic device, such as Figure 6 shown. The electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 complete mutual communication through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the single-wire laser-arc hybrid welding method, which includes: receiving the molten pool image captured by a high-speed camera through an industrial control computer and preprocessing the molten pool image; aligning, registering, and fusing the preprocessed molten pool image through the industrial control computer to generate 3D point cloud data of the molten pool and keyhole; using a surface reconstruction algorithm through the industrial control computer to convert the 3D point cloud data into a continuous 3D surface model; inputting the 3D surface model into a pre-trained convolutional neural network through the industrial control computer to perform real-time discrimination on whether defects occur during the welding process.

[0146] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A laser-arc hybrid welding monitoring device based on molten pool image, characterized in that: include: A laser, which transmits laser light to the laser head via an optical fiber; A laser head, the laser head is located above the molten pool of the weldment and is used to shoot laser toward the molten pool; An arc welding gun, the arc welding gun is located above the molten pool of the weldment, and the arc welding gun is connected to a welding power source; A high-speed camera with multiple viewing angles is installed above and on the side of the molten pool to capture images of the molten pool including the molten pool and the keyhole during the welding process and transmit them to the industrial computer; An industrial computer is used to preprocess the molten pool image, align, register and fuse the preprocessed molten pool image, and generate three-dimensional point cloud data of the molten pool and the keyhole; The three-dimensional point cloud data is converted into a continuous three-dimensional surface model using a surface reconstruction algorithm; the three-dimensional surface model is input into a pre-trained convolutional neural network to perform real-time judgment on whether defects occur in the welding process.

2. The laser-arc hybrid welding monitoring device based on molten pool image according to claim 1 is characterized in that: The high-speed camera comprises: A top-view high-speed camera is installed above the laser head to capture the top images of the molten pool and keyhole during the welding process and transmit them to the industrial computer; The structured light high-speed cameras on both sides are installed on both sides of the arc welding gun to capture the side images of the molten pool and the keyhole during the welding process and transmit them to the industrial computer.

3. The laser-arc hybrid welding monitoring device based on molten pool image according to claim 2 is characterized in that: The top-view high-speed camera captures the top images of the molten pool and the keyhole at a frequency of 200 Hz and transmits them to the industrial computer in real time; The two-side structured light high-speed cameras are installed on both sides of the arc welding gun through mechanical fixtures, and project grating or stripe patterns at a frequency of 200 Hz to capture the reflected molten pool and the side images of the keyhole, and transmit them to the industrial computer in real time.

4. A laser-arc hybrid welding monitoring method based on a molten pool image, used in the laser-arc hybrid welding monitoring device based on a molten pool image as claimed in any one of claims 1 to 3, characterized in that: The method comprises: Receiving the molten pool image taken by the high-speed camera through the industrial computer, and preprocessing the molten pool image; The pre-processed molten pool images are aligned, registered and fused by an industrial computer to generate three-dimensional point cloud data of the molten pool and keyhole; The 3D point cloud data is converted into a continuous 3D surface model using a surface reconstruction algorithm through an industrial computer; The three-dimensional surface model is input into a pre-trained convolutional neural network through an industrial computer to perform real-time judgment on whether defects occur in the welding process.

5. The laser-arc hybrid welding monitoring method based on molten pool image according to claim 4 is characterized in that: The molten pool image includes a top image and structured light images on both sides; The molten pool image is preprocessed, including: Performing denoising and enhancement processing on the top image and the two-side structured light images to obtain a processed top image and a processed two-side structured light images; Performing edge detection on the processed top image to extract contour information of the top image; The processed structured light images on both sides are decoded into structured light patterns, and depth information of the structured light images is extracted.

6. The laser-arc hybrid welding monitoring method based on molten pool image according to claim 5 is characterized in that: The pre-processed molten pool images are aligned, registered and fused by an industrial computer to generate three-dimensional point cloud data of the molten pool and the keyhole, including: The preprocessed top image and the preprocessed two-side structured light images are registered using a feature matching algorithm; According to the registered top image and structured light image, contour information of the top image and depth information of the structured light image are fused to generate three-dimensional point cloud data of the molten pool and the keyhole.

7. The laser-arc hybrid welding monitoring method based on molten pool image according to any one of claims 4 to 6, characterized in that: Use surface reconstruction algorithms to convert 3D point cloud data into a continuous 3D surface model, including: De-noising the three-dimensional point cloud data; Use Poisson surface reconstruction algorithm to perform surface reconstruction based on the denoised 3D point cloud data; The reconstructed surface is smoothed to obtain a continuous three-dimensional surface model.

8. The laser-arc hybrid welding monitoring method based on molten pool image according to any one of claims 4 to 6, characterized in that: The three-dimensional surface model is input into a pre-trained convolutional neural network to perform real-time judgment on whether defects occur in the welding process, including: Converting the three-dimensional surface model into a three-dimensional vector and inputting it into a pre-trained convolutional neural network to obtain an output result; The output result is used to determine whether the welding process has defects. If the output is 0, it means that no welding defects have occurred. If the output is 1, it means that welding defects have occurred.

9. The laser-arc hybrid welding monitoring method based on molten pool image according to any one of claims 4 to 6, characterized in that: The training method of the convolutional neural network includes: The image data and actual defect conditions of each welding experiment are recorded as training data; the experimental parameters of the welding experiment include: the current is between 80-300 A, the laser power is between 2000-5000 W, and the welding speed is between 0.5-3m / min; The training data is divided into a training set, a validation set and a test set. The training set is input into the established initial convolutional neural network. The initial learning rate is set to 0.

05. The learning rate is halved after every 10 rounds of training. The training effect is verified by the validation set after every 5 rounds of training. The number of training rounds is 150. The model is trained using the PyCharm framework to finally obtain a trained convolutional neural network.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the laser-arc hybrid welding monitoring method based on the molten pool image as described in any one of claims 4 to 9 is implemented.

Images