A method for real-time control of welding process based on visual inspection of weld pool
By combining visual inspection and neural networks, the characteristics of the molten pool during the welding process are extracted, and a neural network model is constructed. This solves the problem of deep penetration welding technology relying on human experience, and realizes automated control and improves welding quality.
Patent Information
- Application Number
- CN202410485414.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-22
AI Technical Summary
Existing deep penetration welding technology relies on the welder's personal experience and cannot meet the needs of production automation and intelligence, lacking real-time control methods.
By combining visual inspection technology and neural networks, the welding process is captured by a CCD industrial camera, the features of the molten pool are extracted, and feedback neural networks and Transformer neural networks are constructed to achieve real-time control and intervention of the welding process.
It has enabled the automation of deep penetration welding technology, improved production efficiency and welding yield, and enhanced the accuracy and automation of the welding process.
Smart Images

Figure CN118172358B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of automatic welding equipment, and more specifically to a real-time control method for the welding process based on visual inspection of the weld pool. Background Technology
[0002] Deep penetration welding is a novel and clean welding method with advantages such as no beveling required, one-time penetration of medium-thick plates, high welding efficiency, and low equipment cost. It has unique advantages in welding medium-thick plates. Deep penetration welding can achieve single-sided welding with double-sided forming. When the butt joint gap changes, skilled welders can observe the characteristics of the molten pool and keyholes to adjust the welding process in a timely manner, ensuring the quality of the weld formation and thus obtaining a welded joint with excellent performance.
[0003] However, current deep penetration welding technology relies heavily on the welder's personal experience and skill, which is insufficient to meet the demands of industrial upgrading and transformation in the context of current production automation and intelligentization. Therefore, it is necessary to improve existing technologies to promote the application of deep penetration welding in the field of production automation. Summary of the Invention
[0004] The purpose of this invention is to address the problems in the prior art by providing a real-time control method for the welding process based on visual inspection of the weld pool. This method combines visual inspection technology with neural network technology to achieve real-time control and intervention of the automatic welding process, thereby promoting the automation of deep penetration welding technology.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] A real-time control method for welding process based on visual inspection of the weld pool includes the following steps:
[0007] S1. Filming the manual welding process: During the welding process, a CCD industrial camera captures video images of the welding area and records in real time the adjustments made by the welder to the welding parameters of the welding machine to form a welding parameter-time curve; the welding parameters include welding speed, welding current and welding height.
[0008] S2. Extracting the molten pool feature map: Visual inspection and analysis are performed on the video images to identify the molten pool features in the video images frame by frame in order to extract and form a molten pool feature map; the molten pool features include the gas-liquid interface line, the solid-liquid interface line, and the molten pool boundary line;
[0009] S3. Extract the changes in molten pool features: Compare the changes in molten pool features in the molten pool feature map in chronological order to obtain the changes in molten pool features; at the same time, based on the welding parameter-time curve, obtain the changes in welding parameters corresponding to the changes in molten pool features before and after the changes.
[0010] The changes in molten pool characteristics include changes in the shape and displacement of the gas-liquid interface line, changes in the shape and displacement of the solid-liquid interface line, and changes in molten pool width, molten pool length, and molten pool area caused by changes in the molten pool boundary line; the changes in welding parameters include changes in welding speed, welding current, and welding height.
[0011] S4. Create a sample set: Based on the time correspondence, match the changes in molten pool characteristics with the changes in welding parameters to form the first sample set; match the molten pool feature map with the welding parameters to form the second sample set;
[0012] S5. Construct a neural network and perform pre-training: Construct a feedback neural network with the change in molten pool characteristics as input and the change in welding parameters as output. Use the first sample set to train the feedback neural network to obtain the mapping relationship between the change in molten pool characteristics and the change in welding parameters.
[0013] A Transformer neural network is constructed, with the molten pool feature map as input and welding parameters as output. The Transformer neural network is trained using a second sample set to obtain the mapping relationship between the molten pool feature map and the welding parameters.
[0014] S6. Obtain a reference benchmark: Capture a standard video image of a welder manually welding with a welding machine using a CCD industrial camera, and perform visual inspection and analysis on the standard video image using the same method as in S2. Identify the molten pool features in the standard video image frame by frame to extract and form a standard molten pool feature map, which serves as a reference benchmark during the trial run of the neural network.
[0015] S7. Neural Network Trial Run: Assign weight ratios to the feedback neural network and the Transformer neural network, and then deploy them online; the weight ratio of the feedback neural network is n1, the weight ratio of the Transformer neural network is n2, and n1+n2=1; during the initial neural network trial run, n1 takes the value of 1, and n2 takes the value of 0.
[0016] When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map.
[0017] The real-time molten pool feature map is compared with the standard molten pool feature map in S6 to obtain the molten pool feature deviation; the molten pool feature deviation is input into the feedback neural network to obtain the output welding parameter correction; the welding parameter correction is added to the welding parameters currently in use to obtain the first welding parameter P1; the real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2;
[0018] The first welding parameter P1 and the second welding parameter P2 are weighted and summed to obtain the third welding parameter P3 = n1×P1 + n2×P2; the welding process of the welding machine is intervened in real time based on the third welding parameter P3.
[0019] S8. Use the trial run samples to supplement the training of the neural network: After the neural network has been running for a period of time, according to the time correspondence, match the real-time molten pool feature map used in S7 with the third welding parameter P3 to form a third sample set; use the third sample set to supplement the training of the Transformer neural network.
[0020] S9. Adjust the weight ratio and run again: After the supplementary training is completed, adjust the weight ratio of the feedback neural network and the Transformer neural network, decrease the weight ratio n1 of the feedback neural network and increase the weight ratio n2 of the Transformer neural network; then repeat S7 to S9 multiple times until the weight ratio n1 of the feedback neural network decreases to 0 and the weight ratio n2 of the Transformer neural network increases to 1.
[0021] S10, Neural Network Officially Runs: Set the weight ratio n1 of the feedback neural network to 0, set the weight ratio n2 of the Transformer neural network to 1, and then deploy it online;
[0022] When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map.
[0023] The real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2; based on the second welding parameter P2, the welding process of the welding machine is intervened in real time.
[0024] Furthermore, in step S2, the method for extracting the molten pool feature map includes the following steps:
[0025] S201. Select ROI region: Identify the arc region in the video image and select the ROI region for image processing around the arc region;
[0026] S202, Filtering and Segmentation: Smooth the colors of the image in the ROI region, and then segment the image into several regions according to color similarity;
[0027] S203, Flood Fill: Fill the segmented areas with different and dissimilar colors;
[0028] S204, Edge Detection: Use convolution kernels to calculate the gradient of the image in the horizontal and vertical directions to detect the boundary lines of different color regions in S202, and output the set of all detected boundary lines as a melt pool feature map.
[0029] Further, step S201 includes the following steps:
[0030] S2011. Convert the video image captured by the CCD industrial camera into a grayscale image, and then convert the grayscale image into a binary image using a set grayscale value as a threshold.
[0031] S2012. In real time, traverse each region of the binary image and determine the white blocks in the binary image with an area greater than a certain threshold as electric arc regions.
[0032] S2013. Calculate the centroid coordinates of the arc region, and generate a bounding box of ROI with the centroid coordinates of the arc region as the center. Use the region in the bounding box of ROI as the ROI region for subsequent processing.
[0033] Furthermore, in S2013, the continuous video images are divided into 3-second cycles. When the ROI region of the video image in the current cycle is selected, the ROI box is generated with the centroid of the arc region in all video images in the previous cycle as the center, and the ROI box is periodically and intermittently updated according to this rule.
[0034] Further, step S202 includes the following steps:
[0035] S2021. Establishing an iterative space: On the input image, with any point as the center, establish a spherical space with radius Sr in the color space, where the R, G, and B values of the color space are set to 0-255; establish a spherical space with radius Sp in the coordinate space, where the x and y values of the coordinate space are the length and width of the input image, respectively; the color space and coordinate space constitute a five-dimensional joint iterative space.
[0036] S2022. Obtain the vectors in the iteration space and recalculate the vectors after moving the sphere in the iteration space until convergence: In the constructed iteration space, calculate the mean of the color vectors from all points in the iteration space to the origin of the color space. Use the magnitude of this mean as the distance the origin of the color space is moved, and use the direction of this mean as the direction the origin of the color space is moved. Update the origin of the color space in this way. Then calculate the mean of the color vectors from all points in the iteration space to the updated origin of the color space, and update the origin of the color space again until the mean of the color vectors meets a certain threshold.
[0037] S2023, Region Segmentation: Using color similarity and spatial location similarity as the criteria, points with color similarity and spatial location similarity higher than a set threshold are merged into the same region, thereby segmenting the input image into several regions;
[0038] S2024, Region Merging: From the several regions segmented by S2023, find the regions with fewer than a certain threshold of pixels and merge them into the adjacent regions with the highest color similarity and spatial similarity.
[0039] Further, S204 includes the following steps:
[0040] S2041, Definition and Two convolutional kernels simultaneously create two zero matrices of the same size as the image, used to store the gradient values of the image in the horizontal and vertical directions;
[0041] S2042. Iterate through the three RGB channels of the image in sequence, and perform a horizontal convolution operation on the elements in each channel using G(x) to obtain the gradient in the horizontal direction. Then perform a vertical convolution operation using G(y) to obtain the gradient in the vertical direction.
[0042] S2043. For each pixel in the image, calculate the gradient magnitude and gradient direction of each pixel based on the gradient in the horizontal direction and the gradient in the vertical direction obtained in S2032, so as to extract the boundary lines of each region in the image.
[0043] S2044. The boundary lines extracted in S2033 are processed using the non-maximum suppression method to eliminate fine boundaries.
[0044] S2045. Output the processed image, which retains only the boundary lines, as a molten pool feature map.
[0045] Furthermore, before using a CCD industrial camera to capture video images of the welded area, the following steps are also included:
[0046] Camera calibration is used to determine the size mapping relationship between pixels in video images and real physical space, which is then used for subsequent calculations of molten pool size data.
[0047] Furthermore, in S7, the method for real-time intervention in the welding process of the welding machine based on the third welding parameter P3 is semi-automatic intervention, specifically including:
[0048] When a welder uses a welding machine for manual welding, the welding parameters used by the welder in real time are compared with the third welding parameter P3 output in real time. When the deviation between the two is greater than the set threshold, the welding parameters of the welding machine are forcibly set to the third welding parameter P3.
[0049] Furthermore, in S10, the method of real-time intervention of the welding process of the welding machine according to the second welding parameter P2 is fully automatic intervention, which specifically includes: the welding machine performs fully automatic welding using the real-time output second welding parameter P2 when no one is operating it.
[0050] Furthermore, in S9, after each adjustment of the weight ratio of the neural network and a re-run, the image of the welded part formed by the neural network intervention is compared with the image of the welded part formed by standard manual welding to evaluate the similarity between the two; if the similarity after adjusting the weight ratio is lower than the similarity before adjusting the weight ratio, the weight ratio of the neural network is restored to the value before adjustment and the re-run is performed.
[0051] This invention provides a real-time welding process control method based on visual inspection of the weld pool. It efficiently and comprehensively extracts weld pool features using visual inspection technology, and combines this with deep learning technology to obtain the mapping relationship between weld pool features, their changes, and welding parameters. Furthermore, through repeated trial runs and supplementary training, the neural network model is continuously improved, ultimately achieving automated intervention and control of the welding process. This invention promotes the automation of deep penetration welding technology, helps improve production efficiency and weld yield, and has broad application prospects.
[0052] This invention employs a combination of feedback neural networks and Transformer neural networks to construct a neural network model for automated welding. The Transformer neural network increases the model's receptive field and better captures local information in images, enabling the model to obtain a deeper and more comprehensive mapping relationship and improving the accuracy of deep learning. Simultaneously, this invention addresses the dependence of the Transformer neural network on large sample data by assigning different weight ratios to the feedback neural network and the Transformer neural network. In the initial stage of model deployment, the feedback neural network is dominant to ensure basic model stability. As the model continues to run and undergoes supplementary training, the weight ratio of the Transformer neural network is gradually increased in the later stages to further improve the model's accuracy and automation. This method of combining the feedback neural network and the Transformer neural network with variable weight ratios significantly shortens the training cycle before model deployment and allows the model to continuously improve itself during operation, ultimately achieving a more accurate and automated control process. Attached Figure Description
[0053] Figure 1 This is a flowchart illustrating a real-time control method for welding process based on visual inspection of the weld pool, provided by an embodiment of the present invention.
[0054] Figure 2 This is a flowchart illustrating the image processing steps S201 to S204 in an embodiment of the present invention. Detailed Implementation
[0055] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0056] like Figure 1 As shown in the figure, an embodiment of the present invention provides a real-time control method for welding process based on visual inspection of the weld pool, comprising the following steps:
[0057] S1. Filming the manual welding process: During the welding process, a CCD industrial camera captures video images of the welding area and records in real time the adjustments made by the welder to the welding parameters of the welding machine to form a welding parameter-time curve; the welding parameters include welding speed, welding current and welding height.
[0058] S2. Extracting the molten pool feature map: Visual inspection and analysis are performed on the video images to identify the molten pool features in the video images frame by frame in order to extract and form a molten pool feature map; the molten pool features include the gas-liquid interface line, the solid-liquid interface line, and the molten pool boundary line;
[0059] S3. Extract the changes in molten pool features: Compare the changes in molten pool features in the molten pool feature map in chronological order to obtain the changes in molten pool features; at the same time, based on the welding parameter-time curve, obtain the changes in welding parameters corresponding to the changes in molten pool features before and after the changes.
[0060] The changes in molten pool characteristics include changes in the shape and displacement of the gas-liquid interface line, changes in the shape and displacement of the solid-liquid interface line, and changes in molten pool width, molten pool length, and molten pool area caused by changes in the molten pool boundary line; the changes in welding parameters include changes in welding speed, welding current, and welding height.
[0061] S4. Create a sample set: Based on the time correspondence, match the changes in molten pool characteristics with the changes in welding parameters to form the first sample set; match the molten pool feature map with the welding parameters to form the second sample set;
[0062] S5. Construct a neural network and perform pre-training: Construct a feedback neural network with the change in molten pool characteristics as input and the change in welding parameters as output. Use the first sample set to train the feedback neural network to obtain the mapping relationship between the change in molten pool characteristics and the change in welding parameters.
[0063] A Transformer neural network is constructed, with the molten pool feature map as input and welding parameters as output. The Transformer neural network is trained using a second sample set to obtain the mapping relationship between the molten pool feature map and the welding parameters.
[0064] S6. Obtain a reference benchmark: Capture a standard video image of a welder manually welding with a welding machine using a CCD industrial camera, and perform visual inspection and analysis on the standard video image using the same method as in S2. Identify the molten pool features in the standard video image frame by frame to extract and form a standard molten pool feature map, which serves as a reference benchmark during the trial run of the neural network.
[0065] S7. Neural Network Trial Run: Assign weight ratios to the feedback neural network and the Transformer neural network, and then deploy them online; the weight ratio of the feedback neural network is n1, the weight ratio of the Transformer neural network is n2, and n1+n2=1; during the initial neural network trial run, n1 takes the value of 1, and n2 takes the value of 0.
[0066] When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map.
[0067] The real-time molten pool feature map is compared with the standard molten pool feature map in S6 to obtain the molten pool feature deviation; the molten pool feature deviation is input into the feedback neural network to obtain the output welding parameter correction; the welding parameter correction is added to the welding parameters currently in use to obtain the first welding parameter P1; the real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2;
[0068] The first welding parameter P1 and the second welding parameter P2 are weighted and summed to obtain the third welding parameter P3 = n1×P1 + n2×P2; the welding process of the welding machine is intervened in real time based on the third welding parameter P3.
[0069] S8. Use the trial run samples to supplement the training of the neural network: After the neural network has been running for a period of time, according to the time correspondence, match the real-time molten pool feature map used in S7 with the third welding parameter P3 to form a third sample set; use the third sample set to supplement the training of the Transformer neural network.
[0070] S9. Adjust the weight ratio and run again: After the supplementary training is completed, adjust the weight ratio of the feedback neural network and the Transformer neural network, decrease the weight ratio n1 of the feedback neural network and increase the weight ratio n2 of the Transformer neural network; then repeat S7 to S9 multiple times until the weight ratio n1 of the feedback neural network decreases to 0 and the weight ratio n2 of the Transformer neural network increases to 1.
[0071] S10, Neural Network Officially Runs: Set the weight ratio n1 of the feedback neural network to 0, set the weight ratio n2 of the Transformer neural network to 1, and then deploy it online;
[0072] When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map.
[0073] The real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2; based on the second welding parameter P2, the welding process of the welding machine is intervened in real time.
[0074] During welding, the tungsten needle of the welding torch generates a high-temperature plasma gas stream of tens of thousands of degrees Celsius at the weld location. This plasma gas stream melts the workpiece to form a molten pool, which solidifies to form the weld. Therefore, during welding, the following boundaries define the characteristics of the molten pool: the gas-liquid interface (the boundary between the plasma gas stream and the molten liquid pool), the solid-liquid interface (the boundary between the tail of the molten liquid pool and the solidified weld), and the molten pool boundary lines (i.e., the edge boundaries on both sides of the molten pool). The changes in molten pool characteristics can be specifically manifested as: changes in the shape and displacement of the gas-liquid interface, changes in the shape and displacement of the solid-liquid interface, and changes in the width, length, and area of the molten pool resulting from changes in the molten pool boundary lines.
[0075] The main technical approach of this invention is to identify the above-mentioned molten pool features, further establish a mapping relationship between these molten pool features and their changes and welding parameters (welding speed, welding current, welding height), and finally utilize this mapping relationship to perform real-time automated intervention in the welding process. Traditional CNN convolutional neural networks require a large number of convolutional layers to extract features, which reduces the receptive field, increases computational load, and requires high hardware costs. Therefore, how to identify and extract all molten pool features from video images more quickly and accurately is both a key focus and a challenge.
[0076] like Figure 2 As shown, to solve the above problems, in S2 of this embodiment of the invention, visual inspection and analysis of the video image is performed by the following method to identify the molten pool features in the video image frame by frame in order to extract and form a molten pool feature map:
[0077] S201. Select ROI region: Identify the arc region in the video image and select the ROI region for image processing around the arc region;
[0078] S202, Filtering and Segmentation: Smooth the colors of the image in the ROI region, and then segment the image into several regions according to color similarity;
[0079] S203, Flood Fill: Fill the segmented areas with different and dissimilar colors;
[0080] S204, Edge Detection: Use convolution kernels to calculate the gradient of the image in the horizontal and vertical directions to detect the boundary lines of different color regions in S202, and output the set of all detected boundary lines as a melt pool feature map.
[0081] Specifically, before identifying and extracting the molten pool features, it is necessary to first select the region to be identified, i.e., the Region of Interest (ROI), within the entire camera frame. This simplifies the input image and optimizes subsequent image processing speed. At the start of welding, a bright spot appears in the video image captured by the CCD industrial camera; this bright spot is the arc region, and the molten pool region is directly below it. Based on this characteristic, in embodiment S201 of this invention, the ROI region is selected using the following method:
[0082] S2011. Convert the video image captured by the CCD industrial camera into a grayscale image, and then convert the grayscale image into a binary image using a set grayscale value as a threshold.
[0083] S2012. In real time, traverse each region of the binary image and determine the white blocks in the binary image with an area greater than a certain threshold as electric arc regions.
[0084] S2013. Calculate the centroid coordinates of the arc region, and generate a bounding box of ROI with the centroid coordinates of the arc region as the center. Use the region in the bounding box of ROI as the ROI region for subsequent processing.
[0085] Specifically, after converting the video image into a binary image, the binary image is pure black when no highlights appear; when highlights appear, white patches with the same shape as electric arcs appear against the black background. To prevent these white patches from being generated by noise, only white patches with an area greater than a certain threshold are considered to be electric arc regions. Further calculation of the centroid coordinates of the white patches (i.e., finding the centroid or average position of the white patches) allows for the generation of Regions of Interest (ROI) boxes around the centroid of the white patches.
[0086] As an improvement, in S2013, the continuous video images are divided into cycles of 3 seconds. When the ROI region of the video image in the current cycle is selected, the ROI box is generated with the centroid of the arc region in all video images in the previous cycle as the center. The ROI box is periodically and intermittently updated according to this rule to ensure the continuity and stability of the ROI region.
[0087] Because welding work conditions are generally harsh, and the video images are captured in strong light, the molten pool is affected by the electric arc, causing oscillations and resulting in noise such as ripples, bright spots, and smoke in the video images. To effectively filter this noise and extract stable and clear boundary images, this embodiment of the invention employs the following method for image filtering and segmentation in S202:
[0088] S2021. Establishing an iterative space: On the input image, with any point as the center, establish a spherical space with radius Sr in the color space, where the R, G, and B values of the color space are set to 0-255; establish a spherical space with radius Sp in the coordinate space, where the x and y values of the coordinate space are the length and width of the input image, respectively; the color space and coordinate space constitute a five-dimensional joint iterative space.
[0089] S2022. Obtain the vectors in the iteration space and recalculate the vectors after moving the sphere in the iteration space until convergence: In the constructed iteration space, calculate the mean of the color vectors from all points in the iteration space to the origin of the color space. Use the magnitude of this mean as the distance the origin of the color space is moved, and use the direction of this mean as the direction the origin of the color space is moved. Update the origin of the color space in this way. Then calculate the mean of the color vectors from all points in the iteration space to the updated origin of the color space, and update the origin of the color space again until the mean of the color vectors meets a certain threshold.
[0090] S2023, Region Segmentation: Using color similarity and spatial location similarity as the criteria, points with color similarity and spatial location similarity higher than a set threshold are merged into the same region, thereby segmenting the input image into several regions;
[0091] S2024, Region Merging: From the several regions segmented in S2023, find regions with fewer than a certain threshold of pixels and merge them into the adjacent regions with the highest color similarity and spatial similarity; thereby eliminating regions with too few pixels and finally completing filtering and segmentation.
[0092] To make the boundaries of the various regions segmented in S202 clearer, this embodiment of the invention further fills the segmented regions with different and dissimilar colors in S203. Specifically, this includes the following steps:
[0093] S2031. Perform a Gaussian filter on the image output in S2024 to further integrate the small and similar-colored parts of the image with the surrounding environment.
[0094] S2032. Read in the Gaussian-filtered image and ensure that the input image is not empty. Create a zero matrix slightly larger than the input image to mark the pixels that have been visited. At the same time, define a number of fixed colors to fill the different regions after segmentation.
[0095] S2033. Loop through each pixel in the image. If the current pixel has not been visited, perform a fill operation.
[0096] Furthermore, in S204 of this embodiment of the invention, edge detection and molten pool feature map are generated using the following method:
[0097] S2041, Definition and Two convolutional kernels simultaneously create two zero matrices of the same size as the image, used to store the gradient values of the image in the horizontal and vertical directions;
[0098] S2042. Iterate through the three RGB channels of the image in sequence, and perform a horizontal convolution operation on the elements in each channel using G(x) to obtain the gradient in the horizontal direction. Then perform a vertical convolution operation using G(y) to obtain the gradient in the vertical direction.
[0099] S2043. For each pixel in the image, calculate the gradient magnitude and gradient direction of each pixel based on the gradient in the horizontal direction and the gradient in the vertical direction obtained in S2032, so as to extract the boundary lines of each region in the image.
[0100] S2044. The boundary lines extracted in S2033 are processed using the non-maximum suppression method to eliminate fine boundaries.
[0101] S2045. Output the processed image, which retains only the boundary lines, as a molten pool feature map.
[0102] Through the above steps, the embodiments of the present invention can identify and extract molten pool features such as gas-liquid interface lines, solid-liquid interface lines, and molten pool boundary lines from video images captured by CCD industrial cameras in one go. The feature extraction process is fast, accurate, and comprehensive. The final molten pool feature map can fully reflect the feature changes during the welding process and can effectively meet the needs of subsequent model training and real-time intervention in the welding process.
[0103] It is important to note that the identification of changes in molten pool characteristics involves calculating the width, length, and area of the molten pool, requiring prior knowledge of the dimensional mapping between the camera image and the real physical space. Therefore, before capturing video images of the welding area using a CCD industrial camera, camera calibration is necessary to determine the dimensional mapping between pixels in the video image and the real physical space for subsequent calculations of molten pool dimensions.
[0104] Traditional checkerboard calibration methods are complex and cumbersome, making them less applicable in the welding process of this invention. This invention employs the following method for camera calibration: After fixing the camera, a fixed calibration identification frame is set in the camera's image, the pixel size and coordinates of which are known. Then, a standard sphere with a known cross-sectional area is placed in the image of the calibration identification frame, specifically on the welding torch fixture of the welding machine, ensuring that the distance from the standard sphere to the camera is approximately equal to the distance from the welding area to the camera. After capturing the image of the standard sphere, the ratio of the projected area of the standard sphere to the area of the calibration identification frame can be calculated. Through this ratio conversion, the size conversion ratio between one pixel in the calibration identification frame and the actual physical space can be determined. Subsequently, by counting pixels, the width, length, and area of the molten pool in the actual physical space are calculated.
[0105] The neural network model in this invention combines a feedback neural network and a Transformer neural network. Compared with other neural network models used in the prior art for automatic welding control, the Transformer neural network used in this invention can increase the receptive field of the model and better capture local information in the image, thereby enabling the model to obtain a deeper and more comprehensive mapping relationship and improving the accuracy of deep learning.
[0106] In the Transformer neural network, the input image is first segmented into multiple patches, and then all patches are mapped to feature vectors. Each patch passes through a linear projection layer (equivalent to a fully connected layer) to obtain a feature vector, and these feature vectors form the sequence of basic input units of the image. To incorporate positional information into the model, a positional encoding is added to each feature vector, so that each sequence of basic input units contains not only image information but also its positional information within the original image.
[0107] The feature vector sequence with positional encoding is input into the Transformer encoder. The Transformer encoder consists of multiple Transformer modules, each including a multi-head self-attention mechanism and a feedforward neural network. After passing through the Transformer encoder, an MLP (Multilayer Perceptron) is used as the classification head to map the output of each basic unit sequence to a predefined number of classes.
[0108] Next, the model is trained using the cross-entropy loss function. The loss function measures the difference between the model output and the true label. Stochastic gradient descent is used to minimize this loss, allowing the model to gradually adjust its parameters to improve prediction accuracy.
[0109] However, a drawback of Transformer neural networks is their heavy reliance on large datasets. In the initial stages of model deployment, a large amount of data is unavailable for training the network; therefore, prior experience is crucial to ensure stable model operation. Based on this, this invention, during the trial phase of the neural network, primarily uses the output of the feedback neural network to control welding parameters, supplemented by the output of the Transformer neural network. The two networks are combined and controlled according to their weights to regulate the final output. In the later stages of model deployment, after training with a large number of samples, the weights between the two networks are dynamically adjusted, gradually forming a control method where the Transformer neural network is primary and the feedback neural network is secondary, achieving more accurate welding results and a more intelligent and automated welding intervention process.
[0110] Specifically, in S7, the method of real-time intervention in the welding process of the welding machine based on the third welding parameter P3 is semi-automatic intervention, which specifically includes:
[0111] When a welder uses a welding machine for manual welding, the welding parameters used by the welder in real time are compared with the third welding parameter P3 output in real time. When the deviation between the two is greater than the set threshold, the welding parameters of the welding machine are forcibly set to the third welding parameter P3.
[0112] In S10, the method of real-time intervention of the welding process of the welding machine according to the second welding parameter P2 is fully automatic intervention, which specifically includes: the welding machine performs fully automatic welding using the real-time output second welding parameter P2 when no one is operating it.
[0113] As an improvement, in S9, after each adjustment of the weight ratio of the neural network and a re-run, the image of the welded part formed by the neural network intervention is compared with the image of the welded part formed by standard manual welding to evaluate the similarity between the two; if the similarity after adjusting the weight ratio is lower than the similarity before adjusting the weight ratio, the weight ratio of the neural network is restored to the value before adjustment and the re-run is performed.
[0114] This invention provides a real-time welding process control method based on visual inspection of the weld pool. It efficiently and comprehensively extracts weld pool features using visual inspection technology, and combines this with deep learning technology to obtain the mapping relationship between weld pool features, their changes, and welding parameters. Furthermore, through repeated trial runs and supplementary training, the neural network model is continuously improved, ultimately achieving automated intervention and control of the welding process. This invention promotes the automation of deep penetration welding technology, helps improve production efficiency and weld yield, and has broad application prospects.
[0115] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A method for real-time control of welding process based on visual inspection of the weld pool, characterized in that, Includes the following steps: S1. Filming the manual welding process: During the welding process, a CCD industrial camera captures video images of the welding area and records in real time the adjustments made by the welder to the welding parameters of the welding machine to form a welding parameter-time curve; the welding parameters include welding speed, welding current and welding height. S2. Extracting the molten pool feature map: Visual inspection and analysis are performed on the video images to identify the molten pool features in the video images frame by frame in order to extract and form a molten pool feature map; the molten pool features include the gas-liquid interface line, the solid-liquid interface line, and the molten pool boundary line; S3. Extract the changes in molten pool features: Compare the changes in molten pool features in the molten pool feature map in chronological order to obtain the changes in molten pool features; at the same time, based on the welding parameter-time curve, obtain the changes in welding parameters corresponding to the changes in molten pool features before and after the changes. The changes in molten pool characteristics include changes in the shape and displacement of the gas-liquid interface line, changes in the shape and displacement of the solid-liquid interface line, and changes in molten pool width, molten pool length, and molten pool area caused by changes in the molten pool boundary line; the changes in welding parameters include changes in welding speed, welding current, and welding height. S4. Create a sample set: Based on the time correspondence, match the changes in molten pool characteristics with the changes in welding parameters to form the first sample set; match the molten pool feature map with the welding parameters to form the second sample set; S5. Construct a neural network and perform pre-training: Construct a feedback neural network with the change in molten pool characteristics as input and the change in welding parameters as output. Use the first sample set to train the feedback neural network to obtain the mapping relationship between the change in molten pool characteristics and the change in welding parameters. A Transformer neural network is constructed, with the molten pool feature map as input and welding parameters as output. The Transformer neural network is trained using a second sample set to obtain the mapping relationship between the molten pool feature map and the welding parameters. S6. Obtain a reference benchmark: Capture a standard video image of a welder manually welding with a welding machine using a CCD industrial camera, and perform visual inspection and analysis on the standard video image using the same method as in S2. Identify the molten pool features in the standard video image frame by frame to extract and form a standard molten pool feature map, which serves as a reference benchmark during the trial run of the neural network. S7. Neural Network Trial Run: Assign weight ratios to the feedback neural network and the Transformer neural network, and then deploy them online; the weight ratio of the feedback neural network is n1, the weight ratio of the Transformer neural network is n2, and n1+n2=1; during the initial neural network trial run, n1 takes the value of 1, and n2 takes the value of 0. When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map. The real-time molten pool feature map is compared with the standard molten pool feature map in S6 to obtain the molten pool feature deviation; the molten pool feature deviation is input into the feedback neural network to obtain the output welding parameter correction; the welding parameter correction is added to the welding parameters currently in use to obtain the first welding parameter P1; the real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2; The first welding parameter P1 and the second welding parameter P2 are weighted and summed to obtain the third welding parameter P3 = n1×P1 + n2×P2; the welding process of the welding machine is intervened in real time based on the third welding parameter P3. S8. Use the trial run samples to supplement the training of the neural network: After the neural network has been running for a period of time, according to the time correspondence, match the real-time molten pool feature map used in S7 with the third welding parameter P3 to form a third sample set; use the third sample set to supplement the training of the Transformer neural network. S9. Adjust the weight ratio and run again: After the supplementary training is completed, adjust the weight ratio of the feedback neural network and the Transformer neural network, decrease the weight ratio n1 of the feedback neural network and increase the weight ratio n2 of the Transformer neural network; then repeat S7 to S9 multiple times until the weight ratio n1 of the feedback neural network decreases to 0 and the weight ratio n2 of the Transformer neural network increases to 1. S10, Neural Network Officially Runs: Set the weight ratio n1 of the feedback neural network to 0, set the weight ratio n2 of the Transformer neural network to 1, and then deploy it online; When the welding machine is welding, a real-time video image of the welding area is acquired by a CCD industrial camera, and the same method as in S2 is used to perform visual inspection and analysis on the real-time video image to identify the molten pool features in the real-time video image frame by frame in order to extract and form a real-time molten pool feature map. The real-time molten pool feature map is input into the Transformer neural network to obtain the output second welding parameter P2; based on the second welding parameter P2, the welding process of the welding machine is intervened in real time.
2. The real-time control method for welding process based on visual inspection of the weld pool according to claim 1, characterized in that, In step S2, the method for extracting the molten pool feature map includes the following steps: S201. Select ROI region: Identify the arc region in the video image and select the ROI region for image processing around the arc region; S202, Filtering and Segmentation: Smooth the colors of the image in the ROI region, and then segment the image into several regions according to color similarity; S203, Flood Fill: Fill the segmented areas with different and dissimilar colors; S204, Edge Detection: Use convolution kernels to calculate the gradient of the image in the horizontal and vertical directions to detect the boundary lines of different color regions in S202, and output the set of all detected boundary lines as a melt pool feature map.
3. The real-time control method for welding process based on visual inspection of the weld pool according to claim 2, characterized in that, S201 includes the following steps: S2011. Convert the video image captured by the CCD industrial camera into a grayscale image, and then convert the grayscale image into a binary image using a set grayscale value as a threshold. S2012. In real time, traverse each region of the binary image and determine the white blocks in the binary image with an area greater than a certain threshold as electric arc regions. S2013. Calculate the centroid coordinates of the arc region, and generate a bounding box of ROI with the centroid coordinates of the arc region as the center. Use the region in the bounding box of ROI as the ROI region for subsequent processing.
4. The real-time control method for welding process based on visual inspection of the weld pool according to claim 3, characterized in that, In S2013, continuous video images are divided into 3-second cycles. When the ROI region of the video image in the current cycle is selected, the ROI box is generated with the centroid of the arc region in all video images in the previous cycle as the center, and the ROI box is periodically and intermittently updated according to this rule.
5. The real-time control method for welding process based on visual inspection of the weld pool according to claim 2, characterized in that, S202 includes the following steps: S2021. Establishing an iterative space: On the input image, with any point as the center, establish a spherical space with radius Sr in the color space, where the R, G, and B values of the color space are set to 0-255; establish a spherical space with radius Sp in the coordinate space, where the x and y values of the coordinate space are the length and width of the input image, respectively; the color space and coordinate space constitute a five-dimensional joint iterative space. S2022. Obtain the vectors in the iteration space and recalculate the vectors after moving the sphere in the iteration space until convergence: In the constructed iteration space, calculate the mean of the color vectors from all points in the iteration space to the origin of the color space. Use the magnitude of this mean as the distance the origin of the color space is moved, and use the direction of this mean as the direction the origin of the color space is moved. Update the origin of the color space in this way. Then calculate the mean of the color vectors from all points in the iteration space to the updated origin of the color space, and update the origin of the color space again until the mean of the color vectors meets a certain threshold. S2023, Region Segmentation: Using color similarity and spatial location similarity as the criteria, points with color similarity and spatial location similarity higher than a set threshold are merged into the same region, thereby segmenting the input image into several regions; S2024, Region Merging: From the several regions segmented by S2023, find the regions with fewer than a certain threshold of pixels and merge them into the adjacent regions with the highest color similarity and spatial similarity.
6. The real-time control method for welding process based on visual inspection of the weld pool according to claim 5, characterized in that, S204 includes the following steps: S2041, Definition and Two convolutional kernels simultaneously create two zero matrices of the same size as the image, used to store the gradient values of the image in the horizontal and vertical directions; S2042. Iterate through the three RGB channels of the image in sequence, and perform a horizontal convolution operation on the elements in each channel using G(x) to obtain the gradient in the horizontal direction. Then perform a vertical convolution operation using G(y) to obtain the gradient in the vertical direction. S2043. For each pixel in the image, calculate the gradient magnitude and gradient direction of each pixel based on the gradient in the horizontal direction and the gradient in the vertical direction obtained in S2032, so as to extract the boundary lines of each region in the image. S2044. The boundary lines extracted in S2033 are processed using the non-maximum suppression method to eliminate fine boundaries. S2045. Output the processed image, which retains only the boundary lines, as a molten pool feature map.
7. The real-time control method for welding process based on visual inspection of the weld pool according to claim 2, characterized in that, Before using a CCD industrial camera to capture video images of the welded area, the following steps are also included: Camera calibration is used to determine the size mapping relationship between pixels in video images and real physical space, which is then used for subsequent calculations of molten pool size data.
8. The real-time control method for welding process based on visual inspection of the weld pool according to claim 1, characterized in that, In S7, the method of real-time intervention in the welding process of the welding machine based on the third welding parameter P3 is semi-automatic intervention, specifically including: When a welder uses a welding machine for manual welding, the welding parameters used by the welder in real time are compared with the third welding parameter P3 output in real time. When the deviation between the two is greater than the set threshold, the welding parameters of the welding machine are forcibly set to the third welding parameter P3.
9. The real-time control method for welding process based on visual inspection of the weld pool according to claim 8, characterized in that, In S10, the method of real-time intervention of the welding process of the welding machine according to the second welding parameter P2 is fully automatic intervention, which specifically includes: the welding machine performs fully automatic welding using the real-time output second welding parameter P2 when no one is operating it.
10. The real-time control method for welding process based on visual inspection of the weld pool according to claim 1, characterized in that, In S9, after each adjustment of the weight ratio of the neural network and a re-run, the image of the welded part formed by the neural network intervention is compared with the image of the welded part formed by standard manual welding to evaluate the similarity between the two. If the similarity after adjusting the weight ratio is lower than the similarity before adjusting the weight ratio, the weight ratio of the neural network is restored to the value before adjustment and the re-run is performed.
Citation Information
Patent Citations
Visual localization method based on inverse neural radiation field
CN116630420A
Vision-based working area boundary detection system and method, and machine equipment
WO2020107687A1
