An end-to-end welding process feature extraction and welding control method and system

Abstract

The application discloses an end-to-end welding process feature extraction and welding control method and system, and belongs to the technical field of welding. The feature extraction comprises the following steps: collecting a laser line image; using a first neural network model to infer and output a physical space mapping image; scanning and extracting a laser line center point position and scaling to a physical space to obtain a one-dimensional laser line data array; combining multiple frames of data to form three-dimensional point cloud data; and using a second neural network model to identify and output an upper groove feature line and a lower groove feature line. The application realizes end-to-end feature extraction through a neural network, finally outputs a feature line, does not need calibration, coordinate transformation and other participation in reasoning, and has the advantages of significantly improved reasoning efficiency, strong anti-interference ability, and applicability to welding seam height change, gentle and multi-layer multi-pass welding scenes.

Description

Technical Field

[0001] This invention relates to the field of welding technology, and in particular to an end-to-end welding process feature extraction and welding control method and system. Background Technology

[0002] Currently, in automated welding processes, line structured light modules are used to acquire the contour image of the weld bevel. To accurately and efficiently extract physical dimension information and key feature points from the raw contour lines output by the sensor, existing technical solutions typically employ a two-stage processing mode: first, the contour lines are extracted from pixel space using image processing algorithms; then, the contour lines are transformed into real physical space using calibration and mapping relationships; and finally, key feature points of the bevel are identified based on geometric calculation rules. However, under actual welding conditions, due to factors such as welding spatter, arc light interference, differences in surface reflection at the bevel, and morphological changes between multiple weld layers, the following drawbacks exist: (1) Traditional mapping from pixel space to physical space relies on manual calibration and coordinate transformation model. The preprocessing and filtering steps are time-consuming. Furthermore, when welding conditions (such as workpiece position and ambient light) fluctuate, the mapping accuracy decreases significantly, leading to an increase in the extraction error of the physical contour line.

[0003] (2) After extracting the physical contour line, existing technologies mostly use geometric algorithms based on slope, curvature, or template matching to detect feature points (such as inflection points) of the bevel. However, such algorithms are highly sensitive to the quality of the contour data. When the height change between weld beads is relatively gentle, the geometric feature differences in the area where the feature points are located are weak, and traditional algorithms have difficulty in accurately distinguishing the boundary lines of different weld beads, which can easily lead to missed detections or false detections. In addition, in multi-layer and multi-pass welding scenarios, the width, height, and number of layers of each weld are not the same, and the number of feature points on the bevel contour is not fixed. Traditional geometric detection methods cannot adaptively handle the changes in the number of bevel feature points in multi-layer and multi-pass welding scenarios, resulting in poor recognition stability. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of the prior art and provide an end-to-end welding process feature extraction and welding control method and system.

[0005] The objective of this invention is achieved through the following technical solution: a method for extracting features from an end-to-end welding process, comprising the following steps: Acquire laser line images of the weld bevel surface; The laser line image is inferred using a pre-trained first neural network model, and a physical space mapping image is output; the pixel space in the physical space mapping image has a fixed proportional correspondence with the physical space. The physical space mapping image is scanned, the center point position of each laser line in the physical space mapping image is extracted, and the position index of each laser line center point is scaled to the physical space according to the fixed proportional correspondence to obtain a one-dimensional laser line data array. The array of continuously acquired one-dimensional laser line data frames is combined according to the scanning interval to form three-dimensional point cloud data; The pre-trained second neural network model is used to identify the three-dimensional point cloud data and output the key feature points of the welding bevel, including the upper bevel feature line and the lower bevel feature line.

[0006] In one example, scanning the physical space mapping image and extracting the center point positions of each laser line in the physical space mapping image includes: The physical space mapping image is traversed column by column, and each column is scanned row by row. The pixels in each column are binarized using a preset probability threshold. Pixels that continuously exceed the preset probability threshold are stored in a sliding window. When a pixel is found that is below the preset probability threshold, the median or mean of all row coordinates in the current sliding window is calculated and used as the candidate laser line center point position for the current column. The sliding window is then cleared. The above process is repeated until the candidate laser line center point position corresponding to the first valid column is obtained, which is recorded as the initial seed point. After obtaining the initial seed point, stop the global scan. Using the row coordinates of the initial seed point as a reference, set the local search radius and scan the left and right adjacent columns within the local search radius. During the scanning of adjacent columns within the local search radius, a sliding window mechanism is used to calculate the median or mean and use it as the candidate laser line center point position of the column. The candidate laser line center point position calculated in the previous column is used as the center of the search window for the next column. The process is continued column by column until all columns are covered, and the complete center point positions of each laser line are extracted.

[0007] In one example, before recognizing the 3D point cloud data using a pre-trained second neural network model, the method further includes: The three-dimensional point cloud data is converted into a two-dimensional pseudo-color image, and the pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data. The two-dimensional pseudo-color image is input into the second neural network model so that the second neural network model can identify feature points in the two-dimensional pseudo-color image.

[0008] In one example, when the second neural network outputs the location of key feature points of the weld bevel, it also includes: The location of the output welding defect area includes at least one of the following: undercut defect area, porosity defect area, sidewall groove defect area, and hump defect area.

[0009] In one example, the location of the weld defect area is output, including: The second neural network model identifies defect regions in multiple consecutive frames of two-dimensional pseudo-color images and stores the laser line index number corresponding to the identified defect regions. During the welding robot's movement along the welding direction, a new frame of one-dimensional laser line data array enters the recognition window, and an old frame of one-dimensional laser line data array leaves the recognition window. The updated recognition window is then assembled into a new two-dimensional pseudo-color image and input into the second neural network model for defect area recognition. The recognition results of the new defect area and the corresponding laser line index number are stored. Repeat the above steps until the one-dimensional laser line data array within the recognition window is completely refreshed; Defect regions whose laser line index numbers overlap or are spaced within a preset threshold range in the recognition results of multiple frames are treated as the same defect region for data fusion. The spatial location of the defect region's boundary is then determined, and the spatial location of the defect region's boundary is output.

[0010] It should be further noted that the technical features corresponding to the above feature extraction methods can be combined or replaced to form new technical solutions.

[0011] The present invention also includes an end-to-end welding process feature extraction system, the system comprising: A laser line image acquisition unit is used to acquire laser line images of the weld bevel surface; The first inference unit uses a pre-trained first neural network model to infer the laser line image and outputs a physical space mapping image; the pixel space in the physical space mapping image has a fixed proportional correspondence with the physical space. The post-processing unit is used to scan the physical space mapping image, extract the position of the center point of each laser line in the physical space mapping image, and scale the position index of each laser line center point to the physical space according to the fixed ratio correspondence to obtain a one-dimensional laser line data array. The point cloud generation unit is used to combine multiple frames of continuously acquired one-dimensional laser line data arrays according to the scanning interval to form three-dimensional point cloud data; The second inference unit uses a pre-trained second neural network model to identify the three-dimensional point cloud data and output the key feature points of the welding bevel, including the upper bevel feature line and the lower bevel feature line.

[0012] In one example, the system further includes: An image conversion unit is used to convert three-dimensional point cloud data into a two-dimensional pseudo-color image; the pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data. The second inference unit uses a pre-trained second neural network model to identify the two-dimensional pseudo-color image and outputs the location of key feature points of the weld bevel and the location of the weld defect area. The location of the weld defect area includes at least one of the following: undercut defect area, porosity defect area, sidewall groove defect area, and hump defect area.

[0013] The present invention also includes a welding control method, implemented based on the end-to-end welding process feature extraction method formed by any or a combination of the above examples, further comprising, after outputting the location of the welding defect area: The location and type of the welding defect area are obtained; before the welding torch reaches the welding defect area, the corresponding feedforward control command is generated according to the defect type, which is used to adjust the welding parameters or welding torch posture when the welding torch reaches the welding defect area in order to correct the welding defect.

[0014] In one example, after the second neural network model outputs the locations of key feature points of the weld bevel, it also includes: Based on the spatial relationship between the line structured light module and the welding torch, determine the three-dimensional point cloud distribution window corresponding to the current welding position; The bevel depth is determined based on the height difference between the upper and lower bevel feature lines. The target filling height is determined based on the difference between the bevel depth and the preset reserved depth. The plane containing the target filling height is used as the target filling plane. The volume of the area to be filled is obtained by integrating the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window. Based on the volume of the area to be filled and the constraints of the welding process parameters in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, and the welding torch is controlled to weld the area to be filled according to the welding process parameters.

[0015] The present invention also includes a welding control system, which is implemented based on a feature extraction system, the control system comprising: The weld bead planning unit is used to calculate the number of weld beads in the current weld layer based on the horizontal spacing between the upper and lower bevel feature lines and the preset single-pass welding width; determine the bevel depth based on the height difference between the upper and lower bevel feature lines; determine the target filling depth based on the difference between the bevel depth and the preset reserved depth; and determine the number of weld layers based on the ratio of the target filling depth to the preset single-pass filling thickness; control the welding torch to weld layer by layer and bead by bead according to the number of weld layers, the number of weld beads, and the initial position of the welding wire corresponding to each weld bead; and / or, A defect correction unit is used to acquire the location and type of the welding defect area. Before the welding torch reaches the welding defect area, it generates corresponding feedforward control commands based on the defect type. These commands are used to adjust welding parameters or welding torch posture when the welding torch reaches the welding defect area to correct the welding defect; and / or, The welding planning unit is used to determine the bevel depth based on the height difference between the upper and lower bevel feature lines, and to determine the target filling height based on the difference between the bevel depth and the preset reserved depth. The plane containing the target filling height is used as the target filling plane. The volume of the area to be filled is obtained by integrating the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window. Based on the volume of the area to be filled and the constraint range of the welding process parameters in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, and the welding torch is controlled to weld the area to be filled according to the welding process parameters.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. The first neural network model performs online inference on the laser line image, directly outputting a physical space mapping image with a fixed ratio between the image pixel space and the physical space. The inference stage does not require hand-eye calibration or coordinate transformation, which significantly improves inference speed and reduces CPU computing power consumption. Based on this, the second neural network model outputs feature lines rather than isolated feature points. It can use the continuous distribution pattern of feature lines to eliminate outliers, thereby effectively resisting external interference such as welding spatter, arc light interference, and bevel surface reflection, ensuring the stability and reliability of feature point data and their positions. At the same time, compared with traditional geometric detection algorithms that directly perform point-by-point traversal, difference, or fitting numerical calculations on discrete coordinate points, this application uses the second neural network model to extract feature information under different receptive fields corresponding to different feature layers to achieve key feature point location identification. It is not sensitive to fluctuations in individual data outliers and can accurately output feature point positions even in welding scenarios with gradual changes in weld bead height and multi-layer, multi-pass welding, ensuring the robustness and stability of key feature point extraction of the bevel under complex welding conditions such as strong arc light and spatter.

[0017] 2. A sliding window is used to determine the candidate center point by judging the continuity of a preset probability threshold. Multi-pixel statistics within the window can suppress the influence of noise points on the center position. After obtaining the initial seed point, the search shifts from global scanning to local search. The left and right adjacent columns are scanned only within the narrow band corresponding to the local search radius, which greatly reduces the amount of traversal calculation and improves the calculation efficiency.

[0018] 3. The 3D point cloud data is converted into a 2D pseudo-color image, and the correspondence between depth values ​​and pixel colors is established. The depth information in the 3D space is compressed into the 2D image by color encoding, so that the 3D point cloud data can be input into the convolutional neural network for semantic segmentation processing in 2D image format, thereby ensuring the accuracy of the bevel feature point extraction.

[0019] 4. By performing a single identification on the 3D point cloud data through the second neural network, the locations of key feature points and welding defect areas can be obtained simultaneously, simplifying the data processing flow. Furthermore, the location of the feature points can provide spatial constraint information for the defect area, limiting the defect detection range to the area enclosed by the feature points, thereby reducing the false detection rate.

[0020] 5. By performing region fusion on the recognition results of multiple consecutive frames, the final boundary spatial location of the defect area can be obtained, which can effectively eliminate possible misidentification results in a single frame and improve the reliability of defect area detection.

[0021] 6. For different types of welding defects (porosity, humps, etc.) identified, before the welding torch reaches the defect location, corresponding feedforward control commands are generated according to the defect type. By pre-adjusting welding parameters or welding torch posture, targeted corrections are made for different defects. Active intervention is carried out at the moment the defect occurs, thereby effectively eliminating or reducing welding defects such as porosity and humps, and further improving welding quality.

[0022] 7. By performing volume integration on the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window, the volume of the current area to be filled is accurately calculated in real time. Based on the parameter constraints in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, realizing feedforward planning of welding parameters and ensuring the welding quality of the filling area. Attached Figure Description

[0023] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0024] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the definition of physical image space coordinates of a laser sensor according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a first neural network model architecture provided in an embodiment of the present invention; Figure 4This is a schematic diagram of the original laser line image provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the physical space mapping image corresponding to the original laser line image provided in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the conversion of a multi-frame one-dimensional laser line data array into 3D point cloud data according to an embodiment of the present invention. Figure 7 This is a schematic diagram of defect area annotation provided in an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the process of inputting a pseudo-color image into a convolutional neural network for edge extraction, according to an embodiment of the present invention. Figure 9 This is a schematic diagram of the undercut defect area provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of a porosity defect region provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of a sidewall groove defect area provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the hump defect region provided in an embodiment of the present invention; Figure 13 This is a schematic diagram of the distribution of three-dimensional point cloud within a window and the target filling plane provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of real-time weld bead planning provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of weld center tracking provided in an embodiment of the present invention.

[0025] In the diagram: 1-Bevel defect; 2-Surface pores; 3-Bevel edge bite; 4-Bevel misalignment; 5-Target filling plane. Detailed Implementation

[0026] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In the description of this invention, ordinal numbers (e.g., "first and second," etc.) are used to distinguish objects and are not limited to this order, nor should they be construed as indicating or implying relative importance. Furthermore, the technical features involved in the different embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0028] In one embodiment, an end-to-end welding process feature extraction system is provided. The system includes a laser line image acquisition unit, a first inference unit, a post-processing unit, a point cloud generation unit, and a second inference unit. The laser line image acquisition unit is connected to a welding machine control system. The welding control system is used to control the movement path of the welding torch, the welding speed, and the welding posture, etc.

[0029] The laser line image acquisition unit includes an image acquisition module and a line structured light module. The image acquisition module, such as a camera or video camera, is used to acquire image information during the welding process. Camera parameters are selected according to the needs of the welding scenario, such as resolution and frame rate, to ensure clear capture of dynamic changes in the welding area. The line structured light module includes a laser and a beam shaper connected in sequence. The laser generates multiple line laser beams, which are shaped by the beam shaper and then emitted onto the surface of the welding bevel, forming a contour line consistent with the workpiece cross-section. The beams are reflected off the workpiece surface and imaged by the image acquisition module. The line structured light module and the image acquisition module work together. After the light source in the line structured light module is turned on, the welding control system outputs a trigger signal to the image acquisition module to acquire the laser line image.

[0030] The first inference unit integrates a pre-trained first neural network model, which is used to process and analyze the laser line image. Specifically, the first neural network model can be a semantic segmentation network based on an encoder-decoder structure, such as U-Net, SegNet, or FPN. In this embodiment, it is a neural network model based on a U-Net encoder-decoder structure. This neural network model is used to extract and analyze the features of the laser line, directly outputting a mask image with a resolution of 1024x1024, i.e., a physical space mapping image. The pixel space in this physical space mapping image has a fixed proportional correspondence with the physical space, and the laser line shape in the physical space mapping image represents the cross-sectional contour of the welding bevel in physical space.

[0031] The post-processing unit is used to scan the laser line skeleton of a single pixel on the mask image and extract the center point position of the laser line to obtain the pixel position at the sub-pixel scale. The current sub-pixel position is multiplied by a fixed scaling factor to finally obtain the physical depth data corresponding to the image position, i.e., a one-dimensional laser line data array.

[0032] The point cloud generation unit assembles the continuous one-dimensional laser line data groups scanned by the front-end structured light module at fixed scanning intervals to form three-dimensional point cloud data. In this embodiment, 512 laser line data can be sampled at equal intervals along the walking direction by identifying the one-dimensional laser line data array to form 3D point cloud data characterizing the three-dimensional morphology of the weld bevel surface.

[0033] The second inference unit integrates a pre-trained second neural network model. This model performs inference and recognition on the 3D point cloud data, outputting the locations of key feature points of the welding bevel, including the upper and lower bevel feature lines. Each feature line is composed of continuously arranged feature points, and each discrete point on the feature line corresponds to a feature point at the bevel boundary. The second neural network model can employ a deep learning-based semantic segmentation network, such as a fully convolutional network or U-Net; in this embodiment, a convolutional neural network is used.

[0034] Preferably, the second inference unit performs inference and recognition on the 3D point cloud data, outputting areas including the bevel surface region, bevel side region, weld bead region, and areas related to welding defects such as undercut defects, porosity defects, sidewall grooves, and hump defects. In other words, the second neural network model simultaneously outputs key feature areas of the weld bevel or near feature lines between weld beads, effectively reducing false detections. Simultaneously, it alarms for detected defects and records the welding position, assisting in subsequent defect inspection and spatial location analysis.

[0035] In one embodiment, the system further includes an image conversion unit for converting three-dimensional point cloud data into a two-dimensional pseudo-color image, wherein the pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data. In this embodiment, a normalized pseudo-color coloring scheme is used to correspond the depth values ​​of the 3D point cloud data to the gradient colors in the two-dimensional pseudo-color image, forming a 512x1024 2D RGB image data (two-dimensional pseudo-color image). The second inference unit uses a pre-trained second neural network model to identify the 2D RGB image data and outputs the locations of key feature points of the weld bevel and the locations of weld defect areas.

[0036] The present invention also includes a welding control system, which can be a welding machine control system. The control system is implemented based on the end-to-end welding process feature extraction system described in any of the above embodiments. The control system includes a weld layer / weld bead planning unit and / or a walking tracking unit and / or a defect correction unit and / or a welding planning unit, preferably including a weld layer / weld bead planning unit, a walking tracking unit, a defect correction unit, and a welding planning unit.

[0037] The weld bead planning unit is used to calculate the number of weld beads in the current weld layer based on the horizontal spacing between the upper and lower bevel feature lines (the current total width of the weld bead) and the preset single-pass welding width; determine the bevel depth based on the height difference between the upper and lower bevel feature lines; determine the target filling depth based on the difference between the bevel depth and the preset reserved depth; and determine the number of welding layers based on the ratio of the target filling depth to the preset single-pass filling thickness. The current total width of the weld bead in the bevel width direction is divided equally according to the number of weld beads, and the center point of each division is the initial position of the welding wire corresponding to each weld bead. The welding torch is controlled to weld layer by layer and weld bead by weld bead according to the number of welding layers, the number of weld beads, and the initial position of the welding wire corresponding to each weld bead.

[0038] The walking tracking unit determines the center position of the bevel based on the feature lines of the upper and lower bevels; it calculates the deviation between the center position and the preset reference position; and it generates a welding torch position adjustment command based on the deviation to control the welding torch to move along the weld center. Specifically, the difference between the center point of the upper bevel feature point detected by the walking tracking unit and the point x=0 of the laser in the x direction is used as the input signal of the PID controller, and the output signal is the turning angular velocity of the motion device, thereby realizing the smooth operation of the motion device along the weld center.

[0039] The defect correction unit is used to obtain the location and defect type (welding defect label) of each welding defect area. Before the welding torch reaches the welding defect area, it generates corresponding feedforward control instructions according to the defect type. These instructions are used to adjust the welding parameters or welding torch posture when the welding torch reaches the welding defect area in order to correct the welding defects. Optionally, for areas with porosity defects, the welding torch is controlled to perform a porosity stirring operation when it reaches the defect location, while the dry extension length is reduced to increase the stirring effect of the molten pool and promote the escape of bubbles, thereby achieving in-situ elimination of the defect; for areas with hump defects, welding process parameters such as welding speed, wire feed speed, and heat input are adjusted before the welding torch reaches the defect location to reduce the molten pool temperature and decrease the fluidity of the molten pool, thereby inhibiting the formation of the hump; for areas with undercut defects, the welding torch oscillation amplitude is increased or the dwell time on both sides is extended when the welding torch reaches the defect location, so that the molten metal fully fills the concave area of ​​the bevel sidewall; for areas with sidewall groove defects, the welding torch angle is adjusted when the welding torch reaches the defect location so that the welding wire points towards the sidewall where the groove is located, increasing the heat input and molten metal filling amount of the sidewall, ensuring that the groove is fully fused and filled.

[0040] The welding planning unit is used to determine the bevel depth based on the height difference between the upper and lower bevel feature lines. Based on the difference between the bevel depth and the preset reserved depth, the target filling height is determined, and the plane containing the target filling height is used as the target filling plane. The volume of the area to be filled is obtained by integrating the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window. Based on the volume of the area to be filled and combined with the constraint range of the welding process parameters in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, and the welding torch is controlled to weld the area to be filled according to the welding process parameters.

[0041] Optionally, the first inference unit, post-processing unit, point cloud generation unit, second inference unit, and image conversion unit in the feature extraction system, as well as the weld layer and weld bead planning unit, defect correction unit, and welding planning unit in the welding control system, can be integrated on a single hardware processor or distributed across multiple hardware processors. Preferably, all of the above units are integrated into the processor of the welding machine control system.

[0042] In one embodiment, such as Figure 1 As shown, the present invention also includes an end-to-end welding process feature extraction method, implemented based on the above-mentioned feature extraction system, the method comprising the following steps: S1: Acquire laser line images of the weld bevel surface.

[0043] In step S1, the feature extraction system is started, and the image acquisition module, line structured light module, first neural network model, post-processing unit, point cloud generation unit, and second inference unit are initialized. Camera parameters, such as exposure time and gain, are set; the line structured light module voltage is tested to ensure that each system component functions normally; and image processing and analysis algorithms are loaded. Further, the line structured light module is controlled to start projecting line laser light and outputs a trigger signal to the image acquisition module to acquire the laser line image under the current bevel shape.

[0044] S2: The pre-trained first neural network model infers from the laser line image and outputs a physical space mapping image; the pixel space in the physical space mapping image has a fixed proportional correspondence with the physical space.

[0045] In step S2, laser line images are acquired, and the physical space coordinate mapping images of the laser line images at each time step are calculated using calibration parameters, thus pairing the laser line images with their corresponding physical space coordinate mapping images. Using the paired laser line images as input and the corresponding physical space coordinate mapping images as supervision labels, the difference between the output of the first neural network model and the supervision labels is calculated as the loss function. The loss gradient is backpropagated layer by layer using the backpropagation algorithm to update the parameters of the first neural network model. This process is repeated until the physical space coordinate mapping image output by the first neural network model converges to match the label, thereby achieving offline pre-training of the first neural network model.

[0046] The pre-trained first neural network model performs end-to-end inference on the acquired raw laser line image, outputting a physical space mapping image corresponding to the laser line image. In this embodiment, each pixel position in the output physical space mapping image has a fixed proportional relationship with the real physical space coordinates (fixed proportional correspondence), so there is no need to rely on calibration parameters and coordinate transformation matrices in traditional methods for point-by-point conversion, thereby greatly reducing the efficiency of online inference and meeting the real-time requirements of welding.

[0047] S3: Scan the physical space mapping image to complete the data scan of the entire laser line in the X and Y directions. Extract the center point position of each laser line in the physical space mapping image. Scale the position index of each laser line center point to the physical space according to a fixed proportional correspondence to obtain a one-dimensional laser line data array, i.e., physical depth data.

[0048] In step S3, the physical space mapping image is scanned column by column. The index position of the laser line's center point is found in each column. Once all columns are scanned, the complete center line position of the entire laser line is obtained. The center point position of the laser line obtained from the scan is represented by a pixel index. Multiplying the pixel index by a fixed scaling factor converts the pixel-scale position index into actual coordinate values ​​at the physical scale, ultimately forming a one-dimensional depth data array representing the current bevel cross-sectional profile. Each data point in this one-dimensional depth data array represents the actual physical depth of the bevel at that location.

[0049] S4: Combine the continuously acquired multi-frame one-dimensional laser line data arrays according to the scanning interval to form three-dimensional point cloud data.

[0050] In step S4, the array of multiple frames of one-dimensional laser line data collected at equal intervals along the welding walking direction is arranged sequentially according to the scanning order. Each frame of data corresponds to the two-dimensional cross-sectional profile of the groove at the current position. After splicing the data of each frame along the walking direction, a three-dimensional point cloud data composed of several cross-sectional profile lines is formed, which is used to represent the surface morphology of the welding groove in three-dimensional space, including the geometric information of the bottom of the groove, the sidewalls and the weld bead.

[0051] S5: Use a pre-trained second neural network model to identify the three-dimensional point cloud data and output the key feature points of the welding bevel, including the upper bevel feature line and the lower bevel feature line.

[0052] In step S5, paired 3D point cloud data is used as input, and corresponding region-annotated images are used as supervision labels. The difference between the output of the second neural network model and the supervision labels is calculated as the loss function. The loss gradient is backpropagated layer by layer using the backpropagation algorithm to update the parameters of the second neural network model. This process is repeated until the region classification and recognition result output by the second neural network model converges to match the label, thereby achieving offline pre-training of the second neural network model. The region-annotated images are used to mark the pixel regions corresponding to the bevel surface, bevel side, weld bead, and various welding defects in the image.

[0053] Furthermore, a second neural network model, which has been pre-trained offline, is used to perform pixel-level classification of the 3D point cloud, identifying different regions such as the bevel surface region, the bevel side region, and the weld bead region. By identifying the boundary lines between these different regions, the upper bevel feature line between the bevel surface and the bevel side, and the lower bevel feature line between the bevel side and the weld bead are extracted.

[0054] In this embodiment, the second neural network model outputs feature lines rather than isolated feature points. It can use the continuous distribution pattern of feature lines to eliminate abnormal points, thereby effectively resisting external interference such as welding spatter, arc light interference, and surface reflection of the bevel, ensuring the stability and reliability of feature point data and its position. At the same time, compared with traditional geometric detection algorithms that directly perform point-by-point traversal, difference, or fitting numerical calculations on discrete coordinate points, this application extracts local pixel information through the second neural network model to identify the position of key feature points. It is not sensitive to fluctuations in individual data points, and can accurately output the position of feature points even in welding scenarios with gradual changes in weld height and multi-layer, multi-pass welding, ensuring the robustness and stability of key feature point extraction of the bevel under complex welding conditions such as strong arc light and spatter.

[0055] In one embodiment, based on the above-described end-to-end welding process feature extraction method, the present invention also provides a corresponding experimental environment setup and end-to-end dataset creation method.

[0056] First, hand-eye calibration is completed according to industry standards, and the spatial pose transformation matrix from the line structured light module (line structured light sensor) to the end of the welding torch is calculated.

[0057] Specifically, the calibration of the line structured light module adopts the laser plane calibration method. By determining the standard inflection point of the laser plane on a standard workpiece, and fitting the laser line plane through multiple physical space points on the workpiece, the laser line space plane equation is obtained. The specific implementation method is as follows: (1) Extract five sets of effective three-dimensional feature points of the laser line on the bevel surface. , The laser detection reference plane is constructed by fitting using the least squares method. The simplified model of the plane is as follows:

[0058] In the formula, , , All are laser detection plane fitting coefficients; For the first The three-dimensional spatial coordinates of the laser feature points.

[0059] (2) Least squares optimal solution: The least squares objective function is constructed based on minimizing the sum of squared residuals of the five feature points relative to the fitting plane. :

[0060] By applying the principle of extreme value solving, the optimal fitting coefficients are obtained by taking the partial derivative of the objective function. A patented universal matrix solution method is then used for rapid solution, and a sample matrix is ​​constructed. With observation matrix : , ,

[0061]

[0062] in, It is a coefficient matrix; Let be the transpose of the matrix. The unique optimal plane fitting coefficients can be directly obtained using the above formula, constructing a high-precision laser detection reference plane and providing a benchmark for subsequent weld deviation detection.

[0063] Simultaneously, the intrinsic and extrinsic parameters of the camera lens are calibrated at a fixed sensor position, ultimately establishing the correspondence between points on the laser line plane and CMOS pixels. This process completes the standard label establishment workflow from the input laser line image to the physical space mapping image; a large amount of bevel data of different depths and shapes is collected using the calibrated line structured light sensor (raw laser line data is acquired via camera, and the physical space mapping image is obtained through calibration calculations).

[0064]

[0065] in, This is the scaling factor; , These are the pixel coordinates of the feature points on the camera; Here is the camera intrinsic parameter matrix, and here are the fixed parameters obtained from calibration. The extrinsic parameter matrix for camera rotation; The extrinsic parameter matrix for camera translation; , , These are the world physical coordinates of feature points on the laser plane. The physical image space coordinates of the laser sensor are defined as follows: Figure 2 As shown, Figure 2 middle This refers to the physical spatial location of the laser beam's emission point. The origin of the image coordinate system; , where is the pixel coordinate of the laser line point in the laser line image; PhyPos represents the position coordinate of the laser line point on the laser plane in actual physical space. Figure 2 The horizontal direction is marked with a double arrow indicating 1024 pixels ↔ 102.4 mm, and the vertical direction is marked with a double arrow indicating 512 pixels ↔ 51.2 mm. Both arrows are labeled "img2phy," indicating the transformation relationship between image pixel coordinates and physical space coordinates. Specifically, for the laser line point PhyPos, the calculation expression is: ,in express arrive The fixed offset value, determined by the physical installation position, defines the physical height from the starting point of the sensor's measurement range to the origin of the laser emission, and can be used to calculate the absolute spatial positional relationship. Since typical line structured light sensors have a defined measurement range—for example, starting 100mm from the origin of the laser emission and ending at 151.2mm—and measuring a width between -51.2mm and 51.2mm, this embodiment constructs a 1024x512 pixel image and plots the acquired physical space point data on it. Before plotting, the X and Y coordinates of the physical space are magnified by a factor of 10. At this point, the actual physical space accuracy corresponding to a single pixel is 0.1mm, which fully meets the detection requirements of laser lines in the welding field. If higher accuracy is required, the resolution of the inference output image can be increased.

[0066] Finally, the dataset obtained through the above processing consists of paired original laser line images and physical space mapping images. This dataset is then used to train the first neural network model offline, learning the mapping relationship between a large number of paired original laser line images and physical space mapping images (supervisory labels).

[0067] Furthermore, in actual camera imaging systems, the optical system undergoes a nonlinear distortion process, including radial distortion and tangential distortion. The formula for calculating radial distortion is:

[0068]

[0069] The expression for calculating tangential distortion is:

[0070]

[0071] in, , , These are the first-order, second-order, and third-order radial distortion coefficients, respectively. , Ideal coordinates for the physical image; , Radial correction coordinates; , Tangential correction coordinates; Radial distance; , These are the first-order tangential distortion coefficient and the second-order tangential distortion coefficient, respectively. Radial distortion is a high-order binomial distortion produced by the lens, including barrel distortion and pincushion distortion. This distortion is nonlinear, and in neural network models, nonlinear activation layers such as ReLU can generally achieve the polynomial calculation process.

[0072] Generally, the lenses used in structured light systems are low-distortion lenses. In high-precision sensing applications, the first-order term coefficient of lens distortion is... , Distortion correction has the greatest impact on accuracy. To improve computational speed, higher-order terms, such as second-order terms, are often ignored during distortion correction, leading to errors in the final calculation results. This application uses a complete distortion model, including all higher-order terms, to calculate physical coordinates when creating training labels. This allows the neural network to learn and fully fit the mapping relationship from the original image to the ideal physical space. As a result, high-precision distortion correction can be completed with a single forward propagation during the inference phase, significantly improving physical coordinate accuracy while maintaining a high output frame rate.

[0073] In one embodiment, the first neural network model employs a neural network model based on the U-Net encoding / decoding structure to utilize its deep nonlinear feature extraction capabilities to fit the aforementioned distortion mapping. For example... Figure 3As shown, the first neural network model includes an encoder and a decoder, with a bottleneck layer between them. The encoder includes at least four levels of feature units: a first-level feature unit, a second-level feature unit, a third-level feature unit, and a fourth-level feature unit. The first-level feature unit includes a first convolutional + ReLU activation layer and a first downsampling convolutional layer. The second-level feature unit includes a second convolutional + ReLU activation layer, a first dilated convolutional layer, and a second downsampling convolutional layer. The third-level feature unit includes a third convolutional + ReLU activation layer, a second dilated convolutional layer, and a third downsampling convolutional layer. The fourth-level feature unit includes a fourth convolutional + ReLU activation layer, a third dilated convolutional layer, a fifth convolutional + ReLU activation layer, and a first max-pooling layer. These four feature units perform downconvolutional sampling on the original 224×224 resolution image, successively obtaining multi-scale feature maps with resolutions of 112×112, 56×56, 28×28, and 14×14. Optionally, each level of the encoder's feature units can be configured with an attention module (ECA) for weighted filtering of the feature channels at that layer. Furthermore, the preceding stages of the bottleneck layer include a sixth convolutional + ReLU activation layer, a fourth dilated convolutional layer, and a seventh convolutional + ReLU activation layer. Preferably, the bottleneck layer also includes an intermediate stage, which may include multiple convolutional + ReLU activation layers and dilated convolutional layers. The final stage of the bottleneck layer includes a first attention module (ECA), an eighth convolutional + ReLU activation layer, and a first upsampling convolutional layer. The bottleneck layer is used to achieve global semantic feature fusion and maintain the feature map resolution at 14×14. The decoder comprises four levels: Level 1, Level 2, Level 3, and Level 4. The Level 1 decoder includes a second attention module (ECA), a ninth convolutional layer with ReLU activation, a fifth dilated convolutional layer, and a second upsampling convolutional layer. The Level 2 decoder includes a tenth convolutional layer with ReLU activation, a sixth dilated convolutional layer, and a third upsampling convolutional layer. The Level 4 decoder includes an eleventh convolutional layer with ReLU activation, a seventh dilated convolutional layer, and a fourth upsampling convolutional layer. The Level 4 decoder also includes a fifth upsampling convolutional layer. The Level 1, Level 2, and Level 3 decoders sequentially perform upsampling convolutions on the feature maps, outputting feature maps with resolutions of 28×28, 56×56, and 112×112. Finally, the fifth upsampling convolutional layer restores the image to a 224×224 segmented output image. It should be noted that in this application, the feature units at each level of the encoder are skipped to the decoding units of the same resolution in the decoder, enabling the splicing and fusion of shallow detail features and deep semantic features, compensating for the loss of detail information during the upsampling process. At this point, the first neural network model performs inference on the laser line image, including: (1) Using an encoder to perform such Figure 4The laser line image shown is downsampled to extract deep data features of the laser line image, and the high-resolution shallow features output by each layer of the encoder are spliced ​​with the upsampled deep features of the corresponding layer of the decoder through skip connections. (2) The laser line data features after downsampling are upsampled using a decoder to restore the data features to their original resolution, so as to output a physical space mapping image, such as... Figure 5 As shown. The size of the output physical space mapping image depends entirely on the parameters and number of upsampling layers in the encoder. Therefore, different physical space accuracy requirements can be met by adjusting this parameter.

[0074] In this embodiment, downsampling can effectively filter out interference information such as arc light and spatter in the original image, while upsampling combined with skip connections can retain the fine spatial position information of the laser line. This allows the first neural network model to automatically complete image preprocessing and filtering functions while realizing end-to-end physical space mapping, without having to repeatedly adjust the preprocessing parameters for different working conditions, thus improving the adaptability of welding conditions.

[0075] In one embodiment, scanning a physical space mapping image and extracting the center point positions of each laser line in the physical space mapping image includes: (1) Traverse the physical space mapping image by column and scan each column row by row; use a preset probability threshold (e.g., 0.5) to binarize the pixels in each column and store the pixels that continuously exceed the preset probability threshold in the sliding window; when a pixel appears that is lower than the preset probability threshold, calculate the median or mean of all row coordinates in the current sliding window and use it as the candidate laser line center point position for that column and clear the sliding window; repeat the above process until the candidate laser line center point position corresponding to the first valid column is obtained and recorded as the initial seed point. (2) After obtaining the initial seed point, stop the global scan; based on the row coordinates of the initial seed point, set a local search radius (such as ±N pixels) much smaller than the image height, and scan the left and right adjacent columns within the narrow range of the local search radius.

[0076] (3) During the local scanning process, the sliding window mechanism is still used to calculate the median or mean, and it is used as the candidate laser line center point position of the column to suppress noise points. Taking advantage of the spatial continuity of the laser line, the candidate laser line center point position calculated in the previous column is used as the center of the search window of the next column, and the search is pushed to both sides column by column until all columns are covered, and the complete center point position of each laser line (laser line center coordinate sequence) is output.

[0077] Preferably, the obtained laser line center coordinate sequence data is uniformly scaled by a factor of 10 to obtain one-dimensional coordinate data, i.e., physical space coordinate data with an accuracy of 0.1 mm. The row and column indices of each laser line center point are multiplied by a preset physical space scaling factor to obtain the X-direction and Y-direction coordinate values ​​of each laser line center point in physical space, forming a sequence as follows: Figure 6 The one-dimensional laser line data array.

[0078] In one embodiment, the distance between the laser line pre-scan and the welding wire tip is analyzed through mechanical structure analysis. This distance represents the window range of the pre-scan laser line and can be used for edge feature extraction, defect detection, and parameter planning. This distance is used to retain N laser line data as 3D point cloud data during scanning.

[0079] In one embodiment, before recognizing the 3D point cloud data using a pre-trained second neural network model, the method further includes: (1) Convert the three-dimensional point cloud data into a two-dimensional pseudo-color image. The pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data.

[0080] Specifically, by measuring the depth using a structured light sensor, a pseudo-color mapping table between depth values ​​and colors is established. Based on the depth values ​​of each point in the 3D point cloud data, the corresponding color is found from the pseudo-color mapping table to generate a 2D pseudo-color image. The 3D point cloud data is then represented by the 2D color image.

[0081] More specifically, since the detection range of the laser line is 0–51.2 mm, a gradient mapping relationship from red to purple can be established to map the determined depth data to a single color. Only 512 color types are actually needed, with a depth difference of 0.1 mm between the two colors, consistent with the laser line detection accuracy. To reduce the amount of inference data and parameters in the second neural network model, grayscale images can be used to encode the depth 3D point cloud data.

[0082] Preferably, after completing the above image conversion process, such as Figure 7 As shown, annotation software tools are used to annotate the areas to be identified, including the bevel surface, side surface, bevel defect 1, surface pores 2, bevel undercut 3, and bevel misalignment 4. Polygonal region annotation methods are used to create labels for different target areas. The images of pore areas are often presented as circles with a darker center, corresponding to deep pits. Furthermore, the original 2D pseudo-color images and the region label images are fed into a second neural network model for parameter training and optimization, ultimately resulting in a network model capable of accurately identifying defects and regions.

[0083] (2) Input the two-dimensional pseudo-color image into the second neural network model so that the second neural network model can identify the feature points of the two-dimensional pseudo-color image.

[0084] Specifically, in this embodiment, the second neural network model is a Convolutional Neural Network (CNN), including an input layer, convolutional layers, activation layers, pooling layers, fully connected layers, and an output layer. The convolutional layers, activation layers, and pooling layers are stacked alternately to form a feature extraction module, which extracts feature information step-by-step from the input two-dimensional pseudo-color image. This information is then non-linearly combined and mapped by the fully connected layers, and finally, the output layer provides the classification result. This embodiment utilizes a CNN to scan the entire two-dimensional pseudo-color image and identify feature boundaries such as... Figure 8 As shown, this method locates the bevel surface area, bevel side area, and weld bead area, and extracts the upper and lower bevel lines based on the intersection lines between different areas. The intersection line between the bevel surface and the bevel side is the upper bevel line, and the intersection line between the bevel side and the weld bead is the lower bevel line. Simultaneously, welding defect areas are extracted by scanning semantic segmentation images, including... Figure 9 The area of ​​the undercut defect shown is as follows: Figure 10 The porosity defect area shown is as follows: Figure 11 The sidewall groove defect area shown and as Figure 12 The hump defect area is shown.

[0085] Preferably, the training method of the second neural network model is as follows: the paired two-dimensional pseudo-color image is used as input and the corresponding region labeled image is used as supervision label. The difference between the output of the second neural network model and the supervision label is calculated as the loss function. The loss gradient is backpropagated layer by layer through the backpropagation algorithm and the parameters of the second neural network model are updated. This process is repeated until the region classification and recognition result output by the second neural network model converges to be consistent with the label, thereby realizing the offline pre-training of the second neural network model.

[0086] In one embodiment, the feature data such as defects and bevel lines output by the second neural network model are extracted in the form of a window. During the scanning process, the actual defect area will appear multiple times in consecutive frames, providing multiple sets of data for accurate defect detection. That is, after the defect area is extracted in the current frame, the area continues to be extracted in subsequent consecutive frames until the defect area leaves the scanning window. Since the defect is located in different frames (which can be reconstructed by continuous laser line frame labels), and the pseudo-color image of the area entering the new frame is different, completely different input image data are provided for the neural network to identify defects. By fusing the recognition results of multiple frames, the accuracy and reliability of area recognition are effectively improved. The specific implementation method is as follows: (1) The second neural network model performs defect region identification on multiple consecutive frames of two-dimensional pseudo-color images and stores the laser line index number corresponding to the identified defect region.

[0087] Specifically, after scanning the two laser lines of the window data, a pseudo-color image is constructed for region recognition processing; the index number of the laser line of the identified feature region is stored, for example, the interval from point 513 of the 10th laser line to point 620 of the 15th laser line where a porosity defect is detected.

[0088] (2) During the welding robot’s movement along the welding direction, a new frame of one-dimensional laser line data array enters the recognition window and an old frame of one-dimensional laser line data array leaves the recognition window. The updated recognition window is then assembled into a new two-dimensional pseudo-color image and input into the second neural network model for defect area recognition. The recognition results of the new defect area and the corresponding laser line index number are stored. For example, at this time, it can be found that there is a porosity defect in the interval from the 516th point of the 11th laser line to the 618th point of the 15th laser line.

[0089] (3) Repeat steps (1)-(2) above until the one-dimensional laser line data array in the recognition window is completely refreshed.

[0090] (4) Defect regions that are close on the laser line index in the recognition results of multiple frames before and after are treated as the same defect region for data fusion. The discrete data clustering method is used to locate the spatial position of the defect region boundary, and finally the spatial position of the defect region boundary is output.

[0091] Optionally, a discrete data clustering method can be used to spatially locate the boundary of the defect area, including the following steps: Discrete data clustering methods locate the spatial position of defect regions. In this case, all boundary points belonging to the same defect region in multi-frame recognition results are used as a dataset (DataSet). Clustering algorithms are then used to analyze the spatial distribution of these boundary points to determine the precise location of the defect boundary in three-dimensional space. This includes the following steps: (1) Select from the DataSet 1 point is used as the initial cluster point set PointDataSet: ; (2) Calculate the Euclidean distance from all data points in the DataSet to each cluster point in the PointDataSet; (3) Compare the Euclidean distances from each data point to each cluster point, and find the cluster point with the shortest distance. This data point is classified as the first kind; (4) Based on the current classification results, we obtain For each category of point set, calculate the centroid (baroid) of each category of point set. (5) Compare the centroid position obtained in this calculation with the centroid position of the previous iteration (if it is the first iteration, compare it with the initial cluster point position); if the error between the two centroid positions is less than the preset threshold, the clustering is determined to be converged, the centroid position is output as the precise spatial position of the defect boundary in three-dimensional space, and the clustering ends; if the error between the two centroid positions is greater than or equal to the preset threshold, the PointDataSet is updated to the current centroid position, and the process returns to step (2) to continue the iteration.

[0092] The points on the upper and lower slope lines are actually the feature points that need to be extracted from the laser line. For feature point acquisition from a single image, it is easily affected by the environment, acquisition, and algorithm. This application extracts a feature line, which can make full use of the line distribution to remove abnormal points. At the same time, the laser line data extracted based on the end-to-end neural network model can effectively resist external interference such as arc light, ensuring that the feature point data and its position are stable and reliable.

[0093] The present invention also includes a welding control method, implemented based on an end-to-end welding process feature extraction method formed by any or a combination of the above embodiments, and further comprising, after outputting the location of the welding defect area: (1) Obtain the location and type of welding defect area.

[0094] Specifically, in the classification results of welding defect regions output by the second neural network model through semantic segmentation, each defect region corresponds to a defect type label (such as porosity defect, hump defect, undercut defect, sidewall groove defect, etc.). At the same time, the position coordinates of the region in three-dimensional space are determined. Therefore, the spatial location of each welding defect region and its corresponding defect type can be obtained through the second neural network model.

[0095] (2) Before the welding torch reaches the welding defect area, a corresponding feedforward control command is generated according to the defect type. This command is used to adjust the welding parameters or the welding torch posture when the welding torch reaches the welding defect area in order to correct the welding defect.

[0096] Specifically, for areas with porosity defects, the control system controls the welding torch to perform a porosity stirring operation when it reaches the defect location, while simultaneously reducing the dry extension length to increase the stirring effect of the molten pool and promote the escape of bubbles, thereby achieving in-situ elimination of the defect. For areas with hump defects, welding process parameters such as welding speed, wire feed speed, and heat input are adjusted before the welding torch reaches the defect location to reduce the molten pool temperature and decrease the fluidity of the molten pool, thereby suppressing the formation of hump. For areas with undercut defects, the welding torch oscillation amplitude is increased or the dwell time on both sides is extended when the welding torch reaches the defect location, so that the molten metal fully fills the concave area of ​​the bevel sidewall. For areas with sidewall groove defects, the welding torch angle is adjusted when the welding torch reaches the defect location so that the welding wire points towards the sidewall where the groove is located, increasing the heat input and molten metal filling amount of the sidewall, ensuring that the groove is fully fused and filled.

[0097] In this embodiment, based on the defect detection results, a customized welding scheme for the defect area is achieved by controlling the welding posture and welding parameters of the welding torch. The defect is repaired by adjusting the welding parameters through feedforward, so as to achieve the final goal of welding a perfect bevel.

[0098] In one embodiment, after calculating the number of weld passes in the current weld layer, the method further includes: (1) Determine the three-dimensional point cloud distribution window corresponding to the current welding position based on the spatial position relationship between the line structured light sensor and the welding torch.

[0099] Specifically, based on the fixed front-mounted installation position of the line structured light sensor in front of the welding torch and its known spatial pose relationship with the welding torch, the welding torch position corresponding to the bevel contour point cloud data currently scanned by the line structured light sensor is determined, and the range of point cloud data at this position is the three-dimensional point cloud distribution window.

[0100] (2) Calculate the bevel depth (currently planned welding remaining depth) based on the height difference between the upper bevel feature line and the lower bevel feature line. Determine the target filling height (target welding remaining depth) based on the difference between the bevel depth and the preset reserved depth. Use the plane where the target filling height is located as the target filling plane.

[0101] During the welding process, it is necessary to ensure that after the final layer of filling, a depth feature area is left for laser line scanning and identification. This depth is defined as the target welding remaining depth data, which is used for feature scanning and tracking during the subsequent capping process (too shallow a depth will not be able to accurately detect boundary feature points). In this embodiment, the target welding remaining depth data is obtained by subtracting the preset reserved depth (e.g., 1mm) from the currently detected bevel depth.

[0102] (3) Perform volume integration on discrete points of the bevel contour within the target filling plane and the three-dimensional point cloud distribution window to obtain the accurate volume of the area to be filled.

[0103] The discrete points of the bevel profile are obtained from multiple one-dimensional laser line data arrays, such as... Figure 13 As shown, this is an array of multiple frames of one-dimensional laser line data continuously acquired along the welding direction within the window. Together, they form a 3D point cloud distribution window, and the target filling plane 5 is determined based on the currently detected bevel depth and the preset reserved depth. Furthermore, the window length depends on the length of the molten pool under the current welding process conditions (obtained by capturing real-time images of the molten pool using a molten pool camera and extracting the length of the molten pool contour in the welding direction), thereby effectively controlling the welding process parameters. The volume integral calculation expression is:

[0104] in, The volume of the region to be filled; Fill the height of the target; Index for a single laser line The discrete physical profile height, i.e., the discrete points of the bevel profile; The spacing between discrete laser line points; The physical scanning distance interval between the two laser lines; This is an index of the laser line profile within a certain window range.

[0105] (4) Based on the volume of the area to be filled and the constraints of the welding process parameters in the process database, determine the welding process parameters that meet the filling requirements of the area to be filled, and control the welding torch to weld the area to be filled according to the welding process parameters. Among them, the welding process parameters include welding speed, wire feed speed, heat input, etc.

[0106] In this embodiment, after calculating the volume of the region to be filled, the volume value is used to implement feedforward planning of the welding speed. Since the welding process must meet the requirements for linear heat input, and the process database already imposes dual constraints on welding speed and welding current (uniform wire feed speed), the determination of the welding speed is not a unique solution. Instead, it seeks the optimal solution within the constraints of welding speed and welding current, while ensuring the filling volume V is met. The principle for finding this optimal solution is: to achieve the target filling volume while ensuring that the welding speed and welding current are as far away from the process boundary as possible. That is: first, estimate the required wire feed based on the filling volume V and preset single-pass welding parameters (such as weld cross-sectional area); then, within the constraints of the process database, select a combination of welding speed and welding current that meets the wire feed requirement and is far from the process boundary, thereby generating the feedforward control quantity.

[0107] In one embodiment, the control method further includes: (1) Based on the horizontal distance between the upper bevel feature line (upper bevel feature point) and the lower bevel feature line (lower bevel feature point) (current overall weld width) ) and the preset single-pass welding width Calculate the number of weld beads in the current weld layer.

[0108] The width of a single weld pass can be preset by measuring it through multiple welding experiments. In this embodiment, the ratio of the current total weld pass width to the preset width of a single weld pass is the number of weld passes in the current weld layer.

[0109] (2) Determine the bevel depth based on the height difference between the upper bevel feature line and the lower bevel feature line, determine the target filling depth (target welding remaining depth) based on the difference between the bevel depth and the preset reserved depth, and determine the number of welding layers based on the ratio of the target filling depth to the preset single filling thickness.

[0110] like Figure 14 As shown, This refers to the bevel depth, which is the remaining depth of the currently planned welding. Preset depth; single fill thickness The thickness is determined as follows: Under multiple identical welding conditions, laser line data is collected at corresponding positions before and after welding, and the height difference between the two measurements at corresponding positions is calculated. The height difference obtained from multiple measurements is then statistically averaged, and this average value is used as the single-pass fill thickness. In this embodiment, the number of welding layers is the currently planned remaining welding depth. With preset reserved depth The difference and the thickness of a single-layer weld The ratio, i.e. , The depth to be filled.

[0111] (3) Welding gun is controlled to weld layer by layer and weld by weld according to the number of welding layers, the number of weld passes, and the initial position of the welding wire corresponding to each weld pass.

[0112] Specifically, such as Figure 14 As shown, the current total width of the weld bead in the bevel width direction is... Divide the layers equally according to the number of weld passes to obtain multiple weld passes for each layer. The center point of each equally divided position is the initial position of the welding wire corresponding to each weld bead. Control the welding torch to move sequentially to the first weld bead of the current layer. Welding is performed at the center of the weld (center of the weld) until the weld bead is completed. .

[0113] Preferably, this embodiment only plans the welding data of the current layer to be welded and the current weld bead in real time. After each layer of welding is completed, the welding parameters of the next layer or the next pass are recalculated based on the real-time collected bevel profile data, thereby dynamically adjusting the weld bead height of different layers, such as... Figure 14 As shown, This refers to the single-pass filler thickness of each weld pass in the first weld layer; For the first The single-pass filler thickness of each weld bead in the weld layer. This strategy eliminates the need for pre-planning data for all weld bead layers, effectively avoiding cumulative errors caused by deviations between pre-planned parameters and actual working conditions.

[0114] Preferably, controlling the welding torch to weld along the center of the weld includes: (1) Determine the center position of the slope based on the feature lines of the upper and lower slopes; (2) Calculate the deviation between the center position of the bevel and the preset reference position; (3) Generate welding torch position adjustment command based on deviation value to control the welding torch to move along the center of the weld.

[0115] Specifically, by accurately identifying the location of feature points, the center position of the bevel corresponding to the current welding torch is constrained. When there is a deviation between the bevel center identified by the laser line and the 0 point position of the laser line, the oscillation center position of the welding torch is adjusted through PID control to ensure that the welding torch is in the center of the weld throughout the entire process. The PID control formula is as follows:

[0116] in, This refers to the real-time sampling time; for The amount of welding torch position adjustment output by the PID controller at any given time; This is the proportionality coefficient; The integral coefficient; These are the differential coefficients; for Real-time deviation of weld position; for Historical deviation of weld position at any given time; for Real-time deviation of weld position. In this embodiment, such as... Figure 15 As shown, the deviation value As the input signal to the proportional-integral-derivative (PID) controller, the PID controller outputs the angular velocity or position adjustment of the welding torch, which is then applied to the volume of the area to be filled. During welding, the welding torch is dynamically tracked with high precision along the center of the weld, ensuring that the welding torch is always aligned with the center of the weld during the welding process, thus improving welding accuracy.

[0117] Combining the above embodiments yields a preferred embodiment of the feature extraction method of the present invention, in which the feature extraction method includes the following steps: S10: Acquire laser line images of the weld bevel surface. S20: Use the pre-trained first neural network model to perform online inference on the laser line image and output a physical space mapping image; S30: Scan the physical space mapping image, extract the position of the center point of each laser line in the physical space mapping image, and scale the position index of each laser line center point to the physical space according to the fixed proportional correspondence to obtain a one-dimensional laser line data array. S40: Combine the continuously acquired multi-frame one-dimensional laser line data arrays according to the scanning interval to form three-dimensional point cloud data; S50: Converts 3D point cloud data into a 2D pseudo-color image, where the pixel colors in the 2D pseudo-color image correspond to the depth values ​​of the 3D point cloud data. S60: The pre-trained second neural network model identifies two-dimensional pseudo-color images and outputs the locations of key feature points of the weld bevel and the locations of weld defect areas.

[0118] Combining the above embodiments yields a preferred embodiment of the control method of the present invention, which can be implemented by a welding machine control system. In this case, the control method includes the following steps: S70: Calculate the number of weld beads in the current weld layer based on the horizontal spacing between the upper and lower bevel feature lines and the preset single-pass welding width; determine the bevel depth based on the height difference between the upper and lower bevel feature lines; determine the target filling depth based on the difference between the bevel depth and the preset reserved depth; and determine the number of welding layers based on the ratio of the target filling depth to the preset single-pass filling thickness, thereby generating the planned weld bead layer. S80: Determine the center position of the bevel based on the upper bevel feature line and the lower bevel feature line; calculate the deviation between the center position of the bevel and the preset reference position; generate a welding torch position adjustment command based on the deviation value; and control the welding torch to weld layer by layer and weld by weld along the center of the weld according to the number of welding layers, the number of weld passes, and the initial position of the welding wire corresponding to each weld pass. S90: Based on the spatial relationship between the line structured light module and the welding torch, determine the 3D point cloud distribution window corresponding to the current welding position; determine the bevel depth based on the height difference between the upper and lower bevel feature lines; determine the target filling height based on the difference between the bevel depth and the preset reserved depth; use the plane containing the target filling height as the target filling plane; perform volume integration using the discrete points of the bevel contour within the 3D point cloud distribution window and the target filling plane to obtain the volume of the area to be filled; based on the volume of the area to be filled and combined with the constraint range of the welding process parameters in the process database, determine the welding process parameters that meet the filling requirements of the area to be filled, and control the welding torch to weld the area to be filled according to the welding process parameters; S100: Obtain the location and type of the welding defect area; before the welding torch reaches the welding defect area, generate corresponding feedforward control instructions based on the defect type, which are used to adjust the welding parameters or welding torch posture when the welding torch reaches the welding defect area in order to correct the welding defect.

[0119] The feature extraction method of this invention achieves end-to-end inference from laser line images to physical space mapping images through a first neural network model, eliminating the need for the complex process of real-time calibration parameter calculation in traditional schemes, thus effectively improving the efficiency of physical coordinate transformation. A second neural network model identifies two-dimensional pseudo-color images, simultaneously outputting the locations of key feature points and welding defect areas, simplifying the processing flow compared to the step-by-step processing in traditional schemes. Based on this, weld layer and weld bead planning, weld seam tracking, and volume integration of the area to be filled are performed based on the accurately identified feature point locations, realizing feedforward planning and control of welding path and process parameters. Furthermore, corresponding correction strategies are executed according to the defect type, thereby improving welding quality.

[0120] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method for feature extraction in an end-to-end welding process, characterized in that, Includes the following steps: Acquire laser line images of the weld bevel surface; The laser line image is inferred using a pre-trained first neural network model, and a physical space mapping image is output; the pixel space in the physical space mapping image has a fixed proportional correspondence with the physical space. The physical space mapping image is scanned, the center point position of each laser line in the physical space mapping image is extracted, and the position index of each laser line center point is scaled to the physical space according to the fixed proportional correspondence to obtain a one-dimensional laser line data array. The array of continuously acquired one-dimensional laser line data frames is combined according to the scanning interval to form three-dimensional point cloud data; The pre-trained second neural network model is used to identify the three-dimensional point cloud data and output the key feature points of the welding bevel, including the upper bevel feature line and the lower bevel feature line.

2. The end-to-end welding process feature extraction method according to claim 1, characterized in that, The step of scanning the physical space mapping image and extracting the center point position of each laser line in the physical space mapping image includes: The physical space mapping image is traversed column by column, and each column is scanned row by row. The pixels in each column are binarized using a preset probability threshold. Pixels that continuously exceed the preset probability threshold are stored in a sliding window. When a pixel is found that is below the preset probability threshold, the median or mean of all row coordinates in the current sliding window is calculated and used as the candidate laser line center point position for the current column. The sliding window is then cleared. The above process is repeated until the candidate laser line center point position corresponding to the first valid column is obtained, which is recorded as the initial seed point. After obtaining the initial seed point, stop the global scan. Using the row coordinates of the initial seed point as a reference, set the local search radius and scan the left and right adjacent columns within the local search radius. During the scanning of adjacent columns within the local search radius, a sliding window mechanism is used to calculate the median or mean and use it as the candidate laser line center point position of the column. The candidate laser line center point position calculated in the previous column is used as the center of the search window for the next column. The process is continued column by column until all columns are covered, and the complete center point positions of each laser line are extracted.

3. The end-to-end welding process feature extraction method according to claim 1, characterized in that, Before using the pre-trained second neural network model to identify the 3D point cloud data, the process also includes: The three-dimensional point cloud data is converted into a two-dimensional pseudo-color image, and the pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data. The two-dimensional pseudo-color image is input into the second neural network model so that the second neural network model can identify feature points in the two-dimensional pseudo-color image.

4. The end-to-end welding process feature extraction method according to claim 3, characterized in that, When the second neural network outputs the key feature point locations of the weld bevel, it also includes: The location of the output welding defect area includes at least one of the following: undercut defect area, porosity defect area, sidewall groove defect area, and hump defect area.

5. The end-to-end welding process feature extraction method according to claim 4, characterized in that, The location of the weld defect area is output, including: The second neural network model identifies defect regions in multiple consecutive frames of two-dimensional pseudo-color images and stores the laser line index number corresponding to the identified defect regions. During the welding robot's movement along the welding direction, a new frame of one-dimensional laser line data array enters the recognition window, and an old frame of one-dimensional laser line data array leaves the recognition window. The updated recognition window's multi-frame one-dimensional laser line data array is assembled into a new two-dimensional pseudo-color image and input into the second neural network model for defect area recognition. The recognition results of the new defect area and the corresponding laser line index number are stored. Repeat the above steps until the one-dimensional laser line data array within the recognition window is completely refreshed; Defect regions whose laser line index numbers overlap or are spaced within a preset threshold range in the recognition results of multiple frames are treated as the same defect region for data fusion. The spatial location of the defect region's boundary is then determined, and the spatial location of the defect region's boundary is output.

6. A welding control method, characterized in that, The method for extracting end-to-end welding process features according to any one of claims 1-5, after outputting the location of the welding defect area, further includes: The location and type of the welding defect area are obtained; before the welding torch reaches the welding defect area, the corresponding feedforward control command is generated according to the defect type, which is used to adjust the welding parameters or welding torch posture when the welding torch reaches the welding defect area in order to correct the welding defect.

7. The welding control method according to claim 6, characterized in that, After the second neural network model outputs the locations of the key feature points of the weld bevel, it also includes: Based on the spatial relationship between the line structured light module and the welding torch, determine the three-dimensional point cloud distribution window corresponding to the current welding position; The bevel depth is determined based on the height difference between the upper and lower bevel feature lines. The target filling height is determined based on the difference between the bevel depth and the preset reserved depth. The plane containing the target filling height is used as the target filling plane. The volume of the area to be filled is obtained by integrating the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window. Based on the volume of the area to be filled and the constraints of the welding process parameters in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, and the welding torch is controlled to weld the area to be filled according to the welding process parameters.

8. A feature extraction system for an end-to-end welding process, characterized in that, The system includes: A laser line image acquisition unit is used to acquire laser line images of the weld bevel surface; The first inference unit uses a pre-trained first neural network model to infer the laser line image and outputs a physical space mapping image; the pixel space in the physical space mapping image has a fixed proportional correspondence with the physical space. The post-processing unit is used to scan the physical space mapping image, extract the position of the center point of each laser line in the physical space mapping image, and scale the position index of each laser line center point to the physical space according to the fixed ratio correspondence to obtain a one-dimensional laser line data array. The point cloud generation unit is used to combine multiple frames of continuously acquired one-dimensional laser line data arrays according to the scanning interval to form three-dimensional point cloud data; The second inference unit uses a pre-trained second neural network model to identify the three-dimensional point cloud data and output the key feature points of the welding bevel, including the upper bevel feature line and the lower bevel feature line.

9. The end-to-end welding process feature extraction system according to claim 8, characterized in that, The system also includes: An image conversion unit is used to convert three-dimensional point cloud data into a two-dimensional pseudo-color image; the pixel colors in the two-dimensional pseudo-color image correspond to the depth values ​​of the three-dimensional point cloud data. The second inference unit uses a pre-trained second neural network model to identify the two-dimensional pseudo-color image and outputs the location of key feature points of the weld bevel and the location of the weld defect area. The location of the weld defect area includes at least one of the following: undercut defect area, porosity defect area, sidewall groove defect area, and hump defect area.

10. A welding control system, characterized in that, The control system is implemented based on the end-to-end welding process feature extraction system of claim 8 or 9, and the control system includes: The weld bead planning unit is used to calculate the number of weld beads in the current weld layer based on the horizontal spacing between the upper and lower bevel feature lines and the preset single-pass welding width; determine the bevel depth based on the height difference between the upper and lower bevel feature lines; determine the target filling depth based on the difference between the bevel depth and the preset reserved depth; and determine the number of weld layers based on the ratio of the target filling depth to the preset single-pass filling thickness; control the welding torch to weld layer by layer and bead by bead according to the number of weld layers, the number of weld beads, and the initial position of the welding wire corresponding to each weld bead; and / or, A defect correction unit is used to acquire the location and type of the welding defect area. Before the welding torch reaches the welding defect area, it generates corresponding feedforward control commands based on the defect type. These commands are used to adjust welding parameters or welding torch posture when the welding torch reaches the welding defect area to correct the welding defect; and / or, The welding planning unit is used to determine the bevel depth based on the height difference between the upper and lower bevel feature lines, and to determine the target filling height based on the difference between the bevel depth and the preset reserved depth. The plane containing the target filling height is used as the target filling plane. The volume of the area to be filled is obtained by integrating the discrete points of the bevel contour within the target filling plane and the 3D point cloud distribution window. Based on the volume of the area to be filled and the constraint range of the welding process parameters in the process database, the welding process parameters that meet the filling requirements of the area to be filled are determined, and the welding torch is controlled to weld the area to be filled according to the welding process parameters.

Images