Welding track determination method and device, electronic equipment and medium

Abstract

The invention relates to a method and device for determining a welding track, electronic equipment and a medium, and belongs to the technical field of welding.The method comprises the steps that a left view of a target binocular image of a target steel structure part is input into a well-trained deep learning model to obtain a welding seam area mask image, and the welding seam area mask image is obtained; the target binocular image is shot by a binocular camera installed on the target robot; the left view and the right view of the target binocular image are matched based on a Foundation Stereo model, and a point cloud of the target steel structure part is obtained; segmenting the point cloud of the target steel structure part based on the welding seam area mask graph to obtain a welding seam area point cloud; performing geometric fitting on the point cloud of the weld zone to obtain a weld welding track; and spatial pose information of a welding gun of the target robot is obtained based on the weld joint welding track, and the welding track of the target robot is determined based on the spatial position information. The welding precision of the steel structure part is improved.

Description

Technical Field

[0001] This invention relates to the field of welding technology, and in particular to a method, apparatus, electronic device, and medium for determining a welding trajectory. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing, robotic welding has been widely used in engineering manufacturing fields such as structural steel.

[0003] Path planning for welding robots is a key step in achieving high-quality welding. Currently, traditional trajectory acquisition methods mainly include manual teaching and offline programming based on CAD models. Manual teaching is the most common method for inputting welding paths. It requires a person to guide the robot to the target weld position point by point using a handheld teach pendant, and then record the welding trajectory points one by one. Although this method is intuitive and easy to operate, it has obvious disadvantages: (1) low programming efficiency and long teaching cycle for complex workpieces; (2) high dependence on operator experience, making it difficult to guarantee trajectory quality and consistency; (3) lack of flexibility and versatility, requiring re-teaching every time the workpiece or tooling is changed; (4) inability to cope with trajectory errors caused by workpiece position or weld deviation in actual production, seriously affecting welding accuracy and stability. The offline programming method based on CAD models imports the three-dimensional CAD model of the workpiece into offline programming software, performs virtual trajectory planning and simulation, and uploads the generated program to the robot system. This method improves the efficiency of trajectory compilation and has the advantages of visualization and reusability, but it also has several limitations: (1) It depends on the high-quality three-dimensional model of the workpiece. Once there is a deviation between the actual workpiece and the model (such as slight changes in weld size or position), it will lead to trajectory execution error; (2) It cannot perceive the detailed features of the actual weld surface such as oxidation, deformation, and cracks; (3) The trajectory is rigid and lacks the ability to adapt to complex and irregular welds; (4) The simulation and debugging process is complex and requires high technical skills from personnel.

[0004] In summary, the existing technology lacks a method for determining the welding trajectory to improve welding efficiency and accuracy. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, apparatus, electronic device and medium for determining the welding trajectory in order to solve the problem of low welding efficiency and accuracy of welding robots.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a method for determining a welding trajectory, comprising: The left view of the target binocular image of the target steel structure part is input into the fully trained deep learning model to obtain the weld area mask image. The target binocular image is taken by the binocular camera installed on the target robot. The left and right views of the target binocular image are matched based on the FoundationStereo model to obtain the point cloud of the target steel structure part; The point cloud of the target steel structure part is segmented based on the weld area mask image to obtain the weld area point cloud; Geometric fitting is performed on the point cloud of the weld area to obtain the weld trajectory; The spatial pose information of the target robot's welding torch is obtained based on the weld seam welding trajectory, and the welding trajectory of the target robot is determined based on the spatial position information.

[0007] In one possible implementation, the training process of the fully trained deep learning model includes: Acquire a dataset of weld seam images of steel parts, including various types of weld seam images and weld seam images under various working conditions; The weld seam image dataset is used to train a pre-defined deep learning model to obtain a fully trained deep learning model.

[0008] In one possible implementation, the fully trained deep learning model includes: One or more of YOLOv11, mamba, SSD, and DETR.

[0009] In one possible implementation, the matching of the left and right views of the target binocular image based on the FoundationStereo model to obtain the point cloud of the target steel structure part includes: Based on the FoundationStereo model, stereo matching is performed on the left and right views of the target binocular image to obtain a disparity map; Three-dimensional reconstruction of the disparity map is performed based on the reprojection matrix to obtain the point cloud of the target steel structure part.

[0010] In one possible implementation, segmenting the point cloud of the target steel structure part based on the weld area mask image to obtain the weld area point cloud includes: Based on the weld area mask image, the pixel mask in the left view is projected onto the point cloud of the target steel structure part to obtain the point cloud of the weld area.

[0011] In one possible implementation, the geometric fitting of the point cloud in the weld area to obtain the weld trajectory includes: Curve fitting is performed on the point cloud of the weld area to obtain the initial weld centerline or edge profile; The initial weld center and the edge contour are fitted with a surface to obtain the fitted surface of the weld. The welding trajectory of the weld is obtained based on the fitted surface of the weld.

[0012] In one possible implementation, obtaining the spatial pose information of the target robot welding torch based on the weld seam welding trajectory includes: The welding trajectory of the weld is smoothed based on the B-spline curve fitting algorithm to obtain the processed welding trajectory of the weld. The processed weld trajectory is sampled according to a preset step size or time interval to obtain a trajectory point sequence; The spatial pose information of the target robot's welding torch is determined based on the trajectory point sequence.

[0013] Secondly, the present invention also provides a device for determining a welding trajectory, comprising: The weld seam area mask image acquisition module is used to input the left view of the target binocular image of the target steel structure part into the fully trained deep learning model to obtain the weld seam area mask image. The target binocular image is captured by a binocular camera mounted on the target robot. The point cloud acquisition module for the target steel structure part is used to match the left and right views of the target binocular image based on the FoundationStereo model to obtain the point cloud of the target steel structure part. The weld seam area point cloud acquisition module is used to segment the point cloud of the target steel structure part based on the weld seam area mask image to obtain the weld seam area point cloud. The weld trajectory acquisition module is used to perform geometric fitting on the point cloud of the weld area to obtain the weld trajectory. The robot's welding trajectory determination module is used to obtain the spatial pose information of the target robot's welding torch based on the weld seam welding trajectory, and to determine the target robot's welding trajectory based on the spatial position information. Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the method for determining a welding trajectory as described in any of the above implementations.

[0014] Fourthly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the method for determining a welding trajectory described in any of the above implementations.

[0015] The beneficial effects of this invention are as follows: This invention provides a method for determining welding trajectories. The left view of a target binocular image of a target steel structure part is input into a fully trained deep learning model to obtain a weld area mask. The target binocular image is captured by a binocular camera mounted on a target robot. The left and right views of the target binocular image are matched based on the FoundationStereo model to obtain a point cloud of the target steel structure part. The three-dimensional spatial environment of the area to be welded is accurately perceived through camera vision. The point cloud of the target steel structure part is segmented based on the weld area mask to obtain a weld area point cloud. Geometric fitting is performed on the weld area point cloud to obtain the weld welding trajectory. The spatial pose information of the target robot's welding torch is obtained based on the weld welding trajectory. The welding trajectory of the target robot is determined based on the spatial position information. This invention improves welding efficiency and accuracy by accurately perceiving the three-dimensional spatial environment of the area to be welded through camera vision and realizing automated programmed welding techniques. Attached Figure Description

[0016] Figure 1 A flowchart illustrating an embodiment of a method for determining a welding trajectory provided by the present invention; Figure 2 A detailed flowchart of an embodiment of a welding trajectory determination method provided by the present invention; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S101; Figure 4 A schematic diagram of the structure of a YOLOv11 model, which is an embodiment of the welding trajectory determination method provided by the present invention; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 6 For the present invention Figure 1 A schematic diagram of an embodiment of S104; Figure 7 For the present invention Figure 1 A schematic diagram of an embodiment of S106; Figure 8 An embodiment of a method for determining a welding trajectory provided by the present invention includes an image of the marked area of ​​a structural steel weld. Figure 9 An example diagram of the inference results of a YOLOv11 instance segmentation model for a method for determining welding trajectories provided by the present invention; Figure 10 In one embodiment of the method for determining welding trajectory provided by the present invention, the reasoning result is converted into an image in the form of a mask; Figure 11 An embodiment of a method for determining a welding trajectory provided by the present invention uses a structural steel depth map derived from Foundation Stereo inference; Figure 12 An embodiment of a method for determining a welding trajectory provided by the present invention uses a structural steel point cloud obtained through Foundation Stereo inference; Figure 13 A schematic flowchart of an embodiment of the welding trajectory determination method and apparatus provided by the present invention; Figure 14 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0018] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0019] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Before demonstrating the embodiments, the following terms will be explained.

[0022] Foundation Stereo is an open-source stereo depth estimation foundation model developed by NVIDIA, which focuses on achieving zero-shot generalization capabilities and can be applied across domains without fine-tuning.

[0023] This invention provides a method, apparatus, electronic device, and medium for determining welding trajectories, which will be described below.

[0024] Figure 1 A schematic flowchart of an embodiment of the welding trajectory determination method provided by the present invention is shown below. Figure 1 As shown, the method for determining the welding trajectory includes: S101. Input the left view of the target binocular image of the target steel structure part into the trained deep learning model to obtain the weld area mask. The target binocular image is taken by the binocular camera installed on the target robot. S102. Match the left and right views of the target binocular image based on the FoundationStereo model to obtain the point cloud of the target steel structure part; The FoundationStereo model is a foundational model for stereo depth estimation.

[0025] S103. Segment the point cloud of the target steel structure part based on the weld area mask image to obtain the weld area point cloud; S104. Perform geometric fitting on the point cloud of the weld area to obtain the weld trajectory; S105. Obtain the spatial pose information of the target robot's welding torch based on the weld seam welding trajectory, and determine the target robot's welding trajectory based on the spatial position information.

[0026] Compared with existing technologies, this embodiment provides a method for determining welding trajectories. The left view of a target binocular image of a target steel structure part is input into a fully trained deep learning model to obtain a weld seam mask. The target binocular image is captured by a binocular camera mounted on a target robot. The left and right views of the target binocular image are matched based on the FoundationStereo model to obtain a point cloud of the target steel structure part. The three-dimensional spatial environment of the area to be welded is accurately perceived through camera vision. The point cloud of the target steel structure part is segmented based on the weld seam mask to obtain a weld seam point cloud. Geometric fitting is performed on the weld seam point cloud to obtain the weld welding trajectory. The spatial pose information of the target robot's welding torch is obtained based on the weld welding trajectory. The welding trajectory of the target robot is determined based on the spatial position information. This invention improves welding efficiency and accuracy by accurately perceiving the three-dimensional spatial environment of the area to be welded through camera vision and achieving automated programmed welding.

[0027] In specific embodiments of the present invention, such as Figure 2 The following is a detailed flowchart of this embodiment.

[0028] In a specific embodiment of the present invention, in step S101, a suitable binocular vision camera is selected, and a hardware platform is built. Based on the actual needs of the welding scenario, an industrial-grade binocular vision camera system with high resolution and frame rate is selected. The camera is mounted on the rigid support of the robot, and by adjusting the baseline distance and the field of view of the binocular camera, it is adapted to the working space of the structural steel component, so that the binocular camera can simultaneously capture the complete structural steel component.

[0029] To ensure complete imaging and appropriate resolution of the weld area, this invention considers factors such as workpiece size and camera mounting position when selecting the camera lens focal length. Specifically, assuming the field of view of the workpiece on the camera imaging plane is W, the mounting distance from the camera to the workpiece is D, and the effective size of the camera sensor is L, the required lens focal length f can be approximately determined by the following formula:

[0030] The ratio of D to W reflects the proportion of the workpiece in the image, while L represents the imaging size of the camera sensor. Using this method, the camera lens focal length can be appropriately selected in practical applications to ensure that the weld area is projected completely and clearly onto the sensor.

[0031] Perform camera calibration and obtain camera intrinsic and extrinsic parameters. The stereo camera is calibrated to obtain its intrinsic parameter matrix, distortion coefficients, and the extrinsic parameter relationship between the left and right cameras. The Zhang Zhengyou calibration method can be used for precise calibration of the stereo camera. A high-precision checkerboard calibration board is used during the calibration process, and multiple sets of images are acquired at different angles and distances (at least 15-20 sets of images) to solve for the parameters. The acquired images can be calibrated using MATLAB toolboxes or OpenCV. The obtained parameters are used for image distortion correction and epipolar alignment, thus providing an accurate basis for subsequent image stereo matching and depth estimation.

[0032] Calibration error assessment The quality of the calibration results is evaluated by calculating the mean reprojection error. The calibration results should result in a reprojection error below 0.5 pixels, ideally within 0.2-0.3 pixels, to ensure a denser point cloud. If the mean reprojection error is too large, the process of selecting a suitable binocular vision camera and calibrating the camera needs to be repeated.

[0033] In some embodiments of the present invention, in step S101, such as Figure 3As shown, the training process of the fully trained deep learning model includes: S301. Obtain the weld seam image dataset of the steel parts. The weld seam image dataset includes various types of weld seam images and weld seam images under various working conditions. S302. The weld seam image dataset is used to train the preset deep learning model to obtain a fully trained deep learning model.

[0034] In some embodiments of the present invention, the fully trained deep learning model includes: One or more of YOLOv11, mamba, SSD, and DETR.

[0035] In specific embodiments of the present invention, such as Figure 4 The diagram shows the structure of the YOLOv11 model. This embodiment uses the YOLOv11 model. The following is the data set construction process for this embodiment. Specifically, a large number of weld seam images of structural steel components are collected, covering various weld seam types (such as fillet welds, butt welds, lap welds, and spatial curve welds) and different working conditions (such as different lighting conditions, material textures, and surface reflections). Images can be acquired using a regular camera, and the captured images can be rotated at different angles to enhance the dataset. In the data annotation stage, pixel-level semantic segmentation is used to annotate the weld seam regions, simply by using polygons to frame the weld seam regions. The annotation format is selected according to the format required by the deep neural network semantic segmentation model. The constructed dataset contains diverse samples to improve the model's generalization ability. The dataset is further divided into training, validation, and test sets, with a ratio of 7:2:1.

[0036] This invention employs a deep learning-based instance segmentation model. This type of model can simultaneously output the target location, category label, and corresponding pixel-level segmentation mask in a single stage, thus avoiding the latency and inconsistency problems inherent in traditional two-stage "detection + segmentation" methods. Compared to traditional methods, this type of model maintains high accuracy while offering higher inference speed and lower computational resource consumption, making it suitable for industrial deployments and applicable to structural steel weld seam region identification tasks. Therefore, this embodiment uses YOLOv11 as the instance segmentation model.

[0037] The overall process of the deep learning-based weld area recognition method includes the following steps: Step 1: Preprocess the images by normalizing the images in the captured dataset, adjusting the input size, and performing color space standardization (e.g., converting to RGB). The second step is feature extraction. The preprocessed image is first input into the model backbone network, such as the improved CSPDarkNet, PVTv2, or a lightweight Transformer module. The model automatically extracts multi-scale features from the image through convolutional structures, attention mechanisms, and cross-layer connections, including information such as the shape, edges, brightness gradient, and background differences of the weld. The resulting feature map not only captures the local weld texture details but also retains spatial structure information, providing rich semantic support for subsequent segmentation inference. Furthermore, the parameters of this module are automatically updated through backpropagation during the training phase, enabling the network to adapt to the feature changes of the weld under different working conditions. Step 3: The feature map is fed into the segmentation head via a multi-scale fusion module (such as FPN). The model then outputs the bounding box, category information, and corresponding pixel-level segmentation mask of the weld target. False targets are filtered out by setting a confidence threshold, retaining high-confidence weld regions and outputting the mask of the resulting region. The output mask is represented as a polygonal outline or a binary image for easy subsequent processing. Step 4: Post-processing of the mask and outputting the results. Morphological operations (such as opening / closing operations, edge smoothing, and contour optimization) are performed on the initial mask image. Two-dimensional pixel information of the weld region is extracted through coordinate mapping or indexing. Finally, a visualized weld region map is output.

[0038] Model training and evaluation During training, key hyperparameters such as model structure parameters, optimizer strategies, and the number of training epochs are appropriately set to improve recognition accuracy and model convergence efficiency. Several commonly adjusted parameters include input size (imgsz), batch size, initial learning rate, optimizer, loss function, number of training epochs, and early stopping strategy. These parameters need to be adjusted based on the graphics card's memory. After training, the model is quantitatively evaluated using the following metrics: IoU (Intersection over Union): measures the overlap between the predicted weld seam region and the ground truth label, a key indicator for evaluating segmentation accuracy; F1-score: considers precision and recall, reflecting the overall segmentation quality of the model; Precision and Recall: reflect the model's accuracy and false negative rate when detecting weld seams, respectively; mAP (mean Average Precision): evaluates the model's overall performance at different confidence levels for the detection task; Inference time: records the model's prediction time per image (in milliseconds), used to evaluate its real-time performance in industrial deployment scenarios. A typical IoU ≥ 0.85 and F1-score ≥ 0.90 are sufficient to meet the high-precision requirements of structural steel weld area identification in industrial applications.

[0039] Robustness enhancement measures: The following methods can be used to enhance the model's recognition ability and adaptability in real-world scenarios: Data augmentation can be used during dataset construction, such as introducing random brightness and contrast adjustments, adding noise, random rotation, affine transformations, etc., to simulate different shooting conditions; Before training, the model structure can be optimized, such as combining the characteristics of slender targets in weld seam images and introducing lightweight attention mechanisms (such as SE, CBAM) to enhance detail perception; Multi-scale training and testing can be used to improve the model's recognition accuracy for weld seams of different sizes; K-fold cross-validation and Hard Negative Mining strategies can be used to optimize sample selection to improve generalization ability.

[0040] In some embodiments of the present invention, in step S102, as Figure 5 As shown, the point cloud of the target steel structure part is obtained by matching the left and right views of the target binocular image based on the FoundationStereo model, including: S501. Based on the FoundationStereo model, stereo matching is performed on the left and right views of the target binocular image to obtain a disparity map; S502. Based on the reprojection matrix, perform three-dimensional reconstruction of the disparity map to obtain the point cloud of the target steel structure part.

[0041] In some embodiments of the present invention, the step of segmenting the point cloud of the target steel structure part based on the weld area mask image to obtain the weld area point cloud includes: Based on the weld area mask image, the pixel mask in the left view is projected onto the point cloud of the target steel structure part to obtain the point cloud of the weld area.

[0042] In some embodiments of the present invention, in step S104, as Figure 6 As shown, the geometric fitting of the point cloud in the weld area to obtain the weld trajectory includes: S601. Perform curve fitting on the point cloud of the weld area to obtain the initial weld centerline or edge profile. S602. Perform surface fitting on the initial weld center and the edge contour to obtain the fitted surface of the weld; S603. The welding trajectory of the weld is obtained based on the fitted surface of the weld.

[0043] In a specific embodiment of this invention, to obtain high-quality, dense, and precisely defined 3D point cloud data of the weld area, this invention employs an advanced stereo matching network based on deep learning as the primary stereo matching method. This type of algorithm possesses excellent generalization ability and reconstruction accuracy, exhibiting particularly stable performance in industrial scenarios. Its core idea is to achieve matching results with strong generalization, high robustness in weakly textured regions, and excellent edge reconstruction accuracy through an end-to-end deep learning network, utilizing techniques such as multi-scale cost volume construction, feature enhancement mechanisms, and sparse matching guidance. Compared to traditional semi-global matching (SGBM) methods, this type of algorithm obtains dense point clouds without requiring the adjustment of numerous hyperparameters.

[0044] To address the characteristics of weld seam images, such as monotonous local textures, strong light reflection, and elongated edges, which easily lead to errors during matching, this invention proposes the following optimization strategies: Edge enhancement preprocessing: Before matching, edge enhancement processing (such as using Sobel or Canny operators) is performed on the image to highlight the weld seam boundary texture and improve the cost aggregation accuracy; Weld seam region-guided matching: Utilizing the weld seam segmentation results, higher weights are assigned to the weld seam region during cost calculation, guiding the matching algorithm to focus on key areas; Filtering and consistency verification: After generating the disparity map, left-right consistency detection and weighted least squares (WLS) filtering are performed to remove outliers and smooth the results; Multi-scale matching mechanism: A pyramid-style multi-resolution matching mechanism is adopted, progressively optimizing the matching accuracy from coarse to fine, taking into account both global consistency and local details.

[0045] Through the above optimizations, high-quality weld parallax maps and dense point clouds with less noise, clearer boundaries, and fewer void areas can be obtained.

[0046] Specifically, the point cloud extraction method for local weld seam areas is as follows: A trained deep learning instance segmentation model is employed, inputting the left view from a binocular image. The model outputs a segmented image containing the contour of the weld instance region and a weld region mask of the same size as the original image. In the mask, regions with a pixel value of 1 correspond to the weld, while 0 represents the background. Simultaneously, an advanced deep learning-based stereo matching network performs stereo matching on the left and right view image pairs, outputting a high-precision disparity map. 3D reconstruction is performed on the disparity map using the reprojection matrix Q, obtaining dense point cloud data, where each point corresponds to a pixel in the left view. Based on the generated weld mask, the pixel mask from the left view is projected onto the point cloud space. Only the point cloud coordinates corresponding to pixels with a mask value of 1 in the left view are retained, while irrelevant points outside the mask are removed, achieving accurate extraction of the weld region point cloud. During this process, the mask and disparity map have a pixel-level one-to-one correspondence, thus achieving seamless mapping and ensuring accurate point cloud cropping boundaries and complete region integrity. Further processing is performed on the extracted local weld area point cloud, such as voxel filtering, statistical outlier removal, and curvature calculation, to improve the continuity and noise robustness of the weld point cloud.

[0047] Weld area fitting First, the extracted local weld point cloud needs to be fitted with curves or surfaces to obtain the spatial morphology information of the weld. The fitting algorithm needs to be robust to noise and adaptable to complex shapes. Specific methods include, but are not limited to, the following two: RANSAC-based curve fitting: The core geometry in the weld point cloud is robustly estimated using the Random Sample Consensus (RANSAC) algorithm to remove outlier interference and obtain the initial weld centerline or edge profile.

[0048] Least squares surface fitting: Based on the initial results, a weighted least squares fitting algorithm is used to finely model the local surface, improving the continuity and smoothness of the trajectory geometry. It is applicable to various weld types (such as butt welds, fillet welds, etc.).

[0049] In addition, to enhance the accuracy of spatial fitting, the point cloud can be sliced ​​according to the main direction (such as projecting it onto a two-dimensional plane and then performing local fitting), and then the central trajectory can be reconstructed in three-dimensional space.

[0050] Welding trajectory generation and optimization After obtaining the weld centerline or fitted surface, the welding trajectory is generated based on the geometric information of the trajectory points and the welding process requirements. The following factors need to be considered during the generation of the welding trajectory: a reasonable linear velocity range needs to be set according to the material type, thickness, and equipment parameters to ensure a balance between weld quality and welding efficiency; secondly, the welding torch posture (including pitch and normal angles) needs to be dynamically adjusted based on the spatial tangent and normal directions of the fitted curve to ensure the welding torch is always aligned with the weld at the optimal angle.

[0051] Trajectory smoothing based on B-spline curve fitting The weld centerline generated by fitting may exhibit slight jitter or discontinuity due to point cloud noise or local anomalies in the fitting process. To address this, a B-spline curve fitting algorithm is used to smooth the discrete trajectory point sequence. The specific processing flow is as follows: This embodiment introduces a trajectory smoothing algorithm based on a third-order B-spline curve in the welding trajectory planning. By fitting the initial trajectory point sequence, a high-order continuous trajectory curve is generated, effectively eliminating trajectory jitter caused by point cloud errors and abrupt fitting changes. During the fitting process, an automatic control point generation and node vector optimization strategy is introduced, combined with welding speed for equally spaced sampling, outputting a highly smooth and stable welding trajectory point sequence to ensure the continuity and consistency of the subsequent welding process.

[0052] In some embodiments of the present invention, in step S105, as Figure 7 As shown, obtaining the spatial pose information of the target robot welding torch based on the weld seam welding trajectory includes: S701. The weld trajectory is smoothed based on the B-spline curve fitting algorithm to obtain the processed weld trajectory. S702. The processed weld trajectory is sampled according to a preset step size or time interval to obtain a trajectory point sequence. S703. Determine the spatial pose information of the target robot welding torch based on the trajectory point sequence.

[0053] In a specific embodiment of the present invention, the core of trajectory output lies in converting the fitted three-dimensional trajectory curve into a sequence of discrete path points that the robot can recognize. The specific process is as follows: First, the fitted or smoothed welding trajectory curve is sampled according to a set step size or time interval to obtain the trajectory point sequence. Then, necessary attitude information (such as normal vectors or Euler angles) is added to each trajectory point to form complete spatial pose information of the welding torch, i.e., six-degree-of-freedom data. Then convert the data to CSV format, with the format (x,y,z,roll,pitch,yaw,speed).

[0054] In summary, this embodiment constructs a structural steel weld seam image dataset and employs an advanced deep learning model to achieve high-precision instance segmentation of the weld seam region. Through model optimization and enhancement strategies, it significantly improves the recognition accuracy and robustness for complex backgrounds and different types of weld seams, providing reliable two-dimensional positioning information for subsequent 3D reconstruction and trajectory generation.

[0055] This embodiment covers the entire process of image acquisition, weld recognition, stereo matching, 3D modeling, trajectory planning and output, forming a complete closed loop. It has a high degree of automation and real-time performance, significantly reducing the degree of manual intervention and improving welding quality and production efficiency.

[0056] Example 2: I. Image Acquisition and Data Preparation Thirty sets of checkerboard images were captured using a stereo camera, and the stereo camera was calibrated using Stereo CameraCalibrator in MATLAB. The results were then exported, primarily obtaining the intrinsic parameter matrix K, extrinsic parameters (rotation matrix R, translation matrix T) of the left and right cameras, and the distortion coefficients of the left and right cameras.

[0057] Images of the weld seam area of ​​the structural steel parts were captured using a binocular camera, obtaining left and right views respectively. The image resolution was 1920×1080, and these images were used for point cloud generation. 500 weld seam images were captured using a monocular camera to create a deep learning dataset.

[0058] The labelme software was used to annotate 500 images of the weld area. The annotation results are as follows: Figure 8 As shown, the format is YOLO instance divider (class_id + polygon point coordinates).

[0059] To enhance data diversity and model robustness, data augmentation methods such as rotation, flipping, and brightness perturbation were applied to both the images and their annotations on the 500 already labeled images. The dataset was then expanded to 2000 images and divided into training, validation, and test sets in a 7:2:1 ratio.

[0060] II. Construction and Inference of Weld Region Identification Model This embodiment preferably uses the yolo11m-seg.pt instance segmentation model, and the YOLO network structure is as follows: Figure 4 As shown, the structure includes Backbone (C2f+C3 module), Neck (FPN structure) and Head (Detect + Proto).

[0061] Before training the model, some parameters need to be set: the input size imgsz is set to 640, the batch size is 32, the total number of training epochs is 500, the optimizer is AdamW, the initial learning rate is set to 0.001, and the cosine learning rate decay strategy is adopted.

[0062] After 500 training iterations, the model achieved the following accuracy: mAP50 of 96.6% and IoU of 88.3%. Testing on the test set demonstrated accurate weld contour segmentation with few missed detections and accurate extraction of the weld region mask. The input results are as follows: Figure 9 As shown.

[0063] The left view from the captured left and right views is input into the trained instance segmentation model. The model outputs a segmented image containing the outline of the weld instance region and a weld region mask of the same size as the original image, such as... Figure 10 As shown.

[0064] III. Depth Map Generation and Point Cloud Reconstruction (FoundationStereo) The captured binocular images were then subjected to epipolar correction using the Bouguet method.

[0065] Configure the FoundationStereo environment on a Linux system, inputting the corrected left and right images along with the camera's intrinsic and extrinsic parameters. Adjust the input size of the stereo images based on the graphics card's memory. The model then outputs high-quality depth maps and dense point clouds, such as... Figure 11 as well as Figure 12 As shown.

[0066] IV. Point Cloud Extraction and Processing in Welded Areas Using the aforementioned mask, the point cloud of the structural steel is selected, retaining only the point cloud locations where the mask value is 1, as shown below. Figure 5 The white areas in the image are then removed, and the areas with a mask of 0 are removed, which are the black areas in Figure 5. Further processing is performed on the extracted local weld point cloud, such as voxel filtering, statistical outlier removal, and curvature calculation, to improve the continuity and noise robustness of the weld point cloud.

[0067] V. Welding trajectory fitting and planning The RANSAC algorithm was used to fit the weld centerline, while B-spline smoothing was used for continuous welds.

[0068] The centerline is discretized into a sequence of trajectory points, and each point records its spatial coordinates (X, Y, Z) and normal direction (for welding torch posture).

[0069] The welding trajectory is smoothed a second time after interpolation to avoid abrupt changes in the trajectory.

[0070] VI. Trajectory Output and Robot Interface The final trajectory is exported in .csv format, containing the fields: [Point_ID, X, Y, Z, nx, ny, nz]. It can be read by general robot control systems (such as ABB, UR) for point-to-point or trajectory-following welding tasks.

[0071] To better implement the welding trajectory determination method in the embodiments of the present invention, based on the welding trajectory determination method, correspondingly, as follows: Figure 13 As shown, this embodiment of the invention also provides a welding trajectory determination device. A welding trajectory determination device 1300 includes: The weld area mask image acquisition module 1301 is used to input the left view of the target binocular image of the target steel structure part into a well-trained deep learning model to obtain the weld area mask image. The target binocular image is captured by a binocular camera mounted on the target robot. The point cloud acquisition module 1302 for the target steel structure part is used to match the left and right views of the target binocular image based on the FoundationStereo model to obtain the point cloud of the target steel structure part. The weld area point cloud acquisition module 1303 is used to segment the point cloud of the target steel structure part based on the weld area mask image to obtain the weld area point cloud. The weld trajectory acquisition module 1304 is used to perform geometric fitting on the point cloud of the weld area to obtain the weld trajectory. The robot welding trajectory determination module 1305 is used to obtain the spatial pose information of the target robot welding torch based on the weld seam welding trajectory, and to determine the welding trajectory of the target robot based on the spatial position information. The welding trajectory determination device 1300 provided in the above embodiment can realize the technical solution described in the above welding trajectory determination method embodiment. The specific implementation principle of each module or unit can be found in the corresponding content in the above welding trajectory determination method embodiment, and will not be repeated here.

[0072] like Figure 14 As shown, the present invention also provides an electronic device 1400. The electronic device 1400 includes a processor 1401, a memory 1402, and a display 1403. Figure 14 Only some of the components of the electronic device 1400 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0073] In some embodiments, processor 1401 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 1402 or process data, such as a method for determining welding trajectory in this invention.

[0074] In some embodiments, processor 1401 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1401 may be local or remote. In some embodiments, processor 1401 may be implemented on a cloud platform. In some embodiments, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, or any combination thereof.

[0075] In some embodiments, memory 1402 may be an internal storage unit of electronic device 1400, such as a hard disk or memory of electronic device 1400. In other embodiments, memory 1402 may also be an external storage device of electronic device 1400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on electronic device 1400.

[0076] Furthermore, the memory 1402 may include both internal storage units of the electronic device 1400 and external storage devices. The memory 1402 is used to store application software and various types of data installed on the electronic device 1400.

[0077] In some embodiments, display 1403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1403 is used to display information from electronic device 1400 and to display a visual user interface. Components 1401-1403 of electronic device 1400 communicate with each other via a system bus.

[0078] In one embodiment, when processor 1401 executes a procedure for determining a welding trajectory in memory 1402, the following steps may be performed: The left view of the target binocular image of the target steel structure part is input into the fully trained deep learning model to obtain the weld area mask image. The target binocular image is taken by the binocular camera installed on the target robot. The left and right views of the target binocular image are matched based on the FoundationStereo model to obtain the point cloud of the target steel structure part; The point cloud of the target steel structure part is segmented based on the weld area mask image to obtain the weld area point cloud; Geometric fitting is performed on the point cloud of the weld area to obtain the weld trajectory; The spatial pose information of the target robot's welding torch is obtained based on the weld seam welding trajectory, and the welding trajectory of the target robot is determined based on the spatial position information.

[0079] It should be understood that when the processor 1401 executes a welding trajectory determination program in the memory 1402, in addition to the functions described above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0080] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 1400 mentioned. The electronic device 1400 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1400 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0081] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0082] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of determining a welding trajectory, characterized in that, The method comprises the following steps: inputting a left view of a target binocular image of a target steel structure part into a trained deep learning model to obtain a weld seam region mask, the target binocular image being obtained by a binocular camera mounted on a target robot; matching a left view and a right view of the target binocular image based on a FoundationStereo model to obtain a point cloud of the target steel structure part; segmenting the point cloud of the target steel structure part based on the weld seam region mask to obtain a weld seam region point cloud; geometrically fitting the weld seam region point cloud to obtain a weld seam welding trajectory; obtaining spatial pose information of a welding torch of the target robot based on the weld seam welding trajectory, and determining a welding trajectory of the target robot based on the spatial pose information.

2. The method of determining a weld trajectory according to claim 1, wherein, The training process of the trained deep learning model comprises the following steps: obtaining a weld seam image dataset of a steel part, the weld seam image dataset comprising weld seam images of multiple types and under multiple working conditions; training a preset deep learning model based on the weld seam image dataset to obtain the trained deep learning model.

3. The method of determining a weld trajectory of claim 1, wherein, The trained deep learning model comprises: one or more of YOLOv11, mamba, SSD and DETR.

4. The method of determining a weld trajectory of claim 1, wherein, The matching of the left view and the right view of the target binocular image based on the FoundationStereo model to obtain the point cloud of the target steel structure part comprises the following steps: stereo matching the left view and the right view of the target binocular image based on the FoundationStereo model to obtain a disparity map; performing three-dimensional reconstruction on the disparity map based on a re-projection matrix to obtain the point cloud of the target steel structure part.

5. The method of determining a weld trajectory of claim 1, wherein, The segmentation of the point cloud of the target steel structure part based on the weld seam region mask to obtain the weld seam region point cloud comprises the following steps: projecting a pixel mask in the left view to the point cloud of the target steel structure part based on the weld seam region mask to obtain a weld seam region point cloud.

6. The method of determining a weld trajectory of claim 1, wherein, The geometric fitting of the weld seam region point cloud to obtain the weld seam welding trajectory comprises the following steps: curve fitting the weld seam region point cloud to obtain an initial weld seam center line or an edge contour; surface fitting the initial weld seam center and the edge contour to obtain a fitting surface of the weld seam; obtaining the weld seam welding trajectory based on the fitting surface of the weld seam.

7. The method of determining a weld trajectory of claim 1, wherein, The obtaining of the spatial pose information of the welding torch of the target robot based on the weld seam welding trajectory comprises the following steps: smoothing the weld seam welding trajectory based on a B-spline curve fitting algorithm to obtain a processed weld seam welding trajectory; sampling the processed weld seam welding trajectory at a preset step length or time interval to obtain a trajectory point sequence; determining the spatial pose information of the welding torch of the target robot based on the trajectory point sequence.

8. A welding trajectory determination apparatus characterized by comprising: The method comprises the following steps: a weld seam region mask acquisition module is configured to input a left view of a target binocular image of a target steel structure part into a trained deep learning model to obtain a weld seam region mask, the target binocular image being obtained by a binocular camera mounted on a target robot; a target steel structure part point cloud acquisition module is configured to match a left view and a right view of the target binocular image based on a FoundationStereo model to obtain a point cloud of the target steel structure part; The weld zone point cloud acquisition module is configured to segment the point cloud of the target steel structure part based on the weld zone mask diagram to obtain a weld zone point cloud. The weld seam welding track acquisition module is configured to perform geometric fitting on the weld zone point cloud to obtain a weld seam welding track. The welding track determination module of the robot is configured to obtain spatial pose information of a target robot welding torch based on the weld seam welding track, and determine a welding track of the target robot based on the spatial position information.

9. An electronic device, comprising: The memory is configured to store a program. The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the welding track determination method according to any one of claims 1 to 7. The computer readable program or instruction is stored in the computer readable storage medium, and the program or instruction is executed by the processor to implement the steps of the welding track determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, ​

Images