Welding track generation method and system based on mobile equipment image acquisition

Abstract

The invention provides a welding track generation method and system based on mobile equipment image acquisition, relates to the technical field of welding robot control, and solves the problems of tedious teaching operation, high three-dimensional reconstruction cost, complex visual calibration and inaccurate conversion in an existing scheme. The method comprises the following steps: acquiring a plurality of to-be-welded workpiece images with pose marks by using a mobile device carrying a camera, determining a reference image through a registration relation, calculating a relative pose, and reconstructing dense point clouds of the to-be-welded workpiece; then determining an initial coordinate system of the to-be-welded workpiece according to the pose mark in the reference image, and establishing a mapping relation between the initial coordinate system and a base coordinate system of a working space of the welding robot; and a welding track is extracted from the dense point cloud, a discrete welding point sequence is generated, finally, the discrete welding point sequence is mapped into a robot working space according to the mapping relation, and welding operation is completed. According to the method, the executable welding track of the welding robot can be quickly generated, and an automatic welding task is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of welding robot control, and particularly relates to a welding trajectory generation method and system based on image acquisition of a mobile device. BACKGROUND

[0002] In the field of industrial robot welding, accurate acquisition of the welding path is a core link to ensure welding quality and efficiency. At present, this process mainly relies on traditional technical means, including manual teach pendant operation and three-dimensional reconstruction systems based on structured light or laser scanning. These methods have formed a certain technical foundation in long-term practice, but with the rapid development of manufacturing towards multi-variety, small batch and high customization, their inherent defects have gradually emerged, making it difficult to meet the needs of modern flexible production. Especially in the scene of frequent process changes, the limitations of existing technologies have become a key bottleneck restricting the improvement of welding automation level.

[0003] As a widely used traditional method, the manual teaching method requires the operator to hold the teach pendant and manually guide the robot to move along the weld seam of the workpiece point by point, and to record the posture parameters of each key position in real time. This process highly depends on human experience, and the operation steps are tedious and time-consuming. In complex curved surface or long path welding tasks, personnel need to repeatedly adjust the joint angles of the robot, which is extremely easy to introduce cumulative errors due to fatigue or misoperation, resulting in a decrease in path accuracy. At the same time, this method lacks dynamic response capability to small deformations of the workpiece, and once the production environment is slightly disturbed, the full teaching process needs to be performed again, significantly reducing production efficiency and making it difficult to meet the requirements of high consistency welding.

[0004] In contrast, three-dimensional reconstruction technologies such as structured light or laser scanning attempt to acquire point cloud data of the workpiece surface through automated equipment, and then generate the welding path. However, such systems are usually equipped with high-precision optical sensors and special data processing software, which are costly and complex to deploy. In practical applications, the equipment is sensitive to the initial pose of the workpiece, and if the workpiece has a slight shift or deformation during clamping, the production process often needs to be interrupted for repeated scanning and calibration. In addition, compatibility problems between special software and robot control systems frequently occur, resulting in a lengthy path data conversion process, which further weakens its practicality in dynamic production environments. These factors limit the popularization of this technology in cost-sensitive small and medium-sized enterprises.

[0005] In recent years, the path acquisition method based on visual reconstruction has gradually attracted attention, which uses a camera to collect workpiece images and combines algorithms to solve the pose. Although this direction shows certain potential, most schemes still have obvious shortcomings. On the one hand, the system generally relies on professional industrial cameras and supporting light sources, and the hardware investment has not been significantly reduced; on the other hand, the image calibration process involves complex coordinate system conversion and parameter calibration, the operation threshold is high, and it is easy to be disturbed by environmental light changes. More importantly, the workpiece pose information extracted from the two-dimensional image is difficult to accurately map to the robot control coordinate system, and the path data is often distorted due to calibration errors or algorithm limitations, affecting the reliability of the welding trajectory. These problems make it difficult for visual methods to be stably applied in actual production lines.

[0006] Under the background of today's manufacturing industry emphasizing rapid response and low-cost operation, the above defects seriously hinder the popularization of industrial robots in diversified welding scenes. Especially for production environments with frequent process changes and limited budget, a more flexible, economical and robust alternative is urgently needed to break through the current technical bottleneck and promote the development of welding automation to a higher level. SUMMARY

[0007] The purpose of the present application is to solve the problems of the existing welding trajectory generation method, which is dependent on manual teaching, complicated operation, high cost of three-dimensional reconstruction equipment and limited adaptability, and the calibration of visual methods is complex and the conversion of pose information is not accurate. Therefore, a welding trajectory generation method and system based on mobile device image acquisition are proposed. The present application uses ordinary mobile devices such as mobile phones to take relevant images, combines two-dimensional code calibration and image pose solving, quickly generates welding trajectories executable by welding robots, and realizes automatic welding tasks. The present application is particularly suitable for flexible, small-batch, multi-variety welding production scenes, and has significant convenience and practicality.

[0008] The present application adopts the following technical solutions to achieve the purpose: A welding trajectory generation method based on mobile device image acquisition, comprising the following steps: S1, using a mobile device equipped with a camera, acquiring multiple images of a workpiece to be welded at different shooting angles; the workpiece to be welded is provided with a pose marker in advance; S2, based on the DUSt3R algorithm, the registration relationship between the multiple workpiece-to-be-welded images is used to calculate the camera pose of the remaining images relative to the first workpiece-to-be-welded image as the reference image, and the dense point cloud of the workpiece to be welded is reconstructed; S3, detecting the pose marker in the reference image through an image recognition algorithm, determining the initial coordinate system of the workpiece to be welded according to the position and pose of the pose marker in the reference image; then establishing the mapping relationship between the initial coordinate system and the work space base coordinate system of the welding robot; S4, based on the dense point cloud of the workpiece to be welded, welding trajectories are extracted therefrom and a discrete welding point sequence is generated; coordinates of each welding point in the reference image in the discrete welding point sequence are determined; S5, according to the established mapping relationship, the discrete welding point sequence is mapped into the welding robot workspace to generate control instructions, so that the welding robot performs a welding operation on the workpiece to be welded.

[0009] Preferably, in step S1, the pose marker is a two-dimensional code attached to the surface of the workpiece to be welded, which has a preset size, a high-contrast black-and-white pattern and a unique code identifier, which is used to provide a known geometric reference for calculating the registration relationship and determining the initial coordinate system.

[0010] Specifically, in step S1, the mobile device is a handheld smart terminal, and the operator holds the smart terminal to take pictures at a preset pitch angle around the workpiece to be welded at multiple different preset distances and orientations, thereby obtaining multiple workpiece images taken at different angles.

[0011] Further, in step S2, the DUSt3R algorithm is improved to calculate the camera pose of the remaining images relative to the reference image, specifically including: A lightweight feature extraction module is integrated in front of the Transformer encoder of the DUSt3R algorithm, which uses the SuperPoint algorithm to detect feature points and generate descriptors for each workpiece image, and predefines the corresponding feature point detection threshold and descriptor dimension; The feature descriptors between the multiple workpiece images are initially matched by the GMS algorithm to obtain matching point pairs; the fundamental matrix is calculated based on the matching point pairs, and then the RANSAC algorithm is applied for outlier rejection; before outlier rejection, the iteration number and inlier determination threshold of the RANSAC algorithm are pre-set; Taking the first workpiece image as the reference image, based on the matching point pairs after outlier rejection, the essential matrix of the remaining images relative to the reference image is solved by the five-point algorithm, and the rotation matrix and translation vector are decomposed to obtain the camera pose of the remaining images relative to the reference image. The rotation matrix is represented by Lie algebra and optimized by the LM optimizer, and thus the camera pose of the remaining images relative to the reference image is obtained.

[0012] Specifically, the dense point cloud of the workpiece to be welded is reconstructed, specifically including: inputting each image of the workpiece to be welded into a pre-trained depth estimation model for monocular depth prediction, outputting a corresponding depth map as a depth prediction result; aligning the depth prediction result with the relative value of each camera pose, and constructing a unified 3D volume grid using a weighted TSDF fusion strategy; then using the Marching Cubes algorithm to extract an isosurface from the 3D volume grid to generate an initial point cloud, and applying statistical outlier removal filtering processing, and after presetting the number of neighborhood points and the standard deviation multiple threshold, outputting the final dense point cloud of the workpiece to be welded, and completing the reconstruction.

[0013] Specifically, in step S3, the reference image plane coordinate system is denoted as o(l), the plane coordinate system of the pose marker in the reference image is taken as the initial coordinate system of the workpiece to be welded and denoted as o(p), and the work space base coordinate system of the welding robot is denoted as o(w); by calculating the rotation relationship among o(l), o(p) and o(w), the corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p) and the corresponding positions of each coordinate point in the initial coordinate system o(p) in the work space base coordinate system o(w) of the welding robot are obtained, and a mapping relationship is established.

[0014] Specifically, in step S3, the transformation matrix T(p-w) between the initial coordinate system o(p) and the work space base coordinate system o(w) of the welding robot is established in advance through a hand-eye calibration algorithm, and the initial coordinate system o(p) is directly mapped to the work space base coordinate system o(w) of the welding robot based on the transformation matrix T(p-w); then, based on the corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p), each coordinate point in the reference image plane coordinate system o(l) is also mapped to the work space base coordinate system o(w) of the welding robot; the welding trajectory formed by the coordinate points in the reference image plane coordinate system o(l) forms a corresponding trajectory point path in the work space base coordinate system o(w) of the welding robot.

[0015] Preferably, in step S4, a geometric feature analysis method is used to extract the welding trajectory and generate a discrete welding point sequence, specifically including: performing bilateral filtering on the dense point cloud to suppress noise and retain edge features; calculating the curvature value and normal vector of each point, and screening out weld candidate points based on a preset curvature threshold and normal vector mutation angle; performing linear or quadratic curve fitting on the weld candidate points through the RANSAC algorithm to generate a smooth welding trajectory; sampling on the welding trajectory at a fixed interval or based on a curvature adaptive dynamic interval to generate a discrete welding point sequence.

[0016] Preferably, in step S4, the neural network algorithm is used to extract the welding trajectory and generate a discrete welding point sequence, specifically including: projecting the dense point cloud along the optical axis direction of the reference image into a depth image, and normalizing it into a fixed resolution input tensor; inputting into a pre-trained U-Net architecture convolutional neural network model, which is trained by a welding workpiece dataset to output a semantic segmentation map of the welding trajectory; sequentially performing morphological closing budget, skeletonization processing and center line extraction on the semantic segmentation map output by the model to generate a welding trajectory meeting the preset accuracy requirement; sampling on the welding trajectory at a fixed interval or a dynamic interval based on curvature adaptivity to generate a discrete welding point sequence.

[0017] The application also provides a computer system, which comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the welding trajectory generation method based on mobile device image acquisition.

[0018] In summary, due to the adoption of the technical solution, the application has the following advantages: The application realizes the rapid generation of the welding trajectory of the industrial robot by fusing the image acquisition of the ordinary mobile device, the two-dimensional code calibration and the image pose solving technology, and effectively overcomes the inherent defects of the traditional method, such as the complicated manual teaching and the high equipment cost. The application does not need to rely on the professional teaching device or the structured light scanning system, significantly simplifies the welding trajectory planning process, greatly reduces the technical threshold and the implementation cost, and provides an efficient and reliable solution for the industrial welding field.

[0019] In actual application, the operator only needs to use a conventional smart phone for handheld shooting to complete the image acquisition and processing of the workpiece, avoids the dependence on the fixed shooting environment or the special tooling fixture, makes the deployment process more flexible and fast, and at the same time, the system can automatically complete the three-dimensional model reconstruction of the workpiece, the welding trajectory extraction and the coordinate system conversion, generate the welding instruction that can be directly executed by the welding robot, greatly reduces the manual intervention and the potential operation error, and improves the overall automation level and the task execution efficiency.

[0020] The application can seamlessly adapt to the welding requirements of workpieces of various sizes, shapes and process parameters, and is particularly suitable for the production scene of flexible, small batch and multi-variety. The low-cost and high-practicality characteristics of the application not only meet the actual needs of cost-sensitive enterprises, but also provide rapid response welding support for the industrial environment with frequent process adjustment, thereby significantly enhancing the convenience, economy and market competitiveness of the welding production. BRIEF DESCRIPTION OF DRAWINGS

[0021] The application is further described in detail by the following drawings, specifically including five drawings, as follows: Figure 1A schematic diagram of the overall flow of the welding trajectory generation method of the present application is shown; Figure 2 A schematic diagram of the multi-angle shooting of the workpiece image in the method of the present application is shown; Figure 3 A schematic diagram of the dense point cloud of the workpiece to be welded reconstructed in the method of the present application is shown; Figure 4 A schematic diagram of the coordinate conversion in the method of the present application is shown; Figure 5 A schematic diagram of the extraction of the welding trajectory in the method of the present application is shown. DETAILED DESCRIPTION

[0022] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0024] A welding trajectory generation method based on image acquisition of a mobile device, Fig. 1 shows the overall flow of the method, which can be referred to simultaneously, and the key steps of the method can be summarized as follows: S1, using a mobile device with a camera, acquire multiple images of a workpiece to be welded at different shooting angles; the workpiece to be welded is provided with a pose marker in advance; S2, based on the DUSt3R algorithm, through the registration relationship between the multiple images of the workpiece to be welded, take the first image of the workpiece to be welded as the reference image, calculate the camera pose of the remaining images relative to the reference image, and reconstruct the dense point cloud of the workpiece to be welded; S3, detect the pose marker in the reference image through an image recognition algorithm, determine the initial coordinate system of the workpiece to be welded according to the position and attitude of the pose marker in the reference image, and then establish the mapping relationship between the initial coordinate system and the base coordinate system of the welding robot workspace; S4, based on the dense point cloud of the workpiece to be welded, extract the welding trajectory therefrom and generate a discrete welding point sequence; determine the coordinates of each welding point in the reference image in the discrete welding point sequence; S5. According to the established mapping relationship, the discrete welding point sequence is mapped into the welding robot workspace to generate control instructions, so that the welding robot performs a welding operation on the workpiece to be welded.

[0025] The embodiment will introduce the details or preferred contents of the method in detail according to the above step sequence. First, in step S1, the pose marker is a two-dimensional code attached to the surface of the workpiece to be welded, which has a preset size, a high-contrast black and white pattern and a unique code identifier, which is used to provide a known geometric reference for calculating the registration relationship and determining the initial coordinate system.

[0026] In the embodiment, the mobile device is a handheld smart terminal, such as a smartphone or a tablet device. The operator holds the smart terminal and takes pictures at different preset distances and orientations around the workpiece to be welded at a preset pitch angle, thereby obtaining multiple images of the workpiece to be welded at different shooting angles. Please refer to the schematic diagram of Figure 2 The shooting process does not need to follow a preset path, fixed shooting position or repeated shooting sequence, and the shooting operation does not depend on professional auxiliary equipment, fixtures or auxiliary positioning devices. It does not need to use a tripod, a fixed support, a laser pointer or a special lighting system. It can directly use the natural light of the environment or the built-in flash of the smartphone to complete the acquisition of the corresponding image.

[0027] Enter step S2, which appropriately improves the DUSt3R algorithm to calculate the camera pose of the remaining images relative to the reference image. The ultimate goal is to reconstruct the dense point cloud as shown in Figure 3 The DUSt3R algorithm itself is a dense unsupervised motion recovery structure algorithm based on the Transformer architecture, which can recover the three-dimensional structure and camera pose of the scene from a monocular image sequence without calibration. It has the characteristics of not needing depth truth label and supporting dense reconstruction, so it is suitable for high-precision three-dimensional reconstruction tasks in various scenes.

[0028] In the embodiment, a lightweight feature extraction module is integrated in the front end of the Transformer encoder of the DUSt3R algorithm. The module uses the SuperPoint algorithm to detect feature points and generate descriptors for each workpiece image. The preset feature point detection threshold and descriptor dimension are set. The SuperPoint algorithm used here is a deep learning model for image key point detection and descriptor generation, which provides an end-to-end solution to automatically detect key points in images and generate corresponding descriptors. The algorithm itself is open source and can be improved and extended as a basic model, so it can be applied to the method process of the embodiment. Therefore, it will not be described here. In the embodiment, the feature point detection threshold is set to 0.015-0.025, and the descriptor dimension is 256.

[0029] Subsequently, the feature descriptors between multiple images of the workpieces to be welded are initially matched by a GMS algorithm to obtain matching point pairs; a fundamental matrix is calculated based on the matching point pairs, and an RANSAC algorithm is applied for outlier elimination; before the outlier elimination, the iteration number of the RANSAC algorithm and the inlier determination threshold are preset.

[0030] GMS (Grid-based Motion Statistics) is a computer vision algorithm specially designed for outlier elimination in feature matching; it divides an image into regular grids, and statistically analyzes the motion consistency of feature points in each grid, thereby quickly identifying and filtering out abnormal matching points, and significantly improving the accuracy and robustness of matching. In this embodiment, GMS is used to optimize the feature correlation process of the image sequence to achieve initial matching, and to provide reliable input data support for subsequent reconstruction tasks.

[0031] RANSAC (Random Sample Consensus) is a robust iterative model fitting algorithm, which estimates model parameters by randomly sampling data subsets, and filters inliers based on a consistency threshold to effectively exclude abnormal value interference. It is widely used in computer vision for geometric model estimation, such as homography matrix or fundamental matrix calculation, significantly improving the accuracy and reliability in noisy environments. In this embodiment, RANSAC is used in the outlier elimination process based on its data fitting characteristics, to ensure accurate geometric calibration, and the iteration number range is preset to 100-200 times, and the inlier determination threshold is preset to 1.5-2.0 pixels.

[0032] In this embodiment, the first image of the workpiece to be welded is taken as the reference image, and based on the matching point pairs after the elimination of outliers, the essential matrix of the remaining images relative to the reference image is solved by the five-point algorithm, and the rotation matrix and translation vector are decomposed to obtain the camera pose of the remaining images relative to the reference image.

[0033] Levenberg-Marquardt (LM) optimizer is an iterative algorithm widely used to solve nonlinear least squares problems, suitable for the optimization requirements in this embodiment, which finds a set of parameter values that minimizes the sum of squared errors between predicted and observed values; the iteration number range of the Bundle Adjustment (BA) optimization performed is 50-100 times.

[0034] In this embodiment, the image of each workpiece to be welded is input into a pre-trained depth estimation model for monocular depth prediction, and the corresponding depth map is output as the depth prediction result. The depth estimation model can use the MiDaS v3 model, which has the same input resolution as the original image. The output depth map is upsampled to the original size by bilinear interpolation, and the depth values are mapped to the effective working range of 0.1-10.0 meters through inverse depth normalization processing.

[0035] The depth prediction result is aligned with the relative value of each camera pose, and a unified 3D volume grid is constructed using a weighted TSDF fusion strategy. The TSDF truncation distance is set to 0.02-0.05 meters, and the voxel resolution is set to 2mm x 2mm x 2mm. The weight of each voxel is dynamically adjusted according to the angle between the image view angle and the normal vector.

[0036] Finally, the Marching Cubes algorithm is used to extract isosurfaces from the 3D volume grid, and the initial dense point cloud is generated. Then, the statistical outlier removal (SOR) filtering process is applied, with a preset neighborhood point number of 30 and a standard deviation multiple threshold range of 1.5-2.0. Finally, the dense point cloud of the workpiece to be welded with a noise suppression rate ≥85% is output, and the reconstruction process is completed.

[0037] The reconstructed dense point cloud is used to extract the welding trajectory in step S4, but before the welding trajectory is extracted and applied, the operation in step S3 is also required to establish the mapping of multiple coordinate system spaces and determine their unique and accurate mapping relationship. After the coordinate points with welding trajectories in the reference image are obtained, they can be directly used by the workspace coordinate system of the welding robot.

[0038] In step S3, the reference image plane coordinate system is denoted as o(l), the plane coordinate system of the pose marker in the reference image is denoted as o(p), and the welding robot workspace base coordinate system is denoted as o(w). By calculating the rotation relationship between o(l), o(p), and o(w), the corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p) and the corresponding positions of each coordinate point in the initial coordinate system o(p) in the welding robot workspace base coordinate system o(w) are obtained, and the mapping relationship is established.

[0039] In this embodiment, the transformation matrix T(p-w) between the initial coordinate system o(p) and the welding robot workspace base coordinate system o(w) is established in advance through a hand-eye calibration algorithm. For example, Figure 4As shown, the transformation matrix T(p-w) includes the translation parameter T_2 and the rotation parameter R_2 shown in the figure, based on which the initial coordinate system o(p) is directly mapped to the welding robot workspace base coordinate system o(w). The corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p) can also be realized by another transformation matrix, the parameters of which are the translation parameter T_1 and the rotation parameter R_1, and the principle is similar. Finally, the welding trajectory is reflected in the reference image plane coordinate system o(l) after extraction, and the coordinate points can be mapped to the welding robot workspace base coordinate system o(w). That is, the welding trajectory composed of each coordinate point in the reference image plane coordinate system o(l) forms a corresponding trajectory point path in the welding robot workspace base coordinate system o(w).

[0040] For the step S4 of extracting the welding trajectory of the dense point cloud and generating the discrete welding point sequence, the embodiment provides two preferred schemes, namely a geometric feature analysis method and a neural network algorithm method. The extraction effect can be referred to the schematic diagram of FIG. 6, wherein the highlighted red line at the junction of the vertical surface of the workpiece and the horizontal surface is the extracted welding trajectory, which is similar to the effect presented in an image. Figure 5

[0041] In the embodiment, the geometric feature analysis method is adopted to extract the welding trajectory and generate the discrete welding point sequence, which specifically includes: first, bilateral filtering processing is applied to the obtained dense point cloud data, which can effectively eliminate random noise interference in the point cloud, while ensuring that the geometric features of the edge of the weld area are not blurred; then, the curvature value and normal vector direction of each point in the point cloud are accurately calculated, and all candidate points that may be located in the weld position are systematically identified and screened according to the pre-set curvature threshold range and normal vector direction mutation angle standard; on this basis, the same RANSAC algorithm with strong robustness is used to perform linear or quadratic curve fitting operation on the screened weld candidate points, and abnormal point interference is excluded through iterative optimization, and finally a smooth and continuous welding trajectory that meets the actual welding requirements is generated.

[0042] ​In this embodiment, the neural network algorithm is adopted to extract the welding trajectory and generate the discrete welding point sequence, which specifically includes: first, the dense point cloud data is orthogonally projected along the optical axis direction of the reference image to convert it into a two-dimensional image with depth information, and the image is normalized to an input tensor of uniform size to adapt to the model requirements; then the normalized tensor is input into the U-Net architecture convolutional neural network model which is fully trained on the welding workpiece dataset in advance. The model automatically outputs a high-precision welding trajectory semantic segmentation map by learning a large number of samples, clearly distinguishing the weld area from the background; then the segmentation map is sequentially subjected to morphological closing operation to fill small holes and connect broken parts, skeletonization processing to extract single-pixel width centerline structure, and finally the centerline optimization is performed to generate a welding trajectory that meets the engineering precision requirements.

[0043] Regardless of the way the welding trajectory is extracted, sampling is performed on the welding trajectory at a fixed interval or a dynamic interval based on the curvature to generate a discrete welding point sequence. Then, according to the camera intrinsic matrix and the camera extrinsic parameter of the reference image, the discrete welding point sequence is mapped into the plane coordinate system of the reference image through the perspective projection model, i.e., the coordinates of each welding point in the reference image are determined. Adding that the coordinates can be mapped into the welding robot workspace coordinate system, the generation of the welding trajectory required by the robot is realized.

[0044] Finally, in step S5, the welding robot, such as a six-axis industrial robot, can execute the welding task point by point along the trajectory based on the path point control instruction corresponding to the welding trajectory; each welding point can also correspond to the welding pose of the welding robot, so that the welding pose is accurately transformed as the welding task progresses.

[0045] The above method process of the present embodiment can be applied in a corresponding computer system to make it a welding trajectory generation system based on mobile device image acquisition. The system includes a memory, a processor, and computer programs stored on the memory, and the processor executes the computer programs to implement the steps of the aforementioned welding trajectory generation method based on mobile device image acquisition. In the present embodiment, these computer programs / instructions can be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices that implement the functions specified in one step or multiple steps in the method.

Claims

1. A method for generating welding trajectories based on images acquired from mobile devices, characterized in that, Includes the following steps: S1. Using a mobile device equipped with a camera, acquire multiple images of the workpiece to be welded from different shooting angles; the workpiece to be welded has pose markers pre-set on it; S2. Based on the DUSt3R algorithm, the dense point cloud of the workpiece to be welded is reconstructed by using the registration relationship between multiple images of the workpiece to be welded as the reference image and calculating the camera pose of the remaining images relative to the reference image. S3. Detect the pose marker in the reference image using an image recognition algorithm, and determine the initial coordinate system of the workpiece to be welded based on the position and orientation of the pose marker in the reference image. Then, the mapping relationship between the initial coordinate system and the base coordinate system of the welding robot's workspace is established; S4. Based on the dense point cloud of the workpiece to be welded, extract the welding trajectory and generate a discrete welding point sequence; determine the coordinates of each welding point in the discrete welding point sequence in the reference image. S5. Based on the established mapping relationship, the discrete welding point sequence is mapped to the welding robot's workspace, and control commands are generated to enable the welding robot to perform welding operations on the workpiece to be welded.

2. The welding trajectory generation method according to claim 1, characterized in that: In step S1, the pose marker is a QR code attached to the surface of the workpiece to be welded. The QR code has a preset size, a high-contrast black and white pattern and a unique code identifier, which is used to provide a known geometric reference datum when calculating the registration relationship and determining the initial coordinate system.

3. The welding trajectory generation method according to claim 1, characterized in that: In step S1, the mobile device is a handheld smart terminal. The operator holds the smart terminal and takes pictures of the workpiece to be welded at multiple preset distances and positions with preset pitch angles, thereby obtaining multiple images of the workpiece to be welded from different shooting angles.

4. The welding trajectory generation method according to claim 1, characterized in that, In step S2, the DUSt3R algorithm is improved to calculate the camera pose of the remaining images relative to the reference image, specifically including: A lightweight feature extraction module is integrated into the front end of the Transformer encoder of the DUSt3R algorithm. This module uses the SuperPoint algorithm to perform feature point detection and descriptor generation for each image of the workpiece to be welded, with preset corresponding feature point detection thresholds and descriptor dimensions. The feature descriptors of multiple workpiece images to be welded are initially matched using the GMS algorithm to obtain matching point pairs; the fundamental matrix is ​​calculated based on the matching point pairs, and then the RANSAC algorithm is applied to remove outliers; before removing outliers, the number of iterations of the RANSAC algorithm and the threshold for determining inliers are preset. Using the first image of the workpiece to be welded as the reference image, based on the matching point pairs after removing outliers, the essential matrix of the remaining images relative to the reference image is solved by the five-point algorithm, and the rotation matrix and translation vector are decomposed. The rotation matrix is ​​represented by Lie algebra and local BA optimization is performed by the LM optimizer, thereby obtaining the camera pose of the remaining images relative to the reference image.

5. The welding trajectory generation method according to claim 4, characterized in that, The reconstruction of the dense point cloud of the workpiece to be welded specifically includes: inputting each image of the workpiece to be welded into a pre-trained depth estimation model for monocular depth prediction, outputting the corresponding depth map as the depth prediction result; aligning the depth prediction result with the relative pose values ​​of each camera, and constructing a unified 3D volumetric mesh using a weighted TSDF fusion strategy; then using the Marching Cubes algorithm to extract isosurfaces from the 3D volumetric mesh to generate an initial point cloud, applying statistical outlier removal filtering, and after setting the thresholds for the number of neighborhood points and the standard deviation multiple, outputting the final dense point cloud of the workpiece to be welded, thus completing the reconstruction.

6. The welding trajectory generation method according to claim 1, characterized in that: In step S3, the reference image plane coordinate system is denoted as o(l), the plane coordinate system of the pose marker in the reference image is used as the initial coordinate system of the workpiece to be welded and denoted as o(p), and the base coordinate system of the welding robot workspace is denoted as o(w). By calculating the rotation relationship between o(l), o(p) and o(w), the corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p) and the corresponding positions of each coordinate point in the initial coordinate system o(p) in the welding robot workspace base coordinate system o(w) are obtained, and then a mapping relationship is established.

7. The welding trajectory generation method according to claim 6, characterized in that: In step S3, a transformation matrix T(pw) between the initial coordinate system o(p) and the welding robot workspace base coordinate system o(w) is established in advance using a hand-eye calibration algorithm. Based on this transformation matrix T(pw), the initial coordinate system o(p) is directly mapped to the welding robot workspace base coordinate system o(w). Subsequently, based on the corresponding positions of each coordinate point in the reference image plane coordinate system o(l) in the initial coordinate system o(p), each coordinate point in the reference image plane coordinate system o(l) is also mapped to the welding robot workspace base coordinate system o(w). The welding trajectory formed by each coordinate point in the reference image plane coordinate system o(l) is the corresponding trajectory point path in the welding robot workspace base coordinate system o(w).

8. The welding trajectory generation method according to claim 1, characterized in that, In step S4, a geometric feature analysis method is used to extract the welding trajectory and generate a discrete welding point sequence. Specifically, this includes: performing bilateral filtering on the dense point cloud to suppress noise and retain edge features; calculating the curvature value and normal vector of each point, and selecting candidate weld points based on a preset curvature threshold and normal vector abrupt change angle; performing linear or quadratic curve fitting on the candidate weld points using the RANSAC algorithm to generate a smooth welding trajectory; and sampling on the welding trajectory at a fixed interval or a dynamic interval based on curvature adaptation to generate a discrete welding point sequence.

9. The welding trajectory generation method according to claim 1, characterized in that, In step S4, a neural network algorithm is used to extract the welding trajectory and generate a discrete welding point sequence. Specifically, this includes: projecting the dense point cloud along the optical axis of the reference image into a depth image and normalizing it into an input tensor with a fixed resolution; inputting it into a pre-trained U-Net architecture convolutional neural network model, which is trained on a welding workpiece dataset to output a semantic segmentation map of the welding trajectory; performing morphological closure estimation, skeletonization, and centerline extraction on the semantic segmentation map output by the model in sequence to generate a welding trajectory that meets the preset accuracy requirements; and sampling on the welding trajectory at a fixed interval or a dynamic interval based on curvature adaptation to generate a discrete welding point sequence.

10. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the welding trajectory generation method based on mobile device image acquisition as described in claim 1.

Images