Welding seam area positioning method and device based on multi-camera vision and deep learning

Abstract

The invention discloses a welding seam area positioning method and device based on multi-camera vision and deep learning, and belongs to the technical field of robot welding, and the method comprises the steps: S1, carrying out the internal reference calibration of a plurality of cameras, and constructing a workbench plane model under a camera coordinate system based on a calibration result; establishing a conversion relation between a camera coordinate system and an orbital coordinate system through hand-eye calibration and orbital direction unit vector calibration; s2, concurrently collecting a workpiece stage image, extracting a workpiece mask by adopting an instance segmentation algorithm, and converting the mask from a pixel coordinate system to an orbital coordinate system based on the parameters in the step S1; s3, identifying a welding seam area of the mask by adopting a target detection algorithm, and carrying out optimal coverage photographing planning on a single workpiece to obtain a photographing point coordinate; s4, the actual movement distance of the robot on the track is calculated, and positioning and guiding of the welding seam area are completed; and relatively high positioning efficiency and precision can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of robotic welding technology, and particularly relates to a weld seam area localization method based on multi-camera vision and deep learning. Background Technology

[0002] In industrial automated welding scenarios, the identification and localization of weld seams are prerequisites for achieving high-quality welding operations. Existing weld seam localization technologies are mostly based on monocular camera vision systems, which acquire images of the weldment and use image processing or deep learning algorithms to identify the weld seam area. However, monocular vision systems suffer from problems in practical applications, such as large blind spots, severe occlusion, and poor scene adaptability. Especially when dealing with large and complex welded components, they are prone to incomplete localization and insufficient accuracy.

[0003] Some existing solutions attempt to use a camera mounted on the end effector of a robotic arm to collect data from multiple angles in conjunction with the motion trajectory. However, this approach has the following problems: First, the data collection efficiency is low, requiring multiple movements to take pictures; second, due to the limited space accessible by the robotic arm, some areas may still have blind spots; in addition, since the camera position changes continuously during movement, frequent visual calibration and coordinate system updates are required during long-term operation, increasing system complexity and error accumulation.

[0004] In summary, existing technologies suffer from limitations such as limited data acquisition range, large positioning errors, and poor algorithm robustness, making it difficult to meet the actual needs of locating weld seam areas in complex industrial scenarios.

[0005] Therefore, a new technical solution is needed to solve the above problems. Summary of the Invention

[0006] Purpose of the invention: To solve the above problems, the purpose of this invention is to provide a weld seam area localization method based on multi-camera vision and deep learning, so as to solve the problems of limited acquisition range and large cumulative error in the existing technology.

[0007] The present invention also provides a positioning device and a computer device for performing the weld area positioning method based on multi-camera vision and deep learning.

[0008] Technical Solution: The weld seam region localization method based on multi-camera vision and deep learning disclosed in this invention can adopt the following technical solution:

[0009] A method for locating weld seam areas based on multi-camera vision and deep learning, specifically including the following steps:

[0010] S1. Perform intrinsic parameter calibration on the camera to determine the camera's intrinsic parameter matrix and distortion coefficients;

[0011] Construct the plane equations of the worktable in the camera coordinate system to obtain the unit normal vector of the worktable plane. and the constant term of the plane equation ;

[0012] Hand-eye calibration is performed on the camera and robot to obtain the hand-eye transformation matrix;

[0013] Calculate the unit vector of the orbital direction in the robot's base coordinate system. Use this unit vector to linearly superimpose and correct the translation vector in the aforementioned hand-eye transformation matrix. Use the corrected hand-eye transformation matrix as the transformation matrix from the camera coordinate system to the orbital coordinate system.

[0014] Based on the camera's intrinsic parameter matrix and distortion coefficients, the table plane equation parameters, and the transformation matrix from the camera coordinate system to the orbit coordinate system, the coordinate system mapping relationship between the pixel coordinate system and the orbit coordinate system is finally obtained.

[0015] S2. Acquire worktable images from multiple cameras and use instance segmentation model to extract mask region features of the welding workpiece in pixel coordinate system;

[0016] Using the coordinate system mapping relationship obtained in step S1, the above mask area features are transformed from the pixel coordinate system to the orbit coordinate system, and the scaling factor and yaw angle are calculated to correct the mask area features.

[0017] A depth-first search strategy was used to perform IoU threshold filtering on the corrected mask region features to construct a complete weld mask image.

[0018] S3. Use the target detection model to extract the weld seam region features from the complete weld mask image in step S2; and perform optimal coverage photography planning for multiple weld seam region features of a single weld, finally outputting the coordinates of the shooting point;

[0019] S4. Based on the unit vector of the track direction in step S1, the coordinates of the shooting point in step S3 are projected and corrected to calculate the actual moving distance of the track and the robot, so that the robot and the robot end reach the shooting area of ​​each weld seam and complete the positioning guidance.

[0020] Furthermore, the following coordinate system is defined: pixel coordinate system The origin is the pixel at the top left corner of the intersection of the horizontal and vertical pixel edges of the image captured by the camera. , The axis is set along the direction of increasing pixel column index. The axis is set along the direction of increasing pixel row index; camera coordinate system. With the camera's optical center as the origin ,in The plane is parallel to the camera's imaging plane. The axis is set along the camera's optical axis and points towards the subject; calibration plate coordinate system The origin is the lower left corner of the intersection of the horizontal and vertical edges of the checkerboard on the calibration board. , The shaft is set along the direction of the lateral edge line. The shaft is positioned along the longitudinal edge line. The axes are set along the plane of the vertical calibration plate, and the mapping relationship between this coordinate system and the camera coordinate system is determined by the camera extrinsic parameter matrix; robot base coordinate system. With the geometric center of the robot base as the origin , shaft and The axes are set along two orthogonal directions within the plane of the robot base. The axis is set along a direction perpendicular to the plane; orbital coordinate system The robot is reset to the limit position of the track switch using the track. The robot's base coordinate system at this position is used as the track coordinate system, and the three coordinate axes of this coordinate system are... They are parallel to the corresponding coordinate axes of the robot's base coordinate system.

[0021] Furthermore, the intrinsic parameter calibration of the camera described in step S1 includes: acquiring an image of the calibration board in the camera's field of view, and using a pinhole camera model to establish a mapping relationship between spatial three-dimensional points and pixels, satisfying the following intrinsic and extrinsic parameter projection relationships:

[0022] ,

[0023] ,

[0024] In the formula: K is the intrinsic parameter matrix of the camera; This is the normalized coordinate scale factor; Image distance; Each represents a pixel in and Physical scale in direction; These represent the coordinates of the projected position of the camera's optical center on the pixel plane in the pixel coordinate system; express shaft and The angle between axes; R is the rotation matrix of the camera extrinsic matrix; t is the translation vector of the camera extrinsic matrix; The three-dimensional coordinates of the corner points of the chessboard grid on the calibration plate coordinate system; These are the two-dimensional coordinates of the point in the pixel coordinate system.

[0025] By acquiring images of the calibration board in multiple different poses in space, multiple sets of three-dimensional spatial points in the calibration board coordinate system and two-dimensional imaging points of the spatial points in the pixel coordinate system are obtained. The two are used as observation data and substituted into the above projection relationship to solve for the initial intrinsic parameter matrix and initial extrinsic parameter matrix of the camera, which are then used for subsequent parameter optimization.

[0026] To address camera lens distortion, the following radial and tangential distortion models are used to distort the normalized coordinates in the camera coordinate system, and then the distorted coordinates are transformed to the pixel coordinate system:

[0027] ,

[0028] ,

[0029] ,

[0030] ,

[0031] In the formula: To locate the 3D point coordinates in the camera coordinate system of the camera, r and r are the normalized coordinates in the camera coordinate system before distortion and the distance of that coordinate to the camera center, respectively. Radial distortion coefficient; The tangential distortion coefficient is... These are the normalized coordinates in the distorted camera coordinate system. The two-dimensional predicted coordinates in the pixel coordinate system are obtained by substituting the coordinates in the calibration plate coordinate system into the pinhole camera model, radial distortion model, and tangential distortion model.

[0032] Using the aforementioned observation data, initial intrinsic parameter matrix, and initial extrinsic parameter matrix, a nonlinear optimization method based on minimizing reprojection error is employed to jointly optimize the camera's intrinsic and extrinsic parameters, and the radial and tangential distortion coefficients are solved. The objective function is as follows:

[0033] ,

[0034] In the formula, M is the number of calibration board images acquired; To determine the number of corner points of the plate in the m-th image; These are the actual pixel coordinates of the nth corner point in the mth image; These are the predicted pixel coordinates obtained by projecting from the model parameters; The distortion coefficient vector ; These are the rotation matrix and translation vector corresponding to each calibration board image.

[0035] Furthermore, the construction of the plane equation of the worktable in the camera coordinate system in step S1 includes:

[0036] The calibration board is placed on the worktable, and images of the calibration board in various poses are captured within the fixed field of view of the camera. For each calibration board image, the correspondence between the 3D calibration board points and the pixels of the calibration board image is constructed, and the extrinsic parameter matrix is ​​optimized by minimizing the reprojection error. The objective function is:

[0037] ,

[0038] In the formula, N is the number of calibration board corner points in each calibration board image; Let n be the coordinates of the nth corner point of the calibration board in the pixel coordinate system; Let be the three-dimensional coordinate point of the nth corner point of the calibration plate in the calibration plate coordinate system; This is the standard perspective projection function;

[0039] The optimization process is based on the Perspective-n-Point algorithm, ultimately obtaining the extrinsic parameter matrix of the calibration board coordinate system to the camera coordinate system in each image when the calibration board is located on the worktable. ;

[0040] Based on the above extrinsic matrix The corner points of the calibration plate in the entire calibration plate coordinate system are transformed from 3D world coordinates to camera coordinates, resulting in a set of 3D points in the camera coordinate system. Calculate the mean vector of a three-dimensional point set:

[0041] ,

[0042] Constructing a centralized dataset Calculate the three-dimensional covariance matrix of the point set. :

[0043] ,

[0044] Eigenvalue decomposition of the covariance matrix yields three real eigenvalues. and their corresponding eigenvectors Among them, the smallest eigenvalue Corresponding feature vector The direction that minimizes the variance of the point set is the normal vector of the best-fit plane for the data points. Normalizing this vector yields the unit normal vector of the worktable plane in the camera coordinate system. Where A, B, and C are respectively , and Components along the axial direction;

[0045] Based on the above best-fit plane normal vector Mean of the point set Calculate the constant term of the plane equation:

[0046] ,

[0047] Finally, the parameters and equations of the worktable plane in the camera coordinate system are obtained:

[0048] ,

[0049] In the formula, the parameters of the workbench plane equation include the plane unit normal vector. and the constant term D of the plane equation;

[0050] The above steps are described with a single camera as the object. Each positioning camera performs the steps respectively, and finally obtains the plane equation of the worktable plane in the coordinate system of each camera.

[0051] Furthermore, the step S1 of performing hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix includes:

[0052] First, the trajectory is reverted to its origin to unify the initial position of the trajectory coordinate system. Then, the calibration board is mounted on the robot's end effector, and the robot is controlled to move along the trajectory into the fixed field of view of the camera. Multiple sets of calibration board images are acquired under different robot poses. At this position, hand-eye calibration in an eye-in-hand mode is performed to obtain the camera coordinate system. With the robot base coordinate system Hand-eye transformation matrix It satisfies the following transformation relationship:

[0053] ,

[0054] In the formula, The transformation from the calibration board coordinate system to the camera coordinate system in the m-th calibration board image when the calibration board is located at the robot end effector is obtained using the Perspective-n-Point algorithm. Let be the transformation matrix from the robot base coordinate system to the end effector in the m-th sub-calibration plate image;

[0055] After performing hand-eye calibration, the hand-eye transformation matrix is ​​obtained:

[0056] ,

[0057] In the formula: This is the rotation matrix for transforming the robot's base coordinate system to the camera coordinate system; This is the translation vector used to transform the robot's base coordinate system to the camera coordinate system.

[0058] Furthermore, the step S1, which involves calculating the unit vector of the orbital direction in the robot's base coordinate system, and using this unit vector to linearly superimpose and correct the translation vector in the hand-eye transformation matrix, and then using the corrected hand-eye transformation matrix as the transformation matrix from the camera coordinate system to the orbital coordinate system, includes:

[0059] Record the current orbital position Keeping the robot's end-effector posture and camera field of view unchanged, the control track drives the robot to move continuously in the positive direction with a fixed step distance, and collects several images of the calibration board fixed to the robot's end-effector.

[0060] Based on each calibration board image, the intrinsic parameter matrix K of the corresponding camera and the distortion coefficients are used. The correspondence between the three-dimensional calibration board points and their image pixels, combined with the Perspective-n-Point algorithm, is used to solve for the extrinsic parameter matrix of the calibration board coordinate system in the camera coordinate system. ;

[0061] Based on the above, the hand-eye transformation matrix is ​​obtained. and extrinsic parameter matrix The corner coordinates in the calibration plate coordinate system Coordinates transformed to the robot base coordinate system :

[0062] ,

[0063] Let the three-dimensional coordinates of the nth corner point of the calibration board in the camera base coordinate system be (the coordinates of the nth corner point of the calibration board in the mth image of the calibration board are given by the given coordinates of the nth corner point of the calibration board in the camera base coordinate system). Using the first calibration plate image as the reference image, the point-by-point difference between corresponding corner points in all calibration plate images and the reference image is calculated to obtain the relative translation vector in the robot base coordinate system. Furthermore, the average of all corner point difference vectors is used to fit the total displacement vector along the trajectory.

[0064] ,

[0065] In the formula: M represents the number of calibration board images acquired; N is the number of corner points of the calibration board in each calibration board image;

[0066] Normalize the result to obtain the unit vector of the orbital direction:

[0067] ,

[0068] In the formula, For unit vectors in , and The sum of the squares of the components along the three coordinate axes is 1;

[0069] Based on the current orbital position obtained above Translation vector in the hand-eye transformation matrix A linear superposition correction is performed, and the corrected hand-eye transformation matrix is ​​used as the transformation matrix from the camera coordinate system to the orbit coordinate system:

[0070] ,

[0071] In the formula, This is the rotation matrix from the camera coordinate system to the orbit coordinate system; This is the translation vector from the camera coordinate system to the orbit coordinate system;

[0072] Finally, the pixel coordinate system is obtained. With orbital coordinate system Coordinate system mapping relationship:

[0073]

[0074]

[0075]

[0076] In the formula, To correct the distortion of the original pixel coordinates, the distortion-free normalized three-dimensional coordinates of the camera coordinate system are obtained through iteration. These are the three-dimensional coordinates in the orbital coordinate system obtained from pixel coordinates through a mapping relationship; It is the depth coefficient along the camera optical axis, used to map normalized camera coordinates to three-dimensional points on the worktable plane in the camera coordinate system;

[0077] Repeat step S1 to construct the mapping relationship between the pixel coordinate system and the camera coordinate system of all cameras.

[0078] Furthermore, in step S2, the acquisition of welding workpiece images from multiple cameras and the extraction of mask region features of the welding workpiece in pixel coordinates using an instance segmentation model include:

[0079] The workbench images captured by multiple fixed cameras are used as the basic data. The welding workpiece areas in the images are masked and labeled to form a training sample set.

[0080] The training sample set is used to train an instance segmentation model based on the YOLOv11-Seg architecture. During the training process, the detection and segmentation accuracy of the model is improved by minimizing the bounding box error and the mask loss function, thereby generating a deep learning-based weldment segmentation detector.

[0081] Multiple welding workpieces are placed on the worktable, and images of the worktable are acquired in parallel from multiple cameras. The features of the mask region of the welding workpiece in the pixel coordinate system are obtained by inference through the welding workpiece segmentation detector.

[0082] Furthermore, step S2, which involves transforming the mask region features to the orbital coordinate system using the coordinate system mapping relationship obtained in step S1, and calculating the scaling factor and yaw angle to correct the mask region features, includes:

[0083] Based on the features of the welding workpiece mask region obtained from instance segmentation, the intersection of the center point of the mask region and the upper left corner of the horizontal and vertical edges of the mask region is extracted as the dimension positioning point. The pixel coordinates of the dimension positioning point are denoted as follows: ;

[0084] Using the obtained corresponding camera intrinsic parameter matrix K and distortion coefficients Convert pixel coordinates to camera-normalized coordinates, i.e., unit vectors in the depth direction:

[0085]

[0086] In the formula Normalized coordinates in the camera coordinate system after distortion correction ;

[0087] Combined with the parameters of the worktable plane equation in the camera coordinate system Based on the formula for calculating the intersection of a ray and a plane, the three-dimensional spatial coordinates of the positioning point of this dimension on the worktable plane in the camera coordinate system are obtained as follows:

[0088]

[0089] The three-dimensional coordinates of the above-mentioned positioning points are transformed from the camera coordinate system to the orbital coordinate system using the transformation matrix. Convert to orbital coordinates:

[0090]

[0091] In the formula, Let be the three-dimensional coordinates of the dimension positioning point in the orbital coordinate system, where the center point of the mask region in the orbital coordinate system is denoted as . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is ; Let be the three-dimensional coordinates of the size positioning point in the camera coordinate system, where the center point of the mask region in the camera coordinate system is set to . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is Calculate the mask scaling factor based on the above coordinate points:

[0092]

[0093] The weld workpiece mask region features obtained from the above examples are scaled according to the mask scaling factor, and then... rotation matrix in Extract the yaw angle as the coordinate system around the orbit. Rotation angle of the shaft The calculation formula is as follows:

[0094] ,

[0095] ,

[0096] Constructing an orbital coordinate system Rotation matrix of the axis:

[0097] ,

[0098] The mask region features are rotated to complete the correction of the mask region features.

[0099] Step S2, which involves using a depth-first search strategy to perform IoU threshold filtering on the features of the corrected mask region to construct a complete weldment mask image, includes:

[0100] Let there be any two mask regions Crossover ratio:

[0101] ,

[0102] If satisfied Then the two mask areas They are considered as the same weldment area and merged to form a mask area. ;in The set overlap threshold;

[0103] Based on the above Determine the set of mask regions to construct This set will be used to merge the mask region set of the complete welding mask image;

[0104] A depth-first search algorithm is used to traverse the set S of mask region pairs, identify the connected components in all mask region pairs, establish a chain structure of mask regions, and set it as a connected subgraph. And for the mask region in each connected subgraph Perform a merge process; where J is the sequence number of the connected subgraph corresponding to the workpiece, which also represents the workpiece sequence number; j is the sequence number of the mask region in the connected subgraph;

[0105] For each connected subgraph Record each mask area The corresponding area rectangle positioning point is The set of two-dimensional bounding box positioning points for the mask region in the connected subgraph is: Then the joint enclosing rectangle of the connected subgraph is located at the following point. Defined as:

[0106] ,

[0107] In the formula, This represents the total number of mask regions contained in the connected subgraph; This represents the location point of the bounding rectangle of the k-th mask region;

[0108] Perform mask fusion on the connected subgraph to construct connected regions. That is, a complete welding workpiece mask; the complete welding workpiece is discretized in the orbital coordinate system to construct a set of discrete points in space. And calculate the mask The coordinates of the centroid are used as an estimate of the spatial position of the corresponding weldment.

[0109] ,

[0110] ,

[0111] In the formula, where This represents the number of points in a spatial discrete point set. This represents the geometric coordinates of a discrete point in space within the orbital coordinate system. The coordinates of the centroid of the complete weldment mask in the orbital coordinate system; This represents each of the following in the set F of the two-dimensional bounding box positioning points: of Coordinate values; The center point of the complete weldment mask in the orbital coordinate system Axis coordinates.

[0112] Furthermore, in step S3, the target detection model is used to extract weld region features from the complete weld mask image in step S2, including:

[0113] Provides a dataset of welded workpiece images, with annotations of the weld area on the workpiece images;

[0114] The labeled workpiece image dataset was used to train a target detection model based on the YOLOv11 architecture. During training, the YOLOv11 network optimized the model parameters by minimizing the bounding box regression error and the class recognition error, and finally generated a weld area target detector based on deep learning.

[0115] The complete welding workpiece mask generated in step S2 and constructed according to the orbital coordinate system scale is used as the image input, and then the positioning points are used to... Within the defined bounded rectangular area, with Origin , shaft and Parallel and in the same direction, shaft and In reverse order and opposite direction, establish the workpiece region coordinate system; after reasoning about the joint enclosing rectangular region using the aforementioned weld region target detector, return the weld region rectangular frame. The center point of the rectangular frame of the weld area is:

[0116] ,

[0117] In the formula: The coordinates of the center of the weld area frame in the workpiece area coordinate system;

[0118] Based on the three-dimensional center coordinates of the complete welded workpiece mask image in step S6 The center coordinates of the weld area frame in the orbital coordinate system are:

[0119] ,

[0120] In the formula, The coordinates of the three-dimensional center of the weld area boundary box;

[0121] Obtain bounding box information ;in , ;

[0122] The optimal coverage photography plan for multiple weld seam regions of a single weldment, and the final output of the shooting point coordinates include:

[0123] Let e ​​be the weld region index of a single workpiece. Based on the obtained weld region bounding box information, let the weld region be... In the formula, Define the coordinates of the 2D center point of the region bounding box; define the camera's shooting area in the orbital coordinate system as... In the formula The coordinates of the center point of the bounding box of the shooting area. The width and height of the fixed shooting area;

[0124] Calculate the weld area Let the set of points be the center point of the region and the positions of its four corner points. for:

[0125] ,

[0126] Will Each point in the diagram serves as the coordinate of the center point of the bounding box of the shooting range, and several shooting areas are formed around these center points. Calculate each The number of weld areas included is selected based on the number of areas covered. In As the covered weld area The center point of the photo;

[0127] Repeat this process until all uncovered weld areas are completely covered;

[0128] set up Let the set of camera shooting areas obtained from the above process be an index for the camera shooting area. ,set up The shooting area in the image is an unmerged area. and Let the merged shooting area be any two camera shooting areas in the above set of shooting areas:

[0129] ,

[0130] If the merging conditions are met:

[0131] ,

[0132] The areas captured by the two cameras are then merged into one, and this merged area is then placed back into the original area. In the process of merging photographic areas, a breadth-first search strategy is used to select the set of photographic areas to be merged. The shooting areas in the image are searched and merged.

[0133] The final output is the set of photographic points for a single workpiece in the orbital coordinate system:

[0134] ,

[0135] In the formula, This is the preset photo height.

[0136] Furthermore, in step S4, the coordinates of the shooting points in step S3 are projected and corrected to calculate the actual distance the robot travels along the track, enabling it to reach the shooting areas of each weld seam and complete the positioning guidance, including:

[0137] Based on the unit vector of the orbital direction determined in step S1 The set of shooting coordinate points in the orbital coordinate system obtained in step S3 After projection correction, the actual motion data is:

[0138]

[0139]

[0140]

[0141] In the formula: This represents the projection distance of the set of captured coordinate points onto the orbital coordinate system along the orbital direction vector; This refers to the corresponding point in the robot's base coordinate system after the robot has moved to the aforementioned projection distance. Axial direction and Distance moved along the axis;

[0142] Repeat steps S3 and S4 to obtain the actual motion data of each weld area in all workpieces.

[0143] Beneficial effects: The weld seam area positioning method provided by this invention converts images acquired by multiple cameras into positioning data of the weld seam area through strategies such as image acquisition, workpiece segmentation, coordinate system transformation, weld seam detection, photo planning, and data projection. It solves the problems of limited acquisition range and large cumulative error in the prior art, improves positioning efficiency and accuracy, and can cope with problems such as weldment occlusion and inconsistent multi-view imaging of positioning cameras.

[0144] In a further improvement, as described in step S2, different types of mask annotations are applied to the welding workpiece area in the image to identify different workpiece types. As described in step S3, different types of annotations are applied to welds with different field characteristics to identify different weld types.

[0145] The positioning device for the weld seam region positioning method based on multi-camera vision and deep learning provided by this invention adopts the following technical solution:

[0146] A positioning device for executing the weld area positioning method based on multi-camera vision and deep learning, comprising:

[0147] Calibration and Transformation Correction Module: Used to calibrate the camera's intrinsic parameters, determine the camera's intrinsic parameter matrix and distortion coefficients; construct the planar equation of the worktable in the camera coordinate system; perform hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix; calculate the unit vector of the orbital direction in the robot's base coordinate system, correct the translation vector of the hand-eye transformation matrix, and obtain the transformation matrix from each camera coordinate system to the orbital coordinate system; for the input pixel coordinates, perform pixel coordinate system to camera coordinate system transformation and camera coordinate system to orbital coordinate system transformation using the above parameters.

[0148] Multi-camera image acquisition module: includes multiple camera units and welding worktable; the cameras are mounted on a gantry-type overhead frame fixed to the ground to acquire image data of the welded workpiece and transmit it to the data processing and control unit;

[0149] Image feature extraction module: used to perform instance segmentation on the workbench image captured by the camera, obtain the mask area features of the welding workpiece in pixel coordinate system; perform target detection on the complete weld mask image, and extract the weld area features;

[0150] Welding mask correction module: The positioning points of the extracted welding mask area features are transformed from the pixel coordinate system to the orbit coordinate system. The difference of the welding mask area features in the transformation is calculated, the scaling factor of the mask area features is calculated, and the yaw angle is solved according to the rotation matrix, so as to perform spatial correction on the welding mask area features.

[0151] Mask image construction module: used to construct a complete weldment mask image based on the corrected mask region features. It uses a depth-first search strategy to perform IoU threshold filtering on the above region features. If the IoU is greater than the threshold, it is regarded as the same weldment region and connected components are merged to construct a complete weldment mask image.

[0152] Weld seam area photography planning module: For each uncovered area, calculate the center point of the uncovered area and the positions of the four corner points of the uncovered area as candidate photography centers; calculate the number of weld seam areas covered by each candidate photography center, and select the candidate photography center with the most covered areas as the photography center point of this area, and remove the covered areas from the index of uncovered areas; use a breadth-first search strategy to merge and judge the above photography areas, and finally output the coordinates of the shooting point in the orbital coordinate system;

[0153] Motion distance correction module: Performs projection correction based on the unit vector of the track direction and the obtained set of shooting point coordinates, calculates the projection ratio between the principal axis of track motion and the unit vector of the track direction, and then converts the shooting point coordinates into track motion data and robot pose data for output;

[0154] The track robot control module includes a guide rail, a motor control cabinet, servo motors, a robot control cabinet, and the robot body. The guide rail, motor control cabinet, and servo motors form the track, which has an origin return function to keep the spatial relative relationship between the track coordinate system and each camera coordinate system constant. Based on the target pose information output by the computer device, the track drives the robot body to move and controls the posture and position adjustment of the robot body and its end effector, thereby achieving precise positioning and guidance of the weld seam area.

[0155] Beneficial effects: The positioning device includes a computer program module that executes the above-mentioned weld area positioning method and the track robot control hardware used, which can apply the above-mentioned weld area positioning method to the welding robot hardware system and realize accurate positioning of the weld area obtained by welding in the welding robot system.

[0156] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the weld area positioning method. Attached Figure Description

[0157] Figure 1 This is a schematic diagram of the hardware interaction used in the weld seam area localization method based on multi-camera vision and deep learning of this invention.

[0158] Figure 2 This is a schematic diagram of a workpiece image for the weld area localization method based on multi-camera vision and deep learning according to the present invention.

[0159] Figure 3 This is a schematic diagram of the external coordinate system of the weld area localization method based on multi-camera vision and deep learning of the present invention;

[0160] Figure 4 This is a schematic diagram of calibration data acquisition for the weld area localization method based on multi-camera vision and deep learning of the present invention;

[0161] Figure 5 This is a flowchart of the weld area detection process of the weld area localization method based on multi-camera vision and deep learning of the present invention. Detailed Implementation

[0162] Please combine Figure 1 As shown, the hardware foundation of the weld seam area localization method based on multi-camera vision and deep learning provided by this invention is as follows: a multi-camera vision system and a track-mounted mobile robot work together, wherein the multi-camera vision system is mounted on a gantry-type elevated structure fixed to the ground; the track-mounted mobile robot is mounted on a fixed guide rail and has an origin return function to ensure that the spatial relative relationship between the track coordinate system and the coordinate systems of each camera remains constant; combined with Figure 2 As shown, pixel coordinate system The origin is the pixel at the top left corner of the intersection of the horizontal and vertical pixel edges of the image captured by the camera. , The axis is set along the direction of increasing pixel column index. The axis is set along the direction of increasing pixel row index; camera coordinate system. With the camera's optical center as the origin ,in The plane is parallel to the camera's imaging plane. The axis is set along the camera's optical axis and points towards the subject. The mapping relationship between the camera coordinate system and the pixel coordinate system is determined through intrinsic parameter calibration; calibration plate coordinate system. The origin is the lower left corner of the intersection of the horizontal and vertical edges of the chessboard. , The shaft is set along the direction of the lateral edge line. The shaft is positioned along the longitudinal edge line. The axes are set along the plane of the vertical calibration plate, and the mapping relationship between this coordinate system and the camera coordinate system is determined by the camera extrinsic parameter matrix; robot base coordinate system. With the geometric center of the robot base as the origin , shaft and The axes are set along two orthogonal directions within the plane of the robot base. The axis is set along a direction perpendicular to this plane, and the mapping relationship between this coordinate system and the camera coordinate system is determined through hand-eye calibration; orbital coordinate system The robot is reset to the limit position of the track switch using the track. The robot's base coordinate system at this position is used as the track coordinate system, and the three coordinate axes of this coordinate system are... Each axis is parallel to the corresponding coordinate axis of the robot base coordinate system. The mapping relationship between this coordinate system and the robot base coordinate system is determined by calculating the unit vector of the track direction.

[0163] Please combine Figure 4 and Figure 5 As shown, based on the above hardware, the weld seam region localization method based on multi-camera vision and deep learning includes the following steps:

[0164] S1. Perform intrinsic parameter calibration on the camera to determine its intrinsic parameter matrix and distortion coefficients; construct the planar equations of the worktable in the camera coordinate system; perform hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix; calculate the unit vector of the orbital direction in the robot's base coordinate system, correct the translation vector of the hand-eye transformation matrix, and obtain the transformation matrices from each camera coordinate system to the orbital coordinate system; combine with... Figure 4 As shown, component 1 is a positioning camera, component 2 is a robot, component 3 is a calibration plate, component 4 is a welding workbench, and component 5 is a moving track;

[0165] Specifically, step S1 includes the following steps:

[0166] S1-1, The camera intrinsic parameter calibration includes: combining Figure 4The diagram shown illustrates the acquisition of camera calibration and hand-eye calibration data. Calibration plate 3 is fixed to the end effector of robot 2. Multiple sets of calibration plate images are acquired by adjusting the robot to different poses within the field of view of camera 1. Using a pinhole camera model, a mapping relationship between spatial 3D points and pixels is established, satisfying the following intrinsic and extrinsic parameter projection relationships:

[0167] ,

[0168] ,

[0169] In the formula: K is the intrinsic parameter matrix of the camera; f is the normalized coordinate scale factor; f is the image distance; Each represents a pixel in and Physical scale in direction; These represent the coordinates of the projected position of the camera's optical center on the pixel plane in the pixel coordinate system; express shaft and The angle between axes; R is the rotation matrix of the camera extrinsic matrix; t is the translation vector of the camera extrinsic matrix; The three-dimensional coordinates of the corner points of the chessboard grid on the calibration plate coordinate system; These are the two-dimensional coordinates of the point in the pixel coordinate system.

[0170] By acquiring images of the calibration board in multiple different poses in space, multiple sets of three-dimensional spatial points in the calibration board coordinate system and two-dimensional imaging points of the spatial points in the pixel coordinate system are obtained. The two are used as observation data and substituted into the above projection relationship to solve for the initial intrinsic parameter matrix and initial extrinsic parameter matrix of the camera, which are then used for subsequent parameter optimization.

[0171] To address camera lens distortion, the following radial and tangential distortion models are used to distort the normalized coordinates in the camera coordinate system, and then the distorted coordinates are transformed to the pixel coordinate system:

[0172] ,

[0173] ,

[0174] ,

[0175] ,

[0176] In the formula: Let be the coordinates of a 3D point in the camera coordinate system of the i-th positioning camera. and These are the normalized coordinates in the camera coordinate system before distortion and the distance of that coordinate to the camera center, respectively. Radial distortion coefficient; The tangential distortion coefficient is... These are the normalized coordinates in the distorted camera coordinate system. The two-dimensional predicted coordinates in the pixel coordinate system are obtained by substituting the coordinates in the calibration plate coordinate system into the pinhole camera model, radial distortion model, and tangential distortion model.

[0177] Using the aforementioned observation data, initial intrinsic parameter matrix, and initial extrinsic parameter matrix, a nonlinear optimization method based on minimizing reprojection error is employed to jointly optimize the camera's intrinsic and extrinsic parameters, and the radial and tangential distortion coefficients are solved. The objective function is as follows:

[0178] ,

[0179] In the formula, M is the number of calibration board images acquired; N m To determine the number of corner points of the plate in the m-th image; These are the actual pixel coordinates of the nth corner point in the mth image; These are the predicted pixel coordinates obtained by projecting from the model parameters; The distortion coefficient vector ; These are the rotation matrix and translation vector corresponding to each calibration board image.

[0180] S1-2, The plane equations of the worktable in the camera coordinate system include:

[0181] Combination Figure 4 As shown in the schematic diagram of the data acquisition workbench, the calibration board is placed on the workbench surface, and multiple positions are captured within the fixed field of view of the camera. For each image, the correspondence between the 3D calibration board points and their image pixels is constructed, and the extrinsic parameter matrix is ​​optimized by minimizing the reprojection error. The objective function is:

[0182] ,

[0183] In the formula, N is the number of calibration board corner points in each calibration board image; Let n be the coordinates of the nth corner point of the calibration board in the pixel coordinate system; Let be the three-dimensional coordinate point of the nth corner point of the calibration plate in the calibration plate coordinate system; This is the standard perspective projection function;

[0184] The optimization process is based on the Perspective-n-Point algorithm, ultimately obtaining the extrinsic parameter matrix of the calibration board coordinate system to the camera coordinate system in each image when the calibration board is located on the worktable. ;

[0185] Based on the above extrinsic matrix The corner points of the calibration plate in the entire calibration plate coordinate system are transformed from 3D world coordinates to camera coordinates, resulting in a set of 3D points in the camera coordinate system. Calculate the mean vector of a three-dimensional point set:

[0186] ,

[0187] Constructing a centralized dataset Calculate the three-dimensional covariance matrix of the point set. :

[0188] ,

[0189] Eigenvalue decomposition of the covariance matrix yields three real eigenvalues. and their corresponding eigenvectors Among them, the smallest eigenvalue corresponding feature vector The direction reflecting the minimum variance of the point set is the normal vector of the best-fit plane for the data points. This is used as the unit normal vector of the worktable plane in the camera coordinate system, where A, B, and C are respectively... , and Components along the axial direction;

[0190] Based on the above best-fit plane normal vector Sum of points Calculate the constant term of the plane equation:

[0191] ,

[0192] Finally, the parameters and equations of the worktable plane in the camera coordinate system are obtained:

[0193] ,

[0194] In the formula, the parameters of the workbench plane equation include the plane unit normal vector v and the plane equation constant term D.

[0195] To ensure consistency in the plane orientation, the direction of the normal vector needs to be constrained.

[0196] S1-3, The step of performing hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix includes:

[0197] Combination Figure 4The diagram shown illustrates the acquisition of camera calibration and hand-eye calibration data. First, the trajectory is reverted to its origin to unify the initial position of the trajectory coordinate system. Then, the calibration board is mounted on the robot's end effector, and the robot is controlled to move along the trajectory into the fixed field of view of the camera. Multiple sets of calibration board images are acquired under different robot poses. Finally, the camera coordinate system is obtained. With the robot base coordinate system Hand-eye transformation matrix It satisfies the following transformation relationship:

[0198] ,

[0199] In the formula, The transformation from the calibration board coordinate system to the camera coordinate system in the m-th calibration board image when the calibration board is located at the robot end effector is obtained using the Perspective-n-Point algorithm. The transformation matrix from the robot base coordinate system to the end effector in the m-th sub-calibration plate image is obtained from the robot forward kinematics and is a current technology.

[0200] After performing hand-eye calibration, the hand-eye transformation matrix is ​​obtained:

[0201] ,

[0202] In the formula: This is the rotation matrix for transforming the robot's base coordinate system to the camera coordinate system; Translation vector from robot base coordinate system to camera coordinate system.

[0203] S1-4, The calculation of the unit vector of the orbital direction in the robot base coordinate system, the correction of the hand-eye transformation matrix translation vector, and the acquisition of the transformation matrix from each camera coordinate system to the orbital coordinate system include:

[0204] Combination Figure 4 The diagram shown illustrates the unit vector for calibrating the trajectory direction. The calibration plate is fixed to the robot's end effector to record the current trajectory position. While keeping the robot's end-effector posture and camera field of view unchanged, the control track drives the robot to move continuously in the positive direction with a fixed step distance, and collects several images of the calibration board fixed to the robot's end-effector.

[0205] Based on each calibration board image, the intrinsic parameter matrix K of the corresponding camera and the distortion coefficients are used. The correspondence between the three-dimensional calibration board points and their image pixels, combined with the Perspective-n-Point algorithm, is used to solve for the extrinsic parameter matrix of the calibration board coordinate system in the camera coordinate system. ;

[0206] Based on the above, the hand-eye transformation matrix is ​​obtained. and extrinsic parameter matrix The corner coordinates in the calibration plate coordinate system Coordinates transformed to the robot base coordinate system The conversion relationship is as follows:

[0207] ,

[0208] Let the first The three-dimensional coordinates of the j-th corner point of the calibration plate in the camera base coordinate system in the image of the calibration plate are: Using the first calibration plate image as the reference image, the point-by-point difference between corresponding corner points in all calibration plate images and the reference image is calculated to obtain the relative translation vector in the robot base coordinate system. Furthermore, the average of all corner point difference vectors is used to fit the total displacement vector along the trajectory.

[0209] ,

[0210] In the formula: N represents the number of calibration board images acquired; M is the number of corner points of the calibration board in each image;

[0211] Normalize the result to obtain the unit vector of the orbital direction:

[0212] ,

[0213] In the formula, For unit vectors in , and The sum of the squares of the components along the three coordinate axes is 1;

[0214] Based on the current orbital position scalar obtained above The final transformation matrix from the camera coordinate system to the orbit coordinate system is:

[0215] ,

[0216] In the formula, This is the rotation matrix from the camera coordinate system to the orbit coordinate system; This is the translation vector from the camera coordinate system to the orbit coordinate system;

[0217] Finally, the pixel coordinate system is obtained. With orbital coordinate system Coordinate system mapping relationship:

[0218]

[0219]

[0220]

[0221] In the formula, To correct the distortion of the original pixel coordinates, the distortion-free normalized three-dimensional coordinates of the camera coordinate system are obtained through iteration. These are the three-dimensional coordinates in the orbital coordinate system obtained from pixel coordinates through a mapping relationship; It is the depth coefficient along the camera's optical axis, used to map normalized camera coordinates to three-dimensional points on the worktable plane in the camera coordinate system.

[0222] Repeat step S1 to construct the mapping relationship between the pixel coordinate system and the camera coordinate system of all cameras.

[0223] S2, Combination Figure 5 As shown, multiple camera images of the worktable are acquired, and the mask region features of the welding workpiece in the pixel coordinate system are extracted using an instance segmentation model. The mask region features are transformed to the orbit coordinate system using the parameters obtained in step S1, and the scaling factor and yaw angle are calculated to correct the mask region features. The IoU threshold is used to filter the corrected mask region features using a depth-first search strategy to construct a complete welding workpiece mask image.

[0224] Specifically, step S2 includes the following steps:

[0225] S2-1, The acquisition of welding workpiece images from multiple cameras and the extraction of mask region features of the welding workpiece in pixel coordinate system using an instance segmentation model include:

[0226] An instance segmentation model is constructed. This model uses workbench images captured by multiple fixed cameras as basic data, and performs mask annotation on the welding workpiece areas in the images to form a training sample set.

[0227] In this embodiment, the sample set is used to train an instance segmentation model based on the YOLOv11-Seg architecture. During training, the detection and segmentation accuracy of the model is improved by minimizing the bounding box error and the mask loss function, thereby generating a deep learning-based weldment segmentation detector. The YOLOv11-Seg instance segmentation model is an existing technology that supports multi-task branching capabilities such as object detection and instance segmentation. It also includes key structures such as C3k2 feature blocks, SPPF pooling modules, and C2PSA parallel spatial attention to improve recognition performance.

[0228] Multiple welding workpieces are placed on the worktable, and images of the worktable are acquired in parallel from multiple cameras. The features of the mask region of the welding workpiece in the pixel coordinate system are obtained by inference through the welding workpiece segmentation detector.

[0229] S2-2, The process of transforming the mask region features to the orbital coordinate system using the coordinate system mapping relationship obtained in step S1, and calculating the scaling factor and yaw angle to correct the mask region features includes:

[0230] Based on the features of the welding workpiece mask region obtained from the instance segmentation in step S2-1, the intersection of the center point of the mask region and the upper left corner of the horizontal and vertical edges of the mask region is extracted as the size positioning point. The pixel coordinates of the size positioning point are as follows: ;

[0231] Using the corresponding camera intrinsic parameter matrix K and distortion coefficients obtained in step S1-1 Convert the pixel coordinate system to camera-normalized coordinates, i.e., a unit vector in the depth direction:

[0232]

[0233] In the formula Normalized coordinates in the camera coordinate system after distortion correction ;

[0234] Combined with the table plane equation parameters in the camera coordinate system obtained in step S1-2 Based on the formula for calculating the intersection of a ray and a plane, the three-dimensional spatial coordinates of the positioning point of this dimension on the worktable plane in the camera coordinate system are obtained as follows:

[0235]

[0236] The three-dimensional coordinates of the above-mentioned positioning points are transformed from the camera coordinate system to the orbital coordinate system using the transformation matrix. Convert to orbital coordinates:

[0237]

[0238] In the formula, Let be the three-dimensional coordinates of the dimension positioning point in the orbital coordinate system, where the center point of the mask region in the orbital coordinate system is denoted as . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is . Let be the three-dimensional coordinates of the size positioning point in the camera coordinate system, where the center point of the mask region in the camera coordinate system is set to . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is Calculate the mask scaling factor based on the above coordinate points:

[0239]

[0240] The welding workpiece mask region features obtained from the above examples are scaled according to the mask scaling factor, and then... rotation matrix in Extract the yaw angle as the coordinate system around the orbit. Rotation angle of the shaft The calculation formula is as follows:

[0241] ,

[0242] ,

[0243] Constructing an orbital coordinate system Rotation matrix of the axis:

[0244] ,

[0245] The mask region features are rotated to complete the correction of the mask region features.

[0246] S2-3, The step of using a depth-first search strategy to perform IoU threshold filtering on the features of the corrected mask region to construct a complete weldment mask image includes:

[0247] Let there be any two mask regions Crossover ratio:

[0248] ,

[0249] If satisfied Then the two mask areas They are considered as the same weldment area and merged to form a mask area. ;in The set overlap threshold;

[0250] Based on the above Determine the set of mask regions to construct This set will be used to merge the mask region set of the complete welding mask image;

[0251] A depth-first search algorithm is used to traverse the set S of mask region pairs, identify the connected components in all mask region pairs, establish a chain structure of mask regions, and set it as a connected subgraph. And for the mask region in each connected subgraph Perform merge processing; where is the index of the connected subgraph corresponding to the workpiece, and also represents the workpiece index; j is the index of the mask region in the connected subgraph;

[0252] For each connected subgraph Record each mask area The corresponding area rectangle positioning point is The set of two-dimensional bounding box positioning points for the mask region in the connected subgraph is: Then the joint enclosing rectangle of the connected subgraph is located at the following point. Defined as:

[0253] ,

[0254] In the formula, This represents the total number of mask regions contained in the connected subgraph; This represents the location point of the bounding rectangle of the k-th mask region;

[0255] Perform mask fusion on the connected subgraph to construct connected regions. That is, a complete welding workpiece mask; the complete welding workpiece is discretized in the orbital coordinate system to construct a set of discrete points in space. And calculate the mask The coordinates of the centroid are used as an estimate of the spatial position of the corresponding weldment.

[0256] ,

[0257] ,

[0258] In the formula, where This represents the number of points in a spatial discrete point set. This represents the geometric coordinates of a discrete point in space within the orbital coordinate system. The coordinates of the centroid of the complete weldment mask in the orbital coordinate system; This represents each of the following in the set F of the two-dimensional bounding box positioning points: of Coordinate values; The center point of the complete weldment mask in the orbital coordinate system Axis coordinates.

[0259] S3. Use the target detection model to extract the weld seam region features from the complete weld mask image in step S2; and perform optimal coverage photography planning for multiple weld seam region features of a single weld, finally outputting the coordinates of the shooting point;

[0260] Specifically, step S3 includes the following steps:

[0261] S3-1, Combination Figure 5 As shown, the extraction of weld region features from the complete weld mask image in step S2 using the object detection model includes:

[0262] This study provides a dataset of welded workpiece images and annotates the weld seam regions on these images. Different categories are used to annotate weld seams with different neighborhood characteristics to improve detection accuracy and success rate. During annotation, in addition to the weld seam itself, the workpiece portion within the weld seam's neighborhood is also labeled as part of the weld seam region. This strategy helps improve the appearance differences between different weld seam categories, ensuring the model can accurately distinguish between different weld seam shapes and textures. By enhancing the richness of weld seam regions in the samples, this approach improves the performance of subsequent deep learning-based weld seam target region recognition.

[0263] An object detection model is constructed, using the aforementioned dataset as its foundation. In this example, the detection dataset is used to train an object detection model based on the YOLOv11 architecture. During training, the YOLOv11 network optimizes model parameters by minimizing bounding box regression error and class recognition error, thereby improving the detection accuracy of weld areas. This training process enables the model to efficiently and accurately detect weld areas in new welded workpiece images, meeting the practical application needs in complex welding environments, and ultimately generating a deep learning-based weld area object detector.

[0264] The complete welding workpiece mask generated in step S2-3 and constructed according to the orbital coordinate system scale is used as the image input, and then positioned by the positioning points. Within the defined bounded rectangular area, with Origin , shaft and Parallel and in the same direction, shaft and In reverse order and opposite direction, establish the workpiece region coordinate system; after reasoning about the joint enclosing rectangular region using the aforementioned weld region target detector, return the weld region rectangular frame. The center point of the rectangular frame of the weld area is:

[0265] ,

[0266] In the formula: The coordinates of the center of the weld area frame in the workpiece area coordinate system;

[0267] Since the mask is scaled in step S2-2, the ratio of the mask pixel coordinate system scale to the orbital coordinate system scale is 1; therefore, the width and height of the orbital coordinate system bounding box remain unchanged; based on the three-dimensional center coordinates of the complete welding workpiece mask image in step S6... The center coordinates of the weld area frame in the orbital coordinate system are:

[0268] ,

[0269] In the formula, The coordinates of the three-dimensional center of the weld area boundary box;

[0270] Obtain bounding box information ;in , .

[0271] S3-2, The optimal coverage photography plan for multiple weld seam areas of a single weldment, and the final output of the shooting point coordinates include:

[0272] Let e ​​be the weld region index of a single workpiece. Based on the obtained weld region bounding box information, let the weld region be... In the formula, Define the coordinates of the 2D center point of the region bounding box; define the camera's shooting area in the orbital coordinate system as... In the formula The coordinates of the center point of the bounding box of the shooting area. The width and height of the fixed shooting area;

[0273] Calculate the weld area Let the set of points be the center point of the region and the positions of its four corner points. for:

[0274] ,

[0275] Will Each point in the diagram serves as the coordinate of the center point of the bounding box of the shooting range, and several shooting areas are formed around these center points. Calculate each The number of weld areas included is selected based on the number of areas covered. In As the covered weld area The center point of the photo;

[0276] Repeat this process until all uncovered weld areas are completely covered;

[0277] set up Let the set of camera shooting areas obtained from the above process be an index for the camera shooting area. ,set up The shooting area in the image is an unmerged area. and Let the merged shooting area be any two camera shooting areas in the above set of shooting areas:

[0278] ,

[0279] If the merging conditions are met:

[0280] ,

[0281] The areas captured by the two cameras are then merged into one, and this merged area is then placed back into the original area. In the process of merging photographic areas, a breadth-first search strategy is used to select the set of photographic areas to be merged. The shooting areas in the image are searched and merged;

[0282] The final output is the set of photographic points for a single workpiece in the orbital coordinate system:

[0283] ,

[0284] In the formula, This is the preset photo height.

[0285] S4. Based on the unit vector of the track direction in step S1, the coordinates of the shooting point in step S3 are projected and corrected to calculate the actual moving distance of the track and the robot, so that it can reach the shooting area of ​​each weld seam and complete the positioning guidance.

[0286] Specifically, step S4 includes the following steps:

[0287] S4-1. Project the coordinates of the shooting points in step S3 to calculate the actual distance the robot travels along the track, enabling it to reach the shooting areas of each weld seam and complete the positioning guidance, including:

[0288] Based on the unit vector of the orbital direction determined in steps S1-3 For the set of camera coordinate points in the orbital coordinate system obtained in step S3-2 Projection correction is performed to ensure that the robot can accurately reach the imaging area of ​​each weld seam.

[0289] To ensure that the robot's linear motion axes align with the camera point after it moves along the track, inner product projection is not used. Since the coordinate axes of the robot's base coordinate system and the track coordinate system are aligned, it is necessary to calculate the projection ratio of the principal axes of the track motion to the unit vector of the track direction, thus converting the camera point coordinates into actual motion data. ;

[0290] In this example, let the main axis of the orbital motion be the X-axis, and the linear motion axis of the robot be the Y-axis. The actual motion data is as follows:

[0291]

[0292]

[0293]

[0294] In the formula: This represents the projection distance of the set of captured coordinate points onto the orbital coordinate system along the orbital direction vector; This refers to the corresponding point in the robot's base coordinate system after the robot has moved to the aforementioned projection distance. Axial direction and Distance moved along the axis;

[0295] Repeat steps S3 and S4 to obtain the actual motion data of each weld area in all workpieces so that the hardware can directly read and execute the corresponding motion tasks.

[0296] Please combine Figures 1 to 5 As shown, the present invention also provides an embodiment of a positioning device for performing the above-described weld seam region positioning method based on multi-camera vision and deep learning, the positioning device comprising:

[0297] Calibration and Transformation Correction Module: Used to calibrate the camera's intrinsic parameters, determine the camera's intrinsic parameter matrix and distortion coefficients; construct the planar equation of the worktable in the camera coordinate system; perform hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix; calculate the unit vector of the orbital direction in the robot's base coordinate system, correct the translation vector of the hand-eye transformation matrix, and obtain the transformation matrix from each camera coordinate system to the orbital coordinate system; for the input pixel coordinates, perform pixel coordinate system to camera coordinate system transformation and camera coordinate system to orbital coordinate system transformation using the above parameters.

[0298] Multi-camera image acquisition module: includes multiple camera units and welding worktable; the cameras are mounted on a gantry-type overhead frame fixed to the ground to acquire image data of the welded workpiece and transmit it to the data processing and control unit;

[0299] Image feature extraction module: used to perform instance segmentation on the workbench image captured by the camera, obtain the mask area features of the welding workpiece in pixel coordinate system; perform target detection on the complete weld mask image, and extract the weld area features;

[0300] Welding mask correction module: The positioning points of the extracted welding mask area features are transformed from the pixel coordinate system to the orbit coordinate system. The difference of the welding mask area features in the transformation is calculated, the scaling factor of the mask area features is calculated, and the yaw angle is solved according to the rotation matrix, so as to perform spatial correction on the welding mask area features.

[0301] Mask image construction module: used to construct a complete weldment mask image based on the corrected mask region features. It uses a depth-first search strategy to perform IoU threshold filtering on the above region features. If the IoU is greater than the threshold, it is regarded as the same weldment region and connected components are merged to construct a complete weldment mask image.

[0302] Weld seam area photography planning module: For each uncovered area, calculate the center point of the uncovered area and the positions of the four corner points of the uncovered area as candidate photography centers; calculate the number of weld seam areas covered by each candidate photography center, and select the candidate photography center with the most covered areas as the photography center point of this area, and remove the covered areas from the index of uncovered areas; use a breadth-first search strategy to merge and judge the above photography areas, and finally output the coordinates of the shooting point in the orbital coordinate system;

[0303] Motion distance correction module: Based on the unit vector of the track direction and the obtained set of shooting point coordinates, projection correction is performed, the projection ratio of the main axis of track motion to the unit vector of track direction is calculated, and then the shooting point coordinates are converted into track motion data and robot pose data for output;

[0304] The track robot control module includes a guide rail, a motor control cabinet, servo motors, a robot control cabinet, and the robot body. The guide rail, motor control cabinet, and servo motors form the track, which has an origin return function to keep the spatial relative relationship between the track coordinate system and each camera coordinate system constant. Based on the target pose information output by the computer device, the track drives the robot body to move and controls the posture and position adjustment of the robot body and its end effector, thereby achieving precise positioning and guidance of the weld seam area.

[0305] Furthermore, there are many specific methods and approaches to implement this invention, and the above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention.

Claims

1. A method for locating weld seam areas based on multi-camera vision and deep learning, characterized in that, Specifically, the following steps are included: S1. Perform intrinsic parameter calibration on the camera to determine the camera's intrinsic parameter matrix and distortion coefficients; Construct the plane equations of the worktable in the camera coordinate system to obtain the unit normal vector of the worktable plane. and the constant term D of the plane equation; Hand-eye calibration is performed on the camera and robot to obtain the hand-eye transformation matrix; Calculate the unit vector of the orbital direction in the robot's base coordinate system. Use this unit vector to linearly superimpose and correct the translation vector in the aforementioned hand-eye transformation matrix. Use the corrected hand-eye transformation matrix as the transformation matrix from the camera coordinate system to the orbital coordinate system. Based on the camera's intrinsic parameter matrix and distortion coefficients, the table plane equation parameters, and the transformation matrix from the camera coordinate system to the orbit coordinate system, the coordinate system mapping relationship between the pixel coordinate system and the orbit coordinate system is finally obtained. S2. Acquire worktable images from multiple cameras and use instance segmentation model to extract mask region features of the welding workpiece in pixel coordinate system; Using the coordinate system mapping relationship obtained in step S1, the above mask area features are transformed from the pixel coordinate system to the orbit coordinate system, and the scaling factor and yaw angle are calculated to correct the mask area features. A depth-first search strategy was used to perform IoU threshold filtering on the corrected mask region features to construct a complete weld mask image. S3. Use the target detection model to extract the weld seam region features from the complete weld mask image in step S2; and perform optimal coverage photography planning for multiple weld seam region features of a single weld, finally outputting the coordinates of the shooting point; S4. Based on the unit vector of the track direction in step S1, the coordinates of the shooting point in step S3 are projected and corrected to calculate the actual moving distance of the track and the robot, so that the robot and the robot end reach the shooting area of ​​each weld seam and complete the positioning guidance.

2. The weld seam region localization method based on multi-camera vision and deep learning according to claim 1, characterized in that, Define the following coordinate system: pixel coordinate system The origin is the pixel at the top left corner of the intersection of the horizontal and vertical pixel edges of the image captured by the camera. , The axis is set along the direction of increasing pixel column index. The axis is set along the direction of increasing pixel row index; camera coordinate system. With the camera's optical center as the origin ,in The plane is parallel to the camera's imaging plane. The axis is set along the camera's optical axis and points towards the subject; calibration plate coordinate system The origin is the lower left corner of the intersection of the horizontal and vertical edges of the checkerboard on the calibration board. , The shaft is set along the direction of the lateral edge line. The shaft is positioned along the longitudinal edge line. The axes are set along the plane of the vertical calibration plate, and the mapping relationship between this coordinate system and the camera coordinate system is determined by the camera extrinsic parameter matrix; robot base coordinate system. With the geometric center of the robot base as the origin , shaft and The axes are set along two orthogonal directions within the plane of the robot base. The axis is set along a direction perpendicular to the plane; orbital coordinate system The robot is reset to the limit position of the track switch using the track. The robot's base coordinate system at this position is used as the track coordinate system, and the three coordinate axes of this coordinate system are... They are parallel to the corresponding coordinate axes of the robot's base coordinate system.

3. The weld seam region localization method based on multi-camera vision and deep learning according to claim 2, characterized in that, Step S1, which involves calibrating the camera's intrinsic parameters, includes: acquiring an image of the calibration board within the camera's field of view; and using a pinhole camera model, establishing a mapping relationship between spatial 3D points and pixels, satisfying the following intrinsic and extrinsic parameter projection relationships: ， ， In the formula: K is the intrinsic parameter matrix of the camera; f is the normalized coordinate scale factor; f is the image distance; Each represents a pixel in and Physical scale in direction; θ represents the projected position coordinates of the camera's optical center on the pixel plane in the pixel coordinate system; θ represents... shaft and The angle between axes; R is the rotation matrix of the camera extrinsic matrix; t is the translation vector of the camera extrinsic matrix; The three-dimensional coordinates of the corner points of the chessboard grid on the calibration plate coordinate system; These are the two-dimensional coordinates of the point in the pixel coordinate system. By acquiring images of the calibration board in multiple different poses in space, multiple sets of three-dimensional spatial points in the calibration board coordinate system and two-dimensional imaging points of the spatial points in the pixel coordinate system are obtained. The two are used as observation data and substituted into the above projection relationship to solve for the initial intrinsic parameter matrix and initial extrinsic parameter matrix of the camera, which are then used for subsequent parameter optimization. To address camera lens distortion, the following radial and tangential distortion models are used to distort the normalized coordinates in the camera coordinate system, and then the distorted coordinates are transformed to the pixel coordinate system: ， ， ， ， In the formula: To locate the 3D point coordinates in the camera coordinate system of the camera, r and r are the normalized coordinates in the camera coordinate system before distortion and the distance of that coordinate to the camera center, respectively. Radial distortion coefficient; The tangential distortion coefficient is... These are the normalized coordinates in the distorted camera coordinate system. The two-dimensional predicted coordinates in the pixel coordinate system are obtained by substituting the coordinates in the calibration plate coordinate system into the pinhole camera model, radial distortion model, and tangential distortion model. Using the aforementioned observation data, initial intrinsic parameter matrix, and initial extrinsic parameter matrix, a nonlinear optimization method based on minimizing reprojection error is employed to jointly optimize the camera's intrinsic and extrinsic parameters, and the radial and tangential distortion coefficients are solved. The objective function is as follows: ， In the formula, The number of calibration board images acquired; To determine the number of corner points of the plate in the m-th image; These are the actual pixel coordinates of the nth corner point in the mth image; These are the predicted pixel coordinates obtained by projecting from the model parameters; The distortion coefficient vector ; These are the rotation matrix and translation vector corresponding to each calibration board image.

4. The weld seam region localization method based on multi-camera vision and deep learning according to claim 3, characterized in that, The step S1 of constructing the plane equations of the worktable in the camera coordinate system includes: The calibration board is placed on the worktable, and images of the calibration board in various poses are captured within the fixed field of view of the camera. For each calibration board image, the correspondence between the 3D calibration board points and the pixels of the calibration board image is constructed, and the extrinsic parameter matrix is ​​optimized by minimizing the reprojection error. The objective function is: ， In the formula, N is the number of calibration board corner points in each calibration board image; Let n be the coordinates of the nth corner point of the calibration board in the pixel coordinate system; Let be the three-dimensional coordinate point of the nth corner point of the calibration plate in the calibration plate coordinate system; This is the standard perspective projection function; The optimization process is based on the Perspective-n-Point algorithm, ultimately obtaining the extrinsic parameter matrix of the calibration board coordinate system to the camera coordinate system in each image when the calibration board is located on the worktable. ; Based on the above extrinsic matrix The corner points of the calibration plate in the entire calibration plate coordinate system are transformed from 3D world coordinates to camera coordinates, resulting in a set of 3D points in the camera coordinate system. Calculate the mean vector of a three-dimensional point set: ， Constructing a centralized dataset Calculate the three-dimensional covariance matrix of the point set. : ， Eigenvalue decomposition of the covariance matrix yields three real eigenvalues. and their corresponding eigenvectors Among them, the smallest eigenvalue Corresponding feature vector The direction that minimizes the variance of the point set is the normal vector of the best-fit plane for the data points. Normalizing this vector yields the unit normal vector of the worktable plane in the camera coordinate system. Where A, B, and C are respectively , and Components along the axial direction; Based on the above best-fit plane normal vector Sum of points Calculate the constant term of the plane equation: ， Finally, the parameters and equations of the worktable plane in the camera coordinate system are obtained: ， In the formula, the parameters of the workbench plane equation include the plane unit normal vector. and the constant term D of the plane equation; The above steps are described with a single camera as the object. Each positioning camera performs the steps respectively, and finally obtains the plane equation of the worktable plane in the coordinate system of each camera.

5. The weld seam region localization method based on multi-camera vision and deep learning according to claim 4, characterized in that, Step S1, which involves performing hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix, includes: First, the trajectory is reverted to its origin to unify the initial position of the trajectory coordinate system. Then, the calibration board is installed on the robot's end effector, and the robot is controlled to move along the trajectory into the fixed field of view of the camera, acquiring multiple sets of calibration board images under different robot poses. At this position, hand-eye calibration in an eye-in-hand mode is performed to obtain the camera coordinate system. With the robot base coordinate system Hand-eye transformation matrix It satisfies the following transformation relationship: ， In the formula, The transformation from the calibration board coordinate system to the camera coordinate system in the m-th calibration board image when the calibration board is located at the robot end effector is obtained using the Perspective-n-Point algorithm. Let be the transformation matrix from the robot base coordinate system to the end effector in the m-th sub-calibration plate image; After performing hand-eye calibration, the hand-eye transformation matrix is ​​obtained: ， In the formula: This is the rotation matrix for transforming the robot's base coordinate system to the camera coordinate system; This is the translation vector used to transform the robot's base coordinate system to the camera coordinate system.

6. The weld seam region localization method based on multi-camera vision and deep learning according to claim 5, characterized in that, Step S1, which involves calculating the unit vector of the orbital direction in the robot's base coordinate system, using the unit vector of the orbital direction to linearly superimpose and correct the translation vector in the hand-eye transformation matrix, and using the corrected hand-eye transformation matrix as the transformation matrix from the camera coordinate system to the orbital coordinate system, includes: Record the current orbital position Keeping the robot's end-effector posture and camera field of view unchanged, the control track drives the robot to move continuously in the positive direction with a fixed step distance, and collects several images of the calibration board fixed to the robot's end-effector. Based on each calibration board image, the intrinsic parameter matrix K of the corresponding camera and the distortion coefficients are used. The correspondence between the three-dimensional calibration board points and their image pixels, combined with the Perspective-n-Point algorithm, is used to solve for the extrinsic parameter matrix of the calibration board coordinate system in the camera coordinate system. ; Based on the above, the hand-eye transformation matrix is ​​obtained. and extrinsic parameter matrix The corner coordinates in the calibration plate coordinate system Coordinates transformed to the robot base coordinate system : ， Let the three-dimensional coordinates of the nth corner point of the calibration board in the camera base coordinate system be (the coordinates of the nth corner point of the calibration board in the mth image of the calibration board are given by the given coordinates of the nth corner point of the calibration board in the camera base coordinate system). Using the first calibration plate image as the reference image, the point-by-point difference between corresponding corner points in all calibration plate images and the reference image is calculated to obtain the relative translation vector in the robot base coordinate system. Furthermore, the average of all corner point difference vectors is used to fit the total displacement vector along the trajectory. ， In the formula: M represents the number of calibration board images acquired; N is the number of corner points of the calibration board in each calibration board image; Normalize the result to obtain the unit vector of the orbital direction: ， In the formula, For unit vectors in , and The sum of the squares of the components along the three coordinate axes is 1; Based on the current orbital position obtained above Translation vector in the hand-eye transformation matrix A linear superposition correction is performed, and the corrected hand-eye transformation matrix is ​​used as the transformation matrix from the camera coordinate system to the orbit coordinate system: ， In the formula, This is the rotation matrix from the camera coordinate system to the orbit coordinate system; This is the translation vector from the camera coordinate system to the orbit coordinate system; Finally, the pixel coordinate system is obtained. With orbital coordinate system Coordinate system mapping relationship: ， ， ， In the formula, To correct distortion in the original pixel coordinates, the distortion-free normalized 3D camera coordinates are obtained iteratively. These are the three-dimensional coordinates in the orbital coordinate system obtained from pixel coordinates through a mapping relationship; It is the depth coefficient along the camera optical axis, used to map normalized camera coordinates to three-dimensional points on the worktable plane in the camera coordinate system; Repeat step S1 to construct the mapping relationship between the pixel coordinate system and the camera coordinate system of all cameras.

7. The weld seam region localization method based on multi-camera vision and deep learning according to claim 6, characterized in that, In step S2, acquiring welding workpiece images from multiple cameras and extracting mask region features of the welding workpiece in pixel coordinates using an instance segmentation model includes: The workbench images captured by multiple fixed cameras are used as the basic data. The welding workpiece areas in the images are masked and labeled to form a training sample set. The training sample set is used to train an instance segmentation model based on the YOLOv11-Seg architecture. During the training process, the detection and segmentation accuracy of the model is improved by minimizing the bounding box error and the mask loss function, thereby generating a deep learning-based weldment segmentation detector. Multiple welding workpieces are placed on the worktable, and images of the worktable are acquired in parallel from multiple cameras. The features of the mask region of the welding workpiece in the pixel coordinate system are obtained by inference through the welding workpiece segmentation detector.

8. The weld seam region localization method based on multi-camera vision and deep learning according to claim 7, characterized in that, Step S2, which involves transforming the mask region features to the orbital coordinate system using the coordinate system mapping relationship obtained in step S1, and calculating the scaling factor and yaw angle to correct the mask region features, includes: Based on the features of the welding workpiece mask region obtained from instance segmentation, the intersection of the center point of the mask region and the upper left corner of the horizontal and vertical edges of the mask region is extracted as the dimension positioning point. The pixel coordinates of the dimension positioning point are denoted as follows: ; Using the obtained corresponding camera intrinsic parameter matrix K and distortion coefficients Convert pixel coordinates to camera-normalized coordinates, i.e., unit vectors in the depth direction: ， In the formula Normalized coordinates in the camera coordinate system after distortion correction ; Combined with the parameters of the worktable plane equation in the camera coordinate system Based on the formula for calculating the intersection of a ray and a plane, the three-dimensional spatial coordinates of the positioning point of this dimension on the worktable plane in the camera coordinate system are obtained as follows: ， The three-dimensional coordinates of the above-mentioned positioning points are transformed from the camera coordinate system to the orbital coordinate system using the transformation matrix. Convert to orbital coordinates: ， In the formula, Let be the three-dimensional coordinates of the dimension positioning point in the orbital coordinate system, where the center point of the mask region in the orbital coordinate system is denoted as . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is ; Let be the three-dimensional coordinates of the size positioning point in the camera coordinate system, where the center point of the mask region in the camera coordinate system is set to . The intersection of the horizontal and vertical edges of the mask area at the upper left corner is Calculate the mask scaling factor based on the above coordinate points: ， The weld workpiece mask region features obtained from the above examples are scaled according to the mask scaling factor, and then... rotation matrix in Extract the yaw angle as the coordinate system around the orbit. Rotation angle of the shaft The calculation formula is as follows: ， ， Constructing an orbital coordinate system Rotation matrix of the axis: ， The mask region features are rotated to complete the correction of the mask region features.

9. The weld seam region localization method based on multi-camera vision and deep learning according to claim 8, characterized in that, Step S2, which involves using a depth-first search strategy to perform IoU threshold filtering on the features of the corrected mask region to construct a complete weldment mask image, includes: Let there be any two mask regions Crossover ratio: ， If satisfied Then the two mask areas They are considered as the same weldment area and merged to form a mask area. ;in The set overlap threshold; Based on the above Determine the set of mask regions to construct This set will be used to merge the mask region set of the complete welding mask image; A depth-first search algorithm is used to search the set of mask regions. Perform a traversal, identify the connected components in all mask region pairs, establish a chain structure of mask regions, and set it as a connected subgraph. And for the mask region in each connected subgraph Perform a merge process; where J is the sequence number of the connected subgraph corresponding to the workpiece, which also represents the workpiece sequence number; j is the sequence number of the mask region in the connected subgraph; For each connected subgraph Record each mask area The corresponding area rectangle positioning point is The set of two-dimensional bounding box positioning points for the mask region in the connected subgraph is: Then the joint enclosing rectangle of the connected subgraph is located at the following point. Defined as: ， In the formula, f is the total number of mask regions contained in the connected subgraph; k This represents the location point of the bounding rectangle of the k-th mask region; Perform mask fusion on the connected subgraph to construct connected regions. That is, a complete welding workpiece mask; the complete welding workpiece is discretized in the orbital coordinate system to construct a set of discrete points in space. And calculate the mask The coordinates of the centroid are used as an estimate of the spatial position of the corresponding weldment. ， ， In the formula, where This represents the number of points in a spatial discrete point set. This represents the geometric coordinates of a discrete point in space within the orbital coordinate system. The coordinates of the centroid of the complete weldment mask in the orbital coordinate system; The above two-dimensional bounding box positioning point set F represents each f. j In Coordinate values; The center point of the complete weldment mask in the orbital coordinate system Axis coordinates.

10. The weld seam region localization method based on multi-camera vision and deep learning according to claim 9, characterized in that, Step S3 uses a target detection model to extract weld region features from the complete weld mask image in step S2, including: Provides a dataset of welded workpiece images, with annotations of the weld area on the workpiece images; The labeled workpiece image dataset was used to train a target detection model based on the YOLOv11 architecture. During training, the YOLOv11 network optimized the model parameters by minimizing the bounding box regression error and the class recognition error, and finally generated a weld area target detector based on deep learning. The complete welding workpiece mask generated in step S2 and constructed according to the orbital coordinate system scale is used as the image input, and then the positioning points are used to... Within the defined bounded rectangular area, with Origin , shaft and Parallel and in the same direction, shaft and In reverse order and opposite direction, establish the workpiece region coordinate system; after reasoning about the joint enclosing rectangular region using the aforementioned weld region target detector, return the weld region rectangular frame. The center point of the rectangular frame of the weld area is: ， In the formula: The coordinates of the center of the weld area frame in the workpiece area coordinate system; Based on the three-dimensional center coordinates of the complete welded workpiece mask image in step S6 The center coordinates of the weld area frame in the orbital coordinate system are: ， In the formula, The coordinates of the three-dimensional center of the weld area boundary box; Obtain bounding box information ;in , ; The optimal coverage photography plan for multiple weld seam regions of a single weldment, and the final output of the shooting point coordinates include: Let e ​​be the weld region index of a single workpiece. Based on the obtained weld region bounding box information, let the weld region be... In the formula, Define the coordinates of the 2D center point of the region bounding box; define the camera's shooting area in the orbital coordinate system as... In the formula The coordinates of the center point of the bounding box of the shooting area. The width and height of the fixed shooting area; Calculate the weld area Let the set of points be the center point of the region and the positions of its four corner points. for: ， Will Each point in the diagram serves as the coordinate of the center point of the bounding box of the shooting range, and several shooting areas are formed around these center points. Calculate each The number of weld areas included is selected based on the number of areas covered. In As the covered weld area The center point for taking the photo; Repeat this process until all uncovered weld areas are completely covered; set up Let the set of camera shooting areas obtained from the above process be an index for the camera shooting area. ,set up The shooting area in the image is an unmerged area. and Let the merged shooting area be any two camera shooting areas from the above set of shooting areas: ， If the merging conditions are met: ， The areas captured by the two cameras are then merged into one, and this merged area is then placed back into the original area. In the process of merging photographic areas, a breadth-first search strategy is used to select the set of photographic areas to be merged. The shooting areas in the image are searched and merged; The final output is the set of photographic points for a single workpiece in the orbital coordinate system: ， In the formula, This is the preset photo height.

11. The weld seam region localization method based on multi-camera vision and deep learning according to claim 10, characterized in that, In step S4, the coordinates of the shooting points in step S3 are projected and corrected to calculate the actual distance the robot travels along the track, enabling it to reach the shooting areas of each weld seam and complete the positioning guidance. Based on the unit vector of the orbital direction determined in step S1 The set of shooting coordinate points in the orbital coordinate system obtained in step S3 After projection correction, the actual motion data is: ， ， ， In the formula: This represents the projection distance of the set of captured coordinate points onto the orbital coordinate system along the orbital direction vector; This refers to the corresponding point in the robot's base coordinate system after the robot has moved to the aforementioned projection distance. Axial direction and Distance moved along the axis; Repeat steps S3 and S4 to obtain the actual motion data of each weld area in all workpieces.

12. A positioning device for performing the weld area positioning method based on multi-camera vision and deep learning as described in any one of claims 1 to 11, characterized in that, include: Calibration and Transformation Correction Module: Used to calibrate the camera's intrinsic parameters, determine the camera's intrinsic parameter matrix and distortion coefficients; construct the plane equation of the worktable in the camera coordinate system; and perform hand-eye calibration on the camera and robot to obtain the hand-eye transformation matrix. Calculate the unit vector of the orbital direction in the robot's base coordinate system, correct the translation vector of the hand-eye transformation matrix, and obtain the transformation matrix from each camera coordinate system to the orbital coordinate system; for the input pixel coordinates, perform pixel coordinate system to camera coordinate system transformation and camera coordinate system to orbital coordinate system transformation using the above parameters; Multi-camera image acquisition module: includes multiple camera units and a welding worktable; The camera is mounted on a gantry-type overhead frame fixed to the ground to acquire image data of the welded workpiece and transmit it to the data processing and control unit. Image feature extraction module: used to perform instance segmentation on the workbench image captured by the camera and obtain the mask area features of the welding workpiece in pixel coordinate system; Target detection is performed on the complete weld mask image, and the weld area features are extracted. Welding mask correction module: The module transforms the positioning points of the extracted welding mask area features from the pixel coordinate system to the orbit coordinate system, calculates the difference in the welding mask area features during the transformation, calculates the scaling factor of the mask area features, and solves the yaw angle based on the rotation matrix, thereby spatially correcting the welding mask area features. Mask image construction module: used to construct a complete weldment mask image based on the corrected mask region features. It uses a depth-first search strategy to perform IoU threshold filtering on the above region features. If the IoU is greater than the threshold, it is regarded as the same weldment region and connected components are merged to construct a complete weldment mask image. Weld area photography planning module: For each uncovered area, calculate the center point of the uncovered area and the positions of the four corner points of the uncovered area as candidate photography centers; Calculate the number of weld seam areas covered by each candidate image center, select the candidate image center with the most covered areas as the image center point of this area, and remove the covered areas from the index of uncovered areas; use a breadth-first search strategy to merge and judge the above image areas, and finally output the coordinates of the shooting point in the orbital coordinate system. Motion distance correction module: Based on the unit vector of the track direction and the obtained set of shooting point coordinates, projection correction is performed, the projection ratio of the main axis of track motion to the unit vector of track direction is calculated, and then the shooting point coordinates are converted into track motion data and robot pose data for output; The track robot control module includes a guide rail, a motor control cabinet, servo motors, a robot control cabinet, and the robot body. The guide rail, motor control cabinet, and servo motors form the track, which has an origin return function to keep the spatial relative relationship between the track coordinate system and each camera coordinate system constant. Based on the target pose information output by the computer device, the track drives the robot body to move and controls the posture and position adjustment of the robot body and its end effector, thereby achieving precise positioning and guidance of the weld seam area.

13. A computer apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 11.

Images