Weld joint visual inspection and robot coordinate conversion method and system based on multi-modal feature fusion

Abstract

The invention relates to the technical field of industrial robot visual control, and discloses a welding seam visual detection and robot coordinate conversion method and system based on multi-modal feature fusion, which can realize high-precision automatic detection and positioning of a welding seam in a complex industrial environment and accurately convert a detection result into a space coordinate which can be executed by a robot. According to the scheme, during modeling, images and teaching coordinates are collected, template blocks are cut, multi-modal features are pre-calculated, relevant data are stored after the physical distance is calculated, and indexes are updated; when the welding seam is detected, teaching coordinates are called to cut the reasoning block for detection, offset cutting and a model size backspacing mechanism are supplemented, then the welding seam is determined through multi-modal fusion similarity screening, and pixel coordinates are calculated; and finally, obtaining a depth value, carrying out distortion removal on the pixel coordinates to recover the three-dimensional coordinates of the camera, and carrying out homogeneous transformation to obtain the coordinates of the robot base and outputting the coordinates. The method is suitable for complex industrial environments with multiple welding seams, multiple interferences, high light reflection and the like, and the automation level and the welding precision of the welding robot can be remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of industrial robot vision control technology, specifically to a method and system for weld seam visual inspection and robot coordinate transformation based on multimodal feature fusion. Background Technology

[0002] With the continuous development of industrial automation technology, welding robots are increasingly widely used in manufacturing. Accurate weld inspection and positioning are crucial for ensuring the quality of automated welding. However, existing weld inspection and positioning technologies still face many challenges in real-world industrial scenarios:

[0003] (1) Poor environmental adaptability: Existing weld inspection systems mostly rely on manually set image processing rules, such as edge detection and morphological processing. These methods are poorly adaptable to changes in weld type, are easily affected by environmental factors such as changes in lighting and metal reflection, and have low detection accuracy for irregular welds, making it difficult to meet the requirements of high-precision welding.

[0004] (2) Difficulty in acquiring depth information: The surface of metal workpieces has high reflectivity, which causes a large number of invalid areas, i.e., depth holes, to be generated when the depth camera is acquiring data. Most existing depth map repair algorithms are designed for ordinary scenes and are difficult to effectively handle the depth loss problem unique to metal surfaces, and cannot accurately restore the three-dimensional spatial information of the weld area.

[0005] (3) Problems in multi-target weld identification: In actual welding scenarios, there are often multiple welds with similar shapes within the field of view of a single image. When faced with this situation, existing detection systems lack an effective target differentiation mechanism, making it difficult to accurately identify the specific target weld that needs to be welded, and are prone to misjudgment.

[0006] (4) Positioning accuracy issues: Converting the pixel coordinates detected in the image into base coordinates that the robot can execute requires multiple steps such as camera distortion correction and hand-eye calibration. Existing systems often neglect the correct order of distortion correction or have problems with improper application of calibration parameters when processing this conversion link, resulting in significant deviations in robot positioning.

[0007] (5) Scale adaptability of the detection model: The detection model with a fixed input size has a large difference in detection effect for weld targets of different sizes. Small welds are easy to miss, while large welds may not be detected completely, and there is a lack of adaptive multi-scale detection mechanism.

[0008] Therefore, there is an urgent need for a weld seam inspection and positioning method that can overcome the above-mentioned technical defects in order to improve the automation level and welding accuracy of industrial welding robots. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a method and system for weld visual inspection and robot coordinate transformation based on multimodal feature fusion, which realizes high-precision automatic inspection and positioning of welds in complex industrial environments, and accurately converts the inspection results into spatial coordinates that can be executed by the robot.

[0010] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0011] On the one hand, this invention provides a method for weld seam visual inspection and robot coordinate transformation based on multimodal feature fusion, including the following steps:

[0012] S1. Weld modeling stage:

[0013] S11. Acquire reference images and corresponding depth images, and receive the coordinates of the weld center point from manual instruction;

[0014] S12. Using the teaching point as the center, cut out the template block from the reference image;

[0015] S13. Perform multimodal feature pre-computation on the template image block, including grayscale image conversion, convolutional neural network feature vector extraction, and ORB feature descriptor extraction;

[0016] S14. Calculate the physical distance from the reference line to the first weld point based on the depth value at the corresponding position of the first weld point and the position of the preset reference line;

[0017] S15. Save the template blocks and modeling metadata, and update the template index file corresponding to the product type;

[0018] S2. Weld inspection stage:

[0019] S21. Receive the runtime image and obtain the corresponding teaching point coordinates based on the product type and weld index;

[0020] S22. Crops out the inference patch from the input image, inputs it into the neural network detection model for weld detection, and if no target is detected, it is cropped by shifting left and right along the teaching point coordinates and then re-detected;

[0021] S23. Enable dynamic model size fallback mechanism. When the model with the default size fails to detect the target, automatically switch to a model with a larger input size for secondary detection.

[0022] S24. When multiple candidate targets are detected, the multimodal fusion similarity between each candidate target and the template block is calculated, and the candidate target with the highest similarity is selected as the weld detection result.

[0023] S25. Calculate the pixel coordinates of key welding points based on the weld inspection results;

[0024] S3. Coordinate Transformation Stage:

[0025] S31. Obtain the depth value of the welding key point location;

[0026] S32. Perform distortion correction on pixel coordinates and combine with depth values ​​to recover the 3D points in the camera coordinate system;

[0027] S33. Convert the camera coordinates to robot base coordinates using the Eye-to-Hand homogeneous transformation matrix and output them.

[0028] Furthermore, in step S12, a square image block of a preset size is cropped from the reference image according to the following formula as a template block. When the cropped area exceeds the image boundary, it is filled with black pixels:

[0029] ;

[0030] in, For reference image, Here are the coordinates of the teaching point. Template size The template block obtained by cutting.

[0031] Furthermore, in step S13, the grayscale conversion involves converting the template image from the RGB color space to a grayscale image, using the following formula:

[0032] ;

[0033] in, These are the pixel values ​​for the red, green, and blue color channels, respectively. The converted grayscale value;

[0034] The convolutional neural network feature vector extraction involves using a pre-trained convolutional neural network to extract the depth feature vector of the template image and then normalizing the feature vector.

[0035] The ORB feature descriptor extraction method uses the ORB algorithm to extract key points and their descriptors from the template image.

[0036] Furthermore, in step S14, the physical distance from the reference line to the first weld point is calculated as follows:

[0037] ;

[0038] in, Pixel distance, For depth value, The scaling factor is the calibrated value. Image width, This represents the actual physical distance.

[0039] Furthermore, in step S22, cropping the inference map from the input image includes:

[0040] Using the pixel x-coordinate of the center point of the input image as the vertical coordinate of the cropped image block and the pixel y-coordinate of the teaching point as the horizontal coordinate of the cropped image block, a pre-defined inference image block is cropped from the input image.

[0041] Furthermore, in step S24, calculating the multimodal fusion similarity between each candidate target and the template image patch includes:

[0042] The SSIM algorithm is used to calculate the structural similarity between candidate targets and template tiles:

[0043] ;

[0044] Where SSIM(x,y) represents the structural similarity between the candidate target and the template tile; and These are the mean values ​​of the candidate target and the template tile, respectively. and These represent the variances of the candidate targets and the template tiles, respectively. For covariance; and It is a constant;

[0045] The same convolutional neural network used in the modeling phase is used to extract feature vectors of candidate targets, and the cosine similarity between these vectors and the feature vectors of the template patch is calculated as the deep feature similarity.

[0046] ;

[0047] in, The template patch feature vector, S represents the feature vector of the candidate target; cnn For deep feature similarity;

[0048] Extract the ORB feature descriptors of candidate targets and match them with the ORB descriptors of template patches. Calculate local feature similarity based on the number of matching point pairs and the matching distance.

[0049] ;

[0050] in, The average Hamming distance for matching point pairs, To match the number of point pairs, and The number of key points for candidate targets and template tiles, respectively; S orb For local feature similarity;

[0051] The structural similarity, deep feature similarity, and local feature similarity are weighted and fused:

[0052] ;

[0053] in, , , Let be the weighting coefficient, satisfying .

[0054] Furthermore, in step S25, the key welding points include the left welding point and the right welding point, and the coordinates of the key welding points are calculated as follows:

[0055] ;

[0056] in, The coordinates of the left edge of the detection box. For the detection frame width, and This is the preset horizontal scaling factor. The ordinate of the reference edge, This is the vertical offset.

[0057] Furthermore, in step S31, when the depth value of the welding key point is missing, an adaptive surface fitting method is used for depth repair, including:

[0058] Collect valid depth value points within a preset search range around the target point;

[0059] Outliers are filtered out using the median absolute deviation method, retaining only those that meet the criteria. The data points of the condition, where, For the first The depth value of each point, The median of all depth values. This represents the absolute deviation of the median.

[0060] Elliptical surface models should be used preferentially. Perform least-squares fitting, and if the fitting fails, revert to the circular surface model. ;in, The parameters are those of the model to be fitted. For pixel coordinates, This is the depth value;

[0061] Substitute the target point coordinates into the fitted surface equation to calculate the predicted depth value.

[0062] Alternatively, in step S31, when the depth value of the welding critical point location is missing, the depth value is estimated using a neighborhood statistical method:

[0063] Determine the search direction based on whether the key point is on the left or right;

[0064] A certain number of pixels are taken to the left and right of the row where the key point is located to form a sampling window, and the proportion of effective points within the window is calculated.

[0065] When the proportion of valid points meets the threshold condition, Gaussian weighted spline interpolation is used to estimate the depth value of the target point, and the weighting function is: ;in, For the first The distance from each sampling point to the target point The Gaussian kernel width parameter;

[0066] If the condition is not met, search the neighboring rows or take the non-zero depth value of the preset sequence number along the search direction.

[0067] Furthermore, in step S32, distortion correction is performed on the pixel coordinates, and the 3D points in the camera coordinate system are recovered by combining the depth values, including:

[0068] The original distorted image directly output by the camera is used as the input image;

[0069] Using a pre-calibrated camera intrinsic matrix and distortion coefficient The distortion correction function is called to convert the original pixel coordinates into normalized camera coordinates. :

[0070] ;

[0071] in, These are the original pixel coordinates;

[0072] Then, the 3D points in the camera coordinate system are recovered based on the normalized coordinates and depth values:

[0073] ;

[0074] in, This represents the depth value.

[0075] Furthermore, in step S33, the camera coordinates are converted to robot base coordinates using an Eye-to-Hand homogeneous transformation matrix, including:

[0076] Finally, the homogeneous transformation matrix obtained from the Eye-to-Hand calibration is used to transform the 3D points in the camera coordinate system to the robot base coordinate system:

[0077] ;

[0078] in, It is a 4×4 homogeneous transformation matrix, containing rotation matrices. Translation vector :

[0079] ;

[0080] P base For the robot's base coordinates.

[0081] On the other hand, the present invention also provides a weld visual inspection and robot coordinate transformation system based on multimodal feature fusion, including a modeling module, an inspection module and a coordinate transformation module;

[0082] The modeling module includes:

[0083] The template cropping unit is used to crop template blocks from the reference image based on the coordinates of the teaching points;

[0084] The feature extraction unit is used to perform grayscale conversion, convolutional neural network feature extraction, and ORB feature extraction on the template image.

[0085] The index management unit is used to maintain the storage structure and version index of template files;

[0086] The detection module includes:

[0087] The image preprocessing unit is used to perform image cropping, resizing, and normalization.

[0088] The neural network inference unit is used to perform object detection and output candidate objects;

[0089] The model fallback unit is used to automatically switch to an alternative model when the default model does not detect the target;

[0090] The similarity matching unit is used to calculate the multimodal fusion similarity between each candidate target and the template image patch when there are multiple candidate targets, and select the candidate target with the highest similarity as the detection result.

[0091] The key point calculation unit is used to calculate the coordinates of welding key points based on the position of the detection frame.

[0092] The coordinate transformation module includes:

[0093] Depth repair unit, used to recover missing depth values ​​using surface fitting or neighborhood statistics methods;

[0094] The distortion correction unit is used to convert pixel coordinates into normalized coordinates using calibration parameters;

[0095] The coordinate transformation unit is used to convert camera coordinates to robot base coordinates using an Eye-to-Hand homogeneous transformation matrix.

[0096] The beneficial effects of this invention are:

[0097] (1) High detection accuracy and strong adaptability:

[0098] By adopting a deep learning target detection algorithm to replace traditional image processing methods, the system can adapt to different lighting conditions and weld types, significantly improving detection accuracy. The dynamic model size backoff mechanism enables the system to effectively detect weld targets of different scales, avoiding the problem of missing small targets. The left-right sliding image block cropping strategy can effectively cope with the operation deviation of industrial production lines and achieve accurate location of predetermined welds.

[0099] (2) Accurate recognition of multi-weld seam scenes:

[0100] The multimodal feature fusion similarity matching method comprehensively considers information from three dimensions: structural similarity, deep semantic features, and local geometric features. It can accurately distinguish between different welds with similar shapes and solves the problem of misjudgment in multi-object scenarios by traditional methods.

[0101] (3) Reliable depth information recovery:

[0102] To address the problem of deep holes caused by highly reflective metallic surfaces, an adaptive surface fitting repair algorithm is adopted. This algorithm can automatically select an elliptical or circular surface model for fitting based on the distribution characteristics of surrounding effective points, resulting in high depth recovery accuracy and a low failure rate.

[0103] (4) Precise coordinate transformation specifications:

[0104] The input image is explicitly required to be the original distorted image. Distortion correction is correctly performed during the coordinate transformation process to avoid the cumulative error caused by secondary correction. The complete eye-to-hand calibration and coordinate transformation process ensures the accurate conversion from pixel coordinates to robot base coordinates.

[0105] (5) High system operating efficiency:

[0106] Template feature pre-computation and caching mechanisms avoid redundant computation overhead; the batch vectorized SSIM computation method significantly improves the speed of multi-candidate matching; the asynchronous image saving mechanism avoids blocking the main detection process by file writing; and the lazy model loading strategy optimizes memory usage efficiency.

[0107] (6) Good engineering practicality:

[0108] The modular system design makes each subsystem independently configurable; the comprehensive logging and visualization storage mechanism facilitates troubleshooting and system optimization; and the multi-version template management function supports product switching and template updates. Attached Figure Description

[0109] Figure 1This is a schematic diagram of the weld seam modeling process in an embodiment of the present invention, illustrating the complete process from reference image acquisition to template feature pre-calculation and then to index file management.

[0110] Figure 2 This is a schematic diagram of the weld seam detection process in an embodiment of the present invention, illustrating the processing from image cropping to target detection, then to similarity matching, and finally output of key points.

[0111] Figure 3 This is a schematic diagram of multimodal feature fusion similarity calculation in an embodiment of the present invention, illustrating the extraction and fusion process of three features: SSIM structural similarity, CNN deep feature similarity, and ORB local feature similarity.

[0112] Figure 4 This is a schematic diagram of the dynamic model size rollback mechanism in an embodiment of the present invention, illustrating the process of automatically switching to a larger size model for secondary detection after the default size model fails to be detected.

[0113] Figure 5 This is a schematic diagram illustrating the principle of deep hole repair in an embodiment of the present invention, showing the processing steps of effective point collection, outlier filtering, surface model fitting, and depth value prediction.

[0114] Figure 6 This is a schematic diagram of the Eye-to-Hand coordinate transformation process in an embodiment of the present invention, showing the complete link from the original pixel coordinates through distortion correction and normalization to the recovery of 3D points in the camera coordinate system and finally to the robot base coordinate system.

[0115] Figure 7 This is a schematic diagram of the overall architecture of the weld inspection and positioning system in this embodiment of the invention, showing the three major modules: the modeling subsystem, the inspection subsystem, and the coordinate transformation subsystem, and their interrelationships. Detailed Implementation

[0116] This invention aims to provide a method and system for weld seam visual inspection and robot coordinate transformation based on multimodal feature fusion. It achieves high-precision automatic inspection and localization of weld seams in complex industrial environments and accurately converts the inspection results into robot-executable spatial coordinates. The core idea is to address the key pain points of weld seam inspection and robot localization in complex industrial scenarios through an integrated design of multimodal feature fusion, dynamic scale adaptation, targeted depth repair, and standardized coordinate transformation. By extracting multi-dimensional features in the modeling stage, achieving adaptive scale detection and accurate target matching in the detection stage, and completing targeted depth repair and rigorous link transformation in the coordinate transformation stage, an end-to-end solution is constructed from image acquisition to robot-executable coordinate output, achieving a synergistic improvement in weld seam inspection accuracy, recognition stability, and coordinate transformation accuracy.

[0117] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0118] Example 1

[0119] This embodiment provides a method for weld seam visual inspection and robot coordinate transformation based on multimodal feature fusion, and its specific implementation is as follows:

[0120] I. System Hardware Configuration

[0121] The system hardware includes one industrial RGB-D camera for simultaneously acquiring color and depth images; one edge computing device equipped with a GPU for running neural network inference and image processing algorithms; and one dual-arm six-axis welding robot (within the same coordinate system), with a welding torch and welding rod mounted at each end. The camera is mounted in an eye-to-hand manner, meaning it is fixed above the worktable and does not move with the robotic arm.

[0122] II. System Software Configuration

[0123] The system uses YAML configuration files for parameter management. Parameters related to the detection model include: default model input size set to 256 pixels, alternative model input size set to 512 pixels, template matching tile size set to 512 pixels, detection confidence threshold set to 0.80, and non-maximum suppression threshold set to 0.60. Parameters related to similarity calculation include: GPU is preferred for computation, and SSIM weights are used. Set to 0.4, CNN weights Set to 0.4, ORB weight Set to 0.2. Depth processing parameters include: surface fitting as the repair strategy, and an effective depth range of 300 mm to 800 mm.

[0124] III. Specific Implementation Process of Weld Modeling

[0125] The core objective of this process is to construct a multimodal feature template library for weld seams, providing a benchmark for subsequent detection and matching. (See [link to documentation]). Figure 1 The specific steps are as follows:

[0126] 1. Acquisition of reference images and teaching points

[0127] The operator first places the standard workpiece at the designated position on the worktable, and the control system acquires reference and depth images. Then, through the human-machine interface, the operator clicks on the center position of each weld seam in sequence on the image, and the system records the pixel coordinates of these teaching points. .

[0128] This step lays the foundation for subsequent template creation by acquiring image references and key location information of standard welds.

[0129] 2. Template block trimming

[0130] The system crops images from color images, centered on each teaching point. The pixel image block serves as a template for the weld. In this embodiment... The cutting formula is:

[0131] ;

[0132] If the cropped area extends beyond the image boundary, the excess portion is filled with black pixel values.

[0133] This step extracts images of the core weld area, eliminates irrelevant background interference, and ensures the relevance of the template features.

[0134] 3. Multimodal feature pre-computation

[0135] The system performs multimodal feature pre-calculation for each template tile:

[0136] Grayscale conversion uses a standard weighted formula:

[0137] ;

[0138] CNN feature extraction uses a pre-trained ResNet18 network, removing the final fully connected classification layer and retaining the 512-dimensional feature vector after global average pooling. And perform L2 normalization to make its magnitude length 1:

[0139] ;

[0140] ORB feature extraction uses default parameters to create an ORB detector, extracting keypoint locations and corresponding binary descriptors from the template. .

[0141] This step extracts weld features from different dimensions to provide data support for subsequent multimodal matching, thereby improving the ability to distinguish targets.

[0142] 4. Calculation of the physical distance from the reference line to the weld point

[0143] The system reads the depth value of the first weld seam teaching point location. If the depth value is missing at that location, a surface fitting method is used to repair it. This is combined with the reference lines set by the operator. coordinate Calculate the physical distance from the reference line to the first weld. The calculation formula is:

[0144] ;

[0145] in Pixel distance, This represents the depth value (in meters). For calibration coefficients, This represents the image width.

[0146] This step calculates the physical distance from the reference line to the first weld point. This distance information can be used for subsequent workpiece positioning, offset compensation, and limiting the robotic arm's operating range.

[0147] 5. Template storage and index update

[0148] The system creates a directory for the current product type under the storage path, using a timestamp accurate to the microsecond as the subdirectory name to ensure uniqueness. Template tiles are saved as JPEG image files, and modeling metadata is saved as JSON files. Simultaneously, the index file for this product type is updated, recording the path and time information of all historical modeling versions, and the quick access file is updated to point to the latest version.

[0149] This step enables the orderly management and rapid retrieval of templates, supports product switching and template updates, and ensures the efficiency of the testing process.

[0150] IV. Specific Implementation Process of Weld Inspection

[0151] The core objective of this process is to accurately identify the target weld from real-time acquired images. See [link to relevant documentation]. Figure 2 The specific process for weld inspection is as follows:

[0152] 1. Acquisition of runtime images and teaching points

[0153] After the robot moves to the preset image acquisition position, the system acquires a color image and a depth image of the current workpiece. Based on the product type specified in the production task and the weld number of the current weld, the system looks up the corresponding teaching point coordinates from the index file. .

[0154] This step obtains the actual image data and reference position information of the workpiece to be inspected, providing a basis for subsequent precise cutting and inspection.

[0155] 2. Trimming and Preprocessing of Inference Graphs

[0156] The system crops the image from the input image using the pixel x-coordinate of the center point as the ordinate of the cropped image patch and the pixel y-coordinate of the teaching point as the x-coordinate of the cropped image patch. Inference blocks of pixels, in this embodiment The inference graph is fed into a neural network detection model, which is deployed using the NCNN framework and supports GPU acceleration.

[0157] In the preprocessing stage, the image is resized while maintaining its aspect ratio. Any insufficient areas are filled with gray pixels (value 114) to form a square, and then normalization is performed.

[0158] ;

[0159] The model output, after confidence threshold filtering and NMS processing, yields the detection results, with each detection box represented as follows: , where represents the x and y coordinates of the center point of the detection box, the width and height of the rectangle, and the confidence level, respectively.

[0160] If no valid target is detected, the cropping box is offset by a fixed number of pixels to the left or right along the y-coordinate of the teaching point pixel. In this embodiment, the offset is 192 pixels (i.e., it overlaps with one side of the previous cropping box by 64 pixels), and a total of 5 image blocks are cropped in a single size.

[0161] This step focuses on areas where welds may occur, reducing interference from irrelevant areas, and adjusts the image to a model-compatible format to ensure inference accuracy.

[0162] 3. Neural Network Detection and Model Backoff

[0163] If the default 256 model fails to detect any targets, the system automatically triggers a model fallback mechanism. See [link / reference]. Figure 4 Specifically, this includes: First, checking if a 512-size alternative model path has been configured. If configured, the system uses a lazy loading strategy, loading the 512 model into memory only the first time it is needed. Then, it re-crops the image from the original image using the pixel x-coordinate of the input image center point as the ordinate of the cropped image patch and the pixel y-coordinate of the teaching point as the x-coordinate of the cropped image patch. The pixel-sized tiles are detected using the 512 model. Regardless of whether the 512 model detects the target, subsequent coordinate calculations are based on the 512-pixel tile size.

[0164] This step covers the possible offset range of the weld by multi-position cutting, combined with model size rollback, to ensure that welds of different scales can be effectively detected, thereby improving the detection coverage.

[0165] 4. Multimodal fusion matching

[0166] If the detection result contains multiple candidate boxes, the system initiates a multimodal similarity matching process. (See below) Figure 3 For each candidate box, crop it from the original image based on its center coordinates. Matching tiles for pixels.

[0167] SSIM similarity calculation uses the structural similarity index formula:

[0168] ;

[0169] in, and Images and The mean, and They are variance, For covariance, , It is the stability constant.

[0170] When batch computation is enabled, all candidate patches are grouped into batches and computed simultaneously with the template, leveraging vectorized operations to improve efficiency. CNN feature similarity calculation extracts feature vectors from the candidate patches. And calculate the cosine similarity with the pre-stored template feature vector:

[0171] ;

[0172] ORB feature similarity calculation extracts ORB descriptors for candidate image patches. Brute-force matching is performed against the template descriptor. The result is based on the number of matches. and average matching distance Calculate the score:

[0173] ;

[0174] in, and These represent the number of key points for the two tiles, respectively.

[0175] The fused similarity is obtained by weighting and summing the similarities of the three dimensions according to preset weights.

[0176] ;

[0177] In this embodiment , , ,satisfy .

[0178] Finally, all candidates were ranked in descending order, with fusion similarity as the primary ranking criterion and detection confidence as the secondary criterion. The candidate ranked first was selected as the final detection result.

[0179] This step uses multi-dimensional feature fusion to accurately distinguish welds with similar shapes, solving the problem of misjudgment in multi-target scenarios and ensuring the accuracy of target recognition.

[0180] 5. Calculation of key welding points

[0181] Calculate the key welding points based on the final inspection frame position. Let the inspection frame parameters be... (Global coordinates), the key point calculation formula is:

[0182] ;

[0183] In this embodiment , , Pixels Pixel.

[0184] This step determines the core location of the welding operation, providing a precise pixel coordinate reference for subsequent coordinate transformation and robot operations.

[0185] V. Specific Implementation Process of Coordinate Transformation

[0186] The core objective of this process is to accurately convert pixel coordinates into robot base coordinates. The specific implementation is as follows:

[0187] 1. Depth value acquisition and repair

[0188] The system acquires the depth values ​​corresponding to the pixel positions of the left and right keypoints. Since reflections from the metal surface may cause depth loss, the system first checks if the depth values ​​at the keypoint positions are valid. If the depth value is zero or outside the valid range, the depth repair process is initiated. Depth repair can employ surface fitting or neighborhood statistical repair.

[0189] Surface fitting and repair, see Figure 5 Specifically, it includes:

[0190] (1) Based on key points Centered on the initial search radius Traverse all pixels within the range, collecting pixels with non-zero depth values ​​that are within the valid range. Points within the range are considered as the set of valid points. If the number of valid points is less than the minimum requirement (4 in this embodiment), the search radius is gradually expanded until the condition is met or the maximum number of searches is reached.

[0191] (2) Calculate the median of the depth values ​​for the set of valid points. And the absolute deviation of the median (MAD):

[0192] ;

[0193] (3) Filter out outliers that deviate from the median by more than 3 times the MAD:

[0194] ;

[0195] (4) Extract the coordinates and depth values ​​of the effective points, and fit the equation of the elliptic surface using the least squares method:

[0196] ;

[0197] (5) By judging the discriminant The value is used to verify whether the fitting result satisfies the elliptic condition. If the conditions are not met, revert to the circular surface model:

[0198] ;

[0199] (6) Set the coordinates of the key points Substitute into the fitted surface equation to calculate the predicted depth value .

[0200] Neighborhood statistics repair, specifically including:

[0201] (1) Determine the search direction based on whether the key point is a left point or a right point. , left point (Search to the right), right click (Search to the left).

[0202] (2) Take a certain number of pixels (12 in this embodiment) to the left and right of the row where the key point is located to form a sampling window, for a total of One point.

[0203] (3) Calculate the proportion of valid points within the window. and quantity .

[0204] If the conditions are met:

[0205] ;

[0206] Then Gaussian weighted spline fitting is used, with the weighting function being:

[0207] ;

[0208] in, For the first The distance from each sampling point to the target point This is the Gaussian kernel width parameter.

[0209] (4) If the current row does not meet the fitting condition, then search the adjacent rows above and below in order of distance from nearest to farthest, up to a maximum of searching. OK.

[0210] (5) If none of the rows meet the condition, then find the first row by following the search direction. For each non-zero point, the depth value of that point is used directly as the result.

[0211] This step can solve the problem of missing depth caused by high metal reflectivity, accurately restore the three-dimensional information of the weld, and provide reliable depth data for coordinate transformation.

[0212] Next, we will continue with the Eye-to-Hand coordinate transformation. See below. Figure 6 It includes the processes of distortion removal and normalization, coordinate system transformation, and output.

[0213] 2. Distortion Correction and Normalization

[0214] Read the camera intrinsic parameter matrix from the calibration file. :

[0215] ;

[0216] and distortion coefficient Homogeneous transformation matrix from camera to robot base .

[0217] pixel coordinates of key points The distortion correction function is called to process the data and obtain the normalized camera coordinates. :

[0218] ;

[0219] Special emphasis: The input image must be the original distorted image directly output by the camera. It cannot be pre-processed to remove distortion, otherwise it will lead to errors in coordinate calculation.

[0220] This step eliminates the impact of camera distortion on coordinate calculations, converting pixel coordinates into standardized coordinates in the camera coordinate system, thus laying the foundation for 3D point reconstruction.

[0221] 3. Coordinate system transformation and output

[0222] Calculate the 3D points in the camera coordinate system based on the normalized coordinates and depth values:

[0223] ;

[0224] Use a homogeneous transformation matrix to convert camera coordinates to robot base coordinates:

[0225] ;

[0226] in, It is a 4×4 homogeneous transformation matrix, containing Rotation matrix and 3×1 translation vector :

[0227] ;

[0228] Get the result The first three components are used as the three-dimensional coordinates in the robot's base coordinate system. .

[0229] This step accurately converts pixel coordinates into spatial coordinates that the robot can execute. Finally, the system outputs the base coordinates of the left and right welding key points to the robot control system. The robot plans the welding trajectory and performs the welding operation based on these coordinates. The coordinate values ​​retain 5 decimal places of precision.

[0230] Example 2

[0231] This embodiment provides a weld seam visual inspection and robot coordinate transformation system based on multimodal feature fusion. See [link to documentation]. Figure 1 It comprises three main modules: modeling, detection, and coordinate transformation; and three auxiliary modules: configuration management, log management, and file management. The functions of each module are described below:

[0232] (1) Modeling module

[0233] It includes a template trimming unit, a feature extraction unit, and an index management unit.

[0234] The modeling module loads parameters from the configuration file during system initialization, including the template size. Features such as network weight paths for feature extraction and root directory for storage.

[0235] The template clipping unit receives a list of teaching points and a reference image, performs a center clipping operation on each teaching point, handles boundary overflow cases, and returns a list of template tiles.

[0236] The feature extraction unit loads the pre-trained ResNet18 model and removes the classification layer. For each template, it sequentially performs grayscale conversion, CNN feature extraction (outputting a 512-dimensional normalized vector), and ORB feature extraction, and caches the calculation results in memory for subsequent matching.

[0237] The index management unit uses JSON format to manage template indexes, supporting multi-version history queries and quick access to the latest version.

[0238] (2) Detection module

[0239] It includes an image preprocessing unit, a neural network inference unit, a model backoff unit, a similarity matching unit, and a key point calculation unit.

[0240] The detection module loads the default neural network model (256-bit size) during initialization. If an alternative model path (512-bit size) is specified in the configuration file, the path information is recorded but not loaded yet, adopting a lazy loading strategy.

[0241] The image preprocessing unit performs image cropping, resizing, boundary filling (grayscale value 114), and normalization. Features such as [list of functions] are available, and all preprocessing parameters can be adjusted via configuration files.

[0242] The neural network inference unit encapsulates the model loading, input setup, inference execution, and output parsing operations of the NCNN framework, and supports GPU acceleration options. The output undergoes confidence thresholding and NMS processing.

[0243] The model rollback unit monitors the detection results. When no target is detected, it triggers the alternative model loading and re-detection process and records the actual model size information used.

[0244] The similarity matching unit maintains a reusable feature calculator instance, avoiding the duplication of model object creation. It supports two acceleration modes: batch SSIM computation (using a vectorized implementation of SSIM) and parallel candidate processing. Similarity is fused according to the formula... calculate.

[0245] The key point calculation unit calculates based on configurable scaling parameters. , and offset parameters Calculate the location of key welding points, and support using the upper or lower edge of the detection frame as a reference.

[0246] (3) Coordinate transformation module

[0247] It includes a depth repair unit, a distortion correction unit, and a coordinate transformation unit.

[0248] The coordinate transformation module loads the Eye-to-Hand calibration result file during initialization and parses the camera intrinsic parameter matrix. Distortion coefficient and transformation matrix If the file does not exist or is formatted incorrectly, a warning message will be logged and the coordinate transformation function will be disabled.

[0249] The depth repair unit encapsulates surface fitting (elliptical model). and circle model Two repair strategies are available: neighborhood statistics (Gaussian weighted spline interpolation) and neighborhood statistics (Gaussian weighted spline interpolation). You can choose which strategy to use or set it to not repair and directly return the original value.

[0250] The distortion correction unit wraps OpenCV The function call takes the pixel coordinates of the original distorted image as input and outputs the normalized camera coordinates.

[0251] The coordinate transformation unit implements matrix operations for homogeneous coordinate transformations. The output will retain five decimal places.

[0252] (4) Configuration Management Module: Responsible for loading and parsing YAML format configuration files, mapping configuration items to data class instances, and providing type checking and default value handling.

[0253] (5) Log management module: It uses a standard log library and supports two modes: console output and file rotation. Log level and file size limit can be configured.

[0254] (6) File Management Module: Responsible for saving detection result images and metadata files, supporting both synchronous and asynchronous saving modes, and realizing the function of organizing the directory structure by date and automatically cleaning up expired files.

[0255] Although embodiments of the present invention have been described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present invention, and all such changes and alterations shall not depart from the protection scope of the present invention.

Claims

1. A method for visual inspection of weld seams and robot coordinate transformation based on multimodal feature fusion, characterized in that, Includes the following steps: S1. Weld modeling stage: S11. Acquire reference images and corresponding depth images, and receive the coordinates of the weld center point from manual instruction; S12. Using the teaching point as the center, cut out the template block from the reference image; S13. Perform multimodal feature pre-computation on the template image block, including grayscale image conversion, convolutional neural network feature vector extraction, and ORB feature descriptor extraction; S14. Calculate the physical distance from the reference line to the first weld point based on the depth value at the corresponding position of the first weld point and the position of the preset reference line; S15. Save the template blocks and modeling metadata, and update the template index file corresponding to the product type; S2. Weld inspection stage: S21. Receive the runtime image and obtain the corresponding teaching point coordinates based on the product type and weld index; S22. Crops out the inference patch from the input image, inputs it into the neural network detection model for weld detection, and if no target is detected, it is cropped by shifting left and right along the teaching point coordinates and then re-detected; S23. Enable dynamic model size fallback mechanism. When the model with the default size fails to detect the target, automatically switch to a model with a larger input size for secondary detection. S24. When multiple candidate targets are detected, the multimodal fusion similarity between each candidate target and the template block is calculated, and the candidate target with the highest similarity is selected as the weld detection result. S25. Calculate the pixel coordinates of key welding points based on the weld inspection results; S3. Coordinate Transformation Stage: S31. Obtain the depth value of the welding key point location; S32. Perform distortion correction on pixel coordinates and combine with depth values ​​to recover the 3D points in the camera coordinate system; S33. Convert the camera coordinates to robot base coordinates using the Eye-to-Hand homogeneous transformation matrix and output them.

2. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S12, a square image block of a preset size is cropped from the reference image according to the following formula as a template block. When the cropped area exceeds the image boundary, it is filled with black pixels: ； in, For reference image, Here are the coordinates of the teaching point. Template size The template block obtained by cutting.

3. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S13, the grayscale conversion is to convert the template image from the RGB color space to a grayscale image, and the conversion formula is: ； in, These are the pixel values ​​for the red, green, and blue color channels, respectively. The converted grayscale value; The convolutional neural network feature vector extraction involves using a pre-trained convolutional neural network to extract the depth feature vector of the template image and then normalizing the feature vector. The ORB feature descriptor extraction method uses the ORB algorithm to extract key points and their descriptors from the template image.

4. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S14, the physical distance from the reference line to the first weld point is calculated as follows: ； in, Pixel distance, For depth value, The scaling factor is the calibrated value. Image width, This represents the actual physical distance.

5. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S22, cropping the inference patch from the input image includes: Using the pixel x-coordinate of the center point of the input image as the vertical coordinate of the cropped image block and the pixel y-coordinate of the teaching point as the horizontal coordinate of the cropped image block, a pre-defined inference image block is cropped from the input image.

6. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S24, calculating the multimodal fusion similarity between each candidate target and the template image patch includes: The SSIM algorithm is used to calculate the structural similarity between candidate targets and template tiles: ； Where SSIM(x,y) represents the structural similarity between the candidate target and the template tile; and These are the mean values ​​of the candidate target and the template tile, respectively. and These represent the variances of the candidate targets and the template tiles, respectively. For covariance; and It is a constant; The same convolutional neural network used in the modeling phase is used to extract feature vectors of candidate targets, and the cosine similarity between these vectors and the feature vectors of the template patch is calculated as the deep feature similarity. ； in, The template patch feature vector, S represents the feature vector of the candidate target; cnn For deep feature similarity; Extract the ORB feature descriptors of candidate targets and match them with the ORB descriptors of template patches. Calculate local feature similarity based on the number of matching point pairs and the matching distance. ； in, The average Hamming distance for matching point pairs, To match the number of point pairs, and The number of key points for candidate targets and template tiles, respectively; S orb For local feature similarity; The structural similarity, deep feature similarity, and local feature similarity are weighted and fused: ； in, , , Let be the weighting coefficient, satisfying .

7. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S25, the key welding points include the left welding point and the right welding point. The coordinates of the key welding points are calculated as follows: ； in, The coordinates of the left edge of the detection box. For the detection frame width, and This is the preset horizontal scaling factor. The ordinate of the reference edge, This is the vertical offset.

8. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S31, when the depth value of the welding key point is missing, an adaptive surface fitting method is used for depth repair, including: Collect valid depth value points within a preset search range around the target point; Outliers are filtered out using the median absolute deviation method, retaining only those that meet the criteria. The data points of the condition, where, For the first The depth value of each point, The median of all depth values. This represents the absolute deviation of the median. Elliptical surface models should be used preferentially. Perform least-squares fitting, and if the fitting fails, revert to the circular surface model. ;in, The parameters are those of the model to be fitted. For pixel coordinates, This is the depth value; Substitute the target point coordinates into the fitted surface equation to calculate the predicted depth value.

9. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S31, when the depth value of the welding critical point is missing, the depth value is estimated using a neighborhood statistical method: Determine the search direction based on whether the key point is on the left or right; A certain number of pixels are taken to the left and right of the row where the key point is located to form a sampling window, and the proportion of effective points within the window is calculated. When the proportion of valid points meets the threshold condition, Gaussian weighted spline interpolation is used to estimate the depth value of the target point, and the weighting function is: ;in, For the first The distance from each sampling point to the target point The Gaussian kernel width parameter; If the condition is not met, search the neighboring rows or take the non-zero depth value of the preset sequence number along the search direction.

10. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 1, characterized in that, In step S32, distortion correction is performed on the pixel coordinates, and the 3D points in the camera coordinate system are recovered by combining the depth values, including: The original distorted image directly output by the camera is used as the input image; Using a pre-calibrated camera intrinsic matrix and distortion coefficient The distortion correction function is called to convert the original pixel coordinates into normalized camera coordinates. : ； in, These are the original pixel coordinates; Then, the 3D points in the camera coordinate system are recovered based on the normalized coordinates and depth values: ； in, This represents the depth value.

11. The weld visual inspection and robot coordinate transformation method based on multimodal feature fusion as described in claim 10, characterized in that, In step S33, the camera coordinates are converted to robot base coordinates using the Eye-to-Hand homogeneous transformation matrix, including: Finally, the homogeneous transformation matrix obtained from the Eye-to-Hand calibration is used to transform the 3D points in the camera coordinate system to the robot base coordinate system: ； in, It is a 4×4 homogeneous transformation matrix, containing rotation matrices. Translation vector : ； P base For the robot's base coordinates.

12. A weld seam visual inspection and robot coordinate transformation system based on multimodal feature fusion, characterized in that, It includes a modeling module, a detection module, and a coordinate transformation module; The modeling module includes: The template cropping unit is used to crop template blocks from the reference image based on the coordinates of the teaching points; The feature extraction unit is used to perform grayscale conversion, convolutional neural network feature extraction, and ORB feature extraction on the template image. The index management unit is used to maintain the storage structure and version index of template files; The detection module includes: The image preprocessing unit is used to perform image cropping, resizing, and normalization. The neural network inference unit is used to perform object detection and output candidate objects; The model fallback unit is used to automatically switch to an alternative model when the default model does not detect the target; The similarity matching unit is used to calculate the multimodal fusion similarity between each candidate target and the template image patch when there are multiple candidate targets, and select the candidate target with the highest similarity as the detection result. The key point calculation unit is used to calculate the coordinates of welding key points based on the position of the detection frame. The coordinate transformation module includes: Depth repair unit, used to recover missing depth values ​​using surface fitting or neighborhood statistics methods; The distortion correction unit is used to convert pixel coordinates into normalized coordinates using calibration parameters; The coordinate transformation unit is used to convert camera coordinates to robot base coordinates using an Eye-to-Hand homogeneous transformation matrix.

Images