Non-destructive testing robot weld seam automatic tracking system based on multi-modal sensing

Abstract

The present application relates to the technical field of industrial nondestructive testing, in particular to a nondestructive testing robot welding seam automatic tracking system based on multi-modal sensing. It comprises a data acquisition module, a data preprocessing module, a data fusion module, a trajectory generation module and a motion control module. Point cloud data is obtained by a laser scanner, image information is obtained by an RGB-D camera, and attitude data is obtained by an inertial measurement unit. After denoising, downsampling and correction processing of multi-modal data, data fusion is realized by Kalman filtering, and continuous welding seam spatial trajectory points are output. The trajectory generation module generates end pose sequences through three-dimensional B-spline fitting and surface normal estimation, and adopts quaternion interpolation for attitude smoothing processing. The motion control module combines differential path keeping and hybrid position-force control to ensure that the robot accurately tracks along the welding seam centerline and maintains constant contact force, while having process correction function to improve the stability and reliability of the system in complex welding seam environment.

Description

Technical Field

[0001] This invention relates to the field of industrial nondestructive testing technology, specifically to an automatic weld seam tracking system for nondestructive testing robots based on multimodal sensing. Background Technology

[0002] Weld quality is crucial for the safe operation of industrial equipment and pipelines, making non-destructive testing (NDT) of welds essential in industrial production. However, existing ultrasonic testing robots suffer from low efficiency and complex operation in practical applications. For example, ultrasonic testing robots for long-distance pipelines typically require installation along a track along the pipeline, involving segment-by-segment disassembly and reinstallation, resulting in long inspection cycles, high labor intensity, and low efficiency.

[0003] Furthermore, traditional crawling inspection robots typically rely on preset trajectories for weld seam tracking, making it difficult to adapt to geometric deformations or installation errors that occur during construction or use. Especially for curved weld seams, current technology has not yet achieved real-time trajectory correction, making it difficult for the robot to move accurately along the weld centerline, thus affecting inspection accuracy. On the other hand, the unstable coupling state of ultrasonic testing probes leads to fluctuations in the signal-to-noise ratio, and the detection data is prone to poor coupling, thereby reducing the reliability of the inspection results. Current technology struggles to simultaneously meet the requirements of weld seam trajectory adaptability, automatic tracking of curved weld seams, and stable probe coupling. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an automatic weld seam tracking system for non-destructive testing robots based on multimodal sensing, aiming to solve the problems of low accuracy and unstable trajectory tracking in traditional weld seam inspection.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an automatic weld seam tracking system for non-destructive testing robots based on multimodal sensing, comprising:

[0006] The data acquisition module is used to acquire point cloud data through a laser scanner, acquire RGB-D image data through an RGB-D camera, and acquire attitude data through an inertial measurement unit.

[0007] The data preprocessing module is used to preprocess point cloud data, RGB-D image data and pose data to obtain preprocessed multimodal data;

[0008] The data fusion module is used to synchronize the preprocessed multimodal data and output continuous weld space trajectory points through the Kalman fusion algorithm;

[0009] The trajectory generation module is used to perform three-dimensional B-spline fitting based on the arc length parameterization of the weld space trajectory points, and combine it with the surface normal estimated by the point cloud to generate the end pose sequence to obtain the target motion trajectory.

[0010] The motion control module is used to drive the robot to perform inspection tasks according to the target motion trajectory, including differential path holding control based on lateral deviation and hybrid position-force control of the robotic arm end effector based on feedforward-PID and force feedback, so as to realize automatic weld tracking and non-destructive testing.

[0011] Furthermore, the steps for obtaining preprocessed multimodal data include:

[0012] The point cloud data was denoised, downsampled, and calibrated. The random sampling consistency method was used to fit the bevel surface to extract the feature points of the weld centerline.

[0013] Edge detection, depth map denoising, and feature point matching are performed on RGB-D image data to supplement weld edge feature information;

[0014] The attitude data is subjected to zero-bias correction and low-pass filtering to obtain stable attitude angle and angular velocity information.

[0015] Furthermore, the step of outputting continuous weld space trajectory points includes:

[0016] The preprocessed point cloud data and image data are spatially aligned and time-synchronized to generate fused laser-visual multimodal data.

[0017] The Kalman filter algorithm is used to fuse the laser-vision multimodal data and the attitude data collected by the inertial measurement unit to estimate the three-dimensional position and attitude state of the weld centerline in real time.

[0018] Through the state update and prediction mechanism of Kalman filtering, continuous weld space trajectory points are output.

[0019] Furthermore, the steps to obtain the target's trajectory are as follows:

[0020] The continuous weld space trajectory points output by the Kalman filter are sorted and parameterized according to the arc length parameter;

[0021] Based on parametric trajectory points, a smooth and continuous weld trajectory curve is generated using a three-dimensional B-spline curve fitting method.

[0022] The local normal information of the weld surface is estimated using point cloud data, and the corresponding end pose sequence is generated by combining the fitted curve.

[0023] Furthermore, when generating the end pose sequence, the trajectory generation module employs a quaternion-based pose smoothing method to perform spherical linear interpolation on adjacent poses to avoid abrupt changes in the robot arm's pose and ensure the continuity and stability of the trajectory pose.

[0024] Furthermore, the steps of differential path holding control include:

[0025] The lateral deviation between the weld centerline and the robot's central axis is measured in real time using a laser sensor.

[0026] The deviation is calculated using a PID control algorithm to generate a speed difference.

[0027] Based on the calculated speed difference, adjust the PWM duty cycle of the left and right wheels to dynamically adjust the speed ratio of the left and right wheels;

[0028] The stability of the adjusted wheel speed ratio was verified to ensure that the robot accurately tracks the center line of the weld.

[0029] Furthermore, the steps of hybrid position-force control include:

[0030] The spatial deviation of the weld center is provided by the laser vision module;

[0031] The deviation is solved by using the inverse kinematics of the robotic arm, the angles of each joint are calculated, the positional spatial deviation is compensated, and the positional adjustment of the probe center to align with the weld centerline is achieved.

[0032] The probe contact force feedback value is collected in real time by a six-dimensional force sensor and compared with the preset target force value.

[0033] By using a PID controller to adjust the Z-axis displacement of the robotic arm, a constant contact force can be achieved.

[0034] Furthermore, the hybrid position-force control adaptively adjusts the control strategy according to the weld type. When a planar weld is detected, a constant force control mode is adopted; when a curved weld is detected, a hybrid control mode combining position adjustment and force adjustment is adopted.

[0035] Furthermore, the system also includes a process error correction module, the implementation logic of which is as follows:

[0036] When the laser signal is lost, switch to pure vision mode and detect that the robot's movement speed has dropped to 50% of the preset speed. When the laser signal is restored and lasts for 2 seconds, return to the initial state.

[0037] When the contact force exceeds 50% of the preset initial value, the robotic arm immediately retracts 3mm and recouples. After the force value stabilizes at the preset initial value for 1 second, it returns to the initial state.

[0038] If communication is interrupted, the robot immediately stops moving and is manually remotely returned to its original position by an external controller.

[0039] This invention also proposes an automatic weld seam tracking method for nondestructive testing robots based on multimodal sensing, comprising:

[0040] The data acquisition module is used to acquire point cloud data through a laser scanner, acquire RGB-D image data through an RGB-D camera, and acquire attitude data through an inertial measurement unit.

[0041] Point cloud data, RGB-D image data, and pose data are preprocessed to obtain preprocessed multimodal data;

[0042] The preprocessed multimodal data is synchronized, and continuous weld space trajectory points are output through the Kalman fusion algorithm.

[0043] Based on the arc length parameterization of the weld space trajectory points, a 3D B-spline fitting is performed, and the end pose sequence is generated by combining the surface normal estimated by the point cloud to obtain the target motion trajectory.

[0044] The robot is driven to perform inspection tasks based on the target motion trajectory, including differential path holding control based on lateral deviation and hybrid position-force control of the robotic arm end effector based on feedforward-PID and force feedback, so as to achieve automatic weld tracking and non-destructive testing.

[0045] This invention provides an automatic weld seam tracking system for non-destructive testing robots based on multimodal sensing. It offers the following advantages:

[0046] 1. This invention achieves high-precision, real-time sensing of the weld centerline through multimodal sensing fusion of a laser scanner, an RGB-D camera, and an inertial measurement unit (IMU). The laser provides precise geometric information, the vision system supplements edge features, and the IMU provides stable attitude data, enabling the system to stably acquire continuous spatial trajectories even under complex weld shapes, significantly improving the accuracy and reliability of weld detection.

[0047] 2. This invention employs a Kalman filter algorithm to fuse multimodal data, achieving continuous output of weld seam spatial trajectory points. Through a state prediction and update mechanism, it effectively suppresses the impact of sensor noise and short-term data loss, improving the stability and continuity of trajectory estimation. This provides a reliable foundation for subsequent trajectory generation and motion control, ensuring precise robot tracking along the weld seam.

[0048] 3. This invention generates a smooth end pose sequence by fitting three-dimensional B-splines and estimating surface normals, and performs pose smoothing by combining quaternion-based pose interpolation. This effectively avoids abrupt changes in the robot arm's pose, improves the continuity and stability of the motion trajectory, and enables the probe to be accurately aligned with the weld center, ensuring uniform contact force and stable detection quality during non-destructive testing.

[0049] 4. This invention combines differential path holding control with hybrid position-force control and introduces a process error correction mechanism to achieve coordinated control of the robot's mobile platform and robotic arm. In cases of complex weld seams or sensor malfunctions, the system can automatically adjust wheel speed or force feedback to ensure the accuracy and safety of weld seam tracking, improve overall inspection efficiency, and significantly enhance the system's reliability and applicability. Attached Figure Description

[0050] Figure 1 The structural diagram of the automatic weld seam tracking system for non-destructive testing robots based on multimodal sensing provided by the present invention;

[0051] Figure 2 A flowchart illustrating the steps for outputting continuous weld space trajectory points provided by this invention;

[0052] Figure 3 The flowchart of the differential path holding control steps provided by the present invention;

[0053] Figure 4 A flowchart of the mixed position-force control steps provided by the present invention;

[0054] Figure 5 The flowchart shows the automatic tracking method for weld seams in a non-destructive testing robot based on multimodal sensing provided by the present invention. Detailed Implementation

[0055] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example 1

[0057] This invention provides an automatic weld seam tracking system for nondestructive testing robots based on multimodal sensing, such as... Figure 1 As shown, it includes:

[0058] The data acquisition module is used to acquire point cloud data through a laser scanner, acquire RGB-D image data through an RGB-D camera, and acquire attitude data through an inertial measurement unit.

[0059] Specifically, the data acquisition module includes a laser scanner, an RGB-D camera, and an inertial measurement unit (IMU). These devices are mounted on a mobile platform, which employs a chassis structure combining independently driven tracks and magnetic wheels. The left and right tracks are controlled by independent drive motors, enabling differential steering. The laser scanner, mounted at the front of the chassis, operates at a wavelength of 650nm and generates two-dimensional contour data of the weld and surrounding surfaces. This data is then accumulated through the robot's movement to obtain a three-dimensional point cloud of the weld. The RGB-D camera, with a resolution of 1280×1024, simultaneously acquires color and depth images of the weld area, supplementing the laser scanner's weld feature information under conditions of occlusion or reflection. The inertial measurement unit measures the robot's attitude angles and angular velocity during movement, providing attitude references for multi-sensor data fusion. The attitude angles include roll, pitch, and yaw angles.

[0060] The data preprocessing module is used to preprocess point cloud data, RGB-D image data and pose data to obtain preprocessed multimodal data;

[0061] Furthermore, the steps for obtaining preprocessed multimodal data include:

[0062] The point cloud data was denoised, downsampled, and calibrated. The random sampling consistency method was used to fit the bevel surface to extract the feature points of the weld centerline.

[0063] Edge detection, depth map denoising, and feature point matching are performed on RGB-D image data to supplement weld edge feature information;

[0064] The attitude data is subjected to zero-bias correction and low-pass filtering to obtain stable attitude angle and angular velocity information.

[0065] Specifically, the raw point cloud data output by the line laser scanner first undergoes statistical filtering for noise reduction. This involves calculating the average neighborhood distance of each point and removing outliers that deviate from the mean by more than two standard deviations, thus eliminating isolated points caused by light spot reflection, surface contamination, or sensor jitter. Subsequently, a voxel mesh downsampling method is used to divide the point cloud into fixed-size cubic voxels, with the centroid of each voxel representing that voxel. This reduces the number of points and computational load while preserving the weld structure characteristics. After downsampling, the point cloud data is transformed from the laser scanner coordinate system to the robot body coordinate system using an extrinsic calibration matrix, ensuring consistency between data from different sensors. Finally, a random sampling consistency plane fitting method is used to fit the bevel plane of the weld in the point cloud, removing background points that do not belong to the bevel plane. Based on the fitted bevel geometry, the weld centerline feature points are extracted, providing an accurate three-dimensional geometric basis for subsequent trajectory calculations.

[0066] Color images and depth maps acquired from an RGB-D camera are first processed using the Sobel or Canny operator for edge detection of the weld region, yielding a preliminary outline of the weld in the image. For the depth map data, median filtering is used to remove depth noise caused by reflection, scattering, or sensor ranging errors, while bilateral filtering preserves depth gradient edge features. Then, ORB or SIFT algorithms are used to match feature points from the color image and depth map, associating visually detected weld edge points with point cloud feature points to supplement weld feature information missing during laser scanning under specific poses or occlusion conditions.

[0067] The raw attitude data acquired from the inertial measurement unit (IMU) first undergoes zero-bias correction. This involves collecting the average acceleration and angular velocity over a certain period while the robot is stationary. This average value is used as the zero-bias value and subtracted from subsequent data to eliminate the influence of sensor drift. Next, a first-order low-pass filter or complementary filtering algorithm is used to smooth the attitude angle and angular velocity signals, suppressing high-frequency noise and preserving the true motion trend, thereby obtaining stable attitude information.

[0068] Through the above steps, the system obtains multimodal data that has undergone noise suppression, feature extraction, and coordinate alignment. This preserves the detailed features of the weld space geometry and ensures the spatiotemporal consistency between different data sources, providing high-quality input data for subsequent multi-sensor fusion and trajectory generation.

[0069] The data fusion module is used to synchronize the preprocessed multimodal data and output continuous weld space trajectory points through the Kalman fusion algorithm;

[0070] Furthermore, the steps for outputting continuous weld space trajectory points are as follows: Figure 2 As shown, it includes:

[0071] The preprocessed point cloud data and image data are spatially aligned and time-synchronized to generate fused laser-visual multimodal data.

[0072] The Kalman filter algorithm is used to fuse the laser-vision multimodal data and the attitude data collected by the inertial measurement unit to estimate the three-dimensional position and attitude state of the weld centerline in real time.

[0073] Through the state update and prediction mechanism of Kalman filtering, continuous weld space trajectory points are output.

[0074] Specifically, firstly, the preprocessed point cloud data and RGB-D image data undergo spatial coordinate system transformation. The coordinates of image feature points in the RGB-D camera coordinate system are transformed to the laser scanner coordinate system using an extrinsic calibration matrix, which is obtained from a calibration plate or a known feature object. Then, the ICP iterative nearest point algorithm from the point cloud library is used to register the laser-scanned point cloud with the dense point cloud generated by backprojection of the RGB-D depth map, thereby reducing spatial deviations caused by installation errors and sensor drift.

[0075] To ensure the consistency of multi-sensor data at the fusion time, hardware-triggered synchronization or timestamp alignment is used to unify laser scan data, RGB-D data, and IMU data to the robot system clock. For sensor data with different sampling frequencies, linear interpolation is used to fill in missing data points on the time axis, ensuring that the fusion input is based on the same time reference.

[0076] Spatially and temporally aligned laser-vision multimodal data is used as the observation input to a Kalman filter, and IMU attitude data is used as the predictor for state estimation. The system state vector is defined as follows:

[0077] ;

[0078] in, This represents the three-dimensional position of the weld centerline in the robot coordinate system. These are roll angle, pitch angle, and yaw angle, respectively. The velocity components at the weld joint are given. The prediction model obtains prior estimates of position and attitude by integrating the IMU angular velocity and linear acceleration.

[0079] ;

[0080] in, Represents the state transition matrix. Represents the control matrix. Indicates control input, This represents process noise. The observation model uses the weld center point coordinates and RGB-D edge features from the laser-vision fusion point cloud to calculate the pose as the observation vector. ,in Represents the observation matrix. This represents observation noise. The Kalman filter performs a two-stage iterative process: prediction and update. In the prediction stage, prior values ​​of the weld centerline position and orientation are calculated based on IMU data. In the update stage, the laser-vision fusion result is weighted and fused with the prior values ​​to obtain the optimal estimate. During the Kalman filter iteration process, the system outputs the position in the state vector in real time. As spatial trajectory points of continuous weld seams.

[0081] Through the above processing, the obtained weld space trajectory points have high accuracy and continuity in both three-dimensional position and orientation, which can effectively reflect the spatial morphology of the real weld and provide reliable basic data for subsequent three-dimensional curve fitting and end pose generation.

[0082] The trajectory generation module is used to perform three-dimensional B-spline fitting based on the arc length parameterization of the weld space trajectory points, and combine it with the surface normal estimated by the point cloud to generate the end pose sequence to obtain the target motion trajectory.

[0083] Furthermore, the steps to obtain the target's trajectory are as follows:

[0084] The continuous weld space trajectory points output by the Kalman filter are sorted and parameterized according to the arc length parameter;

[0085] Based on parametric trajectory points, a smooth and continuous weld trajectory curve is generated using a three-dimensional B-spline curve fitting method.

[0086] The local normal information of the weld surface is estimated using point cloud data, and the corresponding end pose sequence is generated by combining the fitted curve.

[0087] Specifically, the continuous weld space trajectory points output by the Kalman filter Sort by acquisition order and calculate the three-dimensional Euclidean distance between adjacent trajectory points. The total arc length of the trajectory is obtained by accumulating the distance. And the cumulative distance of each trajectory point Normalized to parameters This yields the arc-length parameterized trajectory point set. .

[0088] Selecting the B-spline order and node vectors, fitting is performed based on the arc-length parameterized point set, respectively for... Perform independent spline fitting to obtain the set of control points for the 3D B-spline curve. The least squares method is used to solve for the control point positions during the fitting process, and a smoothing factor can be introduced. Noise interference is suppressed, ultimately generating a smooth three-dimensional weld seam trajectory curve. ,in This represents the spatial path along which the robot's end effector moves along the weld seam.

[0089] From the original point cloud data, a local point set within a certain radius near the fitted curve is extracted. The normal vector of the local point cloud is calculated using the normal estimation method of the PCL library, based on k-nearest neighbor search. The normal vector is corrected for directional consistency to ensure the continuity of the normal vector along the trajectory direction.

[0090] According to the fitted curve and the corresponding normal vector Determine the attitude of the end effector, where the position part is taken from... The attitude component is defined by the normal vector as the Z-axis direction at the end point, and the trajectory tangent vector... The X-axis direction, representing the endpoint, is used to determine the Y-axis direction via cross product, forming an orthogonal attitude matrix. Combine position and orientation into a homogeneous transformation matrix:

[0091] ;

[0092] With fixed step size Sampling is performed to output a continuous end-effector pose sequence. This is for the motion control module to call directly.

[0093] Through the above steps, discrete weld trajectory points that may contain noise can be transformed into a smooth and continuous three-dimensional path, and a precise end pose sequence can be generated by combining local surface normal vectors, thereby ensuring the stable fit and correct posture of the robot detection probe along the weld trajectory.

[0094] Furthermore, when generating the end pose sequence, the trajectory generation module employs a quaternion-based pose smoothing method to perform spherical linear interpolation on adjacent poses to avoid abrupt changes in the robot arm's pose and ensure the continuity and stability of the trajectory pose.

[0095] Specifically, each end pose in the trajectory is converted from a rotation matrix to a quaternion representation for interpolation calculation. Then, for two adjacent pose quaternions, the angle between them is calculated, and when the angle is large, the shortest path is selected for interpolation to prevent pose flipping. Next, several intermediate pose points are generated between adjacent poses using a spherical linear interpolation method according to a preset interpolation ratio, thus making the pose transition smoother. When the difference between adjacent poses is extremely small, linear interpolation is used to avoid numerical instability.

[0096] The generated intermediate interpolated quaternions are normalized before use to ensure that the rotational attitude they represent is valid and free of accumulated errors. Finally, the smoothed quaternions are converted back into rotation matrices and combined with the corresponding trajectory position coordinates to form a continuous and smooth end-effector pose sequence. This method effectively avoids shocks and trajectory instability caused by sudden changes in the robotic arm's attitude, ensuring stable contact and consistent attitude of the detection probe during weld tracking and non-destructive testing.

[0097] By using the above-mentioned attitude smoothing method based on quaternion interpolation, the present invention can significantly improve the continuity and stability of the attitude change of the robotic arm end effector, avoid the shaking or detachment of the detection probe from the weld seam caused by sudden attitude changes, and effectively ensure the accuracy of weld seam tracking and the quality of non-destructive testing.

[0098] The motion control module is used to drive the robot to perform inspection tasks according to the target motion trajectory, including differential path holding control based on lateral deviation and hybrid position-force control of the robotic arm end effector based on feedforward-PID and force feedback, so as to realize automatic weld tracking and non-destructive testing.

[0099] Furthermore, the steps of differential path holding control are as follows: Figure 3 As shown, it includes:

[0100] The lateral deviation between the weld centerline and the robot's central axis is measured in real time using a laser sensor.

[0101] The deviation is calculated using a PID control algorithm to generate a speed difference.

[0102] Based on the calculated speed difference, adjust the PWM duty cycle of the left and right wheels to dynamically adjust the speed ratio of the left and right wheels;

[0103] The stability of the adjusted wheel speed ratio was verified to ensure that the robot accurately tracks the center line of the weld.

[0104] Specifically, the laser sensor scans the weld seam area in front in real time, and calculates the lateral deviation Δy of the weld seam centerline in the robot coordinate system using a weld seam feature extraction algorithm. This deviation reflects the relative positional difference between the weld seam center and the robot's central axis. Then, Δy is input to the PID controller, which calculates the corresponding speed difference Δv based on proportional, integral, and derivative parameters to correct the speed difference between the two tracks. Next, the control system dynamically adjusts the duty cycle of the PWM signals of the left and right drive wheels based on Δv to compensate for the wheel speed difference, thereby correcting the robot's direction of travel and gradually bringing it closer to and maintaining it on the weld seam centerline. Finally, during movement, the system verifies the stability of the path adjustment using odometer and IMU attitude data. When the robot's attitude changes too rapidly or exhibits an oscillating trend, the system automatically adjusts the PID parameters or reduces the rate of change of Δv to ensure smooth operation and weld seam tracking accuracy.

[0105] Differential path keeping control detects the lateral deviation between the weld centerline and the robot's central axis in real time, and dynamically adjusts the speed ratio of the left and right wheels using a PID algorithm. This enables the robot to correct yaw in a timely manner during movement and move stably along the weld centerline, thereby significantly improving the accuracy of automatic weld tracking and the stability of path keeping.

[0106] Furthermore, the steps of mixed position-force control are as follows: Figure 4 As shown, it includes:

[0107] The spatial deviation of the weld center is provided by the laser vision module;

[0108] The deviation is solved by using the inverse kinematics of the robotic arm, the angles of each joint are calculated, the positional spatial deviation is compensated, and the positional adjustment of the probe center to align with the weld centerline is achieved.

[0109] The probe contact force feedback value is collected in real time by a six-dimensional force sensor and compared with the preset target force value.

[0110] By using a PID controller to adjust the Z-axis displacement of the robotic arm, a constant contact force can be achieved.

[0111] Specifically, the laser vision module acquires 3D point cloud and image information of the weld area in real time. Through weld feature extraction and spatial positioning algorithms, it calculates the spatial deviation of the weld center relative to the coordinate system of the robotic arm's end effector. Then, this spatial deviation is input into the robotic arm kinematics solution module. Based on the robotic arm's DH parameter model, forward and inverse kinematic equations are established. The Jacobi inverse matrix method is used to iteratively solve for the joint angle changes that can compensate for the deviation, thereby generating the target angles of each joint and achieving end-effector position modulation, so that the probe center is accurately aligned with the weld centerline.

[0112] Based on position modulation, a six-dimensional force sensor measures the contact force between the probe and the weld surface in real time, and compares the force values ​​in each direction, especially the contact force in the Z-axis direction, with the preset target force value to obtain a force error signal. Subsequently, the PID controller calculates the displacement compensation in the Z-axis direction based on this error signal and adds it to the motion command of the robotic arm end effector to achieve dynamic adjustment of the contact force.

[0113] By using a hybrid position-force control method, the probe at the end of the robotic arm can continuously maintain alignment with the weld centerline in three-dimensional space. At the same time, force feedback adjustment ensures that the contact force between the probe and the weld surface remains constant, thereby ensuring the accuracy of the detection path and the stability of the detection signal even under complex weld morphology, and improving the reliability and continuity of non-destructive testing.

[0114] Furthermore, the hybrid position-force control adaptively adjusts the control strategy according to the weld type. When a planar weld is detected, a constant force control mode is adopted; when a curved weld is detected, a hybrid control mode combining position adjustment and force adjustment is adopted.

[0115] By adaptively adjusting the control strategy, the probe can accurately follow the weld path while maintaining a stable contact force, thereby improving the reliability and data quality of non-destructive testing.

[0116] Furthermore, the system also includes a process error correction module, the implementation logic of which is as follows:

[0117] When the laser signal is lost, switch to pure vision mode and detect that the robot's movement speed has dropped to 50% of the preset speed. When the laser signal is restored and lasts for 2 seconds, return to the initial state.

[0118] When the contact force exceeds 50% of the preset initial value, the robotic arm immediately retracts 3mm and recouples. After the force value stabilizes at the preset initial value for 1 second, it returns to the initial state.

[0119] If communication is interrupted, the robot immediately stops moving and is manually remotely returned to its original position by an external controller.

[0120] Process error correction can respond in real time to emergencies such as sensor failure, abnormal force, or communication interruption during the detection process. By automatically switching working modes, adjusting motion parameters, or taking emergency stops, it can prevent equipment damage and decrease detection accuracy, ensuring the safety, stability, and task continuity of the system under complex working conditions.

[0121] Example 2

[0122] This invention also proposes an automatic tracking method for weld seams in a non-destructive testing robot based on multimodal sensing, as shown in Figure 5. In a shipyard's steel structure manufacturing workshop, this solution is applied to the ultrasonic non-destructive testing of vertical weld seams. After the robot starts, a line laser scanner mounted at the front of the mobile platform acquires three-dimensional point cloud data of the weld seam area with an accuracy of ±0.1mm. Simultaneously, an RGB-D camera on top acquires color images and depth information at a resolution of 1280×1024, and an IMU provides platform attitude angle and angular velocity data at a frequency of 100Hz.

[0123] The acquired multimodal data were preprocessed as follows: point cloud data was first downsampled by voxel mesh filtering, then isolated noise points were removed by statistical filtering, and the coordinate system was aligned with the workpiece reference system; then, the random sampling consistency method was used to fit the bevel surface and extract the three-dimensional feature points of the weld centerline; RGB-D data were subjected to edge detection and depth denoising to extract supplementary weld edge features; IMU data were subjected to zero bias correction and low-pass filtering to obtain stable and reliable attitude information.

[0124] The processed point cloud and image data undergo extrinsic parameter calibration and spatial alignment, and are synchronized using timestamps to form fused laser-vision data. This data, along with IMU attitude information, is then input into an extended Kalman filter algorithm. The filter estimates the 3D position and attitude of the weld centerline in real time during the prediction-update loop and outputs a continuous, smooth sequence of trajectory points.

[0125] The trajectory generation module parametrically sorts the trajectory points output by the Kalman filter according to the arc length, and fits them with a 3D B-spline curve to obtain a continuous and smooth weld curve. At the same time, it combines the local surface normal estimated by the point cloud to generate the corresponding end pose sequence, and uses the quaternion spherical linear interpolation method to smooth adjacent poses to avoid abrupt changes in the robot arm's pose.

[0126] During the motion control phase, the mobile platform uses differential path holding control to measure the lateral deviation Δy between the weld centerline and the central axis in real time. The speed difference Δv is calculated by PID control, and the PWM duty cycle of the left and right tracks is adjusted to ensure that the platform always travels along the weld centerline. The robotic arm end effector performs hybrid position-force control. First, it compensates for the spatial position deviation by calculating the joint angle through inverse kinematics to achieve precise alignment of the probe with the weld center. Then, it monitors the contact force between the probe and the workpiece surface in real time through a six-dimensional force sensor and uses PID control to adjust the Z-axis displacement to ensure that the contact force remains constant during the detection process.

[0127] In this embodiment, the robot can stably follow the weld path and maintain a constant contact force between the detection probe and the weld throughout the weld inspection process, achieving the preset detection accuracy. It also has the ability to correct errors in case of laser signal loss, abnormal force value, or communication interruption, ensuring the safety and reliability of vertical weld inspection tasks in complex workshop environments.

[0128] In this embodiment, the robot can stably follow the weld path and maintain a constant contact force between the detection probe and the weld throughout the weld inspection process, achieving the preset detection accuracy. It also has the ability to correct errors in case of laser signal loss, abnormal force value, or communication interruption, ensuring the safety and reliability of vertical weld inspection tasks in complex workshop environments.

[0129] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A non-destructive testing robot weld seam automatic tracking system based on multi-modal sensing, characterized in that, include: The data acquisition module is used to acquire point cloud data through a laser scanner, acquire RGB-D image data through an RGB-D camera, and acquire attitude data through an inertial measurement unit. The data preprocessing module is used to preprocess point cloud data, RGB-D image data and pose data to obtain preprocessed multimodal data; The data fusion module is used to synchronize the preprocessed multimodal data and output continuous weld space trajectory points through the Kalman filter algorithm; The trajectory generation module is used to perform three-dimensional B-spline fitting based on the arc length parameterization of the weld space trajectory points, and combine it with the surface normal estimated by the point cloud to generate the end pose sequence to obtain the target motion trajectory. The motion control module is used to drive the robot to perform inspection tasks according to the target motion trajectory, including differential path holding control based on lateral deviation and hybrid position-force control of the robotic arm end based on feedforward-PID and force feedback, so as to realize automatic weld tracking and non-destructive testing. The steps to obtain the target's trajectory are as follows: The continuous weld space trajectory points output by the Kalman filter are sorted and parameterized according to the arc length parameter; Based on parametric trajectory points, a smooth and continuous weld trajectory curve is generated using a three-dimensional B-spline curve fitting method. The local normal information of the weld surface is estimated using point cloud data, and the corresponding end pose sequence is generated by combining the fitted curve. When generating the end pose sequence, the trajectory generation module uses a quaternion-based pose smoothing method to perform spherical linear interpolation on adjacent poses to avoid abrupt changes in the robot arm's pose and ensure the continuity and stability of the trajectory pose. The steps of differential path holding control include: The lateral deviation between the weld centerline and the robot's central axis is measured in real time using a laser sensor. The deviation is calculated using a PID control algorithm to generate a speed difference. Based on the calculated speed difference, adjust the PWM duty cycle of the left and right wheels to dynamically adjust the speed ratio of the left and right wheels; The stability of the adjusted wheel speed ratio was verified to ensure that the robot accurately tracks the center line of the weld. The steps of mixed position-force control include: The spatial deviation of the weld center is provided by the laser vision module; The deviation is solved by using the inverse kinematics of the robotic arm, the angles of each joint are calculated, the positional spatial deviation is compensated, and the positional adjustment of the probe center to align with the weld centerline is achieved. The probe contact force feedback value is collected in real time by a six-dimensional force sensor and compared with the preset target force value. By using a PID controller to adjust the Z-axis displacement of the robotic arm, a constant contact force can be achieved.

2. The multi-modal sensor based non-destructive inspection robotic weld seam automatic tracking system of claim 1, wherein, The steps to obtain preprocessed multimodal data include: The point cloud data was denoised, downsampled, and calibrated. The random sampling consistency method was used to fit the bevel surface to extract the feature points of the weld centerline. Edge detection, depth map denoising, and feature point matching are performed on RGB-D image data to supplement weld edge feature information; The attitude data is subjected to zero-bias correction and low-pass filtering to obtain stable attitude angle and angular velocity information.

3. The multi-modal sensor based non-destructive inspection robotic weld seam automatic tracking system of claim 1, wherein, The steps for outputting continuous weld space trajectory points include: The preprocessed point cloud data and image data are spatially aligned and time-synchronized to generate fused laser-visual multimodal data. The Kalman filter algorithm is used to fuse the laser-vision multimodal data and the attitude data collected by the inertial measurement unit to estimate the three-dimensional position and attitude state of the weld centerline in real time. Through the state update and prediction mechanism of Kalman filtering, continuous weld space trajectory points are output.

4. The automatic weld seam tracking system for non-destructive testing robots based on multimodal sensing according to claim 1, characterized in that, The hybrid position-force control adaptively adjusts the control strategy according to the weld type. When a planar weld is detected, a constant force control mode is adopted; when a curved weld is detected, a hybrid control mode combining position adjustment and force adjustment is adopted.

5. The multi-modal sensor based non-destructive inspection robotic weld seam automatic tracking system of claim 1, wherein, The system also includes a process error correction module, the implementation logic of which is as follows: When the laser signal is lost, switch to pure vision mode and detect that the robot's movement speed has dropped to 50% of the preset speed. When the laser signal is restored and lasts for 2 seconds, return to the initial state. When the contact force exceeds 50% of the preset initial value, the robotic arm immediately retracts 3mm and recouples. After the force value stabilizes at the preset initial value for 1 second, it returns to the initial state. If communication is interrupted, the robot immediately stops moving and is manually remotely returned to its original position by an external controller.

Images