A multi-modal fusion-based weld magnetic detection system and method

Abstract

The present application relates to a kind of based on multi-modal fusion's weld magnetic detection system and method, system includes: multi-modal sensor component, for obtaining the two-dimensional image data and three-dimensional point cloud data of weld;Industrial control module, for according to two-dimensional image data and three-dimensional point cloud data, using instance segmentation model and three-dimensional point cloud feature extraction network to identify weld area, and extract weld three-dimensional structure information, according to weld three-dimensional structure information to generate detection path, based on detection path to carry out weld defect analysis and positioning;Mechanical arm actuating mechanism, for connecting detection probe, drive detection probe to move according to detection path, and detection probe real-time acquisition weld detection signal and transmission to industrial control module.The present application realizes the full process automation of weld identification, three-dimensional modeling, detection path generation and defect identification, compared with prior art has significant technical advantage and practical application value.

Description

Technical Field

[0001] This invention relates to the field of welded structure production and quality inspection technology, and in particular to a weld magnetic inspection system and method based on multimodal fusion. Background Technology

[0002] In the field of welded structure production and quality inspection, non-destructive testing (NDT) of welds is a crucial link in ensuring structural safety and service performance. Currently, in complex weld structure scenarios, traditional NDT relies heavily on manual handheld inspection, manual path planning, and manual interpretation, exhibiting problems such as cumbersome inspection path planning, insufficient stability of inspection results, and poor consistency among different personnel. To improve the intelligence and automation of inspection, existing technologies are gradually incorporating machine vision and deep learning. Some solutions use a single two-dimensional visual modality to complete weld area recognition, or use basic three-dimensional point cloud processing methods to obtain weld geometric information. A few systems attempt to combine visual and point cloud data for weld inspection. However, existing automated inspection systems mostly rely on manual teaching to generate fixed inspection paths, making it difficult to adapt to dynamic changes in weld morphology, spatial orientation, and surface conditions. Furthermore, they often use simple data stitching or shallow feature fusion to process two-dimensional images and three-dimensional point cloud data, failing to fully coordinate two-dimensional texture features and three-dimensional spatial structural features. This results in significant shortcomings in adaptability to complex scenarios, accuracy of three-dimensional weld structure extraction, and efficient utilization of multi-modal data, making it difficult to meet the requirements for high-precision, high-robust automated magnetic weld inspection.

[0003] In summary, the existing technology has the following problems: 1. Insufficient accuracy in weld seam identification and positioning. Complex weld seams vary greatly in shape, and their surfaces are easily contaminated, obstructed, affected by changes in lighting, weld beads, and uneven roughness, making traditional two-dimensional vision recognition methods prone to false detections and missed detections. It is difficult to accurately segment and locate weld seam areas, leading to deviations in subsequent inspection benchmarks.

[0004] 2. Path planning relies on manual intervention and has poor adaptability. Existing automated inspection systems mostly rely on manual teaching to generate fixed paths, which cannot be adaptively adjusted according to the spatial shape of the workpiece. They are unable to cope with scenarios such as uneven weld surfaces and changes in working conditions, resulting in insufficient path fit and inspection coverage.

[0005] 3. Insufficient acquisition and utilization of 3D structural information. Two-dimensional images cannot reflect 3D geometric information such as weld depth, bevel, reinforcement height, and curvature; 3D point clouds suffer from problems such as high noise, uneven density, and strong background interference. Traditional point cloud processing methods are unable to stably extract the fine structure of the weld, resulting in insufficient accuracy of probe tracking and attitude control.

[0006] 4. Inefficient multimodal data fusion methods. Existing fusion schemes are mostly simple data splicing or shallow feature overlay, failing to establish a deep correlation between the texture details of two-dimensional images and the spatial structure of three-dimensional point clouds. This results in the inability to synergistically complement each other's advantages, leading to low data utilization, weak model robustness, and limited defect detection rate and detection stability.

[0007] 5. Insufficient accuracy in probe attitude and position control. Lacking a closed-loop attitude compensation mechanism driven by the three-dimensional fine structure of the weld, the probe struggles to maintain optimal detection distance and angle, easily leading to lift-off fluctuations, poor coupling, and attitude deviation, directly affecting the quality of the magnetic detection signal and the accuracy of defect location. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing non-destructive testing technologies for welds and to provide a weld magnetic detection system and method based on multimodal fusion. This addresses technical problems such as low weld recognition accuracy, poor adaptability to complex scenarios, reliance on manual path planning, insufficient detection stability and consistency, low utilization of multimodal data, and insufficient probe attitude control accuracy. This invention constructs a deep multimodal fusion architecture combining YOLOv11-Seg 2D segmentation and PointNet++ 3D feature extraction. Combined with dedicated detection tooling and intelligent path planning algorithms, it achieves high-precision automatic identification of weld areas, extraction of fine 3D structures, adaptive path planning for welds in complex spaces, real-time closed-loop control of probe attitude, and intelligent defect analysis and localization. This ensures that the system meets the stringent requirements of high positioning accuracy, high stability, and high consistency for weld non-destructive testing even under complex conditions such as weld deformation, surface contamination, occlusion, and spatial distortion.

[0009] To achieve the above objectives, the present invention provides the following solution: A multimodal fusion-based magnetic detection system for weld seams includes: A multimodal sensor assembly for acquiring two-dimensional image data and three-dimensional point cloud data of the weld seam; The industrial control module is used to identify the weld area based on two-dimensional image data and three-dimensional point cloud data, using an instance segmentation model and a three-dimensional point cloud feature extraction network, and extract the three-dimensional structural information of the weld. It automatically generates a detection path based on the three-dimensional structural information of the weld, and performs weld defect analysis and location based on the detection path. The robotic arm actuator is used to connect to the detection probe, drive the detection probe to move along the detection path, and the detection probe collects the detection signal of the weld in real time and transmits it to the industrial control module.

[0010] Optionally, the multimodal sensor assembly includes: An image acquisition device is used to acquire two-dimensional image data of the weld; wherein, the two-dimensional image data includes: weld surface texture and color; Point cloud acquisition equipment is used to acquire three-dimensional point cloud data of weld surface and spatial structure; the three-dimensional point cloud data includes: weld depth, curvature, and reinforcement height; The mapping unit is used to establish the mapping relationship between the pixel coordinates of a two-dimensional image and the world coordinates of a three-dimensional point cloud, so as to achieve the coordinate unification between two-dimensional image data and three-dimensional point cloud data.

[0011] Optionally, the industrial control module includes: The weld seam recognition submodule is used to achieve automatic recognition of weld seam areas by employing a multimodal fusion method that combines YOLOv11-Seg 2D segmentation with PointNet++ 3D feature extraction. The path planning submodule is used to extract the weld centerline based on the weld identification results and generate the detection trajectory in combination with the probe detection requirements.

[0012] Optionally, the weld recognition submodule includes: The image segmentation unit is used to input two-dimensional image data into the YOLOv11-Seg model. The YOLOv11-Seg model extracts global features of the image through the backbone network, and enhances the feature expression of the weld area by combining the neck feature fusion module. The detection head is used to realize the rapid localization and pixel-level segmentation of the weld area, and obtains a two-dimensional mask of the weld workpiece. The data mapping unit is used to map a two-dimensional mask to a three-dimensional point cloud space and extract the point cloud corresponding to the weld from the three-dimensional point cloud data. The point cloud optimization unit is used to standardize the extracted weld point cloud using a preset optimization method; the preset optimization method includes: statistical filtering, voxel downsampling, and radius filtering. The 3D structure extraction unit is used to input the optimized weld point cloud into the PointNet++ network. The PointNet++ network selects key sampling points through the sampling layer, constructs local neighborhoods through the grouping layer, and achieves hierarchical feature extraction through the feature propagation layer to obtain the 3D structure information of the weld from the point cloud. The 3D structure information of the weld includes: weld centerline, bevel shape, reinforcement height variation and spatial curvature.

[0013] Optionally, the route planning submodule includes: The weld centerline optimization unit is used to smooth the weld centerline curve, remove local fluctuations, and ensure that the trajectory point density meets the motion control requirements of the robotic arm by sampling and densification. The normal vector calculation and attitude planning unit is used to calculate the normal vector of the weld surface for each trajectory point. Combined with the structural parameters of the detection probe, excitation direction requirements, incident attitude constraints and magnetic coupling or acoustic coupling conditions, it automatically plans the attitude of the robotic arm end effector and generates the scanning path. The trajectory coordinate transformation unit is used to transform the scanning path to the robot arm's base coordinate system using a homogeneous coordinate transformation algorithm, so as to obtain the motion trajectory that the robot arm can directly execute.

[0014] Optionally, the industrial control module also includes: The analysis and positioning unit is used to process the real-time acquired detection signals of the weld, combine them with the three-dimensional structural information of the weld, identify the defect type, and locate the defect location; among which, the defect types include: cracks, porosity, and slag inclusions.

[0015] Optionally, the robotic arm actuator is connected to the detection probe via a detection probe mounting fixture; The detection probe mounting fixture is used to fix the detection probe and install it on the end flange of the robotic arm actuator. The probe mounting fixture is made of non-metallic materials and has an internal precision adjustment structure, including an X / Y / Z three-axis translation stage and a two-axis rotary stage, used to adjust the probe's orientation and distance from the weld surface.

[0016] On the other hand, to achieve the above objectives, the present invention provides the following solution: A magnetic detection method for welds based on multimodal fusion includes: The multimodal sensor assembly is activated to simultaneously acquire two-dimensional image data and three-dimensional point cloud data of the weld, generate multimodal raw data, and transmit it to the industrial control computer in real time. The industrial control computer calls the instance segmentation model to process the two-dimensional image data to obtain a two-dimensional mask of the weld seam. Then, the two-dimensional mask of the weld seam is mapped to the three-dimensional point cloud space to extract the corresponding point cloud of the weld seam. The point cloud corresponding to the weld is subjected to statistical filtering, voxel downsampling and radius filtering, and then input into the three-dimensional point cloud feature extraction network to obtain the three-dimensional structural information of the weld. Based on the three-dimensional structural information of the weld, the scanning path and the posture of the robotic arm are automatically generated, and the path is fitted and optimized to generate the detection path. The detection path is converted to the robot arm's base coordinate system, a robot arm motion command file is generated, and then loaded into the robot arm's actuator. The robotic arm actuator scans the weld seam at a constant speed along the detection path, and the detection probe collects the detection signal in real time and transmits it to the industrial control computer. The industrial control computer processes the detection signals, combines them with the three-dimensional structural information of the weld, identifies the defect type, and locates the defect position. Save test data and results, and automatically generate test reports.

[0017] The beneficial effects of this invention are as follows: This invention, by introducing technologies such as YOLOv11-Seg 2D segmentation, PointNet++ 3D feature extraction, deep fusion decision-making, automatic path planning, and intelligent detection analysis, achieves full-process automation of weld recognition, 3D modeling, detection path generation, and defect identification. Compared with existing technologies, it has significant technical advantages and practical application value. The specific technical effects are as follows: 1. Significantly improved weld seam recognition accuracy: Through deep fusion of YOLOv11-Seg and PointNet++, high-precision recognition of weld seam areas can be achieved under conditions of complex surfaces, occlusion, lighting changes, and irregular weld seam shapes. The accuracy of 3D structure extraction is improved, effectively solving the problem of low weld seam recognition accuracy in existing technologies.

[0018] 2. Significantly enhanced path planning adaptability: Based on the 3D fine structure of the weld seam extracted by PointNet++, combined with the dynamic attitude compensation algorithm, the automatically generated scanning path can accurately adapt to complex spatial weld seams (vertical welds, oblique welds, pipe welds, spatial twisted welds, etc.), without the need for manual teaching, overcoming the shortcomings of poor path planning adaptability in existing technologies.

[0019] 3. Stability and consistency of testing meet standards: The entire process is completed automatically by the system, reducing the impact of human factors and meeting the stringent requirements of non-destructive testing for accuracy and stability.

[0020] 4. Maximizing the utilization of multimodal data: The constructed deep fusion architecture fully explores the correlation between the texture details of two-dimensional images and the spatial structural features of three-dimensional point clouds, providing support for accurate defect identification.

[0021] 5. High versatility and practicality: The method of this invention can be used in conjunction with various types of probes or different detection principles, including AC electromagnetic field detection, electromagnetic ultrasound, laser ultrasound, etc. It is suitable for weld inspection of various materials such as austenitic stainless steel, carbon steel, and aluminum alloy, and can be widely used in many fields such as machinery manufacturing, petrochemicals, aerospace, and pipeline construction. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall structure of a weld magnetic detection system based on multimodal fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of multimodal weld seam recognition in an embodiment of the present invention. Figure 3 This is a schematic diagram of the three-dimensional reconstruction and structural extraction of the weld seam according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the automatic path planning principle of an embodiment of the present invention; Figure 5 This is a schematic diagram of three-dimensional trajectory and attitude planning according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the test block and detection area according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the test block detection path generation result according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] This embodiment proposes a weld magnetic detection system based on multimodal fusion, including: A multimodal sensor assembly for acquiring two-dimensional image data and three-dimensional point cloud data of the weld seam; The industrial control module is used to identify the weld area based on two-dimensional image data and three-dimensional point cloud data, using an instance segmentation model and a three-dimensional point cloud feature extraction network, and extract the three-dimensional structural information of the weld. It automatically generates a detection path based on the three-dimensional structural information of the weld, and performs weld defect analysis and localization based on the detection path. The instance segmentation model and the three-dimensional point cloud feature extraction network adopt the YOLOv11-Seg model and the PointNet++ network, respectively. The robotic arm actuator is used to connect to the detection probe, drive the detection probe to move along the detection path, and the detection probe collects the detection signal of the weld in real time and transmits it to the industrial control module.

[0027] Specifically, the weld magnetic inspection system of this embodiment includes a multimodal sensor assembly, a data acquisition and transmission module, a robotic arm actuator, and a detection probe mounting fixture, such as... Figure 1 As shown.

[0028] Furthermore, the multimodal sensor assembly includes: An image acquisition device is used to acquire two-dimensional image data of the weld; wherein, the two-dimensional image data includes: weld surface texture and color; Point cloud acquisition equipment is used to acquire three-dimensional point cloud data of weld surface and spatial structure; the three-dimensional point cloud data includes: weld depth, curvature, and reinforcement height; The mapping unit is used to establish the mapping relationship between the pixel coordinates of a two-dimensional image and the world coordinates of a three-dimensional point cloud, so as to achieve the coordinate unification between two-dimensional image data and three-dimensional point cloud data.

[0029] Specifically, in this embodiment, the multimodal sensor components include: an image acquisition device using a camera to acquire overall images of the weld area and obtain two-dimensional information such as weld surface texture and color; and a point cloud acquisition device, which can be a laser scanner (scanning accuracy ≤0.1mm) or a structured light depth camera, used to acquire point cloud data of the weld surface and spatial structure, obtaining three-dimensional geometric information such as weld depth, curvature, and height. Spatial calibration of the camera and the three-dimensional sensor is completed using the Zhang Zhengyou calibration method, establishing a mapping relationship between the pixel coordinates of the two-dimensional image and the world coordinates of the three-dimensional point cloud, achieving coordinate unification between the two-dimensional image and the three-dimensional point cloud, and laying the foundation for multimodal data fusion.

[0030] Data acquisition and transmission module: Responsible for the synchronous transmission of image data, point cloud data, and detection probe signals. It employs a combination of Gigabit Ethernet and PCIe 4.0 interfaces. Image and point cloud data are transmitted in real-time via Gigabit Ethernet, while detection signals are transmitted to the detection signal acquisition board via the PCIe 4.0 interface. The data cables are highly flexible drag chain cables, with sufficient length reserved according to the workpiece size and the robotic arm's working space to ensure unrestricted operation of the robotic arm in complex postures, while also possessing anti-interference, wear-resistant, and bending-resistant characteristics.

[0031] Furthermore, the industrial control module includes: The weld seam recognition submodule is used to achieve automatic recognition of weld seam areas by employing a multimodal fusion method that combines YOLOv11-Seg 2D segmentation with PointNet++ 3D feature extraction. The path planning submodule is used to extract the weld centerline based on the weld identification results and generate the detection trajectory in combination with the probe detection requirements.

[0032] The weld seam recognition submodule includes: The image segmentation unit is used to input two-dimensional image data into the YOLOv11-Seg model. The YOLOv11-Seg model extracts global features of the image through the backbone network, and enhances the feature expression of the weld area by combining the neck feature fusion module. The detection head is used to realize the rapid localization and pixel-level segmentation of the weld area, and obtains a two-dimensional mask of the weld workpiece. The data mapping unit is used to map a two-dimensional mask to a three-dimensional point cloud space and extract the point cloud corresponding to the weld from the three-dimensional point cloud data. The point cloud optimization unit is used to standardize the extracted weld point cloud using a preset optimization method; the preset optimization method includes: statistical filtering, voxel downsampling, and radius filtering. The 3D structure extraction unit is used to input the optimized weld point cloud into the PointNet++ network. The PointNet++ network selects key sampling points through the sampling layer, constructs local neighborhoods through the grouping layer, and achieves hierarchical feature extraction through the feature propagation layer to obtain the 3D structure information of the weld from the point cloud. The 3D structure information of the weld includes: weld centerline, bevel shape, reinforcement height variation and spatial curvature.

[0033] The route planning submodule includes: The weld centerline optimization unit is used to smooth the weld centerline curve, remove local fluctuations, and ensure that the trajectory point density meets the motion control requirements of the robotic arm by sampling and densification. The trajectory points are obtained by extracting the weld centerline through a trajectory planning algorithm after obtaining the three-dimensional structural information of the weld using PointNet++.

[0034] The normal vector calculation and attitude planning unit is used to calculate the normal vector of the weld surface for each trajectory point. Combined with the structural parameters of the detection probe, excitation direction requirements, incident attitude constraints and magnetic coupling or acoustic coupling conditions, it automatically plans the attitude of the robotic arm end effector and generates the scanning path. The trajectory coordinate transformation unit is used to transform the scanning path to the robot arm's base coordinate system using a homogeneous coordinate transformation algorithm, so as to obtain the motion trajectory that the robot arm can directly execute.

[0035] The industrial control module also includes: The analysis and positioning unit is used to process the real-time acquired detection signals of the weld, combine them with the three-dimensional structural information of the weld, identify the defect type, and locate the defect location; among which, the defect types include: cracks, porosity, and slag inclusions.

[0036] Specifically, in this embodiment, the industrial control computer (ICC) is used to process the acquired multimodal data and non-destructive testing signals, and is the core control unit of the system. The accompanying host computer software integrates the YOLOv11-Seg inference module, the PointNet++ feature extraction module, and the multimodal fusion decision module, enabling functions such as automatic weld seam recognition, 3D path planning, probe trajectory generation, and defect analysis and location. The external casing of the ICC includes an ICC display screen, keyboard, touchpad, and control switches, facilitating parameter setting, process control, and result viewing for operators. The internal casing includes the ICC motherboard, power module, deep learning inference accelerator card (used to improve the inference speed of the YOLOv11-Seg and PointNet++ models to ensure real-time performance), 3D point cloud acquisition card (supporting data acquisition rates of over 1 million points / second to ensure point cloud data integrity), and detection signal acquisition board (sampling rate ≥1MHz to ensure accurate defect signal capture).

[0037] Robotic arm actuator: A 6-axis industrial robotic arm (with repeatability ≤ ±0.02mm) is used to drive the detection probe to move along a planned trajectory. The end flange of the robotic arm is rigidly connected to the mounting fixture of the detection probe, enabling multi-degree-of-freedom attitude adjustment to meet the inspection requirements of complex spatial welds.

[0038] The inspection probe mounting fixture is used to secure non-destructive testing probes (such as AC electromagnetic field probes, other electromagnetic / ultrasonic probes) and stably mount them at the end flange of the robotic arm. The fixture is made of non-metallic materials to avoid interference with the detection signal from metal structures. Internally, the fixture features a precision adjustment mechanism, including an X / Y / Z three-axis translation stage and a two-axis rotary stage, for precisely adjusting the probe's orientation and distance from the weld surface, ensuring the probe is always in the optimal detection position. Externally, the fixture is equipped with an anti-collision protection device that triggers an emergency stop of the robotic arm in the event of an accidental collision between the probe and the workpiece, protecting both the probe and the workpiece.

[0039] The aforementioned automatic detection system is used to perform 3D modeling, automatic path planning, and defect detection of welds.

[0040] 1. Principles of Weld Seam Recognition and 3D Information Extraction This invention employs a multimodal fusion method combining "YOLOv11-Seg 2D segmentation + PointNet++ 3D feature extraction" to achieve high-precision automatic identification of weld seam regions, such as... Figure 2As shown. The specific steps are as follows: (1) Two-dimensional image segmentation: Input the weld seam image acquired by the two-dimensional industrial camera into the YOLOv11-Seg model. The model extracts the global features of the image through the backbone network, and enhances the feature expression of the weld seam area by combining the neck feature fusion module. The detection head is used to realize the rapid positioning and pixel-level segmentation of the weld seam area to obtain the two-dimensional mask of the weld seam workpiece. (2) Two-dimensional-three-dimensional data mapping: Based on the sensor space calibration results completed in the early stage, the two-dimensional mask output by YOLOv11-Seg is mapped to the three-dimensional point cloud space. The corresponding point cloud of the weld seam is accurately extracted from the original point cloud, and the background point cloud interference is removed to reduce the amount of subsequent data processing. (3) Three-dimensional point cloud optimization processing: The extracted weld seam point cloud has problems such as noise and uneven density. Statistical filtering (removing isolated noise points), voxel downsampling (uniforming point cloud density and retaining key structural features) and radius filtering (removing low-density area point clouds) are used to standardize it into a data form that can be used for deep learning analysis. (4) 3D Fine Structure Extraction: The optimized weld point cloud is input into the PointNet++ network. This network selects key sampling points through sampling layers, constructs local neighborhoods using grouping layers, and achieves hierarchical feature extraction through feature propagation layers. It accurately obtains 3D fine structure information such as weld centerline, groove morphology, reinforcement height variation, and spatial curvature from the point cloud, such as... Figure 3 As shown, PointNet++ can adaptively handle the problem of uneven point cloud density, improving the accuracy of 3D structure extraction compared to traditional geometry extraction methods. This structural information will be used as input for subsequent path planning.

[0041] 2. Automatic Path Planning Method For flat weld seams, the path planning is as follows: Figure 4 As shown. This invention extracts the weld centerline based on the geometric features of the weld three-dimensional point cloud, and generates a high-precision detection trajectory in combination with the probe detection requirements. The specific steps are as follows: (1) Optimization of weld centerline: The weld centerline extracted by PointNet++ is smoothed (B-spline curve fitting is used) to remove local fluctuations and ensure the smoothness of the trajectory; the density of trajectory points is ensured to meet the motion control requirements of the robotic arm by sampling densification. (2) Normal vector calculation and attitude planning: For each trajectory point, the normal vector of the weld surface is calculated. Combined with the structural parameters of the detection probe, the excitation direction requirements, the incident attitude constraints and the magnetic coupling or acoustic coupling conditions, the attitude of the end effector of the robotic arm is automatically planned. For complex spatial welds (such as vertical welds, oblique welds, pipe welds, and spatial tortuous welds), the system automatically compensates the probe angle based on the change of weld spatial curvature by attitude interpolation algorithm, so that the probe always remains perpendicular to the weld surface or has a specified tilt angle, thereby ensuring the detection stability. (3) Trajectory coordinate transformation: The generated scanning path is transformed to the robot arm base coordinate system through the coordinate transformation module (based on homogeneous coordinate transformation algorithm) to obtain the motion trajectory that the robot arm can directly execute. The trajectory structure is as follows: Figure 5 As shown. This path planning method requires no manual teaching, is highly adaptable, and can handle complex situations such as spatially distorted welds and the presence of weld beads.

[0042] The core innovation of this invention lies in proposing a multimodal fusion architecture of "YOLOv11-Seg 2D precise segmentation + PointNet++ 3D depth feature extraction," which solves the problems of insufficient multimodal data fusion, low weld seam recognition accuracy, and poor adaptability to complex scenarios in existing technologies. The specific innovations are as follows: Innovative Fusion Strategy: Breaking through the limitations of traditional simple splicing or shallow fusion of multimodal data, a deep fusion architecture of "2D segmentation - 3D feature extraction - feature collaborative decision-making" is constructed. YOLOv11-Seg quickly and accurately locates the weld seam region, eliminates background interference, and provides PointNet++ with an accurate region of interest; PointNet++ deeply mines the 3D spatial structural features of the weld seam. The synergistic effect of the two improves the robustness of weld seam recognition in complex scenarios (such as surface contamination, lighting changes, and weld seam deformation).

[0043] Model Adaptation and Optimization Innovations: Specifically optimized for weld inspection scenarios, YOLOv11-Seg and PointNet++ were implemented. A weld-specific attention mechanism was introduced into YOLOv11-Seg to enhance the representation of weld region features; in PointNet++, the sampling strategy and feature propagation method were optimized to improve adaptability to weld point cloud noise and density unevenness, ensuring the accuracy of 3D structure extraction.

[0044] Collaborative innovation of path planning and attitude control: The three-dimensional fine structure of the weld extracted by PointNet++ is deeply coupled with the probe attitude control. Dynamic attitude compensation of complex spatial welds is achieved through attitude interpolation algorithm to ensure that the probe is always in the optimal detection attitude.

[0045] This embodiment also proposes a weld magnetic detection method based on multimodal fusion, including: The multimodal sensor assembly is activated to simultaneously acquire two-dimensional image data and three-dimensional point cloud data of the weld, generate multimodal raw data, and transmit it to the industrial control computer in real time. The industrial control computer calls the instance segmentation model to process the two-dimensional image data to obtain a two-dimensional mask of the weld seam. Then, the two-dimensional mask of the weld seam is mapped to the three-dimensional point cloud space to extract the corresponding point cloud of the weld seam. The point cloud corresponding to the weld is subjected to statistical filtering, voxel downsampling and radius filtering, and then input into the three-dimensional point cloud feature extraction network to obtain the three-dimensional structural information of the weld. Based on the three-dimensional structural information of the weld, the scanning path and the posture of the robotic arm are automatically generated, and the path is fitted and optimized to generate the detection path. The detection path is converted to the robot arm's base coordinate system, a robot arm motion command file is generated, and then loaded into the robot arm's actuator. The robotic arm actuator scans the weld seam at a constant speed along the detection path, and the detection probe collects the detection signal in real time and transmits it to the industrial control computer. The industrial control computer processes the detection signals, combines them with the three-dimensional structural information of the weld, identifies the defect type, and locates the defect position. Save test data and results, and automatically generate test reports.

[0046] Specifically, the specific operating steps for detection in this embodiment are as follows: (1) Preparation: Fix the workpiece to be inspected on the workbench, start the industrial computer and initialize the two-dimensional industrial camera and three-dimensional sensor, and complete the sensor calibration and data synchronization; according to the size of the workpiece and the probe model, adjust the probe posture and distance to the weld surface through the precision adjustment structure of the tooling, and fix the probe to the end of the robotic arm. (2) Multimodal data acquisition: Start the multimodal sensor assembly, synchronously acquire the two-dimensional image and three-dimensional point cloud of the weld, generate multimodal raw data, and transmit it to the industrial computer in real time through the data acquisition and transmission module. (3) Weld area recognition: The industrial computer calls the YOLOv11-Seg model to process the two-dimensional image, obtain the two-dimensional mask of the weld, and then map it to the three-dimensional point cloud space to extract the corresponding point cloud of the weld. (4) Three-dimensional structure extraction: Perform statistical filtering, voxel downsampling and radius filtering on the weld point cloud, and then input it into the PointNet++ network to obtain the fine three-dimensional structure of the weld (center line, bevel shape, curvature, etc.). (5) Path and posture planning: Based on the three-dimensional structure of the weld, the scanning path and the posture of the robotic arm are automatically generated. The path is optimized by fitting B-spline curves to ensure smooth trajectory. (6) Trajectory conversion and loading: The planned path is converted to the base coordinate system of the robotic arm, the robotic arm motion command file is generated, and loaded into the robotic arm control system. (7) Automatic scanning and detection: The non-destructive testing module is started, and the robotic arm is driven to scan the weld at a constant speed along the planned trajectory. The probe collects the detection signal in real time and transmits it to the industrial control computer. (8) Defect analysis and location: The industrial control computer processes the detection signal, combines the three-dimensional structure information of the weld, identifies the defect type (such as cracks, porosity, slag inclusions, etc.), and accurately locates the defect position. (9) Data storage and report generation: The detection data (including multimodal raw data, detection signals, defect information, etc.) and detection results are stored, and the detection report is automatically generated, which includes information such as defect position, size, type and detection accuracy.

[0047] This embodiment introduces technologies such as YOLOv11-Seg 2D segmentation, PointNet++ 3D feature extraction, deep fusion decision-making, automatic path planning, and intelligent detection analysis to achieve full automation of the weld recognition, 3D modeling, detection path generation, and defect identification process. Compared with existing technologies, it has significant technical advantages and practical application value. The specific technical effects are as follows: 1. Significantly improved weld seam recognition accuracy: Through deep fusion of YOLOv11-Seg and PointNet++, high-precision recognition of weld seam areas can be achieved under conditions of complex surfaces, occlusion, lighting changes, and irregular weld seam shapes. The accuracy of 3D structure extraction is improved, effectively solving the problem of low weld seam recognition accuracy in existing technologies.

[0048] 2. Significantly enhanced path planning adaptability: Based on the 3D fine structure of the weld seam extracted by PointNet++, combined with the dynamic attitude compensation algorithm, the automatically generated scanning path can accurately adapt to complex spatial weld seams (vertical welds, oblique welds, pipe welds, spatial twisted welds, etc.), without the need for manual teaching, overcoming the shortcomings of poor path planning adaptability in existing technologies.

[0049] 3. Stability and consistency of testing meet standards: The entire process is completed automatically by the system, reducing the impact of human factors and meeting the stringent requirements of non-destructive testing for accuracy and stability.

[0050] 4. Maximizing the utilization of multimodal data: The constructed deep fusion architecture fully explores the correlation between the texture details of two-dimensional images and the spatial structural features of three-dimensional point clouds, providing support for accurate defect identification.

[0051] 5. High versatility and practicality: The method of this invention can be used in conjunction with various types of probes or different detection principles, including AC electromagnetic field detection, electromagnetic ultrasound, laser ultrasound, etc. It is suitable for weld inspection of various materials such as austenitic stainless steel, carbon steel, and aluminum alloy, and can be widely used in many fields such as machinery manufacturing, petrochemicals, aerospace, and pipeline construction.

[0052] This embodiment verifies the effectiveness of the multi-modal fusion-based magnetic detection system and method for welds by performing AC electromagnetic field detection on a flat weld test block with artificial defects. The key verifications are weld recognition accuracy, path fit, defect detection rate, and detection consistency.

[0053] II. Experimental Equipment and Parameters Industrial PC CPU: Intel Core i5-14600K; GPU: NVIDIA 3060 12GB; Memory: 16GB DDR4; Storage: 1TB SSD; Binocular camera resolution 1920 1200, frame rate 30fps, lens focal length 16mm; Single-structured light laser with a scanning accuracy of 0.05mm and a point cloud density of 1 million points / second; The robotic arm is a 6-axis industrial robotic arm with a repeatability of ±0.02mm and a maximum working radius of 1.5m. Detection probe, AC electromagnetic field (ACFM) probe; The inspection fixture is made of carbon fiber composite material and is equipped with a three-axis translation stage (X / Y / Z) and a two-axis rotary stage. YOLOv11-Seg model training dataset: contains 1200 images of different weld seam scenarios (contamination, lighting changes, deformation); training parameters: batchsize=4, epoch=100, learning rate 0.001; PointNet++ model training dataset: contains 300 sets of 3D point cloud data of different welds; training parameters: batchsize=4, epoch=80, learning rate 0.0005.

[0054] III. Test Block Parameters test block such as Figure 6 The image shows a test block for a flat weld. The base material of the test block is carbon steel, and the welding wire material has the same properties as the base material, resulting in a stable weld structure. The overall dimensions of the test block are 300mm × 150mm × 10mm, with a weld reinforcement of approximately 2mm and a weld width of approximately 10mm.

[0055] IV. Specific Implementation Steps (1) Equipment installation and initialization: Fix the test block on the workbench to ensure that the test block is flat and does not shake; install the two-dimensional industrial camera and three-dimensional sensor on the sensor fixing bracket, adjust the position of the bracket so that the sensor field of view covers the entire weld area; fix the detection probe to the flange at the end of the robotic arm through the tooling, and adjust the distance between the probe and the weld surface to 2mm through the X / Y / Z three-axis translation stage and two-axis rotary stage of the tooling, and the posture is perpendicular to the weld surface (tilt angle 0°); start the industrial control computer, install and start the host computer software, complete the initialization of the two-dimensional industrial camera and three-dimensional sensor, and complete the sensor calibration through Zhang Zhengyou calibration method to ensure that the coordinates of the two-dimensional image and the three-dimensional point cloud are consistent.

[0056] (2) Parameter settings: Input the detection parameters in the host computer software, including the test block size (300mm×150mm×10mm), weld width (5mm), scanning speed (2mm / s), ACFM probe working frequency (5kHz), defect detection threshold (determined based on previous experiments), etc.

[0057] (3) Multimodal data acquisition: Click the “Acquisition” button on the host computer software to simultaneously start the two-dimensional industrial camera and three-dimensional sensor to acquire two-dimensional images and three-dimensional point cloud data of the test block weld. The data is transmitted to the industrial control computer in real time via gigabit Ethernet.

[0058] (4) Weld seam area recognition: The industrial control computer automatically calls the YOLOv11-Seg model to process the two-dimensional image. The model outputs a two-dimensional mask of the weld seam. Then, based on the sensor calibration results, the two-dimensional mask is mapped to the three-dimensional point cloud space, the corresponding point cloud of the weld seam is extracted, and the background point cloud is removed.

[0059] (5) 3D structure extraction: The extracted weld point cloud is processed. First, statistical filtering (with 20 neighboring points and a standard deviation multiple of 2.0) is used to remove isolated noise points. Then, voxel downsampling (voxel size 0.1 mm) is used to homogenize the point cloud density. Finally, radius filtering (radius 0.3 mm and 5 neighboring points) is used to remove low-density area point clouds. The optimized point cloud is input into the PointNet++ network. The network outputs 3D structural information such as weld centerline, groove shape and spatial curvature. The centerline extraction error is ≤0.1 mm.

[0060] (6) Path and attitude planning: Based on the 3D structure of the weld seam output by PointNet++, the scanning path is automatically generated with a path sampling interval of 0.5mm. The path is optimized by fitting B-spline curves. According to the attitude requirements of the ACFM probe, the normal vector of the weld seam surface at each trajectory point is calculated to plan the attitude of the robotic arm so that the probe is always perpendicular to the weld seam surface. The planned path is converted to the base coordinate system of the robotic arm to generate the robotic arm motion command file and load it into the robotic arm control system.

[0061] (7) Automatic Scanning and Inspection: Click the "Start Inspection" button on the host computer software to start the ACFM inspection module and the robotic arm. The robotic arm scans the weld seam at a constant speed of 2mm / s along the planned trajectory. The probe collects the inspection signal in real time (sampling rate 1MHz) and transmits it to the industrial control computer through the PCIe 4.0 interface. The test block inspection path generation result is as follows: Figure 7 As shown.

[0062] (8) Data saving: After the detection is completed, the system automatically saves the detection data to the specified path.

[0063] V. Analysis of Implementation Results This embodiment verifies the effectiveness of the method and system of the present invention by testing a plate weld test block with artificial defects, proving that the present invention can effectively solve the problems existing in the prior art and meet the high-precision and high-stability testing requirements of complex weld structures.

[0064] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A weld magnetic inspection system based on multimodal fusion, characterized in that, include: A multimodal sensor assembly for acquiring two-dimensional image data and three-dimensional point cloud data of the weld seam; The industrial control module is used to identify the weld area based on two-dimensional image data and three-dimensional point cloud data, using an instance segmentation model and a three-dimensional point cloud feature extraction network, and extract the three-dimensional structural information of the weld. It automatically generates a detection path based on the three-dimensional structural information of the weld, and performs weld defect analysis and location based on the detection path. The robotic arm actuator is used to connect to the detection probe, drive the detection probe to move along the detection path, and the detection probe collects the detection signal of the weld in real time and transmits it to the industrial control module.

2. The weld magnetic inspection system based on multimodal fusion according to claim 1, characterized in that, The multimodal sensor assembly includes: An image acquisition device is used to acquire two-dimensional image data of the weld; wherein, the two-dimensional image data includes: weld surface texture and color; Point cloud acquisition equipment is used to acquire three-dimensional point cloud data of weld surface and spatial structure; the three-dimensional point cloud data includes: weld depth, curvature, and reinforcement height; The mapping unit is used to establish the mapping relationship between the pixel coordinates of a two-dimensional image and the world coordinates of a three-dimensional point cloud, so as to achieve the coordinate unification between two-dimensional image data and three-dimensional point cloud data.

3. The weld magnetic inspection system based on multimodal fusion according to claim 1, characterized in that, The industrial control module includes: The weld seam recognition submodule is used to achieve automatic recognition of weld seam areas by employing a multimodal fusion method that combines YOLOv11-Seg 2D segmentation with PointNet++ 3D feature extraction. The path planning submodule is used to extract the weld centerline based on the weld identification results and generate the detection trajectory in combination with the probe detection requirements.

4. The weld magnetic inspection system based on multimodal fusion according to claim 3, characterized in that, The weld seam recognition submodule includes: The image segmentation unit is used to input two-dimensional image data into the YOLOv11-Seg model. The YOLOv11-Seg model extracts global features of the image through the backbone network, and enhances the feature expression of the weld area by combining the neck feature fusion module. The detection head is used to realize the rapid localization and pixel-level segmentation of the weld area, and obtains a two-dimensional mask of the weld workpiece. The data mapping unit is used to map a two-dimensional mask to a three-dimensional point cloud space and extract the point cloud corresponding to the weld from the three-dimensional point cloud data. The point cloud optimization unit is used to standardize the extracted weld point cloud using a preset optimization method; the preset optimization method includes: statistical filtering, voxel downsampling, and radius filtering. The 3D structure extraction unit is used to input the optimized weld point cloud into the PointNet++ network. The PointNet++ network selects key sampling points through the sampling layer, constructs local neighborhoods through the grouping layer, and achieves hierarchical feature extraction through the feature propagation layer to obtain the 3D structure information of the weld from the point cloud. The 3D structure information of the weld includes: weld centerline, bevel shape, reinforcement height variation and spatial curvature.

5. The weld magnetic inspection system based on multimodal fusion according to claim 4, characterized in that, The route planning submodule includes: The weld centerline optimization unit is used to smooth the weld centerline curve, remove local fluctuations, and ensure that the trajectory point density meets the motion control requirements of the robotic arm by sampling and densification. The normal vector calculation and attitude planning unit is used to calculate the normal vector of the weld surface for each trajectory point. Combined with the structural parameters of the detection probe, excitation direction requirements, incident attitude constraints and magnetic coupling or acoustic coupling conditions, it automatically plans the attitude of the robotic arm end effector and generates the scanning path. The trajectory coordinate transformation unit is used to transform the scanning path to the robot arm's base coordinate system using a homogeneous coordinate transformation algorithm, so as to obtain the motion trajectory that the robot arm can directly execute.

6. The weld magnetic inspection system based on multimodal fusion according to claim 4, characterized in that, The industrial control module also includes: The analysis and positioning unit is used to process the real-time acquired detection signals of the weld, combine them with the three-dimensional structural information of the weld, identify the defect type, and locate the defect location; among which, the defect types include: cracks, porosity, and slag inclusions.

7. The weld magnetic inspection system based on multimodal fusion according to claim 1, characterized in that, The robotic arm actuator is connected to the detection probe via a detection probe mounting fixture. The detection probe mounting fixture is used to fix the detection probe and install it on the end flange of the robotic arm actuator. The probe mounting fixture is made of non-metallic materials and has an internal precision adjustment structure, including an X / Y / Z three-axis translation stage and a two-axis rotary stage, used to adjust the probe's orientation and distance from the weld surface.

8. A magnetic detection method for welds based on multimodal fusion, characterized in that, The method of using the system as described in any one of claims 1-7 includes: The multimodal sensor assembly is activated to simultaneously acquire two-dimensional image data and three-dimensional point cloud data of the weld, generate multimodal raw data, and transmit it to the industrial control computer in real time. The industrial control computer calls the instance segmentation model to process the two-dimensional image data to obtain a two-dimensional mask of the weld seam. Then, the two-dimensional mask of the weld seam is mapped to the three-dimensional point cloud space to extract the corresponding point cloud of the weld seam. The point cloud corresponding to the weld is subjected to statistical filtering, voxel downsampling and radius filtering, and then input into the three-dimensional point cloud feature extraction network to obtain the three-dimensional structural information of the weld. Based on the three-dimensional structural information of the weld, the scanning path and the posture of the robotic arm are automatically generated, and the path is fitted and optimized to generate the detection path. The detection path is converted to the robot arm's base coordinate system, a robot arm motion command file is generated, and then loaded into the robot arm's actuator. The robotic arm actuator scans the weld seam at a constant speed along the detection path, and the detection probe collects the detection signal in real time and transmits it to the industrial control computer. The industrial control computer processes the detection signals, combines them with the three-dimensional structural information of the weld, identifies the defect type, and locates the defect position. Save test data and results, and automatically generate test reports.

Images