A method and system for synchronous three-dimensional perception and detection of internal and external defects in a weld based on a graph neural network
By synchronously acquiring data with a structured optical camera and a phased array ultrasonic probe, cross-modal feature alignment and fusion are achieved using graph neural networks, and three-dimensional reconstruction is performed by combining neural implicit representations and embedding physical law constraints. This solves the problem of synchronous perception and three-dimensional visualization of internal and external defects in weld inspection, and improves the detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-03-09
- Publication Date
- 2026-05-29
AI Technical Summary
Existing weld inspection technologies rely on a single mode, making it difficult to detect internal and external defects simultaneously. The accuracy of multi-source heterogeneous data fusion is low, the three-dimensional visualization reconstruction effect is poor, the rate of missing small and dense defects is high, and the adaptability to dynamic working conditions is weak.
Data is acquired synchronously using a structured optical camera and a phased array ultrasonic probe. Cross-modal feature alignment and fusion are achieved through graph neural networks. Three-dimensional reconstruction is performed by combining neural implicit representations and embedding physical constraints for defect detection.
It achieves high-precision synchronous detection of internal and external defects in welds, improves the integrity of three-dimensional visualization reconstruction and the robustness of detection, and enhances adaptability under dynamic working conditions.
Smart Images
Figure CN122115985A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent manufacturing and nondestructive testing technology, specifically relating to an intelligent detection method and system for internal and external defects in welds based on multimodal data and graph neural networks. This method integrates optical and ultrasonic sensing information, achieves cross-modal feature alignment and fusion through graph structures, and completes three-dimensional reconstruction and quantitative defect detection of the internal and external structures of the weld based on implicit neural representations. It is suitable for online monitoring and health management of welding quality in high-end equipment such as aerospace, shipbuilding, and pipelines. Background Technology
[0002] Simultaneous, accurate, and visualized detection of welding defects is a core challenge in ensuring the safety and manufacturing quality of major engineering projects. Traditional nondestructive testing relies on single-modal data, making it difficult to achieve collaborative perception and three-dimensional visualization of internal and external defects in welds. Existing multimodal fusion technologies in industrial weld inspection scenarios are limited by differences in the spatiotemporal references, physical mechanisms, and feature dimensions of multi-source heterogeneous data, and still suffer from problems such as low fusion accuracy, insufficient reconstruction visualization, high missed detection rate of small and dense defects, and weak adaptive capability under dynamic working conditions.
[0003] In the prior art, patent CN115774014A discloses a deep learning-based weld defect detection method that fuses visual and ultrasonic data, but relies on convolutional neural networks for simple feature stitching, resulting in limited semantic alignment capabilities. Patent CN112258521B proposes a 3D reconstruction system based on multi-sensor data, which achieves point cloud generation but lacks a graph structure association mechanism for cross-modal features. Patent CN121236403A relates to a multimodal fusion detection method based on an attention mechanism, simulating a feature interaction mechanism, but it is limited to surface defect detection and does not fuse internal ultrasonic data with the 3D reconstruction process. Patent CN120491605A discloses a multimodal data fusion method based on tensor decomposition for structural health monitoring, but lacks a dynamic feature aggregation and physical law fusion mechanism guided by graph neural networks. Furthermore, existing methods are mostly limited to a single physical modality or simple multimodal stitching, lacking deep cross-modal semantic alignment guided by graph structures, making it difficult to achieve integrated, high-fidelity 3D visualization reconstruction of internal and external defects, and also failing to consider robustness enhancement through physical law embedding.
[0004] In summary, existing weld inspection technologies still have significant shortcomings in terms of the accuracy of multimodal data fusion, the completeness of 3D visualization reconstruction, and the system's adaptability and physical consistency in dynamic scenarios. Therefore, there is an urgent need for a new intelligent inspection method that can deeply integrate multi-source heterogeneous data, achieve synchronous high-precision 3D visualization of internal and external defects, and be guided by physical laws. Summary of the Invention
[0005] The technical problem to be solved by this invention is to provide an intelligent method that can achieve integrated, high-precision, and visualized synchronous detection of internal and external defects, addressing the problems of existing weld defect detection methods such as reliance on a single mode, difficulty in synchronous detection of internal and external defects, low accuracy of multi-source heterogeneous data fusion, poor three-dimensional visualization reconstruction effect, high missed detection rate of small and dense defects, and weak adaptability to dynamic working conditions.
[0006] To achieve the above objectives, this invention provides a method for simultaneous three-dimensional perception and detection of internal and external defects in welds based on graph neural networks, characterized by the following steps:
[0007] S1. Synchronous acquisition and de-crosstalk processing of optical and ultrasonic images
[0008] Using a structured optical camera and a phased array ultrasonic probe, optical image sequences of the weld surface and ultrasonic image sequences of the interior are acquired synchronously under the control of a unified hardware trigger signal. Then, a blind source separation method is used to de-crosstalk the mixed sensor signals in order to suppress cross-interference caused by electromagnetic coupling and signal leakage.
[0009] S2, Spatiotemporal Alignment and Feature Extraction of Dual-Modal Data
[0010] First, a hardware-triggered synchronization and software-based post-synchronization correction method based on dynamic time warping are employed to achieve time series alignment. A fixed delay compensation calculated from the ultrasonic wave propagation speed is introduced to improve alignment accuracy. Second, spatial coordinate registration of the optical and ultrasonic images is achieved through geometric transformation. Finally, high-dimensional features of both modalities are extracted using convolutional neural networks.
[0011] S3. Cross-modal feature fusion based on graph neural networks
[0012] The extracted optical and ultrasonic features are mapped to nodes in a graph structure, and cross-modal association edges and intramodal temporal edges are constructed. Then, a multi-head attention mechanism is used to model the relationship between nodes, and deep fusion of cross-modal features is achieved through graph convolution operations to generate a unified joint feature representation.
[0013] S4. Three-dimensional reconstruction of the internal and external structure of the weld.
[0014] By representing joint features as conditions, a neural implicit surface reconstruction method is adopted. The symbolic distance value and occupancy probability of any point in the space are regressed through a multilayer perceptron network. During the reconstruction process, optimization is carried out by combining the symbolic distance function and the occupancy network dual constraints. An adaptive sampling strategy based on geometric prior is adopted to increase the sampling density in smooth areas and maintain the baseline sampling density at feature edges, thereby achieving high-fidelity 3D reconstruction of weld surface and internal defect structure.
[0015] S5. Intelligent Defect Detection and Quantification Integrating Physical Laws
[0016] Based on the reconstructed 3D model, a differentiable physical constraint model integrating differential geometry and wave scattering theory is constructed and embedded into the training process of a deep learning detection network. Through physical laws constraining network optimization, the identification, location, and size quantification of weld defects are achieved.
[0017] The present invention also provides a synchronous intelligent detection system for internal and external defects in welds for implementing the above method, characterized in that it comprises:
[0018] The dual-sensor synchronous acquisition module is used to synchronously acquire optical images of the weld surface and ultrasonic images of the interior under hardware triggering, and to perform crosstalk removal processing.
[0019] The data processing and computation module is used to sequentially execute algorithms for spatiotemporal alignment and feature extraction, cross-modal graph neural network fusion, neural implicit 3D reconstruction, and physical constraint defect detection.
[0020] The results output module is used to visualize the 3D defect reconstruction model and generate a structured inspection report.
[0021] The beneficial effects of this invention are as follows:
[0022] 1. By introducing a delay-compensated dynamic time warping algorithm and a graph neural network fusion architecture, the spatiotemporal heterogeneity and semantic gap between optical and ultrasonic data are effectively solved, achieving high-precision alignment and deep fusion of cross-modal features.
[0023] 2. By employing neural implicit representation for 3D reconstruction and combining it with a curvature-driven adaptive sampling strategy, high-completeness and high-fidelity 3D visualization of weakly textured regions and internal structures can be achieved, significantly improving the spatial representation capability of defects.
[0024] 3. By embedding a differentiable physical constraint model into the detection network, which integrates geometric morphology and wave scattering theory, the robustness and quantization accuracy of the method in scenarios with strong interference, small scale, and high density defects are enhanced, while the physical interpretability and industrial applicability of the system are also improved. Attached Figure Description
[0025] Figure 1 This is the overall method flowchart.
[0026] Figure 2 This is a schematic diagram of cross-modal feature extraction.
[0027] Figure 3 This is a schematic diagram of cross-modal feature fusion.
[0028] Figure 4 This is a schematic diagram of graph neural network-based 3D reconstruction.
[0029] Figure 5 This is a schematic diagram of the structural components of the detection system. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.
[0031] This embodiment is implemented on an automated pipeline circumferential weld inspection platform. The object to be inspected is the circumferential weld of an X80 grade oil and gas long-distance pipeline with a wall thickness of 20mm. The system hardware configuration is as follows: Figure 5 As shown, the setup includes a structured light 3D camera (Gocator2350 series), a 32-element phased array ultrasonic probe (center frequency 5MHz), and a high-performance graphics workstation. The methodology used in this study is as follows: Figure 1 As shown.
[0032] S1. Synchronous acquisition and de-crosstalk processing of optical and ultrasonic images
[0033] A synchronization controller generates a 10Hz hardware trigger pulse to synchronously control the exposure of the industrial camera and the excitation of the ultrasonic probe, achieving alignment of the acquisition timing. In a single scan, a sequence of structured light images of the weld surface is acquired simultaneously. and internal ultrasound B-scan image sequence .
[0034] The high-voltage excitation signal from the ultrasonic probe causes electromagnetic crosstalk to the camera's image sensor, manifesting as periodic stripe noise in the optical image. A blind source separation algorithm based on sparsity constraints is employed for crosstalk removal. The two synchronously acquired signals are used to construct an observation matrix. The source signal is separated by optimizing the following objective function:
[0035]
[0036] in, It is a mixed matrix. Let be the source signal matrix to be separated. The regularization term characterizes the sparsity of the signal. The regularization parameter is set to 0.1 in this embodiment. After optimization, an optical image with crosstalk noise removed is obtained. With ultrasound images
[0037] S2, Spatiotemporal Alignment and Feature Extraction of Dual-Modal Data
[0038] To achieve effective fusion of optical surface images and ultrasonic internal images, the inconsistencies in their acquisition timing and spatial coordinate systems must first be addressed. This step sequentially performs temporal alignment, spatial registration, and feature extraction, as detailed below:
[0039] Timing Alignment: The propagation speed of ultrasound in X80 steel is approximately 5900 m / s. Calculate the fixed delay of ultrasound propagation from the probe surface to a depth (10 mm) at the weld center. Using this delay as a priori compensation, and substituting it into the Dynamic Time Warping (DTW) algorithm, the improved objective function is:
[0040]
[0041] in, To align the path, This is a path smoothness penalty term. Its weighting coefficient, in this embodiment, is set to By solving dynamic programming, sub-pixel precision temporal alignment is achieved, resulting in a time-aligned sequence. and .
[0042] Spatial registration: Hand-eye calibration is performed using a checkerboard calibration board to obtain a rigid transformation matrix from the camera coordinate system to the ultrasonic probe coordinate system. . use The ultrasound image sequence was reprojected and unified to the camera coordinate system. .
[0043] Feature extraction: Feature extraction is performed using two independent convolutional neural networks (CNNs), as shown in Figure 2. The optical branch uses a ResNet-34 pre-trained on ImageNet, with the input... Feature maps output from intermediate layers of the ResNet-34 network model The ultrasound branch uses a lightweight custom CNN (containing 4 convolutional layers) as input. Output and Spatially resolution aligned feature maps .
[0044] S3. Cross-modal feature fusion based on graph neural networks
[0045] Will and The feature vector of each spatial location is defined as a graph node. and , construct graph Node set Edge set It includes two types: 1) Cross-modal associative edges: connecting edges at the same spatial location. and ;2) Same-modal temporal edge: Based on the optical flow field of the image sequence, connect the nearest neighbor nodes of the same modality in adjacent frames.
[0046] The multi-head attention (MHA) mechanism is used to calculate the association weights between nodes. For a node... and In the The attention coefficient of each attention head is calculated as follows:
[0047]
[0048] in, It is a learnable linear transformation matrix. For attention vectors, This represents vector concatenation. For nodes The set of neighbors.
[0049] Subsequently, neighborhood information is aggregated using graph convolutional neural networks (GCN) to update node features:
[0050]
[0051] in, The value transformation matrix, For ELU activation function, To focus on the number of attention heads, after two layers of graph convolution, the optical and ultrasonic node features at the same location are added together to generate a deeply fused joint feature representation. Its schematic diagram is as follows Figure 3 As shown.
[0052] S4. Three-dimensional reconstruction of the internal and external structure of the weld.
[0053] Three-dimensional reconstruction using implicit neural expression, such as Figure 4 As shown, a multilayer perceptron (MLP) network is constructed. Its input is the coordinates of a point in space. and the joint features projected onto the feature map from that point. The output is the symbolic distance (sdf) and occupancy probability (occ) of that point.
[0054]
[0055] During network training, a joint loss combining the signed distance function constraint Lsdf and the occupied network constraint Locc is used:
[0056]
[0057] Where ŝdf and ôcc are the true values, , BCE is the binary cross-entropy loss.
[0058] During the sampling phase, a curvature-driven adaptive strategy is employed. This is based on the local curvature predicted by the current network. Dynamically adjust sampling point density :
[0059]
[0060] in, The reference density is 100 points per cubic millimeter in this embodiment. The curvature threshold is set to 0.05. Sampling is densified in smooth regions to enhance details, while baseline sampling density is maintained in feature edge regions. Through training optimization, the isosurface with sdf = 0 is finally extracted, resulting in a 3D mesh model M containing internal and external structures.
[0061] S5. Intelligent Defect Detection and Quantification Integrating Physical Laws
[0062] Reconstructed 3D Model A 3D object detection network (such as PointRCNN) is used for defect detection. To improve the physical plausibility of the detection, differentiable physical constraints are embedded in the network's loss function. :
[0063]
[0064] in, To detect the classification and regression loss of the task itself, These are the weighting coefficients for physical constraints.
[0065] Physical constraint loss It consists of two parts: 1. Geometric consistency loss For each detected defect candidate region, its shape descriptor based on the Riemannian manifold (such as surface area to volume ratio, principal curvature statistics) is calculated and compared with the prior physical morphology of typical welding defects (porosity, cracks, etc.) of the material. The comparison revealed that the penalty deviated significantly from the prior prediction.
[0066]
[0067] in, The function for calculating the shape factor. Mesh for the defect region, 1. The predicted defect type. 2. Signal fidelity loss. Based on the three-dimensional dimensions of the defect candidate region and location Using the Mie scattering and elastic wave diffraction model, the theoretical signal response characteristics that it should have in ultrasound images are simulated. This simulation feature is compared with the original ultrasound data from step S2. The features actually extracted at the corresponding location Comparison:
[0068]
[0069] The network is trained under physical constraints and ultimately outputs defect category labels, three-dimensional bounding boxes in three-dimensional space, and precise size parameters.
[0070] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the invention. Those skilled in the art can adjust the hardware selection, network structure, and parameters without departing from the core principles of the present invention, and such adjustments should be considered within the scope of protection of the present invention.
Claims
1. A method for simultaneous three-dimensional perception and detection of internal and external defects in welds based on graph neural networks, characterized in that, Includes the following steps: S1. Synchronous acquisition and crosstalk removal of optical and ultrasonic images: First, optical images of the weld surface are acquired synchronously using a structured optical camera, and ultrasonic images of the weld interior are acquired using a phased array ultrasonic probe. Then, the acquired optical and ultrasonic images are processed to remove crosstalk, so as to suppress crosstalk caused by electromagnetic coupling and signal leakage during multi-sensor synchronous acquisition. S2. Spatiotemporal alignment and feature extraction of dual-modal data: First, a software synchronization correction method using hardware-triggered synchronization and dynamic time warping algorithm is adopted to align the optical and ultrasound images after crosstalk removal in time series. Then, a geometric transformation algorithm is used to complete the spatial coordinate registration of the two types of images. Finally, convolutional neural networks are used to extract the high-dimensional features of the optical and ultrasound images respectively. S3. Cross-modal feature fusion based on graph neural networks: The extracted optical and ultrasonic features are mapped to nodes in a graph structure, and the multi-head attention mechanism is used to model the correlation between different modal features. The deep fusion of cross-modal features is achieved through graph convolution operations to generate a unified joint feature representation. S4. Three-dimensional reconstruction of the internal and external structure of the weld: Taking the joint feature representation of step S3 as the core input, the neural implicit expression is adopted and the symbolic distance function and the occupancy network are introduced as dual constraints. Combined with the geometric prior-driven adaptive sampling strategy, the cross-modal joint features are transformed into three-dimensional spatial geometric information, accurately characterizing the surface topology and internal defect spatial structure of the weld, and obtaining a three-dimensional geometric model. S5. Intelligent Defect Detection and Quantification Based on Physical Laws: Based on the weld surface topology and internal defect spatial structure information provided by the three-dimensional model in step S4, a differentiable physical constraint model integrating differential geometry and wave scattering theory is constructed and embedded into the loss function of the deep learning detection network. The physical laws constrain the network learning process, driving the network to achieve the identification, location and size quantification of weld defects.
2. The method according to claim 1, characterized in that, S1 includes: In step S1, the crosstalk removal process uses a blind source separation algorithm to separate crosstalk components from the mixed sensor signals; the synchronous acquisition is achieved by controlling the exposure of the industrial camera and the excitation of the ultrasonic probe with a unified hardware trigger signal to achieve millisecond-level synchronization.
3. The method according to claim 1, characterized in that, S2 includes: In step S2, the dynamic time warping algorithm introduces a fixed delay compensation amount calculated from the ultrasonic wave propagation speed to achieve sub-pixel-level temporal alignment of optical and ultrasonic image sequences.
4. The method according to claim 1, characterized in that, S3 includes: In step S3, the nodes in the graph structure include optical feature nodes and ultrasonic feature nodes, and the edges include cross-modal associated edges and intramodal temporal edges; the multi-head attention mechanism is used to calculate the attention weights between nodes and to achieve feature aggregation through graph convolution.
5. The method according to claim 1, characterized in that, S4 includes: The neural implicit reconstruction uses a multilayer perceptron network. The input is spatial coordinates and joint features, and the output is symbolic distance value and occupancy probability. During the reconstruction process, the sampling density is adaptively adjusted according to the local curvature. The sampling density is increased in smooth regions and the baseline density is maintained in feature edge regions to obtain a three-dimensional geometric model.
6. A synchronous intelligent detection system for internal and external defects in welds for implementing the method according to any one of claims 1 to 5, characterized in that, include: The dual-sensor synchronous acquisition module is used to simultaneously acquire optical images of the weld surface and ultrasonic images of the interior, and to perform image de-crosstalk processing. The data processing and computation module is used to execute algorithms for spatiotemporal alignment, feature extraction, cross-modal fusion, 3D reconstruction, and defect detection. The results output module is used to visualize the 3D defect model and generate an inspection report. The dual-sensor synchronous acquisition module includes an industrial camera, a phased array ultrasonic probe, a synchronization controller, and a crosstalk removal processing unit.