Weld defect detection and multi-modal neural network model construction method

Abstract

The invention discloses a weld defect detection and multi-modal neural network model construction method, which belongs to the technical field of weld defect detection, and comprises the following steps: scanning the surface of a weldment by adopting a line light source, obtaining an original image containing contour information and depth data, carrying out image denoising processing, extracting a central track, and obtaining a multi-modal neural network model; outliers are removed by using a point cloud statistical method based on a Gaussian distribution model, farthest point sampling is carried out, and three-dimensional reconstruction and calibration of a welding seam are carried out; according to the method, multi-modal feature learning, information monitoring and quality evaluation technologies of point cloud data, welding images and welding parameter information are fused, intelligent monitoring and diagnosis of multiple types of defects such as cracks, undercuts, gaps and pits in the welding process under the small sample condition are achieved, key process parameters generated by the defects can be traced, and the defect detection accuracy is improved. And the detection accuracy is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of weld defect detection technology, and particularly relates to a method for weld defect detection and multimodal neural network model construction. Background Technology

[0002] Weld defect inspection is the process of checking and testing for various defects that may exist in the weld after welding is completed. These defects may originate from various factors during the welding process, such as improper welding parameter settings, material quality problems, and insufficient welding skills. Weld defects not only affect the mechanical properties and service life of the welded joint, but may also lead to serious safety accidents. Common weld defects include, but are not limited to, cracks, slag inclusions, lack of fusion, incomplete penetration, porosity, and weld dimensions and shapes that do not meet requirements. The presence of these defects weakens the strength, toughness, and sealing of the weld, thereby affecting the safety and reliability of the entire structure. Therefore, defect detection methods are required when conducting weld defect inspection.

[0003] Chinese patent (CN119470616A) discloses a weld defect detection method based on a weak magnetic field sensor array. This method can obtain magnetic induction intensity data after removing the background field through difference calculation and combine it with multiple judgment conditions for defect identification. However, this method fails to integrate multimodal sensor data, making it difficult to accurately trace the cause of defect formation, and it has poor adaptability to complex welding conditions. To solve these problems, a method for weld defect detection and multimodal neural network model construction is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to address the problems that current methods fail to integrate multimodal sensing data, making it difficult to accurately trace the causes of defects and have poor adaptability to complex welding conditions. Therefore, this invention proposes a method for weld defect detection and multimodal neural network model construction.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a method for weld defect detection and multimodal neural network model construction, comprising the following steps:

[0006] S1. Use a line light source to scan the surface of the weldment to obtain the original image and depth data containing contour information;

[0007] S2. Perform image denoising processing;

[0008] S3. Extract the center trajectory;

[0009] S4. Outlier removal is performed using point cloud statistical methods based on Gaussian distribution models.

[0010] S5. Perform farthest point sampling;

[0011] S6. Perform three-dimensional reconstruction and calibration of the weld seam;

[0012] S7. Establish a neural network for weld seam point cloud;

[0013] S8. Perform weld defect detection based on neural networks.

[0014] As a further description of the above technical solution:

[0015] In step S1, a line light source is used to scan the surface of the weldment to obtain the original image and depth data containing contour information. This method has low requirements for the surface to be tested and has the advantages of high speed, high efficiency and high precision.

[0016] As a further description of the above technical solution:

[0017] In step S2, image denoising is performed. The image denoising algorithms used include mean filtering, median filtering, Gaussian filtering, and adaptive median filtering. The window size of the adaptive median filtering is dynamically adjusted according to the width of the weld seam light band, with a maximum window size of 15×15 pixels.

[0018] As a further description of the above technical solution:

[0019] In step S3, the center trajectory is extracted using the Steger algorithm. The Steger algorithm achieves high-precision light band center extraction based on the Hessian matrix. It constructs the Hessian matrix by calculating the second derivative of the image, performs eigenvalue decomposition, determines the direction of the light band center, and achieves sub-pixel-level positioning through Taylor expansion. This algorithm exhibits high precision and strong robustness when dealing with complex light band shapes and image noise. The prior constraint of the weld direction is introduced in the Hessian matrix calculation to optimize the robustness of the light band center extraction.

[0020] As a further description of the above technical solution:

[0021] In S5, sampling of the farthest point is performed. Specifically, a starting point is selected from the point cloud data, and then the point farthest from the current point set is selected iteratively until the set number is met. This sampling method can ensure the maximization of spatial coverage and the uniformity of point distribution.

[0022] As a further description of the above technical solution:

[0023] In step S6, the weld seam is reconstructed and calibrated in three dimensions. The specific steps are: extracting weld seam features, reconstructing in three dimensions, and registering and fusing point clouds. In the process of reconstructing the weld seam in three dimensions, the quality of the point cloud data directly affects the accuracy and reliability of the reconstruction model.

[0024] As a further description of the above technical solution:

[0025] In step S6, weld feature extraction is performed to identify and describe the geometric features, shape information and other key attributes of the weld. The feature extraction methods used include normal estimation, curvature calculation, cross-sectional area calculation, maximum point calculation, average height calculation and feature point extraction.

[0026] As a further description of the above technical solution:

[0027] In step S6, three-dimensional reconstruction is performed, and the surface reconstruction methods used include Poisson reconstruction and Delaunay triangulation.

[0028] As a further description of the above technical solution:

[0029] In step S6, point cloud registration and fusion are performed. During the welding process, due to differences in perspective, time, and sensors, the obtained point cloud data often exhibit inconsistencies in position, orientation, and scale. Therefore, it is necessary to accurately align these point clouds and fuse them into a unified model. The first step in point cloud registration is initial alignment, which typically employs a feature-based registration method, such as the Random Sample Consensus (RANSAC) algorithm. This algorithm randomly selects a set of feature point pairs from the source and target point clouds, calculates their transformation matrix to achieve initial alignment, and then applies the transformation to the source point cloud and evaluates the distance between it and the target point cloud. The alignment quality is assessed by the number of interior points and a distance threshold, and the transformation matrix is ​​iteratively optimized to ultimately achieve high-precision registration. Fine registration not only improves the accuracy of point cloud alignment but also ensures that the geometric features of each part of the weld bead can be accurately matched.

[0030] As a further description of the above technical solution:

[0031] In step S7, a weld seam point cloud neural network is established, including a PointNet network. In step S8, weld seam defect detection based on neural networks is performed, including weld defect classification using deep convolutional neural networks, weld defect classification based on PointNet++, and weld defect classification based on DGCNN. Deep convolutional neural networks can naturally process data through end-to-end integration of multi-layer features, and the depth of feature extraction is enhanced by increasing the number of layers. The training of PointNet++ uses the same dataset as PointNet, and uses negative log-likelihood loss as the loss function. The optimizer uses the Adam algorithm to further improve the stability and accuracy of the model. The training of DGCNN also uses the same dataset as PointNet, and uses labeled smooth cross-entropy loss as the loss function. The optimizer uses the Adam algorithm, and random dropping of input points during training improves the generalization ability of the model.

[0032] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0033] The detection method proposed in this invention integrates point cloud data, welding images, and welding process parameters to construct a multimodal feature learning and quality assessment system. This system is capable of identifying various weld defects such as cracks, undercut, voids, and pits even under small sample conditions. Furthermore, the system can analyze the causes of defects by combining key process parameters (including current, voltage, wire feed speed, and welding speed), effectively improving detection accuracy and welding quality control. Attached Figure Description

[0034] Figure 1 This is a flowchart of a method for weld defect detection and multimodal neural network model construction. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figure 1 This invention provides a technical solution: a method for weld defect detection and multimodal neural network model construction, comprising the following steps:

[0037] S1. The surface of the weldment is scanned by a line light source to obtain the original image and depth data containing contour information. It has low requirements for the surface to be tested and has the advantages of high speed, high efficiency and high precision.

[0038] S2. Perform image denoising processing, including mean filtering, median filtering, Gaussian filtering and adaptive median filtering.

[0039] S3. The center trajectory is extracted using the Steger algorithm. The Steger algorithm is based on the Hessian matrix to achieve high-precision light band center extraction. It constructs the Hessian matrix by calculating the second derivative of the image, performs eigenvalue decomposition, determines the direction of the light band center, and achieves sub-pixel level positioning through Taylor expansion. This algorithm shows high precision and strong robustness when dealing with complex light band shapes and image noise. The prior constraint of the weld direction is introduced in the Hessian matrix calculation to optimize the robustness of the light band center extraction.

[0040] S4. Outlier removal is performed using point cloud statistical methods based on Gaussian distribution models.

[0041] S5. Perform farthest point sampling. Specifically, select a starting point from the point cloud data, and then iteratively select the point farthest from the current point set until the set number is met. This sampling method can ensure the maximization of spatial coverage and the uniformity of point distribution.

[0042] S6. Perform 3D reconstruction and calibration of the weld seam. The specific steps are as follows: extract weld seam features, perform 3D reconstruction, and perform point cloud registration and fusion. In the 3D reconstruction of the weld seam, the quality of the point cloud data directly affects the accuracy and reliability of the reconstruction model. Extract weld seam features to identify and describe the geometric features, shape information, and other key attributes of the weld seam. The feature extraction methods used include normal estimation, curvature calculation, cross-sectional area calculation, maximum point calculation, average height calculation, and feature point extraction. Perform 3D reconstruction using surface reconstruction methods including Poisson reconstruction and Delaunay reconstruction. Triangulation is used for point cloud registration and fusion. During the welding process, due to differences in perspective, time, and sensors, the obtained point cloud data often exhibits inconsistencies in position, orientation, and scale. Therefore, it is necessary to accurately align these point clouds and fuse them into a unified model. The first step in point cloud registration is initial alignment, which typically employs feature-based registration methods, such as the Random Sample Consensus (RANSAC) algorithm. This algorithm randomly selects a set of feature point pairs from the source and target point clouds, calculates their transformation matrix to achieve initial alignment, and then applies the transformation to the source point cloud and evaluates the distance between it and the target point cloud. The alignment quality is assessed by the number of interior points and a distance threshold, and the transformation matrix is ​​iteratively optimized to ultimately achieve high-precision registration. Fine registration not only improves the accuracy of point cloud alignment but also ensures that the geometric features of each part of the weld bead can be accurately matched.

[0043] S7. Establish a weld seam point cloud neural network, including the PointNet network;

[0044] S8. Perform weld defect detection based on neural networks, including weld defect classification using deep convolutional neural networks, weld defect classification based on PointNet++, and weld defect classification based on DGCNN. Deep convolutional neural networks can naturally process data through end-to-end integration of multi-layer features, and the depth of feature extraction can be enhanced by increasing the number of layers. The PointNet++ training uses the same dataset as PointNet, and uses negative log-likelihood loss as the loss function. The optimizer uses the Adam algorithm to further improve the stability and accuracy of the model. The DGCNN training also uses the same dataset as PointNet, and uses labeled smooth cross-entropy loss as the loss function. The optimizer uses the Adam algorithm, and random dropping of input points during training improves the generalization ability of the model.

[0045] In this embodiment, the method integrates multimodal feature learning, information monitoring, and quality assessment technologies based on point cloud data, welding images, and welding parameter information. This enables intelligent monitoring and diagnosis of various types of defects in the welding process, such as cracks, undercuts, voids, and pits, under small sample conditions. It identifies the causes of defects caused by key parameters such as current, voltage, wire feed speed, and welding speed, greatly improving the defect detection accuracy of the method.

[0046] In this embodiment, weld defect detection can also be performed through vibration analysis, acoustic emission detection, infrared thermal imaging, machine vision and image processing and multimodal data fusion.

[0047] Vibration analysis, in particular, involves applying excitation to the weld area and measuring its vibration response to analyze the dynamic characteristics of the weld structure. Defects in the weld often alter the stiffness or damping characteristics of the structure, which manifests in the vibration response. Vibration data can be collected using accelerometers or other sensors, and potential defects can be identified through signal processing and analysis techniques.

[0048] Acoustic emission detection uses an array of acoustic emission sensors to capture these signals and locate defects by analyzing their characteristics.

[0049] Infrared thermal imaging can capture the heat distribution in a weld area. Defects in the weld may affect the heat conduction path, thus appearing as temperature anomalies in the infrared image. The weld can be thermally stimulated (e.g., heated), and then the changes in temperature distribution in the infrared image can be observed to identify potential defects.

[0050] Machine vision, combined with image processing, utilizes high-resolution cameras and image processing algorithms to perform visual inspections of weld surfaces. While this method is primarily suitable for surface defect detection, combining it with techniques such as deep learning can improve detection accuracy and efficiency. Image segmentation, feature extraction, and classification algorithms can be used to automatically identify and locate defects on weld surfaces.

[0051] By fusing the aforementioned non-modal data with conventional detection data (such as ultrasound, X-rays, etc.), the accuracy and reliability of defect detection can be further improved. Through data fusion algorithms (such as Bayesian networks, neural networks, etc.), data information from different modalities can be integrated and analyzed to obtain more comprehensive defect information.

[0052] In S8 above, the Adam algorithm used by the optimizer has been optimized and improved, and the specific improvements are as follows:

[0053] Improvements were made at the algorithm level, specifically:

[0054] 1. Correction of Second-Order Moment Estimation: Addressing Convergence Defects

[0055] AMSGrad

[0056] Improvement: The replacement index has averaged its historical maximum.

[0057] vt=max(β2V t−1 +(1−β2)g t 2 ,v t−1 );

[0058] Effect: Prevents premature decay of the adaptive learning rate and ensures convergence to a better solution (especially for non-convex optimization).

[0059] 2. Weight decay decoupling: improving generalization ability

[0060] AdamW

[0061] Improvement: Separate weight decay from gradient update:

[0062] Results: Avoids the conflict between adaptive learning rate and weight decay, significantly improving the generalization performance of large models such as Transformer.

[0063] 3. Momentum Mechanism Optimization: Accelerating Convergence

[0064] Nadam

[0065] Improvement: Introduce Nesterov momentum look-ahead gradient:

[0066] M t =β1M t +(1−β1)g t

[0067] Effects: Reduces oscillations, provides a smoother path in high curvature loss surfaces, and accelerates convergence by 10%~30%.

[0068] 4. Enhanced noise robustness

[0069] AdaBelief

[0070] Improvement: Replace the squared gradient with gradient bias.

[0071] S t =β2S t−1 +(1−β2)(g t -M t ) 2

[0072] Results: More stable for noisy data and mini-batch training, with improved convergence speed and reduced fluctuations.

[0073] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for weld defect detection and multimodal neural network model construction, characterized in that: Includes the following steps: S1. Use a line light source to scan the surface of the weldment to obtain the original image and depth data containing contour information; S2. Perform image denoising processing; S3. Extract the center trajectory; S4. Outlier removal is performed using point cloud statistical methods based on Gaussian distribution models. S5. Perform the farthest point sampling; S6. Perform three-dimensional reconstruction and calibration of the weld seam; S7. Establish a neural network for weld seam point cloud; S8. Perform weld defect detection based on neural networks.

2. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S1, a line light source is used to scan the surface of the weldment to obtain the original image and depth data containing contour information. This method has low requirements for the surface to be tested and has the advantages of high speed, high efficiency and high precision.

3. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S2, image denoising is performed, and the image denoising algorithms used include mean filtering, median filtering, Gaussian filtering, and adaptive median filtering.

4. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S3, the center trajectory is extracted using the Steger algorithm. The Steger algorithm achieves high-precision light band center extraction based on the Hessian matrix. It constructs the Hessian matrix by calculating the second derivative of the image, performs eigenvalue decomposition, determines the direction of the light band center, and achieves sub-pixel-level positioning through Taylor expansion. This algorithm exhibits high precision and strong robustness when dealing with complex light band shapes and image noise. The prior constraint of the weld direction is introduced in the Hessian matrix calculation to optimize the robustness of the light band center extraction.

5. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In S5, sampling of the farthest point is performed. Specifically, a starting point is selected from the point cloud data, and then the point farthest from the current point set is selected iteratively until the set number is met. This sampling method can ensure the maximization of spatial coverage and the uniformity of point distribution.

6. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S6, the weld seam is reconstructed and calibrated in three dimensions. The specific steps are: extracting weld seam features, reconstructing in three dimensions, and registering and fusing point clouds. In the process of reconstructing the weld seam in three dimensions, the quality of the point cloud data directly affects the accuracy and reliability of the reconstruction model.

7. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S6, weld feature extraction is performed to identify and describe the geometric features, shape information and other key attributes of the weld. The feature extraction methods used include normal estimation, curvature calculation, cross-sectional area calculation, maximum point calculation, average height calculation and feature point extraction.

8. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S6, three-dimensional reconstruction is performed, and the surface reconstruction methods used include Poisson reconstruction and Delaunay triangulation.

9. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S6, point cloud registration and fusion are performed. During the welding process, due to differences in perspective, time, and sensors, the obtained point cloud data often exhibit inconsistencies in position, orientation, and scale. Therefore, it is necessary to accurately align these point clouds and fuse them into a unified model. The first step in point cloud registration is initial alignment, which typically employs a feature-based registration method, such as the Random Sample Consensus (RANSAC) algorithm. This algorithm randomly selects a set of feature point pairs from the source and target point clouds, calculates their transformation matrix to achieve initial alignment, and then applies the transformation to the source point cloud and evaluates the distance between it and the target point cloud. The alignment quality is assessed by the number of interior points and a distance threshold, and the transformation matrix is ​​iteratively optimized to ultimately achieve high-precision registration. Fine registration not only improves the accuracy of point cloud alignment but also ensures that the geometric features of each part of the weld bead can be accurately matched.

10. The method for weld defect detection and multimodal neural network model construction according to claim 1, characterized in that, In step S7, a weld seam point cloud neural network is established, including a PointNet network. In step S8, weld seam defect detection based on neural networks is performed, including weld defect classification using deep convolutional neural networks, weld defect classification based on PointNet++, and weld defect classification based on DGCNN. Deep convolutional neural networks can naturally process data through end-to-end integration of multi-layer features, and the depth of feature extraction is enhanced by increasing the number of layers. The training of PointNet++ uses the same dataset as PointNet, and uses negative log-likelihood loss as the loss function. The optimizer uses the Adam algorithm to further improve the stability and accuracy of the model. The training of DGCNN also uses the same dataset as PointNet, and uses labeled smooth cross-entropy loss as the loss function. The optimizer uses the Adam algorithm, and random dropping of input points during training improves the generalization ability of the model.

Images