Weld tracking and evaluation method and system based on machine vision

Abstract

The application provides a welding seam tracking and evaluation method and system based on machine vision, relates to the field of machine vision, and solves the technical problems that the prior art cannot accurately reconstruct the three-dimensional appearance and geometric features of a welding seam, resulting in a large deviation of welding seam quality evaluation results and a high misjudgment rate. The method comprises the following steps: acquiring a multispectral image sequence of a welding seam area, and performing spatiotemporal adaptive weighted fusion to generate a fused image with high signal-to-noise ratio; inputting the fused image into a deep learning segmentation network trained by physical laws to segment the contour of a two-dimensional welding seam; reconstructing a three-dimensional point cloud model of the welding seam based on the contour of the two-dimensional welding seam and multi-view geometric information; extracting multi-scale geometric features of the welding seam according to the three-dimensional point cloud model and the accurate contour of the welding seam; and inputting the multi-scale geometric features into a pre-trained graph neural network to output a defect index of the welding seam. The application is used in the welding seam tracking and evaluation process.

Description

Technical Field

[0001] This application relates to the field of machine vision, and in particular to a method and system for weld seam tracking and evaluation based on machine vision. Background Technology

[0002] In the field of modern intelligent manufacturing and automated welding, weld seam tracking and evaluation technology based on machine vision has become a key means to improve welding quality and achieve closed-loop control. Currently, the industry generally uses structured light, laser scanning, or passive binocular / multi-view vision systems to observe weld seams online, and combines image processing or point cloud analysis algorithms to extract the weld seam position, contour, and geometric features.

[0003] In recent years, with the development of deep learning, some studies have attempted to introduce convolutional neural networks (CNNs) or Transformer models for weld seam segmentation and centerline prediction, initially achieving end-to-end perception from image to 3D shape. However, these methods mostly focus on ideal or high-contrast scenes. In actual welding environments, factors such as strong arc light, smoke, metal spatter, surface reflection, and workpiece occlusion severely interfere with image quality, resulting in insufficient robustness of traditional vision algorithms, while purely data-driven models have limited generalization ability due to the lack of physical constraints.

[0004] Therefore, this application provides a machine vision-based method and system for weld seam tracking and evaluation. Summary of the Invention

[0005] This application provides a machine vision-based method and system for weld tracking and evaluation, which solves the technical problem that existing technologies cannot accurately reconstruct the three-dimensional morphology and geometric features of welds, resulting in large deviations and high misjudgment rates in weld quality evaluation results.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, it provides a machine vision-based method for weld seam tracking and evaluation, including: During the welding process, a multispectral image sequence of the weld area is acquired simultaneously. The multispectral image sequence includes images in the visible light band, near-infrared band, and short-wave infrared band. The multispectral image sequence is subjected to spatiotemporal adaptive weighted fusion to generate a fused image with a high signal-to-noise ratio; The fused image is input into a deep learning segmentation network trained by physical laws to segment the outline of the two-dimensional weld. Based on the outline of the two-dimensional weld and the multi-view geometric information, the three-dimensional point cloud model of the weld is reconstructed in real time. Based on the three-dimensional point cloud model and the precise contour of the weld, multi-scale geometric features of the weld are extracted; the multi-scale geometric features are used to characterize the geometric parameters related to the structural integrity of the weld joint. The multi-scale geometric features are input into a pre-trained graph neural network, which outputs a defect index of the weld, which is used to assess the degree of defect in the weld.

[0007] Based on the above technical solutions, in the machine vision-based weld seam tracking and evaluation method provided in this application, traditional methods are limited by single-spectrum imaging and are easily interfered with by strong arc light, smoke, and metal reflection in online welding quality evaluation, resulting in unstable weld seam contour extraction and affecting the reliability of 3D reconstruction and defect identification. To solve this problem, this solution proposes a multi-spectral fusion-driven intelligent weld seam perception and evaluation framework. Visible light, near-infrared, and short-wave infrared images are acquired simultaneously. Utilizing the complementarity of each band's thermal radiation, surface reflection, and penetration capabilities, a high signal-to-noise ratio image is generated through spatiotemporal adaptive weighted fusion, significantly suppressing interference and enhancing weld seam edge information. Physical laws (such as heat conduction models and prior knowledge of molten pool morphology) are introduced to guide the training of a deep learning segmentation network, improving the generalization and robustness of contour segmentation. Furthermore, high-precision 3D point clouds are reconstructed in real time using multi-view geometric constraints, and a multi-scale anisotropic neighborhood is constructed based on the weld seam orientation to extract geometric features that reflect key structural parameters such as weld height, width, and curvature. A quantitative defect index is output by modeling the topological relationships between features through a graph neural network. This solution deeply integrates multispectral sensing, physical priors, and graph structure learning, enabling the stable extraction of weld geometric semantics from highly disturbed environments and accurate assessment of its structural integrity, providing a reliable basis for closed-loop control of welding in intelligent manufacturing.

[0008] In conjunction with the first aspect above, in one possible implementation, the method for acquiring the high signal-to-noise ratio fused image includes: For each frame of the multispectral image sequence, a corresponding spatial quality map is generated through a branch of a convolutional neural network. The spatial quality map is used to characterize the local reliability of each pixel region in the image affected by welding interference. For each band of image sequence, the global temporal stability score, which characterizes the current imaging stability of the band, is obtained by calculating the structural similarity between the current frame and historical frames. The spatial quality map and the temporal stability score are input into a nonlinear gating unit to generate a spatiotemporally adaptive weighted map. Using the spatiotemporal adaptive weight map, the current frame images of all bands are weighted and summed to generate the fused image with high signal-to-noise ratio.

[0009] In conjunction with the first aspect above, in one possible implementation, the training process of the deep learning segmentation network includes constructing a physical consistency loss function, which characterizes the following differences: Based on the reconstructed three-dimensional model of the weld, its actual spatial curvature distribution is calculated; Based on preset welding process parameters, the theoretical spatial curvature distribution of the weld is calculated using a heat conduction physical model. The physical consistency loss function is a measure of the difference between the actual spatial curvature distribution and the theoretical spatial curvature distribution; By minimizing the physical consistency loss function, the parameters of the deep learning segmentation network are optimized so that the predicted weld contour conforms to the thermodynamic physical laws of welding in three-dimensional space.

[0010] In conjunction with the first aspect above, in one possible implementation, the method for obtaining the theoretical spatial curvature distribution includes: Based on the welding current, voltage, welding speed and material thermophysical parameters, the steady-state temperature field of the weld pool is calculated using the Rosenthal moving heat source model. Based on the temperature field, the isotherms corresponding to the molten pool boundary are determined, and the isotherms are fitted to the theoretical profile curve of the weld cross-section. The theoretical contour curve is parameterized in three-dimensional space to obtain the theoretical spatial trajectory of the weld centerline. By performing differential geometric operations on the theoretical spatial trajectory, its local curvature is calculated, thereby obtaining the curvature distribution of the theoretical space.

[0011] In conjunction with the first aspect above, in one possible implementation, calculating the actual spatial curvature distribution of the three-dimensional weld profile includes: For each point in the 3D point cloud model, obtain its corresponding predicted position uncertainty metric; based on the predicted position uncertainty metric, dynamically calculate the neighborhood search radius of the point through a preset mapping function; wherein, the neighborhood search radius is positively correlated with the predicted position uncertainty; Based on the neighborhood search radius, local neighborhoods of the target point are constructed at multiple preset scales, and the ratio of the minimum eigenvalue to the sum of eigenvalues ​​of the covariance matrix is ​​calculated by principal component analysis at each scale to obtain the local curvature response at that scale. The initial curvature value of the target point is obtained by weighted averaging of the local curvature response at multiple scales. Based on the angle between the normal vectors of the target point and its neighboring points, a smoothing constraint is applied to the initial curvature value to generate a corrected spatial curvature value. Iterate through all points and output the actual spatial curvature distribution corresponding to the predicted weld contour.

[0012] In conjunction with the first aspect above, in one possible implementation, the method for obtaining the predicted location uncertainty measure includes: Obtain the corresponding two-dimensional contour point of each point in the three-dimensional point cloud model in the multi-view image; Based on the segmentation confidence or edge gradient magnitude of the two-dimensional contour points, estimate their two-dimensional positioning uncertainty; Using camera calibration parameters and the Jacobian matrix of multi-view triangulation, the two-dimensional positioning uncertainty is propagated to three-dimensional space to obtain the position covariance matrix of the point. The measurement of the predicted location uncertainty is the trace or the largest eigenvalue of the covariance matrix.

[0013] In conjunction with the first aspect above, in one possible implementation, the reconstructed three-dimensional point cloud model of the weld seam includes: Based on the contour of the two-dimensional weld, a narrow band of reliable matching region is generated on the image from each viewpoint, around the center line of the weld; the center line of the weld contains multiple center line points, each center line point being associated with a corresponding spatial position uncertainty; When performing multi-view stereo matching, for pixels located within the reliable matching area, a parallax search range with a preset resolution and an enhanced matching cost are calculated. In the matching cost aggregation and optimization stage, an anisotropic regularization term is introduced, the weight of which is determined by the contour of the two-dimensional weld; wherein, there is a first smoothing weight in the direction parallel to the weld direction, and a second smoothing weight in the direction perpendicular to the weld direction, and the second smoothing weight is greater than the first smoothing weight, so as to enhance the geometric fidelity of the weld edge. Based on the optimized disparity map and combined with camera calibration parameters, a three-dimensional point cloud model of the weld is obtained through triangulation calculation.

[0014] In conjunction with the first aspect above, in one possible implementation, the method for obtaining the narrowband reliable matching region includes: Obtain the weld centerline predicted by an artificial intelligence model. The weld centerline contains multiple centerline points, each of which is associated with a corresponding spatial position uncertainty. The centerline of the weld is projected onto the image plane of each viewpoint to obtain the pixel trajectory of the centerline under each viewpoint. Calculate the local image gradient magnitude of each centerline pixel in each viewpoint, and determine the local image confidence level of that point based on the gradient magnitude; Based on the spatial location uncertainty and the local image confidence level, the narrow band half-width at the center line pixel is calculated; wherein, the narrow band half-width increases with increasing uncertainty and decreases with decreasing image confidence level; Centered on the centerline pixel, a line segment with a length twice the width of the narrow band is generated along the direction perpendicular to the tangent of the centerline at that point. Line segments with local image credibility below a preset credibility threshold are removed, and the union of the remaining line segments is taken as the narrowband credibility matching region under that viewpoint.

[0015] In conjunction with the first aspect above, in one possible implementation, the multi-scale geometric feature extraction method for the weld seam includes: Based on the precise two-dimensional contour of the weld, the center line of the weld is fitted in the three-dimensional point cloud model, and the tangential, normal, and subnormal directions of each point on the center line are calculated. Using the weld centerline as a reference, multiple elliptical cylindrical analysis neighborhoods of preset scales are set along the normal and tangential directions respectively, forming a multi-scale, anisotropic local point cloud subset; In the local point cloud subset at each scale, the width distribution characteristics along the normal direction, the smoothness characteristics along the tangential direction, and the curvature and co-height statistical characteristics along the height direction are calculated respectively to obtain multiple sets of scale-related geometric descriptors. The multiple sets of geometric descriptors are weighted and fused across scales to generate multi-scale geometric feature vectors.

[0016] Secondly, this application provides a machine vision-based weld seam tracking and evaluation system, comprising: an acquisition module, an analysis module, and a defect evaluation module; wherein, the acquisition module is used to acquire a multispectral image sequence of the weld seam region during the welding process; the analysis module is used to perform spatiotemporal adaptive weighted fusion on the multispectral image sequence to generate a fused image with a high signal-to-noise ratio; input the fused image into a deep learning segmentation network trained by physical laws to segment the contour of the two-dimensional weld seam; based on the contour of the two-dimensional weld seam and multi-view geometric information, reconstruct a three-dimensional point cloud model of the weld seam in real time; extract multi-scale geometric features of the weld seam according to the three-dimensional point cloud model and the precise contour of the weld seam; the defect evaluation module is used to input the multi-scale geometric features into a pre-trained graph neural network and output the defect index of the weld seam.

[0017] This application provides a machine vision-based method and system for weld seam tracking and evaluation, which can effectively cope with complex interferences such as strong arc light, spatter, smoke, and high metal reflectivity during the welding process, significantly improving the stability and accuracy of weld seam perception. By simultaneously acquiring multispectral image sequences in visible light, near-infrared, and short-wave infrared, and employing a spatiotemporal adaptive weighted fusion strategy, the complementary advantages of different bands in terms of penetration, thermal radiation response, and edge contrast are fully utilized to generate a fused image with high signal-to-noise ratio and complete detail preservation. Based on this, welding physics laws (such as molten pool dynamics and heat conduction characteristics) are introduced as prior knowledge to guide the training of a deep learning segmentation network, enabling the model to not only rely on data-driven approaches but also possess a physical consistency understanding of the real weld morphology, thereby accurately segmenting the two-dimensional weld seam contour. Combined with multi-view geometric constraints, real-time high-precision reconstruction of the weld seam's three-dimensional point cloud model is achieved. Furthermore, multi-scale geometric features covering local curvature, width variation, and excess height distribution are extracted and input into a pre-trained graph neural network to model the spatial topological relationship of the weld seam structure and output a quantifiable defect index. This method achieves intelligent operation across the entire chain from perception and reconstruction to evaluation, providing highly reliable technical support for closed-loop control, quality prediction, and process optimization in the welding process, and has significant engineering application value.

[0018] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a machine vision-based weld seam tracking and evaluation method provided in this application embodiment; Figure 2 A flowchart illustrating an analysis method for the actual spatial curvature distribution of a three-dimensional weld contour provided in this application embodiment; Figure 3 This is a flowchart illustrating a method for obtaining a three-dimensional point cloud model of a weld seam, as provided in an embodiment of this application. Detailed Implementation

[0020] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The machine vision-based weld seam tracking and evaluation method provided in this application embodiment can be applied to a machine vision-based weld seam tracking and evaluation system, which includes: an acquisition module, an analysis module, and a defect evaluation module. The acquisition module is used to acquire multispectral image sequences of the weld area during the welding process; The analysis module is used to perform spatiotemporal adaptive weighted fusion of multispectral image sequences to generate a fused image with a high signal-to-noise ratio; The fused image is input into a deep learning segmentation network trained by physical laws to segment the outline of the two-dimensional weld. Based on the contour of the two-dimensional weld and multi-view geometric information, a three-dimensional point cloud model of the weld is reconstructed in real time. Based on the 3D point cloud model and the precise contour of the weld, multi-scale geometric features of the weld are extracted. The defect assessment module is used to input multi-scale geometric features into a pre-trained graph neural network and output the defect index of the weld.

[0022] To address the technical problem that existing technologies cannot accurately reconstruct the three-dimensional morphology and geometric features of welds, leading to large deviations and high misjudgment rates in weld quality assessment results, embodiments of this application provide a machine vision-based weld tracking and assessment method. This method includes: acquiring a multispectral image sequence of the weld region during the welding process; performing spatiotemporal adaptive weighted fusion on the multispectral image sequence to generate a fused image with a high signal-to-noise ratio; and inputting the fused image into a deep learning segmentation network trained according to physical laws to segment the contour of the two-dimensional weld. Based on the 2D weld contour and multi-view geometric information, a 3D point cloud model of the weld is reconstructed in real time. Multi-scale geometric features of the weld are extracted based on the 3D point cloud model and the precise weld contour. These multi-scale geometric features are input into a pre-trained graph neural network, which outputs a defect index of the weld. Based on this, the technical solution effectively suppresses interference from strong welding arc light and fumes through multispectral image fusion, significantly improving the robustness of weld contour segmentation. A deep learning network guided by physical laws enhances the model's generalization ability to real weld morphology. High-precision 3D point cloud reconstruction is achieved using multi-view geometric information, and multi-scale geometric features that comprehensively characterize the weld structure integrity are extracted. Finally, a graph neural network is used to model the spatial correlation between features, outputting a quantified defect index. The entire process enables stable and accurate perception of the weld geometric state from complex interference environments, providing reliable support for online welding quality assessment and intelligent control.

[0023] like Figure 1 As shown in the embodiments of this application, the machine vision-based weld seam tracking and evaluation method includes: S201. Obtain a multispectral image sequence of the weld area during the welding process.

[0024] The multispectral image sequence includes images in the visible light band, near-infrared band, and short-wave infrared band.

[0025] In some implementations, multispectral image sequences are obtained by simultaneously acquiring radiation information of the welding area in the visible (400-700nm), near-infrared (NIR, 780-1100nm), and short-wave infrared (SWIR, 1.4-2.5μm) bands, in order to capture the weld surface morphology, molten pool thermal radiation distribution, and internal thermal conduction characteristics of the material, respectively.

[0026] It should be noted that different wavelengths have different sensitivities to physical phenomena during the welding process: the visible light band is mainly used to observe the geometric profile of the weld and the spatter state; the near-infrared band has a strong response to the thermal radiation of the high-temperature molten pool and is suitable for dynamic monitoring of the molten pool; while the short-wave infrared band can penetrate some smoke and plasma interference and reflect the thermal diffusion behavior of the material on the subsurface, which helps to identify the root forming quality.

[0027] For example, a multi-camera fusion system can be used, in which one camera is equipped with an RGB sensor for visible light imaging and another is equipped with an InGaAs sensor for near-infrared and short-wave infrared imaging. Frame-level synchronization is achieved through hardware triggering, thereby constructing a spatiotemporally aligned multispectral image sequence to provide a data foundation for subsequent multi-scale geometric and thermodynamic joint analysis.

[0028] S202. Perform spatiotemporal adaptive weighted fusion on the multispectral image sequence to generate a fused image with high signal-to-noise ratio.

[0029] In some implementations, spatiotemporal adaptive weighted fusion of multispectral image sequences is performed, including: dynamically calculating the confidence weight of each pixel in different bands based on the local contrast, thermal radiation intensity, and temporal stability of each band image, and performing sliding window aggregation on high-confidence frames along the time dimension to generate a fused image with high signal-to-noise ratio, clear geometric structure, and complete preservation of thermal features.

[0030] It should be noted that interference factors such as strong arc light, smoke and dust obstruction, and violent fluctuations in the molten pool exist during the welding process, which may cause information loss or noise surges in single-band images at different times. By introducing a spatiotemporal adaptive mechanism, instantaneous interference can be effectively suppressed, the visibility of key areas (such as bevel edges and fusion lines) can be enhanced, and the problems of detail blurring or artifacts caused by traditional fixed-weight fusion can be avoided.

[0031] S203. Input the fused image into a deep learning segmentation network trained by physical laws to segment the outline of the two-dimensional weld.

[0032] In some implementations, the fused image is input into a deep learning segmentation network trained by physical laws. This network, based on conventional semantic segmentation architectures (such as U-Net or DeepLab), embeds physical constraints such as welding heat conduction, molten pool fluid dynamics, or weld formation geometry priors as regularization terms or loss functions. This guides the network to more accurately predict two-dimensional weld contours that conform to actual welding physical behavior during training.

[0033] It should be noted that traditional data-driven segmentation models are susceptible to noise, changes in lighting, or rare defect morphologies, causing the contour prediction to deviate from the actual weld geometry. However, introducing physical laws (such as molten pool symmetry, bevel angle limitations, and the diffusion range of the heat-affected zone) can significantly improve the model's generalization ability and geometric rationality under complex working conditions, and it is particularly robust in small sample or weakly labeled scenarios.

[0034] S204. Based on the contour of the two-dimensional weld and multi-view geometric information, a three-dimensional point cloud model of the weld is reconstructed in real time.

[0035] In some implementations, based on the contour of the two-dimensional weld and multi-view geometric information, a high-precision three-dimensional point cloud model of the weld is generated in real time by fusing the camera pose parameters corresponding to the multispectral fusion image with stereo matching or structured light projection data and utilizing the triangulation principle or deep learning-driven multi-view stereo (MVS) algorithm. Among these, the two-dimensional contour is used as a semantic prior to constrain the effective area of ​​depth estimation, thereby improving the geometric integrity and edge sharpness of the reconstruction.

[0036] It should be noted that welding sites often have strong reflections, smoke and dust obstruction, and high temperature disturbances, which makes traditional 3D reconstruction methods prone to producing voids or noise in key areas of the weld (such as root gaps and undercuts). However, by co-optimizing semantic segmentation results with multi-view geometric information, interference matching in non-weld areas can be effectively eliminated, and the focus can be placed on the recovery of high-value structural features, thereby improving reconstruction robustness while ensuring real-time performance.

[0037] S205. Based on the 3D point cloud model and the precise contour of the weld, extract the multi-scale geometric features of the weld.

[0038] Among them, multi-scale geometric features are used to characterize the geometric parameters related to the structural integrity of weld joints.

[0039] In some implementations, based on the 3D point cloud model and the precise contour of the weld, multi-scale geometric features of the weld are extracted using multi-scale surface analysis methods (such as local normal vector estimation, principal curvature calculation, Gaussian-mean curvature decomposition) and region aggregation strategies (such as sliding sphere neighborhood and adaptive voxel partitioning). These features cover surface roughness at the microscale, the rate of change of weld height and weld width at the mesoscale, and joint alignment deviation and groove filling continuity at the macroscale, which together constitute a geometric parameter system that can discriminate the structural integrity of the weld joint.

[0040] It should be noted that a single-scale geometric description is insufficient to fully reflect the impact of welding defects (such as lack of fusion, undercut, and collapse) on the structural load-bearing capacity; while multi-scale feature fusion can simultaneously capture local mutations and overall deformation, improve the sensitivity to potential failure modes, and provide high-dimensional and interpretable input for subsequent quality assessment or process closed-loop control.

[0041] S206. Input the multi-scale geometric features into a pre-trained graph neural network and output the defect index of the weld.

[0042] Among them, the weld defect index is used to assess the degree of defect in the weld.

[0043] In some implementations, multi-scale geometric features are input into a pre-trained Graph Neural Network (GNN). This network models the 3D point cloud or segmented geometric features of the weld as a graph structure, where nodes represent local geometric regions (such as voxel blocks, contour segments, or key points), and edges are constructed based on spatial adjacency, curvature similarity, or process continuity. The GNN aggregates multi-scale contextual information through a message passing mechanism, learns defect-sensitive high-order geometric relationships, and regresses to output a standardized weld defect index to quantify the overall defect degree of the weld.

[0044] It should be noted that traditional defect assessment methods based on rules or shallow machine learning are difficult to effectively model the nonlocal dependencies and topological relationships between weld geometric features. Graph neural networks, on the other hand, are naturally adapted to irregular point cloud data and can capture implicit relationships such as those between unfused areas and abrupt changes in adjacent weld height, thereby improving the ability to identify complex or hidden defects. In addition, by jointly pre-training physical simulation and measured data on a large number of labeled weld samples, GNNs can be generalized to different joint types and welding process scenarios.

[0045] For example, the weld can be divided into several nodes along its length, with each node embedding its corresponding multi-scale geometric feature vector (such as curvature statistics, bevel symmetry, local volume deviation, etc.), and edge connections can be constructed through K-nearest neighbors or Delaunay triangulation; multi-layer feature propagation and fusion can be performed using graph convolutional networks (GCN) or graph attention networks (GAT), and graph-level representation can be generated through global readout operations, and a defect index in the range of 0–1 can be output through a fully connected regression head. The higher the value, the greater the possibility of structural defects such as undercut, incomplete penetration, and collapse. This index can be directly used for automated quality inspection grading or rework decision-making.

[0046] Based on the above technical solutions, the machine vision-based weld seam tracking and evaluation method provided in this application addresses the limitations of traditional methods in online welding quality evaluation. Traditional methods, relying on single-spectrum imaging, are susceptible to interference from strong arc light, smoke, and metal reflection, leading to unstable weld seam contour extraction and consequently affecting the reliability of 3D reconstruction and defect identification. To solve this problem, this solution proposes a multi-spectral fusion-driven intelligent weld seam perception and evaluation framework. Visible, near-infrared, and short-wave infrared images are acquired simultaneously. Utilizing the complementarity of each band's thermal radiation, surface reflection, and penetration capabilities, a high signal-to-noise ratio image is generated through spatiotemporal adaptive weighted fusion, significantly suppressing interference and enhancing weld seam edge information. Physical laws (such as heat conduction models and prior knowledge of molten pool morphology) are introduced to guide the training of a deep learning segmentation network, improving the generalization and robustness of contour segmentation. Furthermore, high-precision 3D point clouds are reconstructed in real-time using multi-view geometric constraints, and a multi-scale anisotropic neighborhood is constructed based on the weld seam orientation to extract geometric features reflecting key structural parameters such as weld height, width, and curvature. Finally, a quantitative defect index is output by modeling the topological relationships between features using a graph neural network. This solution deeply integrates multispectral sensing, physical priors, and graph structure learning, enabling the stable extraction of weld geometric semantics from highly disturbed environments and accurate assessment of its structural integrity, providing a reliable basis for closed-loop control of welding in intelligent manufacturing.

[0047] In one possible implementation of this application embodiment, the above-mentioned S202 can be specifically implemented by the following S301-S304, which will be described in detail below: S301. For each frame of the multispectral image sequence, generate the corresponding spatial quality map through a branch of a convolutional neural network.

[0048] Among them, the spatial quality map is used to characterize the local reliability of each pixel area in the image under welding interference (such as strong arc light, spatter, and smoke obstruction).

[0049] In some implementations, the convolutional neural network branch can adopt a lightweight encoder-decoder structure (such as a variant of U-Net), taking the original multispectral frame as input and outputting a single-channel confidence map with the same size as the input. Its value range is usually normalized to [0,1], and a high value indicates that the region is clearly imaged and less affected by interference.

[0050] It should be noted that different spectral bands (such as visible light, near-infrared, and thermal infrared) have significantly different sensitivities to welding interference. Therefore, the spatial quality map needs to have band adaptive capability to accurately reflect the true distribution of local imaging quality in each spectral band.

[0051] For example, band embedding or channel attention mechanisms (such as the SE module) can be introduced into the network to enable the model to dynamically adjust the feature extraction strategy according to the input band, thereby generating a more discriminative spatial quality map.

[0052] S302. For each band of image sequence, by calculating the structural similarity between the current frame and historical frames, a global temporal stability score characterizing the current imaging stability of that band is obtained.

[0053] The temporal stability score is used to quantify the consistency of the current frame with the historical normal imaging state in the time dimension, reflecting whether the band is in a stable observation window.

[0054] In some implementations, the Structural Similarity Index (SSIM) or its improved versions (such as MS-SSIM, CW-SSIM) can be used to calculate the average similarity between the current frame and several historical reference frames within the sliding window, and the result can be compressed into a single scalar as a global temporal stability score.

[0055] It should be noted that the welding process itself is dynamic (such as molten pool oscillation and welding torch movement). If the original pixels are used directly for comparison, normal dynamic changes may be misjudged as unstable. Therefore, the historical reference frames should select data from the steady-state stage of the process, or the similarity should be calculated after alignment through motion compensation.

[0056] For example, optical flow can be used to perform motion correction on historical frames, and then the compensated SSIM mean can be calculated to obtain a time stability assessment that is more sensitive to real imaging disturbances (rather than process dynamics).

[0057] S303. Input the spatial quality map and temporal stability score into a nonlinear gated unit, a spatiotemporal adaptive weighted map.

[0058] Among them, the spatiotemporal adaptive weight map integrates pixel-level spatial reliability and temporal global stability, and is used to guide information selection in the subsequent fusion process.

[0059] In some implementations, the nonlinear gating unit can be designed as a modulation module consisting of fully connected layers or small convolutional networks. After broadcasting the scalar temporal stability score to the spatial dimension, it is fused element-wise with the spatial quality map (e.g., by multiplication, weighted summation, or sigmoid gating) to output a weight map of the same size as the image.

[0060] It should be noted that simple linear combinations may not be able to fully model the coupling relationship between spatiotemporal factors (for example, even if the spatial quality of a certain region is high, if the overall band time is unstable, its credibility should be suppressed). Therefore, nonlinear activation (such as tanh, softmax) or attention mechanisms need to be introduced to achieve dynamic modulation.

[0061] For example, the temporal stability score can be used as a conditional input, and the feature channels of the spatial quality map can be adjusted through conditional batch normalization or FiLM (Feature-wise Linear Modulation) mechanism to generate a context-aware spatiotemporal weight map.

[0062] S304. Using a spatiotemporal adaptive weighted graph, the current frame images of all bands are weighted and summed to generate a fused image with a high signal-to-noise ratio.

[0063] Among them, the high signal-to-noise ratio fused image aims to retain the least disturbed and most information-rich areas in each band, while suppressing the effects of noise, overexposure, or occlusion.

[0064] In some implementations, the current frame of each band is multiplied element-wise with its corresponding spatiotemporal adaptive weight map, and then the results of all bands are weighted and summed in the channel dimension or stitched together and then convolved to fuse, outputting a single or multi-channel fused image.

[0065] It should be noted that directly weighting the original pixel intensity may lead to fusion bias due to differences in dimensions or dynamic range between bands; therefore, it is advisable to first normalize or logarithmically transform the images of each band to ensure that the weights are applied to comparable feature spaces.

[0066] For example, images of different bands can be mapped to a unified perceptual feature space, such as by extracting shallow features through a shared convolutional layer, and then fusion is performed by applying a weight map. A lightweight reconstruction head can then be used to restore high-fidelity output, thereby improving the overall signal-to-noise ratio and robustness while preserving details.

[0067] Based on the above technical solutions, existing welding visual inspection technologies are susceptible to varying degrees of noise in different bands of multispectral images under complex interference such as strong arc light and fumes. Furthermore, a single fixed fusion strategy cannot account for dynamic changes in the spatiotemporal dimensions, resulting in low signal-to-noise ratio and loss of key geometric information in the fused image, thus limiting the accuracy of subsequent segmentation and 3D reconstruction. To address this problem, this solution innovatively constructs a spatiotemporal adaptive fusion mechanism: a spatial quality map representing the reliability of local pixels is generated through CNN branches, and a global time fraction reflecting the overall stability of the bands is calculated by combining structural similarity; the two are deeply fused through a nonlinear gating unit to dynamically generate spatiotemporal weights that accurately suppress interference and retain effective information, achieving high signal-to-noise ratio fusion. This solution abandons the limitations of traditional static weighting, starting from the essence of physical interference, and realizes "on-demand" access to multispectral information, significantly improving the robustness and fidelity of the fused image, laying a reliable data foundation for subsequent high-precision weld evaluation.

[0068] In one possible implementation of this application embodiment, the training process of the deep learning segmentation network in S203 above includes constructing a physical consistency loss function, which represents the following differences, as detailed in S401-S404.

[0069] S401. Based on the reconstructed three-dimensional model of the weld, calculate its actual spatial curvature distribution, such as... Figure 2 As shown, the details are as follows: S4011. For each point in the 3D point cloud model, obtain its corresponding prediction location uncertainty metric, as follows: S40111. Obtain the corresponding two-dimensional contour point of each point in the three-dimensional point cloud model in the multi-view image.

[0070] The correspondence is used to establish the relationship between three-dimensional geometry and two-dimensional perceptual information, which is the basis for subsequent uncertainty propagation.

[0071] In some implementations, three-dimensional points can be projected onto image planes of various viewpoints using known camera intrinsic and extrinsic parameters through back projection combined with visibility judgment, and nearest neighbor or sub-pixel matching points can be found in the pre-segmented two-dimensional contour mask of the weld seam as the corresponding two-dimensional contour points.

[0072] It should be noted that due to occlusion, reflection, or contour breakage, some 3D points may not have a valid correspondence from certain viewpoints. In this case, only visible and highly confident views should be used in subsequent calculations to avoid introducing false constraints.

[0073] S40112. Analyze the uncertainty of two-dimensional positioning based on the segmentation confidence or edge gradient magnitude of two-dimensional contour points.

[0074] Among them, the two-dimensional positioning uncertainty reflects the reliability of the position of the contour point on the image plane, which is affected by imaging noise, blurring or segmentation error.

[0075] In some implementations, if a deep learning segmentation model (such as U-Net) is used, the confidence level of the corresponding position in the output probability map can be directly extracted; if based on traditional edge detection (such as Canny, Sobel), the edge sharpness can be measured by the local gradient magnitude or the eigenvalue of the Hessian matrix, thereby quantifying the positioning accuracy.

[0076] It should be noted that high gradient magnitudes usually correspond to clear edges, but strong welding arc light may cause false edges. Therefore, uncertainty should be corrected in combination with contextual semantics (such as whether it is located in the prior area of ​​the weld).

[0077] S40113. Using camera calibration parameters and the Jacobian matrix of multi-view triangulation, the two-dimensional positioning uncertainty is propagated to three-dimensional space to obtain the position covariance matrix of the point.

[0078] The covariance matrix represents the range of spatial location uncertainty caused by two-dimensional observation noise at a three-dimensional point, and is an important measure of the quality of geometric reconstruction.

[0079] In some implementations, a nonlinear observation model of the triangulation problem is constructed, the model is linearized, the Jacobian matrix is ​​calculated, and the three-dimensional covariance is obtained by using the covariance propagation formula.

[0080] It should be noted that when the angle between the viewpoints is too small or they are coplanar, the Jacobian matrix approaches singularity, leading to covariance inflation. In this case, regularization should be introduced or low-information views should be removed to ensure numerical stability.

[0081] For example, an analytical covariance approximation can be made using weighted least squares triangulation (such as the Hartley-Sturm algorithm), or the covariance can be estimated empirically by repeatedly reprojecting the two-dimensional point positions through Monte Carlo perturbation and statistically analyzing the three-dimensional point distribution.

[0082] S40114. The measure of uncertainty in the predicted location is the trace or the largest eigenvalue of the covariance matrix.

[0083] Among them, the uncertainty metric is used to quantify the degree of positional deviation of the three-dimensional point as a whole or in the most unfavorable direction, which facilitates subsequent threshold judgment or weighted processing.

[0084] In some implementations, the trace of the covariance matrix (i.e., the sum of variances in all directions) reflects the overall uncertainty, while the largest eigenvalue corresponds to the square of the standard deviation in the direction of the principal axis of uncertainty, focusing more on the upper limit of the error in the most sensitive dimension.

[0085] It should be noted that traces are sensitive to isotropic noise and are suitable for global evaluation; while the maximum eigenvalue is better able to reveal potential structural deviations (such as stretching along the line of sight) and is suitable for reliability judgment of key geometric features (such as bevel edges).

[0086] S4012. Based on the predicted location uncertainty metric, dynamically calculate the neighborhood search radius of the point using a preset mapping function.

[0087] Among them, the neighborhood search radius is positively correlated with the uncertainty of the predicted location.

[0088] The neighborhood search radius is used to define the spatial range of local geometric analysis, and its size directly affects the robustness and resolution of differential geometric features such as curvature.

[0089] In some implementations, linear or nonlinear mapping functions can be used to convert the uncertainty measure (such as covariance trace) of each point into an adaptive radius, ensuring that a larger neighborhood is used in high uncertainty regions to enhance statistical stability.

[0090] It should be noted that if the radius is too small, noise will dominate the curvature estimation; if it is too large, nonlocal structures (such as opposite bevels or spatter) may be mixed in, leading to feature distortion. Therefore, the mapping function needs to be calibrated in combination with point cloud density and typical weld geometry.

[0091] S4013. Based on the neighborhood search radius, construct the local neighborhood of the target point at multiple preset scales, and calculate the ratio of the minimum eigenvalue to the sum of eigenvalues ​​of the covariance matrix at each scale through principal component analysis to obtain the local curvature response at that scale.

[0092] Among them, the local curvature response is used to characterize the degree of surface curvature at different spatial granularities and is the core indicator of multi-scale geometric analysis.

[0093] In some implementations, a multi-scale spherical neighborhood is generated by scaling the radius of the basic neighborhood with the target point as the center. PCA is performed on the point set in each neighborhood to obtain three eigenvalues ​​of the covariance matrix, and the curvature response is calculated. The closer the value is to 1, the flatter it is, and the closer it is to 0, the sharper or linear it is.

[0094] It should be noted that curvature at a single scale is easily affected by uneven sampling or local anomalies, while a multi-scale strategy can take into account both detail sensitivity and structural consistency, and is especially suitable for complex morphologies in welds that have both smooth molten pool areas and steep bevel edges.

[0095] S4014. Perform a weighted average of the local curvature responses at multiple scales to obtain the preliminary curvature value of the target point.

[0096] The preliminary curvature value incorporates information from different observation scales, aiming to improve the ability to discriminate true geometric shapes.

[0097] In some implementations, fixed weights (such as uniform averaging) or adaptive weights (such as those based on the number of neighborhood points at each scale, the number of eigenvalue conditions, or the degree of matching with the base radius) can be used to weight and fuse the multi-scale curvature response.

[0098] It should be noted that simple averaging may weaken the discriminative signal at key scales; ideal weights should emphasize scales with high signal-to-noise ratios and strong structural consistency, and suppress unreliable responses caused by neighborhood contamination or sparsity.

[0099] S4015. Based on the angle between the normal vectors of the target point and its neighboring points, apply a smoothing constraint to the initial curvature value to generate the corrected spatial curvature value.

[0100] Among them, the smoothing constraint utilizes the prior knowledge of normal vector continuity to suppress curvature jumps caused by isolated noise points or segmentation errors.

[0101] In some implementations, the normal vector of each point is estimated by PCA or local plane fitting; for the target point and its k nearest neighbors, similarity weights are constructed based on the angle between the normals, and the preliminary curvature values ​​in the neighborhood are weighted and averaged to obtain the corrected curvature.

[0102] It should be noted that there may be real abrupt changes in the normal direction in the weld area (such as unfused boundaries), and excessive smoothing will mask the defect characteristics; therefore, a normal angle threshold should be set (e.g., if >45°, it will not participate in smoothing) to preserve structural discontinuities.

[0103] For example, for a suspected biting point, if the normal angle of most points in its neighborhood is less than 30°, smoothing is allowed to suppress random fluctuations; however, if there is a clear directional boundary, its high curvature response is retained as a defect clue.

[0104] S4016. Traverse all points and output the actual spatial curvature distribution corresponding to the predicted weld contour.

[0105] The actual spatial curvature distribution constitutes a key input for assessing the geometric integrity of welds and can be used to identify typical defects such as abnormal weld height, collapse, and undercut.

[0106] In some implementations, the corrected curvature values ​​are organized by point index into an attribute array aligned with the original point cloud, and can be further projected or resampled along the weld centerline to generate a one-dimensional curvature profile to support quantitative analysis.

[0107] It should be noted that curvature itself does not have an absolute physical unit. It is necessary to set a threshold range in conjunction with the process specifications (such as curvature ∈ [0.2, 0.4] in the normal excess height area and >0.6 in the bite area) and to make a judgment in conjunction with other features (such as normal consistency and volume deviation) to avoid false alarms.

[0108] For example, in the inspection of butt welds, if multiple consecutive points have a curvature exceeding 0.65 and are accompanied by normal divergence, they are judged as undercut defects; while at the toe of a fillet weld, moderately high curvature is a normal geometric feature and needs to be distinguished by contextual semantics.

[0109] Based on the above technical solutions, traditional methods for calculating the curvature of weld point clouds typically employ fixed neighborhood radii and single-scale analysis, which struggles to address the uneven point cloud quality caused by strong interference in welding scenarios. In high-noise or low-confidence regions, fixed small neighborhoods are susceptible to outliers, leading to distorted curvature estimations, while fixed large neighborhoods may be mixed with non-local structural information, causing feature ambiguity. To address this issue, this solution proposes an uncertainty-guided multi-scale adaptive curvature calculation method. This method quantifies the positional uncertainty of each 3D point and dynamically adjusts its neighborhood search radius accordingly; the higher the uncertainty, the larger the neighborhood, enhancing statistical robustness. Local curvature responses are calculated and weighted at multiple scales, balancing details and structure. A smoothing constraint based on normal vector consistency is introduced to effectively suppress curvature jumps caused by noise while preserving true geometric discontinuities (such as defect edges). This technical solution, starting from the inherent reliability of point cloud data, achieves accurate and robust extraction of geometric features in complex welding environments, providing a reliable basis for subsequent high-precision physical consistency verification and defect assessment.

[0110] S402. Based on preset welding process parameters, the theoretical spatial curvature distribution of the weld is calculated using a heat conduction physical model, as follows: S4021. Based on the welding current, voltage, welding speed and material thermophysical parameters, calculate the steady-state temperature field of the weld pool using the Rosenthal moving heat source model.

[0111] Among them, the steady-state temperature field describes the spatial temperature distribution of the material around the molten pool when it reaches thermal equilibrium under ideal continuous welding conditions, and is the basis for deriving the theoretical weld geometry.

[0112] In some implementations, Rosenthal's classic analytical solution (such as the exponential decay form under the assumption of a two-dimensional or three-dimensional Gaussian heat source) is used. The input parameters (current I, voltage U, velocity v, thermal conductivity k, specific heat capacity c, density ρ, etc.) are substituted into the formula to calculate the temperature T(x,y,z) at any point (x,y,z) in space. Usually, the welding torch is used as the origin of the moving coordinate system.

[0113] It should be noted that the Rosenthal model is based on many simplifying assumptions (such as an infinitely large homogeneous medium, neglecting the latent heat of phase transformation, steady-state conditions, etc.), which may lead to significant deviations in high-energy-density welding (such as laser welding) or thin-plate welding; therefore, its results are more suitable for qualitative trend analysis or as initial guesses.

[0114] S4022. Based on the temperature field, determine the isotherms corresponding to the molten pool boundary, and fit the isotherms to the theoretical profile curve of the weld cross-section.

[0115] The isotherm is usually taken as the melting point temperature (or solidus temperature) of the material. Its projection on the cross section perpendicular to the welding direction is regarded as the theoretical molten pool boundary, representing the weld geometry under ideal undisturbed conditions.

[0116] In some implementations, samples satisfying ( ) are extracted from multiple cross-sections (e.g., every 1 mm along the weld length). The point set of ) The melting point temperature is used to fit a smooth closed curve using spline interpolation (such as B-spline, NURBS) or parametric functions (such as ellipse, hyperbolic cosine) to obtain the theoretical profile of the cross section.

[0117] It should be noted that the actual molten pool is affected by surface tension, gravity, Marangoni convection, etc., and its shape often deviates from the symmetrical shape predicted by Rosenthal (such as collapse or finger-shaped melt depth). Therefore, the theoretical profile should be regarded as an ideal reference driven by process parameters rather than the actual morphology.

[0118] S4023. Perform three-dimensional spatial parameterization on the theoretical profile curve to obtain the theoretical spatial trajectory of the weld centerline.

[0119] The weld centerline is defined as the line connecting the geometric centers (such as the centroid or the intersection of the axis of symmetry) of the theoretical profiles of each cross section along the weld length direction, representing the spatial orientation of the ideal weld bead.

[0120] In some implementations, the center point coordinates of the fitted profile of each cross section are extracted sequentially, connected in the order of the welding path, and global smooth interpolation is performed using cubic splines or cyclotron curves to generate a continuously differentiable three-dimensional parametric curve.

[0121] It should be noted that if the welding path itself contains corners, bevels, or curved transitions, the Rosenthal model alone cannot automatically capture the geometric constraints of the path; in this case, the preset welding torch trajectory must be used as a priori to guide the construction of the centerline.

[0122] S4024. By performing differential geometric operations on the theoretical spatial trajectory, its local curvature is calculated to obtain the theoretical spatial curvature distribution.

[0123] Among them, the theoretical spatial curvature distribution reflects the degree of curvature that the ideal weld centerline should have under given process parameters, and can be used to compare with the actual measured curvature to identify geometric deviations.

[0124] In some implementations, the first and second derivatives of the parameterized trajectory are calculated, and the local curvature is calculated point by point using the curvature formula to form a one-dimensional curvature sequence along the weld length.

[0125] It should be noted that the theoretical curvature is determined only by the welding path and heat input, and does not include manufacturing errors (such as clamping deviations and robot vibration); therefore, its main value lies in providing an "expected" benchmark, rather than an absolute defect criterion.

[0126] For example, when welding a straight butt joint, the theoretical centerline is a straight line and the curvature is zero everywhere; if the actual curvature deviates significantly, it may indicate misalignment, collapse or tracking failure; while in pipe bending welding, the theoretical curvature is non-zero and needs to match the geometric curvature of the design path.

[0127] S403, the physical consistency loss function is a measure of the difference between the actual spatial curvature distribution and the theoretical spatial curvature distribution.

[0128] Among them, the physical consistency loss function is used to quantify the degree of inconsistency between the prediction results of the deep learning model and the prior knowledge of welding thermodynamics in terms of geometric shape. It is a key supervision signal to guide the network to learn physically reasonable outputs.

[0129] In some implementations, the difference between two curvature distribution sequences (after alignment along the weld centerline) can be calculated using metrics such as L2 norm (mean squared error), L1 norm (mean absolute error), or weighted cosine similarity. If the curvature exists as a point attribute, it can be calculated point by point on the corresponding point set and summed or averaged.

[0130] It should be noted that directly comparing the original curvature values ​​may be affected by scale or local noise. Therefore, it is often necessary to normalize the curvature (such as zero mean and unit variance) or introduce a multi-scale matching strategy (such as combining low-frequency trends and high-frequency details separately) to enhance the robustness and physical sensitivity of the loss function.

[0131] S404. By minimizing the physical consistency loss function, the parameters of the deep learning segmentation network are optimized so that the predicted weld contour conforms to the thermodynamic and physical laws of welding in three-dimensional space.

[0132] The optimization process embeds physical priors into a data-driven learning framework, enabling the network to not only fit labeled data but also generalize to unseen conditions while maintaining geometric rationality.

[0133] In some implementations, the physical consistency loss is combined with the traditional segmentation loss (such as Dice Loss, cross-entropy) in a weighted manner to form the total loss function, and the network weights are updated jointly by backpropagation and gradient descent (such as Adam optimizer).

[0134] It should be noted that this method is a variant of Physics-Informed Neural Networks (PINNs), but it is necessary to ensure that the entire process (from 2D segmentation → 3D reconstruction → curvature calculation → loss feedback) is end-to-end differentiable; otherwise, a two-stage training or surrogate gradient strategy is required.

[0135] Based on the above technical solutions, traditional deep learning weld segmentation methods rely solely on data-driven approaches and lack constraints on the welding physics process. This leads to situations where, under complex interference, the predicted contours may appear reasonable on the image, but the reconstructed 3D morphology may violate thermodynamic laws (such as exhibiting non-physical, severe distortions or discontinuities), thus affecting the accuracy of the assessment. To address this issue, this solution innovatively introduces a physical consistency loss function: by combining a heat conduction model (such as the Rosenthal model) with process parameters, the theoretical spatial curvature distribution that the weld should possess under ideal conditions is calculated; simultaneously, the corresponding actual spatial curvature distribution is calculated from the actual reconstructed 3D model. By minimizing the difference between these two methods, the segmentation network is optimized, embedding welding physics priors into the training process. This approach enables the network to learn not only pixel-level visual patterns but also geometric generation mechanisms that conform to physical laws, thereby ensuring the rationality and realism of the predicted contours in 3D space and significantly improving the reliability of subsequent 3D reconstruction and quality assessment.

[0136] In one possible implementation of the embodiments of this application, such as Figure 3 As shown, the above S204 can be implemented through the following S501-S504, which are explained in detail below: S501. Based on the contour of the two-dimensional weld, generate a narrow band of reliable matching region around the center line of the weld on the image from each viewpoint.

[0137] The weld centerline includes multiple centerline points, each of which is associated with a corresponding spatial positional uncertainty.

[0138] Specifically, the following steps are included: S5011. Obtain the weld centerline predicted by the artificial intelligence model. The weld centerline contains multiple centerline points, and each centerline point is associated with a corresponding spatial position uncertainty.

[0139] Among them, the weld centerline is a skeleton curve in three-dimensional space that represents the direction of the weld. Each point on it not only contains coordinate information, but also carries positional uncertainty quantified by model output (such as covariance matrix, confidence level or Monte Carlo sampling variance).

[0140] In some implementations, the centerline can be generated by point cloud-based graph neural networks, Transformer, or U-Net3D segmentation post-processing (such as skeleton extraction or centerline fitting), while the uncertainty is estimated through Bayesian deep learning, ensemble prediction, or error propagation mechanisms.

[0141] It should be noted that positional uncertainty reflects the model's confidence in local geometric judgments. Areas with high uncertainty may be caused by occlusion, low texture, splashing interference, or sparse training data, and should be handled carefully in subsequent multi-view fusion.

[0142] S5012. Project the weld centerline onto the image plane of each viewpoint to obtain the centerline pixel trajectory under each viewpoint.

[0143] The projection process utilizes calibrated camera intrinsic and extrinsic parameters (i.e., rigid body transformation from the world coordinate system to the image coordinate system) to map the three-dimensional centerline points into two-dimensional pixel coordinates, forming the corresponding centerline trajectories under each viewpoint.

[0144] It should be noted that, due to potential depth ambiguity or projection occlusion at the centerline point, some points may fall outside the image or be occluded by the workpiece itself from a specific viewpoint. In such cases, they should be marked as invalid points and skipped in subsequent steps.

[0145] S5013. Calculate the local image gradient magnitude of each centerline pixel in each viewpoint, and determine the local image confidence level of that point based on the gradient magnitude.

[0146] Local image confidence is used to measure whether an image has clear and stable edge or texture features at a given pixel location, thus reflecting the quality of the observation at that point.

[0147] In some implementations, the input image is first convolved with Sobel, Scharr or Canny gradient operators to obtain the gradient components in the horizontal and vertical directions. Then the gradient magnitude is calculated and normalized as the confidence score (or mapped to the [0,1] interval via sigmoid).

[0148] It should be noted that high gradient amplitudes usually correspond to strong structural features such as weld edges and fusion lines, but may also come from noise, spatter, or background interference; therefore, further filtering can be achieved by combining neighborhood consistency (such as gradient direction consistency) or frequency domain analysis.

[0149] S5014. Based on the spatial location uncertainty and local image reliability, calculate the narrowband half-width at the centerline pixel. The narrowband half-width increases with increasing uncertainty and decreases with decreasing image reliability.

[0150] Among them, the narrowband half-width defines the width of the local search or matching window around the centerline trajectory on the image plane, which is used to balance the reliability of geometric priors and image evidence.

[0151] It should be noted that this mapping function dynamically adjusts the narrowband half-width at each centerline pixel by comprehensively considering the positional uncertainty in 3D space and the local confidence in the 2D image: when the model is more uncertain about the spatial position of a point (e.g., due to occlusion or low-confidence prediction), the narrowband half-width increases accordingly to provide a larger tolerance range; however, if the point lacks reliable visual features in the image (e.g., weak gradient, blurred texture), even with high positional uncertainty, the search area should not be blindly expanded, but rather the narrowband half-width should be reduced to avoid introducing noise or mismatches. This two-way adjustment mechanism ensures that the narrowband is neither too conservative and misses true edges, nor too expanded and includes unreliable regions, thus achieving an adaptive balance between geometric priors and image evidence.

[0152] S5015. Using the center line pixel as the center, generate a line segment with a length of twice the width of the narrow band along the direction perpendicular to the tangent of the center line at that point.

[0153] Among them, the line segments constitute the basic unit of the narrow band area, and their direction is perpendicular to the local direction of the center line, ensuring the coverage of the potential true edge position in the direction of the weld cross section.

[0154] It should be noted that at locations with high curvature (such as corners), tangent estimation is susceptible to the effects of discrete sampling. Savitzky-Golay filtering or spline interpolation can be used to improve directional stability. In addition, line segments should be limited to the effective area of ​​the image.

[0155] S5016. Remove line segments whose local image credibility is lower than the preset credibility threshold, and take the union of the remaining line segments as the narrowband credibility matching region under this view.

[0156] Among them, the narrow-band reliable matching region is used for subsequent multi-view feature matching, 3D reconstruction or contour optimization, and only local regions with sufficient image evidence and reasonable geometric priors are retained.

[0157] In some implementations, a confidence threshold (such as 0.3) is set, and all line segments are filtered according to the confidence of their center point; the retained line segments can be rasterized or masked to form binary image regions, or they can be represented as a set of line segments for subsequent geometric processing.

[0158] It should be noted that this rejection mechanism effectively suppresses invalid searches in areas with smoke, reflection, or obstruction, improving matching efficiency and accuracy; however, the threshold setting needs to balance recall and precision, and being too strict may lead to the loss of real weld edges.

[0159] S502. When performing multi-view stereo matching, for pixels located within the reliable matching area, a parallax search range with a preset resolution and an enhanced matching cost are calculated.

[0160] The reliable matching region is generated by the preceding steps (such as S5016) and contains only local narrow bands in the image where both geometric and visual evidence are reliable, thereby concentrating high-precision matching resources on the critical areas of the weld.

[0161] In some implementations, finer disparity search step sizes are set for these pixels (such as subpixel level or 0.25 pixel resolution), and enhanced matching cost functions are used, such as combining Census transform, normalized cross-correlation (NCC), gradient consistency, or learned features (such as MC-CNN features) to improve robustness; at the same time, the upper and lower bounds of the disparity search can be limited, and invalid disparity intervals can be dynamically cropped based on narrow-band half-width.

[0162] It should be noted that this strategy significantly reduces computational complexity (because it only processes local areas) while avoiding unreliable matching in low-texture or highly interfering areas (such as arc light and spatter), effectively improving the integrity and accuracy of the parallax map in the weld area.

[0163] S503. In the matching cost aggregation and optimization stage, an anisotropic regularization term is introduced. The weight of the anisotropic regularization term is determined by the contour of the two-dimensional weld. Specifically, it has a first smoothing weight in the direction parallel to the weld direction and a second smoothing weight in the direction perpendicular to the weld direction, with the second smoothing weight being greater than the first smoothing weight.

[0164] Among them, the anisotropic regularization term is used to guide the parallax optimization process to respect the physical geometry of the weld: moderate parallax variation is allowed along the weld direction (such as due to surface undulations), while stronger parallax consistency is forced in the vertical direction (i.e. the direction of the weld cross-section) to maintain the sharpness and continuity of the weld edges.

[0165] It should be noted that this direction-sensitive smoothing strategy effectively suppresses parallax noise in the vertical direction (such as jumps caused by reflections) while preserving the true height variation along the weld direction (such as excess height fluctuations), avoiding edge blurring or contour distortion caused by traditional isotropic smoothing.

[0166] S504. Based on the optimized disparity map and combined with camera calibration parameters, a three-dimensional point cloud model of the weld is obtained through triangulation calculation.

[0167] The triangulation process utilizes multi-view geometry to convert the disparity value of each pixel into three-dimensional spatial coordinates, thereby reconstructing a high-precision point cloud of the weld surface.

[0168] In some implementations, the standard binocular triangulation formula is used: given the pixel coordinates (u,v) and disparity d of the left image, the depth Z = Bf / d is calculated by combining the baseline length B and the focal length f, and then the world coordinates are obtained by backprojection through the camera intrinsic parameters; if it is a multi-view system, multiple disparity maps can be fused and the accuracy can be improved by least squares optimization.

[0169] It should be noted that since the disparity map has been optimized within a reliable narrow band region, the resulting point cloud mainly covers the weld body and adjacent areas. The data is sparse but highly accurate, making it suitable for subsequent use in dimensional measurement, defect detection, or comparison with CAD models. At the same time, it should be noted that occlusion areas may cause holes in the point cloud, which requires interpolation or prior completion.

[0170] Based on the above technical solutions, traditional methods often suffer from large matching search spaces and high mismatch rates when reconstructing 3D models of weld seams using multi-view stereo vision. This is due to the strong reflectivity, smoke, and low texture characteristics of the welding area, especially in the direction perpendicular to the weld seam, where depth jumps are prone to occur, severely affecting reconstruction accuracy. To address this issue, this solution proposes a contour-guided anisotropic stereo matching strategy: a narrow-band reliable matching region is generated based on the 2D segmentation results, and the spatial uncertainty of the centerline point is combined to effectively constrain the disparity search range, reduce computational complexity, and suppress interference from non-weld seam areas. In the cost optimization stage, an anisotropic regularization term driven by the weld seam direction is introduced, allowing moderate smoothing along the weld seam direction (first weight), while applying stronger smoothing constraints in the vertical direction (second weight is larger) to match the prior geometric continuity of the weld seam, suppressing depth noise while preserving the true contour boundary. This solution deeply integrates semantic segmentation results with stereo matching, guiding 3D reconstruction from the perspective of physical structural characteristics, significantly improving the integrity, smoothness, and geometric fidelity of the weld seam point cloud, laying a solid foundation for subsequent high-precision curvature analysis and quality assessment.

[0171] In one possible implementation of this application embodiment, the above-mentioned S205 can be specifically implemented by the following S601-S504, which will be described in detail below: S601. Based on the precise two-dimensional contour of the weld, fit the weld centerline in the three-dimensional point cloud model, and calculate the tangential, normal, and sub-normal directions of each point on the centerline.

[0172] Among them, the weld centerline is a three-dimensional skeleton curve that characterizes the weld orientation. It is extracted by geometric fitting method using high-precision point cloud combined with two-dimensional contour prior (such as narrow band region or segmentation results) to establish a local coordinate system to support subsequent anisotropic analysis.

[0173] In some implementations, the two-dimensional contour is back-projected onto the three-dimensional point cloud space, and neighboring points are selected to form an initial sequence of center points. Then, B-spline fitting, moving least squares (MLS) curve fitting, or principal component analysis sliding window method are used to generate a smooth center line. The tangent is obtained by the first derivative of the center line parameterization, and the normal is estimated by combining the local point cloud distribution or surface normal. The subnormal is determined by the cross product of the tangent and the normal, thus forming the Frenet-Serret frame.

[0174] It should be noted that accurate local reference frames are crucial for subsequent feature extraction. If the normal deviates from the direction of the actual weld cross section, it will lead to distortion in the measurement of width or height. Therefore, at locations with drastic changes in curvature (such as corners or arc initiation points), regularization or direction propagation strategies need to be introduced to improve stability.

[0175] S602. Using the weld centerline as a reference, set multiple elliptical cylindrical analysis neighborhoods of preset scales along the normal and tangential directions to form a multi-scale, anisotropic local point cloud subset.

[0176] Among them, the elliptical cylindrical analysis neighborhood is a spatial sampling region locally defined along the centerline. Its cross-section is an ellipse, with the major axis along the normal (covering the weld width direction), the minor axis along the sub-normal (thickness direction), and the axial direction along the tangential (weld length direction), thus adapting to the typical flat geometry of the weld.

[0177] In some implementations, for each sampling point on the centerline, multiple elliptical cylinders of different scales (e.g., small: ±2mm, medium: ±5mm, large: ±10mm) are constructed, with their normal radius being greater than their tangential radius (e.g., a ratio of 2:1) to emphasize the importance of the transverse structure of the weld. The point cloud falling within these elliptical cylinders constitutes a local subset for multi-scale feature extraction.

[0178] It should be noted that using an elliptical rather than a circular cross section can better fit the actual shape of the weld and avoid introducing redundant or noise points in non-critical directions (such as along the plate thickness direction); at the same time, multi-scale design takes into account both local details (such as undercut) and global trends (such as overall reinforcement height changes).

[0179] S603. In the local point cloud subset at each scale, calculate the width distribution characteristics along the normal direction, the smoothness characteristics along the tangential direction, and the curvature and co-height statistical characteristics along the height direction to obtain multiple sets of scale-related geometric descriptors.

[0180] Among them, geometric descriptors are key indicators for quantifying weld morphology from different physical dimensions: normal width reflects the fusion state, tangential smoothness characterizes continuity, and height direction features are related to weld formation quality (such as reinforcement height and collapse).

[0181] In some implementations, the width feature can be defined as the projected span of the local point cloud in the normal direction or the density-weighted mean; the tangential smoothness can be measured by the centerline curvature, the standard deviation of the height difference between adjacent neighborhood centers, or the first-order difference norm; the curvature in the height direction (usually the vertical component in the world coordinate or local normal-subnormal plane) can be estimated by fitting a local surface (such as a quadratic surface), and the residual height is calculated relative to the parent material reference plane as the mean, variance, or maximum value.

[0182] It should be noted that all features should be calculated in a unified local coordinate system to eliminate the influence of attitude; in addition, to improve robustness, RANSAC can be used to remove outliers or kernel density estimation can be used instead of simple statistics.

[0183] S604. Perform cross-scale weighted fusion of multiple sets of geometric descriptors to generate multi-scale geometric feature vectors.

[0184] Among them, the multi-scale geometric feature vector integrates weld morphology information under different receptive fields, which not only retains detail sensitivity but also has global context awareness, making it suitable for subsequent classification, regression or anomaly detection tasks.

[0185] Some implementations employ a scale-based weighting strategy: for example, smaller-scale features are given higher weights for defect identification, while larger-scale features are used for overall quality rating; the weights can be fixed (e.g., Gaussian decay), learned (e.g., attention mechanisms), or dynamically adjusted based on the signal-to-noise ratio (e.g., higher point cloud density results in higher weights). Fusion methods include stitching, weighted averaging, or hierarchical aggregation (e.g., pyramid pooling).

[0186] It should be noted that cross-scale fusion needs to address the issue of inconsistent feature dimensions, which usually requires normalization of each descriptor (such as Z-score or Min-Max). In addition, information redundancy should be avoided, and the dimensionality can be further compressed through principal component analysis or feature selection.

[0187] Based on the above technical solutions, traditional three-dimensional weld evaluation methods typically employ isotropic or single-scale local geometric analysis, which struggles to effectively characterize the complex morphological features of welds that exhibit significant differences across different directions (such as width, reinforcement height, and orientation). This is especially problematic when defects or process variations exist, easily overlooking subtle anomalies at critical scales. To address this issue, this solution proposes a multi-scale anisotropic geometric feature extraction method guided by the weld centerline. This method accurately fits the three-dimensional centerline and establishes a local directional reference system, defining physically meaningful normal, tangential, and sub-normal directions. Then, an elliptical cylindrical neighborhood that extends along these directions is constructed, ensuring the analysis region closely matches the actual geometric orientation and morphology of the weld. Based on this, direction-sensitive geometric descriptors reflecting width variations, surface smoothness, and reinforcement height curvature are extracted, and multi-level information is integrated through a cross-scale fusion mechanism. This technical solution overcomes the limitations of traditional spherical or cubic neighborhoods, achieving refined, directional, and multi-scale characterization of weld local geometric properties, significantly improving the accuracy and robustness of subsequent quality assessment and defect identification.

[0188] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

Claims

1. A machine vision-based weld seam tracking and evaluation method, characterized in that, include: Acquire multispectral image sequences of the weld area during the welding process; The multispectral image sequence is subjected to spatiotemporal adaptive weighted fusion to generate a fused image with a high signal-to-noise ratio; The fused image is input into a deep learning segmentation network trained by physical laws to segment the outline of the two-dimensional weld. Based on the outline of the two-dimensional weld and the multi-view geometric information, the three-dimensional point cloud model of the weld is reconstructed in real time. Based on the three-dimensional point cloud model and the precise contour of the weld, multi-scale geometric features of the weld are extracted; the multi-scale geometric features are used to characterize the geometric parameters related to the structural integrity of the weld joint. The multi-scale geometric features are input into a pre-trained graph neural network, which outputs a defect index of the weld, which is used to assess the degree of defect in the weld.

2. The machine vision-based weld seam tracking and evaluation method according to claim 1, characterized in that, The method for acquiring the high signal-to-noise ratio fused image includes: For each frame of the multispectral image sequence, a corresponding spatial quality map is generated through a branch of a convolutional neural network. The spatial quality map is used to characterize the local reliability of each pixel region in the image affected by welding interference. For each band of image sequence, the global temporal stability score, which characterizes the current imaging stability of the band, is obtained by calculating the structural similarity between the current frame and historical frames. The spatial quality map and the temporal stability score are input into a nonlinear gating unit to generate a spatiotemporally adaptive weighted map. Using the spatiotemporal adaptive weight map, the current frame images of all bands are weighted and summed to generate the fused image with high signal-to-noise ratio.

3. The machine vision-based weld seam tracking and evaluation method according to claim 1, characterized in that, The training process of the deep learning segmentation network includes constructing a physical consistency loss function, which characterizes the following differences: Based on the reconstructed three-dimensional model of the weld, its actual spatial curvature distribution is calculated; Based on preset welding process parameters, the theoretical spatial curvature distribution of the weld is calculated using a heat conduction physical model. The physical consistency loss function is a measure of the difference between the actual spatial curvature distribution and the theoretical spatial curvature distribution; By minimizing the physical consistency loss function, the parameters of the deep learning segmentation network are optimized so that the predicted weld contour conforms to the thermodynamic physical laws of welding in three-dimensional space.

4. The machine vision-based weld seam tracking and evaluation method according to claim 3, characterized in that, The method for obtaining the theoretical spatial curvature distribution includes: Based on the welding current, voltage, welding speed and material thermophysical parameters, the steady-state temperature field of the weld pool is calculated using the Rosenthal moving heat source model. Based on the temperature field, the isotherms corresponding to the molten pool boundary are determined, and the isotherms are fitted to the theoretical profile curve of the weld cross-section. The theoretical contour curve is parameterized in three-dimensional space to obtain the theoretical spatial trajectory of the weld centerline. By performing differential geometric operations on the theoretical spatial trajectory, its local curvature is calculated, and the curvature distribution of the theoretical space is obtained.

5. The machine vision-based weld seam tracking and evaluation method according to claim 3, characterized in that, The actual spatial curvature distribution of the three-dimensional weld profile includes: For each point in the 3D point cloud model, obtain its corresponding predicted position uncertainty metric; based on the predicted position uncertainty metric, dynamically calculate the neighborhood search radius of the point through a preset mapping function; wherein, the neighborhood search radius is positively correlated with the predicted position uncertainty; Based on the neighborhood search radius, local neighborhoods of the target point are constructed at multiple preset scales, and the ratio of the minimum eigenvalue to the sum of eigenvalues ​​of the covariance matrix is ​​calculated by principal component analysis at each scale to obtain the local curvature response at that scale. The initial curvature value of the target point is obtained by weighted averaging of the local curvature response at multiple scales. Based on the angle between the normal vectors of the target point and its neighboring points, a smoothing constraint is applied to the initial curvature value to generate a corrected spatial curvature value. Iterate through all points and output the actual spatial curvature distribution corresponding to the predicted weld contour.

6. The machine vision-based weld seam tracking and evaluation method according to claim 5, characterized in that, The method for obtaining the measurement of predicted location uncertainty includes: Obtain the corresponding two-dimensional contour point of each point in the three-dimensional point cloud model in the multi-view image; Based on the segmentation confidence or edge gradient magnitude of the two-dimensional contour points, the uncertainty of two-dimensional positioning is analyzed. Using camera calibration parameters and the Jacobian matrix of multi-view triangulation, the two-dimensional positioning uncertainty is propagated to three-dimensional space to obtain the position covariance matrix of the point. The measurement of the predicted location uncertainty is the trace or the largest eigenvalue of the covariance matrix.

7. The machine vision-based weld seam tracking and evaluation method according to claim 1, characterized in that, The three-dimensional point cloud model of the reconstructed weld includes: Based on the contour of the two-dimensional weld, a narrow band of reliable matching region is generated on the image from each viewpoint, around the center line of the weld; the center line of the weld contains multiple center line points, each center line point being associated with a corresponding spatial position uncertainty; When performing multi-view stereo matching, for pixels located within the reliable matching area, a parallax search range with a preset resolution and an enhanced matching cost are calculated. In the matching cost aggregation and optimization stage, an anisotropic regularization term is introduced, the weight of which is determined by the contour of the two-dimensional weld; wherein, it has a first smoothing weight in the direction parallel to the weld direction and a second smoothing weight in the direction perpendicular to the weld direction, and the second smoothing weight is greater than the first smoothing weight. Based on the optimized disparity map and combined with camera calibration parameters, a three-dimensional point cloud model of the weld is obtained through triangulation calculation.

8. The machine vision-based weld seam tracking and evaluation method according to claim 7, characterized in that, The method for obtaining the narrowband reliable matching region includes: Obtain the weld centerline predicted by an artificial intelligence model. The weld centerline contains multiple centerline points, each of which is associated with a corresponding spatial position uncertainty. The centerline of the weld is projected onto the image plane of each viewpoint to obtain the pixel trajectory of the centerline under each viewpoint. Calculate the local image gradient magnitude of each centerline pixel in each viewpoint, and determine the local image confidence level of that point based on the gradient magnitude; Based on the spatial location uncertainty and the local image reliability, calculate the narrow band half-width at the center line pixel; Centered on the centerline pixel, a line segment with a length twice the width of the narrow band is generated along the direction perpendicular to the tangent of the centerline at that point. Line segments with local image credibility below a preset credibility threshold are removed, and the union of the remaining line segments is taken as the narrowband credibility matching region under that viewpoint.

9. The machine vision-based weld seam tracking and evaluation method according to claim 1, characterized in that, The multi-scale geometric feature extraction method for the weld seam includes: Based on the precise two-dimensional contour of the weld, the center line of the weld is fitted in the three-dimensional point cloud model, and the tangential, normal, and subnormal directions of each point on the center line are calculated. Using the weld centerline as a reference, multiple elliptical cylindrical analysis neighborhoods of preset scales are set along the normal and tangential directions respectively, forming a multi-scale, anisotropic local point cloud subset; In the local point cloud subset at each scale, the width distribution characteristics along the normal direction, the smoothness characteristics along the tangential direction, and the curvature and co-height statistical characteristics along the height direction are calculated respectively to obtain multiple sets of scale-related geometric descriptors. The multiple sets of geometric descriptors are weighted and fused across scales to generate multi-scale geometric feature vectors.

10. A machine vision-based weld seam tracking and evaluation system, operating based on the machine vision-based weld seam tracking and evaluation method according to any one of claims 1-9, characterized in that, It includes an acquisition module, an analysis module, and a defect assessment module; The acquisition module is used to acquire multispectral image sequences of the weld area during the welding process; The analysis module is used to perform spatiotemporal adaptive weighted fusion on the multispectral image sequence to generate a fused image with a high signal-to-noise ratio; The fused image is input into a deep learning segmentation network trained by physical laws to segment the outline of the two-dimensional weld. Based on the outline of the two-dimensional weld and the multi-view geometric information, the three-dimensional point cloud model of the weld is reconstructed in real time. Based on the three-dimensional point cloud model and the precise contour of the weld, the multi-scale geometric features of the weld are extracted; The defect assessment module is used to input the multi-scale geometric features into a pre-trained graph neural network and output the defect index of the weld.

Images