Weld defect intelligent identification method and system based on multi-modal fusion

Abstract

The invention relates to a weld defect intelligent identification method and system based on multi-modal fusion. The method comprises the following steps: acquiring multi-modal data of the same welding seam area to obtain a multi-modal image set, enhancing a feature map of each modal image in the multi-modal image set through a preset feature extraction network to obtain an enhanced feature, and extracting a multi-scale depth feature of the enhanced feature; on this basis, cross-modal feature alignment is realized by using deformable convolution, and a physical constraint mechanism is introduced to generate a fusion feature map; candidate defect areas are generated based on the fusion features, feature vectors and position information of the candidate defect areas are extracted, and a defect relation graph is constructed; relational reasoning optimization feature representation is carried out through a graph neural network, collaborative judgment of defect types, positions and incidence relations is achieved, the identification accuracy of a symbiotic defect group in a complex industrial scene is remarkably improved, physical relevance between defects is reliably quantified, and the anti-interference capacity is enhanced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of weld defect recognition, and particularly relates to a weld defect intelligent recognition method and system based on multi-modal fusion. BACKGROUND

[0002] In the field of welding manufacturing, the accurate detection of weld quality is directly related to the structural safety and service life of major equipment. Traditional detection methods usually rely on a single non-destructive testing modality, such as only using X-ray imaging or ultrasonic scanning for defect recognition. Such single-modality detection system has essential limitations: X-ray imaging is sensitive to volume defects such as pores but difficult to capture microscopic cracks, and ultrasonic is significantly responsive to layered defects but easily disturbed by material grain noise. More importantly, when facing defect groups generated by the welding heat process, such as the complex defect network of heat cracks accompanied by pores, slag and un-melted, existing technologies can only provide isolated defect coordinates and category judgments, and completely fail to reveal the physical correlation between defects.

[0003] The lack of defect correlation information described above leads to the inability to distinguish between randomly dispersed defects and defect clusters with causal relationships, and makes it difficult to predict potential failure paths, causing the detection report to be disconnected from the structural safety assessment. SUMMARY

[0004] Therefore, it is necessary to provide a weld defect intelligent recognition method and system based on multi-modal fusion in view of the above technical problems.

[0005] In a first aspect, the application provides a weld defect intelligent recognition method based on multi-modal fusion, comprising:

[0006] S1, collecting multi-modal data of the same weld area to obtain a multi-modal image set; enhancing the feature maps of each modality image in the multi-modal image set through a pre-set feature extraction network to obtain enhanced features, and extracting multi-scale deep features of the enhanced features;

[0007] S2, performing cross-modal feature alignment and fusion on the multi-scale deep features through deformable convolution and physical constraint mechanism to generate a fusion feature map;

[0008] S3, generating candidate defect regions based on the fusion feature map, and extracting feature vectors and position information of each candidate defect region; constructing a defect relation graph based on the feature vectors and position information of the candidate defect regions;

[0009] S4, performing feature optimization and relationship reasoning on the defect relation graph through a graph neural network to generate a relationship-enhanced defect feature representation; performing defect classification and positioning based on the defect feature representation to output the recognition result of the weld defect.

[0010] In a second aspect, the application also provides a welding seam defect intelligent identification system based on multi-modal fusion, which is used to implement the method in the first aspect, and the system comprises:

[0011] a multi-modal data acquisition and feature enhancement module, which is used to acquire multi-modal data of the same welding seam area to obtain a multi-modal image set, and to enhance feature maps of each modal image in the multi-modal image set through a preset feature extraction network to obtain enhanced features and extract multi-scale deep features of the enhanced features;

[0012] a cross-modal feature fusion module, which is used to perform cross-modal feature alignment and fusion on the multi-scale deep features through deformable convolution and a physical constraint mechanism to generate a fusion feature map;

[0013] a candidate defect analysis and relationship construction module, which is used to generate candidate defect regions based on the fusion feature map and extract feature vectors and position information of each candidate defect region, and to construct a defect relationship graph based on the feature vectors and position information of the candidate defect regions;

[0014] a defect feature optimization and classification and positioning module, which is used to perform feature optimization and relationship reasoning on the defect relationship graph through a graph neural network to generate relationship-enhanced defect feature representations, and to perform defect classification and positioning based on the defect feature representations to output an identification result of the welding seam defect.

[0015] The above-mentioned welding seam defect intelligent identification method and system based on multi-modal fusion acquire multi-modal data of the same welding seam area to obtain a multi-modal image set, enhance feature maps of each modal image in the multi-modal image set through a preset feature extraction network to obtain enhanced features, and extract multi-scale deep features of the enhanced features. On this basis, deformable convolution is used to realize cross-modal feature alignment, and a physical constraint mechanism is introduced to generate a fusion feature map. Candidate defect regions are generated based on the fusion feature, and feature vectors and position information thereof are extracted, based on which a defect relationship graph is constructed. Finally, relationship reasoning is performed through a graph neural network to optimize feature representations, and the collaborative determination of defect categories, positions and associated relationships is realized, so that the recognition accuracy of symbiotic defect groups is significantly improved in complex industrial scenarios, the physical correlation between defects is reliably quantified, and the anti-interference capability is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0017] Figure 1A flowchart of a welding seam defect intelligent identification method based on multi-modal fusion provided by the present application is shown in the figure.

[0018] Figure 2 A flowchart of constructing a defect relationship graph in an alternative embodiment of the present application is shown in the figure.

[0019] Figure 3 A structural diagram of a welding seam defect intelligent identification system based on multi-modal fusion provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0021] Reference Figure 1 It shows a flowchart of a welding seam defect intelligent identification method based on multi-modal fusion provided by the present application, which comprises the following steps:

[0022] S1, collect multi-modal data of the same welding seam area to obtain a multi-modal image set; enhance the feature maps of each modal image in the multi-modal image set through a pre-set feature extraction network to obtain enhanced features, and extract multi-scale deep features of the enhanced features.

[0023] Specifically, a non-local mean filter is applied to the collected original image to remove random noise to obtain a denoised image; adaptive histogram equalization is applied to the denoised image to enhance the contrast to obtain an enhanced image; the enhanced image is adjusted to a uniform resolution pixel to obtain a normalized image; the welding seam geometric features in the normalized image are extracted as control points, and an affine transformation matrix T is calculated, wherein T satisfies wherein a, b, c, d, e and f are affine transformation parameters, which can be determined by minimizing the Euclidean distance between the control points; finally, the affine transformation matrix T is applied to the normalized image obtained above to obtain a spatially aligned multi-modal image set. The multi-modal data includes metallographic images, ultrasonic images, radiographic images and penetration images.

[0024] Then, the multi-scale deep features F of the obtained modal images are enhanced by spatial attention, which can be generated by 1x1 convolution and Sigmoid activation function to generate a spatial attention map M s :

[0025]

[0026] wherein Conv 1×1 represents a 1x1 convolution operation, and sigma represents a Sigmoid activation function.

[0027] Then, the input feature map F is enhanced by channel attention, first through a squeeze-excitation module composed of a global average pooling and two fully connected layers, and then normalized by a layer normalization operation to generate a channel attention map M c :

[0028]

[0029] wherein LayerNorm represents a layer normalization operation; SEBlock represents a channel attention mechanism module, and the calculation process of SEBlock(F) can be as follows:

[0030] 1) Global average pooling is performed on the input feature map F to compress the CxHxW feature map into a Cx1x1 vector z:

[0031]

[0032] wherein H and W represent the height and width of the feature map, respectively.

[0033] 2) The channel attention weight is calculated by two fully connected layers W1 and W2 and a ReLU activation function:

[0034]

[0035] wherein , , and r is the dimension reduction ratio.

[0036] 3) The channel attention weight s is returned as the output of the SEBlock function.

[0037] Finally, the spatial attention map M s , the channel attention map M c , and the input feature map F are multiplied elementarily to obtain the enhanced feature map F out :

[0038]

[0039] wherein represents an element-wise multiplication operation.

[0040] S2, multi-scale deep features are aligned and fused across modalities by deformable convolution and physical constraint mechanism to generate a fused feature map.

[0041] Specifically, deformable convolution learns the offset of sampling points to adapt to the geometric deviation between multi-modal data. For example, the “projected features” of the ray image and the “reflected features” of the ultrasound image may have a position difference, and the offset can dynamically correct such deviation.

[0042] In specific calculations, the offset sampling points can be fused through bilinear interpolation to capture the deformation characteristics such as bending and irregularity of the defects. For the design of the cross-modal feature alignment loss, regions with large density differences (for example, a gray difference > 30, which can be automatically determined by the Otsu algorithm) in the radiographic image and strong reflection regions in the ultrasonic image can be selected as candidate regions, the L2 distance of the features of these regions is calculated, and the model is forced to learn the cross-modal consistent defect features (such as the feature alignment of the "black line" crack in the radiographic image and the "strong reflection" crack in the ultrasonic image). In feature fusion, the microstructure features of the metallographic image, the surface contour features of the penetration image, and the output features of the deformable convolution are spliced, and then the dimension is reduced to 512 channels through 1x1 convolution. While preserving multi-modal information, the computational load and feature expression ability are balanced.

[0043] S3, generating candidate defect regions based on the fused feature map, and extracting feature vectors and position information of each candidate defect region; constructing a defect relation graph based on the feature vectors and position information of the candidate defect regions.

[0044] Specifically, based on the fused feature map, a region proposal network can be used to generate candidate defect regions. The region proposal network uses a sliding window mechanism to generate candidate regions of different scales and aspect ratios on the fused feature map, and scores each candidate region to select regions that may contain weld defects. Meanwhile, for each candidate defect region, a specific feature extraction method is used, such as convolution operation and pooling operation on the candidate region, to extract its feature vector. The feature vector contains defect feature information within the candidate region and can represent the defect type and severity of the region. In addition, the position information of each candidate defect region is also recorded, which can be represented in the form of a bounding box, clearly indicating the specific position of the defect in the weld image, providing accurate region and position information for subsequent defect relation graph construction.

[0045] Each candidate defect region is regarded as a node in the graph, and the feature of the node is represented by the corresponding feature vector. Then, edges between nodes are established according to the spatial positional relationship between candidate defect regions, feature similarity, etc. For example, if two candidate defect regions are close in position or their feature vectors have high similarity in some metric, an edge is established between them, indicating that there may be an association between the two defect regions. In this way, a graph structure is constructed that reflects the mutual relationship between weld defects, providing a data basis for subsequent graph neural network processing.

[0046] S4, optimizing and reasoning the features of the defect relation graph through the graph neural network to generate relation-enhanced defect feature representation; based on the defect feature representation, performing defect classification and positioning to output the recognition result of the weld defect.

[0047] Specifically, the graph neural network is used for feature optimization and relationship reasoning of the defect relationship graph. The graph neural network updates the feature representation of each node by aggregating the features of the node itself and the features of adjacent nodes, thereby realizing feature optimization. For example, the graph convolution network performs weighted summation operation on the node features according to the connection relationship between the edges of the nodes, so that each node can fuse the information of its neighbor nodes, enrich its own feature representation, and obtain a relationship-enhanced defect feature representation. While updating the node features, the graph neural network can also learn the complex relationship between the nodes and reason out the potential association and interaction between the defects, further mining the intrinsic characteristics of the weld defects, and providing more accurate features for subsequent defect classification and positioning.

[0048] Based on the relationship-enhanced defect feature representation, a classification algorithm and a positioning algorithm are used to realize the classification and positioning of the weld defects. The classification algorithm can determine the category of the weld defects, such as pores, cracks, and incomplete fusion, according to the similarity between the defect feature representation and the feature model of each category of defects. The positioning algorithm determines the specific position of the defect in the weld according to the position information in the defect feature representation, and combines techniques such as bounding box regression to output the category, position, and association relationship of the weld defect, thereby providing accurate basis for the evaluation of the weld quality and subsequent repair work.

[0049] The above-mentioned method for intelligent recognition of weld defects based on multi-modal fusion collects metallographic, ultrasonic, radiographic, and penetrant four modal data of the weld area and performs spatial alignment preprocessing to form a multi-modal image set that can be cooperatively analyzed. Then, the parallel feature extraction network is used to implement double enhancement of spatial attention and channel attention on each modal feature map, and the multi-scale deep features with significant defect response characteristics are extracted through element-level fusion. On this basis, cross-modal feature alignment is realized by using deformable convolution, and a physical constraint mechanism is introduced to generate a fusion feature map. Based on the fusion feature, a candidate defect region is generated, and its feature vector and position information are extracted, and a defect relationship graph is constructed accordingly. Finally, the graph neural network is used for relationship reasoning and optimization of feature representation, and the cooperative determination of the defect category, position, and association relationship is realized, thereby significantly improving the recognition accuracy of the symbiotic defect group in complex industrial scenes, reliably quantifying the physical association between defects, and enhancing the anti-interference ability.

[0050] In an optional embodiment, cross-modal feature alignment and fusion are performed on the multi-scale deep features by using deformable convolution and a physical constraint mechanism to generate a fusion feature map, including the following steps:

[0051] S11, aligning any two modal features F A and F B, a joint feature representation is generated by channel concatenation operation; a pixel-level offset for each spatial position is predicted by processing the joint feature representation through a convolutional layer; the pixel-level offset is applied to a deformable convolution operation to perform spatial transformation on the feature F B to obtain an aligned feature.

[0052] Specifically, the multi-scale deep features contain feature information of different modal images at multiple scales. By performing channel concatenation operation on any two modal features F A and F B , they are concatenated in the channel dimension to generate a joint feature representation. For example, assuming the shape of F A is HxWxC1 and the shape of F B is HxWxC2, the shape of the joint feature representation after channel concatenation is HxWx(C1+C2). In this way, the information of the two modal features can be integrated together to provide a richer feature basis for subsequent processing.

[0053] The convolutional layer is composed of multiple convolutional kernels that slide on the joint feature representation and perform convolution operation. The mathematical expression of the convolution operation can be represented as:

[0054]

[0055] where, is the value of the output feature map at position (i,j), is the value of the joint feature representation at position (i,j) and channel k, is the weight of the convolutional kernel at channel k, and b is the bias term. Through the convolution operation, the model can learn the local features and spatial information in the joint feature representation and predict a pixel-level offset for each spatial position. This offset is a two-dimensional vector indicating the direction and distance of movement for each pixel point in space.

[0056] The core idea of deformable convolution is to dynamically adjust the sampling position of the convolutional kernel according to the offset. Specifically, for each sampling point (i,j) in the feature F B , according to the predicted offset (Δi,Δj), it is transformed to a new position (i+Δi,j+Δj). Then, an interpolation operation (bilinear interpolation can be used) is performed at this new position to obtain the corresponding feature value. After such spatial transformation, the feature F B is adjusted to a state that is more aligned with the feature F A , obtaining an aligned feature. Spatial transformation can make the features more aligned in space, reduce the feature misalignment problem caused by welding heat process and other factors, and improve the matching accuracy and consistency of the features.

[0057] S12, introduce the heat affected zone prior mask, and perform element-level multiplication on the aligned features and the heat affected zone prior mask to obtain physically constrained features; wherein the heat affected zone prior mask is a binary spatial mask used to identify the metal region in the weld area that is affected by the welding heat process but not melted.

[0058] Specifically, to further constrain and optimize the results of feature alignment, a heat affected zone prior mask is introduced. The heat affected zone prior mask is a binary spatial mask, and its shape is the same as the feature map, i.e., [H, W]. In the weld area, the metal region affected by the welding heat process but not melted is marked as 1, and other regions are marked as 0. Element-level multiplication is performed on the aligned features and the heat affected zone prior mask, i.e., pixel-by-pixel multiplication. This operation only retains the region features in the aligned features corresponding to the heat affected zone, and the feature values of other regions are set to 0. In this way, the features are constrained using the physical characteristics of the heat affected zone, highlighting the defect features related to the heat affected zone and reducing the interference of irrelevant regions to obtain physically constrained features.

[0059] S13, repeat steps S11 to S12 for all modal pairs in the multi-scale depth features to obtain multiple groups of physically constrained features; and perform channel concatenation and convolution fusion on the physically constrained features and the multi-scale depth features to generate a fused feature map.

[0060] Specifically, the above process is repeated for all modal pairs in the multi-scale depth features, i.e., channel concatenation, convolution to predict offset, deformable convolution spatial transformation, and physical constraint are performed for each modal pair combination. If there are four modalities, there are six modal pair combinations (metallography-ultrasound, metallography-ray, metallography-penetration, ultrasound-ray, ultrasound-penetration, and ray-penetration), and finally six groups of physically constrained features are obtained.

[0061] If there are six modal pair combinations, and assuming that the number of channels of each group of physically constrained features is C and the number of channels of the multi-scale depth features is D, the shape of the concatenated features is [H, W, 6C+D]. Then, the concatenated features are processed through convolution fusion operation. Convolution fusion operation can use multiple convolution layers, and each convolution layer contains a certain number of convolution kernels. For example, one convolution layer can be used to reduce the number of channels to an intermediate value, and another convolution layer can be used to further fuse the features. Convolution fusion can make features from different sources interact and integrate fully, extract more representative and discriminative feature information, and remove redundant information, finally generating a fused feature map. This fused feature map integrates the advantages of multi-modal images and more accurately reflects the characteristics of weld defects, providing a reliable foundation for subsequent candidate defect region generation and defect classification and positioning.

[0062] In this embodiment, through channel splicing, convolutional prediction offset, deformable convolution spatial transformation and physical constraints, precise alignment and fusion of multi-modal features can be achieved. The feature is constrained by the heat affected zone prior mask, highlighting the defect features related to the heat affected zone, effectively reducing the interference of irrelevant areas, improving the relevance and effectiveness of the features, and thus improving the accuracy of weld defect recognition.

[0063] In an alternative embodiment, the heat affected zone prior mask is a spatial distribution binary mask calculated by a Goldak double ellipsoid heat source model. The calculation process of the heat affected zone prior mask is as follows:

[0064]

[0065]

[0066]

[0067] wherein, is the heat source distribution of the front ellipsoid, representing the heat flux density in the front direction of the welding direction; is the heat source distribution of the rear ellipsoid, representing the heat flux density in the rear direction of the welding direction; is the heat affected zone formed by superimposing the rear, representing the total heat flux density; and are energy distribution coefficients; is the welding heat input power, and are the long semi-axes of the front ellipsoid and the rear ellipsoid, respectively, representing the welding direction; and are the other two semi-axes of the ellipsoid, representing the transverse and depth directions, respectively; , , are spatial coordinates.

[0068] When is greater than the heat flux threshold corresponding to the melting point of the material, the value of the heat affected zone prior mask at the corresponding spatial position is 1, otherwise it is 0.

[0069] Specifically, the model calculates the heat flux density in the front and rear directions of the welding direction, i.e. the heat source distribution of the front ellipsoid and the heat source distribution of the rear ellipsoid , and then adds them to obtain the total heat flux density .

[0070] In the formula, and are energy distribution coefficients, used to distribute the proportion of welding heat input power Q between the front ellipsoid and the rear ellipsoid. and are the long semi-axes of the front and back ellipsoids, respectively, extending along the welding direction, determining the distribution range of the heat source in the welding direction. And b and c are the other two semi-axes of the ellipsoid, corresponding to the transverse and depth directions, respectively, affecting the distribution shape of the heat source in the transverse and depth directions. x, y, z are the spatial coordinates, used to determine the position of each point in the weld area.

[0071] When the calculated value is greater than the heat flux threshold value corresponding to the melting point of the material, it indicates that the spatial position is in the heat-affected zone, and the value of the corresponding heat-affected zone prior mask is set to 1; otherwise, the mask value is 0.

[0072] The heat-affected zone prior mask obtained by this method is a binary spatial mask, which can accurately identify the metal region in the weld area that is affected by the welding heat process but not melted. This prior mask generation method based on physical model provides important physical constraints for subsequent feature alignment and fusion, which helps to improve the accuracy and reliability of weld defect recognition.

[0073] In practical applications, the values of each parameter in the above formula can be determined according to the specific welding process parameters and material properties, so as to generate a prior mask that matches the actual weld heat-affected zone. For example, for different welding heat input power Q, energy distribution coefficients ff and fr, ellipsoid semi-axis lengths a, b, c, etc., they can be adjusted and set according to experimental data or engineering experience to ensure that the generated mask can accurately reflect the distribution of the weld heat-affected zone.

[0074] In this embodiment, the heat-affected zone prior mask is calculated based on the Goldak double-ellipsoid heat source model, which can accurately identify the metal region in the weld area that is affected by the welding heat process but not melted. This prior mask generation method based on physical model provides important physical constraints for feature alignment and fusion, which helps to further improve the accuracy and reliability of feature alignment and enhance the robustness of weld defect recognition.

[0075] Referring to Figure 2 , in an alternative embodiment, based on the feature vector and position information of the candidate defect region, a defect relation graph is constructed, including the following steps:

[0076] S21, define each candidate defect region as a node, initialize the node feature as the corresponding feature vector, form a node set; according to the position information of the candidate defect region, establish a spatial adjacent edge connection between the nodes of the node set; according to the feature vector of each node in the node set, establish a feature similar edge connection between the nodes of the node set.

[0077] ​Specifically, in the process of constructing the defect relation graph, each candidate defect region is first regarded as an independent node, and the feature vector of each node is initialized as the feature vector extracted from the fused feature map, thereby forming a node set containing information of all candidate defect regions. Specifically, the extraction of the feature vector can be realized by applying a region of interest (ROI) pooling operation on the candidate defect region, which pools the candidate region feature maps of different sizes into a fixed-size feature vector. In this way, each feature vector contains information about the shape, texture and semantics of the candidate defect region.

[0078] The center point of the candidate box can be calculated according to the position information of the candidate defect region, and the Euclidean distance between the center points of the candidate boxes corresponding to each candidate defect region is calculated. When the Euclidean distance between any two candidate defect regions is less than a preset threshold D, a spatial proximity edge is established. The threshold D can be set based on the weld width.

[0079] Specifically, the position information is given in the form of a bounding box, including the coordinates of the upper left corner and the lower right corner of the bounding box. Through these four coordinate values, the center point coordinates of each bounding box can be calculated. Then, the Euclidean distance between the center points of the candidate boxes corresponding to all candidate defect regions is calculated. If the Euclidean distance between the center points of any two candidate defect regions is less than a pre-set threshold, a spatial proximity edge is established between the two candidate defect regions. The threshold is set based on the width of the weld to ensure that only spatially close candidate defect regions are connected. In this way, local defect regions that may have relevance can be captured.

[0080] The cosine similarity between each feature vector can be calculated, and when the cosine similarity is higher than a preset threshold, a feature similarity edge is established.

[0081] Specifically, the cosine similarity measures the angle difference between two vectors, ranging from [-1, 1]. When the cosine similarity of two feature vectors is higher than a preset threshold, a feature similarity edge is established, indicating that the two candidate defect regions have high similarity in the feature level and may have the same defect type or similar texture patterns. The selection of this threshold can be based on experiments and domain knowledge to balance sensitivity and accuracy.

[0082] S22, applying a pre-trained classifier to process the feature vector of each node to obtain the defect type of each candidate defect region; according to the pre-set defect type association table in the welding process knowledge base, when the prior association strength between the defect types of any two candidate defect regions is greater than a first preset threshold, a physical prior edge is established.

[0083] Specifically, to exploit the physical correlations between defect types, a pre-trained classifier is applied to each node to predict the defect type of the candidate defect region. The pre-trained classifier can be a convolutional neural network trained on a large amount of labeled data, which can accurately identify the type of weld defects such as porosity, crack, incomplete fusion, etc. Then, according to the pre-defined defect type correlation table in the welding process knowledge base, when the prior correlation strength between the defect types of any two candidate defect regions is greater than a first threshold, a physical prior edge connection is established. The prior correlation strength can be determined based on the physical relationship between different defect types in the welding process, for example, cracks often occur with incomplete fusion, so the correlation strength between them is higher. In this way, defect regions with physical correlations can be connected, which helps to understand the intrinsic relationship between defects.

[0084] S23, based on the distance corresponding to the spatial proximity edge connection, calculate the spatial proximity relationship weight; based on the similarity corresponding to the feature similarity edge connection, calculate the feature similarity relationship weight; based on the prior correlation strength corresponding to the physical prior edge connection, calculate the physical prior relationship weight.

[0085] Specifically, after establishing three different types of edge connections, the weight of each edge is calculated. For spatial proximity edge connection, the spatial proximity relationship weight is calculated based on the corresponding Euclidean distance. The Euclidean distance can be mapped to a weight value, so that the weight between nodes with closer distance is higher. Similarly, for feature similarity edge connection, the feature similarity relationship weight is calculated based on the corresponding cosine similarity. The cosine similarity can be mapped to a weight, or normalized to be within the range of [0, 1]. For physical prior edge connection, combined with the pre-defined mapping rule, the physical prior relationship weight is obtained based on the corresponding prior correlation strength. The prior correlation strength can be a pre-defined value determined according to the statistical data or expert experience in the welding process knowledge base.

[0086] S24, for each node pair, calculate the comprehensive edge weight according to the spatial proximity relationship weight, the feature similarity relationship weight and the physical prior relationship weight; based on the comprehensive edge weight of each node pair, obtain the edge weight matrix.

[0087] Specifically, for each node pair, combined with the above three weights, the comprehensive edge weight is calculated according to the weighted formula. The weighted formula can be designed according to the specific application requirements, which can be a simple linear combination or a more complex nonlinear combination. For example, the comprehensive edge weight can be represented as: comprehensive edge weight = a x spatial proximity relationship weight + β x feature similarity relationship weight + γ x physical prior relationship weight. Wherein, a, β and γ are weight coefficients, used to balance the contribution of the three different types of weights. These coefficients can be determined through experiments or optimization algorithms to achieve the best graph construction effect.

[0088] S25, integrate the spatial adjacent edge connection, the feature similar edge connection and the physical prior edge connection to obtain an edge connection relationship; based on the node set and the edge connection relationship, apply an edge weight matrix to obtain a defect relationship graph.

[0089] Specifically, the generation of the defect relationship graph can be performed in the following ways:

[0090] 1) Integration based on "or". For any two node pairs, as long as there is at least one connection among the three edge connections (spatial adjacent edge, feature similar edge and physical prior edge), it is considered that there is an edge connection between the two nodes. The specific integration steps are as follows:

[0091] a. Create an edge set, which is initially empty.

[0092] b. Traverse all node pairs: if there is a spatial adjacent edge connection between the node pairs, add the edge to the edge set; if there is a feature similar edge connection between the node pairs, add the edge to the edge set; if there is a physical prior edge connection between the node pairs, add the edge to the edge set.

[0093] c. Finally, all edges in the edge set collectively constitute the complete edge connection relationship.

[0094] The edge connection relationship integrated in this way is relatively comprehensive and can include all possible associations.

[0095] 2) Integration based on weighted sum. This way considers the weights of the three edge connections, calculates a comprehensive edge weight through weighted sum, and decides whether to establish an edge connection according to the comprehensive edge weight. The specific integration steps are as follows:

[0096] a. Pre-set the weight coefficients α, β and γ of the three edge connections (for example, determine their relative importance according to experience or experiment).

[0097] b. For each node pair: obtain the spatial adjacent relationship weight w space of the spatial adjacent edge connection; obtain the feature similar relationship weight w feature of the feature similar edge connection; and obtain the physical prior relationship weight w prior of the physical prior edge connection.

[0098] c. Calculate the comprehensive edge weight: w total = α × w space + β × w feature + γ × w prior .

[0099] d. If w total is greater than a certain pre-set comprehensive weight threshold, an edge connection is established between the node pair, and w totalAs the weight of the edge.

[0100] This way can filter out more important edge connections according to the integrated edge weights, making the graph structure more focused on key relationships.

[0101] 3) Multi-path graph neural network based on edge type. If subsequent processing is required using graph neural networks, a multi-path graph neural network based on edge type can be used to integrate edge connection relationships. This way does not directly merge edge connections, but designs specific propagation mechanisms for different types of edges in the graph neural network, allowing each type of edge to play a unique role in information propagation. The integrated edge connection relationship is reflected in the propagation process of the graph neural network, and different types of edges will update and aggregate node features according to their characteristics. The specific implementation steps are as follows:

[0102] a. Construct an independent adjacency matrix for each edge connection type (spatial proximity edge, feature similarity edge, physical prior edge).

[0103] b. In each layer of the graph neural network, information propagation is performed for each edge connection type, for example: for spatial proximity edges, a graph convolutional network (GCN) layer is used to aggregate information between nodes based on spatial proximity relationships; for feature similarity edges, another GCN layer is used to aggregate information between nodes based on feature similarity relationships; for physical prior edges, a third GCN layer is used to aggregate information between nodes based on physical prior relationships.

[0104] c. Fuse the node features obtained by different edge connection types, for example, through concatenation, weighted summation, etc., to obtain updated node features.

[0105] d. Repeat the above process, and after multiple layers of graph neural network propagation and fusion, finally obtain node feature representations containing the influence of various edge connection relationships, thereby indirectly implementing the integration of edge connection relationships.

[0106] This way can fully utilize different types of edge connection information, making the modeling of defect relationship graphs by graph neural networks more comprehensive and accurate, but it is relatively complex to implement, requiring the design and training of corresponding multi-path graph neural network architectures.

[0107] In the embodiment, the candidate defect area is defined as a node and a node set is formed, so that the characteristics and position information of each candidate defect area can be clearly represented. The edge connection relationship is established in multiple ways, including a spatial proximity edge, a feature similarity edge and a physical prior edge, and the spatial position, feature similarity and physical correlation between defects are comprehensively considered, so that the defect relation graph constructed can more comprehensively and accurately reflect the various correlation characteristics between the weld defects, provide more rich and comprehensive structured information basis for the feature optimization and relation reasoning of the defect relation graph by the hollow neural network, and help to more accurately identify the category, position and correlation of the weld defects.

[0108] In an alternative embodiment, S23 comprises the following steps:

[0109] S31, for any two nodes i and j, a spatial proximity relationship weight is calculated by a Gaussian distance decay function; the expression of the spatial proximity relationship weight is:

[0110]

[0111] wherein, is the Euclidean distance between the center points of nodes i and j, is a distance decay coefficient.

[0112] Specifically, the calculation formula of the Euclidean distance is:

[0113]

[0114] wherein, and are the center point coordinates of nodes i and j. These coordinates can be calculated by the bounding box information of the candidate defect area, and the bounding box information can include the left upper corner and right lower corner coordinates, and the center point coordinates can be obtained by taking the average value.

[0115] is a distance decay coefficient, used to control the decay rate of the Gaussian function. The distance decay coefficient can be set according to the threshold value D, and the threshold value D can be determined based on the weld width. For example, if the threshold value D is set as T, then can be set as T / 3 or other values determined by experience or experiment. Through the formula, the spatial proximity relationship weight will exponentially decay with the increase of the distance between nodes, and the weight of nodes with closer distance is higher, and vice versa.

[0116] S32, the cosine similarity between the node feature vectors is taken as the feature similarity relationship weight; the expression of the feature similarity relationship weight is:

[0117]

[0118] wherein, and are the feature vectors corresponding to nodes i and j respectively; the feature similarity relationship weight is set to 0 when the cosine similarity is less than a second preset threshold.

[0119] Specifically, and are the feature vectors corresponding to nodes i and j respectively, and the feature vectors are extracted from the fusion feature map and contain shape, texture and semantic information of the candidate defect region. The dot product calculates the similarity between two vectors, while the denominator is the normalization of the vector length, ensuring that the cosine similarity is in the range [-1, 1].

[0120] When the calculated cosine similarity is less than the second preset threshold, the feature similarity relationship weight is set to 0. The second preset threshold can be determined based on experiments and domain knowledge, and can be between 0.5 and 0.8, with the specific value depending on the distribution of the feature vector and the difference in defect types. For example, if the second preset threshold is set to 0.7, then only when the cosine similarity of the feature vectors of the two nodes is greater than or equal to 0.7, it is considered that they have significant similarity at the feature level, otherwise the similarity is insufficient and the weight is set to 0.

[0121] S33, the prior association strength between the defect types corresponding to nodes i and j is taken as the physical prior relationship weight; wherein, when the prior association strength is less than a first preset threshold, the physical prior relationship weight is set to 0.

[0122] Specifically, the prior association strength is determined according to the pre-set defect type association table in the welding process knowledge base. For example, the prior association strength between cracks and incomplete fusion may be high, while the association strength between pores and slag inclusion may be low.

[0123] When the prior association strength is less than the first preset threshold, the physical prior relationship weight is set to 0. The first preset threshold can also be determined based on welding process knowledge and experience, and can be between 0.6 and 0.9. For example, if the third threshold is set to 0.8, then only when the prior association strength between the defect types corresponding to the two nodes is greater than or equal to 0.8, it is considered that they have significant association in physics, otherwise the weight is set to 0.

[0124] In this embodiment, the spatial proximity relationship weight is calculated using a Gaussian distance decay function, which can reasonably allocate weights according to the Euclidean distance between nodes, so that spatially adjacent node pairs have a greater influence on each other, and are more in line with the actual situation. Taking the cosine similarity as the feature similarity relationship weight and setting the weight to 0 when it is lower than the threshold value can highlight the association between node pairs with high feature similarity and reduce the interference of nodes with dissimilar features. The introduction of prior association strength as the physical prior relationship weight and the setting of threshold screening can make full use of welding process knowledge, emphasize the relationship between defects with physical association, and further optimize the quality of the defect relationship graph, providing more accurate and meaningful edge weight information for subsequent graph neural network processing, which helps to improve the accuracy and reliability of the weld defect recognition.

[0125] In an alternative embodiment, the defect relationship graph is optimized in feature and inferred in relationship by a graph neural network to generate a relationship-enhanced defect feature representation, including the following steps:

[0126] S41, for each node i and neighbor node j in the defect relationship graph, the attention weight of node i to neighbor node j is calculated by splicing the linear transformation results of the node feature vectors.

[0127] Specifically, the expression of the attention weight is:

[0128]

[0129] wherein, represents the node feature vector of node i, represents the node feature vector of neighbor node j, W is a trainable weight matrix, a is a trainable attention vector, and || represents the feature vector splicing operation.

[0130] When performing this step, first, the feature vectors of node i and neighbor node j are linearly transformed respectively. Specifically, the trainable weight matrix W is used to transform the node feature vectors and to obtain and . The purpose of this step is to map the original feature vectors to a new feature space in order to better capture the correlation between nodes.

[0131] Next, the transformed feature vectors and are spliced to obtain a joint feature vector. The splicing operation can integrate the information of the two feature vectors together to form a more comprehensive representation for subsequent attention weight calculation.

[0132] Then, the concatenated joint feature vector is dot multiplied with the trainable attention vector a. This dot product operation measures the correlation between the concatenated feature vector and the attention vector, generating an unnormalized attention score for each neighbor node j.

[0133] Finally, the unnormalized attention scores are normalized using the softmax function to obtain the final attention weights The softmax function converts the input into a probability distribution, ensuring that the sum of all attention weights is 1 and each weight value is within the range [0, 1].

[0134] S42, for each node i, aggregate the node feature vectors of the neighbor nodes with the attention weights as weighting coefficients to generate an updated node feature vector.

[0135] Specifically, the expression of the updated node feature vector is:

[0136]

[0137] where, denotes the neighbor set of node i.

[0138] When performing this step, first traverse all neighbor nodes of node i where, denotes the neighbor set of node i.

[0139] Then, for each neighbor node j, multiply the linearly transformed result of its feature vector with the corresponding attention weight This step weights the linearly transformed result of the neighbor node's feature vector by the attention weight, making important neighbor nodes have a greater impact on the center node.

[0140] Next, sum the linearly transformed results of all weighted neighbor node feature vectors to obtain an aggregated value.

[0141] Finally, use the ReLU activation function to perform a nonlinear transformation on the aggregated value to obtain the updated node feature vector ReLU activation function can introduce nonlinearity and enhance the expression ability of the model.

[0142] S43, stack the processing procedures of steps S41 and S42 L times, and update the node feature through residual connection in each stacking process to output the node feature representation of the next layer, and finally output the node feature representation of the Lth layer.

[0143] Specifically, the expression of the node feature representation is:

[0144]

[0145] wherein, denotes the layer index, .

[0146] In the first stacking process, the current feature representation of node i in the first layer is first obtained. .

[0147] Then, the attention weight is calculated and the neighbor node features are aggregated to obtain a new feature vector according to the method of steps S41 and S42.

[0148] Next, the current feature representation is added to the new feature vector to form the node feature representation of the first layer. The introduction of the residual connection can alleviate the gradient vanishing problem, facilitating the training and optimization of the model. Wherein, denotes the trainable weight matrix of the first layer, denotes the attention weight of node i to neighbor node j in the first layer, and the calculation method is the same as step S41.

[0149] Finally, after L times of stacking processing, the node feature representation of the Lth layer is output. Through multiple stacking, the model can gradually aggregate more extensive neighbor information and capture more complex inter-node relationships.

[0150] S44, extend the attention calculation of steps S41 to S43 to K independent parallel branches, independently calculate the node feature update in each branch, output K groups of independent updated features; integrate the independent updated features to output the final node feature representation.

[0151] Specifically, the expression of the independent updated feature is:

[0152]

[0153] wherein, denotes the group index, .

[0154] In each branch, the processing procedures of steps S41 to S43 are independently executed, i.e., the attention weight is calculated, the neighbor node features are aggregated, and the attention mechanism is stacked. Each branch has independent trainable parameters, such as the weight matrix and the attention vector , wherein k denotes the index of the branch.

[0155] For each branch, the input node feature vector is output after independent attention calculation and feature aggregation process. In this way, K branches will output K sets of independent node feature representations, each set of feature representations capturing the correlation information between nodes from different angles.

[0156] For the node feature representation of the intermediate layer, the output features of the K branches are spliced and merged along the feature dimension. The splicing operation can integrate the feature vectors output by different branches to form a higher-dimensional feature representation, thereby retaining the different information captured by each branch.

[0157] For the node feature representation of the output layer, the output features of the K branches are averaged and merged. The averaging operation can integrate the results of each branch to obtain a more stable and robust feature representation as the final node feature output.

[0158] In this embodiment, the attention mechanism dynamically assigns different weights to the neighbor nodes of each node, so that important neighbor nodes contribute more to the feature update of the center node, enhancing the model's attention to key information. Stacking the attention mechanism and introducing residual connections can gradually aggregate more extensive neighbor information while alleviating the gradient vanishing problem, improving the training effect and performance of the model. Extending the attention calculation to multiple independent parallel branches and integrating the independently updated features can capture the correlation information between nodes from different angles, enhancing the model's expressive power and robustness. The final output node feature representation is more rich and stable, which helps to more accurately classify and locate the weld defects, and improves the overall performance and effect of the weld defect intelligent recognition method.

[0159] The above-mentioned weld defect intelligent recognition method based on multi-modal fusion collects and pre-processes the metallographic, ultrasonic, radiographic and penetration four modal data of the weld area to form a multi-modal image set that can be cooperatively analyzed; then the parallel feature extraction network is used to implement spatial attention and channel attention double enhancement on each modal feature map, and the multi-scale deep features with significant defect response characteristics are extracted through element-level fusion; on this basis, cross-modal feature alignment is realized by using deformable convolution, and a fusion feature map is generated by introducing a heat affected zone physical constraint mechanism; based on the fusion feature, the candidate defect region is generated and its feature vector and position information are extracted, and a defect relation graph with triple relations of fusion space proximity, feature similarity and physical priori is constructed accordingly; finally, the graph neural network is used for relationship reasoning and optimization of feature representation, realizing the cooperative determination of defect category, position and correlation, thereby significantly improving the recognition accuracy of coexisting defect groups in complex industrial scenes, reliably quantifying the physical correlation between defects, and enhancing the anti-interference ability.

[0160] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0161] Based on the same inventive concept, the embodiments of the present application also provide a system for implementing the above-mentioned multi-modal fusion-based intelligent weld defect recognition method. The problem-solving implementation scheme provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more multi-modal fusion-based intelligent weld defect recognition system embodiments provided below can be referred to the limitations of the multi-modal fusion-based intelligent weld defect recognition method described above, which will not be repeated here.

[0162] In an exemplary embodiment, as shown in Figure 3 A multi-modal fusion-based intelligent weld defect recognition system 30 is provided for implementing the method in the above method embodiments, which includes:

[0163] A multi-modal data acquisition and feature enhancement module 31 is configured to acquire multi-modal data of the same weld area to obtain a multi-modal image set; and enhance feature maps of each modality image in the multi-modal image set through a pre-set feature extraction network to obtain enhanced features and extract multi-scale deep features of the enhanced features.

[0164] A cross-modal feature fusion module 32 is configured to perform cross-modal feature alignment and fusion on the multi-scale deep features through deformable convolution and physical constraint mechanism to generate a fusion feature map.

[0165] A candidate defect analysis and relationship construction module 33 is configured to generate candidate defect regions based on the fusion feature map, and extract feature vectors and position information of each candidate defect region; and construct a defect relationship graph based on the feature vectors and position information of the candidate defect regions.

[0166] A defect feature optimization and classification positioning module 34 is configured to perform feature optimization and relationship reasoning on the defect relationship graph through a graph neural network to generate a relationship-enhanced defect feature representation; and perform defect classification and positioning based on the defect feature representation to output a recognition result of the weld defect.

[0167] The above-described embodiments only express several implementation manners of the application, the description is more specific and detailed, but it cannot be understood as the limitation of the patent scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the application, several modifications and improvements can be made, which are within the protection scope of the application.

Claims

1. A method for intelligent identification of weld defects based on multimodal fusion, characterized in that, The method includes: S1. Collect multimodal data of the same weld area to obtain a multimodal image set; enhance the feature maps of each modality image in the multimodal image set through a preset feature extraction network to obtain enhanced features, and extract the multi-scale depth features of the enhanced features; S2. Through deformable convolution and physical constraint mechanisms, cross-modal feature alignment and fusion are performed on the multi-scale deep features to generate a fused feature map; S3. Generate candidate defect regions based on the fused feature map, and extract the feature vectors and location information of each candidate defect region; construct a defect relationship map based on the feature vectors and location information of the candidate defect regions; S4. Perform feature optimization and relationship reasoning on the defect relationship graph using a graph neural network to generate a defect feature representation with enhanced relationships; based on the defect feature representation, perform defect classification and localization, and output the identification result of weld defects.

2. The method according to claim 1, characterized in that, The process involves cross-modal feature alignment and fusion of the multi-scale deep features using deformable convolution and physical constraint mechanisms to generate a fused feature map, including: S11, for any two modal features F in the multi-scale depth features A and F B A joint feature representation is generated through channel concatenation; this joint feature representation is then processed by a convolutional layer to predict the pixel-level offset at each spatial location; and the pixel-level offset is applied to a deformable convolutional operation to refine the feature F. B Perform spatial transformation to obtain alignment features; S12. Introduce a heat-affected zone prior mask, and perform element-wise multiplication between the alignment feature and the heat-affected zone prior mask to obtain the physically constrained feature; wherein, the heat-affected zone prior mask is a binary spatial mask used to identify the metal area in the weld region that is affected by the welding heat process but has not melted. S13. Repeat steps S11 to S12 for all modal pairs in the multi-scale depth features to obtain multiple sets of physically constrained features; perform channel splicing and convolution fusion on the physically constrained features and the multi-scale depth features to generate the fused feature map.

3. The method according to claim 2, characterized in that, The prior mask for the heat-affected zone is a spatially distributed binary mask calculated using the Goldak double ellipsoidal heat source model. The calculation process for the prior mask for the heat-affected zone is as follows: in, The heat source distribution on the fore-ellipsoid represents the heat flux density in front of the welding direction; The heat source distribution on the rear ellipsoid represents the heat flux density behind the welding direction; The heat-affected zone formed by the superposition of two ellipsoids is represented by the total heat flux density. and Energy distribution coefficient; For welding heat input power, and These are the semi-major axes of the anterior and posterior ellipsoids, respectively, indicating the welding direction; and These are the other two semi-axes of the ellipsoid, representing the horizontal and depth directions, respectively; , , Spatial coordinates; when When the heat flux threshold is greater than the melting point of the material, the a priori mask value of the heat-affected zone at the corresponding spatial location is 1; otherwise, it is 0.

4. The method according to claim 1, characterized in that, The step of constructing a defect relationship graph based on the feature vectors and location information of the candidate defect regions includes: S21. Define each candidate defect region as a node, initialize the node features to the corresponding feature vector, and form a node set; establish spatial proximity edges between the nodes in the node set according to the location information of the candidate defect region; establish feature similarity edges between the nodes in the node set according to the feature vectors of each node in the node set. S22. Apply a pre-trained classifier to each node to process the feature vector and obtain the defect type of each candidate defect region. According to the preset defect type association table in the welding process knowledge base, when the prior association strength between the defect types of any two candidate defect regions is greater than the first preset threshold, establish a physical prior edge connection. S23. Calculate the spatial proximity relation weight based on the distance corresponding to the spatial proximity edge connection; calculate the feature similarity relation weight based on the similarity corresponding to the feature similarity edge connection; calculate the physical prior relation weight based on the prior association strength corresponding to the physical prior edge connection. S24. For each node pair, calculate the comprehensive edge weight based on the spatial proximity relation weight, the feature similarity relation weight, and the physical prior relation weight; obtain the edge weight matrix based on the comprehensive edge weight of each node pair; S25. Integrate the spatial proximity edge connection, the feature similarity edge connection, and the physical prior edge connection to obtain the edge connection relationship; based on the node set and the edge connection relationship, apply the edge weight matrix to obtain the defect relationship graph.

5. The method according to claim 4, characterized in that, S23 includes: S31. For any two nodes i and j, calculate the spatial proximity weight using the Gaussian distance decay function; the expression for the spatial proximity weight is: in, Let be the Euclidean distance between the center points of nodes i and j. This is the distance attenuation coefficient; S32. The cosine similarity between the feature vectors of each node is used as the feature similarity relation weight; the expression for the feature similarity relation weight is: in, and These are the feature vectors corresponding to nodes i and j, respectively; when the cosine similarity is less than the second preset threshold, the feature similarity relation weight is set to 0; S33. The prior association strength between the defect types corresponding to nodes i and j is used as the physical prior relationship weight; wherein, when the prior association strength is less than the first preset threshold, the physical prior relationship weight is set to 0.

6. The method according to any one of claims 1 to 5, characterized in that, The step of performing feature optimization and relationship reasoning on the defect relationship graph using a graph neural network to generate a relationship-enhanced defect feature representation includes: S41. For each node i and its neighboring node j in the defect relationship graph, calculate the attention weight of node i to its neighboring node j by concatenating the linear transformation results of the node feature vectors. S42. For each node i, using the attention weight as a weighting coefficient, aggregate the node feature vectors of neighboring nodes to generate an updated node feature vector. S43. Stack the processing steps S41 and S42 L times. In each stacking process, update the node features through residual connections and output the node feature representation of the next layer. Finally, output the node feature representation of the Lth layer. S44. Extend the attention calculation in steps S41 to S43 into K independent parallel processing branches. Calculate node feature updates independently in each branch and output K sets of independent update features. Integrate the independent update features and output the final node feature representation.

7. A multimodal fusion-based intelligent weld defect identification system, used to implement the method according to any one of claims 1 to 6, characterized in that, The system includes: The multimodal data acquisition and feature enhancement module is used to acquire multimodal data of the same weld area to obtain a multimodal image set; the feature maps of each modality image in the multimodal image set are enhanced by a preset feature extraction network to obtain enhanced features, and the multi-scale depth features of the enhanced features are extracted; The cross-modal feature fusion module is used to perform cross-modal feature alignment and fusion on the multi-scale deep features through deformable convolution and physical constraint mechanisms to generate a fused feature map; The candidate defect analysis and relationship construction module is used to generate candidate defect regions based on the fused feature map, and extract the feature vectors and location information of each candidate defect region; and construct a defect relationship map based on the feature vectors and location information of the candidate defect regions. The defect feature optimization and classification localization module is used to optimize the features and reason about the relationship of the defect relationship graph through a graph neural network to generate a defect feature representation with enhanced relationship; based on the defect feature representation, it performs defect classification and localization and outputs the identification result of weld defect.

Images