Resistance spot welding quality detection method and system based on physical guidance multi-mode

Abstract

The invention relates to a resistance spot welding quality detection method and system based on physical guidance multi-mode. The method comprises the following steps: firstly, acquiring multi-sensor data of a visual mode and a time sequence mode of resistance spot welding and finishing preprocessing; then constructing a multi-view image encoder group to extract visual modal data features, obtaining unified visual features through fusion, and processing the preprocessed time sequence modal data through a time sequence encoder to generate a time sequence feature sequence; then, a hierarchical attention convergence network is built, the unified visual features serve as query vectors, the time sequence feature sequences serve as key vectors and value vectors to be input, and physically-guided multi-modal fusion features are obtained; and finally, analyzing the fusion features by using a multi-task prediction network, and synchronously outputting a welding quality classification result of resistance spot welding and a physical performance regression prediction value. Compared with the prior art, the method has the advantages that the accuracy and reliability of resistance spot welding quality detection are remarkably improved, and the like.

Description

Technical Field

[0001] This invention relates to the field of welding quality inspection technology, and in particular to a method and system for inspecting resistance spot welding quality based on physical-guided multimodal methods. Background Technology

[0002] Resistance spot welding (RSW) is widely used in automotive manufacturing and aerospace. With the development of Industry 4.0, various sensors are integrated into production lines to monitor welding quality. Existing quality inspection methods are mainly divided into image-based inspection and process signal-based inspection. However, existing technologies have the following limitations: 1. Limitations of a single mode: It is difficult to detect surface defects such as spatter by relying solely on process parameters, and it is impossible to reveal the dynamic mechanism of melt nucleus formation by relying solely on images.

[0003] 2. Simple multimodal stitching: Existing multimodal fusion methods (such as simple stitching or symmetric attention) ignore the physical causal relationship that "the welding process determines the welding result, and the welding image is the final manifestation of the process", and lack interpretability.

[0004] 3. Severe class imbalance: There are far more qualified samples than defective samples in industrial sites, resulting in poor model recognition of rare defects (such as cold solder joints and burn-through), and the simple classification task is difficult to improve the regression prediction accuracy of physical properties (such as shear force).

[0005] For example, the invention patent with publication number CN119927394A discloses a resistance spot welding quality inspection method based on edge computing. This method constructs a dynamic resistance curve by acquiring voltage and current data and uses an embedded neural network for quality prediction. However, this method has significant limitations: the scheme relies only on time-series signals of a single mode such as voltage and current, and cannot acquire key visual information such as weld surface morphology and heat distribution, making it difficult to detect surface defects such as spatter and burn-through; the scheme uses a simple neural network structure, lacking the ability to model the complex correlation between multimodal data, and cannot realize the physical-guided diagnostic logic of tracing the cause from the effect; the scheme only performs a single quality classification task, without considering the synergistic optimization with physical performance regression prediction, and lacks a special processing mechanism for the problem of imbalanced categories in industrial data; the system architecture in this scheme is relatively simple, lacks modular design, and is difficult to adapt to the diverse needs of complex industrial environments.

[0006] In summary, current resistance spot welding quality inspection technology suffers from limitations in single-modal information, lack of a physically guided multimodal fusion mechanism, insufficient ability to handle class imbalance, and lack of multi-task collaborative prediction capabilities. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a physical-guided multimodal resistance spot welding quality inspection method and system.

[0008] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a method for detecting the quality of resistance spot welding based on physical guided multimodal methods is provided, the method comprising the following steps: S1. Acquire multimodal sensor data of the resistance spot weld to be inspected and preprocess it. The multimodal sensor data includes visual modal data and temporal modal data. S2. Construct a multi-view image encoder group, input visual modal data into the multi-view image encoder group, output feature extraction results, fuse the feature extraction results, and thus generate unified visual features. S3. Construct a time encoder, input the preprocessed time modal data into the time encoder, and output a time feature sequence; S4. Construct a hierarchical attention aggregation network; use unified visual features as query vectors and temporal feature sequences as key and value vectors, input them into the hierarchical attention aggregation network, and output physically guided multimodal fusion features; S5. The multi-task prediction network is used to process the physical guidance multimodal fusion features, and the welding quality classification results and physical performance regression prediction values ​​of resistance spot welding are output as the resistance spot welding quality detection results.

[0009] As a preferred technical solution, the prediction model is composed of a multi-view image encoder group, a temporal encoder, a hierarchical attention aggregation network, and a multi-task prediction network. The method also includes a model training process: during training, a training dataset is established using sample multimodal sensor data, and this training dataset is used as input to execute steps S1-S5. The prediction model is trained end-to-end using a hybrid loss function to obtain the trained prediction model. During prediction, the data to be detected is input into the trained prediction model, and the welding quality classification result and physical performance regression prediction value are output.

[0010] As a preferred technical solution, the hybrid loss function includes classification loss, regression loss, and invariant regularization loss; The classification loss uses focus loss, which reduces the weight of easily classified samples by using the focus parameter γ, and focuses on difficult-to-classify samples such as cold solder joints and spatter; the regression loss uses mean squared error loss, which is used to supervise the prediction of weld nugget diameter and tensile shear force.

[0011] As a preferred technical solution, when performing step S1 on the training dataset, the preprocessing also includes data balancing processing of the training dataset based on a hybrid resampling strategy. The specific process is as follows: using the Synthetic Minority Over-sampling Technique (SMOTE) with adaptive neighborhood parameters, the number of samples of each class in the training dataset is counted to determine the number of samples of the least number class, and the K-nearest neighbor parameter is set according to the number of samples of the least number class; then, based on the K-nearest neighbor parameter, new synthetic samples are generated in the feature space through linear interpolation.

[0012] As a preferred technical solution, the visual modal data includes infrared thermal imaging images, RGB images of the front of the solder joint and RGB (Red-Green-Blue) images of the back of the solder joint; the temporal modal data includes current, voltage, pressure, electrode displacement, welding time and plate thickness parameters during the welding process.

[0013] As a preferred technical solution, preprocessing includes: resizing and normalizing the image data and performing Z-Score standardization on the temporal parameters.

[0014] As a preferred technical solution, the multi-view image encoder group in S2 includes three pre-trained multi-view image encoders with independent parameters; the three pre-trained multi-view image encoders are used to extract infrared thermal imaging images, solder joint front RGB images and solder joint back RGB images respectively. The feature extraction results in S2 include infrared features, frontal features, and backal features; The specific process of fusing the feature extraction results in S2 to generate unified visual features includes: A three-way parallel inter-view cyclic cross-attention mechanism is used to process infrared features, frontal features, and back features. The three paths include: The infrared-dominated path uses infrared features as the query vector and the concatenation of front and back features as the key and value vectors. The front-dominant path uses front features as the query vector and the concatenation of infrared and back features as the key and value vectors. The back-side dominant path uses back-side features as the query vector and the concatenation of front-side features and infrared features as the key vector and value vector. Finally, the outputs of these three paths are concatenated, and the concatenation result is fused through a fully connected layer to generate a unified visual feature that includes complementary information on thermal distribution and surface morphology.

[0015] As a preferred technical solution, in S3, the specific process of generating the time-series feature sequence includes: in the time encoder, using a bidirectional long short-term memory network to extract bidirectional dynamic features of the time-series modal data, and injecting sinusoidal position codes as time position information into the extraction results, and the injected sequence is the time-series feature sequence.

[0016] As a preferred technical solution, the hierarchical attention aggregation network in S4 specifically performs the following operations: constructing an asymmetric attention mechanism, using unified visual features as query vectors and temporal feature sequences as key and value vectors; mapping the query vector, key vector, and value vector to multiple attention subspaces through a linear projection matrix, calculating the attention weights of the unified visual features and features at each time step in the temporal feature sequence within each subspace; weighting and aggregating the temporal features according to the attention weights, and concatenating the outputs of multiple attention heads to obtain physically guided multimodal fusion features.

[0017] According to another aspect of the present invention, a physical-guided multimodal resistance spot welding quality inspection system is provided, the system comprising a data acquisition module, a data preprocessing module, a visual feature processing module, a temporal feature processing module, a physical-guided fusion module, and a multi-task prediction module; The data acquisition module is used to collect multimodal sensor data for resistance spot welding; The data preprocessing module performs preprocessing operations on multimodal sensor data; The visual feature processing module includes a multi-view image encoder group, which is used to process visual modal data and output feature extraction results, and fuse the feature extraction results to generate unified visual features; The temporal feature processing module includes a temporal encoder, which processes the preprocessed temporal modal data and outputs a temporal feature sequence; The physics-guided fusion module includes a hierarchical attention aggregation network, which processes unified visual features as query vectors and temporal feature sequences as key and value vectors to output physics-guided multimodal fusion features. The multi-task prediction module, including a multi-task prediction network, is used to process physical-guided multimodal fusion features, and simultaneously outputs the welding quality classification results and physical performance regression prediction values ​​for resistance spot welding.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention breaks through the information barrier of a single modality by fusing multi-source data from visual and temporal modalities. It combines a hierarchical attention aggregation network to construct a physically guided multimodal fusion mechanism, achieving accurate alignment and in-depth mining of cross-modal features. This avoids the problems of insufficient feature utilization and large prediction bias caused by fusion without physical guidance. At the same time, relying on the collaborative prediction design of a multi-task prediction network, and with the technical strategy of data balancing and hybrid loss function, it takes into account the detection performance in class imbalance scenarios, and makes up for the one-sidedness of the results of single-task prediction. It improves the accuracy and reliability of resistance spot welding quality detection from multiple dimensions such as information fusion, feature utilization, scenario adaptation and task collaboration.

[0019] 2. In this invention, the collaborative processing of visual modal data and temporal modal data is achieved by organically combining a multi-view image encoder group, a temporal encoder, a hierarchical attention aggregation network, and a multi-task prediction network. This processing not only integrates multimodal information but also simulates the diagnostic logic of tracing causes from effects. It uses post-weld image features to dynamically focus on key moments in the welding process, significantly improving the interpretability and prediction accuracy of the model. This fundamentally solves the technical problems of limited information and low prediction accuracy in traditional single-modal methods.

[0020] 3. In this invention, a hybrid loss function is designed that combines focus loss, mean squared error loss, and invariance regularization loss, and a SMOTE oversampling technique based on adaptive neighborhood parameters is used for data balancing. By combining SMOTE and focus loss, the problem of extreme imbalance in industrial data is effectively solved. Experiments show that this strategy not only significantly improves the defect recall rate, but also significantly reduces the mean absolute error of shear force prediction through feature sharing, achieving synergistic gains from the hybrid training strategy.

[0021] 4. This invention clearly defines multimodal data types including infrared thermal imaging, front and back RGB images, and process parameters. It employs standardized preprocessing, a three-way parallel inter-view cyclic cross-attention mechanism, bidirectional LSTM temporal modeling, and asymmetric attention aggregation. This achieves comprehensive monitoring and deep fusion of welding process heat distribution, surface morphology, and process parameters. Cyclic cross-attention fully leverages the complementary relationships of multi-view information, bidirectional temporal modeling captures the dynamic evolution of the welding process, and asymmetric attention enables precise cross-modal feature alignment.

[0022] 5. In this invention, by constructing an end-to-end prediction model, it supports the simultaneous output of classification and regression results. The introduction of invariance loss makes the model robust to non-essential disturbances such as shooting angle and lighting. At the same time, it achieves the synchronous output of quality classification and physical performance quantification, meeting the needs of comprehensive monitoring in industrial sites. Through multi-task joint learning, it achieves feature sharing and mutual performance promotion.

[0023] 6. This invention designs a modular architecture for a resistance spot welding quality inspection system, including functional modules such as data acquisition, preprocessing, visual processing, time-series processing, fusion, and prediction. The modular design achieves system maintainability and scalability, with clearly defined responsibilities and standardized interfaces for each module, facilitating engineering deployment and subsequent functional upgrades. This provides a complete solution framework for the industrial application of resistance spot welding quality prediction. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the steps of a resistance spot welding quality inspection method based on physical guidance multimodal methods in this invention; Figure 2 This is a schematic diagram of the overall architecture of the prediction model in the embodiment; Figure 3 This is a diagram illustrating the training process of the prediction model in the embodiment; Figure 4 This is a structural diagram of the hierarchical attention aggregation network in the embodiment; Figure 5 This is a schematic diagram illustrating the principle of the hierarchical attention aggregation network in this invention; Figure 6 This is a flowchart illustrating the hybrid training strategy and loss function calculation in the embodiment. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] This scheme first acquires multimodal data of resistance spot welding, including infrared thermography, post-weld front and back images, and welding process time-series signals. The data is then balanced using a hybrid resampling strategy. A visual Transformer-based image encoder is constructed to extract multi-view visual features, and a cross-attention mechanism is used to generate unified visual features. A time-series encoder is constructed to extract the temporal dynamic features of the process signals. A hierarchical attention aggregation network is used, with the unified visual features as the query vector, to perform asymmetric attention weighting on the temporal dynamic features as the key vector, achieving physically guided feature fusion. Based on a multi-task prediction head, welding quality classification results and physical performance regression prediction values ​​are simultaneously output. This scheme effectively solves the problems of sample imbalance and the difficulty of multimodal heterogeneous data fusion by using image results to guide process signal feature extraction and combining a hybrid training strategy. This significantly improves the accuracy of identifying defects such as cold welds and spatter, as well as the prediction accuracy of weld nugget diameter and joint tensile strength.

[0027] Example 1 In this embodiment, a physical-guided multimodal resistance spot welding quality inspection method is adopted, and the method steps are as follows: Figure 1 As shown, it specifically includes: S1. Acquire multimodal sensor data of the resistance spot weld to be inspected and preprocess it. The multimodal sensor data includes visual modal data and temporal modal data. S2. Construct a multi-view image encoder group, input visual modal data into the multi-view image encoder group, output feature extraction results, fuse the feature extraction results, and thus generate unified visual features. S3. Construct a time encoder, input the preprocessed time modal data into the time encoder, and output a time feature sequence; S4. Construct a hierarchical attention aggregation network; use unified visual features as query vectors and temporal feature sequences as key and value vectors, input them into the hierarchical attention aggregation network, and output physically guided multimodal fusion features; S5. The multi-task prediction network is used to process the physical guidance multimodal fusion features, and the welding quality classification results and physical performance regression prediction values ​​of resistance spot welding are output as the resistance spot welding quality detection results.

[0028] The aforementioned multi-view image encoder group, temporal encoder, hierarchical attention aggregation network, and multi-task prediction network together constitute the prediction model, as shown in the figure. Figure 2 As shown, the visual coding branch, the temporal coding branch, and the connection between the two through the physical-guided fusion via the HAAN module are illustrated. The method also includes the model training process, the specific implementation of which is as follows: Figure 3 As shown, the complete technical path from data acquisition, SMOTE resampling, dual-path encoding, HAAN fusion to multi-task prediction is as follows: During training, a training dataset is established using sample multimodal sensor data, and this training dataset is used as input to execute steps S1-S5. The prediction model is trained end-to-end using a hybrid loss function to obtain the trained prediction model. During prediction, the data to be detected is input into the trained prediction model, and the welding quality classification result and physical performance regression prediction value are output.

[0029] The hybrid loss function includes classification loss, regression loss, and invariance regularization loss. The classification loss uses focus loss, which reduces the weight of easily classified samples by using the focus parameter γ, and focuses on difficult-to-classify samples such as cold welds and spatter. The regression loss uses mean squared error loss, which is used to supervise the prediction of weld nugget diameter and tensile shear force.

[0030] When performing step S1 on the training dataset, the preprocessing also includes data balancing of the training dataset based on a hybrid resampling strategy. The specific process is as follows: using the synthetic minority class oversampling technique with adaptive neighborhood parameters, the number of samples of each class in the training dataset is counted to determine the number of samples of the least number class, and the K nearest neighbor parameter is set according to the number of samples of the least number class; then, based on the K nearest neighbor parameter, new synthetic samples are generated in the feature space through linear interpolation.

[0031] The specific implementation process of this method is as follows: Multimodal data acquisition and standardization preprocessing: This embodiment uses a dataset containing 495 welding samples. For each welding sample, data for the following four modalities were obtained: Infrared thermal imaging (IR Image): Reflects the final thermal distribution state of the solder joint; Front visible light image (RGB Front Image): Reflects the surface texture and electrode indentations on the front of the solder joint; Back image (RGB Back Image): Reflects the penetration of light on the back of the solder joint and the heat-affected zone; Process Parameters: Includes physical signals in seven dimensions during the welding process, including current (A), voltage (V), pressure (Force), electrode displacement, welding time, plate thickness A, and plate thickness B.

[0032] The data preprocessing steps are as follows: Image preprocessing: All images were resized to 224×224 pixels. The training set images were randomly horizontally flipped (probability 0.5) and randomly rotated. Data augmentation strategies were employed to improve the model's generalization ability. Normalization was then performed, with the mean set to [0.485, 0.456, 0.406] and the standard deviation set to [0.229, 0.224, 0.225].

[0033] Time series data standardization: Z-score standardization is performed on the 7-dimensional process parameters. The mean of each parameter in the training set is calculated. and standard deviation For input values Perform conversion ,in To prevent tiny quantities with a denominator of zero.

[0034] Label processing: Classification labels are mapped to numerical IDs (Good: 0, Bad: 1, Explode: 2); regression labels (melting core diameter, tensile shear force) are also Z-score normalized to facilitate model convergence.

[0035] Hybrid resampling based on adaptive SMOTE: To address the scarcity of defect samples in industrial data, hybrid resampling is implemented during the data loading phase: Specifically, count the number of samples in each category in the training set to determine the minimum number of samples in each category. Set the K-nearest neighbor parameter. By utilizing the synthetic minority class oversampling technique, new minority class samples are generated in the feature space to balance the distribution of each class, thereby eliminating the model's bias towards majority class samples. In this embodiment, after the original training set is processed by SMOTE, the number of samples in each class reaches a balanced state.

[0036] Constructing a multi-view visual feature deep fusion encoder based on ViT: This step corresponds to the Advanced Image Fusion Encoder class and create_vit_encoder method (i.e., the advanced image fusion encoder class and the method for constructing a visual Transformer encoder) in this solution. Unlike traditional convolutional neural networks (CNNs), this embodiment uses a self-attention-based visual transformer (Vision Transformer, ViT) to capture long-distance dependencies in solder joint images and achieves information complementarity through a specific inter-view interaction mechanism.

[0037] Image segmentation and location embedding: The input infrared thermal image (IR), frontal RGB image (RGB-F), and rear RGB image (RGB-B) are fed into three independent ViT backbone networks.

[0038] Block processing: Divide the 224×224×3 input image into N fixed-size 16×16 patches, where N=196.

[0039] Linear projection: Each flattened tile is mapped to a latent vector of dimension D=768 through a linear mapping layer.

[0040] Position encoding: In order to preserve spatial structure information, a learnable position embedding and a category token ([CLS] token) are superimposed on each latent vector to form an input sequence.

[0041] Transformer encoder stack: The input sequence is processed by a stacked Transformer encoder with L=12 layers. Each encoder layer contains two sub-layers: Multi-head Self-Attention Layer (MSA): Used to capture global relationships between tiles; the calculation formula is as follows. Multilayer Perceptron (MLP): consists of two fully connected layers and the GELU activation function.

[0042] Residual and Normalization: Each sub-layer is followed by a residual connection and layer normalization. The final extracted [CLS] token vector (with a 256-dimensional mapping) serves as the global feature representation for this perspective.

[0043] Inter-view cyclic cross-attention fusion: To explore the potential correlation between infrared thermal distribution and visible light surface texture, a three-way parallel cross-attention fusion module was designed (corresponding to the CrossAttention class in the code): Infrared dominant path: based on infrared characteristics The query vector is a concatenation of front and back features. Using key and value vectors, an attention mechanism is employed to extract texture features related to areas of abnormal thermal distribution in visible light images.

[0044] Positive dominant path: based on positive characteristics For Query, with Key / Value; Dominant path on the back: based on dorsal features For Query, with Key / Value; Unified Fusion: The outputs of the three cross-attention pathways are concatenated and fused into a final unified visual feature through a fully connected layer. .

[0045] The formula for calculating cross attention is: Finally, the outputs of the three cross-attention pathways are concatenated and compressed back to 256 dimensions using a linear layer, a ReLU activation function, and layer normalization to obtain unified visual features. .

[0046] Constructing an enhanced bidirectional temporal dynamic encoder: This step corresponds to the EnhancedTemporalAttentionEncoder class (i.e., the enhanced temporal attention encoder class) in this scheme. Considering the characteristics of rapid transient changes and strong dependencies in welding process signals, this embodiment designs a bidirectional recurrent network structure that incorporates position encoding.

[0047] Multidimensional time-series signal construction: Obtain 7-dimensional process parameters during welding: P = [current, voltage, pressure, displacement, time, plate thickness A, plate thickness B]. Construct a time-series matrix of length T. .

[0048] Bidirectional Long Short-Term Memory (Bi-LSTM) Feature Extraction: Input sequence X into two stacked layers of bidirectional LSTM (Long Short-Term Memory network).

[0049] Bidirectional mechanism: Bi-LSTM includes a forward layer and a backward layer. The forward layer processes the sequence from t=1 to T to capture the cumulative effect of weld nugget growth; the backward layer processes the sequence from t=T to 1 to capture the cooling springback characteristics at the end of the weld.

[0050] Feature concatenation: combining the forward hidden states of each time step and backward hidden state The features are concatenated to generate a dynamic feature vector containing complete contextual information. The hidden layer dimension is set to 128, and the output feature dimension is mapped to 256.

[0051] Sinusoidal Positional Encoding: While LSTMs are capable of processing sequences, they are prone to losing absolute temporal position information in long sequences. This embodiment innovatively injects sinusoidal position encoding (corresponding to the PositionalEncoding class) after the LSTM output to enhance the model's ability to perceive specific welding stages (such as the pre-pressure stage, energizing stage, and sustaining stage). The final output time-series feature sequence This will serve as the key and value for the HAAN network in the subsequent step S5.

[0052] Constructing an image-guided hierarchical attention aggregation network (HAAN) as follows: Figure 4 As shown, and its principle is illustrated as follows. Figure 5 As shown, this demonstrates in detail how image features (Query) dynamically focus on key physical events (such as splash features and pressure fluctuations) in a time-series sequence (Key / Value) through an attention mechanism: Figure 5 In this model, fused image features (Query, dimension [Batch, 1, 256]) are used as the query, and time-series feature sequences (Key / Value, dimension [Batch, 8, 256]) are used as the key and value. Attention weights are calculated through attention, which guide the model to focus on key events in the time series (such as splash features at time t3 and pressure fluctuations at time t5). After feature aggregation (output dimension [Batch, 1, 256]), the features are added to the original fused image features, resulting in a fused feature with dimension [Batch, 1, 512]. This achieves accurate guidance and effective fusion of image features with time-series features.

[0053] This step corresponds to the Hierarchical Attention Aggregation module in this scheme. This module aims to achieve deep physical fusion of visual and temporal modalities. Unlike traditional symmetrical feature stitching, this scheme proposes an image-guided cross-attention mechanism. This mechanism uses the post-weld image as the result anchor point, dynamically focusing on key moments in the welding process (such as the current peak stage), thereby generating a joint representation with high information density. The HAAN module utilizes the physical prior that "the result image is determined by the process," and uses image features to backtrack abnormal fluctuations in the process data (such as current surges during spatter), thereby achieving physically guided feature fusion. The specific execution process is as follows: Constructing an asymmetric query-key-value system: In order to establish a physical logic that uses the result to deduce the process, this step breaks the conventional self-attention mechanism and constructs an asymmetric input system: Query vector (Query, Q): Selects uniform visual features (256 dimensions). This feature encodes the final visual morphology of the solder joint, such as weld nugget size, spatter marks, or surface cracks.

[0054] Key vector (Key, K) and value vector (Value, V): Select the time-series feature sequence F T (Dimension is L×256, where L is the time step). This sequence encodes the dynamic evolution of process parameters such as current and voltage over time.

[0055] Multi-head attention computation: To capture potential physical correlations from different subspaces, this embodiment employs a multi-head attention mechanism. The number of attention heads is set to h = 8. The calculation process is as follows: Linear projection: First, through a learnable projection matrix , , Map the input features to h distinct subspaces: , , in, = 1,…, h represents the first, second, and third digits of the first digit. One point of attention.

[0056] Relevance-scaled dot product: Within each subspace, calculate the relevance score (Attention Scores) between visual features and process features at each time step. in, This is a scaling factor used to prevent gradient vanishing due to excessively large dot products.

[0057] Multi-head concatenation and linear fusion: Concatenate the outputs of h heads and then project the results using the output projection matrix. Perform linear fusion to obtain initial fusion features: The formula mathematically describes how the model integrates information from different subspaces (e.g., one head focuses on current peaks, and another head focuses on pressure abrupt changes).

[0058] Physical mechanism explanation and dynamic focus: This mechanism has clear physical interpretability: the model automatically calculates the attention weights of the current weld point to each time step in the welding process time series (K) based on its visual morphology (Q).

[0059] Example effect: For example, when visual features When the morphological characteristics of "explosion" are displayed, the attention mechanism will automatically assign extremely high weights to time steps in the time series where the current fluctuates violently or the pressure drops suddenly (i.e., the dot product of Q and K at that moment is large).

[0060] This means that the model can automatically identify and extract the key moment features that lead to the current welding quality result, realizing causal tracing from static results to dynamic processes.

[0061] Feature aggregation and residual connections: To prevent degradation caused by increasing network depth, residual connections and layer normalization are introduced after the multi-head attention output. The final generated As a high-dimensional fusion feature that incorporates temporal dynamic causal information, it not only preserves the global information of the image but also aggregates the process fragment information most relevant to quality, providing high-density information support for multi-task prediction in the subsequent S5 step.

[0062] Multi-task prediction head design: Unify visual features With physical guidance timing characteristics The features are concatenated to obtain a 512-dimensional multimodal fusion feature. This feature is then input into two parallel multilayer perceptrons (MLPs). Classifier header: The structure is Linear(512) 256) ReLU Dropout(0.1) Linear(256 3) Output the predicted probability of the welding quality category.

[0063] Regressor: The structure is the same as the classification head, but it outputs two continuous values ​​in the output layer, which correspond to the normalized predicted values ​​of the melt core diameter (mm) and tensile shear force (N), respectively.

[0064] The training process for collaborative models based on a data-algorithm hybrid strategy is as follows: Figure 6 As shown, Figure 6The text illustrates the SMOTE data balancing, model forward propagation, and hybrid backpropagation process based on focus loss and mean squared error loss: Figure 6 In this process, the original data undergoes process data preprocessing to obtain imbalanced training data, which is then oversampled by SMOTE to generate a balanced training set. This balanced training set is then input into Advanced ViT-HAAN Net (Advanced Vision Transformer Hybrid Attention Alignment Network) to obtain predictions. Loss calculation is performed, specifically using a combination of Focal Loss and MSE Loss (Mean Squared Error Loss). The model is optimized through Backpropagation & Update. The figure also compares Standard Cross Entropy Loss and Focal Loss. Both are based on Model's Predicted Probability and Ground Truth Label. The loss is calculated using the cross-entropy calculation module for Good (easy samples), Explode (medium samples), and Bad (difficult samples). Focal Loss introduces an additional modulation factor to adjust the sample loss weights, thereby solving the problem of imbalance between data categories and difficulty.

[0065] This step employs a hybrid training strategy of "data-level resampling + algorithm-level reweighting." SMOTE increases the diversity of minority class samples at the data layer, while focus loss increases the weight of hard-to-separate samples at the algorithm layer. The synergistic effect of these two approaches generates a gain effect, enabling the fine-grained classification features learned by the model to be transferred and improve the accuracy of regression tasks. This addresses the severe class imbalance problem in resistance spot welding industrial data and improves the model's recognition rate of welding defects (such as incomplete welds and spatter) and the prediction accuracy of physical properties (weld nugget diameter, shear force) through multi-task collaboration. This step specifically includes the following sub-steps: Data hierarchy, based on adaptive SMOTE dynamic balancing: To address the situation where there are far more normal solder joint samples than defective solder joint samples (such as cold solder joints and spatter) in the original training set, a synthetic minority class oversampling technique is implemented at the model input.

[0066] Principle: For each minority class sample in the feature space Calculate its k nearest neighbors. Then randomly select a sample from these k nearest neighbors. ,exist and New synthetic samples are randomly generated on the connection lines. .

[0067] Adaptive strategy: An adaptive neighborhood parameter strategy is adopted. The default setting is k=5; if the number of samples for a certain defect category... Then adaptive adjustment This is to prevent the algorithm from failing due to insufficient samples.

[0068] Function: By generating synthetic samples at the feature level, rather than simply copying them, it increases the diversity of minority class samples, prevents model overfitting, and establishes the data foundation for model training.

[0069] At the algorithm level, a multi-task hybrid loss function is constructed: Construct a hybrid objective function that includes classification loss, regression loss, and invariant regularization loss. Perform end-to-end supervised training on the model: in: Classification loss ( Focal Loss is used: To further address the problem of mining hard samples, Focal Loss is used instead of the traditional cross-entropy loss. Its calculation formula is as follows: Focusing parameters Set to 2.0, balance factor The model is dynamically adjusted based on category frequency. When the model achieves high classification accuracy for a particular sample... At that time, modulation factor Approaching 0 reduces the weight of simple samples, forcing the model to focus on "marginal defect samples" (such as tiny splashes or critical cold welds) with low prediction probabilities during training.

[0070] Regression loss ( Mean squared error loss (MSE) is used: Used to monitor the prediction of melt core diameter and tensile shear force. The calculation formula is: in, The normalized true physical property value. These are the model's predicted values. Weighting coefficients. Set it to 1.0.

[0071] Invariance regularization loss ( Robust constraints are adopted: To eliminate the interference of industrial field shooting angle and lighting changes on prediction, isovariance constraints are introduced.

[0072] Performing geometric transformations (such as flipping or rotating) on ​​the same batch of input images X yields... .

[0073] Computational model and Prediction consistency: in, The KL divergence of the categorical distribution. This is the regression output vector. Weight coefficients. Set it to 0.5.

[0074] Synergistic gain and parameter optimization: Based on the above hybrid strategy, the AdamW optimizer is used for iterative parameter updates.

[0075] Synergistic gain mechanism: SMOTE increases the visibility of the number of defect samples, while the focus increases the learning depth of defect features. The combination of the two produces a synergistic effect, enabling the model to learn fine-grained features that distinguish critical defects (such as subtle thermal trace differences).

[0076] Backpropagation optimization: Experiments show that this fine-grained classification feature, backpropagated through shared encoder networks (such as S3 and S4), significantly enhances the model's sensitivity to physical performance regression tasks. For example, accurate identification of the "splash" feature directly corrects the model's prediction of a sudden drop in "tensile shear force," thereby achieving a dual improvement in classification and regression accuracy.

[0077] Training parameters: Initial learning rate set to The batch size is 4, the training rounds are 60, and mixed precision (AMP) is used to accelerate computation.

[0078] This scheme employs a physical-guided fusion mechanism, which not only integrates multimodal information but also simulates the diagnostic logic of tracing causes from effects. It utilizes post-weld image features to dynamically focus on key moments in the welding process, significantly improving the interpretability and prediction accuracy of the model.

[0079] This scheme adopts a hybrid training strategy with synergistic gains. By combining SMOTE and focus loss, it effectively solves the problem of extreme imbalance in industrial data. Experiments show that this strategy not only significantly improves the defect recall rate, but also significantly reduces the mean absolute error of shear force prediction through feature sharing.

[0080] This scheme introduces invariance loss to make the model robust to non-essential disturbances such as shooting angle and lighting, while simultaneously achieving the output of quality classification and physical performance quantification, meeting the needs of comprehensive monitoring in industrial sites.

[0081] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for quality inspection of resistance spot welding based on physical-guided multimodal methods, characterized in that, The method steps include: S1. Acquire multimodal sensor data of the resistance spot weld to be detected and preprocess it. The multimodal sensor data includes visual modal data and temporal modal data. S2. Construct a multi-view image encoder group, input visual modal data into the multi-view image encoder group, output feature extraction results, fuse the feature extraction results, and thus generate unified visual features. S3. Construct a time encoder, input the preprocessed time modal data into the time encoder, and output a time feature sequence; S4. Construct a hierarchical attention aggregation network; use unified visual features as query vectors and temporal feature sequences as key and value vectors, input them into the hierarchical attention aggregation network, and output physically guided multimodal fusion features; S5. The multi-task prediction network is used to process the physical guidance multimodal fusion features, and the welding quality classification results and physical performance regression prediction values ​​of resistance spot welding are output as the resistance spot welding quality detection results.

2. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 1, characterized in that, The multi-view image encoder group, temporal encoder, hierarchical attention aggregation network, and multi-task prediction network together constitute the prediction model; The method also includes a model training process: during training, a training dataset is established using sample multimodal sensor data, and the training dataset is used as input to execute steps S1-S5. The prediction model is trained end-to-end using a hybrid loss function to obtain the trained prediction model; during prediction, the data to be detected is input into the trained prediction model, and the welding quality classification result and physical performance regression prediction value are output.

3. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 2, characterized in that, The hybrid loss function includes classification loss, regression loss, and invariant regularization loss; The classification loss uses focus loss, which reduces the weight of easily classified samples by using the focus parameter γ, and focuses on difficult-to-classify samples, including cold solder joints or spatter. The regression loss uses mean squared error loss, which is used to supervise the prediction of weld nugget diameter and tensile shear force.

4. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 3, characterized in that, When the training dataset is executed in step S1, the preprocessing also includes data balancing processing of the training dataset based on a hybrid resampling strategy. The specific process is as follows: using the synthetic minority class oversampling technique with adaptive neighborhood parameters, the number of samples of each class in the training dataset is counted to determine the number of samples of the least number class, and the K nearest neighbor parameter is set according to the number of samples of the least number class; then, based on the K nearest neighbor parameter, new synthetic samples are generated in the feature space through linear interpolation.

5. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 1, characterized in that, The visual modal data includes infrared thermal imaging images, RGB images of the front and back of the solder joint; the temporal modal data includes parameters such as current, voltage, pressure, electrode displacement, welding time, and plate thickness during the welding process.

6. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 5, characterized in that, The preprocessing includes: resizing and normalizing the image data and performing Z-score normalization on the time-series parameters.

7. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 6, characterized in that, The multi-view image encoder group in S2 includes three pre-trained multi-view image encoders with independent parameters; the three pre-trained multi-view image encoders are used to extract infrared thermal imaging images, solder joint front RGB images and solder joint back RGB images, respectively. The feature extraction results in S2 include infrared features, frontal features, and backal features; The specific process of fusing the feature extraction results to generate unified visual features in S2 includes: A three-path parallel inter-view cyclic cross-attention mechanism is used to process infrared features, frontal features, and back features. The three paths include: The infrared-dominated path uses infrared features as the query vector and the concatenation of front and back features as the key and value vectors. The front-dominant path uses front features as the query vector and the concatenation of infrared and back features as the key and value vectors. The back-side dominant path uses back-side features as the query vector and the concatenation of front-side features and infrared features as the key vector and value vector. Finally, the outputs of these three paths are concatenated, and the concatenation result is fused through a fully connected layer to generate a unified visual feature that includes complementary information on thermal distribution and surface morphology.

8. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 1, characterized in that, In S3, the specific process of generating the time-series feature sequence includes: in the time encoder, using a bidirectional long short-term memory network to extract bidirectional dynamic features of the time-series modal data, and injecting sinusoidal position codes as time position information into the extraction result, and the injected sequence is the time-series feature sequence.

9. The method for detecting resistance spot welding quality based on physical-guided multimodal methods according to claim 1, characterized in that, The hierarchical attention aggregation network in S4 specifically performs the following operations: constructing an asymmetric attention mechanism, using the unified visual features as the query vector and the temporal feature sequence as the key vector and value vector; mapping the query vector, key vector, and value vector to multiple attention subspaces through a linear projection matrix, calculating the attention weights of the unified visual features and the features at each time step in the temporal feature sequence within each subspace; weighting and aggregating the temporal features according to the attention weights, and concatenating the outputs of multiple attention heads to obtain physically guided multimodal fusion features.

10. A physical-guided multimodal resistance spot welding quality inspection system, characterized in that, The system operates using a physical-guided multimodal resistance spot welding quality inspection method as described in any one of claims 1-9. The system includes a data acquisition module, a data preprocessing module, a visual feature processing module, a temporal feature processing module, a physical-guided fusion module, and a multi-task prediction module. The data acquisition module is used to collect multimodal sensor data for resistance spot welding; The data preprocessing module performs preprocessing operations on the multimodal sensor data; The visual feature processing module includes a multi-view image encoder group, which is used to process visual modal data and output feature extraction results, and fuse the feature extraction results to generate unified visual features. The temporal feature processing module includes a temporal encoder, which is used to process the preprocessed temporal modal data and output a temporal feature sequence; The physical guidance fusion module includes a hierarchical attention aggregation network, which processes unified visual features as query vectors and temporal feature sequences as key and value vectors to output physical guidance multimodal fusion features. The multi-task prediction module includes a multi-task prediction network, which processes the physical guidance multimodal fusion features and outputs the welding quality classification results and physical performance regression prediction values ​​of resistance spot welding.

Images