Resistance spot welding quality prediction and optimization method based on multi-modal data and cross-modal attention network fusion

Abstract

The invention relates to a resistance spot welding quality prediction and optimization method based on multi-modal data and cross-modal attention network fusion, and the method comprises the steps: collecting resistance spot welding data which comprises a technological parameter sequence, a visual image and a thermal imaging image; inputting the resistance spot welding data into a resistance spot welding quality prediction model to obtain a resistance spot welding quality prediction result, and determining an optimal process parameter combination according to the resistance spot welding quality prediction result; an image feature extraction module in the resistance spot welding quality prediction model is used for conducting feature extraction on the visual image and the thermal imaging image to generate visual image features and thermal imaging features, a process parameter feature extraction module is used for conducting feature extraction on the process parameter sequence to generate parameter features, and a resistance spot welding quality prediction model is obtained. And a feature fusion module based on a cross-modal attention mechanism performs feature fusion on the visual image features, the thermal imaging features and the parameter features to generate fusion features.

Description

Technical Field

[0001] This invention relates to the field of resistance spot welding quality technology, and in particular to a method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks. Background Technology

[0002] In the process of modern high-end manufacturing accelerating towards intelligence and precision, metal joining technology, as a key foundation for constructing complex equipment structures, directly impacts product reliability through its quality control level. Resistance spot welding, as one of the most widely used solid-state joining processes, dominates in fields such as automotive manufacturing and aerospace due to its advantages including no filler material, minimal joint deformation, and strong adaptability to automation. However, precise control of welding quality during resistance spot welding remains a critical challenge for the industry. The mechanical properties and microstructural integrity of the weld joints directly affect the safety and service life of equipment under complex operating conditions, necessitating innovative technologies to overcome the bottlenecks of traditional detection and prediction methods.

[0003] The mainstream testing methods in the field of metal joining quality control can be divided into two categories: destructive testing and non-destructive testing. While destructive testing can obtain accurate quality data of weld joints through mechanical experiments and metallographic analysis, it has limitations such as high sample consumption, long testing cycles, and difficulty in achieving full-process monitoring. Non-destructive testing achieves non-contact online testing through physical means such as sound waves and magnetic fields, but it has significant shortcomings in areas such as identifying minute defects and quantitatively evaluating the mechanical properties of joints. With the development of industrial IoT and intelligent manufacturing technologies, quality prediction methods based on process parameters and process data have gradually become a research hotspot. These methods construct quality prediction models by real-time acquisition of multi-source data such as electrical signals, temperature fields, and image information during the welding process to achieve early prediction of welding quality and process optimization, which is of great significance for improving the controllability of the metal joining process.

[0004] Existing technical solutions mainly fall into two categories: mechanism-driven and data-driven. Mechanism-driven methods, based on heat transfer, metallurgical dynamics, and materials mechanics theories, construct physical models of the welding process, offering high prediction accuracy and interpretability under simple operating conditions. However, when faced with strong nonlinear coupling problems in real-world industrial scenarios such as welding dissimilar materials and complex multilayer plate structures, the adaptability and accuracy of traditional mechanism models significantly decrease. Data-driven methods, on the other hand, rely on machine learning and deep learning technologies to mine potential correlations between process parameters and quality indicators from massive amounts of historical data. They can effectively handle high-dimensional heterogeneous data and exhibit strong generalization capabilities under complex operating conditions, gradually becoming the mainstream trend in industrial applications. Nevertheless, existing technical solutions still have many problems that urgently need to be addressed:

[0005] (1) Insufficient ability to collaboratively model multi-source data. Most existing studies rely on single-modal data to build models. Among them, existing technical solutions mainly focus on process signal analysis such as dynamic resistance, while deep learning models focus on defect detection of image data. Such single-modal modeling methods are difficult to capture the complex mapping relationship under the coupling of multiple variables, thus limiting the improvement of prediction accuracy.

[0006] (2) Limited ability to analyze high-dimensional nonlinear features. Existing technical solutions rely on manually designed features. When processing high-dimensional, noisy, and nonlinear data in the welding process, they are not capable of analyzing the implicit correlations between parameters and are difficult to adapt to the generalization requirements under complex working conditions.

[0007] (3) Weak cross-modal feature adaptive fusion capability. Existing technical solutions lack effective cross-modal correlation modeling mechanisms, making it difficult to dynamically mine complementary features between image modalities and parameter modalities. Key quality-related information is weakened or omitted during cross-modal transmission, resulting in insufficient collaborative representation capability of multi-source features. Summary of the Invention

[0008] To address the problems of existing technologies, this invention aims to provide a method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks. This method utilizes Industrial Internet of Things (IIoT) technology to collect multimodal data in real time, including dynamic resistance curves, infrared images of the weld nugget, and temperature field distribution. A cross-modal attention mechanism is used to establish correlation mappings between different modalities, enabling efficient utilization of complementary information from multiple data sources. This overcomes the limitations of high-dimensional feature analysis capabilities by constructing a hybrid feature extraction network based on BiLSTM and CNN. Multi-level convolutional operations of CNN capture multi-scale spatial features of image modalities, while BiLSTM analyzes the dependencies between various industrial process parameters. Furthermore, a bidirectional cross-modal attention module is constructed to dynamically fuse image-guided parameter attention and parameter-guided image attention. The model is guided to focus on key parameter features by analyzing defect regions in the weld nugget image, while simultaneously locating important regions in the image based on process parameters, resulting in complementary enhancement of cross-modal features.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks includes:

[0011] Collect resistance spot welding data, which includes: process parameter sequence, visual images and thermal imaging images;

[0012] The resistance spot welding data is input into the resistance spot welding quality prediction model to obtain the resistance spot welding quality prediction result. Based on the resistance spot welding quality prediction result, the optimal combination of process parameters is determined. The resistance spot welding quality prediction model is obtained by optimizing the model hyperparameters using a training set combined with a loss function. The training set includes: the original resistance spot welding data.

[0013] The image feature extraction module in the resistance spot welding quality prediction model extracts features from the visual image and the thermal imaging image respectively to generate visual image features and thermal imaging features. The process parameter feature extraction module extracts features from the process parameter sequence to generate parameter features. The feature fusion module based on the cross-modal attention mechanism fuses the visual image features, the thermal imaging features and the parameter features to generate fused features.

[0014] Optionally, obtaining the training set includes:

[0015] The original resistance spot welding data is acquired. The image data of the original resistance spot welding data is enhanced with a contrast-limited adaptive histogram equalization algorithm. The image data is divided into non-overlapping sub-blocks. Histogram equalization is performed independently on each sub-block. The noise amplification is avoided by limiting the histogram clipping threshold. Finally, bilinear interpolation is used to eliminate the boundary artifacts of the sub-blocks. Guided filtering and sharpening operations are used to fuse image details. Furthermore, mosaic random stitching technology is introduced to expand the dataset.

[0016] The raw resistance spot welding data after image data processing is subjected to box plot detection and outlier removal. Then, Gaussian filtering algorithm is used to smooth the two quality indicators of tensile test strength and weld nugget diameter. All data is then uniformly scaled to the target range to obtain the training set.

[0017] Optionally, the image feature extraction module includes: multiple image feature extraction units;

[0018] Each image feature extraction unit includes: a convolutional layer, a pooling layer, and a fully connected layer.

[0019] Optionally, generating the visual image features and the thermal imaging features includes:

[0020] After the visual image and the thermal image are sequentially passed through the convolutional layer for effective filling convolution, the feature distribution is stabilized by batch normalization, and a nonlinear transformation is introduced by the ReLU activation function. Then, the feature is reduced in dimensionality by the pooling layer to generate a three-dimensional feature map.

[0021] The three-dimensional feature map is transformed into a one-dimensional vector, which is then mapped to the feature space of the target dimension through the fully connected layer. ReLU activation is then used to enhance the non-linear expressive power of the features, and the output result is obtained.

[0022] The output results corresponding to each image feature extraction unit are stabilized by layer normalization to generate the visual image features and the thermal imaging features.

[0023] Optionally, generating the parameter features includes:

[0024] The process parameter sequence is input into the process parameter feature extraction module. The LSTM unit in the process parameter feature extraction module processes the forward and reverse information of the time series respectively, and fuses the bidirectional hidden states to capture the long-term dependence and short-term fluctuations in the sequence, thereby obtaining the parameter features.

[0025] Optionally, generating the fusion feature includes:

[0026] The visible image features and the thermal imaging features are used as query pairs and key-value pairs, respectively. The feature weights are dynamically adjusted through a cross-modal attention scoring matrix, and the first image fusion feature is generated by weighted summation.

[0027] The visible image features and the thermal imaging features are used as key-value pairs and query pairs, respectively. The feature weights are dynamically adjusted by calculating the cross-modal attention score matrix, and a second image fusion feature is generated by weighted summation.

[0028] The first image fusion feature and the second image fusion feature are weighted and fused to obtain the final image fusion feature;

[0029] The final image fusion features are reshaped into a two-dimensional spatial feature map. The parameter features are mapped to attention query vectors through linear transformation. At the same time, the spatial features of the two-dimensional spatial feature map are mapped to key vectors and value vectors. The parameter and image attention matrix are calculated, and the parameter-guided image features are obtained by weighted summation. The fusion features are then generated by combining the parameter features.

[0030] Optionally, the feature weights can be dynamically adjusted by calculating the cross-modal attention score matrix:

[0031] ;

[0032] in, , For learnable weight matrix, The dimension of the key vector.

[0033] Optionally, obtaining the final image fusion features includes:

[0034] ;

[0035] in, For the final image fusion features, For visual image features, For the second image fusion feature, For thermal imaging features, This is the first image fusion feature.

[0036] To achieve the above objectives, the present invention also provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method.

[0037] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor of the steps of the method.

[0038] The beneficial effects of this invention are as follows:

[0039] This invention possesses the ability to efficiently fuse multi-source, multi-modal data, and can process diverse inputs such as process parameter data, images, and text. Through deep integration of multi-dimensional information, it can accurately capture more complex behavioral patterns, effectively improving both the accuracy and comprehensiveness of quality prediction.

[0040] This invention has the ability to dynamically allocate feature weights, eliminating the need for manual feature design, effectively extracting key influencing factors, and maintaining stable prediction accuracy even under varying operating conditions.

[0041] This invention integrates advanced quality index optimization algorithms, which can further realize dynamic optimization of quality indexes and reverse iteration of process parameters based on the trained prediction model, thereby achieving a complete and comprehensive process parameter analysis process.

[0042] This invention features excellent scalability and integrability, enabling easy integration with existing data analysis systems. This makes it easier to deploy and apply quality prediction and optimization systems in real-world industrial environments. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1This is a flowchart of the resistance spot welding quality prediction and optimization method based on the fusion of multimodal data and cross-modal attention network according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the resistance spot welding quality prediction model according to an embodiment of the present invention;

[0046] Figure 3 This is a comparison image before and after image enhancement according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the image feature extraction module according to an embodiment of the present invention;

[0048] Figure 5 The following are trend graphs of the predicted and actual values ​​of each model in the embodiments of the present invention: (a) is a trend graph of the predicted and actual value curves of MBNet, (b) is a trend graph of the predicted and actual value curves of BiLSTM, (c) is a trend graph of the predicted and actual value curves of CNN, and (d) is a trend graph of the predicted and actual value curves of Random Forest (RF).

[0049] Figure 6 This diagram illustrates the prediction performance of different models in this invention on a test set. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1As shown, this embodiment discloses a method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks. The method includes: collecting resistance spot welding data, which includes process parameter sequences, visual images, and thermal imaging images; inputting the resistance spot welding data into a resistance spot welding quality prediction model to obtain resistance spot welding quality prediction results; the resistance spot welding quality prediction model is obtained by optimizing model hyperparameters using a training set combined with a loss function; the training set includes the original resistance spot welding data; using the image feature extraction module in the resistance spot welding quality prediction model to extract features from the visual images and thermal imaging images respectively, generating visual image features and thermal imaging features; using the process parameter feature extraction module to extract features from the process parameter sequences, generating parameter features; and using a feature fusion module based on a cross-modal attention mechanism to fuse the visual image features, thermal imaging features, and parameter features to generate fused features.

[0053] Specifically, this embodiment discloses a method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks, including:

[0054] S1 First, by synchronously collecting process parameter sequences (time series data such as welding current and electrode pressure), visual and thermal imaging images (geometric morphology and temperature field distribution of the weld area), and optional physical field data during the resistance spot welding process, a multi-source heterogeneous dataset is constructed and data preprocessing is performed.

[0055] To address the shortcomings of multi-source data collaborative modeling and limited ability to analyze high-dimensional features, S2 constructs a resistance spot welding quality prediction model (MBNet) based on the fusion of multimodal data and cross-modal attention networks. Figure 2 As shown, the network framework mainly consists of three parts: an image feature extraction module based on a convolutional neural network, a process parameter feature extraction module based on a bidirectional long short-term memory network, and a feature fusion module based on a cross-modal attention mechanism.

[0056] During the training phase, S3 utilizes a convolutional neural network to extract multi-level features from preprocessed visual and thermal imaging images. It captures basic geometric features such as weld point edge texture and high-level semantic spatial features such as weld nugget shape and energy distribution through convolutional kernels of different scales. At the same time, it employs a bidirectional long short-term memory network to process dynamic process parameter sequences. Through forward and backward units, it captures the long-term cumulative effect and short-term fluctuation dependence between parameters, and outputs the spatial feature matrix of the image mode and the temporal feature vector of the parameter mode, respectively.

[0057] S4 flattens the spatial features output by CNN and the parametric features output by BiLSTM, and calculates the interaction weights of the two types of features by scaling the dot product attention formula to construct a cross-modal attention matrix. Based on the attention matrix, the spatial features and process parameter features are dynamically weighted and fused so that the fused features retain both the spatial details of the image modality (such as surface defects of the solder joint) and the long-term and short-term dependencies of the parameter modality, thereby achieving adaptive complementary enhancement of multi-source heterogeneous features.

[0058] In the model optimization phase, S5 calculates the loss between the predicted value and the true value, uses the Keras Tuner search algorithm to optimize the model hyperparameters, and completes the training of the overall model after multiple iterations.

[0059] During the testing phase, S6 uses the test dataset as input to the trained model for prediction and evaluates the model performance through multiple performance metrics.

[0060] In the parameter optimization phase, S7 uses optimization algorithms such as genetic algorithms to process real sample data and continuously iterates and updates the parameter combinations. Simultaneously, the iterated parameters are input into the trained prediction model to obtain the predicted quality index. Constraint values ​​are calculated, and it is determined whether the optimization should end. If the convergence requirement is met, a representative solution is output through a clustering algorithm to obtain the optimal combination of process parameters.

[0061] Further, obtaining the training set includes: acquiring raw resistance spot welding data; enhancing local image contrast using a contrast-limited adaptive histogram equalization algorithm on the image data of the raw resistance spot welding data; dividing the image data into non-overlapping sub-blocks; performing histogram equalization independently on each sub-block; avoiding noise amplification by limiting the histogram clipping threshold; finally, eliminating boundary artifacts of the sub-blocks using bilinear interpolation; fusing image details using guided filtering and sharpening operations; and further introducing mosaic random stitching technology to expand the dataset; using box plot detection to remove outliers from the raw resistance spot welding data after image data processing; then using a Gaussian filtering algorithm to smooth the two quality indicators of tensile test strength and weld nugget diameter; and uniformly scaling all data to the target interval to obtain the training set.

[0062] Specifically, in step S1, the dataset originates from an actual industrial environment equipment acquisition platform. During actual acquisition, factors such as fluctuations in process parameters and equipment performance degradation may cause anomalies in quality indicator measurements; therefore, preprocessing of the raw data is necessary. This method uses box plots to detect and remove outliers, and then employs a Gaussian filtering algorithm to smooth the tensile test strength and melt diameter quality indicators. Assuming the Gaussian kernel window size is 2k+1, the center position is i, and the standard deviation is σ, then the Gaussian weight at position j is:

[0063] (1);

[0064] in, The Gaussian weights at position j are used, and the smoothing degree is controlled by adjusting σ. This represents the j-th position within the current Gaussian window. The center position of the Gaussian window. This is half the width of the window;

[0065] The smoothed sequence values ​​are a weighted average: (2).

[0066] Effectively reduce data noise interference. To eliminate the difference in units between variables and accelerate the model training convergence process, the min-max normalization technique is further adopted to uniformly scale all data to the [0,1] interval.

[0067] To enhance the stability of feature extraction and alleviate the scarcity of defective image samples for image data, this invention constructs a multi-stage image enhancement strategy. The comparison before and after image enhancement is shown below. Figure 3 As shown in the diagram. Specifically, this invention enhances local image contrast through a contrast-limited adaptive histogram equalization algorithm. The image is divided into non-overlapping sub-blocks, and histogram equalization is performed independently on each sub-block to enhance local contrast. Noise amplification is avoided by limiting the histogram cropping threshold, and bilinear interpolation is used to eliminate sub-block boundary artifacts, highlighting key region features. Furthermore, guided filtering and sharpening operations are used to fuse image details and enhance edge information. Finally, a mosaic random stitching technique is introduced to effectively expand the diversity of image data and improve the model's generalization ability.

[0068] Furthermore, in step S1, when collecting data from multiple modalities, the steps also include: preprocessing the collected data, that is, cleaning the collected dataset and normalizing the resulting process parameter dataset to the range [0,1], while using image enhancement technology to process the image data to make it suitable for network input.

[0069] Furthermore, the image feature extraction module includes: multiple image feature extraction units; each image feature extraction unit includes: a convolutional layer, a pooling layer, and a fully connected layer.

[0070] Generating the visual image features and the thermal imaging features includes: sequentially passing the visual image and the thermal imaging image through the convolutional layer for effective padding convolution operations, stabilizing the feature distribution through batch normalization, introducing a nonlinear transformation using the ReLU activation function, and then performing feature dimensionality reduction through the pooling layer to generate a three-dimensional feature map; converting the three-dimensional feature map into a one-dimensional vector, mapping it to the feature space of the target dimension through the fully connected layer, and enhancing the nonlinear expressive power of the features through ReLU activation to obtain the output result; stabilizing the distribution of output features for the output result corresponding to each image feature extraction unit using layer normalization to generate the visual image features and the thermal imaging features.

[0071] Specifically, further, the image feature extraction module based on convolutional neural networks constructed in step S2 adopts a structure of multiple convolutional layers and pooling layers, as shown in the following figure. Figure 4 As shown.

[0072] Furthermore, step S3 specifically includes the following steps:

[0073] The image processing module consists of nine layers, including three convolutional layers, three pooling layers, and three fully connected layers. Upon input, the image is resized to a uniform 64x64 size using a resize operation, providing a consistent tensor input for subsequent convolutional operations. The parameters in each layer are not fixed and can be adjusted based on the task and input characteristics. In the first convolutional layer, the number of filters is adaptively adjusted with a stride of 8, ranging from 8 to 32, while the kernel size is selected from {5, 7, 8}. Assume the input feature map is... The convolution kernel is Where k is the kernel size and Cout is the number of channels. Then each element of the output feature map Y is:

[0074] (3);

[0075] in, For feature map In spatial location Output channels The element value at that position, It is a linear rectified activation function. For batch normalization operations, The size of the convolution kernel. , It is the spatial location index inside the convolution kernel ( q corresponds to the high dimension of the kernel, and q corresponds to the wide dimension of the kernel, with values ​​ranging from 0 to k-1. This represents the number of channels in the input feature map. This refers to the channel index of the input feature map. Given the element value of the input feature map X at position (i+p, j+q) and channel d. Here are the convolution kernel parameters, corresponding to the weights of the convolution kernel at position (p,q) and from input channel d to output channel c. The bias parameter is the one corresponding to output channel c.

[0076] After effective padding convolution with a stride of 1, batch normalization stabilizes the feature distribution, ReLU activation introduces a non-linear transformation, and then 2×2 max pooling is used to achieve feature dimensionality reduction and spatial invariance enhancement, generating the following feature map:

[0077] (4);

[0078] in, To output the element value of the feature map at spatial location (i,j) and channel c. For max pooling operation, , , , The elements of the input feature map X for the pooling operation correspond to the four spatial locations covered by the 2×2 pooling window.

[0079] Similarly, the number of filters in the second convolutional block is adjusted to the range of 32 to 64, with a stride of 16, and the kernel size is limited to {5, 7}. The number of filters in the third convolutional block is increased to 64 to 128, with a stride of 32, and the kernel size is reduced to {3, 5}. After feature extraction, the 3D feature map is transformed into a 1D vector through Flatten, mapped to a 128 to 256-dimensional feature space through a fully connected layer with a stride of 64, and ReLU activation is used to enhance the non-linear expressive power of the features. Finally, layer normalization is used to stabilize the distribution of the output features. That is, the two types of image data are input separately, and then passed through the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer, the third convolutional layer, the third pooling layer, and finally through three fully connected layers and layer normalization to output the features.

[0080] Further, generating the parameter features includes: inputting the process parameter sequence into the process parameter feature extraction module, using the LSTM unit in the process parameter feature extraction module to process the forward and reverse information of the time series respectively, and fusing the bidirectional hidden states to capture long-term dependencies and short-term fluctuations in the sequence, thereby obtaining the parameter features.

[0081] Specifically, the bidirectional Long Short-Term Memory (LSTM) network is an extension of the LSTM network. It uses two LSTM units, one forward and one backward, to process the forward (past → future) and reverse (future → past) information of the time series, respectively, ultimately fusing the bidirectional hidden states to capture long-term dependencies and short-term fluctuations in the sequence. Its core advantage lies in solving the gradient vanishing problem of traditional RNNs and bidirectionally modeling the contextual dependencies of time series, making it suitable for data with dynamic temporal characteristics, such as welding process parameters.

[0082] Bidirectional Long Short-Term Memory (LSTM) networks consist of forward LSTM and backward LSTM, processing sequences x1, x2, ..., x T hour:

[0083] Forward hidden state calculation:

[0084] (5);

[0085] in, For the processed positive information, For long short-term memory networks used for forward sequence modeling, The input data at time t in the sequence. This is the positive hidden state at time t-1, representing the historical information of the sequence up to time t-1;

[0086] Backward hidden state calculation:

[0087] (6);

[0088] in, This is the processed reverse information. For Long Short-Term Memory network modules used for reverse sequence modeling, This is the reverse hidden state at time t+1, representing the subsequent information of the sequence from the end to time t+1.

[0089] After fusion of the two hidden states, we get: .

[0090] Using the above formula, the bidirectional long short-term memory network can effectively capture the temporal dependence of dynamic parameters during resistance spot welding, providing a deep temporal representation for subsequent cross-modal feature fusion.

[0091] Further, generating the fusion feature includes: using the visual image feature and the thermal imaging feature as query pairs and key-value pairs respectively, dynamically adjusting the feature weights through a cross-modal attention score matrix, and generating a first image fusion feature through weighted summation; using the visual image feature and the thermal imaging feature as key-value pairs and query pairs respectively, dynamically adjusting the feature weights through a cross-modal attention score matrix, and generating a second image fusion feature through weighted summation; performing weighted fusion of the first image fusion feature and the second image fusion feature to obtain the final image fusion feature; reshaping the final image fusion feature into a two-dimensional spatial feature map, mapping the parameter features to attention query vectors through linear transformation, and mapping the spatial features of the two-dimensional spatial feature map to key vectors and value vectors, calculating the parameter and image attention matrix, and obtaining parameter-guided image features through weighted summation, and further combining the parameter features to generate the fusion feature.

[0092] Specifically, through the above formula, the bidirectional long short-term memory network can effectively capture the temporal dependence of dynamic parameters in the resistance spot welding process, providing a deep representation of the time dimension for subsequent cross-modal feature fusion.

[0093] Furthermore, in step S4, a cross-modal attention mechanism is designed to fuse image features and process parameter features. This invention employs a cross-attention mechanism and a parameter-guided attention mechanism to achieve efficient fusion of multi-source information. While both visual images and thermal imaging can reflect basic information such as the location of the weld nugget, they still have different focuses. This complementarity requires the fusion process to retain specific features while suppressing redundant information. Traditional stitching or fixed-weighted fusion is difficult to dynamically adapt to the feature importance under different welding scenarios, while the cross-attention mechanism achieves adaptive feature optimization through dynamic weight allocation and cross-modal association mining. The core of the cross-attention mechanism lies in establishing inter-modal association mapping through a query-key-value triple structure. In the image feature fusion stage, visual image features are... With thermal imaging features Using query and key-value pairs respectively, cross-modal attention score matrices are calculated. To achieve dynamic adjustment of feature weights:

[0094] (7);

[0095] in , For learnable weight matrix, This is the dimension of the key vector. This matrix represents... Each feature vector in The correlation strength of all feature vectors in the dataset is used to generate a fused feature by weighted summation. ,in, This is the output weight.

[0096] Similarly, by capturing the hidden correlation between the melt nucleus location and the temperature field through bidirectional attention flow, feature discriminativeness is enhanced. Using thermal imaging as the query and visual images as the key-value pairs, [the following can be generated]. Final image fusion features:

[0097] (8);

[0098] After image feature fusion is completed, a process parameter-guided attention mechanism is introduced to utilize the parameter features extracted by BiLSTM. As a query, it guides the model to focus on the image region most relevant to the current process parameters. Specifically, it uses the fused image features... Reconstructing it into a two-dimensional spatial feature map, and then transforming it linearly... Mapped to attention query vector, and simultaneously Spatial feature mapping is performed as key vectors and value vectors. Parameters and image attention matrices are calculated, and parameter-guided image features are obtained through weighted summation. Through a cascaded design of cross-attention and parameter-guided attention, the model can dynamically filter redundant information and enhance complementary features among image modalities, and also leverage the physical priors of process parameters to guide feature selection, ultimately generating fused features:

[0099] (9);

[0100] Furthermore, in step S5, the optimization phase, Keras Tuner is used to automate the hyperparameter search. The mathematical definition of the learning rate using logarithmic scaling is:

[0101] learning_rate LogUniform (10);

[0102] in, This is the lower limit of sampling for a log-uniform distribution. This represents the upper limit of sampling for a logarithmically uniform distribution;

[0103] Its probability density function is used to provide a theoretical sampling basis for the automated search of hyperparameters (which here match the learning rate mentioned earlier):

[0104] (11);

[0105] in, To model hyperparameter vectors.

[0106] The mapping relationship between the hyperparameter vector θ and the validation set loss L(θ) is modeled using a Gaussian Process (GP), and the next search point is selected using the expected improvement function.

[0107] (12);

[0108] in, This is the current optimal loss on the validation set. The optimization objective of model training is to minimize the mean squared error on the validation set.

[0109] MSE (13);

[0110] in, It is the true welding quality index of the i-th verification sample. These are model predictions. The total number of samples in the validation set.

[0111] The validation set loss is calculated by substituting the predicted values ​​generated by the trained model on the validation set input and the true labels of the validation set into the loss function MSE of the corresponding task.

[0112] The auxiliary evaluation metric is used to measure the mean absolute deviation between the predicted and actual values, enhancing the robustness of the loss function. It is defined as follows: (14).

[0113] Furthermore, in step S6, during the testing phase, after the proposed method has been trained, the model will input the test set into the trained model to generate predicted values.

[0114] Furthermore, in step S7, the parameter optimization stage, genetic algorithms and clustering algorithms were used to iterate and extract representative solutions for key parameters. Specifically, when designing the genetic algorithm, the optimization objective and constraints were first defined, and a grid search algorithm was used to automatically update the genetic algorithm parameters. After a certain number of iterations, the clustering algorithm was used to cluster the parameter combinations, ensuring that the quality influence patterns within each cluster were consistent. The cluster center was selected as the representative solution, i.e., the optimal parameters in the current scenario. This represents the trained and tested model, which first outputs the predicted welding quality indicators. These predictions provide a basis for evaluating the quality indicators for subsequent clustering of process parameters. The genetic algorithm optimizes parameters with the goal of "improving the predicted quality indicators." After the optimized process parameters are verified through actual processing, the corresponding processing data (visible images, thermal images, process parameters) are fed back to the model training process as new samples to iteratively update the model, ultimately forming a closed-loop technical system of "model prediction → parameter optimization → processing verification → model iteration."

[0115] This embodiment also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method.

[0116] This embodiment also provides a computer-readable storage medium storing a computer program, the computer program being executed by a processor of the steps of the method.

[0117] The method proposed in this invention was systematically validated on the Resistance Spot Welding Insights dataset. This dataset, constructed around the resistance spot welding process, was collected using a Chicago Electric 61205 mini resistance welding machine, encompassing 495 samples. The dataset contains rich information dimensions, including key variables such as pressure, welding time, electrode angle, electrode force, current, and material thickness at the process parameter level; visual and thermal images at the image level; and tensile strength (PT) and weld nugget diameter (ND) as core evaluation parameters for quality indicators. During the experiment, the dataset was divided into training and testing sets in a 4:1 ratio to ensure the effectiveness of model training and evaluation. To quantitatively evaluate the performance of different models, this invention uses mean absolute error (MAE), root mean square error (RMSE), and coefficient of determination (R²). 2 () is used as the core evaluation indicator.

[0118] Method validation: This invention was compared with several single-source data-driven models, including BiLSTM and Random Forest for processing process parameters, and CNN for processing image data. The comparison results are shown in Table 1.

[0119] Table 1 compares the performance of the proposed method with several single-source data models.

[0120]

[0121] This invention shows that the average MAE of CNNs reaches 0.3001, and the average R... 2 With an R value of only 0.0619, the overall fitting effect is weak. While BiLSTM can perform basic predictions, there is still significant room for improvement in accuracy and fitting ability. Random Forest performs well in tensile strength prediction, but its R value is low in the melt pool diameter prediction task. 2The average performance (MAE) is only 0.7386, indicating a performance imbalance across tasks. In contrast, MBNet, by fusing complementary information from images and process parameters, achieves an average MAE of 0.0663, approximately 27% lower than Random Forest. It also maintains good prediction performance across different tasks, including tensile strength and melt pool diameter, achieving an average R-value of 0.9227. 2 This fully demonstrates that the multi-source fusion strategy can effectively overcome the limitations of single-modal data, achieving more accurate and robust performance in predicting complex welding process parameters. The following line graphs of predicted and actual values ​​will further validate this conclusion from a trend-matching perspective.

[0122] The trends of the predicted values ​​and the actual values ​​for each model are as follows: Figure 5 As shown. Figure 5 (a) Visually demonstrates that MBNet's predicted curves and actual curves highly overlap in tensile tests and melt pool diameter tasks, indicating its good prediction accuracy and stability. Meanwhile... Figure 5 In (b), although BiLSTM can capture the general trend of data changes, in the range where the true value fluctuates drastically, such as the part where the value rises and falls sharply in the tensile test, the predicted curve deviates significantly from the true curve, reflecting the insufficient ability to fit the dynamic process. Figure 5 (c) The prediction curve of the CNN tends to be flat overall. It cannot reproduce the peak features of the true value, nor can it track subtle fluctuations. This shows that CNNs that rely solely on image data are difficult to effectively model the dynamic relationship of welding parameters. Figure 5 While the predicted curves of the Random Forest (RF) in (d) overlap with the actual values ​​in some sections, significant deviations occur at key fluctuation points in tasks such as the molten pool diameter, further confirming the limitations of single-source models in cross-task adaptability. In summary, the line graphs, from a trend-matching perspective, intuitively verify that the multi-source fusion MBNet exhibits superior accuracy and robustness compared to single-source models in predicting complex welding process parameters.

[0123] Figure 6The predictive performance of different models on the test set is presented. Experimental results show that the proposed method exhibits significant advantages in prediction accuracy and goodness of fit. In the tensile strength prediction task, the MAE value is as low as around 0.05, indicating that the average absolute deviation between the predicted result and the true value is at a low level. Meanwhile, the R² index reaches above 0.95, indicating that the model has a good fit. In the prediction of the melt core diameter, an R² value above 0.88 is also achieved. Compared with the comparative methods, this method achieves the smallest MAE and RMSE in both quality index predictions, while maintaining the highest R², demonstrating excellent overall predictive performance. From the average index, the proposed method has an average R² of 0.9227, representing a 4.12%-5.84% improvement compared to the comparative models. The average MAE value is 0.0663, with a maximum reduction of 30.94% compared to other models, fully validating the effectiveness of combining the image feature extraction capability of the LeNet architecture with the sequence dependency modeling capability of BiLSTM.

[0124] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

[0125] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention networks, characterized in that, include: Collect resistance spot welding data, which includes: process parameter sequence, visual images and thermal imaging images; The resistance spot welding data is input into the resistance spot welding quality prediction model to obtain the resistance spot welding quality prediction result. Based on the resistance spot welding quality prediction result, the optimal combination of process parameters is determined. The resistance spot welding quality prediction model is obtained by optimizing the model hyperparameters using a training set combined with a loss function. The training set includes: the original resistance spot welding data. The image feature extraction module in the resistance spot welding quality prediction model extracts features from the visual image and the thermal imaging image respectively to generate visual image features and thermal imaging features. The process parameter feature extraction module extracts features from the process parameter sequence to generate parameter features. The feature fusion module based on the cross-modal attention mechanism fuses the visual image features, the thermal imaging features and the parameter features to generate fused features.

2. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 1, characterized in that, Obtaining the training set includes: The original resistance spot welding data is acquired. The image data of the original resistance spot welding data is enhanced with a contrast-limited adaptive histogram equalization algorithm. The image data is divided into non-overlapping sub-blocks. Histogram equalization is performed independently on each sub-block. The noise amplification is avoided by limiting the histogram clipping threshold. Finally, bilinear interpolation is used to eliminate the boundary artifacts of the sub-blocks. Guided filtering and sharpening operations are used to fuse image details. Furthermore, mosaic random stitching technology is introduced to expand the dataset. The raw resistance spot welding data after image data processing is subjected to box plot detection and outlier removal. Then, Gaussian filtering algorithm is used to smooth the two quality indicators of tensile test strength and weld nugget diameter. All data is then uniformly scaled to the target range to obtain the training set.

3. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 1, characterized in that, The image feature extraction module includes: multiple image feature extraction units; Each image feature extraction unit includes: a convolutional layer, a pooling layer, and a fully connected layer.

4. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 3, characterized in that, Generating the visual image features and the thermal imaging features includes: After the visual image and the thermal image are sequentially passed through the convolutional layer for effective filling convolution, the feature distribution is stabilized by batch normalization, and a nonlinear transformation is introduced by the ReLU activation function. Then, the feature is reduced in dimensionality by the pooling layer to generate a three-dimensional feature map. The three-dimensional feature map is transformed into a one-dimensional vector, which is then mapped to the feature space of the target dimension through the fully connected layer. ReLU activation is then used to enhance the non-linear expressive power of the features, and the output result is obtained. The output results corresponding to each image feature extraction unit are stabilized by layer normalization to generate the visual image features and the thermal imaging features.

5. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 1, characterized in that, Generating the parameter features includes: The process parameter sequence is input into the process parameter feature extraction module. The LSTM unit in the process parameter feature extraction module processes the forward and reverse information of the time series respectively, and fuses the bidirectional hidden states to capture the long-term dependence and short-term fluctuations in the sequence, thereby obtaining the parameter features.

6. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 1, characterized in that, Generating the fusion feature includes: The visible image features and the thermal imaging features are used as query pairs and key-value pairs, respectively. The feature weights are dynamically adjusted through a cross-modal attention scoring matrix, and the first image fusion feature is generated by weighted summation. The visible image features and the thermal imaging features are used as key-value pairs and query pairs, respectively. The feature weights are dynamically adjusted by calculating the cross-modal attention score matrix, and a second image fusion feature is generated by weighted summation. The first image fusion feature and the second image fusion feature are weighted and fused to obtain the final image fusion feature; The final image fusion features are reshaped into a two-dimensional spatial feature map. The parameter features are mapped to attention query vectors through linear transformation. At the same time, the spatial features of the two-dimensional spatial feature map are mapped to key vectors and value vectors. The parameter and image attention matrix are calculated, and the parameter-guided image features are obtained by weighted summation. The fusion features are then generated by combining the parameter features.

7. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 6, characterized in that, Dynamic adjustment of feature weights is achieved by calculating the cross-modal attention score matrix: ； in, , For learnable weight matrix, The dimension of the key vector.

8. The method for predicting and optimizing resistance spot welding quality based on the fusion of multimodal data and cross-modal attention network according to claim 6, characterized in that, Obtaining the final image fusion features includes: ； in, For the final image fusion features, For visual image features, For the second image fusion feature, For thermal imaging features, This is the first image fusion feature.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

Images